View Procedure
Summative Assessment Procedure for Phases 1-3
Definitions
Assessment plan — A detailed description of how learning will be assessed following a period of instruction.
Borderline-regression method — The borderline-regression method is an examinee-centred approach commonly used to determine the pass score for an OSCE station. Examinees receive a numerical score and global grading of performance for each station. The numerical scores are regressed on grades and the pass score for the station is set as the predicted numerical score corresponding to the borderline grade.
Burr Method — A standard-setting method used to determine the pass mark for an assessment. Using this approach, the upper limit of the pass mark is set to the median percent-correct score of the class minus 10%, or 70%, whichever is lower. The pass mark is identified by applying this upper limit to the cumulative frequency curve of the achieved percent-correct scores of the class as shown in Appendix A.
Course Assessment Report — A report on the assessment results for a given course. The report includes the methods of formative and Summative Assessment, a statistical summary of the class results for all Summative Assessment methods, comparison of the mean course score with those from previous years, a summary of item analysis results for any MCQ examinations, and a summary of learner feedback on assessment from course evaluation reports.
Examination blueprint — A listing of the learning objectives included in an examination, together with the weighting of the content areas on the overall examination
Objective Structured Clinical Examination (OSCE) — An examination used to assess a learner’s clinical competence in a simulation-based setting.
Phase 1-3 — The first two (2) years of the Doctor of Medicine (MD) program.
Phase Assessment Lead — A faculty member of the Phase Management Team with responsibility for assessment implementation in the Phase.
Phase Assessment Working Group (PAWG) — A working group of the Phase Management Teams. The PAWG manages assessment implementation for the Phase and regularly reviews formative and summative items and examinations. They consist of Phase Assessment Lead, the education specialist for assessment, and the instructional designer assigned to the phase.
Rubric — A scoring tool that explicitly represents the performance expectations for an assignment or other form of assessment.
Student Assessment Subcommittee (SAS) — A subcommittee of the Undergraduate Medical Studies (UGMS) committee. The SAS evaluates, monitors, and advises the UGMS on student assessment, assessment-related policy and assessment implementation for the curriculum leading to the MD degree.
Summative Assessment — An assessment of learner achievement at the end of a period of instruction.
Undergraduate Medical Studies (UGMS) Committee — A standing committee of Faculty Council, that has responsibility for the design, management, integration, evaluation, and enhancement of the curriculum for the MD program.
Written Examination — An examination used for Summative Assessment, which utilizes multiple-choice questions or other item formats.
Procedure
Overview
The Committee on the Accreditation of Canadian Medical Schools (CACMS) Element 9.8 requires that “A medical school has a policy and process in place for ensuring timely summative assessment of medical student achievement in each required learning experience.”
The exam review process described here is distinct from (and not intended to replace or contravene) General University Regulations 6.8.3 and 6.8.4 governing access to and request for rereading of final examination scripts.
Purpose
This procedure outlines the approach to Summative Assessment for all Phase 1-3 courses in the MD curriculum.
Scope
Courses in Phases 1-3 of the MD Program.
Procedure
A.0 Assessment Planning
A.1. The Assessment Plan for each course must provide the method(s) of assessment, outline each assessment’s contribution to the final course grade, the examination schedule and due dates for any assignments or other course work.
A.2. The Assessment Plan must specify the criteria for passing the course or rotation and the conditions for required re-assessment (see Section J).
B.0 Assessment Oversight
B.1. Prior to the administration of any written assignments, the instructor or relevant Phase Lead must file a Rubric with the Student Assessment Subcommittee (SAS) for all written summative assignments. The Student Assessment Subcommittee (SAS) annually completes a quality assurance review of all Rubrics.
B.2. After the administration of assessments, the SAS will review the Examination Blueprint for each summative multiple-choice question (MCQ) examination.
B.3. The Clinical Skills Lead provides a blueprint for the OSCE to SAS.
C.0 Administration of Written Examinations
C.1. All internal summative Written Examinations will be administered by the UGME office. This includes examination delivery, collation of learner scores and final reporting of scores to learners.
C.2. All summative Written Examinations are conducted under supervised conditions as described in the Exam Invigilation Procedure.
C.3. All summative Written Examinations for Phases 1-3 will include an opportunity for learners to submit written requests for the SAS/PAWG to review the content or format of specific questions (e.g. using challenge cards).
D.0 Deferral of Assessments
D.1. Learners seeking to defer a summative written examination or other assessment must follow the Undergraduate Medical Education Deferred Examination Policy.
E.0 Mark/Grade Release
E.1. Individual assignment marks and grades must be released by the Assessment Academic Program Assistant (APA) in the UGME office to the learners via One45. Marks and grades are not to be released directly to learners by faculty.
E. 2. Final numerical marks for each Summative Assessment are provided to learners via One45. Additional assessment statistics (e.g. the class mean) may be included to provide learners with further context on their performance.
F.0 Questions for these Examinations
F.1. Faculty members provide exam questions through the following processes:
All faculty members are required to provide two formative exam questions, two summative exam questions and one reassessment exam question per hour of instruction.
a. Faculty members will be directed to submit their questions prior to the start of the theme.
b. Questions may be drawn from the summative and formative item banks or new items may be created.
c. Formative questions may not be used on summative exams.
d. Questions must be linked to session objectives and conform to the guidelines of the Phase Assessment Working Group (PAWG), which are based on the Medical Council of Canada (MCC) and National Board of Medical Examiners (NBME) guidelines.
F.1.1. The Associate Dean, UGME, is notified of the faculty members who have not submitted questions within five working days of the initial notification. The UGME team contacts faculty members regarding question submission.
F.1.2. If questions have not been submitted seven working days before the examination, the Phase Assessment Lead contacts individual faculty members regarding question submission.
F.1.3. If a faculty member does not provide a sufficient number of exam questions and/or does not provide question revisions as requested by the PAWG (see F.2.2) for two consecutive academic years, the Associate Dean, UGME, will follow-up with the faculty member and the Discipline Chair/Associate Dean, as needed.
F.2. Examination questions are reviewed by the appropriate PAWG through the following process:
F.2.1. The PAWG reviews questions to ensure that they meet recommended guidelines.
F.2.2. The PAWG contacts faculty members about any questions that do not meet recommended guidelines.
F.2.3. The PAWG selects questions for formative, summative and reassessment examinations.
F.3. The Instructional Designers compile examination questions electronically using the exam software as directed by PAWG.
F.4. The Instructional Designers test the exam for proper functioning prior to delivery.
G.0 Standard setting for examinations;
G.1. The pass mark for Summative Assessments is 70%. The pass mark for summative Written Examinations is determined by the Burr method if one or more learners achieve a percent-correct score of less than 70% on the written examination.
G.2. Standard setting process for Summative Written Examinations:
G.2.1. All learners achieving a percent-correct score of less than 70% on an examination are notified the day of the exam that they might be required to write a reassessment examination.
G.2.2. After the review of the examination items and scoring is complete and any credits are awarded in the examination, the pass mark for the summative written examination is determined using the Burr method.
G.2.3. After the final pass score is determined, learners achieving less than the final pass mark, as determined by the Burr method, are notified that they are required to write the reassessment examination.
G.3. Standard setting process for the OSCE:
G.3.1 The pass score for each OSCE station is determined using the Borderline-Regression Method. The overall pass score for the OSCE examination is set as the mean of the individual station pass scores.
G.3.2 The criteria for determining the OSCE grade of competent, based on the pass scores for each station and the OSCE overall, are outlined in the course assessment plan.
H.0 Review of Examination Items and Scoring
H.1 The PAWG, will undertake a post-examination review prior to the final determination of learner scores.
H.1.1 Statistical item analysis (where available) and requests for review of specific questions (i.e., challenge cards) are taken into consideration to determine the answer key.
H.1.2 Final decisions for crediting/re-scoring are made by the PAWG in consultation, as appropriate, with the faculty member responsible for the construction of the original assessment item, other content experts and/or specialists in construction of examination questions.
H1.3 All scoring changes for individual examination questions are applied to the whole class.
H.2 A learner may review a summative written examination through the following process:
H.2.1 The examination review request is made through the UGME office. Learners who failed the examination are given priority to review examinations.
H.2.2 The examination review occurs in a meeting of the learner with an APA of the UGME office. In this meeting:
i. The learner may request and receive a copy of an individualized Coaching Report that provides a listing of the questions that the learner answered incorrectly on the examination.
ii. The learner’s incorrect response is provided.
iii. The correct answer is not provided.
H.2.3 The learner reviews the Coaching Report in the presence of the APA.
i. The learner is not permitted to record or transmit any information during this review process.
ii. A learner may be allowed to use a personal computer to view a presentation stored on D2L, but this computer use is only permitted under the supervision of the APA.
H.2.4 Once the Coaching Report is reviewed, the entire Coaching Report is returned to the APA.
i. The learner is given a copy of the Coaching Report that contains a listing of the Blueprint ID (e.g., numbered learning objective) of each question incorrectly answered.
ii. No questions, choices or answers appear on this Coaching Report.
iii. The learner may retain this Coaching Report.
H.3 A learner may request to retain a copy of the Coaching Report that contains a listing of the Blueprint ID (e.g., numbered learning objective) for each question answered incorrectly from the UGME Assessment APA.
H.4 Learners seeking reconsideration of the assessment/reassessment result or a reread of a written examination or paper must follow the procedure outlined in the University Calendar.
I.0 Summative Reassessment
I.1. Reassessment is required if a learner achieves less than 70% (or the pass mark determined by the Burr method in the case of summative Written Examinations, whichever is lower) on any Summative Assessment.
I.2. Reassessment is required if a learner does not achieve a grade of competent in the OSCE or other Summative Assessment in the Clinical Skills course.
I.3. A learner may be reassessed only once for any failed assessment.
I.4. The pass mark for the reassessment will be 70%, or the pass mark determined by the Burr method in the case of Written Examinations.
I.5. For the OSCE, the pass mark determined through the Borderline-Regression Method will apply for the reassessment.
I.6. Reassessment for a written examination is a comparable written examination.
I.6.1. Reassessment examinations include at least 40% new items.
I.6.2. The reassessment examination is scheduled for at least one week after the original examination.
I.7. Appropriate reassessment for written assignments and other summative assessments is determined by the faculty member or course lead who set the original assessment.
I.8. In cases where a learner scores lower on the reassessment than on the original Summative Assessment, the higher mark will stand and be used in calculations of course marks and grades.
J.0 Grading
J.1. Final grades in all courses are determined using the mechanism detailed in the relevant course Assessment Plan.
J.2. If the Burr method is used to determine the pass mark for a summative written examination, the percent-correct score for each learner will be converted to a scaled score. For the scaled score, the pass mark as determined by the Burr method is set to 70 and percent-correct scores are converted using linear transformation. Scaled scores will then be used to calculate the overall course mark and final grade.
K.0 Program Evaluation
K.1. At the end of a course, the Education Specialist for Assessment compiles a Course Assessment Report that is reviewed by the SAS.
K.2. The SAS will provide the Course Assessment Report to the Phase Lead with a copy to the Program Evaluation Subcommittee (PESC).
K.3. The Phase Lead will respond to the SAS within one month and provides an action plan for any items requiring review.
K.4. The SAS reviews the response by the Phase Lead and makes recommendations to the UGMS based on the action plan, including possible changes to the Assessment Plan for the next academic year.
K.4.1. The SAS may recommend a formal review of the assessment plans and methods to the UGMS. This formal review would normally be conducted by the Program Evaluation Subcommittee.
Related Links
Summative Assessment Policy Phases 1-4
Examination Invigilation Procedure