Use of a committee review process to improve the quality of course examinations

P. M. Wallach, L. M. Crespo, K. Z. Holtzman, R. M. Galbraith, D. B. Swanson

Research output: Contribution to journalReview article

30 Citations (Scopus)

Abstract

Purpose: In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. Methods: Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. Results: The mean rating for items from 2000 to 2001 was 2.51 ± 1.27; analogous values for 2001-2002 were 3.16 ± 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 ± 1.15 (t = 10.11; p<0.0001). Conclusion: Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.

Original languageEnglish (US)
Pages (from-to)61-68
Number of pages8
JournalAdvances in Health Sciences Education
Volume11
Issue number1
DOIs
StatePublished - Feb 1 2006

Fingerprint

Coroners and Medical Examiners
Advisory Committees
Guidelines
examination
medical examiner
Medical Staff
director
rating scale
experience
rating
staff
Values
Group

Keywords

  • Assessment
  • Curricular reform
  • Integrated exams
  • Multiple-choice questions
  • Quality assessment score

ASJC Scopus subject areas

  • Nursing(all)
  • Education

Cite this

Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2006). Use of a committee review process to improve the quality of course examinations. Advances in Health Sciences Education, 11(1), 61-68. https://doi.org/10.1007/s10459-004-7515-8

Use of a committee review process to improve the quality of course examinations. / Wallach, P. M.; Crespo, L. M.; Holtzman, K. Z.; Galbraith, R. M.; Swanson, D. B.

In: Advances in Health Sciences Education, Vol. 11, No. 1, 01.02.2006, p. 61-68.

Research output: Contribution to journalReview article

Wallach, PM, Crespo, LM, Holtzman, KZ, Galbraith, RM & Swanson, DB 2006, 'Use of a committee review process to improve the quality of course examinations', Advances in Health Sciences Education, vol. 11, no. 1, pp. 61-68. https://doi.org/10.1007/s10459-004-7515-8
Wallach, P. M. ; Crespo, L. M. ; Holtzman, K. Z. ; Galbraith, R. M. ; Swanson, D. B. / Use of a committee review process to improve the quality of course examinations. In: Advances in Health Sciences Education. 2006 ; Vol. 11, No. 1. pp. 61-68.
@article{026c45f01a1441e7b531626d2aa17c27,
title = "Use of a committee review process to improve the quality of course examinations",
abstract = "Purpose: In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. Methods: Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. Results: The mean rating for items from 2000 to 2001 was 2.51 ± 1.27; analogous values for 2001-2002 were 3.16 ± 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 ± 1.15 (t = 10.11; p<0.0001). Conclusion: Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.",
keywords = "Assessment, Curricular reform, Integrated exams, Multiple-choice questions, Quality assessment score",
author = "Wallach, {P. M.} and Crespo, {L. M.} and Holtzman, {K. Z.} and Galbraith, {R. M.} and Swanson, {D. B.}",
year = "2006",
month = "2",
day = "1",
doi = "10.1007/s10459-004-7515-8",
language = "English (US)",
volume = "11",
pages = "61--68",
journal = "Advances in Health Sciences Education",
issn = "1382-4996",
publisher = "Springer Netherlands",
number = "1",

}

TY - JOUR

T1 - Use of a committee review process to improve the quality of course examinations

AU - Wallach, P. M.

AU - Crespo, L. M.

AU - Holtzman, K. Z.

AU - Galbraith, R. M.

AU - Swanson, D. B.

PY - 2006/2/1

Y1 - 2006/2/1

N2 - Purpose: In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. Methods: Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. Results: The mean rating for items from 2000 to 2001 was 2.51 ± 1.27; analogous values for 2001-2002 were 3.16 ± 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 ± 1.15 (t = 10.11; p<0.0001). Conclusion: Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.

AB - Purpose: In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. Methods: Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. Results: The mean rating for items from 2000 to 2001 was 2.51 ± 1.27; analogous values for 2001-2002 were 3.16 ± 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 ± 1.15 (t = 10.11; p<0.0001). Conclusion: Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.

KW - Assessment

KW - Curricular reform

KW - Integrated exams

KW - Multiple-choice questions

KW - Quality assessment score

UR - http://www.scopus.com/inward/record.url?scp=33645467642&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33645467642&partnerID=8YFLogxK

U2 - 10.1007/s10459-004-7515-8

DO - 10.1007/s10459-004-7515-8

M3 - Review article

VL - 11

SP - 61

EP - 68

JO - Advances in Health Sciences Education

JF - Advances in Health Sciences Education

SN - 1382-4996

IS - 1

ER -