Validity and reliability of portfolio assessment of student competence in two dental school populations: A four-Year study

Cynthia C. Gadbury-Amyot, Michael Scott McCracken, Janet L. Woldt, Robert L. Brennan

Research output: Contribution to journalArticle

15 Citations (Scopus)

Abstract

The purpose of this study was to empirically investigate the validity and reliability of portfolio assessment in two U.S. dental schools using a unified framework for validity. In the process of validation, it is not the test that is validated but rather the claims (interpretations and uses) about test scores that are validated. Kane's argument-based validation framework provided the structure for reporting results where validity claims are followed by evidence to support the argument. This multivariate generalizability theory study found that the greatest source of variance was attributable to faculty raters, suggesting that portfolio assessment would benefit from two raters' evaluating each portfolio independently. The results are generally supportive of holistic scoring, but analytical scoring deserves further research. Correlational analyses between student portfolios and traditional measures of student competence and readiness for licensure resulted in significant correlations between portfolios and National Board Dental Examination Part I (r=0.323, p<0.01) and Part II scores (r=0.268, p<0.05) and small and non-significant correlations with grade point average and scores on the Western Regional Examining Board (WREB) exam. It is incumbent upon the users of portfolio assessment to determine if the claims and evidence arguments set forth in this study support the proposed claims for and decisions about portfolio assessment in their respective institutions.

Original languageEnglish (US)
Pages (from-to)657-667
Number of pages11
JournalJournal of dental education
Volume78
Issue number5
StatePublished - May 1 2014

Fingerprint

Dental Schools
Reproducibility of Results
Mental Competency
Students
Licensure
school
Population
Tooth
student
Research
evidence
examination
interpretation

Keywords

  • Assessment
  • Competency-based education
  • Critical thinking
  • Dental education
  • Educational measurements
  • Multivariate generalizability theory
  • Portfolio assessment
  • Problem-solving
  • Self-assessment

ASJC Scopus subject areas

  • Education
  • Dentistry(all)

Cite this

Validity and reliability of portfolio assessment of student competence in two dental school populations : A four-Year study. / Gadbury-Amyot, Cynthia C.; McCracken, Michael Scott; Woldt, Janet L.; Brennan, Robert L.

In: Journal of dental education, Vol. 78, No. 5, 01.05.2014, p. 657-667.

Research output: Contribution to journalArticle

Gadbury-Amyot, Cynthia C. ; McCracken, Michael Scott ; Woldt, Janet L. ; Brennan, Robert L. / Validity and reliability of portfolio assessment of student competence in two dental school populations : A four-Year study. In: Journal of dental education. 2014 ; Vol. 78, No. 5. pp. 657-667.
@article{e8d53073542746938e9fcb80077012f3,
title = "Validity and reliability of portfolio assessment of student competence in two dental school populations: A four-Year study",
abstract = "The purpose of this study was to empirically investigate the validity and reliability of portfolio assessment in two U.S. dental schools using a unified framework for validity. In the process of validation, it is not the test that is validated but rather the claims (interpretations and uses) about test scores that are validated. Kane's argument-based validation framework provided the structure for reporting results where validity claims are followed by evidence to support the argument. This multivariate generalizability theory study found that the greatest source of variance was attributable to faculty raters, suggesting that portfolio assessment would benefit from two raters' evaluating each portfolio independently. The results are generally supportive of holistic scoring, but analytical scoring deserves further research. Correlational analyses between student portfolios and traditional measures of student competence and readiness for licensure resulted in significant correlations between portfolios and National Board Dental Examination Part I (r=0.323, p<0.01) and Part II scores (r=0.268, p<0.05) and small and non-significant correlations with grade point average and scores on the Western Regional Examining Board (WREB) exam. It is incumbent upon the users of portfolio assessment to determine if the claims and evidence arguments set forth in this study support the proposed claims for and decisions about portfolio assessment in their respective institutions.",
keywords = "Assessment, Competency-based education, Critical thinking, Dental education, Educational measurements, Multivariate generalizability theory, Portfolio assessment, Problem-solving, Self-assessment",
author = "Gadbury-Amyot, {Cynthia C.} and McCracken, {Michael Scott} and Woldt, {Janet L.} and Brennan, {Robert L.}",
year = "2014",
month = "5",
day = "1",
language = "English (US)",
volume = "78",
pages = "657--667",
journal = "Journal of Dental Education",
issn = "0022-0337",
publisher = "American Dental Education Association",
number = "5",

}

TY - JOUR

T1 - Validity and reliability of portfolio assessment of student competence in two dental school populations

T2 - A four-Year study

AU - Gadbury-Amyot, Cynthia C.

AU - McCracken, Michael Scott

AU - Woldt, Janet L.

AU - Brennan, Robert L.

PY - 2014/5/1

Y1 - 2014/5/1

N2 - The purpose of this study was to empirically investigate the validity and reliability of portfolio assessment in two U.S. dental schools using a unified framework for validity. In the process of validation, it is not the test that is validated but rather the claims (interpretations and uses) about test scores that are validated. Kane's argument-based validation framework provided the structure for reporting results where validity claims are followed by evidence to support the argument. This multivariate generalizability theory study found that the greatest source of variance was attributable to faculty raters, suggesting that portfolio assessment would benefit from two raters' evaluating each portfolio independently. The results are generally supportive of holistic scoring, but analytical scoring deserves further research. Correlational analyses between student portfolios and traditional measures of student competence and readiness for licensure resulted in significant correlations between portfolios and National Board Dental Examination Part I (r=0.323, p<0.01) and Part II scores (r=0.268, p<0.05) and small and non-significant correlations with grade point average and scores on the Western Regional Examining Board (WREB) exam. It is incumbent upon the users of portfolio assessment to determine if the claims and evidence arguments set forth in this study support the proposed claims for and decisions about portfolio assessment in their respective institutions.

AB - The purpose of this study was to empirically investigate the validity and reliability of portfolio assessment in two U.S. dental schools using a unified framework for validity. In the process of validation, it is not the test that is validated but rather the claims (interpretations and uses) about test scores that are validated. Kane's argument-based validation framework provided the structure for reporting results where validity claims are followed by evidence to support the argument. This multivariate generalizability theory study found that the greatest source of variance was attributable to faculty raters, suggesting that portfolio assessment would benefit from two raters' evaluating each portfolio independently. The results are generally supportive of holistic scoring, but analytical scoring deserves further research. Correlational analyses between student portfolios and traditional measures of student competence and readiness for licensure resulted in significant correlations between portfolios and National Board Dental Examination Part I (r=0.323, p<0.01) and Part II scores (r=0.268, p<0.05) and small and non-significant correlations with grade point average and scores on the Western Regional Examining Board (WREB) exam. It is incumbent upon the users of portfolio assessment to determine if the claims and evidence arguments set forth in this study support the proposed claims for and decisions about portfolio assessment in their respective institutions.

KW - Assessment

KW - Competency-based education

KW - Critical thinking

KW - Dental education

KW - Educational measurements

KW - Multivariate generalizability theory

KW - Portfolio assessment

KW - Problem-solving

KW - Self-assessment

UR - http://www.scopus.com/inward/record.url?scp=84900411522&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84900411522&partnerID=8YFLogxK

M3 - Article

C2 - 24789826

AN - SCOPUS:84900411522

VL - 78

SP - 657

EP - 667

JO - Journal of Dental Education

JF - Journal of Dental Education

SN - 0022-0337

IS - 5

ER -