Mathematical Sciences-Colloquium-Sophie Litschwartz (Harvard Graduate School of Education) "Correcting Bias in Test Score Distributions Due to Rescoring and Retesting Policies"

Monday, December 16, 2019
11:00 am

Location:

Floor/Room #: 
203

Speaker: Sophie Litschwartz (Harvard Graduate School of Education)

Title: Correcting Bias in Test Score Distributions Due to Rescoring and Retesting Policies

Abstract: ​Pass/fail exams frequently have testing policies that either rescore or retest only initially
failing students. This asymmetric treatment of passing and failing students distorts the test score
distribution and has the potential to bias secondary analysis of test scores if not adequately
accounted for. In this study, I develop a method for adjusting test score analysis to account for
rescoring and retesting policies. I combine Classical Test Theory and simulation to model directly how the combination of rescoring/retesting policies and measurement error affects test score distributions. This model provides a basis for adjusting the estimates of different quantities of interest and prediction intervals for these adjustments. More broadly, the method I develop in
this paper allows us: 1) to visualize the discontinuous distributions that result from
rescoring/retesting; 2) quantify the effect of rescoring/retesting policies and measurement error on parameters of interest; 3) test for real changes in quantities of interest (e.g. pass rate) in the case of retesting; 4) test for scoring irregularities in the case of rescoring. Using simulated data of a pass/fail exam, I demonstrate how retesting affects interpretations of passing rates and standards. I also use this method to reexamine an investigation of “score scrubbing” (i.e. where teachers were believed to be manipulating the grading to put near passing students just over the pass threshold) on the New York Regent Exam. I show that rescoring policies can inflate perceived scrubbing rates by a factor of three, from 12% to 36%.