Investigation the Effects of the Raters' Qualifications on Determining Cutoff Scores with Generalizability Theory

Abstract

Rater group, who give judgments about cutting scores, is the most important factor of the standard setting procedures. Reliability of the cutoff score(s), will be directly affected by the consistency of the raters' judgments. In this study, performance of school teachers and domain experts were compared within the scope of the Generalizability Theory during the standard setting procedure. With this comparison, it was aimed to demonstrate the effect of rater qualifications on cutoff scores. 1-0 and Nedelsky methods which are two of the student centered standard setting methods were adopted in this study. The data that had been collected from raters was analyzed with single and multiple facet crossed designs. According to results of the analyses that were conducted under different designs, it was seen that domain experts gave more consistent judgments compared to school teachers; but this distinction was not so notable. More detailed findings such as G and Phi coefficients of decision studies were also covered in the subsequent sections.