An educational testing company hired five appraisers to evaluate the written portion of a standardized essay test. To complete the training, each appraiser must be able to rate essays consistently with standard ratings. Each appraiser rates 15 essays that have a known standard rating.
The sample size of 15 essays is too small for a proper attribute agreement analysis, but is suitable to illustrate how to create a data collection worksheet in Minitab.
The 5 appraisers assess 15 essays one time, for a total of 75 assessments. The run order lists the order in which the assessments should be made. Duncan rates sample 2, then rates sample 13. By default, runs are randomized within appraisers, so Appraiser 1 assesses all the samples first, Appraiser 2 assess them second, and so on. Hayes starts his assessments with run 16. Assessment values may be recorded in C4 and compared to the standards in C5.