An educational testing company hired five appraisers to evaluate the written portion of a standardized essay test. To complete the training, each appraiser must be able to rate essays consistently with standard ratings. Each appraiser rates 52 essays that have a known standard rating.
You can use this data to demonstrate Attribute Agreement Analysis.Because the data include a known standard for each sample, the test engineer can assess the consistency and correctness of ratings compared to the standard as well as compared with other appraisers.
Worksheet column | Description |
---|---|
Appraiser | The name of the appraiser evaluating the essay: Simpson, Montgomery, Holmes, Duncan, or Hayes |
Sample | The essay (sample) identifier: 1 to 52 |
Rating | The appraiser's rating for the sample essay: -2, -1, 0, 1, or 2 |
Standard | The known standard rating for the sample essay: -2, -1, 0, 1, or 2 |