Example of Create Attribute Agreement Analysis Worksheet

An educational testing company hired five appraisers to evaluate the written portion of a standardized essay test. To complete the training, each appraiser must be able to rate essays consistently with standard ratings. Each appraiser rates 15 essays that have a known standard rating.

The test engineer wants to create a data collection worksheet to collect the ratings from the appraisers in random order.

Note

The sample size of 15 essays is too small for a proper attribute agreement analysis, but is suitable to illustrate how to create a data collection worksheet in Minitab.

Choose Stat > Quality Tools > Create Attribute Agreement Analysis Worksheet.
From the drop-down list, select Sample standard/attribute in numeric values.
In Number of samples, enter 15.
Under Numeric Standard, type a number into each cell, 2, -1, 0,-2, 0, 1, 2, 0, -1, 1,-2, 0, 2, -1, and 1.
In Number of appraisers, enter 5.
Under Appraiser Name, type a name into each cell: Duncan, Hayes, Holmes, Montgomery, and Simpson.
In Number of replicates, enter 1.
Click OK.

Results

The 5 appraisers assess 15 essays one time, for a total of 75 assessments. The run order lists the order in which the assessments should be made. Duncan rates sample 2, then rates sample 13. By default, runs are randomized within appraisers, so Appraiser 1 assesses all the samples first, Appraiser 2 assess them second, and so on. Hayes starts his assessments with run 16. Assessment values may be recorded in C4 and compared to the standards in C5.

If you perform the example steps, the data collection order in your worksheet will be different than the order in this worksheet, because the order for each worksheet is random. If you want to generate the same order more than one time, set the base of the random number generator.

C1	C2-T	C3-T	C4	C5
RunOrder	Samples	Appraisers	Assessments	Standards
1	2	Duncan		-1
2	13	Duncan		2
3	8	Duncan		0
4	10	Duncan		1
5	1	Duncan		2
6	14	Duncan		-1
…	…	…		…