Example of Attribute Agreement Analysis

Fabric appraisers at a textile printing company rate the print quality of cotton fabric on a 1 to 5 point scale. The quality engineer wants to assess the consistency and correctness of the appraisers' ratings. The engineer asks four appraisers to rate print quality on 50 samples of fabric twice, in random order.

Because the data include a known standard for each sample, the quality engineer can assess the consistency and correctness of ratings compared to the standard as well as compared to other appraisers.

Open the sample data, TextilePrintQuality.MWX.
Choose Stat > Quality Tools > Attribute Agreement Analysis.
In Data are arranged as, select Attribute column and enter Response.
In Samples, enter Sample.
In Appraisers, enter Appraiser.
In Known standard/attribute, enter Standard.
Select Categories of the attribute data are ordered.
Click OK.

Interpret the results

Within Appraisers table: Because each appraiser provides two or more ratings for each sample, the engineer can evaluate the consistency of each appraiser.; All of the appraisers have good match rates, from Amanda with 100% to Eric with 86%.; The p-value for Fleiss' kappa statistics is 0.0000 for all appraisers and all responses, with α = 0.05. Therefore, the engineer rejects the null hypothesis that the agreement is due to chance alone.; Because this example has ordinal ratings, the engineer examines Kendall's coefficient of concordance. The Kendall's coefficient of concordance for all appraisers ranges between 0.98446 and 1.000, which indicates a high level of agreement.
Each Appraiser vs Standard table: Because there is a known standard for each sample, the engineer can evaluate the accuracy and consistency of each of the appraiser's ratings.; Each appraiser rated 50 fabric samples (# Inspected). Amanda correctly judged 47 samples across trials (#Matched), for 94% matched. Eric correctly judged 41 samples across trials, for 82% matched.; The p-value for Fleiss' kappa is 0.0000 for all appraisers and all responses, with α = 0.05. Therefore, the engineer rejects the null hypothesis that the agreement is due to chance alone.; The Kendall's correlation coefficient for all appraisers ranges between 0.951863 and 0.975168, which confirms the high level of agreement with the standard.
Between Appraisers table: The Between Appraisers table shows that the appraisers agree on their ratings for 37 of the 50 samples.; The overall kappa value of 0.881705 indicates a good level of absolute agreement of the ratings between appraisers. The Kendall's coefficient of concordance of 0.976681 confirms this strong association.; The between-appraisers statistics do not compare the appraisers' ratings to the standard. Although the appraisers' ratings may be consistent, these statistics do not indicate whether the ratings are correct.
All Appraisers vs Standard table: Because there is a known standard for each sample, the engineer can evaluate the accuracy of all the appraisers' ratings.; The appraisers matched 37 of 50 ratings of all appraiser assessments with the known standard, for 74.0% matched.; The overall kappa value of 0.912082 indicates a good level of absolute agreement of the ratings between appraisers and with the standard. The Kendall's coefficient of concordance of 0.965563 confirms this strong association.; The all appraisers versus standard statistics do compare the appraisers' ratings to the standard. The engineer can conclude that the appraisers' ratings are consistent and correct.

Note

The p-value of 0.0000 in the output is rounded. But you can safely conclude that the p-value is very low and is < 0.00005.

Appraiser	# Inspected	# Matched	Percent	95% CI
Amanda	50	50	100.00	(94.18, 100.00)
Britt	50	48	96.00	(86.29, 99.51)
Eric	50	43	86.00	(73.26, 94.18)
Mike	50	45	90.00	(78.19, 96.67)

Appraiser	Response	Kappa	SE Kappa	Z
Amanda	1	1.00000	0.141421	7.0711
	2	1.00000	0.141421	7.0711
	3	1.00000	0.141421	7.0711
	4	1.00000	0.141421	7.0711
	5	1.00000	0.141421	7.0711
	Overall	1.00000	0.071052	14.0741
Britt	1	1.00000	0.141421	7.0711
	2	0.89605	0.141421	6.3360
	3	0.86450	0.141421	6.1129
	4	1.00000	0.141421	7.0711
	5	1.00000	0.141421	7.0711
	Overall	0.94965	0.071401	13.3002
Eric	1	0.83060	0.141421	5.8733
	2	0.84000	0.141421	5.9397
	3	0.70238	0.141421	4.9666
	4	0.70238	0.141421	4.9666
	5	1.00000	0.141421	7.0711
	Overall	0.82354	0.071591	11.5034
Mike	1	1.00000	0.141421	7.0711
	2	0.83060	0.141421	5.8733
	3	0.81917	0.141421	5.7924
	4	0.86450	0.141421	6.1129
	5	0.86450	0.141421	6.1129
	Overall	0.87472	0.070945	12.3295

Appraiser	Coef	Chi - Sq	DF	P
Amanda	1.00000	98.0000	49	0.0000
Britt	0.99448	97.4587	49	0.0000
Eric	0.98446	96.4769	49	0.0001
Mike	0.98700	96.7256	49	0.0001

Appraiser	# Inspected	# Matched	Percent	95% CI
Amanda	50	47	94.00	(83.45, 98.75)
Britt	50	46	92.00	(80.77, 97.78)
Eric	50	41	82.00	(68.56, 91.42)
Mike	50	45	90.00	(78.19, 96.67)

Appraiser	Response	Kappa	SE Kappa	Z
Amanda	1	1.00000	0.100000	10.0000
	2	0.83060	0.100000	8.3060
	3	0.81917	0.100000	8.1917
	4	1.00000	0.100000	10.0000
	5	1.00000	0.100000	10.0000
	Overall	0.92476	0.050257	18.4006
Britt	1	1.00000	0.100000	10.0000
	2	0.83838	0.100000	8.3838
	3	0.80725	0.100000	8.0725
	4	1.00000	0.100000	10.0000
	5	1.00000	0.100000	10.0000
	Overall	0.92462	0.050396	18.3473
Eric	1	0.91159	0.100000	9.1159
	2	0.81035	0.100000	8.1035
	3	0.72619	0.100000	7.2619
	4	0.84919	0.100000	8.4919
	5	1.00000	0.100000	10.0000
	Overall	0.86163	0.050500	17.0622
Mike	1	1.00000	0.100000	10.0000
	2	0.91694	0.100000	9.1694
	3	0.90736	0.100000	9.0736
	4	0.92913	0.100000	9.2913
	5	0.93502	0.100000	9.3502
	Overall	0.93732	0.050211	18.6674

Example of Attribute Agreement Analysis

Interpret the results

Note

Assessment Agreement

Fleiss’ Kappa Statistics

Kendall’s Coefficient of Concordance

Assessment Agreement

Fleiss’ Kappa Statistics

Kendall’s Correlation Coefficient

Assessment Agreement

Fleiss’ Kappa Statistics

Kendall’s Coefficient of Concordance

Assessment Agreement

Fleiss’ Kappa Statistics

Kendall’s Correlation Coefficient

Appraiser	Coef	SE Coef	Z
Amanda	0.967386	0.0690066	14.0128
Britt	0.967835	0.0690066	14.0193
Eric	0.951863	0.0690066	13.7879
Mike	0.975168	0.0690066	14.1256

Response	Kappa	SE Kappa	Z
1	0.954392	0.0267261	35.7101
2	0.827694	0.0267261	30.9695
3	0.772541	0.0267261	28.9058
4	0.891127	0.0267261	33.3429
5	0.968148	0.0267261	36.2248
Overall	0.881705	0.0134362	65.6218

Response	Kappa	SE Kappa	Z
1	0.977897	0.0500000	19.5579
2	0.849068	0.0500000	16.9814
3	0.814992	0.0500000	16.2998
4	0.944580	0.0500000	18.8916
5	0.983756	0.0500000	19.6751
Overall	0.912082	0.0251705	36.2362