What is an attribute agreement analysis (also called attribute gage R&R study)?

Use attribute agreement analyses to evaluate the agreement of subjective nominal ratings or subjective ordinal ratings by multiple appraisers and to determine how likely your measurement system is to misclassify a part.

Nominal data: Are categorical variables that have multiple levels of a characteristic with no natural ordering, such as (for a study of food texture) crunchy, mushy, and crispy.
Ordinal data: Are categorical variables that have three or more levels of a characteristic with a natural ordering, such as strongly disagree, disagree, neutral, agree, and strongly agree.

Use attribute agreement analysis to answer questions such as:

Does the appraiser agree with himself on all trials?
Does the appraiser agree with the known standard on all trials?
Do all appraisers agree with themselves (within appraiser) and others (between appraisers) on all trials?
Do all appraisers agree with themselves, with others, and with the standard?

For example, 5 appraisers visually inspecting fabric for defects. Because fabric defects are difficult to define, you must rely on the appraisers to compare fabric samples to standards. Your measurement system will not be acceptable if the fabric quality rating depends on which appraiser is evaluating it. To assess how well the appraisers are performing, you plan an attribute agreement analysis to study the agreement between 5 appraisers evaluating 10 pieces of fabric, with 3 ratings each. Each fabric sample has a corresponding standard.

If there is substantial agreement among the appraisers, there is the possibility, although not a guarantee, that the ratings are accurate. If there is not agreement among the appraisers, you cannot rely on the ratings.

Note

Attribute Agreement Analysis is different from Attribute Gage Study (Analytic Method), which is a method to examine the bias and repeatability of an attribute measurement system.