Difference between attribute agreement analysis in the Assistant versus the Stat menu

The primary difference is the way each command defines a match in replicated experiments, where each appraiser assesses each subject multiple times. Assistant > Measurement Systems Analysis (MSA) > Attribute Agreement Analysis defines each replicate as 1 opportunity for a match. Conversely, Stat > Quality Tools > Attribute Agreement Analysis defines each set of replicates as 1 opportunity for a match.

For example, suppose each appraiser assesses each subject 3 times. If Appraiser A correctly matches the standard 2 out of 3 times for Subject A, then Stat > Quality Tools > Attribute Agreement Analysis awards Appraiser A 0 matches out of 1 opportunity assessing Subject A. But Assistant > Measurement Systems Analysis (MSA) > Attribute Agreement Analysis awards Appraiser A 2 matches out of 3 opportunities assessing Subject A.

Likewise, if Appraiser B assesses Subject B and correctly matches the standard 3 out of 3 times, then Stat > Quality Tools > Attribute Agreement Analysis awards Appraiser B 1 match out of 1 opportunity assessing Subject B. But Assistant > Measurement Systems Analysis (MSA) > Attribute Agreement Analysis awards Appraiser B 3 matches out of 3 opportunities assessing Subject B.

The different ways of counting matches may cause the same data to yield different agreement percentages, when analyzed using both commands.

Other differences exist as well:
  • Attribute agreement analysis in the Assistant requires binary ratings and a known standard. In the Stat menu, ratings may have more than 2 levels, may be nominal or ordinal, and a known standard is optional.
  • Attribute agreement analysis in the Assistant does not assess between-appraiser agreement. It only assesses agreement between each appraiser and the standard, and between all appraisers and the standard. The Stat menu offers these assessments too, plus assessment of between-appraiser agreement.
  • The Stat menu calculates p-values to assess the statistical significance of the observed agreement. The Assistant does not calculate p-values, but uses confidence intervals to assess the statistical significance of the observed agreement.
By using this site you agree to the use of cookies for analytics and personalized content.  Read our policy