Use Kendall's statistic with ordinal data of three or more levels.
In the description of the method, without loss of generality, we assume that a single rating on each subject is made by each rater, and there are k raters per subject. Then, to calculate Kendall's correlation coefficient, the k raters represent the k trials made by all the raters.
When the true standard is known, Minitab estimates Kendall's correlation coefficient by calculating the average of the Kendall's coefficients between each appraiser and the standard.
The Kendall's correlation coefficient for the agreement of the trials with the known standard is the average of the Kendall correlation coefficients across trials.
Minitab calculates Kendall's coefficient between each trial and the standard using:
|TX|| number of pairs tied on X = 0.5 Σi ni+ (ni+– 1)|
|TY|| number of pairs tied on Y = 0.5 Σj n+j (n+j– 1)|
|C||number of concordant pairs = Σi<kΣj<l nij nkl|
|D||number of discordant pairs = Σi<kΣj>l nij nkl|
|ni+||number of observations in the ith row|
|n+j||number of observations in the jth column|
|nij|| observations in the cell corresponding to ith row and jth column|
|nkl|| observations in the cell corresponding to kth row and lth column|
|n++|| total number of observations|
A. Agresti (1984). Analysis of Ordinal Categorical Data, John Wiley & Sons.