What is the adjusted p-value in multiple comparisons?

Use for multiple comparisons in ANOVA, the adjusted p-value indicates which factor level comparisons within a family of comparisons (hypothesis tests) are significantly different. If the adjusted p-value is less than alpha, then you reject the null hypothesis. The adjustment limits the family error rate to the alpha level you choose. If you use a regular p-value for multiple comparisons, then the family error rate grows with each additional comparison. The adjusted p-value also represents the smallest family error rate at which a particular null hypothesis will be rejected.

It is important to consider the family error rate when making multiple comparisons because your chances of committing a type I error for a series of comparisons is greater than the error rate for any one comparison alone.

Example of adjusted p-values

Suppose you compare the hardness of 4 different blends of paint. You analyze the data and get the following output:

One-way ANOVA: Hardness versus Paint

Tukey Pairwise Comparisons

Tukey Simultaneous Tests for Differences of Means Difference SE of Difference of Levels of Means Difference 95% CI T-Value Blend 2 - Blend 1 -6.17 2.28 (-12.55, 0.22) -2.70 Blend 3 - Blend 1 -1.75 2.28 ( -8.14, 4.64) -0.77 Blend 4 - Blend 1 3.33 2.28 ( -3.05, 9.72) 1.46 Blend 3 - Blend 2 4.42 2.28 ( -1.97, 10.80) 1.94 Blend 4 - Blend 2 9.50 2.28 ( 3.11, 15.89) 4.17 Blend 4 - Blend 3 5.08 2.28 ( -1.30, 11.47) 2.23 Adjusted Difference of Levels P-Value Blend 2 - Blend 1 0.061 Blend 3 - Blend 1 0.868 Blend 4 - Blend 1 0.478 Blend 3 - Blend 2 0.245 Blend 4 - Blend 2 0.002 Blend 4 - Blend 3 0.150 Individual confidence level = 98.89%

Tukey Simultaneous 95% CIs

You choose an alpha of 0.05 which, in conjunction with the adjusted p-value, limits the family error rate to 0.05. At this level, the differences between blends 4 and 2 are significant. If you lower the family error rate to 0.01, the differences between blends 4 and 2 are still significant.