What is the adjusted p-value in multiple comparisons?

Use for multiple comparisons in ANOVA, the adjusted p-value indicates which factor level comparisons within a family of comparisons (hypothesis tests) are significantly different. If the adjusted p-value is less than alpha, then you reject the null hypothesis. The adjustment limits the family error rate to the alpha level you choose. If you use a regular p-value for multiple comparisons, then the family error rate grows with each additional comparison. The adjusted p-value also represents the smallest family error rate at which a particular null hypothesis will be rejected.

It is important to consider the family error rate when making multiple comparisons because your chances of committing a type I error for a series of comparisons is greater than the error rate for any one comparison alone.

Example of adjusted p-values

Suppose you compare the hardness of 4 different blends of paint. You analyze the data and get the following output:

Tukey Simultaneous Tests for Differences of Means
Difference of Levels
Difference of Means
SE of Difference
T-Value
Adjusted P-Value
Individual confidence level = 98.89%

You choose an alpha of 0.05 which, in conjunction with the adjusted p-value, limits the family error rate to 0.05. At this level, the differences between blends 4 and 2 are significant. If you lower the family error rate to 0.01, the differences between blends 4 and 2 are still significant.

By using this site you agree to the use of cookies for analytics and personalized content.  Read our policy