What is the adjusted p-value in multiple comparisons?

Use for multiple comparisons in ANOVA, the adjusted p-value indicates which factor level comparisons within a family of comparisons (hypothesis tests) are significantly different. If the adjusted p-value is less than alpha, then you reject the null hypothesis. The adjustment limits the family error rate to the alpha level you choose. If you use a regular p-value for multiple comparisons, then the family error rate grows with each additional comparison. The adjusted p-value also represents the smallest family error rate at which a particular null hypothesis will be rejected.

It is important to consider the family error rate when making multiple comparisons because your chances of committing a type I error for a series of comparisons is greater than the error rate for any one comparison alone.

Example of adjusted p-values

Suppose you compare the hardness of 4 different blends of paint. You analyze the data and get the following output:

Tukey Simultaneous Tests for Differences of Means

Difference of Levels	Difference of Means	SE of Difference	95% CI	T-Value	Adjusted P-Value
Blend 2 - Blend 1	-6.17	2.28	(-12.55, 0.22)	-2.70	0.061
Blend 3 - Blend 1	-1.75	2.28	(-8.14, 4.64)	-0.77	0.868
Blend 4 - Blend 1	3.33	2.28	(-3.05, 9.72)	1.46	0.478
Blend 3 - Blend 2	4.42	2.28	(-1.97, 10.80)	1.94	0.245
Blend 4 - Blend 2	9.50	2.28	(3.11, 15.89)	4.17	0.002
Blend 4 - Blend 3	5.08	2.28	(-1.30, 11.47)	2.23	0.150

You choose an alpha of 0.05 which, in conjunction with the adjusted p-value, limits the family error rate to 0.05. At this level, the differences between blends 4 and 2 are significant. If you lower the family error rate to 0.01, the differences between blends 4 and 2 are still significant.