The type I error rates associated with the multiple comparisons are often used to identify significant differences between specific factor levels in an ANOVA. The individual and simultaneous confidence level = 1 - the error rate.
The individual error rate is the maximum probability that one or more comparisons will incorrectly conclude that the observed difference is significantly different from the null hypothesis.
The family error rate is the maximum probability that a procedure consisting of more than one comparison will incorrectly conclude that at least one of the observed differences is significantly different from the null hypothesis. The family error rate is based on both the individual error rate and the number of comparisons. For a single comparison, the family error rate is equal to the individual error rate which is the alpha value. However, each additional comparison causes the family error rate to increase in a cumulative manner.
It is important to consider the family error rate when making multiple comparisons because your chances of committing a type I error for a series of comparisons is greater than the error rate for any one comparison alone.
Individual error rates are exact in all cases. Family error rates are exact for equal group sizes. If group sizes are unequal, the true family error rate for Tukey, Fisher, and MCB will be slightly smaller than stated, resulting in conservative confidence intervals. The Dunnett family error rates are exact for unequal sample sizes.
You do a one-way ANOVA to examine steel strength from five different steel plants using 25 samples from each plant.
You decide to examine all 10 comparisons between the five plants to determine specifically which means are different. If you assign an alpha of 0.05 to each of the 10 comparisons (the individual error rate), Minitab calculates a family error rate of 0.28 for the set of 10 comparisons. However, if you want the entire set of comparisons to have a family error rate of 0.05, then Minitab automatically assigns each individual comparison an alpha of 0.007.
Tukey's method, Fisher's least significant difference (LSD), Hsu's multiple comparisons with the best (MCB), and Bonferroni confidence intervals are methods for calculating and controlling the individual and family error rates for multiple comparisons.