Stat > Power and Sample Size > General Full Factorial Design > Options
Significance level
Usually, a significance level (denoted as α or alpha) of 0.05 works well. A significance level of 0.05 indicates that the risk of concluding that a difference exists—when, actually, no difference exists—is 5%. It also indicates that the power of the test is 0.05 when there is no difference.
Choose a higher significance level, such as 0.10, if you are willing to increase the risk of concluding that the main effect of a factor is statistically significant—when, actually, no effect exists—so that you have greater power to detect an effect that is important. For example, a chemical engineer designs an experiment to study the effect of 5 factors on the yield of a substance. The engineer prefers to consider small or insignificant effects further rather than to remove a factor that could be important. Therefore, the engineer chooses a significance level of 0.10 to be more certain of detecting a factor that is important.
Choose a lower significance level, such as 0.01, to be more certain that you do not conclude that a difference is statistically significant when the difference is not. For example, a scientist at a pharmaceutical company designs an experiment to study the effect of 5 factors on a new drug. For further experimentation, the scientist prefers to study only factors that are important. The scientist chooses a significance level of 0.01 to be more certain not to conclude that an effect that does not exist is statistically significant.
