What is a hypothesis test?

A hypothesis test is rule that specifies whether to accept or reject a claim about a population depending on the evidence provided by a sample of data.

A hypothesis test examines two opposing hypotheses about a population: the null hypothesis and the alternative hypothesis. The null hypothesis is the statement being tested. Usually the null hypothesis is a statement of "no effect" or "no difference". The alternative hypothesis is the statement you want to be able to conclude is true based on evidence provided by the sample data.

Based on the sample data, the test determines whether to reject the null hypothesis. You use a p-value, to make the determination. If the p-value is less than the significance level (denoted as α or alpha), then you can reject the null hypothesis.

A common misconception is that statistical hypothesis tests are designed to select the more likely of two hypotheses. However, in designing a hypothesis test, we set the null hypothesis up as what we want to disapprove. Because we fix the significance level to be small before the analysis (usually, a value of 0.05 works well), when we reject the null hypothesis, we have statistical proof that the alternative is true. Conversely, if we fail to reject the null hypothesis we do not have statistical proof that the null hypothesis is true. This is because we have not fixed the probability that we falsely accepting the null hypothesis to be small.