The great chess grandmaster Siegbert Tarrasch once said that if you pay careful attention to development, then when the time came that you needed a combination, it will be there. Tarrasch's comment came to mind as I was reevaluating, yet again, the way I introduce hypothesis testing.

Recall the comments in Is Statistics Hard?

Statistical methods are convoluted! In order to show an effect exists,

• statistics begins by assuming there is no effect.
• Prior to collecting data, rules are chosen to decide whether whether the data are consistent with the assumption of no effect.
• If the data are found to be inconsistent with the assumption, the assumption must be false and there is, in fact, an effect!

Before considering formal hypothesis testing, let's see how those quoted comments relate to confidence intervals. We wished to compare two population means. We did so by constructing a confidence interval for their difference.

• We calculate the difference in sample means.
• We know that the sample mean and population mean have to be within a certain distance of each other.
• The confidence interval for the difference in population means is the set of values within that certain distance of the difference in sample means.

The way the game is played, the data are said to be consistent with any value for the population mean difference that is in the CI.

• In order to show that there is a difference in population means, it is necessary for 0 to be excluded from the confidence interval for the mean difference. If 0 is not in the CI, then the data are found to be inconsistent with a population mean difference of 0. Therefore, there is evidence that the difference is, in fact, not 0.
• If 0 is included, then, no matter what other values are in the interval, we can't escape the fact that the data are consistent with a mean difference of 0. Since the data are not inconsistent with a population mean difference of 0, we cannot say that the difference is not 0.

Insofar as comparing population means is concerned, a formal test of the hypothesis that the means are equal will be equivalent to looking at whether 0 is in the confidence interval.