Experiment designs include many types of tests. Before running an experiment, a researcher must design the experiment, including the tests he wishes to use in the data analysis procedure after the test. In some circumstances, the data analysis indicates that there may be some interesting information that cannot be analyzed through the preplanned tests. In such a case, Tukey's HSD test comes in handy, allowing the researcher to further research the matter even after data has been collected and analysis run.
Tukey's HSD test is a post-hoc test, meaning that it is performed after an analysis of variance (ANOVA) test. This means that to maintain integrity, a statistician should not perform Tukey's HSD test unless she has first performed an ANOVA analysis. In statistics, post-hoc tests are used only for further data analysis; these types of tests are not pre-planned. In other words, you should have no plans to use Tukey's HSD test before you collect and analyze the data once.
The purpose of Tukey's HSD test is to determine which groups in the sample differ. While ANOVA can tell the researcher whether groups in the sample differ, it cannot tell the researcher which groups differ. That is, if the results of ANOVA are positive in the sense that they state there is a significant difference among the groups, the obvious question becomes: Which groups in this sample differ significantly? It is not likely that all groups differ when compared to each other, only that a handful have significant differences. Tukey's HSD can clarify to the researcher which groups among the sample in specific have significant differences.
Tukey's HSD test works through defining a value known as the Honest Significant Difference (HSD). This value is a number that acts as a distance between groups. It is calculated by the following procedure. Divide the mean squared error within from the ANOVA analysis by the total number of data points for a given group. Take the square root of the resulting value. Finally, multiply this result by the studentized range statistic (you can look up this statistic in a table provided by virtually every experimental design textbook). This result is the Honest Significant Difference, and it represents the minimum distance between two group means that must exist before the difference between the two groups is to be considered statistically significant.
Like other post-hoc tests, the Tukey HSD test is weak. What this means is that if a test of the difference of two specific means were designed prior to collecting data, it would more likely yield significant results than Tukey's HSD test would. Post-hoc tests are conservative by their nature, so regardless of what two groups the Tukey HSD test is looking at, it will always be weaker than preplanned tests, erring on the side of not rejecting the initial hypothesis that the group means are equal.
- "The Collected Works of John W. Tukey"; David Brillinger; 1984