Which statistical test should I use?

Start with the first question and follow the arrows until you reach a test. The test at the end leads to its page, with the null hypothesis, assumptions, formula and R and Python code. All tests are in the statistical tests catalog.

What do you want to do?

Compare groups, or one group with a reference value
What kind of response did you measure?
- Numeric: a measurement, such as weight, time or score
  
  How many groups or measurements?
  
  One group, against a reference value
  
  Are the data approximately normal, or is the sample large (n ≳ 30)?
  
  Yes
  
  One-sample t Means
  
  Compare the mean of a sample with a reference value when the population standard deviation is unknown.
  
  Check first: Shapiro-Wilk
  
  No
  
  Sign test Ranks
  
  Test the median of a sample without assuming anything about the shape of the distribution; it only uses whether each value is above or below.
  
  Two groups
  
  Are the groups made of different individuals or the same individuals?
  
  Different individuals (independent)
  
  Are the data approximately normal in each group, or are the samples large?
  
  Yes
  
  Two-sample t Means
  
  Compare the means of two different groups, such as treatment and control.
  
  Use Welch’s version, which does not require equal variances.
  
  Check first: Shapiro-Wilk
  
  No, or the response is ordinal
  
  Mann-Whitney U Ranks
  
  Compare two independent groups when the data are not normal, have extreme values or are ordinal.
  
  The same individuals, such as before and after (paired)
  
  Are the differences between pairs approximately normal?
  
  Yes
  
  Paired t Means
  
  Compare two measurements taken on the same individuals, such as before and after an intervention.
  
  Check first: Shapiro-Wilk
  
  No, but they are symmetric
  
  Wilcoxon signed-rank Ranks
  
  Compare two paired measurements without assuming the differences are normal.
  
  No, and not symmetric either
  
  Sign test Ranks
  
  Test the median of a sample without assuming anything about the shape of the distribution; it only uses whether each value is above or below.
  
  Three or more groups
  
  Are the groups made of different individuals or the same individuals?
  
  Different individuals (independent)
  
  Are the residuals approximately normal?
  
  Yes
  
  One-way ANOVA Means
  
  Compare the means of three or more independent groups at once, without inflating the type I error with several t-tests.
  
  Check first: Shapiro-Wilk, Levene
  
  If H₀ is rejected: Tukey HSD
  
  No, or the response is ordinal
  
  Kruskal-Wallis Ranks
  
  Compare three or more independent groups without assuming normality: it is the ANOVA done on ranks.
  
  If H₀ is rejected: Dunn’s test
  
  The same individuals, under several conditions or time points
  
  Are the residuals approximately normal?
  
  Yes
  
  Repeated measures ANOVA Means
  
  Compare three or more conditions measured on the same individuals, such as several time points in a follow-up.
  
  Check first: Mauchly’s sphericity
  
  No, or the response is ordinal
  
  Friedman Ranks
  
  Compare three or more conditions measured on the same individuals without assuming normality.
  
  Two factors at once, such as dose and sex
  
  Two-way ANOVA Models and regression
  
  Assess at once the effect of two factors on a numeric response, and whether they interact, such as dose and sex in a trial.
  
  Check first: Shapiro-Wilk, Levene
- Categorical: yes or no, or categories
  
  How many groups or measurements?
  
  One group, against a reference proportion
  
  Binomial / one-proportion Z Proportions
  
  Compare the proportion in a sample with a reference value, such as a pass rate against 50%.
  
  One group, with several categories against expected frequencies
  
  Chi-square goodness of fit Goodness of fit and normality
  
  Check whether the observed frequencies in categories match the ones a theory expects, such as a fair die.
  
  Two independent groups
  
  Are all expected counts at least 5?
  
  Yes
  
  Two-proportion Z Proportions
  
  Compare the success rate of two independent groups, such as the conversion of two versions of a web page.
  
  Also works: Chi-square of independence
  
  No
  
  Fisher’s exact Association and correlation
  
  Test association in a 2 × 2 table with small counts, where the chi-square test does not hold.
  
  Two measurements on the same individuals
  
  McNemar Proportions
  
  Compare a proportion measured twice on the same individuals, such as opinion before and after a campaign.
  
  Three or more independent groups
  
  Chi-square of homogeneity Proportions
  
  Compare the distribution of a categorical variable across three or more independent groups.
  
  If H₀ is rejected: Marascuilo
  
  Three or more measurements on the same individuals
  
  Cochran’s Q Proportions
  
  Compare a proportion measured under three or more conditions on the same individuals, such as passes in several exams.
- The spread, not the center
  
  How many groups or measurements?
  
  One group, against a reference variance
  
  Chi-square for one variance Variances
  
  Test whether the variance of a process is at the specified value, as in quality control.
  
  Two independent groups
  
  Are the data normal?
  
  Yes
  
  F for two variances Variances
  
  Compare the spread of two independent groups.
  
  No, or it cannot be assured
  
  Levene Variances
  
  Check whether two or more groups have the same variance, for example before an ANOVA.
  
  Two measurements on the same individuals
  
  Pitman-Morgan Variances
  
  Compare the variances of two paired measurements, such as the precision of two instruments applied to the same samples.
  
  Three or more independent groups
  
  Levene Variances
  
  Check whether two or more groups have the same variance, for example before an ANOVA.
Measure the relationship between two variables
What kind of variables are they?
- Both numeric
  
  Is the relationship linear, without strong extreme values?
  
  Yes
  
  Pearson correlation Association and correlation
  
  Measure and test the linear relationship between two quantitative variables.
  
  No, or the variables are ordinal
  
  Is the sample small, or does it have many ties?
  
  Yes
  
  Kendall’s tau Association and correlation
  
  Measure the agreement in order between two variables, counting concordant and discordant pairs.
  
  No
  
  Spearman correlation Association and correlation
  
  Measure the monotonic relationship between two variables, even if non-linear or ordinal: it is Pearson computed on ranks.
- Both categorical
  
  Are all expected counts at least 5?
  
  Yes
  
  Chi-square of independence Association and correlation
  
  Check whether two categorical variables are associated, in a contingency table.
  
  No, in a 2 × 2 table
  
  Fisher’s exact Association and correlation
  
  Test association in a 2 × 2 table with small counts, where the chi-square test does not hold.
- One binary and one numeric
  
  Point-biserial correlation Association and correlation
  
  Measure the relationship between a binary and a quantitative variable; it is equivalent to the two-sample t-test.
- Two raters classifying the same items
  
  Cohen’s kappa Association and correlation
  
  Measure the agreement between two raters who classify the same items into categories, discounting the agreement expected by chance.
Check an assumption or the distribution of the data
What do you want to check?
- Whether the data are normal
  
  How large is the sample?
  
  Small or medium
  
  Shapiro-Wilk Goodness of fit and normality
  
  Check whether a sample comes from a normal distribution; it is the most powerful normality test for small and medium samples.
  
  Also works: Anderson-Darling, Lilliefors
  
  Large, as in econometrics
  
  Jarque-Bera Goodness of fit and normality
  
  Test normality through skewness and kurtosis; common in econometrics, with large samples.
  
  With very large n, any test rejects irrelevant departures: look at the Q-Q plot too.
- Whether the data follow another continuous distribution
  
  Are the parameters of the distribution known in advance?
  
  Yes
  
  Kolmogorov-Smirnov Goodness of fit and normality
  
  Compare a sample with a fully specified theoretical distribution.
  
  No, they are estimated from the data
  
  Anderson-Darling Goodness of fit and normality
  
  Test goodness of fit to a distribution, giving more weight to the tails than Kolmogorov-Smirnov.
- Whether the category frequencies are the expected ones
  
  Chi-square goodness of fit Goodness of fit and normality
  
  Check whether the observed frequencies in categories match the ones a theory expects, such as a fair die.
- Whether an extreme value is an outlier
  
  Grubbs Goodness of fit and normality
  
  Check whether the most extreme value in a sample is an outlier, that is, too far away to come from the same normal distribution as the others.
- Whether the groups have the same variance
  
  Levene Variances
  
  Check whether two or more groups have the same variance, for example before an ANOVA.
- Sphericity, before a repeated measures ANOVA
  
  Mauchly’s sphericity Variances
  
  Check sphericity before a repeated measures ANOVA: the variances of the differences between conditions must be equal.
Test a regression model
What do you want to test?
- Whether an explanatory variable has an effect
  
  How was the model fitted?
  
  Least squares (linear regression)
  
  t for a coefficient Models and regression
  
  Test whether an explanatory variable has an effect in a regression, holding the others fixed.
  
  Maximum likelihood, such as logistic regression
  
  Wald Models and regression
  
  Test a coefficient of a model fitted by maximum likelihood using only the full model.
  
  Also works: Likelihood ratio
- Whether the model as a whole explains anything
  
  Overall regression F Models and regression
  
  Test whether the regression model as a whole explains anything, or whether all coefficients may be zero.
- Whether a larger model is worth it over a smaller one
  
  Likelihood ratio Models and regression
  
  Compare two nested models fitted by maximum likelihood, as in a logistic regression.
- The fit of a logistic regression
  
  Hosmer-Lemeshow Models and regression
  
  Check whether the probabilities predicted by a logistic regression match the observed frequencies, grouping the cases by risk band.
- The residuals
  
  What problem do you suspect?
  
  Autocorrelation, in data over time
  
  Durbin-Watson Models and regression
  
  Check for first-order autocorrelation in the residuals of a regression, common with data over time.
  
  Variance that changes with the fitted values
  
  Breusch-Pagan Models and regression
  
  Check whether the variance of the residuals of a regression is constant (homoscedasticity).
Analyze a time series
What do you want to know?
- Whether the series is stationary
  
  Augmented Dickey-Fuller Time series
  
  Check whether a time series has a unit root, that is, whether it is non-stationary.
  
  The two have opposite null hypotheses: running both gives a firmer conclusion.
  
  Also works: KPSS
- Whether there is still autocorrelation, or it is white noise
  
  Ljung-Box Time series
  
  Check whether a series, or the residuals of a model, still have autocorrelation over several lags.

Before following the tree

Normal or not? With large samples, tests of means tolerate non-normality well. When in doubt, look at the histogram and the Q-Q plot before relying on a normality test alone.
Independent or paired? If each value in one group has a natural partner in the other, such as the same patient before and after, the samples are paired.
The tree is a starting point. It covers the usual cases; the study design and the research question may call for another test.