Choosing a statistical test

The diagram below is intended as a guide to choosing an appropriate statistical test to investigate the relationship between two or more variables for a given dataset. It focuses on determining what type of test is available based on the nature of the data. Answering the questions about the dependent and independent variables will guide you to a specific test or method to use. It is important to note that in most cases more than one option is available. To help simplify decision making only a single choice is provided here.

Dependent and independent variables

The diagram asks you to make choices based on the nature of dependent and independent variables. In many situations that distinction is clear. If you are interested in testing how different experimental conditions change an outcome measure of interest, it is pretty clear that the outcome measure is the dependent variable and all other variables will be treated as independent.

Things aren't always that clear cut. Especially in situations where you are dealing with two variables it is possible that there is no clear distinction between them. For example, if you are interested in how closely related two different outcome measures are, neither of them really is an independent variable. For the purpose of the below diagram you are free to choose either one as dependent variable. Note, however, that if the two variables are of different type, the statistical analysis methods at your disposal will differ depending on which one you treat as independent variable.

Measurement scales

While traversing the diagram you will be asked to indicate the level at which your dependent and independent variables are measured. These questions rely on Stevens' classification of measurement scales. This classification system recognises four types of measurements, nominal, ordinal, interval, and ratio scales.

Types of scales

Nominal measures distinguish between a fixed number of discrete classes that are identified by their names. No numeric value is attached to the different classes and they can only be compared for equality.

Ordinal measures consist of discrete categories that can be ranked. This allows for comparison of values to determine whether one of them is larger than the other but the magnitude of the difference between two observations is not meaningful.

Interval measures are numeric quantities for which differences can be computed and compared in a meaningful way. However, such scales do not include a zero point that corresponds to an absence of the measured quantity. As a result of this ratios of interval measures are not meaningful, so it isn't possible to say that one value is twice as large as another.

Ratio scale measurements are numeric quantities that do include a meaningful zero and allow for ratios between values them to be computed and compared.

Using different types of scales

Looking at the above description you may note that these measurement scales fall into two groups. Nominal and ordinal variables take on values from a limited number of categories and may be described as categorical variables. Methods that indicate that they require nominal measurements will generally also work for ordinal data, although any ordering of values will be ignored. Similarly, interval and ratio scales can collectively be described as being numeric. Methods designed for interval scale variables will also work for ratio scale measurements.

In some situations it is common practice to treat measures that really are ordinal as if they were interval values. For example, Likert scale items are generally ordinal. However, it is common to aggregate individual items into scales by adding or averaging individual items, thereby treating them as if they were on an interval scale. The resulting aggregate scores are then typically treated as interval measurements. If the underlying assumption that a single step increase for each item corresponds to a unit change of the measured construct is reasonable, this approach is justified.

What you won't find here

The methods covered here are by no means exhaustive. The focus on choosing a test procedure based on the properties of the data means that little consideration is given to specific research questions or common practices in a given field of research. The recommended methods are of general utility and appropriate under the indicated conditions. However, it is entirely possible that a better method exists for your specific study. Below are a few notes on methods and circumstances not covered here.

Multivariate responses

You may have more than one response measure. If these measures are largely independent of each other it is perfectly fine to analyse them separately. If that is what you want to do, simply use the diagram below repeatedly, using each measure as dependent variable in turn. It may, however, be beneficial to consider a joint analysis of your measures, treating them as a single multivariate response. This situation is not covered here.

Structural Equation Models

Structural equation models (SEMs) are not included in the diagram below. They combine factor analysis and multiple regression techniques to model structural relationships between observed and latent variables. When combined with longitudinal data SEMs, unlike the methods covered here, can be used to model causal relationships but care needs to be taken in the interpretation of such models. Even carefully designed experiments can usually not rule out all alternative interpretations, leaving some doubts regarding causality.

Time series

Time series consist of regular measurements taken over an extended period of time. Their analysis usually focuses on the identification of short term, long term and seasonal trends. Although they share some common features with longitudinal data (which may be analysed with some of the methods below), time series focus on the detailed analysis of single units based on frequent observations whereas longitudinal studies track changes in populations, often using relatively few timepoints per individual.

Loading...