Natural Experiments and Quasi-Experiments

Natural Experiments

You have just learned that it is hard to infer causality from an observational study. With observational studies, it is usually much harder to control for confounding variables. However, researchers have come up with ways to control for confounding variables even when treatment assignment is not under the control of the researcher. One example of this is a research design called the natural experiment.

We have a natural experiment when variation in the independent variable is randomly assigned, but not by the researcher. One example of a natural experiment was conducted in Brazil: researchers wanted to know if voters punish politicians when they discover that politicians are involved in corruption.

The researchers noticed that the federal government of Brazil conducted audits to find out whether or not the mayors of local towns are involved in corruption. The federal government did not have the capacity to conduct such audits in every single town, so they had to choose a group of towns to be audited.

To avoid suspicion that the audits were used to persecute opponents, the government decided that the group of towns to be audited would be randomly selected. This gave researchers an opportunity to test the following question: what happens to the reelection chances of mayors who get audited, when compared to mayors who do not get audited?

They found that, when compared to mayors who do not get audited, honest mayors who were audited and came clean were more likely to be reelected. By contrast, corrupt mayors who were audited and got caught were less likely to be reelected than mayors who were not audited.
We know that these outcomes were caused by the audits because audits were randomly assigned. Because of random assignment, the researchers were able to rule out confounding variables: towns that were audited are similar, on average, to towns that were not audited.

When a researcher notices that variation in an independent variable is random, and uses this random assignment to draw causal inference, we see an example of a natural experiment. The researchers exploited the fact that the Brazilian government randomly assigned variation in which towns get audited--the independent variable of interest.

Quasi-Experiments

A quasi-experiment is an experiment that looks like a true, randomized experiment, but lacks random assignment. One example of a quasi-experiment is when treatment is not randomly assigned, but we have reasons to believe that the treatment group is similar to the control group.

For example, think about a high school that wants to measure the impact of the Covid pandemic on student performance. At this school, the class of 2019 suffered no impact from the pandemic, the class of 2020 suffered some impact, and the class of 2021 suffered a lot of impact. So long as we believe that students in the classes of 19, 20 and 21 are similar to each other, a comparison between these three classes should reveal something about the impact of the pandemic on student performance.

This research design has its flaws because assignment to treatment is not random. For example, the senior year of the class of 2021 had the added stress of a hectic election campaign and its violent aftermath. These additional stressors are potential confounds in this research design. But because we cannot randomly assign the impact of the pandemic on students' lives, a research design that compares across graduating classes may be as good as it gets.

Dig Deeper

Natural Experiments