GoKawiilNatural field experiments might involve studying how shoppers respond to in-store changes, such as moving where products are displayed.Credit: Robert Nickelsberg/Getty
Public policy is full of initiatives that did not work out as hoped. Take Scared Straight, a programme run by more than 30 US states between 1978 and 2015 that aimed to dissuade at-risk teenagers from becoming hardened criminals by bringing them face-to-face with people incarcerated in maximum-security prisons1,2. The programme was extended after a pilot project, the subject of a 1978 documentary, found that 80–90% of teenage participants stayed out of trouble. But the intervention did not work when scaled up. In some places, criminal behaviour among teenagers even rose.
Similarly, many childhood-development interventions that have proved effective in one place have failed to deliver comparable results elsewhere. Deworming children in schools, for instance, substantially reduced absenteeism in Kenya but has shown mixed or weaker effects in other settings. School meal programmes in Burkina Faso increased student attendance but had limited impacts on outcomes in other countries3.
Why science has a credibility problem — and how to address it
This generalizability problem arises, in part, because human behaviour differs across populations and situations. People live in complex social environments in which labels, stakes and scrutiny shape every decision. But those contexts are often overlooked. Conventionally, research participants have come from Western, educated, industrialized, rich and democratic (WEIRD) populations4, which are unrepresentative of other groups on measures ranging from visual perception to moral reasoning and cooperation. What might work for them might not apply to other populations.
In my view, one solution to the problem is to use a greater number of natural field experiments. In such studies, participants go about their everyday activities unaware that they are being observed by researchers, while some feature of their environment is varied. By studying people in their natural setting, assuming that strict ethical rules are followed, researchers can be more confident that their findings will be relevant to that group.
Three developments make scientists better placed to take advantage of such approaches. First, attention on the replication crisis in academia has coincided with a growing understanding that results often fail to generalize beyond the narrow populations typically recruited for laboratory studies. Second, the technology sector is running thousands of natural field experiments to obtain reliable information about their customers, establishing infrastructure and methods that academics can use. Third, a growing body of research into generalizability has provided frameworks for predicting when and why results will fail to apply across populations and settings5,6.
Here I outline how academics can embed natural field experiments in their work.
A three-stage problem
Difficulties in replicating studies have been recognized across the sciences, from social science to biomedicine. Reforms in how research is done can help researchers to generate more-reliable results and repeat others’ work more easily. This includes pre-registration of hypotheses and methods, larger samples, open data and transparent reporting. But, in fields related to human behaviour, replication only requires researchers to obtain the same result with the same kinds of people in the same setting, not to test whether it holds for people elsewhere.
... continue reading