There is a well-known tale of a Sufi philosopher named Nasreddin who lived in the 13th century in what would be present-day Turkey. In this story, Nasreddin enlists the help of a friend to help him find his lost key late at night. They search for the key near a lit lamp post in the street for over an hour. Finally, the friend asks “Are you sure you lost your key here?” to which Nasreddin points to a nearby area shrouded in the darkness and says, “No, I lost the key over there, but the light here is better”. Of course, Nasreddin’s response is preposterous and sends his friend into a fit. After all, you can’t find your keys where you didn’t lose them, no matter how bright the light may be.
Although ridiculous, the story harbors a kernel of truth about avoiding hard problems. By way of analogy, in modern evaluation methods we want to use the bright light of randomized controlled trials or similar unbiased research methods that can yield studies with a high-degree of internal validity. Similarly, when we want to consider the weight of evidence for a given intervention through a meta-analysis, we want to include the “best” studies and exclude those where the research methodology is open to skepticism.
But for many research questions, we have to recognize that we left the keys in the dark – and the light from quantitative research methods such as RCTs have difficulty in and of themselves illuminating the very complex questions we are trying to answer. I had this experience evaluating the effects of a UNICEF funded school intervention in Asia. On face value the effort was sensible, but as I visited the schools, I quickly realized that many interventions were simultaneously occurring and these were layered on many past interventions. While it would have been nice from a research perspective if the intervention had been in a vacuum, the school administrators were understandably enough happy to allow any aid group to provide resources and funding for the school. Rigorously evaluating any one intervention, in the context of dozens of simultaneous programs, became demonstrably problematic. A good example of this is written about in the book “Evidence-based Policy: The Promise of Systematic Review” edited by Ray Pawson (2006):
Consider, for instance, the residents of the UK's ‘sink estates’. These citizens may have been on the receiving end of a succession of area-based initiatives such as Single Regeneration Budgets (SRB), City Challenges (CC), European Social Funds-Community Empowerment Fund (ESF-CEF) and, more recently, the New Deal for Communities (NDC) and the Neighbourhood Renewal Strategy (NRS). Social problems cluster in such localities and there may also have been a Health Action Zone (HAZ), an Education Action Zone (EAZ) or a Sure Start initiative (SS). The young people in the area may have had the benefit of a Connexions (CXS) office in which youth and careers advice workers operate in tandem. The areas are deprived and so too are the individual residents, who may also come under the purview of the other New Deals for lone parents, the unemployed, disabled people and so forth (NDLP, NDU, NDDP). As well as salvation by acronym, remember that community members are also beneficiaries of the mainstream provisions of the welfare state, as well as many local government initiatives.
In this example, trying to cleanly separate out the effects of any one intervention is a tall order. Of course the researchers being paid to do just that may tend to downplay the complexity inherent in some of these interventions. Staggering causal complexity gets reduced to a few lines in a section on research limitations.