Synthesizing Evidence in Public Policy Contexts: The Challenge of Synthesis When There Are Only a Few Studies

For a variety of reasons, researchers and evidence-based clearinghouses

synthesizing the results of multiple studies often have very few studies

that are eligible for any given research question. This situation is less

than optimal for meta-analysis as it is usually practiced, that is, by

employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to

synthesize. As background, we review the synthesis practices used in

several projects done at the behest of governmental agencies and private

foundations. We then discuss the strengths and limitations of different

approaches to meta-analysis in a limited information

environment. Using examples from the U.S. Department of Education’s

What Works Clearinghouse as case studies, we conclude with a discussion

of Bayesian meta-analysis as a potential solution to the challenges

encountered when attempting to draw inferences about the effectiveness

of interventions from a small number of studies.