Authors: Vigneri, M., Masset, E., Exley, J., White, H.
Epidemiologists and economists often evaluate similar projects and policies using different tools and approaches. In some cases, the tools are the same but are used in different ways. Finally, researchers of the two disciplines may employ the same tools but using a different terminology. For example, instrumental variable approaches are standard in econometrics but rarely used by epidemiologists. Similarly, power calculations for the determination of sample size that are common in epidemiology, are not the subject of any major econometrics textbook. Other examples include the use of OLS versus non-linear regressions, the analysis of multiple versus single outcomes, the randomisation of offering of an intervention and many more.
These differences lead to misunderstandings between researchers of the two disciplines. In addition, as the results of evaluations by epidemiologists and economists differ without be strictly comparable, the public is confused and unable to interpret the results or to reconcile the differences between evaluations by economists and epidemiologists. Building on work conducted separately by mixed teams of epidemiologists and economists (a group based at LSHTM looking at differences in the practice of RCTs by economists and epidemiologists, and a group involving other Cedil ILT members looking at similarities in approaches to design and analysis of evaluations), the present paper aims at resolving some of these differences and misunderstanding.
The goal of the paper is twofold. First, we want to identify the area of overlap between methods and approaches used by the two disciplines and to make an attempt at harmonising the evaluation terminology they use in order to achieve a mutual understanding. Second, we want to identify those approaches and methods that markedly differ between the two disciplines and to exploit these differences to generate a debate that will lead to a more integrated approach to the evaluation of social policies. We are aware that the heterogeneity of approaches within each discipline is at times larger than the differences between the two disciplines. For this reason, we will carefully delimit the scope of our work to methods and approaches used in the design and analysis of experiments and quasi-experiments for the evaluation of projects and social policies.
The first step in our project will be the creation of a glossary of impact evaluation terms that includes tools used by both disciplines with an attempt to harmonisation. For this exercise, we will use existing glossaries of impact evaluation as a starting point. The second step will be a review of a few examples of project types evaluated by economists and by epidemiologists (in area such as, for example, WASH, nutrition, and cash transfers) in order to identify main methodological differences in the conduct of impact evaluations. The review will be informed by existing texts on the topic such as Impact Evaluation by S. Khandker or the Impact evaluation toolkit by the JPal, for an economics example, and will include quantitative as well as qualitative approaches. The differences highlighted by the review will then be the subject of a discussion workshop between epidemiologists and economists, which will result in a new table-map of overlaps and disagreements. A short paper will summarise the process followed, identify the opportunities for the different disciplines to learn from each other, and will conclude with a road map of initiatives that will lead to a more integrated approach to evaluations between the two disciplines.
The outputs of this project will include:
- A cross-disciplines unified glossary of impact evaluation terms
- A series of case-studies of evaluations of similar (preferably the same) projects by the two disciplines
- A workshop resulting in a table of main methodologic overlaps and differences between the disciplines
- A summary paper proposing a road map for a more unified approach
The final paper will be available on the website end of September 2018