Authors: Wood, B., Jimenez, E., Masset, E., Vigneri, M.
The goal of the paper is to identify factors that lead to successful impact evaluations. Lessons from successes and failures can be used by researchers and funders to better design and monitor impact evaluation studies in low and middle income countries. It is well known that impact evaluations often do not achieve their intended aims, but how is success defined? The paper will begin by formulating a “theory of change” of successful impact evaluations and will identify the potential threats to success. We tentatively conceptualise success along three sequential dimensions: planning stage (which includes, among others, formulation of hypotheses, stakeholders engagement and evaluation design), implementation (which includes project implementation, survey design, data collection and data analysis) and dissemination (publication, dissemination of results and their impact on policy). Different factors may affect the success of the intervention at each stage such as project implementation failures, poor take-up, flaws in survey design or data analysis.
We anticipate focusing our conceptual work on the first dimension of success: the planning stage. There is considerable work on factors affecting the success of evaluation research at the implementation stage. Cases of success and failure in evaluation research have been widely documented in the literature and can only be summarised here. On the other hand, the impact of the intervention on policy is likely to be beyond the scope of our study. The planning stage, including aspects of transparency in conducting research, formulation of designs and analysis plans, and stakeholders engagement are relatively understudied and will considered in greater detail.
Many factors affect success but they cannot all be addressed without making the cost of evaluations prohibitive. In addition, factors affecting success are not the same in all contexts and some contexts and sectors of interventions are notoriously more ‘difficult’ than others. The paper will therefore: a) try to identify the most important key elements of success, so that impact evaluations can be conducted successfully in a cost-effective way and b) identify the factors that hinder the implementation of impact evaluations in difficult areas of intervention such as, for example, conflict, humanitarian assistance, migration, and environment.
Finally, another focus of the paper will be the identification of guidelines for stopping ongoing impact evaluations. Impact evaluations are rarely implemented according to plan and researchers often have to make a difficult choice between closing an evaluation project or changing it radically. In these cases, managers and researchers struggle to make decisions. Each decision is necessarily contingent on the particular project and context. However, we hope to define some general guidance to help making these decisions by reviewing the practice in this area by major funders of evaluations.
The paper will rely on two main sources of data: a literature review and a portfolio analysis of impact evaluations supported by main funding bodies.
– Literature review. We are planning to review the literature on successful impact evaluations by systematically searching impact evaluation blogs. Much of the literature on the topic is quoted in blogs or contained in blogs contributions and we expect this search to deliver a good amount of information
– Portfolio analysis. We will analyse impact evaluation portfolios held by 3ie, DFID and DIME. Evaluations will be coded as successful/unsuccessful to a different degree and factors relating to design and implementation will be investigated to explain the outcome
The paper will conclude with a set of recommendations for researchers and funders on the main determinants of failure and success of impact evaluation, and will formulate a checklist of initiatives to take, at design, implementation, monitoring stage, in order to prevent studies from failing.
The final paper will be available on the website end of September 2018