Name of paper: A map of evidence maps relating to lower-and middle income countries
Can be found here: http://gapmaps.3ieimpact.org/evidence-maps/map-maps
List of authors: Phillips, D., Coffey, C., Tsoli, S., Stevenson, J., Waddington, H., Eyers, J., Snilstveit, B.
Over the last decade, there has been an increase in publication of impact evaluations and systematic reviews on development interventions in lower-and middle-income countries. (L&MICs). This growth has created new challenges for ensuring existing evidence is accessible to decision-makers, avoiding the duplication of new studies and addressing evidence gaps. In response, researchers, governments and non-governmental organisations (NGOs) are increasingly investing in evidence mapping exercises. Organisations such as 3ie, Sightsavers and the International Rescue Committee, are some of many that have published evidence maps.
This report summarises the findings of a ‘map of maps’ designed to catalogue evidence maps relating to development interventions in L&MICs. It adopts the 3ie evidence gap map (EGM) methodology to identify, categorise and display evidence maps within a framework of intervention sectors adapted from the World Bank’s sectoral categorisation and outcomes classified according to the Sustainable Development Goals (SDGs). The report summarises the EGM’s scope and methodology and describes the trends and characteristics of the body of evidence maps found. It also offers some recommendations for future directions in the production of evidence maps.
We found 55 completed and 18 ongoing evidence maps that met our inclusion criteria. While this represents a significant body of evidence maps overall, in many sectors we identified few or no maps. Moreover, most sectors in the framework cover a very broad range of interventions, so although there are a relatively high number of maps for some sectors, there are still likely to be many types of interventions in these sectors where evidence has not been mapped. Overall, the distribution of maps across intervention sectors is relatively uneven, with ‘health, nutrition and population’ the sector with the highest number of evidence maps, followed by ‘agriculture and rural development’, ‘education’ and ‘climate change and environment’. We did not identify any evidence maps that examine ‘transportation’, and only one evidence map in each of the ‘energy’ and ‘economic policy’ sectors. A relatively smaller number of maps have been completed on topics such as ‘information and communications technology’, ‘humanitarian programmes’, and ‘urban development’.
SDG 3 on health and SDG 4 on education are the most frequently covered SDGs. However, no studies report on targets associated with SDG 17 – Global Partnership, while there are several further SDGs where relatively few evidence maps are available. These include WASH (SDG 6), energy (SDG 7), infrastructure (SDG 9), urban and rural development (SDG 11), consumption (SDG 12), climate change (SDG 13), and sustainable use and management of the oceans (SDG 14).
Publication of evidence maps is growing, with the number of studies being published roughly doubling each year from 2014 through to 2016. Map funding and publication is dominated by a few organisations, with DFID and USAID being the most active funders and 3ie, the IRC and the Centre for Environmental Evidence through its journal ‘Environmental Evidence’, the most active publishers.
A majority of maps consider equity in some way. They either have an explicit focus on a specific dimension of inequity such as sex or age, or report on research disaggregated by one or more population group in their analysis. The highest number of studies with some form of equity focus analyse included studies according to sex-based differences, inequality based on age, socioeconomic status, and educational status.
Included evidence maps employ a diverse range of methodologies in terms of the types of studies they include, as well as the scope of their search, critical appraisal, data extraction, and analysis and presentation.
While we did not conduct a critical appraisal of included maps, we extracted data on methods that provide some indication of quality in terms of the comprehensiveness of maps’ search strategy, any restrictions placed on study inclusion and whether maps focussed on describing a body of evidence, rather than drawing conclusions about the overall findings of included studies.
Maps most commonly either include all types of study designs or focus on impact evaluations and systematic reviews only. Most maps searched the unpublished as well as the published literature, while a majority also have some form of inclusion restriction based on publication date, language or publication type.
In terms of data analysis and presentation, most maps present included studies in some form of interventions-outcomes matrix, often accompanied by a narrative analysis and supporting histograms, tables and charts. The evidence maps are largely limited to describing the characteristics of studies they include and only a few report on findings regarding intervention effects. Where maps do describe findings from included studies it is typically in relation to systematic review findings only and on a study-by-study basis. However, there are a small number of maps that synthesise findings of included studies informally or otherwise provide conclusions about the findings of the body of evidence as a whole. The fact that only a limited number of studies did this is encouraging, as doing so adds confusion around the objectives of evidence maps, and blurs the distinction of evidence maps and other forms of evidence synthesis.
Conclusions
Implications for policy
Evidence maps provide policymakers with thematically focussed collections of knowledge that can help inform policy and programming. They are well-calibrated to identify evidence from existing high quality syntheses that can inform existing policy questions or to help determine where new research is needed. However, maps are not designed to provide new syntheses addressing specific policy questions.
Implications for research: methods
The varying objectives that maps can address and the challenges inherent in mapping different topics mean that maps will continue to adopt differing approaches to sourcing and presenting evidence. However, like any other systematic approach, maps should specify their methods in advance via a study protocol and transparently report final methods in a way that can be replicated by others. They should also be careful not to make generalised claims about the findings of a body of evidence based on an informal synthesis.
Implications for research: substantive focus of future maps
An increasing number of evidence maps with a focus on evidence relating to L&MICs are being published. However, some significant gaps remain in terms of intervention sectors and outcomes covered. Sectors where there are currently limited or no evidence maps include transportation, urban development, economic policy, energy, disaster risk reduction, and other adaptive measures. SDGs for which comparatively few maps have been undertaken include SDG7 on energy, SD13 on climate change and SDG14 on marine environment. Nevertheless, even within sectors or SDGs with the highest concentration of evidence maps, there are likely to be gaps. We encourage researchers and commissioners to inspect the online interactive map accompanying this report to identify specific gaps in priority areas and explore the size of the literature before pursuing new evidence maps.