Aysegül Kayaoglu, Ghassan Baliki & Tilman Brück | 1 December 2022
Every day, humanitarian assistance helps millions of people in humanitarian emergencies and conflict settings around the world survive. Financial resources are too little to meet all needs, so identifying whom to help and how to do so effectively is an important challenge for the humanitarian sector. In recent years, researchers have developed new approaches to support humanitarian practitioners in addressing this challenge. Researchers have developed new ways to collect data, new methods to identify how support helps people and new ways to match divergent data sources for better learning. The ISDC – International Security and Development Center convened a two-day workshop in Berlin in July 2022 to discuss these new approaches. The workshop brought together impact evaluation researchers, humanitarian program implementers, and donors for shared learning.
Organized by ISDC, with support from the Centre of Excellence for Development Impact and Learning (CEDIL) and UK Aid, the workshop reviewed available tools, their application in humanitarian emergency and conflict settings (HECS), and how they could be improved by novel approaches. The overall message of the workshop was clear: innovative and more flexible approaches to learning about improving humanitarian assistance in HECS are methodologically, ethically and practically challenging – but they are both necessary and eminently feasible.
How to sleep well after conducting a randomized controlled trial in a humanitarian emergency and conflict setting
The main problem impact evaluators studying HECS must overcome is the difficulty of using randomized controlled trials (RCTs) — otherwise the ultimate gold standard in relevant research – particularly in the emergency phase of a humanitarian crisis. Imagine designating a control group, a must for any RCT, in a region with catastrophic destruction due to a disaster or an armed conflict, which would mean excluding that group from aid delivery and leaving its members to their demise. And this is just one of the many complications (e.g., context volatility or difficulty in physical collecting data) involved in carrying out an impact evaluation in HECS.
However, researchers are still expected to defy the odds and not simply resort to monitoring dashboards and ex-post assessments. The task, as arduous as it may be, remains the same — eliminating all sorts of bias and providing the highest quality evidence to measure if and how an intervention causally contributes to achieving targeted outcomes. Clearly, the workshop participants had a lot to cover. And they did.
In response to the difficulty of obtaining a control group, or in other words, a counterfactual, the workshop participants identified four potential remedies.
Firstly, in the immediate aftermath of a humanitarian crisis, factorial designs, where the question of which intervention has a bigger impact rather than what is the impact of a certain intervention, is recommended.
Secondly, through later stages, phased-in designs, where crisis-stricken individuals are included in the planned assistance program on a staggered roll-out basis so that late-comers act as a control group for those treated earlier, would be better to adopt.
Thirdly, if the information is available on a large number of individuals over a long period of time prior to the intervention, researchers can also assign weights to statistically chosen individuals among them and create an optimally estimated control group.
Finally, geographic information systems offer a tremendous potential to process spatial data, where observations are numerically identified with geographic coordinates with the help of advanced software. They may help construct valid counterfactuals in projects where physiographic characteristics matter.
Nonetheless, workshop participants conceded that conducting an RCT may sometimes be entirely unfeasible in HECS, particularly during the relief and recovery phases right after a humanitarian crisis emerges. And when that is the case, peer-reviewed journals should be more inclusive towards allowing sub-optimal designs for publication to prevent evidence gaps and encourage future scientists by not disregarding their hard work.
Be flexible and ready to redesign your research
Even with a great counterfactual at the start though, the result of an impact evaluation researcher’s work in HECS could still be an underpowered study where sample sizes suffer unforeseen declines. To be on the safe side, therefore, researchers would be well advised to oversample first and then stick with power analyses through their research to monitor the statistical validity of their design.
When necessary — and believe us, it often is — impact evaluators should also be ready to redesign their research in later stages. With such flexibility, not only will they preserve statistical efficiency but also bolster understanding and remain ethically acceptable. Adaptive strategies can really work wonders in HECS since they will allow for the reallocation of participants and resources as well as cherry-picking the most effective interventions for each group.
Impact evaluations as a collaborative learning process
It is obvious that a two-way transfer of information between researchers and implementers is a must for rapid improvements to be possible. In this regard, following an emergency response preparedness approach might help them also to respond to sudden shocks in a systematic and collaborative way. With such teamwork, it will be much easier to run multi-site trials so that meta-analyses can be conducted and generalizable lessons can be drawn. Only then can a study’s findings in one humanitarian setting be transferable to another, workshop participants agreed.
Another synergy highlighted at the workshop was between the use of quantitative and qualitative research methods. When combined, these approaches can help scientists derive stronger and more representative conclusions on the assistance program whose true impact they are interested in finding out. The bottom line on this particular subject is invest in collecting both quantitative and qualitative data and you will see how they complement each other, enabling a much fuller picture than they could possibly have alone. And no, that is not more work, it is the job itself.
Machine learning may prove transformational in data collection
Speaking of data collection, one could not help but note how this has become an ever-changing field over the last few decades with the introduction and increased penetration of new technologies into our lives. One aspect that received particular attention from across the discussion hall at the workshop was machine learning.
Remote data collection methods are often much more feasible than conducting face-to-face surveys. However, they may discriminate against target groups with little access to communication tools. Geospatial data takes a front seat with the potential it holds for studying even vast terrains of HECS over a much longer period. That is where machine learning comes into the picture.
Rather than trying to find their way through low-resolution geospatial data, which is akin to looking for a needle in a haystack, researchers could use high-resolution and precise data to train machine learning algorithms to equip themselves with a “magic compass” that is much easier to navigate than the former. This would allow for addressing common budget constraints.
With geospatial data enhanced by machine learning, researchers can also stay on top of the implementation of their designs and those of fellow scientists. When such transparency and exchange of real-time data is made a reality, researchers will be able to account for whom each intervention is administered, when and where, and by whom.
There is a way to make impact evaluations in HECS work
Impact evaluations of humanitarian interventions are indeed possible if we innovate and choose context-appropriate designs, draw on novel data and machine learning, and have a flexible and open mindset on what can be achieved and learned. Importantly, it can happen if qualitative and quantitative researchers and implementers show the commitment to working together collaboratively for developing adaptive strategies.
Cover photo: Turkish workers load bags of flour at the Turkish-Syrian border. Credit: Felton Davis
Learn more about this work at our 15 December webinar: SEEDS for Recovery: The impact of agricultural interventions in Syria