Noam Angrist and Rachael Meager | 16 December 2022
More students are enrolled in school than at any time in history, yet progress in learning remains limited (Angrist et al. 2021). Worldwide, 617 million school-age youth are unable to reach minimum proficiency levels in basic reading and mathematics (UIS, 2017).
One of the educational reforms that has successfully addressed this “learning crisis” is targeting instruction to a child’s learning level rather than grade. This pedagogical shift has been shown to dramatically improve learning across multiple contexts in India, Kenya. and Ghana (J-PAL, 2013). A model for this intervention called Teaching at the Right Level (known as TaRL), has been developed by Pratham in India and shown to more than double the number of students who could read a paragraph or story (Banerjee et al, 2017).
The consistency in positive effects of targeted instruction has received significant attention in the academic literature and in policy circles. A recent report by the World Bank and United Kingdom Foreign Commonwealth and Development Office (FCDO) highlights the approach as a cost-effective solution to address the global learning crisis (Global Education Evidence Advisory Panel 2020). However, there is substantial variation in the degree of effectiveness of TaRL and other similar programs – ranging in effect from 0.07 to 0.78 standard deviations – which remains underexplored. Understanding more about this variation could reveal important information on how generalizable TaRL is and factors that can optimize effectiveness with multiple ambitious scale-up efforts underway.
In our CEDIL project, we have been systematically analyzing the generalizability and heterogeneity of the evidence on targeted instruction. And in our recent syntheses working paper and evidence brief we share our findings from aggregating data across 10 studies and program or contextual variations arms covering nearly 100,000 students.
Key success indicators for the targeted instructional approach
Our results reveal a few striking patterns. First, we find that both teachers and volunteers are highly effective, with volunteers particularly effective.
Second, we find that there are two factors that explain nearly all of the variation across sample studies that we assessed as part of our analysis: the type of delivery model (whether it is volunteer or teacher) and the degree of take-up and fidelity of the program. These are the most important ingredients in achieving the largest frontier impacts in the literature and reveal that the effects can be highly generalizable across contexts when accounting for these two dimensions. This is not obvious ex-ante, as other factors like baseline learning levels, year of delivery, and geography are often assumed to be more important.
Implications for future research
Our results show what drives differences in TaRL effectiveness, which can enable scaling efforts to achieve the largest effects in the literature. Moreover, we demonstrate the first-order importance of quantifying implementation to understanding how generalizable effects are across contexts. Accounting for implementation remains rare in meta-analysis. However, it can be as simple as accounting for implementation by estimating treatment-on-the-treated effects – which capture effects for those who actually received the program – in addition to the more typical intention-to-treat effects, which capture effects for those randomized to the program. More elaborate Bayesian models can also be used to fully account for implementation.
Optimizing implementation on a scaling program with A/B tests
Given the importance of implementation, we report results on an effort to optimize implementation on a scaling Teaching at the Right Level program in Botswana. In Botswana, Youth Impact, one of the largest NGOs in the country, is collaborating with the government to test and scale TaRL. A rapid trial was conducted to optimize TaRL implementation fidelity to target instruction to children’s level as best as possible, achieved through more detailed learning assessments and innovative methods for grouping students relative to standard implementation. Youth Impact refers to these rapid randomized optimizations as “A/B tests” and conducts them on an ongoing basis to maximize cost-effectiveness at scale. Results show that these small, extremely cheap innovations to grouping students led to an additional 0.22 SD gains relative to standard implementation (already highly effective to begin with). This reveals concrete mechanisms to enhance implementation of programming on the path to scale, with highly cost-effective returns.
Policy implications
The results in this paper have significant implications for policy. Targeted instructional approaches have been featured in several multilateral policies and have been cited on lists of evidence-based interventions as a high-potential to address the global learning crisis. These approaches are on track to reach over 60 million children in South Asia and sub-Saharan Africa by 2025 (see map below). The results in this paper inform the generalizability of targeted instruction, contextual factors and program components that mediate the largest effects in the literature, and provide practical guidance as targeted instruction approaches are adapted and scaled-up in new contexts.
Figure 1: Ongoing TaRL scale-up efforts
Cover image: Youth Impact