How Biased are Observational Methods in Practice? Accumulating Evidence Using Randomised Controlled Trials with Imperfect Compliance

How Biased are Observational Methods in Practice? Accumulating Evidence Using Randomised Controlled Trials with Imperfect Compliance

Date: 5 April 2023

Time: 12.45 – 14.00

Registration link:Free to register on Zoom

Project Page: CEDIL Project Page

In this webinar, we will be exploring the CEDIL project led by Gharad Bryant. Consider a policy maker choosing between programs of unknown impact. She can inform her decision using observational methods, or by running a randomised controlled trial (RCT). The proponents of RCTs would argue that observational approaches suffer from bias of an unknown size and direction, and so are uninformative. Our study treats this as an empirical claim that can be studied. By doing so we hope to increase the value of observational data and studies, as well as better inform the choice to undertake RCTs. We propose a large-scale, standardised, hands-off approach to assessing the performance of observational methods. First, we collect and categorise data from a large number of RCTs in the past 20 years. Second, we implement new methods to understand the size and direction of expected bias in observational studies, and how bias depends on measurable characteristics of programmes and settings. We find that observational methods have mean zero bias, but the bias distribution has high variance such that any given observational study is likely to have high bias.

Gharad Bryan

Associate Professor of Economics

Bryan is a an Associate Professor at the London School of Economics and Political Science, and an Associate of the STICERD Economic Organisation and Public Policy programme. His research interests include development economics, behavioural economics, and experimental economics.

Edoardo Masset

Deputy Director, CEDIL

Before joining CEDIL, Edoardo was Deputy Director and head of the London office of the International Initiative for Impact Evaluation, which he joined after working for seven years as a Research Fellow at the Institute of Development Studies at the University of Sussex. Edoardo is an agricultural and development economist with extensive experience conducting impact evaluations, researching development interventions and consulting for a variety of institutions, including the World Bank.

Edoardo also has experience of research synthesis work through a number of systematic reviews on a wide range of topics such as nutrition, health insurance and cost-effectiveness. His core research interests include rural development, child nutrition, poverty and inequality, and the analysis of household surveys.

Jon De Quidt

Associate Professor

Jonathan de Quidt is Assistant Professor prior to which he spent the period 3 March 2014 – 31 August 2015 as a Post Doc here at the IIES. He received his PhD from the London School of Economics. His research interests lie broadly in behavioral and development economics and combine experimental work and applied theory.

Leave a Reply