WEBVTT

1
00:00:00.000 --> 00:00:08.700
NANCY CARTWRIGHT: You know that the everything went very strange. I've been in dozens of these apps. There's always something new that goes wrong. Sorry.

2
00:00:09.510 --> 00:00:21.990
NANCY CARTWRIGHT: So topic today is using middle level theory 10 steps for for constructing a middle level theory of change and middle level theory is a hodgepodge

3
00:00:22.770 --> 00:00:35.820
NANCY CARTWRIGHT: So the term means tons of different things and it encompasses lots and lots of different things. There are two kinds of middle level theory or two senses middle level theory that matter for our enterprise here.

4
00:00:36.360 --> 00:00:45.480
NANCY CARTWRIGHT: One is middle level causal principles and the other are middle level what I'm calling process or causal process tracing theories of change.

5
00:00:46.440 --> 00:01:02.580
NANCY CARTWRIGHT: So let's have a look. Middle is, on the one hand it's between the middle between the abstract and the concrete and it's also middle between being fairly universal and wide ranging and being very, very narrow in its range of application.

6
00:01:03.420 --> 00:01:09.180
NANCY CARTWRIGHT: These are obviously sliding scales and there's no particular place that's middle

7
00:01:10.260 --> 00:01:13.140
NANCY CARTWRIGHT: It's not a technical term. It's a guiding idea.

8
00:01:14.610 --> 00:01:24.300
NANCY CARTWRIGHT: Now I'm going to give you some examples. A high level, what I think I was a high level theory and social theory is agents act to maximize their expected utility

9
00:01:25.530 --> 00:01:35.580
NANCY CARTWRIGHT: That's wide ranging and it uses abstract language utility and at least some of my economist friends think it's universal

10
00:01:36.960 --> 00:01:46.290
NANCY CARTWRIGHT: Now, there are various grades of middle level theory of what you know more concrete versions of this, for instance. People respond to incentives. That's wave

11
00:01:47.310 --> 00:01:56.400
NANCY CARTWRIGHT: And instantiate of maximizing your expected utility more concretely people offered conditional cash transfers to do something tend to do that.

12
00:01:56.700 --> 00:02:07.740
NANCY CARTWRIGHT: That's even more concrete parents enrolled children in school have offered educational CCT that's a more concrete still and more narrow ranging um

13
00:02:08.220 --> 00:02:16.170
NANCY CARTWRIGHT: So where are the boundaries are. It doesn't really matter. It's just get the idea that you can do things at different levels and with different ranges.

14
00:02:17.160 --> 00:02:26.940
NANCY CARTWRIGHT: Okay, so these are more range restricted and there shouldn't be a very THERE THEY USE CONCRETE language that was a holdover from something outside. Okay.

15
00:02:28.980 --> 00:02:42.720
NANCY CARTWRIGHT: Then there are things that are fairly concrete in particular and local me. Here's a very local one in Brazil's bolsa program conditional allowances given preferentially the female heads of household through citizens cards that operate, etc.

16
00:02:43.200 --> 00:02:49.230
NANCY CARTWRIGHT: Induce parents to enroll children in school. So we have the same kind of principle, that becomes more and more and more concrete.

17
00:02:49.500 --> 00:02:58.020
NANCY CARTWRIGHT: And more and more narrow as we go down and this one's tied to specific times and places and it uses concrete operational wise and body language. Okay.

18
00:02:58.380 --> 00:03:04.950
NANCY CARTWRIGHT: Now middle level theory is useful and developing planning and evaluation. It helps you to

19
00:03:05.640 --> 00:03:12.120
NANCY CARTWRIGHT: Predict effectiveness and local settings suggest program design features. It gives you information from monitoring.

20
00:03:12.420 --> 00:03:25.470
NANCY CARTWRIGHT: As it gives you assumptions to be tested. Well, doesn't give you, it helps you find assumptions to be tested in an evaluation helps identify evaluation questions and it helps you interpret evaluation findings.

21
00:03:26.190 --> 00:03:29.670
NANCY CARTWRIGHT: It helps with those, you know, it's not a miracle cure. It's just that it's a tool.

22
00:03:31.230 --> 00:03:43.380
NANCY CARTWRIGHT: As I want to talk just a little about middle level causal principles, because these are vex subjects I know in evaluation and in social science in in life in general about causality and principles and things

23
00:03:44.010 --> 00:03:56.010
NANCY CARTWRIGHT: Just some things that I think we need to be alert to these are often familiar behavioral principles, people have to maximize their expected utilities and respond to incentives. Those are familiar behavioral principles.

24
00:03:57.060 --> 00:04:03.480
NANCY CARTWRIGHT: And they're also sometimes the result of social science research, we had more time I give you many examples of those

25
00:04:04.560 --> 00:04:09.330
NANCY CARTWRIGHT: Now generally a middle level causal principle. It's a catering service.

26
00:04:09.360 --> 00:04:10.170
Principle, you might

27
00:04:11.580 --> 00:04:22.740
NANCY CARTWRIGHT: It doesn't tell you what will result mean doesn't tell you if this cause happens this effect will happen. Um, but rather it describes what the cause tends to this is the language that I learned from john Stuart Mill.

28
00:04:23.910 --> 00:04:29.550
NANCY CARTWRIGHT: Okay. For one thing, you might have the cause might be present. You're sitting there, but you have to trigger it.

29
00:04:30.330 --> 00:04:37.320
NANCY CARTWRIGHT: But another is that generally the indicated effect will not actually be the observed outcome.

30
00:04:37.620 --> 00:04:45.810
NANCY CARTWRIGHT: Because in this is just one cause among many of the top rating. So what you're doing is you're picking out what the you know what this causes pushing towards

31
00:04:46.530 --> 00:04:52.800
NANCY CARTWRIGHT: Rather than what's actually going to happen because for that you need the bigger picture of everything that's going on. So still

32
00:04:53.400 --> 00:05:06.870
NANCY CARTWRIGHT: Even though you don't get. I mean, the outcome that occurs and given setting. When the call is indicated one still, it could be what the cause is pushing to some middle global calls or principles describe causes that operate like that.

33
00:05:08.460 --> 00:05:16.230
NANCY CARTWRIGHT: Okay, so today we want to talk about 10 steps for constructing middle level calls or process tracing theories of change.

34
00:05:16.830 --> 00:05:24.510
NANCY CARTWRIGHT: I recommend causal process, the tracing theories of change. I put that big long entity in front of them are two separate them you know again theories of change.

35
00:05:24.750 --> 00:05:31.350
NANCY CARTWRIGHT: Serving a lot, lot of functions and mean different things to different people. So, you know, here's a specific kind of thing.

36
00:05:32.100 --> 00:05:40.830
NANCY CARTWRIGHT: We're to talk about that I think is really helpful. In fact, my own view is it's almost necessary for program planning and evaluation.

37
00:05:41.730 --> 00:05:53.940
NANCY CARTWRIGHT: But that's the optimistic view right um and I also want to talk about constructing is and also how we use middle level causal principles as guides and the construction of them back and forth.

38
00:05:54.930 --> 00:06:06.960
NANCY CARTWRIGHT: So first step is to specify the overall middle level program theory. So what program is expected to do and why. So for educational

39
00:06:07.920 --> 00:06:16.590
NANCY CARTWRIGHT: Conditional cup cup for an educational conditional cash transfer program, the overall theory is something like this conditional

40
00:06:17.460 --> 00:06:24.210
NANCY CARTWRIGHT: Transfers cash transfers conditional on a children attending school tends to increase school enrollments.

41
00:06:24.900 --> 00:06:40.860
NANCY CARTWRIGHT: And that's because household spending their children's interests failing strong barriers to the contrary, and the cash overcomes the financial barriers. So there's the simple overall program theory and you want to specify it certainly going to help you later on.

42
00:06:41.970 --> 00:06:46.890
NANCY CARTWRIGHT: Then, second is to produce a step by step diagram. So I've got a three step one here.

43
00:06:48.510 --> 00:06:56.520
NANCY CARTWRIGHT: If you look in the documents, you know, I mean, there are many, many, many steps in between. And it's useful again it as many steps as possible because

44
00:06:57.090 --> 00:07:03.930
NANCY CARTWRIGHT: Every time there's a, you know, cause one step is supposed to be producing another producing producing other to get to the final outcome.

45
00:07:04.200 --> 00:07:14.010
NANCY CARTWRIGHT: On anything that goes along wrong along the way it's going to break that chain. So you want to actually have identified what those steps are so you can watch for them and predict for them.

46
00:07:14.910 --> 00:07:22.380
NANCY CARTWRIGHT: So here's one. Um, let me see if I can get your pictures out of the way so I can actually see up say

47
00:07:23.580 --> 00:07:26.730
NANCY CARTWRIGHT: Well transfers transfers.

48
00:07:27.810 --> 00:07:37.410
NANCY CARTWRIGHT: Are trans casual transfer the poor households, the transfers are labeled as education related and the conditionality is announced and that tends to that.

49
00:07:38.100 --> 00:07:56.820
NANCY CARTWRIGHT: Induces parents to try to enroll their children. And that gets supposed to the next level, nice, wonderful final step is more children are enrolled. Okay, so that's the second step is to do the step by step diagram. The third is to describe the causal principles at work at each step.

50
00:08:00.150 --> 00:08:01.620
NANCY CARTWRIGHT: You folks don't want to minimize

51
00:08:03.000 --> 00:08:11.970
NANCY CARTWRIGHT: Okay, so just looking at one step. The first step. I mean, we only had three steps in this diagram, the simple one, but

52
00:08:13.230 --> 00:08:25.260
NANCY CARTWRIGHT: One and one crime cooperate together to push towards to and the causal principle by which it does so it's not just an accident that if you do

53
00:08:26.670 --> 00:08:40.800
NANCY CARTWRIGHT: cash transfers and you label them. It's not just an accident that that induces that households to send their children to try to send their children to school. I mean, there's a reason for that. And that's expressed in the causal principles. So one in one prime

54
00:08:42.300 --> 00:08:46.080
NANCY CARTWRIGHT: Imply lead to to buy because labeling a good

55
00:08:47.910 --> 00:08:58.200
NANCY CARTWRIGHT: You know, leads to increase the increased importance of that good and that leads to the increased spending on it even absent a requirement to do so. So that's one concept principle, then

56
00:08:59.010 --> 00:09:08.040
NANCY CARTWRIGHT: There's another causal principal work that takes all three factors together and pushes towards. Step two.

57
00:09:09.120 --> 00:09:12.660
NANCY CARTWRIGHT: So conditioning the cash transfers on a known requirement.

58
00:09:14.160 --> 00:09:26.580
NANCY CARTWRIGHT: Of produces a cost to not meeting the requirement and that induces a substitution effect you know substitution effect means that if you increase the cost of a good that encourages choosing some other one

59
00:09:26.970 --> 00:09:40.650
NANCY CARTWRIGHT: And that leads to attempts to meet the requirement. So we now see that you know what the causal principles that often, as I said, simple behavioral ones or ones that are the result of social science research.

60
00:09:41.910 --> 00:09:55.080
NANCY CARTWRIGHT: But for each step, they'll there should be causal principles that are operating. I mean, that's the reason why one step should lead to the next. And you should be describing them okay now.

61
00:09:56.130 --> 00:10:11.220
NANCY CARTWRIGHT: The causal principles. The reason I'm so keen on them is because they play a central role in figuring out what information is needed for each step to occur. So that's what we're going to look at next. So the next thing I think it's useful to go on for

62
00:10:12.390 --> 00:10:15.240
NANCY CARTWRIGHT: This P talk process traits and

63
00:10:17.220 --> 00:10:27.300
NANCY CARTWRIGHT: Is to add support factors at that, um, you know, the actions that are cited you never say enough the actions that are cited are seldom enough to produce the influence expected

64
00:10:28.320 --> 00:10:35.070
NANCY CARTWRIGHT: They need lots of other factors to be in place in order for them to work. These are I call them support factors.

65
00:10:35.730 --> 00:10:43.740
NANCY CARTWRIGHT: And many other people do too. Um, there's sometimes called moderators or interactive variables. So here's an example.

66
00:10:44.550 --> 00:10:52.350
NANCY CARTWRIGHT: To get from one and one prime to to remember the principle is conditioning on conditioning cash transfers on a requirement.

67
00:10:53.100 --> 00:10:58.140
NANCY CARTWRIGHT: Will produce a substitution effect which will produce attempts to meet the requirement. That's the console principle.

68
00:10:58.470 --> 00:11:09.750
NANCY CARTWRIGHT: Now, not because of principle is only going to be able to operate if the conditions are both known and understood. So that's one of the sport factors that has to be in place that might not mentioned

69
00:11:10.800 --> 00:11:20.340
NANCY CARTWRIGHT: Also, there has to be a credible threat of enforcement or there won't be any belief in there won't be any substitution believe in a substitution effect. And so no attempt to meet the requirements.

70
00:11:21.390 --> 00:11:27.120
NANCY CARTWRIGHT: We didn't really look at two to three, but just, you know, for further illustration

71
00:11:27.690 --> 00:11:42.180
NANCY CARTWRIGHT: Know to is that the households are intend to enroll the children and three is that you get more moments and so that just assumes the simple principle that presenting a child for enrollment will lead it to be enrolled.

72
00:11:43.770 --> 00:11:52.980
NANCY CARTWRIGHT: That assumes that the school is competent to enroll students and that places are available. Um, and we have seen cash transfer

73
00:11:55.410 --> 00:12:10.620
NANCY CARTWRIGHT: Programs, where people a lot of effort was set up to pay people to send their children to clinics that didn't exist. So my team on some of these support factors might seem obvious, but it's worth rehearsing them so caution.

74
00:12:11.820 --> 00:12:26.250
NANCY CARTWRIGHT: As we know, chains are only as strong as the weakest link. So each step has to have it support play factors in place to produce the next step and if any support factors missing anywhere. The whole process collapses.

75
00:12:27.300 --> 00:12:36.030
NANCY CARTWRIGHT: Unless you can find a substitute for it. So I think understanding what the support factors are at each step is really important matter.

76
00:12:36.930 --> 00:12:50.850
NANCY CARTWRIGHT: The next thing is to consider what the retailer's there might be. And to add them in a derailleur, it's just a term I use for things that can intervene to prevent the effect. So things are going along fairly well you've got all the support factors in place and then

77
00:12:51.960 --> 00:13:06.330
NANCY CARTWRIGHT: Well, the child protection example that I use with any Monroe is everything's going on, fine. And then the mothers violent boyfriend moves back in. So in our, in our example, for instance, we might have that the household accepts the cash transfer

78
00:13:08.070 --> 00:13:14.610
NANCY CARTWRIGHT: The household understands that it's meant for sending children to school, they intend to do so and then

79
00:13:15.360 --> 00:13:26.070
NANCY CARTWRIGHT: The whaler that the household is into temptation spinning wheels to other priorities and that's something that can readily come along and the real the process, which seems to be

80
00:13:26.550 --> 00:13:39.510
NANCY CARTWRIGHT: Off to a good start. So, um, the next thing is to think about what kind of safeguards, you can have because there'll be a lot of the whalers hanging around in the background and also ones that you don't. You haven't even thought about

81
00:13:40.860 --> 00:13:46.560
NANCY CARTWRIGHT: I think of safeguards his walls that do their best to exclude the whalers so

82
00:13:47.850 --> 00:13:54.060
NANCY CARTWRIGHT: Here's something that is typically done mothers maybe are very often, or have in in certain

83
00:13:54.720 --> 00:14:12.390
NANCY CARTWRIGHT: Circumstances and settings more concerned about the education of their children and fathers and they may be less likely to give in to temptation spending or you on to other priorities. So if you were in those circumstances, a good safeguard is to give the cash transfers to mothers.

84
00:14:13.620 --> 00:14:20.010
NANCY CARTWRIGHT: So then we end up with a diagram that looks like this. At each step when you have all the causes.

85
00:14:20.760 --> 00:14:26.220
NANCY CARTWRIGHT: That all the things the support factors and Main calls that have to act together in a circle.

86
00:14:26.640 --> 00:14:35.610
NANCY CARTWRIGHT: And you throw the circle to indicate which ones actually have to be acting together and sometimes over overlapping circles, you know, because some support factors will help our one

87
00:14:36.600 --> 00:14:47.580
NANCY CARTWRIGHT: One cause push to that effect and also help another cause push to that effect. So, um, but it's important to have them all together to indicate that those that those are the ones that have to act together.

88
00:14:48.330 --> 00:14:57.390
NANCY CARTWRIGHT: Which is a different issue from you might have a totally separate causal principle work with some other group. And so we'd have independent

89
00:14:58.560 --> 00:14:59.340
NANCY CARTWRIGHT: Causal groups.

90
00:15:00.600 --> 00:15:07.050
NANCY CARTWRIGHT: And you, I'd like to do the directors with these jagged edges just indicate that they're going to come in and

91
00:15:08.100 --> 00:15:17.190
NANCY CARTWRIGHT: A bit like a saw and cut the process and then walls to just one way of picturing these you can also just do it by

92
00:15:18.870 --> 00:15:23.460
NANCY CARTWRIGHT: Verbally, but I think having a diagram really helps. OK.

93
00:15:24.480 --> 00:15:32.970
NANCY CARTWRIGHT: The next thing that I think it's important to do is to allow for causal loops at some time it's sometimes we forget these and

94
00:15:35.340 --> 00:15:44.130
NANCY CARTWRIGHT: Sometimes we're lucky here. They work to our advantage. But often, often when there are negative causal those. I'm going to give you an illustration of a positive one.

95
00:15:45.000 --> 00:15:55.620
NANCY CARTWRIGHT: So, as I say, they could be positive or negative. But let's look at a positive one as more children are enrolled in school that it almost inevitably will lead to

96
00:15:56.190 --> 00:16:02.970
NANCY CARTWRIGHT: Children being withdrawn. Now, when the children are withdrawn that's supposed to lead to the cash transfers being stopped for them.

97
00:16:04.170 --> 00:16:15.360
NANCY CARTWRIGHT: Now if you couple that with a to helping factors or support factors that there's wide notice of this and and an understanding of the reason that that's the reason that the cash transfer was thought

98
00:16:15.870 --> 00:16:23.370
NANCY CARTWRIGHT: Then that will lead to an increase credibility of the threat to stops the cash transfers of the conditions aren't met

99
00:16:23.850 --> 00:16:39.990
NANCY CARTWRIGHT: And that will lead to produce an incentive to keep children in school and that should keep more children enrolled. So here we have a positive causal Luke. You're all familiar. I'm sure you've encountered many, many negative one. So it's just important to

100
00:16:41.400 --> 00:16:45.150
NANCY CARTWRIGHT: Think through what kind of causal loops. There might be in the offing.

101
00:16:46.830 --> 00:16:47.340
NANCY CARTWRIGHT: Now,

102
00:16:48.390 --> 00:16:54.330
NANCY CARTWRIGHT: Something we're all familiar with the need to do as best we can specify the expected range of application.

103
00:16:55.350 --> 00:17:17.100
NANCY CARTWRIGHT: Where should this middle level P talk apply and you can get indicators for this. One of the reasons for having done all the work and steps one through seven. You can get indicators for where it should work from the overall program theory, the individual middle level causal principles.

104
00:17:18.150 --> 00:17:25.710
NANCY CARTWRIGHT: And the supporters, the retailers and safeguards each one of these gives you a clue as to where the

105
00:17:26.640 --> 00:17:33.630
NANCY CARTWRIGHT: The middle level P talk will apply. So think about the range of application. The program is going to be risky anywhere.

106
00:17:34.020 --> 00:17:43.440
NANCY CARTWRIGHT: That the middle level principles don't work, and I've got middle level there with an exclamation point, because I will remind me. These are middle level they're ones that hold some places and all the others.

107
00:17:44.220 --> 00:17:47.220
NANCY CARTWRIGHT: Lot of cultural principles are very widespread.

108
00:17:47.790 --> 00:17:56.670
NANCY CARTWRIGHT: That are things that are widespread in Britain and America are quite different and other cultural settings. So there's a lot of middle chemicals or principles and you have to think

109
00:17:56.940 --> 00:18:08.730
NANCY CARTWRIGHT: Which ones will hold where and which ones need triggering so the program won't work where wherever any of the middle level principles that are necessary for it. Don't apply won't work.

110
00:18:09.390 --> 00:18:20.940
NANCY CARTWRIGHT: If support features are missing. As we noted anywhere if you can substitute for them and they're necessary to get the next step, you're not going to get the final step if you don't get any of the middle ones you miss any of the middle ones.

111
00:18:21.900 --> 00:18:28.800
NANCY CARTWRIGHT: And it won't the programs risk anywhere that unguarded the retailers are likely. So, so this is

112
00:18:29.400 --> 00:18:37.320
NANCY CARTWRIGHT: Sort of illustrating the negative, it's a lot easier to say where it won't apply them where it will, but you know when you can get

113
00:18:37.770 --> 00:18:57.360
NANCY CARTWRIGHT: Are fairly reassured that the middle level principles necessarily will hold and the support factors are in place and that you've set up walls against the whalers. Those are places where the programs are likely to work. So that's just think, for instance, about what you can learn

114
00:18:58.530 --> 00:19:08.760
NANCY CARTWRIGHT: From one of these, the overall middle level program theory. Remember that that theory was that conditional cash transfers remove financial barriers.

115
00:19:09.270 --> 00:19:22.380
NANCY CARTWRIGHT: So conditional educational cash transfers won't work. We're households don't send their children to school because they think it's low quality don't trust it or disapprove of school education.

116
00:19:23.280 --> 00:19:41.340
NANCY CARTWRIGHT: Rather than because they lack resources and again a simple example. But once you put all these simple lessons together. I think have much more powerful case and we often do support factors, the retailers and safeguards, as an example, um,

117
00:19:42.390 --> 00:19:56.400
NANCY CARTWRIGHT: Remember that as support factors that transfers or low cost, the recipients in school places are available. Well, the educational CCTV CC T program won't work where there's no way to achieve these

118
00:19:57.600 --> 00:20:18.570
NANCY CARTWRIGHT: Okay, so that was to try to specify where and why it might white working and we're not ninth step joy implications for evaluation questions and monitoring and evaluation indicators. So let's look at evaluation questions first.

119
00:20:19.680 --> 00:20:22.080
NANCY CARTWRIGHT: What about the availability of school seats.

120
00:20:24.150 --> 00:20:29.790
NANCY CARTWRIGHT: The beneficiaries understanding of the conditions are conditions monitored and enforced.

121
00:20:30.810 --> 00:20:34.830
NANCY CARTWRIGHT: Finding ex post that essential features we're missing.

122
00:20:35.790 --> 00:20:47.460
NANCY CARTWRIGHT: Is evidence that the program had little responsibility for the results, even if the expected outcome occurred. So sometimes something else produces the outcome, what you want to know is how much did this program contribute

123
00:20:48.180 --> 00:20:58.860
NANCY CARTWRIGHT: And if essential features on that also be necessary. Along the way, are missing, then that gives you reason to think the program wasn't very responsible for the outcome.

124
00:21:01.140 --> 00:21:11.670
NANCY CARTWRIGHT: So we can also you should also try to draw implications for monitoring and evaluation indicators. So for instance, the number of intended beneficiary is participating

125
00:21:12.150 --> 00:21:27.870
NANCY CARTWRIGHT: The parental knowledge and understanding of the scheme, the verification of receipt or the transfer how well conditions are monitored and enforced and measure the outcomes on school and Walmart and attendance. I guess that's the obvious one that everyone does do

126
00:21:28.920 --> 00:21:31.380
NANCY CARTWRIGHT: Okay, so we have indicators at various levels.

127
00:21:32.610 --> 00:21:45.240
NANCY CARTWRIGHT: And this is important to remember back we at the very beginning, these, these theories middle level theories right there's high theory very precise theory middle level theory.

128
00:21:45.780 --> 00:21:58.230
NANCY CARTWRIGHT: So theories come at various levels and even within a level, you know, they have gradations so indicators, then we'll also mirror that they'll come at various levels of abstraction.

129
00:22:00.060 --> 00:22:14.340
NANCY CARTWRIGHT: Of abstraction versus concreteness, and wide ranging versus narrow range. And so in particular I'm particularly concerned about these process tracing theories of change because

130
00:22:15.390 --> 00:22:21.690
NANCY CARTWRIGHT: I think it's really necessary to try to have them at two levels. The first is

131
00:22:22.680 --> 00:22:41.220
NANCY CARTWRIGHT: If you're selling a program broadly, then you need a middle level P talk that's mental apply across a range of settings. And that means that you'll probably be using fairly abstract language and highly, highly abstract causal principles.

132
00:22:42.270 --> 00:23:02.940
NANCY CARTWRIGHT: But you know, that's got to be made concrete as we saw in the early slides about the balsa program. It really has to be concrete ised in some way and operationalize on if we're going to build a real program and real place. So we need our local P talk for specific setting now.

133
00:23:04.680 --> 00:23:20.700
NANCY CARTWRIGHT: The middle level P toxic just general types of monitoring and evaluation indicators and these can be adapted and concrete eyes to specific context through and we use this word thickening but to get from the more abstract more general theory to the more concrete one

134
00:23:22.140 --> 00:23:31.230
NANCY CARTWRIGHT: You thick and then you can also take all these indicators and and taking them so that they become more concrete and operationalize double

135
00:23:32.550 --> 00:23:33.300
NANCY CARTWRIGHT: So 10th.

136
00:23:35.040 --> 00:23:48.930
NANCY CARTWRIGHT: Step is to draw implications for future program design and middle level P talks help with applying evaluation findings from one setting to another. That's the middle level ones do that. Okay.

137
00:23:49.380 --> 00:23:57.600
NANCY CARTWRIGHT: I think they do it this way, how the findings in an observed setting. So you have if you if you're looking right you've got a

138
00:23:58.860 --> 00:24:02.670
NANCY CARTWRIGHT: Concrete pretty good local level.

139
00:24:03.810 --> 00:24:05.940
NANCY CARTWRIGHT: theory of change for that case.

140
00:24:08.040 --> 00:24:20.820
NANCY CARTWRIGHT: You from from that with a lot of theorizing, you come up with a more abstract more general middle level P talk that's supposed to be you expect to be

141
00:24:21.480 --> 00:24:36.990
NANCY CARTWRIGHT: To hold a more more settings in the original one. When you've got that middle level P talk, you can then concrete is it right to a new local setting. So I think of the process of getting from

142
00:24:38.400 --> 00:24:49.620
NANCY CARTWRIGHT: Finding is an observed setting to findings in a new setting, known as being a horizontal one but as being this up, down, one that you really want to develop from the one

143
00:24:50.700 --> 00:25:04.980
NANCY CARTWRIGHT: That si P talk works in the observed setting you really want to be able to create something that it's a more general understanding of why that's working the way it is. And then you can use that general understanding to

144
00:25:05.730 --> 00:25:12.270
NANCY CARTWRIGHT: Devise local ones, which, you know, then may or may not be able to obtain in the setting, you're looking at

145
00:25:13.500 --> 00:25:21.090
NANCY CARTWRIGHT: So implications of findings. Remember, we're going up and down to to get them. So consider finding in the observed setting.

146
00:25:23.190 --> 00:25:34.920
NANCY CARTWRIGHT: You notice that households didn't take part because transfers were made to bank accounts and household didn't have bank accounts. Well, simple middle level theory when extracts from that with

147
00:25:36.330 --> 00:25:39.750
NANCY CARTWRIGHT: Not, you know, not a lot of other theorizing. Is that

148
00:25:40.890 --> 00:25:51.600
NANCY CARTWRIGHT: Well, we seem to have lost it that transfers of should be made in a way that's accessible. Ah, in a way that's accessible to people.

149
00:25:53.640 --> 00:25:55.680
NANCY CARTWRIGHT: And let me just see where we are here.

150
00:25:57.270 --> 00:25:59.700
NANCY CARTWRIGHT: I think we are just about

151
00:26:01.140 --> 00:26:02.070
NANCY CARTWRIGHT: Yes, done.

152
00:26:03.330 --> 00:26:04.530
NANCY CARTWRIGHT: Final advice.

153
00:26:05.820 --> 00:26:06.330
NANCY CARTWRIGHT: Okay.

154
00:26:11.940 --> 00:26:14.460
NANCY CARTWRIGHT: Okay. They oh I saw it. Of course, it was obvious.

155
00:26:15.600 --> 00:26:21.600
NANCY CARTWRIGHT: The new settings, the time in place of transfer needs to be convenient and accessible for recipients there. Then we have final advice.

156
00:26:23.010 --> 00:26:35.640
NANCY CARTWRIGHT: to advise for two groups of people for those designing programs for repeated use please build a middle level P talk that can be hard. It needs, probably some new theorizing and research.

157
00:26:36.360 --> 00:26:41.850
NANCY CARTWRIGHT: But don't talk the job because I think it's not fair to leave it to local decision makers.

158
00:26:42.570 --> 00:26:50.880
NANCY CARTWRIGHT: Who are generally in a less good position to figure out, you know, all those bits and pieces that have to be in place at the program processes to carry through there.

159
00:26:51.780 --> 00:27:01.860
NANCY CARTWRIGHT: So I think that program designers really have the job of assisting by providing maximal information and then for those who are deliberating at the local level.

160
00:27:03.030 --> 00:27:14.850
NANCY CARTWRIGHT: Well, I think the best really good thing to do is to build a good local level P talk. That's what you ought to be doing if your decisions are to be both reliable and credentialed

161
00:27:16.830 --> 00:27:20.850
NANCY CARTWRIGHT: How it doesn't allow me to use the philosophical work warranted. If there's to be good.

162
00:27:22.230 --> 00:27:25.500
NANCY CARTWRIGHT: sufficiently good reasons to believe in them. Okay.

163
00:27:26.250 --> 00:27:34.560
NANCY CARTWRIGHT: A good middle level to talk is an invaluable starting point for this right so you have a good going to build a local one you're going to have a big, big advantage if

164
00:27:34.980 --> 00:27:45.750
NANCY CARTWRIGHT: If the program designer has provided good middle level theory and you need to just figure out what those concepts and steps a month to in your setting and then see whether or not you can get that pair.

165
00:27:46.620 --> 00:28:03.720
NANCY CARTWRIGHT: So you want to thicken the middle level one to fit local circumstances that probably requires a great deal of local knowledge and that may take additional research and a final reminder is it's always a good idea to do this with key stakeholders. So thank you.

166
00:28:10.710 --> 00:28:22.380
Campbell Collaboration: I'm stressing the people's we have time for some questions and answers. I was just in people stick their hands up like that. You can't see the hand raising symbol and my bar at the bottom.

167
00:28:24.180 --> 00:28:28.800
Campbell Collaboration: But you can also send questions in the chat or so you'd like to ask a question.

168
00:28:30.120 --> 00:28:32.160
Campbell Collaboration: And there's a very long question just come in.

169
00:28:34.380 --> 00:28:35.340
Campbell Collaboration: And answer you can you

170
00:28:36.660 --> 00:28:40.740
Campbell Collaboration: You can click on the chat simply at the bottom. Your said

171
00:28:42.090 --> 00:28:43.800
Campbell Collaboration: I'll read the first one. The first one from here.

172
00:28:44.700 --> 00:28:47.010
NANCY CARTWRIGHT: And see on the bottom or at the top.

173
00:28:47.670 --> 00:28:48.030
The

174
00:28:49.320 --> 00:28:49.800
Campbell Collaboration: Bottom.

175
00:28:51.420 --> 00:28:53.370
NANCY CARTWRIGHT: Monica to everyone at my bottom

176
00:28:54.000 --> 00:28:54.420
Okay.

177
00:28:55.920 --> 00:28:57.690
NANCY CARTWRIGHT: Okay, well, I'll just listen.

178
00:28:58.230 --> 00:29:00.120
Campbell Collaboration: Oh, exactly. So

179
00:29:01.650 --> 00:29:08.850
Campbell Collaboration: This being Goldman, so I'm trying to relate this to normal work on developing theories of change. And it seems to me, adding some rigor.

180
00:29:09.150 --> 00:29:16.560
Campbell Collaboration: And the low level is a theory of action in common parlance is preventive action components. So the question is one standard

181
00:29:16.830 --> 00:29:29.970
Campbell Collaboration: In those correctly. We often get the question was demonstrating this and a normal conventional theory of change approach and is just implies some level of rigor and then we'll call it a lower level is what he calls a theory of action or theory from action.

182
00:29:30.750 --> 00:29:32.670
NANCY CARTWRIGHT: Okay, basically.

183
00:29:33.990 --> 00:29:37.260
NANCY CARTWRIGHT: What I like to say is, I mean, what means what it says.

184
00:29:38.430 --> 00:29:38.730
NANCY CARTWRIGHT: So,

185
00:29:39.840 --> 00:29:48.420
NANCY CARTWRIGHT: Then I looked. We looked through and found lots and lots of you folks and people in other areas using

186
00:29:48.630 --> 00:30:00.090
NANCY CARTWRIGHT: theory of change in a huge variety of different ways. So I'm not able to tell you how contrast with the usual sense of theories of change, because that depends on what you know group you're in

187
00:30:01.980 --> 00:30:18.720
NANCY CARTWRIGHT: I'm perfectly happy that what at the lower level, it's often what would be a theory of change. I think this has more details in it and more specific kinds of details than theories of change often do

188
00:30:20.640 --> 00:30:34.860
NANCY CARTWRIGHT: So it seemed. I mean, so it's not. And also, as I say, this is not new total news. This is drawn from a lot of standard work on what you're already doing plus standard work on what you know on causation.

189
00:30:35.460 --> 00:30:45.150
NANCY CARTWRIGHT: That involves, you know, understanding that there's got to be a causal process and support factors. So I think I'd rather like to avoid

190
00:30:46.080 --> 00:30:54.450
NANCY CARTWRIGHT: Questions of language and is it like this or like that, you know, if I had a specific case in front and we could look and see how it differs from this and not watered ads.

191
00:30:55.470 --> 00:30:58.890
NANCY CARTWRIGHT: But it does intend to add detail and

192
00:31:00.540 --> 00:31:08.610
NANCY CARTWRIGHT: Is meant to be rigorously based on you know there's a reason to introduce support factors, because that's part of

193
00:31:09.330 --> 00:31:21.960
NANCY CARTWRIGHT: You know, standard really good theories of causality that causes 10 salient causes tend not to operate alone, but really there's a whole set of necessary factors that have to operate together acceptor, etc.

194
00:31:23.760 --> 00:31:34.350
Campbell Collaboration: So saw something that said in her presentation that well, where's where's the middle because there's a top is the bottom as the middle and she's, it doesn't really matter.

195
00:31:34.950 --> 00:31:41.460
Campbell Collaboration: Just put it wearing suits you have your purpose and i i Italian simple I'm telling sympathetic to that so

196
00:31:42.090 --> 00:31:51.330
Campbell Collaboration: Don't get to hop around the antique issues and also just stress that when neither prescribing know proscribing any approach.

197
00:31:51.660 --> 00:32:00.870
Campbell Collaboration: mid level theory, something's being tried outings failed, and then I open to see, well, now let 1000 flowers bloom. Let's try different ways to

198
00:32:01.410 --> 00:32:17.160
Campbell Collaboration: Discover this and see what seems to work. And so, not to my presentations. I'll things we thought about and thinking about him. We have to abide by Patrick financing, we should we have a helpful to you. I do have a different take on it then feel free to

199
00:32:18.090 --> 00:32:25.980
NANCY CARTWRIGHT: Let me also add part of the point though talking about middle level theory and making a big fuss about it when it comes to the causal principles.

200
00:32:26.280 --> 00:32:46.080
NANCY CARTWRIGHT: Is that I think people have generally found some kinds of theory, not very helpful. And so I want to point out that this isn't things like grumpy and Marxist theory, this is rather pretty low level stuff that you actually know how to make yourself.

201
00:32:47.190 --> 00:32:47.370
Yeah.

202
00:32:48.450 --> 00:32:50.460
Campbell Collaboration: So thanks nexus from guilt on

203
00:32:51.600 --> 00:33:01.890
Campbell Collaboration: So he says the causal diagram I want to present it. You've got to basically simplified limited boxes and arrows that you need to be essential ones. So it actually echoes

204
00:33:02.670 --> 00:33:15.300
Campbell Collaboration: Lobby quote I like very much news a lot. Whenever I can from Joe Robinson economic philosophy. When she says a map of it. So it was it a model that tries to replicate reality is as useful as a macro scale one to one. So,

205
00:33:16.170 --> 00:33:25.110
Campbell Collaboration: He said to just simplify stuff and so girl says so. How is it you make a difference between support factors and do it. But importantly, really, to get in there.

206
00:33:26.700 --> 00:33:41.310
Campbell Collaboration: But obvious and those are not obvious, but much more important. So how do you organize the processed reduced number conditions of factors in a group setting. So these different stages, you've got our design. Okay, I'm going to mention this one not mentioned that one.

207
00:33:42.720 --> 00:33:46.740
NANCY CARTWRIGHT: That really does seem to me that it depends on knowing how important

208
00:33:47.490 --> 00:33:52.110
NANCY CARTWRIGHT: It is that this or how likely this factor is this fail if you don't mention it.

209
00:33:52.440 --> 00:34:03.990
NANCY CARTWRIGHT: So, I mean, there's a lot of just generally the philosophers examples is you don't normally mentioned that there's oxygen in the air when you're talking about lighting a match, right, what you mentioned is that the matches and down.

210
00:34:05.580 --> 00:34:15.420
NANCY CARTWRIGHT: What makes for those differences and what makes for the differences is an evaluation of both how important this factory is because many factors.

211
00:34:16.230 --> 00:34:25.650
NANCY CARTWRIGHT: If this one fails. There's just another one in the background that would do just as well and it's gonna, it's actually they are to so many of the factors have

212
00:34:26.790 --> 00:34:36.210
NANCY CARTWRIGHT: Backup in the background. Many of them are very prevalent and you don't need to worry about them and the others. Then if you don't mention them.

213
00:34:37.620 --> 00:34:42.870
NANCY CARTWRIGHT: And there's a good chance they'll fail. He seems to me you didn't do your job.

214
00:34:46.080 --> 00:34:46.290
NANCY CARTWRIGHT: So,

215
00:34:47.370 --> 00:34:53.940
NANCY CARTWRIGHT: You got to make it simple enough for people to do. But if you make it too simple, and you leave stuff up. That's like we'd actually matter.

216
00:34:56.880 --> 00:34:58.050
Campbell Collaboration: Susie wants to say something.

217
00:34:58.140 --> 00:35:06.900
Susie Burdett: Sorry, I had realized the settings hadn't allow people to write in the chat that they should be able to now. So if you had a question. You can type now.

218
00:35:08.250 --> 00:35:18.810
Campbell Collaboration: We'll be able to handle, but I'll try and pick out one soon difference to what's already been our supporters by diminished your question, you do think it was substantially different. We will be able to see the nature on

219
00:35:19.890 --> 00:35:24.600
Campbell Collaboration: So this one's from streaming ROI. So it's in the middle of a theory examples, getting the beginning

220
00:35:24.960 --> 00:35:39.990
Campbell Collaboration: Simple statements full sentences like is a principle, but then in the process causal tracing have a whole diagram. So what's the relationship between the two is middle theory just a sentence summarizes the whole diagram or is it what what is it

221
00:35:40.920 --> 00:35:52.050
NANCY CARTWRIGHT: Um, yes. So that's a whole program theory is usually a short version that assumes that look a lot of the steps in between a pretty automatic

222
00:35:53.220 --> 00:36:03.030
NANCY CARTWRIGHT: So, it usually explains why you think this initial push off cause has is you know in many circumstances likely to lead to the effect

223
00:36:03.840 --> 00:36:14.610
NANCY CARTWRIGHT: Then the rest of it is filling in. Well, exactly what are the circuit steps by which will do so because even though this looks like as a fairly widespread.

224
00:36:15.240 --> 00:36:20.040
NANCY CARTWRIGHT: general principle that you can see instantiated in a variety of places.

225
00:36:20.730 --> 00:36:29.760
NANCY CARTWRIGHT: There are a lot of places. It's not. And there are a lot of places you might expect it to be that it's not so the, the, it's a more detailed theory of I think of

226
00:36:30.360 --> 00:36:42.750
NANCY CARTWRIGHT: Of, you know what's going to happen between the cause and the effect that is often does happen. And that's why the overall program theory is, you know, a general one and and can be stated shortly.

227
00:36:43.740 --> 00:36:55.560
NANCY CARTWRIGHT: These steps often happen and can often be relied on what are they, because if you're putting program in a new place, you have to worry about whether they actually will happen and so

228
00:36:56.730 --> 00:36:59.130
NANCY CARTWRIGHT: Anyway, that's my view of the relation between them.

229
00:37:00.240 --> 00:37:12.450
Campbell Collaboration: So you're going to like this question from Matthew and say he says in john area work. There's very few actual cause of this gospel principles are well established apparently there are lots of them. They're all just experts opinion.

230
00:37:14.040 --> 00:37:17.160
Campbell Collaboration: You judge the causal principle is actually warranted.

231
00:37:17.970 --> 00:37:18.570
NANCY CARTWRIGHT: Ah,

232
00:37:19.110 --> 00:37:19.800
Campbell Collaboration: I find like it.

233
00:37:21.840 --> 00:37:34.020
NANCY CARTWRIGHT: And in particular, because a lot of the causal principles are warranted are likely to hold in some places and not others. So is it worth recording

234
00:37:37.170 --> 00:37:45.750
NANCY CARTWRIGHT: The more interesting question is when is it warranted likely to be warranted here and there, most of the ones that get floated

235
00:37:46.440 --> 00:38:06.120
NANCY CARTWRIGHT: Are perfectly sensible middle level million mill john Stuart Mill Creek type tendency principles. Yeah, I mean that's something that happens, it's part of human nature. It's a part of, you know, a common part of social structure. The real issue is what makes it likely to hold here, right.

236
00:38:07.740 --> 00:38:12.570
NANCY CARTWRIGHT: You know what, why is there any reason to think that this is what will go on here rather than there I'm

237
00:38:13.290 --> 00:38:24.480
NANCY CARTWRIGHT: Just your Elster does have a lot of work on these middle level causal principles which he calls mechanisms and again mechanism is used in 1000 different ways so

238
00:38:25.080 --> 00:38:34.320
NANCY CARTWRIGHT: I always call them Oscar mechanisms but Elster talks a lot about you know how they get warranted. But as I say, he doesn't talk about the issue that really matters is

239
00:38:35.190 --> 00:38:42.510
NANCY CARTWRIGHT: A lot of them. There's positive research that they hold sometimes and not others. And you really have gone on to something. That's a fact.

240
00:38:43.110 --> 00:38:57.930
NANCY CARTWRIGHT: Sometimes about human nature or it's a disposition that we have or disposition that institutions have but that disposition isn't 100% widespread it needs triggering on it. May it it's instantiate some different things in different ways.

241
00:38:59.880 --> 00:39:04.350
Campbell Collaboration: And thanks, Nancy nationals mobile transition opposite to what reporters over crossed

242
00:39:05.700 --> 00:39:14.310
Campbell Collaboration: So I started to she's wondering whether the mid level process tracing cool because the process and transplant president and that will be talks.

243
00:39:16.230 --> 00:39:22.830
Campbell Collaboration: Whether they're simply the boundary objects as described by Susan the star in Jesus Christ, man. So I'm not familiar with that.

244
00:39:23.130 --> 00:39:35.580
Campbell Collaboration: And the point of there is a portable between different disciplines and settings as you as you stretch yourself. And so they have potential to become standardized or stylized factual stylized was principles we accept us okay now we know that

245
00:39:36.480 --> 00:39:44.730
Campbell Collaboration: Or if you are the things that should keep evolving as new pixels come along and feed into them and get the principal evolves over time.

246
00:39:45.930 --> 00:39:47.490
NANCY CARTWRIGHT: Well, I don't think those are

247
00:39:48.600 --> 00:39:49.860
NANCY CARTWRIGHT: I don't think those are

248
00:39:51.060 --> 00:39:59.550
NANCY CARTWRIGHT: Watch McCullough contradict each other. They are boundary objects. I mean, I would classify them as boundary objectively star used to be a colleague of mine.

249
00:40:00.690 --> 00:40:02.700
NANCY CARTWRIGHT: So by half. I happen to know

250
00:40:05.130 --> 00:40:12.030
NANCY CARTWRIGHT: That they are boundary objects but boundary objects can and cannon. I mean, they get stabilized.

251
00:40:13.350 --> 00:40:30.240
NANCY CARTWRIGHT: But things that get stabilized also evolve me they're fairly stable. They're agree they're accepted like electrons, which means something very different now than what they meant 20 years ago from what they meant 50 years ago, but they were fairly stabilized, a long time ago.

252
00:40:31.470 --> 00:40:32.700
NANCY CARTWRIGHT: And their boundary objects.

253
00:40:33.450 --> 00:40:37.500
NANCY CARTWRIGHT: I would say maybe at this point. These are bound to objects. Yes.

254
00:40:38.550 --> 00:40:41.220
Campbell Collaboration: Next question. No.

255
00:40:45.690 --> 00:40:58.530
Campbell Collaboration: Okay. So he's asking about thickening and so you're talking about thickening he understands it's being sort of increasing the complexity of the they've changed it moves down from a middle level to a specific level and

256
00:40:59.220 --> 00:41:11.640
Campbell Collaboration: In a middle level theory or summarizing across different classes of intervention. And his question is, what constitutes a class. So in your presentation, you have conditional cash transfers being a class of interventions as far as you've got

257
00:41:12.240 --> 00:41:24.150
Campbell Collaboration: Transferred strict conditionality as well monsoon post and you have another one in which conditionality as we go portmanteau new forced other silver still the same class interventions. How do you decide what constitutes class of interventions.

258
00:41:24.840 --> 00:41:38.970
NANCY CARTWRIGHT: I think you will only decide when you know what affects you want to produce and how much information you want to convey about how those effects will be produced. So

259
00:41:39.870 --> 00:41:51.150
NANCY CARTWRIGHT: It's a very likely that conditional cash transfers for education will have a lot of different causal principles work at at various steps.

260
00:41:51.780 --> 00:42:10.230
NANCY CARTWRIGHT: Than once for sending kids to clinics to get vaccinated. And if you really want to be helpful for people who are setting up a conditional cash transfer program for vaccination. You want to classify more narrowly

261
00:42:11.220 --> 00:42:28.260
NANCY CARTWRIGHT: But if you, you know, you're just trying to sell the idea that conditional cash transfer explain why conditional cash transfers might be useful, and that there's actually some steps that will happen in the middle you can lump them all together, the more concrete, you get in your

262
00:42:29.730 --> 00:42:41.940
NANCY CARTWRIGHT: middle level theory, the more helpful it is to the you know the local designer. But of course, the less generalizable, the less of general use. It is. It's just a trade off.

263
00:42:45.540 --> 00:42:56.040
Campbell Collaboration: To Ryan has a couple of questions. I'm just going to ask one of them right now, but I'm going to talk to them myself. Are you Nancy I deliberately making a distinction between causal principles and tools and mechanisms, and if so, what is it,

264
00:42:57.240 --> 00:43:09.420
NANCY CARTWRIGHT: I don't use the word causal mechanism. I'm by no principles also using a very a variety of ways, but I don't use the term mechanism because it has lots of different meetings and

265
00:43:10.950 --> 00:43:20.250
NANCY CARTWRIGHT: So y'all Elster uses mechanism for the terms that I'm calling middle level. I mean, he thinks middle level causal principles.

266
00:43:20.820 --> 00:43:33.750
NANCY CARTWRIGHT: Pick up mechanisms. I think they normally pick up social and institutional and individual dispositions, but I just try to avoid the word because, I mean, so if you could elaborate you know what

267
00:43:34.380 --> 00:43:40.320
NANCY CARTWRIGHT: You know what he has in mind by mechanism. I can then say whether causal principles, a mechanism or not.

268
00:43:43.380 --> 00:43:44.340
Campbell Collaboration: Thanks. I'm

269
00:43:44.550 --> 00:43:49.890
NANCY CARTWRIGHT: Going to be difficult there. It's just that they're just is quite different things meant by them.

270
00:43:50.760 --> 00:43:57.900
Campbell Collaboration: So James can stack is asking, why does time come in as most the time dimension in the in the SI

271
00:43:58.980 --> 00:43:59.970
Campbell Collaboration: P talk approach.

272
00:44:02.400 --> 00:44:11.190
NANCY CARTWRIGHT: Well, you know, you start at the beginning. And it takes a lot of times for all these steps to happen and it does matter that you should have some I mean by

273
00:44:12.420 --> 00:44:19.230
NANCY CARTWRIGHT: sketching out the steps I think it often helps you figure out whether or not you could achieve the result in the time expected

274
00:44:20.400 --> 00:44:33.930
NANCY CARTWRIGHT: So that I mean the canonical example here was the California class size reduction program where the theory is that if you reduce class size and they had the money to do so educational

275
00:44:35.970 --> 00:44:42.450
NANCY CARTWRIGHT: Outcomes will improve. And it might well have worked. I mean, there's a lot of debate about this. In general, but it might well have worked

276
00:44:43.470 --> 00:44:47.370
NANCY CARTWRIGHT: If all the steps could have been fitted in in their proper order.

277
00:44:48.450 --> 00:44:56.520
NANCY CARTWRIGHT: But, you know, there wasn't enough time for the program to be announced the schools to be able to

278
00:44:58.560 --> 00:45:11.850
NANCY CARTWRIGHT: announce that they wanted to hire, because of course you need twice as many teachers. If you have twice as many classes, etc, etc. So the, the time was very important there and mapping out exactly what the P talk would look like.

279
00:45:12.450 --> 00:45:24.750
NANCY CARTWRIGHT: Would make it absolutely clear that that you couldn't get the results in time and actually people. Some people did that it's they went ahead with the program anyway because it was so popular.

280
00:45:25.950 --> 00:45:39.870
Campbell Collaboration: And thanks, last question. I have a question that I tend to ask them all from really so you've said that this is something you think program managers really ought to do that is do their job properly. So how do we how do we make that happen.

281
00:45:41.700 --> 00:45:42.480
NANCY CARTWRIGHT: I don't have a clue.

282
00:45:45.210 --> 00:45:52.410
NANCY CARTWRIGHT: But if I had a little control over funding a program, you know, I keep pushing the people to think harder and harder about it.

283
00:45:55.350 --> 00:46:06.960
NANCY CARTWRIGHT: I think actually talking about it more does help them. I've been working with Eileen Monroe works on signs of safety and child protection and now she's got quite keen on

284
00:46:08.550 --> 00:46:18.150
NANCY CARTWRIGHT: Making the program design more detailed like this just on account of the the until she feels that sort of intellectual force behind the approach.

285
00:46:18.720 --> 00:46:27.510
NANCY CARTWRIGHT: But you know, that's appealing intellectually. I have no idea how you set up social structures to to encourage it. If you think it's a good thing.

286
00:46:29.490 --> 00:46:35.730
Campbell Collaboration: Thank you, Nancy. A lot of questions as two more questions in the chat. Maybe. Soon as you mentioned, we have a note of them.

287
00:46:36.150 --> 00:46:52.800
Campbell Collaboration: And Nancy my film and she wants some leisure leisure is look at them and save them useful to reflect upon we're going to turn out my own presentation, which is primarily about how the setup projects are planning to use mid, mid level theory.