/ Par / by Sebastian Schienle

Are control groups ethical? An example from our work in Uganda

Randomized controlled trials are an essential tool for us to measure the impact of the programs we support. They should be for development programs in general.

Randomized controlled trials (RCTs) are called the gold standard of impact evaluations in development projects, because they are able to identify causal relationships between development interventions and the benefits achieved for recipients. Yet, one of the challenges of impact measurements through RCTs is that they require control groups, i.e. a group of individuals or households that is similar to the recipients of the program, but who do not receive the program’s benefit themselves. We are regularly asked whether such a setup is, in fact, ethical and whether it can be morally justified to exclude certain people from a program.

Very simply put, the answer in our view is "yes". In almost all cases, both in international development and in the work of non-profits in the developed world, resources are limited and insufficient to cover every individual in need.

Our cash transfer program with GiveDirectly in Uganda reached 3,400 households across 44 villages in the Iganga district. The program and accompanying RCT were set up as follows:

  • GiveDirectly gave transfers to the poorest 20% of households in these villages (based on a census of households at the beginning of the program and eligibility criteria developed by GiveDirectly - for more details, see the comprehensive Pre-Analysis Plan), but did not include them in the randomization or the study.
  • The wealthiest 35% of households did not receive any transfers and were not included in the RCT.
  • Between the ~20th and ~65th percentiles, GiveDirectly then individually randomized, within the village, matching households, so that for each household in the village that received transfers, there was a control household of the same poverty level in the same village who did not receive a transfer.

Even without the accompanying RCT, GiveDirectly would have had to make a choice which villages and households to enroll, thus excluding others who would certainly also have benefitted from the program. We do acknowledge, however, that members of the control group are different from others who haven’t been enrolled in the program in two main aspects:

  1. By design, they are very similar to program participants and would have very likely been included in the program had it been implemented without the RCT.
  2. They are more closely involved in the program than others, through surveys and other evaluation activities, as program outcomes and impacts are tracked both for program beneficiaries and members of the control group.

Both aspects require some level of compromise. On the first one, an RCT often requires the project area to be slightly larger in order to include the control group. Leaving aside the financial costs of an RCT for a moment, the targeted number of 3,400 recipient households remained constant - i.e. households in the control group were “replaced” by others who also matched the program’s selection criteria. However, due to the matched pairs in our RCT, some households in the control group who did not receive transfers were slightly poorer than some beneficiaries of the program. This means that as a result of the RCT, some households received transfers who would not have been part of the program without the RCT, while some others who would have otherwise been included did not receive any cash.

The second aspect can also be challenging. Even if they wouldn't have been program beneficiaries without the RCT, control group members are, after all, in close contact with the researchers. But that’s precisely why academic studies require ethical approval to protect participants in both the treatment and control groups. (I.e. those actively participating in a project and those monitored to compare results with.)

So whether control groups can be justified really boils down to two considerations:

1. What value does the RCT generate?
If the RCT brings about new insights into how impactful and cost-effective a development intervention really is, we believe both the added cost of the RCT and the (possibly negative) impact on members of the control group who do not benefit from the program aren’t just ethical, but are necessary. Necessary to inform funding decisions in such a way that the dollars we and others invest to help people really make a difference.

2. Can the impact on the control group be minimized?
While we believe that the use of RCTs and control groups are justified overall, smart research design can address concerns over the impact on members of the control group even further and should be considered when RCTs are set up.

In the case of our cash transfer program, GiveDirectly and research partner IDInsight decided to exclude the poorest 20% of households from the RCT, as explained above. While this limits what can be said about this particular group of participants in the RCT, we felt this is an appropriate compromise to avoid that some of the poorest members of a community are excluded from immediately receiving a cash transfer.

In addition, we decided to include all 1,800 households of the control group in a second wave of the program after the conclusion of the RCT and to offer them the same cash transfers of ~$1,000 that their neighbors had received a year earlier.

In the past, GiveDirectly has often used proxy indicators (like the type of houses people own or their roofing materials) to identify which households to enroll. While not all households may have agreed with the selection criteria, they provided a recognizable distinction between program recipients and other households in the community. In the case of an RCT, the random assignment to either the treatment or control group blurs this distinction. As a result, the cash transfers rather became a win in the lottery – or, as many of the villagers perceived it, a blessing or punishment from God. Accordingly, we felt it was the right and necessary thing to go back and ensure control households move from the perception of being punished to being blessed.

We continue to believe that RCTs are a very helpful and necessary tool to enable funders to direct their resources towards proven interventions that deliver real impact. While some concerns over the effects on the individuals in control groups are justified, many of them can be addressed or at least minimized. The endline survey of our RCT with GiveDirectly was wrapped up in June, and we look forward to the insights we can learn from the results once they become available later in the summer. In the meantime, control group households are being enrolled to receive their transfers between July and September 2018 so that in the end, everyone benefits: 5,200 impoverished households in Uganda who will have been “blessed” with $1,000 in cash; GiveDirectly who will gain additional insights into how to further improve the effectiveness of their program; and us and other funders who obtain additional evidence on the impact that cash has on so many lives.