Last week, the United Kingdom’s Independent Commission for Aid Impact (ICAI) released its report on the UK Department for International Development (DFID)’s efforts to fight corruption in poor countries. The report, which got a fair amount of press attention (see here, here, here, and here), was harshly critical of DFID. But the report itself has already been criticized in return, by a wide range of anticorruption experts. Heather Marquette, the director of the Developmental Leadership Program at the University of Birmingham, described the ICAI report as “simplistic,” “a mess,” and a “wasted opportunity” that “fails to understand the nature of corruption.” Mick Moore, head of the International Centre for Tax and Development at the Institute for Development Studies, said that the report was “disingenuous[]” and “oversimplif[ied],” and that it “threatens to push British aid policy in the wrong direction.” Charles Kenny, a senior fellow at the Center for Global Development, called the report a “wasted opportunity” that “has failed to significantly add to our evidence base,” largely because “ICAI’s attitude to what counts as evidence is so inconsistent between what it asks of DFID and what it accepts for itself.”

Harsh words. Are they justified? After reading the ICAI report myself, I regret to say the answer is yes. Though there are some useful observations scattered throughout the ICAI report, taken as a whole the report is just dreadful. Despite a few helpful suggestions on relatively minor points, neither the report’s condemnatory tone nor its primary recommendations are backed up with adequate evidence or cogent reasoning. It is, in most respects, a cautionary example of how incompetent execution can undermine a worthwhile project.

The report is far too long (and too convoluted) to try to address every flaw. Let me focus on what I take to be the report’s three main criticisms and associated substantive recommendations. (To clarify, I’m not counting vague and general admonitions to “articulate and implement a detailed plan,” without specifying the content, as a genuine recommendation worth engaging.) Those three main points are as follows:

1. The ICAI report criticizes DFID for focusing on governance, accountability, and transparency generally, rather than designing programs to target corruption specifically.
2. The ICAI report recommends that DFID shift more of its attention to petty bribery and youth anticorruption education.
3. The ICAI report recommends that DFID implement targeted and ongoing “micro-surveys” of stakeholders and intended beneficiaries to assess the effectiveness of its programs, modeled on the surveys that ICAI itself conducted when assessing the effectiveness of various DFID programs in Nigeria and Nepal.

I won’t spend too much time on the first recommendation, except to note that it’s not self-evidently correct. I personally tend to be sympathetic to the claim that corruption should be viewed as a standalone problem, but there is a plausible view out there that has maintained the opposite: that anticorruption should be viewed as part of a more holistic “good governance” agenda, which should be framed in more positive terms–promoting virtues like transparency and accountability–rather than in terms of detecting and punishing misbehavior. ICAI just asserts that DFID is making a mistake in focusing on governance generally rather than anticorruption specifically, without ever acknowledging that there’s a debate about this, much less giving a cogent argument as to why anticorruption should be treated as a standalone goal. But as this seems to be the least meaningful of ICAI’s criticisms, let me turn to the other two.

The ICAI report repeats, over and over and over again, that DFID’s programs are not sufficiently focused on “the poor.” I lost count of the number of times I saw some version of that criticism in the report. Even when acknowledging what would seem like positive aspects of DFID’s anticorruption activities — like supporting the fight against money laundering, promoting transparency in the extractive sector, and improving anticorruption law enforcement institutions — the ICAI report invariably follows up on this half-hearted praise by saying something like “But DFID has not focused on how all this will help the poor.” (I’m paraphrasing, but there’s language like this throughout). I eventually figured out (I think) what ICAI wants: (1) More focus by DFID on petty, retail corruption, of the sort that poor people are most likely to experience directly in their daily lives; and (2) more DFID-supported efforts “to educate young people about the ill effects of corruption” in order to promote “a self-initiated cultural shift away from corruption.” A few reactions here:

• The idea that fighting money laundering, promoting transparency in the extractive sector and elsewhere, improving the effectiveness of anticorruption enforcement agencies, and similar activities is not relevant (or is only marginally relevant) to “the poor” is wrongheaded. As many have pointed out, corruption is often systemic, frequently requiring a holistic and multifaceted response, and also the types of corruption that poor people are most likely to encounter directly are often embedded in systems supported by grand corruption. So to dismiss so many of DFID’s projects as not relating to the problems of “the poor” is just silly.
• ICAI also says at several points that there is “no evidence” that DFID’s programs have alleviated the corruption problems that the poor experience. In a very limited sense that’s true. We don’t have very good methods for measuring, for example, how stepped-up enforcement of anti-money laundering rules will reduce the amount of public money diverted from public health or education programs to officials’ private bank accounts. But there are nonetheless good reasons to believe that there is likely to be a positive impact, and in fact ICAI acknowledges at several points evidence that DFID programs have had a positive effect on intermediate outcomes (say, convictions or asset seizures) that are plausibly related to the ultimate outcome we care about. Moreover, ICAI itself acknowledges that anticorruption interventions must operate on longer timescales — 10-15 years, according to ICAI — and attributing causation will always be difficult, so it seems unfair and inconsistent to suggest that programs DFID is already implementing, which would have to operate on similar time scales, do not seem to be helping the poor given the existing evidence.
• Related to the previous point, despite criticizing DFID for an absence of evidence, ICAI offers no evidence whatsoever that its proposed interventions, such as its youth educational initiatives (or related suggestions to “leverage emerging technologies and social networks that have a possibility to bring about behavior change on a mass scale”) will have their desired effect. As I’ve pointed out before, there’s actually fairly little evidence on the impact of educational interventions–which is why we may want to consider randomized trials. But despite ICAI’s complaint that we lack evidence that DFID’s programs actually help the poor, all we have in favor of the particular interventions that ICAI thinks are more appropriate is ICAI’s ipse dixit. (And that’s when we can even figure out exactly what ICAI thinks DFID should be doing. Usually we don’t even get that — just a blanket statement that more should be done to “specifically target the everyday corruption experienced by the poor” without explaining what, programmatically, that would mean.)
• More generally, even if we can all get behind the idea that petty corruption is an important problem, that still doesn’t tell us where DFID’s relative emphasis should be. ICAI’s report suggests — perhaps intentionally, perhaps not — that DFID should shift its focus from things like extractive industry transparency, improving courts and prosecutors, and fighting money laundering to, instead, focusing on youth education and campaigns to end petty bribery. That’s not obviously correct, and indeed the weight of opinion among anticorruption experts seems to be moving in the opposite direction: toward the view that petty corruption, though destructive, is often only a symptom of deeper systemic problems, and that grand corruption–say, the theft of public resources from government programs–is much more harmful to the poor than the petty bribes that people may need to pay in everyday transactions, even if the latter are more visible. Now, maybe ICAI isn’t saying that DFID should do less of what it’s currently doing, but rather that it should add more initiatives specifically focused on petty bribery. Maybe. But (a) that’s hardly the tenor of the report, which mostly says things like, DFID is doing X, there’s no evidence X helps the poor, DFID should do Y; and (b) in an environment of limited resources, tradeoffs are inevitable, so a useful assessment would need to make some comparative judgments about priorities.

What about the third main recommendation, that DFID should improve learning about its programs by soliciting continual feedback through low-cost surveys of the sort that the ICAI’s contractors conducted in Nigeria and Nepal about DFID’s programs there? I’m all for more information-gathering and regular feedback, and stakeholder survey can often be valuable. But this was perhaps the most infuriating part of the ICAI report, because the actual surveys that ICAI’s contractors conducted were so transparently incompetent, and such a clear waste of resources, that it’s a bit galling for ICAI to point to them as a model of what DFID ought to be doing. I’ve already gone on for longer than I’d intended in this post, so let me leave Nepal to one side and just run through the Nigeria surveys that ICAI holds up as a template for DFID to follow:

• The first survey was supposed to assess how well a DFID-supported police training program succeeded in reducing police corruption. The ICAI contractors surveyed 500 randomly selected residents in each of five districts, four of which had the training program and one of which (the “control”) had not. The respondents were asked whether the police asked for bribes more often or less often than they did two years ago. The survey found most people thought things were getting worse, and there was no statistically significant difference between the one control district and the four others. This was actually the best of the ICAI surveys, in that it conducted a random survey of a large population and had (apparently) a high response rate. But for that reason, it is not an example of the sort of thing that can be done continuously and at low cost, as ICAI claims. Moreover, the results don’t actually tell us what we need to know (or what ICAI claims they tell us), for several reasons: (1) the treatment and control groups were not randomly selected, and there is only one control, so comparisons are likely invalid; (2) self-reported perceptions may not correspond to reality; (3) a training program may well take time to have an effect on behavior, which may in turn take time to have an effect on perceptions.
• To assess a DFID-funded prosecutor training program, ICAI sent an online survey to prosecutors, and got 68 responses (a response rate of 18%). That on its face is a reason to be skeptical of any results–it’s a small number of respondents, probably not randomly selected. (Elsewhere in the report, ICAI asserts that “[w]here the response rates [to surveys] were low … their usefulness to corroborate or challenge our other findings was given appropriate weight,” but in fact the report seems to ignore that caveat, at no other point even drawing attention to low response rates.) There is also no control group for this survey. And although some of the questions seem relevant, others (such as whether members of different anticorruption agencies are now working together more effectively) seem to have little to do with the nature of the training program.
• Likewise, a survey of investigators who went through a DFID-sponsored training program had a low (and almost certainly non-random) response rate of 27% (48 total responses), but that didn’t stop ICAI from reaching sweeping conclusions like: “[T]he overall impact of [DFID’s] efforts has been limited.” That pessimistic conclusion, by the way, appears not to be based on the actual survey responses, which show that over 90% of the 48 respondents reported better understanding of how corruption cases should be investigated and that the quality of the evidence in such cases had improved, and about 85% of respondents reported that more cases had been brought to trial since the training.
• If you thought the sample sizes for the previous two surveys were bad, the survey of judges consisted of only four judges (contacted via email)–yet, amusingly, ICAI still felt the need to represent their responses in the form of a bar graph. And the question the judges were asked was whether “the overall corruption environment in Nigeria has improved over the past three years.” Three of the four judges said no–which the intrepid ICAI researchers seem to treat as evidence that DFID’s training programs for prosecutors and investigators aren’t working. I assume I don’t need to explain why that’s silly, right?
• OK, this may be my favorite: To get a sense of whether a DFID-supported program to increase transparency in the oil sector was effective, ICAI’s contractors sent a survey via Twitter (I swear I’m not making this up) to 2,591 people who had participated in an online discussion associated with the program, and received 36 responses. Most researchers would be so discouraged by a 1.4% response rate that they would give up. But not ICAI! The report dutifully indicates (with bar graphs!) how many of these 36 Twitter users have participated in a campaign or joined an organization related to the oil sector, and how many think the Nigerian government is planning in advance what the country will do when the oil revenue has dried up. From these percentages, ICAI infers…. actually, I’m not sure what ICAI thinks we’re supposed to learn from this information, and they never tell us in the report. But apparently DFID should be doing more surveys like this, to improve “learning”!

There’s really not much point in going on and on about this. The level of incompetence evidence in this report is just stunning. This is not to say that DFID’s approach is the right one. I’m sure it has problems, maybe deep problems, and could use significant improvement. But the ICAI report doesn’t help much in figuring out what those problems are.