How I Learned To Stop Worrying and Love SDG 16

A few weeks back, I posted a skeptical commentary about the integration of anticorruption into the new Sustainable Development Goals and associated targets, in particular Target 16.5 (“substantially reduce corruption and bribery in all their forms”). Rick was even harsher. The premise of most of my criticism (and Rick’s) was that progress on Target 16.5 was likely to be measured using changes in countries’ scores on Transparency International’s Corruption Perceptions Index (CPI). It turns that this premise was (probably) incorrect.

I had based my assumption on the lengthy report released last June by the Sustainable Development Solutions Network (SDSN)—a report which had been commissioned by the UN’s Inter-Agency Expert Group on SDG Indicators (IAEG-SDG). But as Transparency International Senior Policy Coordinator Craig Fagan helpfully pointed out in his comment on Rick’s post, the more recent official information released by IAEG-SDG in September 2015 does not indicate that the CPI will be used as the principal measure for Target 16.5. Rather, the IAEG-SDG document lists as the proposed indicator the “percentage of persons who had at least one contact with a public official, who paid a bribe to a public official, or were asked for a bribe by these public officials, during the last 12 months.” (The relevant material is on page 225.) This still isn’t finalized, but it certainly appears that the IAEG is poised to endorse an experience/survey-based measure for Target 16.5, rather than the CPI-style perception index.

Is this perfect? No, certainly not. But it’s a lot better than what I’d feared. A few further thoughts on this:

  • First, some might object to this sort of experience-based measure as too narrow. And indeed it’s true that there’s a bit of a mismatch between the broadly-stated target (reducing corruption and bribery in all their forms) and the proposed measure, which at best only measures certain forms of petty bribery. But this doesn’t bother me that much. It’s usually better, in my view, to adopt a narrower target that one can actually measure than to try to capture a whole host of disparate phenomena in a single quantitative index. And even though petty bribery is not the only, or even the most damaging, form of corruption, it’s certainly important. Perhaps the title of the target should be adjusted (though that may not be possible at this stage). But the fact that we might ultimately be unable to measure reductions in corruption and bribery in all their forms doesn’t seem like a good argument against measuring reductions in corruption and bribery in some of their forms.
  • Second, a more serious concern—which Rick pointed out in his response to Craig’s comment—is that there are a number of quite important methodological concerns about these experience surveys. The most significant of these is that people do not always answer honestly when asked about illicit activities like corruption. That by itself might not be so serious, except that the tendency to be candid about such matters may vary substantially across countries/cultures, and even within countries may be correlated with characteristics of the respondents in ways that bias the results. There’s quite a bit of research now that confirms that this is a problem (see here, here, here, here, and here). Rick points out (rightly, in my view) that this makes it extremely difficult to do cross-country comparisons using experience surveys: If 20% of respondents in Country X say they paid a bribe within the last year, and only 10% of respondents in Country Y say that they did so, this could mean that bribery is more widespread in Country X, but it could also mean that respondents in Country X are more willing to admit paying bribes, even if the actual bribery rates are the same (or even if bribery is actually more prevalent in Country X). But I’m less convinced than Rick that this is a fatal problem for using experience surveys to measure within-country progress toward Target 16.5. If the other factors that might affect willingness to answer truthfully remain constant over time within individual countries, then changes in reported bribery experience over time within a single country may still be meaningful, even if comparisons across countries are not meaningful. In this regard, experience surveys and the CPI can perhaps be thought of as opposites in terms of what they can or can’t be used for: for all its flaws, I think the CPI can still be used to get a (rough) sense of cross-country variation in (perceived) corruption levels, but within-country changes in the CPI are not meaningful; with corruption experience surveys, I tend to think that within-country changes can provide useful information, but cross-country comparisons do not.
  • Third, though, the preceding optimistic assessment of the utility of over-time, within-country changes in bribery experience survey results is based on the premise that there are no other within-country changes are affecting the willingness to respond truthfully to bribery questions. But that’s not necessarily the case. For example, suppose a government launches an aggressive anticorruption campaign, including a significant educational component. This could cause self-reported bribery to decrease, even if actual bribery does not change, if the campaign increases the shame associated with corruption. Alternatively, such a campaign could also increase self-reported bribery (again, even if actual bribery is not changing), if, for instance, the campaign causes people to realize that certain payments that they thought were official charges are actually unlawful bribes, or if the campaign makes people feel empowered to complain openly about bribe demands. To my knowledge, as yet we have very little research examining the factors that may affect willingness to answer corruption experience questions, so we would need to be very cautious when interpreting results until we know more.
  • Fourth, and related to the above points, it should probably go without saying that in some countries it will be very difficult to conduct such surveys, or to have any confidence in the results.

Still, although there are a number of serious issues that still need to be addressed, I tend to think it would be a good thing if we got more data, on a yearly basis from a large number of countries, about individual/household/firm-level experience with bribery. As long as we interpret that data cautiously, it might indeed provide useful information. And if the adoption of SDG 16 and Target 16.5 provides an impetus to invest more in gathering such data (and to scrutinize it more carefully, and to pay more attention to concerns like some of those sketched above), then that would be all to the good. So while I still find myself unable to muster the enthusiasm for SDG 16 that I observe among many in the anticorruption community (I still don’t get why people are referring to this as a transformative, watershed, breakthrough moment), I’m much more optimistic that Target 16.5 won’t do harm, and might actually do a modest amount of good. I’m grateful to Craig for correcting my original error, and I look forward to seeing what the IAEG-SDG actually ends up doing.

7 thoughts on “How I Learned To Stop Worrying and Love SDG 16

  1. Matthew, very nice discussion. I realized after posting my note that so long as the non-reporting bias was constant within country, comparisons over time would not be affected. But I had forgotten about the evidence that non-reporting may be correlated with income, education, and other variables. So changes in the percentages of those who say they paid a bribe will have to control for these factors. But like you, even given all these caveats, the petty bribery surveys are probably the best we have. Or at least can afford to conduct on any kind of regular basis.

    Two countries where it will be interesting to watch the impact of SDG 16 are China and India. In their February 2015 volume Paying Bribes for Public Services, Richard Rose and Caryn Peiffer report that neither country has surveyed citizens about their experience with petty bribery. The new anticorruption government in Delhi just did (“One-third of Delhi households paid bribe last year,” The Hindu, October 20, http://www.thehindu.com/news/cities/Delhi/onethird-of-delhi-households-paid-bribe-last-year/article7781563.ece) but I have not seen any other state surveys nor a nationwide one. Nor have I seen any survey of Chinese citizens. Both governments were actively involved in the development of the SDGs. Does this mean we can look forward to seeing such surveys in the near future?

  2. Pingback: How I Learned To Stop Worrying and Love SDG 16 | Anti Corruption Digest

  3. It is heartening that SDG 16 will use experience-based and not perception indicators as the primary tool for measuring corruption. There is increasing evidence for large divergence between corruption levels identified by perception based studies vs. experience-based studies. However, there could be following issues in largely relying on experience based studies:

    1. Logistics: Experience-based studies will have large resource requirements and operational complexities involved in conducting surveys of sufficient scale,especially reaching out to the poor citizens.

    2. Petty corruption vs. grand corruption: Assuming that experience-based survey elicit unbiased and non-evasive replies from citizens, obtaining similar responses from businesses and investors will be difficult. To get a comprehensive view of corruption in the country, we may still need to rely on perception based indices.

  4. I agree that measuring a subset of the problem is far better than not focusing on the problem or not measuring at all (though we need to be careful that this does not draw resources away from fighting other types of corruption that are hard to measure but important to tackle; in the case of the MDG about primary school attendance there is some evidence that countries over-focused on the measurable enrollment, and under-prioritized teacher quality and whether students were really learning much of anything.)

    With regard to your methodological concerns, I agree that they are reasons to be wary, but I think there are ways to get people to report more honestly on anonymous surveys. A good example would be the tactics Professor Daniel Corstange at Columbia used when he wanted to determine how many people had traded their electoral vote for cash. Like corruption, vote-selling could be associated with shame (or fear of punishment) and could be hard to get people to answer honestly. Therefore, he asked a control group “How many of the following have you done in the past 12 months,” followed by a list of roughly ten innocent activities. Another group was asked the same question, but vote-selling was added to the list along with the same activities the other group was asked about. That way, by measuring the uptick between the two groups, you could calculate how many people sold their votes without asking anyone to admit, even anonymously, that they definitely sold their votes.

    • Yes, great point. In fact, I was thinking of writing a post at some point specifically on this approach (sometimes referred to as an “unmatched count technique” (UCT), or simply as the “list technique”), and its potential application to corruption experience surveys. I’m not familiar with the Corstrange paper (I’ll look it up right away), but I’ve seen the Duke economist Ed Malesky and his collaborators make use of this approach in a couple of very interesting papers about Vietnam. But I think it’s still a rarity to use this approach (as opposed to asking about “people like you/firms like yours”) as a way to address the concern about respondents’ reluctance to admit to illegal or unethical activity.

  5. Pingback: Innovative or Ineffective?: Performance-Based Lending as an Anticorruption Tool | GCNI

  6. Pingback: The U4 Proxy Challenge and the Search for New Corruption Indicators | Anti Corruption Digest

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s