In my post last week, I fired off a knee-jerk reaction to Transparency International’s latest Corruption Perceptions Index (CPI). My message of that post was simple and straightforward: We shouldn’t attach much (or perhaps any) importance to short-term changes in any individual country or region’s CPI score, and the bad habit of journalists—and to some extent TI itself—of focusing on such changes is both misleading and counterproductive.

Since I was trying to get that post out quickly, so as to coincide with the release of the CPI, I published it before I’d had a chance to read carefully all of the material TI published along with the new CPI, and I promised that once I’d had a chance to look at those other materials, I would follow up if I had anything else to say. I’ve now had that chance, and I do have a few additional thoughts. The short version is that the way TI itself chose to present and discuss the implications of the 2018 CPI, in the accompanying materials, is both better and worse than I’d originally thought.

So, first, the bad news:

TI’s main press release, its “global analysis,” and its separate “regional analysis” summaries, are even sloppier than I’d originally realized in making bold and mostly unsubstantiated claims about what we learn from the new CPI numbers—even though, I hasten to add, I do agree with most of their bottom-line conclusions.

TI seems to have decided that the big headline stories that we should all take away from this year’s CPI are, first, that efforts to control corruption are, with only a very few exceptions, failing; second, that—again with only a few exceptions—democracy is needed to reduce corruption; and third, that corruption weakens democracy. These points may or may not be true, but the CPI data doesn’t really establish them in the way that TI’s press release and global and regional reports suggest.

Let me start with the issue about whether corruption is staying the same or getting worse in most countries. My last post emphasized that movement (or non-movement) in the CPI score over a relatively short period doesn’t tell us much of anything. I don’t want to belabor that point here, but I do want to emphasize just how bad these TI reports are on this. The opening sentence of the global analysis document is that the 2018 CPI sows that, with only a few exceptions, “most countries are failing to make serious inroads against corruption.” Well, no, the data doesn’t show that, because this sort of data is not well-suited to detecting even substantial improvements over a relatively short time-frame. And as for the issue of which countries are getting better and worse: To its credit, this year TI calculated which countries’ CPI score changes from previous years were statistically significant at conventional levels, and included this list its downloadable data sets. And in the global analysis of the 2018 CPI, TI only mentions that 36 countries have seen statistically significant changes since 2012 (20 improvers and 16 decliners). There are still reasons to doubt whether even statistically significant changes are substantively meaningful, and we shouldn’t treat the absence of a statistically significant change as evidence that no change took place, but this is still progress, so credit where credit is due. But then in its regional analysis pieces, TI proceeds to completely disregard its own careful attention to the statistical significance of various changes. To wit:

• In its Americas analysis, TI emphasizes CPI improvements in Argentina, Ecuador, and El Salvador, and declines in Chile, Mexico, and Nicaragua. The latter three have indeed seen statistically significant declines since 2012, but the improvements in the former three are not statistically significant.
• In its Asia Pacific analysis, TI highlights improvements in Vanuatu, the Solomon Islands, and South Korea, as well as declines in Bangladesh, Maldives, Timor-Leste, and Vietnam. None of these changes is statistically significant by TI’s own calculations.
• In its Eastern Europe and Central Asia analysis, TI emphasizes declining scores in Azerbaijan, Kazakhstan, Kosovo, Montenegro, Russia, and Serbia. Yet none of these countries’ CPI scores changed significantly since 2012, according to TI’s data.
• In its Middle East/North Africa analysis, TI pointed to score improvements in Egypt and Morocco, even though neither country’s score change was statistically significant. This report also highlighted score declines in Libya, Syria, and Yemen; the latter two countries have indeed seen statistically significant score drops since 2012, though the former hasn’t.
• In its Sub-Saharan Africa analysis, TI highlighted two countries, Cote d’Ivoire and Senegal, that have indeed seen statistically significant score improvements since 2012. But the report also focused on score improvements in three other countries—Eritrea, Gambia, and Seychelles—that are not statistically significant by TI’s own calculations. This report also notes five countries that saw their CPI scores decline: Burundi, Congo, Ghana, Liberia, and Mozambique. Of these five, TI calculates that three of the declines (Congo, Liberia, Mozambique) are statistically significant, while the other two (Burundi, Ghana) are not.
• In its Western Europe analysis, TI notes that the UK has improved over time, and indeed the UK has seen a statistically significant improvement in its CPI score since 2012. However, TI also makes much of the fact that between 2017 and 2018 the UK’s score dropped by a couple of points, even though this change is nowhere near statistically meaningful. TI also points to four countries that have seen CPI declines (Bulgaria, Hungary, Malta, Romania), but in only one of these countries (Hungary) has the drop since 2012 been statistically significant.

It may seem like I’m nitpicking, and perhaps that’s true. But these observations underscore the fact that even as TI has gotten both more methodologically sophisticated and more careful in some aspects of its CPI presentation, other parts—especially those intended for more widespread media consumption—haven’t changed, and continue to try to squeeze out of the data evidence that isn’t really there.

That same basic point applies to the other big headlines that TI wants to emphasize this year, concerning the relationship between corruption and democracy. As noted above, TI wants to make two points here. First, TI argues, in the opening words of its global analysis of the 2018 CPI, that the new CPI “reveals that the continued failure of most countries to significantly control corruption is contributing to a crisis in democracy around the world.” Second, TI also wants to argue that democracy is key to controlling corruption, with TI’s new Chair, Delia Rubio, stating in the main press release that TI’s “research makes a clear link between having a healthy democracy and successfully fighting public sector corruption,” because “[c]orruption is much more likely to flourish where democratic foundations are weak”—a sentiment echoed in several of the regional reports as well.

I’m certainly sympathetic to both of these arguments, but the claim that the CPI demonstrates their veracity is a vast overstatement. At best, the data shows a correlation, one that is actually much more complicated than even a simple linear regression might suggest. (This is something I’ve written about before—see this chapter.) And TI’s global report makes the frankly misleading claim that “[t]here are no democracies that score below 50 on the CPI.” That’s a bizarre claim, because the countries that score below 50 on the CPI include India, Argentina, Brazil, Indonesia, Greece, the Philippines, Peru, and South Africa, all of which are usually considered democracies. The explanation is that TI is using, for its “democracy” classification, a scheme developed by the Economist Intelligence Unit that differentiates “full democracies” from “flawed democracies,” and considers, in making this distinction, not only the freedom and fairness of elections, but also broader considerations like how well the government functions.

The bigger point here is that in its press release, global analysis, and regional analyses, TI seems to want to rely on fairly simply correlations between CPI scores and democracy scores to make strong claims about causal relations between democracy and corruption—claims that may well be true, but might not be, at least not entirely, and in any event don’t follow automatically and straightforwardly from the data.

But that brings me to the good news, which is that, along with the press release, the global analysis, and the regional analysis, TI has also included a “research analysis” (co-authored by Coralie Pring and Jon Vrushi), using the new CPI as a springboard to discuss more generally the relationship between corruption and the apparent “crisis of democracy” in many countries. (Full disclosure: I saw and provided some comments on an earlier and very different draft of this document.) This research analysis is rather brief, but it’s overall quite good. I don’t agree with everything in it, there are a few places where I would have added some qualifications, and the statistical analysis is very simple–though still much better than anything in the other TI documents accompanying the CPI. But overall this essay is a thoughtful, nuanced discussion that is well worth reading. The main takeaway points, in the authors’ own words, are as follows:

• when corruption seeps into the democratic system, corrupt leaders may seek to prevent democratic checks and balances so that they can continue to remain in power unpunished

• countries which recently transitioned to democratic governance often did not develop effective anti-corruption and integrity mechanisms, and now find themselves stuck in a cycle of high corruption and low performing democratic institutions

• some populist leaders who have come to power by capitalising on public disgust with corruption, ironically, now seek to undermine anti-corruption mechanisms and democratic institutions.

The authors use the CPI when developing these arguments, and as noted above they do a very simple linear regression (with one control variable, for economic/human development), but they don’t over-rely on the simple data analysis to develop their larger point. Indeed, this report is commendable—and a notable contrast to the other TI documents in this collection—in its modesty, noting, for example, that the linear regressions showing a negative relationship between democracy scores and the CPI are “very simple and not sufficient to explain whether corruption leads to democratic decline or whether democratic decline leads to more corruption,” and noting elsewhere that the academic literature is divided “on whether democracy is a necessary condition in the fight against corruption.” But despite these qualifications, the authors muster a number of examples to show how, in many countries, “corrupt leaders have undermined democratic institutions in order to protect themselves from prosecution and to keep stealing state resources.” And they also emphasize—rightly in my view—that one of the reasons that democratization hasn’t achieved the corruption-control benefits that many had hoped for (and that TI’s other documents seem to imply) is that in many newly democratized countries, the absence of effective anticorruption measures during the transitional period led to a situation in which “intense partisan competition … [led] to higher rates of corruption as new political parties promise[d] state jobs, contracts and other resources to their potential supporters,” which in turn “may have contributed to how little progress has been made in these countries to improve the quality of their democracies.”

Long story short, if you read one thing from TI about the 2018 CPI (besides that data itself), look at the research analysis. Admittedly, it isn’t really about the CPI—it could have been written as a standalone document, frankly—but it’s still useful and insightful, and the care and caution with respect to drawing bold conclusions from the limited quantitative data is something TI would do well to emulate in its other materials.