Today, Transparency International released its new Corruption Perceptions Index (CPI) for 2018. At some point, hopefully soon, I’ll have time to look closely at the new data and accompanying materials, and if I have something to say about it, I’ll post it here. But that will probably take a while, and since the media coverage of the CPI is usually pretty intense in the first few days after the release, and dissipates in a week or two, I wanted to get out at least one post right now, on the day of the release, with a plea to everyone out there–especially journalists, but civil society activists and others as well:

DO NOT COMPARE ANY GIVEN COUNTRY’S CPI SCORE TO LAST YEAR’S SCORE TO MAKE CLAIMS ABOUT WHAT’S HAPPENING IN THE FIGHT AGAINST CORRUPTION.

Just don’t do it. Don’t. I know the temptation can seem overwhelming. Who’s up? Who’s down? Things are getting better! Things are getting worse! Nothing is changing! So many stories can be written based on these changes (or non-changes).

But these sorts of comparisons are virtually all completely useless, and probably counterproductive.

In past years I’ve dwelt at length on the reasons one can’t usually learn anything useful from comparing a country’s CPI score in a given year to its score in previous years. (See, for example, here, here, here, here, here, and here.) In brief, score changes can sometimes be driven by technical changes (such as changes in the underlying sources used to calculate the scores), and there’s so much statistical noise in the CPI estimates that small changes are often not statistically meaningful–and even for those changes that are statistically significant at conventional levels, one would expect that in a sample of 180 countries, some would exhibit “statistically significant” changes even if it were all random noise. More substantively, perceptions of corruption are slow to change, and influenced, at least in the short term, by factors other than the underlying level of corruption, such as whether a major scandal has gotten a lot of press coverage.

TI itself has gotten a bit better in how it presents and frames CPI changes, and is certainly better than a lot of the journalists and others who focus on small, statistically meaningless changes in scores or rankings as if they told us something useful. But TI continues to be a bit careless about this, especially in the materials intended for media consumption. For example, in this year’s CPI press release, TI emphasizes the fact that Brazil’s score dropped two points, from 37 to 35, even though by TI’s own analysis we don’t know whether this was a real change or just statistical noise, and if it was indeed a worsening it may well be the result not of worsening corruption but of exposure of corruption that took place in previous years, when Brazil’s score was (somewhat) higher.

Moreover, the title of TI’s press release, “Corruption Perceptions Index 2018 Shows Anti-Corruption Efforts Stalled in Most Countries,” is just wrong, and in a way that matters:

• First, because there’s so much statistical noise, the fact that most countries haven’t exhibited a statistically significant improvement doesn’t mean things haven’t improved. That’s just an inherent problem with noisy data: small movements might not mean anything, because it might just be noise, but at the same time the noise in the data may prevent us from detecting genuine improvements.
• Second, and perhaps more important, anticorruption efforts take time to be effective, and it would be a mistake to write them off as a failure just because they don’t produce an immediate, detectable improvement in a country’s CPI score. I know TI means well here–they want to emphasize the seriousness of the problem and the urgency of devoting more energy and resources to combating corruption. But as we’ve noted on this blog previously (see, for example, here and here), interpreting a lack of change in the CPI as evidence that anticorruption efforts aren’t working can be counterproductive: They can be demoralizing, and breed more cynicism and fatalism, attitudes which themselves make corruption harder to control. Now, if the CPI data really did show conclusively that anticorruption efforts aren’t working, then the fact that spreading that message might have demoralizing effects might not be a good argument for suppressing or downplaying this truth. But since a lack of statistically significant changes in the CPI data does not clearly show that anticorruption efforts have “stalled,” putting this spin on the data is a rhetorical choice, and I fear a misguided one.

I’ve said it before and I’ll say it again (and again and again and again, until the message gets through): the CPI is useful for many things, but making year-to-year comparisons in any individual country’s score is not one of them.