As most people who follow this blog are likely aware, Transparency International released the 2015 Corruption Perceptions Index (CPI) last week. There is, of course, a lot to talk about here, and I’m sure many commentators and scholars will spend a lot of time poring over the new data and debating its significance. Given my previous criticisms of the CPI’s suggestion that scores for the same country can be compared across time (see here, here, here, and here), that was naturally the first thing I focused on. I was hoping that TI might take up some version of my suggestion to report statistical confidence intervals in an easy-to-see place in the main data table, or, even better, test for statistically significant changes in scores across years. Alas, TI didn’t do either of those things. (The confidence intervals are still available, but you have to download the data to find them.) TI did, however, report that since 2012, some countries had improved, while others had deteriorated. In particular, TI noted three countries (Greece, Senegal, and the UK) had improved their CPI scores since 2012, while five countries (Australia, Brazil, Libya, Spain, and Turkey) had seen a notable worsening.

Because of last year’s fiasco with China (where TI emphasized a decline in China’s CPI score that turned out to be bogus), I was initially skeptical. So, I went ahead and implemented the procedure that I outlined in my post from a few months ago to see whether, for these eight countries, there really was a genuine, statistically meaningful change in the CPI score. I was pleasantly surprised to discover that in all eight of the countries that TI identified, the change in the CPI score between 2012 and 2015 was indeed statistically significant at conventional levels, and do not seem to have been driven by the addition or subtraction of sources in the later year, or by a large anomalous jump in a single source. (Though it’s perhaps worth noting that in the case of Brazil – which TI particularly emphasizes in its press release – the change is just barely significant at conventional levels, and of the seven sources used to construct the score, although four indicate moderate to large declines, two show no change and one actually rates Brazil as improving slightly from 2012 to 2015.) So, while I still have a number of criticisms (about which more below), I’ll gladly give credit where credit is due: In this year’s publicity materials, TI has indeed identified countries where there is statistically significant change in CPI scores, generally driven by changes in several of the underlying data sources. I hope that in future years, TI will go further (and save me some time) by simply including in the main data table not only the confidence interval for the current year, but also a simple three-category indicator (up, down, null) for whether there has been a statistically significant change in the CPI in the past three years. (This is important because of the way the CPI is covered by mainstream journalists: Though researchers might dig into the data tables, most journalists or casual readers just look for year-to-year changes.)

Now, I did say I still had some concerns, so in the interest of continued constructive engagement, let me lay out why I still don’t think we should treat within-country year-to-year changes in CPI scores as terribly meaningful:

• First, as I’ve discussed before, the intertemporal comparability of the CPI depends entirely on the intertemporal comparability of the underlying sources – and I have essentially no confidence that these underlying sources consistently anchor the scales in such a way that over-time comparisons are possible. As I’ve discussed this at length before, I won’t bother belaboring it here, except to say that I do wish TI would acknowledge this issue in its FAQs, rather than simply stating that the 2012 methodology change solves the inter-temporal comparability problem.
• Second, although TI’s main press release emphasizes countries where there has indeed been a statistically significant change from 2012 to 2015, TI doesn’t seem to have quite purged itself of the habit of playing up changes that turn out not to be statistically meaningful. For example, in the most recent version of the FAQs, TI identifies Guatemala and Lesotho, along with Brazil, as the “biggest decliners.” It turns out, however, that there has been no statistically significant change in either country’s score since 2012. (And, putting aside statistical tests, a look at the raw numbers over the last four years suggests random fluctuation rather than a consistent direction of movement. For Lesotho, the last four CPI scores are 45, 49, 49, 45, while for Guatemala they are 33, 29, 32, 28.)
• Third, even though for the eight countries TI identified, the changes between 2012 and 2015 pass the usual test for statistically significance, it’s important to keep in mind that this does not mean that there was actually a real change in these countries. Even if all year-to-year fluctuations in the CPI were driven entirely by random noise in the measurements, we would still expect to see at least some countries exhibit a change that appears to be statistically significant. Remember, “statistical significance” here means an outcome that is unlikely to occur by chance in an individual case. But with a large enough number of cases, it’s not only possible, but in fact highly likely that at some unlikely outcomes will occur in some of those cases. Think about it this way: If I’m wondering whether a coin is fair, and I flip it five times get heads every time, I might reasonably suspect that it’s not a fair coin; after all, the probability of five heads in a row with a fair coin is only 1 in 32. But if I flip each of 200 coins five times, there’s quite a good chance I’ll get all heads for a few of those coins – indeed, it’s quite unlikely that I wouldn’t get at least one or two all-heads results for a couple of those coins. It’s the same issue with the countries that have “statistically significant” CPI changes: With 176 countries, even if there are no real changes and all year-to-year score fluctuations are due to random measurement error, it’s quite likely that for a handful of countries, we’ll get the equivalent of five heads in a row (the random measurement errors all going in the same direction). If we’ve got some specific reason to expect that (perceived) corruption might have changed in a given country (or class of countries), it makes sense to at least look to see if there are statistically significant changes in the CPI scores. But it’s hazardous to go through the list of countries, find the ones where there’s a change, and then try to come up with a story to explain the change.
• Fourth, the fact that Brazil and Guatemala were two of the countries where TI identified a marked worsening in the CPI score between 2014 and 2015 underscores a serious concern that others have raised about the CPI: Sometimes progress in exposing and fighting corruption can result in an increase in the perception of corruption. Though I’m of course speculating here, there’s good reason to believe that the spike in perception of corruption in Brazil has been driven in large part by the ongoing Petrobras scandal (see here and here), while in Guatemala the increased perception of widespread corruption probably has a lot to do with the CICIG-led investigation into high-level bribery, which ultimately forced the president to resign (see here and here). In both cases, the big development was not so much an increase in corruption, but rather a breakthrough investigation that exposed that corruption. In both countries, my sense is that these dramatic developments are in fact evidence that both countries are now doing a better job than they were a few years ago in addressing systemic, high-level corruption. But on the CPI, these improvements register as worsening of corruption perceptions. This, I hasten to add, is not a critique of the CPI as such, and TI has been very good about emphasizing the difference between perceptions and reality. But it’s a point that nonetheless often gets lost. And as Rick and others have cautioned, a danger here is that anticorruption advocates and reformers will become demoralized if too much emphasis is placed on the CPI, as their very successes may result in a worsening of their scores.
• Finally, given the subject of my last post, I thought I’d take a look at whether Malaysia’s score was affected by the recent scandal involving $781 million mysteriously appearing in the Prime Minister’s personal bank account. (The recent announcement that the money was from the Saudi royal family, and was apparently not illegal, would not have been picked up in the sources used to calculate the CPI, but the scandal has been in the news since last summer.) Basically, no change at all: Malaysia’s 2015 CPI score was 50, not a statistically meaningful change from its 2014 score (52), and very close to its 2013 and 2012 scores (50 and 49, respectively). Why did the Petrobras scandal in Brazil result in a significant deterioration in Brazil’s CPI score, while a seemingly comparable scandal in Malaysia didn’t make any difference at all? Who know – but the difference should certainly give us pause before we attach too much significance to these changes, or try to come up with just-so stories to explain why they occurred.