For all its flaws, I’ve long been of the view that Transparency International’s annual Corruption Perceptions Index (CPI) has, on balance, made a positive contribution to our understanding of corruption, and the fight against it. (A couple of my sympathetic treatments can be found here and here.) Although some in the media (and, depressingly, some in academic and policy circles) misuse the index, TI has generally been quite clear about what the CPI numbers do and do not tell us.  And to its great credit, TI has proven remarkably receptive to criticism: each year TI’s annual CPI report has become better, clearer, more nuanced, and more transparent in its limitations.

Until this year. The 2014 CPI came out yesterday, and I’m disappointed at how TI has taken a big step backward, making the meaning of its scores less transparent, and choosing to play for catchy headlines rather than to deepen understanding.

The source of my displeasure with TI is the decision to eliminate an important innovation in the 2012 CPI: the inclusion in the main results of a confidence interval–the range in which we can be reasonably certain (90% confident)–that the true CPI score would fall if it weren’t for the inherent noise in the data. (The 2012 and 2013 CPI Results also helpfully noted the number of sources used to generate the CPI estimate for each individual country.) This is really important, because a perception index, based on the aggregation of a bunch of separate underlying perception indexes, is inherently noisy, and one would expect some random fluctuations, across countries and across years, even if no real change in corruption perceptions were actually occurring. The inclusion of the confidence interval helps consumers of the index distinguish between those CPI differences across countries, or those within-country differences across years, that we can be confident reflect genuine substantive differences, as opposed to random noise. And indeed, as I noted in another post a little while back, when one takes the confidence intervals supplied in the 2012 and 2013 CPI data into account, it turns out that there are very few instances of statistically significant within-country change across those two years, even if one assumes (problematically) that CPI scores are comparable across time.

In its 2014 CPI, TI has dropped–without explanation or comment–the lower and upper confidence bounds (as well as the number of sources used for each country) from the results, reverting to the pre-2012 practice of listing only the point estimate. If you look hard enough you can still find the confidence intervals for 2014: Go to the In Detail page, click on the “Download Information Package” button on the right-hand side menu, unzip the Excel file, and you can find them. But TI certainly isn’t calling attention to them or making them easy for the casual reader to locate–in contrast to all the fancy graphics and visualization tools used to present the point estimate data.

And it gets worse than that: In TI’s press release accompanying the 2014 CPI, TI claims that the CPI fell (that is, perceived corruption got worse) in several countries, most notably China (dropping from 40 in 2013 to 36 in 2014), but also Turkey, Angola, Malawi, and Rwanda, while the “biggest improvers,” according to TI, were Afghanistan, Côte d´Ivoire, Egypt, Jordan, Mali, Saint Vincent & the Grenadines, and Swaziland. The press lapped this right up, particularly the bit about China, with numerous stories claiming that TI’s findings indicate that China’s anticorruption crackdown is not working (see here, here, and here). But without knowing the confidence interval, we don’t know how certain we can be that these changes reflect real movement in perceptions rather than random noise.

Fortunately, because we do have the confidence intervals for 2014 (from the hard-to-find spreadsheet), as well as the 2013 confidence intervals, we can see whether the changes TI highlights are in fact statistically significant at conventional levels. They are not. Take China: though the 2013 CPI estimate was 40, the range (that is, the 90% confidence interval) was 35-45; the 2014 CPI score (36) falls within that interval. Moreover, the 2014 confidence interval for China is 32-40. So, yes, it’s possible that China’s CPI score worsened, but there’s so much overlap between the confidence intervals that most social scientists would be extremely cautious about concluding there was in fact a downward movement. Maybe there was, and maybe that’s even our best guess, but to just assert that China got worse without even acknowledging this issue is borderline professional malpractice.

Moreover, if one takes the time to dig into the individual sources used to construct China’s CPI scores for 2013 and 2014, one finds evidence that the apparent change was probably driven more by idiosyncratic noise in the data sources than by genuine changes in corruption perceptions. The 2013 CPI for China was constructed using nine sources, eight of which were used in constructing the 2014 China CPI. (The ninth source used in 2013 was TI’s 2011 Bribe Payers’ Survey, which I assume was dropped in 2014 because it hadn’t been updated.) Of those eight common sources, three of them show no change between 2013 and 2014, two of them show a small improvement, two show a small worsening, and only one (Global Insight’s Country Risk Ratings) shows a large worsening. Also, it happens that China’s performance on the TI Bribe Payers Survey–included in the 2013 CPI but not the 2014 CPI–was quite high relative to China’s scores on all the other indexes used. So, the big downward movement in China’s 2014 CPI appears not to be the result of an across-the-board perception that corruption worsened, reflected in most sources, but rather (1) a very large, apparently idiosyncratic decline in a single source, and (2) the dropping from the CPI of a source in which China scored unusually well. Just for fun, I also calculated whet the CPI scores would have been without those two sources, by taking the average of the other seven underlying source scores for China in 2013 and 2014. They were almost exactly the same: approximately 37 in both years.

For what it’s worth, I also checked the other 11 countries that TI singled out as big movers between 2013 and 2014.  For every single one of them, the 2013 and 2014 confidence intervals overlapped (usually by quite a bit), and indeed in all but two of those countries the 2014 point estimate falls within the 2013 confidence interval. (For two countries, Côte d´Ivoire and Swaziland, the 2014 CPI score was above the high end of the 2013 interval, but only by a point, and in both cases the confidence intervals overlapped by a substantial amount.) So in fact there is very little evidence of statistically significant movements in any of these countries, either.

To be clear, absence of evidence is not evidence of absence, and it’s entirely possible that there were indeed real changes in these or other countries. It would have been just as bad if TI had said, “These confidence intervals overlap, therefore there was no change in perceived corruption.” (And if my prior post seemed to imply that view, let me set the record straight here.)

The thing I’m most upset about is TI’s decision to eliminate confidence intervals from the headline results, after having included them in two previous CPI reports. Failure to include confidence intervals in the beginning is totally understandable. But I can think of no good reason for eliminating them from the main results and burying them in the technical support material. And I can think of no good reason to issue a press release that trumpets the changes without even acknowledging the substantial uncertainty in the data. Has the new leadership at TI decided that nuance, precision, and (dare I say) transparency are less important than dramatic headlines? I sincerely hope not, and I hope someone at TI (perhaps among our blog readers?) can offer some explanation for this unwelcome change.