Transparency International Makes Its Data Less Transparent: Why TI Should Be Ashamed of Its 2014 CPI Report

For all its flaws, I’ve long been of the view that Transparency International’s annual Corruption Perceptions Index (CPI) has, on balance, made a positive contribution to our understanding of corruption, and the fight against it. (A couple of my sympathetic treatments can be found here and here.) Although some in the media (and, depressingly, some in academic and policy circles) misuse the index, TI has generally been quite clear about what the CPI numbers do and do not tell us.  And to its great credit, TI has proven remarkably receptive to criticism: each year TI’s annual CPI report has become better, clearer, more nuanced, and more transparent in its limitations.

Until this year. The 2014 CPI came out yesterday, and I’m disappointed at how TI has taken a big step backward, making the meaning of its scores less transparent, and choosing to play for catchy headlines rather than to deepen understanding.

The source of my displeasure with TI is the decision to eliminate an important innovation in the 2012 CPI: the inclusion in the main results of a confidence interval–the range in which we can be reasonably certain (90% confident)–that the true CPI score would fall if it weren’t for the inherent noise in the data. (The 2012 and 2013 CPI Results also helpfully noted the number of sources used to generate the CPI estimate for each individual country.) This is really important, because a perception index, based on the aggregation of a bunch of separate underlying perception indexes, is inherently noisy, and one would expect some random fluctuations, across countries and across years, even if no real change in corruption perceptions were actually occurring. The inclusion of the confidence interval helps consumers of the index distinguish between those CPI differences across countries, or those within-country differences across years, that we can be confident reflect genuine substantive differences, as opposed to random noise. And indeed, as I noted in another post a little while back, when one takes the confidence intervals supplied in the 2012 and 2013 CPI data into account, it turns out that there are very few instances of statistically significant within-country change across those two years, even if one assumes (problematically) that CPI scores are comparable across time.

In its 2014 CPI, TI has dropped–without explanation or comment–the lower and upper confidence bounds (as well as the number of sources used for each country) from the results, reverting to the pre-2012 practice of listing only the point estimate. If you look hard enough you can still find the confidence intervals for 2014: Go to the In Detail page, click on the “Download Information Package” button on the right-hand side menu, unzip the Excel file, and you can find them. But TI certainly isn’t calling attention to them or making them easy for the casual reader to locate–in contrast to all the fancy graphics and visualization tools used to present the point estimate data.

And it gets worse than that: In TI’s press release accompanying the 2014 CPI, TI claims that the CPI fell (that is, perceived corruption got worse) in several countries, most notably China (dropping from 40 in 2013 to 36 in 2014), but also Turkey, Angola, Malawi, and Rwanda, while the “biggest improvers,” according to TI, were Afghanistan, Côte d´Ivoire, Egypt, Jordan, Mali, Saint Vincent & the Grenadines, and Swaziland. The press lapped this right up, particularly the bit about China, with numerous stories claiming that TI’s findings indicate that China’s anticorruption crackdown is not working (see here, here, and here). But without knowing the confidence interval, we don’t know how certain we can be that these changes reflect real movement in perceptions rather than random noise.

Fortunately, because we do have the confidence intervals for 2014 (from the hard-to-find spreadsheet), as well as the 2013 confidence intervals, we can see whether the changes TI highlights are in fact statistically significant at conventional levels. They are not. Take China: though the 2013 CPI estimate was 40, the range (that is, the 90% confidence interval) was 35-45; the 2014 CPI score (36) falls within that interval. Moreover, the 2014 confidence interval for China is 32-40. So, yes, it’s possible that China’s CPI score worsened, but there’s so much overlap between the confidence intervals that most social scientists would be extremely cautious about concluding there was in fact a downward movement. Maybe there was, and maybe that’s even our best guess, but to just assert that China got worse without even acknowledging this issue is borderline professional malpractice.

Moreover, if one takes the time to dig into the individual sources used to construct China’s CPI scores for 2013 and 2014, one finds evidence that the apparent change was probably driven more by idiosyncratic noise in the data sources than by genuine changes in corruption perceptions. The 2013 CPI for China was constructed using nine sources, eight of which were used in constructing the 2014 China CPI. (The ninth source used in 2013 was TI’s 2011 Bribe Payers’ Survey, which I assume was dropped in 2014 because it hadn’t been updated.) Of those eight common sources, three of them show no change between 2013 and 2014, two of them show a small improvement, two show a small worsening, and only one (Global Insight’s Country Risk Ratings) shows a large worsening. Also, it happens that China’s performance on the TI Bribe Payers Survey–included in the 2013 CPI but not the 2014 CPI–was quite high relative to China’s scores on all the other indexes used. So, the big downward movement in China’s 2014 CPI appears not to be the result of an across-the-board perception that corruption worsened, reflected in most sources, but rather (1) a very large, apparently idiosyncratic decline in a single source, and (2) the dropping from the CPI of a source in which China scored unusually well. Just for fun, I also calculated whet the CPI scores would have been without those two sources, by taking the average of the other seven underlying source scores for China in 2013 and 2014. They were almost exactly the same: approximately 37 in both years.

For what it’s worth, I also checked the other 11 countries that TI singled out as big movers between 2013 and 2014.  For every single one of them, the 2013 and 2014 confidence intervals overlapped (usually by quite a bit), and indeed in all but two of those countries the 2014 point estimate falls within the 2013 confidence interval. (For two countries, Côte d´Ivoire and Swaziland, the 2014 CPI score was above the high end of the 2013 interval, but only by a point, and in both cases the confidence intervals overlapped by a substantial amount.) So in fact there is very little evidence of statistically significant movements in any of these countries, either.

To be clear, absence of evidence is not evidence of absence, and it’s entirely possible that there were indeed real changes in these or other countries. It would have been just as bad if TI had said, “These confidence intervals overlap, therefore there was no change in perceived corruption.” (And if my prior post seemed to imply that view, let me set the record straight here.)

The thing I’m most upset about is TI’s decision to eliminate confidence intervals from the headline results, after having included them in two previous CPI reports. Failure to include confidence intervals in the beginning is totally understandable. But I can think of no good reason for eliminating them from the main results and burying them in the technical support material. And I can think of no good reason to issue a press release that trumpets the changes without even acknowledging the substantial uncertainty in the data. Has the new leadership at TI decided that nuance, precision, and (dare I say) transparency are less important than dramatic headlines? I sincerely hope not, and I hope someone at TI (perhaps among our blog readers?) can offer some explanation for this unwelcome change.

10 thoughts on “Transparency International Makes Its Data Less Transparent: Why TI Should Be Ashamed of Its 2014 CPI Report

  1. This is a fascinating and ironic development, and I think it showcases what may be an internal divide within TI. I, am a big proponent of Open Governance and Open Data, and I know TI is as well. See links below, but briefly: TI is a participant in the International Aid Transparency Initiative, has several chapters exploring the benefits of open data transparency, and has even published a press release calling on the Open Government Partnership to improve its civil society involvement. As the elements of TI that are responsible for these initiatives are no doubt well aware, “garbage in, garbage out” can cripple an otherwise admirable commitment to transparency. Perhaps the left hand should be paying more attention to the right hand…

    http://www.transparency.org/whoweare/accountability/iati/2/
    http://blog.transparency.org/2012/12/13/does-open-data-make-development-more-accountable-the-case-of-colombia/
    http://www.opengovpartnership.org/tags/transparency-international-georgia
    http://www.transparency.org/news/pressrelease/open_government_partnership_has_to_improve_its_civil_society_participation

  2. This is a very interesting point. Ironically, a Wall Street Journal article on Tuesday cited your earlier post on the intertemporal difficulty (so to speak), but was dominated by reporting on China’s fall down the ranking. I feel like trying to convince commentators to look beyond the headline numbers might be even more difficult than asking them not to compare the yearly reports to each other.

  3. Dear Matthew,
    It would have added some value if you also had commented something on the narrowing down of the confidence range from 35-45 in 2013 to 32-40 in 2014. How would you compensate for the overlapping of the confidence range with narrowing of the confidence range. Does not it speak of data precision? By the way, I am not a statistician.

    • Good question. Because TI actually dropped a source between 2013 and 20144, one might have thought that the confidence range would expand, because the 2014 score was based on fewer sources. I haven’t gone back to try to reconstruct the confidence intervals from the underlying data, but here’s my best guess as to what’s going on:

      Although the 2014 CPI for China had one fewer source, that source was an outlier in 2013. (The source score for the TI Bribe Payers index in 2013 was 55, 6 points above the next-highest source, and 15 points above the mean.) Big outliers tend to expand the confidence interval, because they indicate more noise in the data.

      So, as with the scores themselves, most of the change in the confidence interval for China’s CPI score between 2013 and 2014 seems to be driven by dropping this one source (in which China got an unusually favorable score in 2013) when calculating the index.

  4. Dear Matthew,

    Many thanks for pointing out that the detailed information on the CPI scores’ confidence intervals is somewhat hidden in the CPI data package on our website. Over the coming days, we at TI will work on making this information more accessible to visitors.
    As you pointed out in your blog, we are deeply committed to ensuring full transparency of the CPI as well as to being clear about its limitations. This doesn’t mean that we don’t sometimes fall short of these standards. Thanks again for alerting us.

    • I appreciate your prompt and gracious response. Despite the sharp tone of the most recent post, I continue to hold TI in high regard, and I hope my criticisms prompt a productive dialogue.

      I do think that, if I am correct that the big movement in China’s score was due primarily to the dropping of the TI Bribe Payers’ Survey, it might be worth some kind of press release or statement on that subject, because the release of the 2014 CPI did elicit so much media attention precisely on this point. To be clear, I strongly agree with many of TI’s recent statements about the ways in which China’s current anticorruption campaign falls short (particularly concerns about its top-down nature, marginalization of civil society, limits on transparency, and apparent bias in enforcement targets). But insofar as the apparent drop in China’s CPI score was the Big Story — and insofar as TI itself pushed that interpretation — an acknowledgement and (partial) retraction by TI would do a lot of good, I think.

      Of course, you might well disagree with that assessment, but I’ll just put it out there. Again, I look forward to further exchanges.

  5. I recently did a study where I found a strong correlation between countries’ scores in the 2013 CPI and how old the surveys used at the time were. Countries with newer (or more frequently renewed) surveys had higher scores than countries with older (or rarely renewed) surveys by as much as 40 points. It was a Luxembourg vs. South Sudan type situation, and it seems relevant along with your example about China.

  6. In the interests of transparency, it might be useful for the author to declare any association or paid work with either academic institutions in China and/or Transparency International itself. Practice what you preach, and all that.

    • Fair enough! I have done a small amount of paid consulting work for TI, and I have done several short visits at the Peking University School of Transnational Law in Shenzhen (but have not have any direct contact with the Chinese government). I think the substantive points I make should stand or fall on their merits, but I agree with you that it’s only proper I disclose this, and allow readers to make their own judgments.

  7. Pingback: La mappa della #corruzione probabilmente dice una cosa diversa da quello che sembra » The Soviet Unit

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s