As regular readers of this blog know, I’ve been (figuratively) pounding my fists on the table for a while now about various misuses and misinterpretations of Transparency International’s Corruption Perceptions Index (CPI), particularly in the context of misleading year-to-year comparisons (see here, here, here, here, here, and here). Perhaps I’m overemphasizing a relatively small issue, but it seems that the problem just won’t go away.
Case in point: A piece in last Friday’s New York Times by Carol Giacomo – a member of the New York Times’ editorial board – on recent developments in Indonesia. Most of the piece is a perfectly fine discussion of recent troubling events involving conflict between the Indonesian anticorruption agency (the KPK) and the Indonesian police. But near the end, in discussing the broader implications of recent events for anticorruption efforts in Indonesia, Ms. Giacomo writes:
Transparency International, which annually rates countries on corruption in their public sectors, says Indonesia has improved its performance on the organization’s “corruption perception index” from 1.9 in 2003 to 34 in 2014[.]
Almost everything about that statement is flawed.
- First, Transparency International has not (so far as I can tell) made any statement about how Indonesia’s performance on the CPI has improved between 2003 and 2014; the only link Ms. Giacomo provides is to TI’s general 2014 CPI results page. TI has made some statements about countries whose CPI performance has improved or worsened (and I’ve criticized those statements), but those recent statements have not included any commentary on Indonesia.
- Second, TI changed its methodology for calculating CPI scores in 2012, and to try to make sure no one did exactly what Ms. Giacomo did – compare pre-2012 scores to post-2012 scores – TI actually changed the scale from a 0-10 scale to a 0-100 scale. So when Ms. Giacomo says Indonesia’s TI score improved from 1.9 to 34, and makes this seem like a jump of 32.1 points, she’s actually comparing two entirely different scales without seeming to realize it. The last time I caught this same error in a prominent outlet, the piece had been written by an intern, so I tried not to be too harsh and left the author’s name out of my post. But in this case we’re talking about a prominent journalist who really ought to know better.
- Third, and in connection with the previous point, TI itself would not want Indonesia’s 2003 score to be compared to Indonesia’s score in any other year, because as TI has acknowledged very publicly, pre-2012 CPI scores are not comparable over time.
- Finally, as I’ve tried to point out in several recent posts (see here, here, and here), post-2012 CPI scores are not comparable across time either, though failure to recognize that point is more understandable, as my critique is fairly recent and has not (yet) been widely recognized or acknowledged by TI itself.
None of this matters all that much to the main substance of Ms. Giacomo’s piece – which of course raises the question of why she bothered to mention the change in CPI scores at all. Maybe there has been an improvement on perceptions of corruption in Indonesia, and maybe much of that change is due to the KPK. Indeed, I happen to believe both things are true. But not because of a meaningless comparison between Indonesia’s 2014 and 2003 CPI scores. Really, an outlet that considers itself the paper of record ought to do better on things like this.
GAB | The Global Anticorruption Blog 
I very much agree that it is problematic that so many news outlets/commentators have incorrectly relied upon Transparency International’s Corruption Perceptions Index in order to create comparisons that are, at best, inaccurate and, at worst, misleading. However, given the prevalence of this practice (and the fact that it doesn’t appear likely to abate any time soon), I wonder if a potential solution to this problem may be to encourage Transparency International to more prominently include a disclaimer regarding the fact that its rankings cannot be applied across years on its webpage. Obviously commentators should be sure to do their own research to ensure that the numbers they are reporting are correct and I’m loath to suggest that Transparency International should be forced to make it even easier for them to do so in most cases (particularly when some of the changes it implemented in 2012, particularly the shift from a 0-10 to 0-100 scale, should at least have made a reader suspicious that these numbers might not be comparable). However, it’s not difficult to see how this kind of error could occur. It seems relatively intuitive that an annual ranking system would permit authors to compare a country’s perceived corruption across various years and the main page and results pages of these reports don’t appear to provide information regarding the fact that these results can’t be compared in this way. It’s only once you turn to the “in detail” section that this disclaimer appears. All of this is not to suggest that commentators shouldn’t be held accountable for making these kinds of unforced errors – they should be. However, to the extent that we think this practice is problematic it seems like Transparency International may be well-positioned to provide an additional safeguard against these kinds of errors.
This seems spot-on. Yes, journalists should do more than just “I need a number to back up this assertion, let me pull one from TI”-style reporting, but this mistake happens so often that it really does seem like TI should follow this suggestion.