In post last week, I emphasized what lots of others have already tried (without much apparent success) to point out: Prior to 2012, Transparency International’s Corruption Perception Index (CPI) scores are not comparable over time.  The fact that a country’s score from one year to the next goes up or down might reflect an actual change in perceived corruption, but might be due to a whole host of other factors (changing aggregation methodology, changing scope of country coverage, change in perceived corruption of other countries, etc.), such that simple year-to-year comparisons are unreliable.  In making this point, I was not criticizing TI itself, which has been quite clear that the pre-2012 CPI scores cannot be compared across years.

But what about 2012 and after?  In 2012, TI announced, with much fanfare, that starting with the 2012 CPI and henceforth, scores across years would be comparable, due to changes in methodology (described here).

Is this right?  If it were, it would be a huge benefit both to scholars and policy reformers who want to evaluate changes over time–and the impact of various interventions.  Alas, after reading through TI’s discussion of the revised methodology, I regret to say the answer is probably no.

As I mentioned in my previous post, there are a number of reasons that the pre-2012 CPI is not comparable across years.  Some of these — the ones TI itself has emphasized — have to do with TI’s process for aggregating data from underlying sources.  TI’s 2012 revisions to its methodology are meant to address those concerns, and though I’m not a statistical expert, the description of the revised techniques seemed to me at least plausible.

But as I also noted in my last post, the bigger problem (at least as I see it) is that the underlying data sources used to produce the CPI may not be comparable from year to year.  If they’re not — perhaps due to many of the same methodological issues that TI noted with respect to the old version of the CPI — then the CPI itself will inherit those problems, and also not be year-to-year comparable.

Transparency International lives up to its name in the transparency of its methods, helpfully listing all the data sources included in the CPI here.  Most (though not all) of these are expert rankings that produce an ordinal scale–possibly intended to have cardinal significance, but that’s not always clear.  Even if in principle these expert rankings are supposed to be comparable from year to year, there are good reasons to doubt whether they actually are: implicit relative comparisons are likely — perhaps relative to other countries, or perhaps to some implicitly-perceived global average or standard. The particular experts surveyed may change year to year. As with CPI, the addition (or occasional subtraction) of countries from particular data sources may also affect year-to-year scores for particular countries even absent a change in actual corruption perceptions.

So despite TI’s improvements to its methodology, I remain concerned that the underlying data is not comparable across time.  That said, the methodological revisions are definitely an improvement, and cross-year comparisons with post-2012 CPI data are perhaps not quite as problematic.