OK, in my post from a few weeks back, I asserted that year-to-year changes in a country’s Corruption Perceptions Index (CPI) score are not meaningful, even after the thoughtful and welcome changes that Transparency International made to its methodology in 2012. My concern was — and remains — that the underlying data sources that TI uses to create the index are themselves not likely to be comparable across years, which means that the CPI inherits the problem. But for purposes of this post, I’m going to completely disregard my own warning in that earlier post, and take a look at whether there have been fact been any notable changes in individual countries’ perceived corruption between 2012 and 2013.  Based on a very quick scan of the data, the answer appears to be (mostly) no.

Now, when we compare 2012 and 2013 CPI scores, we don’t just want to see if the scores are different. As TI itself emphasizes, there’s a lot of “noise” associated with corruption perception measurement, and for some countries there’s more noise than for others. What we want to know is whether there’s a statistically significant difference between the two scores. By convention, we usually say that the difference between X and Y is “statistically significant” (at the 5% level, which is admittedly an arbitrary cutoff) if the chance that X and Y were actually drawn from the same probability distribution is lower than 5%.

Fortunately, TI made another useful change to the CPI in 2012: In addition to reporting the CPI score as a single number (a “point estimate”) on a 0-100 scale, the CPI now reports a “confidence interval,” with a lower and upper bound. The confidence interval tells us (given certain assumptions that we need not go into here) that there’s a 95% chance that the “true” CPI score falls within those bounds.

Now, just to be clear, to rigorously test whether there’s a statistically significant difference between a country’s 2012 and 2013 CPI scores, one can’t just compare the 95% confidence intervals. But, as a rough-and-ready first cut, this is a reasonable way to see whether there’s strong evidence of real movement in corruption perceptions. So that’s what I did. And it turns out there’s almost no evidence of statistically significant change.

For almost all of the countries included in the 2012 and 2013 CPI, either the 2012 point estimate falls within the 2013 confidence interval, or the 2013 point estimate falls within the 2012 confidence interval, or both. For only five countries — Australia, Guinea Bissau, Senegal, Syria, and Yemen — did the point estimate for 2012 fall outside the 2013 confidence interval, and the 2013 point estimate fall outside the 2012 confidence interval. And for all five of these countries, the confidence intervals for these two years overlapped. That said, in two cases (Australia and Senegal) the overlap was very small: In Australia, the estimated confidence interval in 2012 was 83-86, while in 2013 it was 79-83; in Senegal, the estimated 2012 confidence interval was 33-39, and in 2013 it was 39-43. So if we put aside all our other concerns about the reliability of these data, there’s some evidence of a meaningful improvement in corruption perceptions in Senegal, and a meaningful worsening in Australia. But not much else.

To be clear, I’m not sure there’s that much point in spending a lot of time on these comparisons, for the reasons I laid out in my earlier post. But really, the main point I want to get across here is that, even if one puts those objections aside, when making year-to-year CPI comparisons, one must heed TI’s own advice and pay attention to the margin of error.  Failing to do so can lead to all sorts of conclusions — Canada and Iceland got more corrupt! Greece and Latvia reduced corruption! — that do not seem to be supported by sufficiently strong evidence.