Tag Archives: Cote d’Ivoire

More on the 2017 Corruption Perceptions Index, and the Relationship Between Media/Civil Society Freedom and Corruption

Posted on by
1

The rest of the anticorruption commentariat (and the mainstream media) may have already moved on from the publication of Transparency International’s 2017 Corruption Perception Index (CPI), but I wanted to follow up on my other posts from earlier this month (here and here) to discuss one other aspect of the new CPI. The general overview, press release, and other supporting materials that accompanied the latest CPI stress as their main theme the importance of a free press and a robust, independent civil society in the fight against corruption. As TI states succinctly in the overview page for the 2017 CPI, “[A]nalysis of the [CPI] results indicates that countries with the least protection for press and non-governmental organisations (NGOs) also tend to have the worst rates of corruption.” And from this observation, TI argues that in order to make progress in the fight against corruption, governments should “do more to encourage free speech, independent media, political dissent and an open an engaged civil society,” and should “minimize regulations on media … and ensure that journalists can work without fear of repression or violence.” (TI also suggests that international donors should consider press freedom relevant to development aid or access to international organizations, a provocative suggestion that deserves fuller exploration elsewhere.)

Speaking in broad terms, I agree with TI’s position, and I’m heartened to see TI making an effort to use the publicity associated with the release of the CPI to push for concrete improvements on a particular area of importance, rather than simply stressing the bad effects of corruption (such as the alleged adverse impacts on inequality and poverty), or devoting undue attention to (statistically meaningless) movements in country scores from previous years. Whether TI succeeded in leveraging the CPI’s publicity into more attention to the freedom of the media and civil society is another story, but the effort is commendable.

That said, I spent a bit of time digging into the supporting research documents that TI provided on this issue, and I find myself in the uncomfortable position of finding the proffered evidentiary basis for the link between a free press/civil society and progress in the fight against corruption problematic, to put it mildly—even though my own reading of the larger academic literature on the topic makes me think the ultimate conclusion is likely correct, at least in broad terms. That latter fact, coupled with my recognition that the materials I’m evaluating are advocacy documents rather than academic research papers, makes me reluctant to criticize too harshly. Nonetheless, on the logic that it’s important to hold even our friends and allies accountable, and that in the long term promoting more careful and rigorous analysis will produce both more suitable policy prescriptions and better advocacy, I’m going to lay out my main difficulties with TI’s data analysis on the press freedom-corruption connection: Continue reading →

The New Corruption Perceptions Index Identifies Countries with Statistically Significant Changes in Perceived Corruption–Should We Credit the Results?

Posted on by
3

As most readers of this blog are likely aware, last month Transparency International (TI) released the 2017 edition of its important and influential Corruption Perceptions Index (CPI). As usual, the publication of the CPI triggered a fair bit of media coverage, much of it focused on how various countries ranked, and how individual country scores had changed from one year to the next (see, for example, here, here, here, and here).

There’s a lot to say about the most recent CPI—I may devote a post at some point to TI’s interesting decision to focus the press release accompanying the publication of the 2017 CPI less on the index itself than on the connection between (perceived) corruption and a lack of adequate freedom and protections for the media and civil society. But in this preliminary post, I want to take up an issue that regular GAB readers will know has been something of a fixation of mine in past years: the emphasis—in my view mostly misplaced—on how individual country CPI scores have changed from year to year.

In prior posts, I’ve raised a number of related but distinct concerns about the tendency of some commentators—and, more disturbingly, of some policymakers—to attach great significance to whether a country’s CPI score has gone up or down relative to previous years. For one thing, the sources used to construct the CPI for any given country may change from year to year—and adding or dropping an idiosyncratic source can have a substantial effect on the aggregate CPI score. For another, even when the underlying sources don’t change, we don’t know whether those sources are on the same implicit scale from year to year. And even if we put these problems to one side, a focus on changes in the final CPI score can sometimes obscure the statistical uncertainty associated with the estimated CPI—these scores can be noisy enough that changes in scores, even those that seem large, may not be statistically meaningful according to the conventional tests. Although TI always calculates statistical confidence intervals, in prior years these intervals have been buried in hard-to-find Excel spreadsheets, and the changes in CPI scores that TI highlights in its annual press releases haven’t always been statistically significant by TI’s own calculations. In an earlier post, I suggested that at the very least, TI should provide an easy-to-find, easy-to-read table assessing which changes in country scores are statistically significant at conventional levels, preferably over a 4-year period (as 1-year changes are both harder to detect if trends are gradual, and less interesting).

Apparently some folks within TI were thinking along similar lines, and I was pleased to see that in the 2017 CPI includes a reasonably prominent link to a spreadsheet showing those countries for which the 2017 CPI score showed a “statistically significant difference” from that country’s CPI score in each of five comparison years (2012, 2013, 2014, 2015, and 2016).

I’ve still got some criticisms and concerns, which—in the spirit of constructive engagement—I’ll turn to in just a moment. But before getting to that, let me pause to note my admiration for TI as an organization, and in this case its research department in particular, for constantly working to improve both the CPI itself and how it is presented and interpreted. It’s easy for folks like me to criticize—and I’ll continue to do so, in the interests of pushing for further improvements—but it’s much more challenging to absorb the raft of criticisms from so many quarters, sift through them, and invest the necessary time and resources to adapt and adjust from year to year. So, in case any folks at TI are reading this, let me first acknowledge and express my appreciation for how much work (often thankless) goes into the creation and continued improvement of this valuable tool.

Having said that, let me now proceed to raising some comments, questions, and concerns about TI’s claims about countries that appear to have experienced statistically meaningful changes in their CPI scores over the last five years. Continue reading →

The 2014 CPI Data Demonstrates Why, Even Post-2012, CPI Scores Cannot Be Compared Over Time

Posted on by
5

A little while back, I expressed some skepticism about whether Transparency International’s Corruption Perceptions Index (CPI) scores can be compared across time, even after TI changed its methodology in 2012 and claimed that its new scores would now be comparable across years.  More recently, I criticized TI’s 2014 CPI for burying the information on the margins of error associated with the CPI values, and for wrongly asserting that changes in the CPI score between 2013 and 2014 for certain countries (most notably China) were substantively meaningful.  (In fact, not only does the change in China’s score between 2013 and 2014 seem not to be statistically significant, but the change was due almost entirely to the dropping of a source in which China did abnormally well in 2013, and an abnormally large movement in a single other source.) I decided to follow up on this by taking a closer look at the other ten countries that TI singled out as having experienced significant CPI changes (in either direction) between 2013 and 2014.

Upon closer examination, I’m even more certain that CPI scores cannot be compared over time. I’m also more confident in my judgment that TI has been unforgivably sloppy — and downright misleading — in how it, and its representatives, have portrayed the substantive significance of these CPI changes. It turns out that the problem I found with the China calculations was not unusual. For almost all of the eleven countries TI identified as big movers, the CPI changes were driven by (1) the addition or elimination of sources from year to year for particular countries, and/or (2) abnormally large (indeed, implausibly large) movements in a single source. Until TI fixes its methodology, the safest thing to do is to ignore year-to-year changes in the CPI. And for the sake of preserving its own integrity and credibility, TI should either (A) persuasively explain why I am wrong in my analysis of the data (in which case I will gladly concede error), or (B) issue some sort of retraction or correction to its earlier press releases, and either drop the claim that post-2012 CPI scores can be compared across time or fix its methodology going forward.

Allow me to elaborate my analysis of the data: Continue reading →