The rest of the anticorruption commentariat (and the mainstream media) may have already moved on from the publication of Transparency International’s 2017 Corruption Perception Index (CPI), but I wanted to follow up on my other posts from earlier this month (here and here) to discuss one other aspect of the new CPI. The general overview, press release, and other supporting materials that accompanied the latest CPI stress as their main theme the importance of a free press and a robust, independent civil society in the fight against corruption. As TI states succinctly in the overview page for the 2017 CPI, “[A]nalysis of the [CPI] results indicates that countries with the least protection for press and non-governmental organisations (NGOs) also tend to have the worst rates of corruption.” And from this observation, TI argues that in order to make progress in the fight against corruption, governments should “do more to encourage free speech, independent media, political dissent and an open an engaged civil society,” and should “minimize regulations on media … and ensure that journalists can work without fear of repression or violence.” (TI also suggests that international donors should consider press freedom relevant to development aid or access to international organizations, a provocative suggestion that deserves fuller exploration elsewhere.)

Speaking in broad terms, I agree with TI’s position, and I’m heartened to see TI making an effort to use the publicity associated with the release of the CPI to push for concrete improvements on a particular area of importance, rather than simply stressing the bad effects of corruption (such as the alleged adverse impacts on inequality and poverty), or devoting undue attention to (statistically meaningless) movements in country scores from previous years. Whether TI succeeded in leveraging the CPI’s publicity into more attention to the freedom of the media and civil society is another story, but the effort is commendable.

That said, I spent a bit of time digging into the supporting research documents that TI provided on this issue, and I find myself in the uncomfortable position of finding the proffered evidentiary basis for the link between a free press/civil society and progress in the fight against corruption problematic, to put it mildly—even though my own reading of the larger academic literature on the topic makes me think the ultimate conclusion is likely correct, at least in broad terms. That latter fact, coupled with my recognition that the materials I’m evaluating are advocacy documents rather than academic research papers, makes me reluctant to criticize too harshly. Nonetheless, on the logic that it’s important to hold even our friends and allies accountable, and that in the long term promoting more careful and rigorous analysis will produce both more suitable policy prescriptions and better advocacy, I’m going to lay out my main difficulties with TI’s data analysis on the press freedom-corruption connection:

First, TI’s analysis of the negative correlation between perceived corruption and various measures of press or civil society freedom (the World Justice Project’s Freedom of Assembly and Association score in its Rule of Law Index, the Reporters Without Borders World Press Freedom Index, the Varieties of Democracy (VOD) Freedom of Expression Index, and the VOD Civil Society Participation Index) looks only at the simple bivariate correlation. In other words, there are no control variables included in the analysis. This is a fairly big problem, and one that would lead all but the most naïve readers to discount, perhaps entirely, the significance of the correlations TI finds. After all, there are a whole host of other factors that might simultaneously affect a country’s level of press freedom/civil society participation and that country’s level of (perceived) corruption, producing a strong correlation even if it were actually the case that press or civil society freedom had little or no impact on corruption levels. The most obvious confounding factor is wealth (usually measured by per capita GDP), which, as I noted in my last post, has a very strong correlation with the CPI, and I’d be willing to bet also correlates reasonably strongly with the various press and civil society measures that TI uses. But there are other possible confounding factors as well, such as historical experience, geography, and other institutional features. Now, one of the reasons I think TI’s conclusion is ultimately correct, despite this problem, is that there are a number of other studies that do control for other factors and continue to find a statistically significant negative correlation between press freedom index scores and perceived corruption scores (see, for example, here, here, here, and here). While that research isn’t dispositive and open questions remain (see, for example, here, here, and here), it does seem to establish more convincingly the basic point TI is trying to advance here. But that makes it even more puzzling to me why TI would rely on simple, unconvincing bivariate correlations when trying to make this point.

Second, and partly related to the preceding point, the fact that in a cross-sectional analysis the media/civil society environment is correlated with perceived corruption does not necessarily mean that improvements in the former will lead to improvements in the latter. Yet in its narrative discussion, TI makes precisely this argument, using a few examples in either direction. On the one hand, TI asserts, Hungary and Brazil “are key examples of the relationship between civil rights and corruption.” Hungary, the document notes, recently “enacted a series of measures to restrict press freedom” and has also proposed legislation that would restrict the operation of NGOs. “We see in our latest CPI.” TI continues, that Hungary’s score “has declined from 55 in 2012 to 45 in 2017.” As for Brazil, TI notes that “[c]ivil society’s ability to participate in decision making in the country has reduced recently” and that Brazil “is also a dangerous place for journalists, with 20 killed in the last six years.” These factors, the TI report implies, explain why “Brazil’s CPI score also declined from 43 in 2014 to 37 in 2017.” The lead example in the other direction is Côte D’Ivoire, which, the TI document notes, “has experienced greater civic participation in politics and progress on human rights in recent years,” and has also “improved its CPI score from 27 in 2013 to 36 in 2017.” There are a few big problems with this:

• First of all, by TI’s own analysis, none of the three countries listed as lead examples actually experienced a statistically significant change in its CPI score over the relevant time period. The changes in scores that the TI document notes might well be statistical noise. In my post earlier this month I offered qualified praise to TI for taking the time and trouble to identify those changes that were statistically meaningful, and to drop from its discussion undue attention to statistically insignificant fluctuations. I was therefore both surprised and disheartened to see that in this other document—the one that’s supposed to provide the evidentiary basis for TI’s featured policy recommendations—we see a reversion to past bad habits, and an inexplicable unawareness of TI’s own analysis of which countries have in fact experienced statistically meaningful changes in their CPI scores.
• The Brazil example is particularly puzzling to me, because whatever may be going on with the CPI scores (and again, the answer may be nothing, or at least nothing statistically meaningful), the big news coming out of Brazil in the past few years is surely the country’s major progress against corruption. (Indeed, my working hypothesis for why Brazil’s CPI score may have worsened, if it’s not just statistical noise, is that the progress in investigations has exposed more corruption, thus worsening perceptions of corruption even as things are getting somewhat better in terms of actually doing something about it.)
• Moreover, TI’s supporting graphs (which I can’t link to directly, but you can get to by clicking the link labeled “download full analysis” about 2/3 of the way through the analysis document) includes charts not only for the bivariate correlations between the 2017 CPI and the four measures of press/civil society freedom, but also scatterplots showing the relationship between the change in the CPI score (between 2012 and 2017) and the change in each of the four press/civil society variables (though it’s not clear what the starting year is, as not all the measures go back to 2012). Statistical significance tests aren’t reported, but from eyeballing the charts (what my old statistics professor jokingly characterized as “ocular analysis”), the correlation between score changes appears to be essentially zero. That does not mean that changes in the media or civil society environment don’t affect corruption levels. Such effects might well not be captured by the sorts of simple scatterplots that TI provides, especially if the effects are more long-term. But it’s borderline irresponsible for TI to cherry-pick three examples to illustrate a causal relationship that TI’s own analysis fails to find in the aggregate data.
• Just for fun, I took a look at the set of 15 countries that TI asserts have experienced statistically significant changes in the CPI score between 2012 and 2017, and compared those changes to the changes in the Reporters Without Borders Press Freedom Index. The latter only goes back to 2015, so it’s not really a great comparison, but according to TI only one country (Guyana) experienced a statistically significant change in its CPI score between 2015 and 2017. I just wanted to see if the big CPI movers that TI identifies also had notable movements in their press freedom scores over a partially overlapping time period. And the answer was basically no. I had to drop Saint Lucia because it’s not in the Press Freedom Index. Of the other 14 countries, only three had movements of more than three points on the Press Freedom Index’s 100-point scale. Turkey’s press freedom score worsened by almost nine points, and its CPI score worsened by about nine points, while Syria’s Press Freedom score dropped by over four points and its CPI score dropped by about 12 points. Both of these results are consistent with the idea that worsening press freedom goes hand in hand with worsening perceived corruption. But Belarus saw its Press Freedom score worsen by about as much as Syria’s (approximately 4.5 points), while its CPI score improved by 13 points. (Oh, and as for Guyana, the one country that according to TI saw a statistically significant change in its CPI score between 2015 and 2017–a nine-point improvement–there was virtually no change in its Press Freedom Index score, which improved by less than half a point.) Again, this is just for fun – what really matters is whether there are patterns in the aggregate data, and there don’t seem to be. But it does suggest that for the country that TI itself identifies as statistically significant CPI movers, there doesn’t seem to be much evidence that they’ve seen big or consistent changes in their press freedom environment, contrary to the story TI seems to want to tell with its handful of examples.

Again, I want to make clear that I do think that, at least in the long term, a robust and free press, together with an active and independent civil society, do contribute substantially to the fight against corruption. (And even if they didn’t, I’d strongly support measures to improve press and associational freedom.) But I do think influential civil society groups like TI should make that case using the best available evidence, and do the best that they can—subject, of course, to their resource limitations and their distinctive advocacy missions—to model best practices in their analysis and presentation of evidence. I suspect that part of the problem is that TI’s researchers are trying to make use of the new 2017 CPI data in their analyses, but don’t have sufficient time to do more than the simple bivariate correlations and scatterplots of the sort described above. I understand the impulse to derive results from the new data, since that data is after all the focus of these announcements, but I’d nonetheless offer the friendly suggestion that in the future, rather than trying to run preliminary analyses using the brand-new CPI data, TI might rely more extensively on the existing research literature, much of it employing previous versions of the CPI, to advance the substantive points that consumers of each new CPI ought to know.