One of my all-time favorite academic papers — which should be required reading not only for those who work on anticorruption, but on any topic where people casually throw around statistics — is Marc Galanter‘s 1993 article News from Nowhere: The Debased Debate on Civil Justice. Professor Galanter’s paper doesn’t have anything directly to do with international corruption. Rather, he sets out to debunk a series of widely-held but mostly-false beliefs about civil litigation in the United States, and in the process he traces the origins of many of the statistics often cited in debates about that topic. He finds that many of these statistics come from, well, nowhere. Here’s my favorite example: Around the time Professor Galanter was writing, it was common to hear claims that the civil justice system costs $80 billion in direct litigation costs; indeed, that figure appeared in an official report from the President’s Council on Competitiveness. The report’s only source for that estimate, however, was an article in Forbes; Forbes, in turn, had drawn the figure from a 1988 book by Peter Huber. But Huber himself hadn’t done any direct research on the costs of the system. Rather, Huber’s only source for the $80 billion figure was an article in Chief Executive magazine, which reported that at a roundtable discussion, a CEO claimed that “it’s estimated” (he didn’t say by whom) that insurance liability costs industry $80 billion per year. So: A CEO throws out a number at a roundtable discussion, without a source, it gets quoted in a non-scholarly magazine, repeated (and thus “laundered”) in what appears to be a serious book, and then picked up in the popular press and official government reports as an important and troubling truth about the out-of-control costs of the US civil justice system.
I thought about Galanter’s book the other day when I was reading the Poznan Declaration on “Whole-of-University Promotion of Social Capital, Health, and Development.” The Declaration itself is about getting universities to commit to integrating anticorruption and ethics into their programs; I may have something to say about the substance of the declaration itself in a later post. But the following assertion in the Declaration caught my eye: “Despite the relative widespread implementations of anti-corruption reforms and institutional solutions, no more than 21 countries have enjoyed a significant decrease in corruption levels since 1996, while at the same time 27 countries have become worse off.” Wow, I thought, that seems awfully precise, and if it’s true it’s very troubling. Despite the fact that I spend a fair amount of time reading about the comparative study of corruption, that statistic is news to me. It turns out, though, that it’s news from nowhere.
In contrast to Professor Galanter’s more intrepid detective work, I just looked up the single source cited to support the “21 got better, 27 got worse” claim. It’s from a 2013 article by Alina Mungiu-Pippidi in the Journal of Democracy. That article contains the following sentences, in the fourth paragraph of the introduction:
Of the 21 countries that have made significant progress on control of corruption since 1996, 12 are electoral democracies—but so are 10 of the 27 countries where control of corruption has weakened. (A far larger number has not registered any significant change.)
No source is cited for these figures. I doubt Professor Mungiu-Pippidi just made them up, but she doesn’t tell us where they came from, so there’s no way to evaluate them. My best guess–and it’s only a guess–is that the numbers are based on a comparison of the World Governance Indicators “Control of Corruption” score between 1996 and 2012, since elsewhere in the article Professor Mungiu-Pippidi runs a few regressions using the WGI score as the corruption measure.
The problem is that when I looked through the WGI webpage — admittedly casually — I couldn’t get anywhere close to reproducing her numbers. As was true in my earlier post on comparing the 2012 and 2013 Transparency International Corruption Perception Index scores, I just did a quick-and-dirty scan to identify the countries for which the WGI 90% confidence intervals for Control of Corruption in 1996 and 2012 do not cross. (Important disclaimer #1: I know that’s not, strictly speaking, the right way to test whether there’s a statistically significant difference between the 1996 and 2012 scores. But this should still be OK as a first cut, and it’s the same procedure the WGI’s creators recommend. Important disclaimer #2: I did this by eyeballing the graphs you can pull off the WGI website, rather than going through the data.) I counted 13 countries where it looked pretty clear that the 2012 score was significantly better than the 1996 score (the confidence intervals don’t cross), and 11 countries where the score appeared significantly lower (though four of these were pretty close to the line). So I can’t figure out how Professor Mungiu-Pippidi gets around twice as many countries with a statistically significant change.
And then there’s another problem: Let’s assume that I’ve guessed right, and Professor Mungiu-Pippidi is counting countries where there’s a statistically significant change between the 1996 and 2012 WGI corruption scores. Let’s also put aside the fact that I couldn’t reproduce anything close to her count. Even if she’s right that in only 21 and 27 countries, respectively, was there a statistically significant decrease/increase in corruption (as measured by WGI) this does not mean that there has been no actual change in any other country; measurement error problems–about which the WGI is commendably candid–may make genuine changes hard to detect. I strongly suspect Professor Mungiu-Pippidi would agree. The problem, though, is that the Poznan Declaration makes a seemingly small but crucial misrepresentation of the figures Professor Mungiu-Pippidi reports: It says that “no more than 21 countries have enjoyed a significant decrease in corruption levels since 1996″ (my emphasis), where in fact it would be more accurate to say that “at least 21 countries” have enjoyed such a decrease.
I don’t mean to beat up on Professor Mungiu-Pippidi, even if I do think it’s a bit careless to throw these numbers around without explaining where they came from and thinking through the methodological difficulties. After all, in the context of her original article, the numbers were just to set the stage for the larger thesis that democratic elections don’t automatically lead to less corruption. To make that point, she just wanted to point out that both democracies and non-democracies have seen improvements and backsliding with respect to anticorruption, and for that point the casual “head count” is not so problematic. And I don’t want to beat up on the Poznan Declaration either: In context, these statistics are relatively a minor part of the document, and the misrepresentation I noted above is an understandable, and seemingly harmless, mistake of the sort anyone (me included) might make.
But this nonetheless illustrates the news-from-nowhere problem: Professor Mungiu-Pippidi throws out those numbers, in passing and without citation or explanation, in a scholarly paper that’s mostly about something else. The numbers then get picked up in another document — the Poznan Declaration — and framed in a way that’s stronger than the original source, in a context that seems to suggest (again mostly in passing) that existing anticorruption reform measures are not working well. The next step in this process, if it’s not arrested in its tracks, is for people to start citing the Poznan Declaration for the claim that existing anticorruption measures are not working, are a waste of time and resources, etc. News from nowhere.
This isn’t the first time I’ve written about issues like this on this blog. A while back I asked where the “$1 trillion bribes paid annually” number came from; while blog readers helpfully identified the source, it turned out that most subsequent discussion had left out very important qualifications about the degree of uncertainty. And on a few other occasions I’ve criticized some advocates for sharing proceeds of anti-bribery settlements–with whom I generally sympathize–for playing fast and loose with the numbers (see here, here, and here). Maybe I’m making too much out of a relatively small thing. But Professor Galanter’s terrific article offers an important cautionary example. Exaggerated estimates of the cost of the US civil justice system may well have distorted the public debate in ways that were unhelpful, or even harmful. Seemingly precise but misleading statistics on corruption may do much the same thing, even if they are offered in good faith and with the best of intentions.
GAB | The Global Anticorruption Blog 
I forwarded this post to someone involved in Prof. Mungiu-Pippidi’s research team. They pointed me to his paper for more background on the methods: http://anticorrp.eu/wp-content/uploads/2013/08/D3_1Global-comparative-trend-analysis-report1.pdf The original “21 positive, 29 negative” statement is on page 9. It is preceded by a short section on the limitations of the data. The Poznan Declaration should have cited that paper, not the the other one.
While I agree with your unease about this kind of sound-bite numbers, I don’t think this case is nearly as problematic as the “1 trillion” guesstimate you analyzed in the earlier blog post.
Wonderful, thank you so much! I will review the paper you sent. I’m very interested to see how the researchers arrived at these numbers, and whether my speculations were at all accurate.
Thanks again, to you and your contact on Professor Mungiu-Pippidi’s research team, for forwarding me the link to the Global Comparative Trend Analysis Report. As you say, this report does indeed discuss the methodology in much greater detail. As I’d conjectured, the numbers are based on the World Governance Indicators (though the comparison was 1996 to 2011, not 1996 to 2012). It’s clear that there was actually a process that produced these numbers — they weren’t plucked out of the air — and the report itself is commendable in emphasizing the importance of paying attention to the statistical confidence intervals (in contrast to, for example, the recent release of the 2014 CPI from Transparency International, which I criticized a couple weeks back). All that said, after having gone through the report, and compared it to the WGI’s own data page, I’ll confess I’m still baffled as to how Professor Mungiu-Pippidi’s research team came up with the numbers, or the countries, that it did. I went through each of the 48 countries the report identified as having experienced a statistically significant change in the WGI control of corruption score between 1996 and 2011, and then looked at the WGI data page for each of those countries, to see if the confidence intervals overlapped or not. Here’s what I found:
Of the 21 countries that the report says experienced a statistically significant improvement between 1996 and 2011, I found that to be true for only three countries (Serbia, Liberia, and Tanzania, and Tanzania is a close call where I’ll give the report the benefit of the doubt). For most of the others, it isn’t even close — just eyeballing the data on the WGI page shows, for most of these countries, that the point estimates are fairly close, and the confidence intervals overlap by a large margin (often with upwards of 50%, and sometimes upwards of 75%, of the ranges appearing to overlap).
Of the 27 countries the report identifies as having experienced a statistically significant decline, I again looked at the WGI data page, and this time found that for only five of these countries (Fiji, Yemen, Eritrea, Zimbabwe, and Cote d’Ivoire) did the 1996 and 2011 confidence intervals not overlap. And again, for most of these other countries, it wasn’t even close — the 1996 and 2011 confidence intervals usually overlapped by a substantial amount, sometimes more than half or three-quarters of the range.
These are not small discrepancies. It’s a complete and total failure to reproduce the results, based on re-examination of (I assume) exactly the same numbers. On top of that, for what it’s worth, in eye-balling of the data I also came across several countries where there did seem to be a statistically significant (often quite large) change in the WGI corruption score between 1996 and 2011 (including Angola, Nepal, Sierra Leone, Sri Lanka, and Thailand), but that were not identified as significant changers in the report.
This problem is perhaps slightly different from the “news from nowhere” problem I focused on in the original post, but it is a significant problem nonetheless. Perhaps I’m wrong — I could be misreading the report, or the WGI data, or perhaps the report employed a more rigorous statistical technique. But if anyone from the original research team (or anyone else) could address this, that would be great.
Thanks for the follow-up! I just looked at the WGI website. You’re right to point out that the confidence intervals for the percentile ranks often overlap for 1996 and 2011.
However, the claims on page 9 of the paper are based on the change in the numerical scores, which can be displayed via the “Table View” on the WGI page. There you don’t have visual confidence intervals, but standard errors for the scores. As far as I can tell, those standard errors refer to the numerical scores, not the percentile ranks. I just checked for three random countries (China, Egypt, UK), and there the numbers are in line with those in the paper.
So as I understand it, percentile ranking is a less precise indicator than the score. And the authors of the report could have done a better job of distinguishing between the two (especially since they also use the percentile ranks later in the paper). But I think their data is correct.
Ah! Thanks so much! My mistake! I was misreading the WGI page – I thought it was displaying the intervals for scores, not rank position. I’ll double-check to verify, but I’m sure you’re right. I humbly acknowledge my error. Though my larger concern about News from Nowhere still applies to other parts of the anticorruption debate, it seems that you are correct that this is indeed News from Somewhere.
Maybe I’m obsessing too much, but I went through the WGI data to try to replicate the results in the report. I compared the 1996 and 2011 WGI control-of-corruption data, to calculate, for each country, whether there is a statistically significant difference (at the 90% level) between the 1996 and 2011 scores. Rather than checking to see whether the 90% confidence intervals do not overlap (which is a sufficient but not necessary condition for the scores to have a statistically significant difference), I actually did what you’re supposed to do — I only hope I did it right! — by calculating the t-statistic for each country (the difference in the means, divided by the square root of the sum of the squares of the standard errors); if the absolute value of the t-score exceeds the critical value of 1.282 (for 90% threshold). (It’s possible I should have used a different threshold because of the smaller number of degrees of freedom, which might explain the minor discrepancies I note below, but bracket that for now.) Doing so, I was able to replicate — almost but not quite exactly — the findings in the report. It certainly took some doing, and it would have been nice if the methodology section of the report (or perhaps an appendix) made the procedures a bit clearer, or provided a table or something, but that point aside, I’m now satisfied that the mistake was mine, the report is basically correct (insofar as we trust the WGI scores, but that’s another story).
That said, I did encounter a few discrepancies, which I feel like I should mention, if only to justify all the time I invested (wasted?) in going through the Excel file and trying to replicate the report’s result. In case anyone else is interested, here goes:
First, I found statistically significant changes in four countries that I think the report didn’t identify: Congo, Ethiopia, and Hong Kong all seemed to show a statistically significant improvement in the WGI score between 1996 and 2011, while the Czech Republic experienced a statistically significant decrease.
Second, for two of the countries that the report listed as having had a statistically significant change — Tanzania (improving) and Namibia (worsening) — my attempted replication found no statistically significant difference (though the t-values, 1.202 for Namibia and 1.864 for Tanzania) were close to the critical value of 1.282. If I was using the wrong threshold for this data that might explain the discrepancy, though lowering the threshold would move several other countries not noted by the report into the “statistically significant change” category, so that’s not a fully satisfying explanation.)
Third, and least importantly, the report includes four small nations (St. Lucia, St. Vincent & Grenadines, St. Kitts & Nevis, and Cape Verde) that did not have 1996 WGI scores, so I couldn’t replicate them. I assume the report’s authors used a later year as the start date for these countries.
Those small issues aside, I feel much more confident that the numbers quoted in the Poznan Declaration do have some evidentiary basis, though the WGI might be problematic in other respects.
Thanks for helping me work through this.
Thank you for the post Matthew!
As the author of the Poznan Declaration I feel somewhat guilty of citing figures that I didn’t follow up on fully.
While I agree on the general problem of the matter – that figures gets laundered when cited – and that this is something we indeed should be very cautious about, I’d like two stress two points in this particular case.
Frist, I do believe Professor Alina Mungiu-Pippidi has a solid basis for these figures and that they’re actually news from somewhere (Cheers Mathis I’ll have a look at that paper as well)!
Second, as regards the declaration I’m not arguing that present anti-corruption measures are a waste of time and resources (i.e. should be scrapped), quite on the contrary, I argue that they are absolutely essential, but may also need to be complemented with an inclusion of this work in university curricula in order to be fully effective.
Whether or not it’s 13/11 or 21/27, considering the efforts made, we know corruption to be a very sticky problem. One that likely needs to be tackled in a multi system approach.
Best,
Marcus,
Thanks for your thoughtful reply. I think we’re in agreement on the general problem. Let me say a quick additional word on each of your two points:
On the first, I was very optimistic when Mathis forwarded me the link to the report that there was indeed a solid basis for these numbers, and that the only problem (a real one, but much milder) was Professor Mungiu-Pippidi’s failure to include an appropriate citation in the Journal of Democracy article. Alas, as I just noted in reply to Mathis’s comment, I went through the report carefully and checked each country identified against the WGI data available from the WGI’s page (link in my original post), and I could not come close to replicating the results. Perhaps you, as an independent third party, could take a look at this and see if you can figure out what’s going on? All the countries (21 up, 27 down) are specifically identified in the report, and the WGI page has a helpful interactive visualization tool that makes it easy to compare each country’s 1996 and 2011 control-0f-corrruption scores and associated confidence intervals.
On your second point, I completely agree! In context — if one reads the whole declaration — it’s quite clear you’re calling for a multi-pronged approach. And for that reason it was perhaps unfair of me to use the Poznan Declaration as an illustration of the “News from Nowhere” problem in anticorruption research — there are far more egregious examples. That said, my concern is precisely how these numbers often get oversimplified, taken out of context, shorn of nuance, etc. I could easily see someone reading the Poznan Declaration and seizing on that one paragraph, citing it in some other document saying something like, “We know anticorruption reforms aren’t working — see Poznan Declaration, page whatever”. And I should also say that I’m not immune to this problem either. I probably should have been much clearer that I wasn’t singling out the Poznan Declaration for special criticism, so much as I was using it as an example (the one most readily at hand) of a much larger and more general concern.
Finally, I should say that — in case it’s not clear to other readers from the post or this exchange — the numbers I’m focusing on are really peripheral to the main point of the Poznan Declaration itself; that Declaration would be much the same even without the paragraph I focused on. And I do hope in a subsequent post to address the substance of the Poznan Declaration, and give it the attention it deserves.