One of my all-time favorite academic papers — which should be required reading not only for those who work on anticorruption, but on any topic where people casually throw around statistics — is Marc Galanter‘s 1993 article News from Nowhere: The Debased Debate on Civil Justice. Professor Galanter’s paper doesn’t have anything directly to do with international corruption. Rather, he sets out to debunk a series of widely-held but mostly-false beliefs about civil litigation in the United States, and in the process he traces the origins of many of the statistics often cited in debates about that topic. He finds that many of these statistics come from, well, nowhere. Here’s my favorite example: Around the time Professor Galanter was writing, it was common to hear claims that the civil justice system costs $80 billion in direct litigation costs; indeed, that figure appeared in an official report from the President’s Council on Competitiveness. The report’s only source for that estimate, however, was an article in Forbes; Forbes, in turn, had drawn the figure from a 1988 book by Peter Huber. But Huber himself hadn’t done any direct research on the costs of the system. Rather, Huber’s only source for the $80 billion figure was an article in Chief Executive magazine, which reported that at a roundtable discussion, a CEO claimed that “it’s estimated” (he didn’t say by whom) that insurance liability costs industry $80 billion per year. So: A CEO throws out a number at a roundtable discussion, without a source, it gets quoted in a non-scholarly magazine, repeated (and thus “laundered”) in what appears to be a serious book, and then picked up in the popular press and official government reports as an important and troubling truth about the out-of-control costs of the US civil justice system.
I thought about Galanter’s book the other day when I was reading the Poznan Declaration on “Whole-of-University Promotion of Social Capital, Health, and Development.” The Declaration itself is about getting universities to commit to integrating anticorruption and ethics into their programs; I may have something to say about the substance of the declaration itself in a later post. But the following assertion in the Declaration caught my eye: “Despite the relative widespread implementations of anti-corruption reforms and institutional solutions, no more than 21 countries have enjoyed a significant decrease in corruption levels since 1996, while at the same time 27 countries have become worse off.” Wow, I thought, that seems awfully precise, and if it’s true it’s very troubling. Despite the fact that I spend a fair amount of time reading about the comparative study of corruption, that statistic is news to me. It turns out, though, that it’s news from nowhere.
In contrast to Professor Galanter’s more intrepid detective work, I just looked up the single source cited to support the “21 got better, 27 got worse” claim. It’s from a 2013 article by Alina Mungiu-Pippidi in the Journal of Democracy. That article contains the following sentences, in the fourth paragraph of the introduction:
Of the 21 countries that have made significant progress on control of corruption since 1996, 12 are electoral democracies—but so are 10 of the 27 countries where control of corruption has weakened. (A far larger number has not registered any significant change.)
No source is cited for these figures. I doubt Professor Mungiu-Pippidi just made them up, but she doesn’t tell us where they came from, so there’s no way to evaluate them. My best guess–and it’s only a guess–is that the numbers are based on a comparison of the World Governance Indicators “Control of Corruption” score between 1996 and 2012, since elsewhere in the article Professor Mungiu-Pippidi runs a few regressions using the WGI score as the corruption measure.
The problem is that when I looked through the WGI webpage — admittedly casually — I couldn’t get anywhere close to reproducing her numbers. As was true in my earlier post on comparing the 2012 and 2013 Transparency International Corruption Perception Index scores, I just did a quick-and-dirty scan to identify the countries for which the WGI 90% confidence intervals for Control of Corruption in 1996 and 2012 do not cross. (Important disclaimer #1: I know that’s not, strictly speaking, the right way to test whether there’s a statistically significant difference between the 1996 and 2012 scores. But this should still be OK as a first cut, and it’s the same procedure the WGI’s creators recommend. Important disclaimer #2: I did this by eyeballing the graphs you can pull off the WGI website, rather than going through the data.) I counted 13 countries where it looked pretty clear that the 2012 score was significantly better than the 1996 score (the confidence intervals don’t cross), and 11 countries where the score appeared significantly lower (though four of these were pretty close to the line). So I can’t figure out how Professor Mungiu-Pippidi gets around twice as many countries with a statistically significant change.
And then there’s another problem: Let’s assume that I’ve guessed right, and Professor Mungiu-Pippidi is counting countries where there’s a statistically significant change between the 1996 and 2012 WGI corruption scores. Let’s also put aside the fact that I couldn’t reproduce anything close to her count. Even if she’s right that in only 21 and 27 countries, respectively, was there a statistically significant decrease/increase in corruption (as measured by WGI) this does not mean that there has been no actual change in any other country; measurement error problems–about which the WGI is commendably candid–may make genuine changes hard to detect. I strongly suspect Professor Mungiu-Pippidi would agree. The problem, though, is that the Poznan Declaration makes a seemingly small but crucial misrepresentation of the figures Professor Mungiu-Pippidi reports: It says that “no more than 21 countries have enjoyed a significant decrease in corruption levels since 1996″ (my emphasis), where in fact it would be more accurate to say that “at least 21 countries” have enjoyed such a decrease.
I don’t mean to beat up on Professor Mungiu-Pippidi, even if I do think it’s a bit careless to throw these numbers around without explaining where they came from and thinking through the methodological difficulties. After all, in the context of her original article, the numbers were just to set the stage for the larger thesis that democratic elections don’t automatically lead to less corruption. To make that point, she just wanted to point out that both democracies and non-democracies have seen improvements and backsliding with respect to anticorruption, and for that point the casual “head count” is not so problematic. And I don’t want to beat up on the Poznan Declaration either: In context, these statistics are relatively a minor part of the document, and the misrepresentation I noted above is an understandable, and seemingly harmless, mistake of the sort anyone (me included) might make.
But this nonetheless illustrates the news-from-nowhere problem: Professor Mungiu-Pippidi throws out those numbers, in passing and without citation or explanation, in a scholarly paper that’s mostly about something else. The numbers then get picked up in another document — the Poznan Declaration — and framed in a way that’s stronger than the original source, in a context that seems to suggest (again mostly in passing) that existing anticorruption reform measures are not working well. The next step in this process, if it’s not arrested in its tracks, is for people to start citing the Poznan Declaration for the claim that existing anticorruption measures are not working, are a waste of time and resources, etc. News from nowhere.
This isn’t the first time I’ve written about issues like this on this blog. A while back I asked where the “$1 trillion bribes paid annually” number came from; while blog readers helpfully identified the source, it turned out that most subsequent discussion had left out very important qualifications about the degree of uncertainty. And on a few other occasions I’ve criticized some advocates for sharing proceeds of anti-bribery settlements–with whom I generally sympathize–for playing fast and loose with the numbers (see here, here, and here). Maybe I’m making too much out of a relatively small thing. But Professor Galanter’s terrific article offers an important cautionary example. Exaggerated estimates of the cost of the US civil justice system may well have distorted the public debate in ways that were unhelpful, or even harmful. Seemingly precise but misleading statistics on corruption may do much the same thing, even if they are offered in good faith and with the best of intentions.