As discussed on this blog and elsewhere, one of the big concerns about the most popular cross-country datasets on corruption (the Transparency International Corruption Perceptions Index (CPI), the World Bank Institute’s Worldwide Governance Indicators (WGI), etc.) is that they are based (largely or entirely) on perceptions of corruption. As Rick noted in a recent post, and as the critical literature has pointed out ad nauseam, perceptions, while perhaps important in their own right, are not necessarily based in reality. Indeed, some recent research (including, but certainly not limited to, nice papers by Claudio Weber Abramo, by Mireille Razafindrakoto and Francois Rouband, and by Richard Rose and William Mishler) indicates that national corruption perceptions are only weakly correlated with survey results asking about individuals’ personal experience with bribery. This raises serious questions about whether the perception-based indicators are useful either for general assessment or for testing hypotheses about the causes or consequences of corruption.
But might there be more objective measures that could be used to assess whether the corruption perceptions indices are picking up something real? Off the top of my head, I can think of four quite clever recent papers that demonstrate a strong correlation between a subjective corruption perception index and some more objective measure of dishonest behavior. I’m sure there are more, but let me note the four examples that I can think of, and then say a bit on what this might mean for the use of perception-based indicators in empirical corruption research.
- First, a well-known paper Raymond Fisman and Edward Miguel examined the propensity of UN diplomats to abuse their diplomatic immunity to violate New York City parking rules, and found that this propensity was greater for diplomats from countries ranked as more corrupt according to the WGI Control of Corruption index. (However, it’s worth noting that once New York got permission from the State Department to pull the plates of serial violators, compliance by all diplomats increased.)
- Second, another paper by Fisman and Shang-Jin Wei examined evidence of illegal export of antiques and other cultural property, by comparing the reported value of exports with the reported value of imports. (The logic is that there’s a strong incentive to misrepresent a relic’s provenance and value at the export stage, in order to avoid national export restrictions, but the incentive is to be truthful at the import stage.) The gap between the value of exports and imports (controlling for a variety of factors) is an implicit measure of unlawful activity, and probably corruption. The paper finds that the size of the gap is strongly (and negatively) correlated with the WBI Control of Corruption index, even after controlling for a variety of other factors.
- Third, a forthcoming paper by Jason DeBacker, Bradley Heim, and Ahn Tran looks at corporate tax evasion by foreign firms operating in the United States. The paper finds that, at least for small- and medium-sized firms, the extent of tax evasion is negatively correlated with the CPI score for the firm’s home country (again, after controlling for a range of other factors, including the IRS’s own assessment of how likely an audit is to uncover additional tax liability).
- Fourth, a recent working paper by Laarni Escresa and Lucio Picci (which I posted about earlier this week) looks at enforcement data for the FCPA and similar foreign anti-bribery laws, and finds that the likelihood of a particular country’s officials being implicated in an FCPA action (after controlling for U.S. exports to that country, and several other factors) is strongly correlated with the CPI. (My earlier post raises some questions about the assumptions underlying this paper, but I will put those aside for the moment.)
So, it looks like there’s at least some evidence that perception-based corruption measures are fairly strongly correlated with more objective quantifications of a range of dishonest behaviors (none of which look like they have much to do with one another, except insofar as all of them involve some form of cheating). What should we take away from this? Here are a few tentative thoughts:
- First, maybe the strong form of the perceptions-are-not-reality criticism is overblown. True, perceptions are not reality, but we now have some decent (though admittedly imperfect) evidence that perceptions are at least correlated with reality. Now, this doesn’t change the fact that several studies (noted above) found little correlation between corruption perceptions and self-reported victimization. But there’s some evidence that people don’t answer such surveys honestly (even when asked about victimization rather than willing participation in bribery)—see, for example, Aart Kraay and Peter Murrell’s recent paper. And other research, for example a recent paper by Nicholas Charron, finds that at least in some countries, self-reported corruption victimization really does correlate strongly with corruption perceptions.
- Second, the fact that perception-based indices seem to be correlated with more objective corruption measures does NOT mean that those perception measures are unbiased. This is an important distinction—perhaps one that’s obvious to the statistics nerds, but not necessarily to others who may be interested in using research that relies on perception measures. Suppose, for example, that we want to see whether democracy is correlated with control of corruption. Suppose further (1) that the CPI is correlated with true corruption, (2) that democracy is NOT correlated with true corruption, but (3) that for other reasons (such as bias on the part of the evaluators) democracies tend to get better CPI scores. A simple analysis would report a positive correlation between democracy and corruption, even though (by supposition in this example) that correlation is spurious. So proponents of the perception measures might want to hold off on doing their victory dance, even if recent research does establish that the perception-based measures seem to be picking up something “real.”
- Finally, it’s worth considering two other common criticisms of the perception-based measures: (1) that they create an illusion of precision (though this is less true of the WGI, which include margins of error, than for the CPI), and (2) that they aren’t based on a clearly-articulated, theoretically-informed definition of “corruption” (as Michel noted in a previous post). It seems to me that the validity of these criticisms is not affected one way or the other by research showing that perception-based corruption indices are correlated with various objective measures of different sorts of dishonesty.
GAB | The Global Anticorruption Blog 
Matthew,
I think it’s fascinating that 3 of the 4 cited papers that provide the (correlational) objective validation to the CPI use measures of “corruption” that would NOT involve bribery, in contrast to the experience-with-personal-bribery papers that you cite at the beginning. Specifically, violating parking rules (maybe abuse-of-office-for-private-gain corruption, but no quid-pro-quo; and not even illegal corruption), tax evasion (same, but illegal), and lying about exports (same, but illegal) do not (to me) seem to involve bribery. Thus one way to interpret the results of the studies as consistent is that perceptions of corruption are (1) correlated with corruption (broadly defined) but (2) not very well correlated with corruption (narrowly, quid-pro-quo bribery defined).
Yes, that’s an interesting point. One quick clarification: Fisman & Wei interpret the export gap as evidence of corruption (that is, bribery) occurring in the exporting state. They don’t know this for sure, because they can’t observe directly the reason for the underreporting, but they conjecture (reasonably) that what’s going on is probably that the exporters are bribing the customs officials to mis-report the exported goods as low-value mass market goods rather than valuable antiques.
The other thing is that even though your interpretation might be consistent with the evidence in some respects, it’s questionable given that people usually assume that perception indexes like the CPI are more likely to pick up perceptions of bribery (particularly bribery that affects firms) than other forms of corruption.
I really enjoyed reading about the studies you cited. What interesting ways to get at the problem.
But I share your ultimate concerns, especially about bias in your second bulleted observation, and about the limits of these new studies to tell us much. “Some” correlation doesn’t mean “much” correlation, and without precision I’m still curious what use anyone could get out of these perception measures (in terms of inferring about actual corruption). In one sense, the idea that perceptions are related to reality seems so intuitive that many people (including myself) probably interpreted prior studies’ inability to prove a correlation to be absence of evidence, not evidence of absence. Without precision (which as Michael has pointed out in prior posts, is complicated by inconsistency in the definitions the studies use) I’m not sure we’ve gotten much further in finding a use for these studies when it comes to thinking about actual corruption.
What do you think we get out of the very general insight that there is “some” correlation? Does it actually make these studies more useful?
Good question – and a hard one to answer. Here’s my instinctive response, though this is all very tentative:
I’m more worried about bias than pure measurement error. If I recall, the correlation between the perception indexes and the more objective measures in these studies is actually reasonably strong. I don’t have the papers in front of me, so I could be misremembering, and there are definitely some differences, but I think most of the time we’re closer to “much” correlation, rather than just “some”. But there’s still a lot of noise, and the big issue (for me) is whether there are biases in the data of the sort that I describe in the post. If there are, then we could reach incorrect conclusions about important questions about corruption’s causes or effects.
Whether those biases actually exist is the question we would need to answer. If we have good reasons (based on data or theory or both) to think that the perception measures, though perhaps imprecise, are unbiased, then we can still attempt to do causal inference and learn useful information. If not, then we’re in trouble.
But the studies I mentioned make me feel somewhat more optimistic about the potential usefulness of the perception indexes than I used to be. The advantage of these indexes is that they are a LOT cheaper to collect, and have much broader coverage, then many of the more objective and/or accurate alternatives. As researchers, we always face a cost-benefit calculation: No measure is perfect, and getting a better measure takes time and resources, so the question is whether a particular measure is good enough, given our objectives, that we will go ahead and use it despite its imperfections. When we’re using something like a perception indicator, if you see studies – like those I mentioned in the first paragraph in the post – indicating very little correlation with a possibly superior measure, I start to get less comfortable with studies that rely on the perception indicator. By the same logic, when I see studies, like the four I focused on in the rest of the post, that show a fairly strong correlation with a possibly superior measure, I start to feel a little better about studies that rely on the perception indicator. But only a little, for all the reasons I mentioned, potential bias chief among them.
How are the reported amounts of organised crime in the US gathered: X$Billion a year. It can only be perceived. I don’t think the FBI go around asking people how may bags of heroin they bought in a year and make some sort of total from that. They don’t, at least I don’t think they do, ask organised crime for their annual returns.
Corruption figures are much the same because it is of the same nature as organised crime: secretive and requiring proof by prosecution to be formally measured.
What is very interesting in all of it is the comparison. No one says to the FBI your figures are only perceived therefore unreliable and perhaps organised crime does not exist or is not a serious problem. No one says these things.
But those in the fight against corruption seem to have two problems. 1. Yes, they want a better measure so that they can more accurately quantify and propose cures. 2. But, more disturbingly, they feel duty bound to come up with a better measure, because some are dismissing corruption as a serious issue in development. In cricket, the term is the anticorruptionists are on the back foot because they have only perceived not some real measures of corruption. So while they should strive for 1., they should not concede 2. Especially, in light of above statement on the difficulty of measuring any crime.
A further point is a legal point. If I loan some of another person’s money to another third person knowing that he will steal some of it, am I liable. Government officials under some statutes are liable for the funds they administer. If the chairman of a multi-national oil company does that then he is prosecuted. I think the administration of aid and development money must be brought onto the same legal platform.
Perhaps above is somewhat controversial. However I do feel that aid and development has to be legally addressed to get it focused. You could start with advertising campaigns. Are they misleading. Is that corruption.
Pingback: The 2016 CPI and the Value of Corruption Perceptions – CEGET