Matthew wrote last month about the February competition the U4 Anticorruption Resource Center and the United Kingdom’s Department for International Affairs sponsored to spur creation of new measures of corruption. What he did not say is that one subtext for the contest was the growing frustration with the use of Transparency International’s Corruption Perceptions Index (CPI) to measure the actual level of corruption in a country.
This is not the fault of TI. The organization is careful to say on its web site that its index does not measure the actual extent of corruption and goes to great lengths to explain how the index is constructed, stressing that it is a ranking of how corrupt countries are perceived to be using the opinions of business people and country experts. Indeed the title selected, “Corruption Perceptions Index,” couldn’t be any clearer about what is being measured. But journalists and academics frequently treat the index as if it measured actual corruption, rather than perceived corruption, or assume that perceptions match reality fairly closely. And that’s where problems may arise.
When the CPI first appeared in 1995, the distinction between perceptions and actual levels of corruption seemed of little concern: a close connection between the two was assumed. In other words, a measure of corruption perceptions was taken to be a good proxy, or stand-in, for a measure of actual corruption.
Recent research, however, has shown this assumption does not hold. In a 2006 paper Daniel Treisman reported that in countries where perceived corruption levels were high, there was little relationship between perceptions of corruption and citizens’ experience with bribery. He also found that perceptions of corruption levels in a country were influenced by the country’s level of economic development and democratization; with all other things equal, wealthier, more democratic countries are perceived to be less corrupt than poorer, non-democratic ones. Work by Dilyan Donchev and Gergely Ujhelyi reaches the same conclusion about perception bias, adding that survey respondents’ education, age, and employment also affect their perceptions of corruption. Other studies showing the divergence between actual and perceived levels of corruption can be found here, here, and here.
The response of many anticorruption activists to these findings is often: “So what? Hasn’t the index been extraordinarily successful in prompting action on corruption?” The answer is an unqualified “yes.” A country’s low score on the index draws media attention and often serves as a rallying point for reformers. The index’ annual appearance puts tremendous pressure on policymakers in low scoring countries to act, and the world community owes Transparency International a debt of gratitude for creating and publicizing the index.
But it is one thing to serve as a stimulus to action, and quite another to serve as a measure of the effectiveness of that action. As much as the CPI has succeeded at the former, it is ill-suited for the latter. And as the priority has shifted from pressuring countries to act to ensuring the actions they take are effective, attention to measurement has become ever more critical. In a later post I will provide some examples of how continued use of the index as a performance measure can frustrate progress on the very objective for which it was created.
GAB | The Global Anticorruption Blog 
Rick, thanks for this useful background information. You’re right that TI’s webpage clearly and repeatedly notes that it describes perceptions, not realities (although it also notes, “The CPI is the most widely used indicator of corruption worldwide,” full stop). And I look forward to your next post about ways of using CPI.
But I wonder if you agree with another set of claims on CPI’s FAQ page. TI claims, first, that “[t]here is no meaningful way to assess absolute levels of corruption in countries or territories on the basis of hard empirical data,” and second, that “[c]apturing perceptions of corruption of those in a position to offer assessments of public sector corruption is the most reliable method of comparing relative corruption levels across countries.” While TI recognizes what it does and doesn’t measure, the FAQ page does suggest a somewhat broader view of what CPI proves. Do you agree with TI that their measures are the best at proving actual corruption, or do they phrase these claims too strongly?
Thanks for the comment. Glad you liked the post.
On your questions about the statements in the FAQs that accompany the CPI. With a slight caveat, I agree with the first and sharply disagree with the second. TI is correct that there is no meaningful way to compare corruption levels across nations. Take the case of two countries equal in every way save in the first ten individuals each accept a $10,000 bribe from a foreign investor whereas in the second one person takes a $100,000 bribe from the investor. Which country is more corrupt? Or are they equally corrupt? Whatever your answer, what is the objective, non-contested criteria upon which you base your answer?
It follows from what I say above that I disagree with the statement that the CPI or similar perception based measures are “the most reliable method of comparing relative corruption levels across countries.” It is a meaningless statement since there is no way to measure corruption levels across countries. Just because policymakers want such a measure doesn’t mean such a measure can be conjured up. For years parents and schools wanted a measure of how smart children are that could be compared across all kids. As the experience with IQ and the subsequent learning shows, not only is such a measure without meaning but the use of IQ has done much damage.
I appreciate TI’s dilemma when it comes to what to say about the CPI. It did stimulate much useful thought about corruption; the subsequent research establishing the points above puts the organization in an awkward position. Perhaps it would be best if the organization confined its comments to the single fact that the CPI measures perceptions and leave it to others to enlarge on its problems.
I’m very sympathetic to these critiques, as you know. But we might want to be careful not to conflate two related but distinct concerns here:
The first is the simple observation that perceptions can be wrong.
The second is the fact that “corruption” is a complex, multi-dimensional concept, which may not be reducible to a single measurable dimension.
It’s certainly possible to have one problem without having the other. Your point about IQ is more about the second issue, not the first. IQ tests may have been quite accurate in measuring what they measured (they were “objective” measures, not subjective perception-based measures); the problem is that what they were measuring was not “intelligence”, because there is no single measurable concept of “intelligence” (or so the argument goes). I take it Michael was also making something like this point in Monday’s post.
The point about perception may be different, in that even if it were the case that corruption were a coherent measurable concept (so that we could talk about country X being more or less corrupt than country Y), perceptions might still be inaccurate.
By the way, even though I share your concerns, we need to be careful not to overstate the impossibility of making statements like Country X has more corruption than Country Y. Suppose that in Country X 100 people take bribes of $100,000 each, while in Country Y 5 people take bribes of $20 year. I feel pretty comfortable saying corruption is a worse problem in X than in Y — just as I feel pretty confident saying that corruption is a bigger problem in Nigeria than in Sweden (or, for that matter, a bigger problem in New Jersey than in Oregon). But of course that’s just a perception, and it could be wrong!
Matthew, would your conclusion about corruption levels in country X and country Y be different if the 100 people taking the bribes in X were taking them to speed the processing through customs of life saving medicines while the bribes in Y were to redirect water supplied by the municipal waterworks from the poor who use it for drinking to the rich who need it to wash their cars?
Rick,
With respect, I think that question mostly misses the point of what I’m trying to convey. I don’t deny that corruption is a complex, multidimensional concept, and that for that reason comparisons are difficult (because one country might have more or worse corruption on one dimension, while another has more or worse corruption on another dimension). My point was that although this _may_ be a problem, sometimes one jurisdiction is sufficiently worse on all or most relevant dimensions, such that we can say confidently that corruption in that jurisdiction is worse than in some other jurisdiction. The fact that it’s hard to decide conclusively whether Tom Brady or David Ortiz is a better athlete doesn’t imply that we can’t say, with supreme confidence, that both of them are better athletes than I am.
The question is whether corruption is something that can ever, or often, be a sufficiently coherent concept that we can do cross-jurisdictional comparisons. (Again, this is a distinct question from whether we can measure the various dimensions of corruption accurately enough, whether via perceptions or otherwise.) I’m still struggling with this question. I don’t think the fact that one can come up with clever hypotheticals in which the different dimensions or aspects of corruption point in opposite directions is sufficient.
The point of the hypothetical was that “corruption” is multi-dimensional and while one country might score lower on one dimension than another, it might score higher on a second. The question then becomes how you weight the scores on the different dimensions to produce a single number.
Matthew is (probably) right that Tom Brady and David Ortiz are each better athletes than he, but if we are looking for a measure of athleticism, we would want to know, between the two, which is better. How would one compare Ortiz’ skill at throwing a baseball with Brady’s skill at throwing a football? How would one compare base running with broken field running? And assuming we can combine all these traits into a single measure of athleticism, how much more athletic than Matthew is each one of them?
Matthew and I may differ on the measurement issue because we mean different things when we say “measurement.” I am using it in a quantitative sense whereas Matthew seems to be using it in a comparative sense. I am asking how much more corrupt Nigeria is than Sweden; he is content to ask which is more corrupt.
Even if one is willing to settle for measuring corruption on an ordinal scale, I don’t think the measurement issue goes away. Save for the easy cases — the athleticism of Brady and Ortiz compared to Matthew; corruption in Sweden compared to that in Nigeria – the weighting problem remains. And I have my doubts about even some of the easy cases – Matthew are you ready to suit up?
Mr. Messick,
If CPI cannot be trusted as it measures the “perceived corruption” rather than actual then what Indexes would you be willing to trust, and recommend for others?
Thanks for the question although I am not quite sure what you are looking for. Are you asking about measures of corruption? If you are, then my question back is what do you want to measure — level of corruption in a country? In a sector? And why do you want the measure?
I am looking for a precise measure of actual corruption in public sector, which -if possible- measures access to public services in return for bribes, and\or corruption of legislation, a measure that mentions developing countries as well as developed ones. If I may give an example, it would be “Rule of law Index” (parts concerned with corruption in the report), and Worldwide governance index’s “control of corruption” measure.