Guest Post: Going Beyond Bribery? Improving the Global Corruption Barometer

Coralie Pring, Research Expert at Transparency International, contributes today’s guest post:

Transparency International has been running the Global Corruption Barometer (GCB) – a general population survey on corruption experience and perception – for a decade and a half now. Before moving ahead with plans for the next round of the survey, we decided to review the survey to see if we can improve it and make it more relevant to the current corruption discourse. In particular, we wanted to know whether it would be worthwhile to add extra questions on topics like grand corruption, nepotism, revolving doors, lobbying, and so forth. To that end, we invited 25 academics and representatives from some of Transparency International’s national chapters to a workshop last October to discuss plans for improving the GCB. We initially planned to focus on what we thought would be a simple question: Should we expand the GCB survey to include questions about grand corruption and political corruption?

In fact, this question was nowhere near simple to answer and it really divided the group. (Perhaps this should have been expected when you get 25 researchers in one room!) Moreover, the discussion ended up focusing less on our initial query about whether or how to expand the GCB, and more on two more basic questions: First, are citizen perceptions of corruption reflective of reality? And second, can information about citizen corruption perceptions still be useful even if they are not accurate?

Because these debates may be of interest to many of this blog’s readers, and because TI is still hoping to get input from a broader set of experts on these and related questions, we would like to share a brief summary of the workshop exchange on these core questions. Continue reading

In Bribery Experience Surveys, Should You Control for Contact?

Perception-based corruption indicators, though still the most widely-used and widely-discussed measures of corruption at the country level, get a lot of criticism (some of it misguided, but much of it fair). The main alternative measures of corruption include experience surveys, which ask a representative random sample of firms or citizens about their experience with bribery. Corruption experience surveys are neither new nor rare, but they’re getting more attention these days as researchers and advocates look for more “objective” ways of assessing corruption levels and monitoring progress. Indeed, although some early discussions of measurement of progress toward the Sustainable Development Goals (SDGs) anticorruption target (Target 16.5) suggested—much to my chagrin—that changes in Transparency International’s Corruption Perceptions Index (CPI) score would be the main measure of progress, more recent discussions appear to indicate that in fact progress toward Goal Target 16.5 will be assessed using experience surveys (see here and here).

Of course, corruption experience surveys have their own problems. Most obviously, they typically only measure a fairly narrow form of corruption (usually petty bribery). Also, there’s always the risk that respondents won’t answer truthfully. There’s actually been quite a bit of interesting recent research on that latter concern, which Rick discussed a while back and that I might post about more at some point. But for now, I want to put that problem aside to focus on a different challenge for bribery experience surveys: When presenting or interpreting the results of those surveys, should one control for the amount of contact the respondents have with government officials? Or should one focus on overall rates of bribery, without regard for whether or how frequently respondents interacted with the government?

To make this a bit more concrete, imagine two towns, A and B, each with 1,000 inhabitants. Suppose we survey every resident of both towns and we ask them two questions: First, within the past 12 months, have you had any contact with a government official? Second, if the answer to the first question was yes, did the government official demand a bribe? In Town A, 200 of the residents had contact with a government official, and of these 200, 100 of them reported that the government official they encountered solicited a bribe. In Town B, 800 residents had contact with a government official, and of these 800, 200 reported that the official solicited a bribe. If we don’t control for contact, we would say that bribery experience rates are twice as high in Town B (20%) as in Town A (10%). If we do control for contact, we would say that bribery experience rates were twice as high in Town A (50%) as in Town B (25%). In which town is bribery a bigger problem? In which one are the public officials more corrupt?

The answer is not at all obvious; both controlling for contact and not controlling for contact have potentially significant problems: Continue reading

The 2016 CPI and the Value of Corruption Perceptions

Last month, Transparency International released its annual Corruption Perceptions Index (CPI). As usual, the release of the CPI has generated widespread discussion and analysis. Previous GAB posts have discussed many of the benefits and challenges of the CPI, with particular attention to the validity of the measurement and the flagrant misreporting of its results. The release of this year’s CPI, and all the media attention it has received, provides an occasion to revisit important questions about how the CPI should and should not be used by researchers, policymakers, and others.

As past posts have discussed, it’s a mistake to focus on the change in each country’s CPI score from the previous year. These changes are often due to changes in the sources used to calculate the score, and most of these changes are not statistically meaningful. As a quick check, I compared the confidence intervals for the 2015 and 2016 CPIs and found that, for each country included in both years, the confidence intervals overlap. (While this doesn’t rule out the possibility of statistically significant changes for some countries, it suggests that a more rigorous statistical test is required to see if the changes are meaningful.) Moreover, even though a few changes each year usually pass the conventional thresholds for statistical significance, with 176 countries in the data, we should expect some of them to exhibit statistical significance, even if in fact all changes are driven by random error. Nevertheless, international newspapers have already begun analyses that compare annual rankings, with headlines such as “Pakistan’s score improves on Corruption Perception Index 2016” from The News International, and “Demonetisation effect? Corruption index ranking improves but a long way to go” from the Hidustan Times. Alas, Transparency International sometimes seems to encourage this style of reporting, both by showing the CPI annual results in a table, and with language such as “more countries declined than improved in this year’s results.” After all, “no change” is no headline.

Although certain uses of the CPI are inappropriate, such as comparing each country’s movement from one year to the next, this does not mean that the CPI is not useful. Indeed, some critics have the unfortunate tendency to dismiss the CPI out of hand, often emphasizing that corruption perceptions are not the same as corruption reality. That is certainly true—TI goes out of its way to emphasize this point with each release of a new CPI— but there are at least two reasons why measuring corruption perceptions is valuable: Continue reading

Guest Post: The Metaphysics of “Corruption” (or, The Fundamental Challenge to Comparative Corruption Measurement)

GAB is pleased to welcome back Jacob Eisler, Lecturer at Cambridge University, who contributes the following guest post:

A couple months back, Matthew Stephenson and Michael Johnston engaged in a lively debate on the question of if aggregate-level data of corruption is useful, focusing on the appropriate level of methodological skepticism that should be directed towards large-scale efforts to quantify corruption (see here, here, here, and here). While this debate touched on a number of fascinating questions regarding how to best treat data regarding corruption, it has drifted away from why Michael had a concern with overly aggressive quantification in the first place: Actually addressing corruption requires a “standard of goodness,” and the difficulty in coming up with such a standard explains why the social sciences have faced a “longstanding inability to come to a working consensus over how to define corruption.” In other words, when we talk about corruption, we are inevitably talking about something bad that suggests the vitiation or distortion of something good. It is difficult to conceptualize corruption except as a distortion of a non-objectionable political process—that is, political practice undertaken with integrity. This need not mean that there must be some shared first-order property of good governance; but it does suggest that there is a shared property to distorted or corrupted governance that must derive from some shared property of all politics.

If this idea of a “shared feature” is taken seriously, it would suggest those who argue for the value of comparative corruption metrics are making a very strong claim: that if you are comparing corruption within a country, or across countries, all the relevant polities and types of practice must have some shared feature, deviation from which counts as corruption. This shared feature in turn would be an aspect of governance. It could be any number of constants in human society – a constant feature of morality in governance, or tendencies of human anthropology. But in any case, this is a very distinctive and powerful claim, and one that requires strong assumptions or assertions regarding the nature of governance. To weave this back to the original dispute, our willingness to rely on quantitative metrics should depend on our level of commitment to our faith in this constant feature of politics that makes corruption a transferable, or, more aggressively put, “universal” thing. Our use of these homogenizing empirical metrics implies that we are committed to the robustness of the constant feature. Yet it doesn’t seem like this conceptual work has been done. Continue reading

Guest Post: When and How Will We Learn How To Curb Corruption?

GAB is pleased to welcome Finn Heinrich, Research Director at Transparency International, who contributes the following guest post:

Listening to conversations about corruption among global policy-makers, corruption researchers, and anticorruption activists alike, I can’t help but notice that the focus of anticorruption research and policy is changing. The 1990s focused mainly on demonstrating that corruption exists and finding ways to measure it (largely through perception-based indicators), and the early 2000s were about assessing corruption risks in specific countries, sectors, or communities, and assessing the performance of anticorruption institutions. More recently, researchers (and their funders and clients) are shifting from the “Where is corruption?” question toward the “How can we fight corruption?” question. They ask: Do we know what works, when, where, and under which circumstances in curbing a specific type of corrupt behavior?

Answering such questions is extremely challenging. Corruption’s clandestine nature makes it difficult to measure, data is often of low quality or simply not available for time-series or cross-sectional analysis beyond aggregate country-level indicators. Furthermore, anticorruption interventions often lack an underlying theory of change which would be needed to design robust research evaluations to find out whether they worked and if so, how (and if not, why not). And we lack realistic but parsimonious causal models which can take account of contextual factors, which are so important to understand and tackle corruption, as corruption is an integral part of broader social and political power structures and relationships which differ across contexts. Similarly, there is a lack of exchange between micro-level approaches focusing on specific, usually local anticorruption interventions, on the one hand, and the macro-level literature on anti-corruption strategies and theories, on the other.

While we at Transparency International certainly do not have any ready-made solutions for these extremely tricky methodological and conceptual issues, we are committed to joining others in making headway on them and have therefore put the “what works” question at the heart of our organizational learning agenda by engaging in reviews of the existing evidence as well as ramping up impact reviews of some of our own key interventions. For example, we have just released a first rapid evidence review on how to curb political corruption, written by David Jackson and Daniel Salgado Moreno, which showcases some fascinating evidence from the vibrant field of political anticorruption research. We are also working with colleagues from Global Integrity on a more thorough evidence review on corruption grievance as a motivator for anti-corruption engagement and are planning further evidence reviews and impact evaluations.

As we start to get our feet wet and figure out how to best go about generating and making sense of the existing evidence on what works in anti-corruption, we are keen to engage with the broader anticorruption research community. Maybe there are others out there who have some ideas about how to go about learning about what works in fighting corruption? If so, please use the comment box on this blog or get in touch directly at acevidence@transparency.org.

Are Aggregate Corruption Indicators Coherent and/or Useful?: Further Reflections

Last week, I used Professor Michael Johnston’s recent post on the methodological and conceptual problems with national-level perceived corruption indicators as an opportunity to respond to some common criticisms of research that relies on these indicators. In particular, I have frequently heard (and interpreted Professor Johnston as advancing) two related criticisms: (1) composite indicators of “corruption” are inherently flawed because “corruption” is a multifaceted phenomenon, comprised of a range of diverse activities that cannot be compared on the same scale, let alone aggregated into a single metric; and (2) corruption is sufficiently diverse within a single country that it is inappropriate to offer a national-level summary statistic for corruption. (These points are related but separate: One could believe that corruption is a sufficiently coherent concept that one can sensibly talk about the level of “corruption,” but still object to attempting to represent an entire country’s corruption level with a single number; one could also endorse the idea that national-level summary statistics can be useful and appropriate, even when there’s a lot of intra-country variation, but still object to the idea that “corruption” is a sufficiently coherent phenomenon that one can capture different sorts of corruption on the same scale.) For the reasons I laid out in my original post, while I share some of the concerns about over-reliance on national-level perceived corruption indicators, I think these critiques—if understood as fundamental conceptual objections—are misguided. Most of the measures and proxies we use in studying social phenomena aggregate distinct phenomena, and in this regard (perceived) corruption is no different from war, wealth, cancer, or any number of other objects of study.

Professor Johnston has written a nuanced, thoughtful reply (with a terrific title, “1.39 Cheers for Quantitative Analysis”). It is clear that he and I basically agree on many of the most fundamental points. Still, I think there are still a few places where I might respectfully disagree with his position. I realize that this back-and-forth might start to seem a little arcane, but since so much corruption research uses aggregate measures like the Corruption Perceptions Index (CPI), and since criticisms of these measures are likewise so common, I thought that perhaps one more round on this might not be a bad idea.

Let me address the two main lines of criticism noted above, and then make some more general observations. Continue reading

The Level-of-Aggregation Question in Corruption Measurement

Recently I learned that CDA Collaborative (a nonprofit organization that works on a variety of development and conflict-resolution projects) has launched a new blog on corruption. Though it’s a new platform, they already have a few of interesting posts up, and it’s worth a look.

While I’m always happy to advertise new platforms in the anticorruption blogosphere, in this post I mostly want to focus on the first entry in the CDA’s new blog, a post by Professor Michael Johnston entitled “Breaking Out of the Methodological Cage.” It’s basically a critique of the anticorruption research literature’s alleged (over-)reliance on quantitative methods, in particular cross-national regression analyses using country-level corruption indices (such at the Corruption Perceptions Index (CPI) or Worldwide Governance Indicators (WGI) graft index). There are some things in Professor Johnston’s post that I agree with, and much that I disagree with. I want to focus on one issue in particular: the question of the right unit of analysis, or level of aggregation, to use when attempting to measure corruption.

Professor Johnston has two related complaints (or maybe two variants on the same underlying complaint) regarding these national-level perceived corruption measures. First, he complains these “[o]ne dimensional indices tell us … that corruption is the same thing everywhere, varying only in amount[.]”  In other words, corruption indices lump a whole bunch of disparate phenomena together under the same umbrella term “corruption,” ignoring the internal diversity of that category. Second, he contends that “relying … on country-level data is to assume that corruption is a national attribute, like GDP per capita” when in fact “corruption arises in highly specific processes, structural niches, and relationships.” Corruption, he explains, is not an attribute of countries, but of more specific contexts, involving “real people … in complex situations[.]”

Respectfully, I think that these points are either wrong or irrelevant, depending on how they are framed. Continue reading