Last week Matthew asked if he were the only one who wasn’t excited about Sustainable Development Goal 16. At first glance it is hard to understand why he would ask such a question. One of 17 goals approved September 25 by the United Nations General Assembly to end poverty by 2030, SDG 16 establishes an ambitious agenda for improving the way the nations of the world govern their citizens by, among other measures, requiring concerted global action to “substantially reduce corruption and bribery in all their forms.” How could anyone, particularly one who works on corruption issues, not be ecstatic that the 193 member-states of the United Nations unanimously endorsed this objective? And indeed numerous anticorruption advocates have already celebrated its approval (click here for Transparency International’s enthusiastic endorsement).
Although the opening of Matthew’s post was low-key (am I the only one not excited?), readers quickly learned that he was in fact severely critical of SDG 16’s corruption and bribery target because of the way progress towards realizing it is to be measured: by changes in a nation’s score on Transparency International’s Corruption Perceptions Index. Matthew nicely summarized why this is insane on technical grounds. Here I explain why using the CPI to measure progress is not only insane but represents a major setback in the fight against corruption.
Some years back I spent a week with senior officials of a newly-created anticorruption agency. The purpose of my visit was to help them think through the many critical issues a new anticorruption agency must resolve — from recruitment of staff to an organization chart to the balance between enforcement and prevention. In the end, however, we spent the bulk of the time dissecting the CPI.
The reason was that the annual CPI rating had appeared just before I arrived, and the country’s score had declined slightly. As a result, civil society was up in arms and the prime minister was demanding the agency raise the country’s score by next year. It thus fell to me to explain why the index was not a valid, reliable measure of the level of corruption and that it was indeed likely that many of the steps the agency might take to reduce corruption would, by generating press coverage about corruption in the country, lower the country’s score on next year’s index.
It took time to convince the several senior staff of this point as they came from different disciplines and different arguments appealed to different individuals. The last to be convinced was the agency’s head, and his poignant reaction when he fully grasped what I was saying sticks with me to this day. “Well,” he said with a look of resignation, “now I understand what you are saying but the prime minister never will. Nor will civil society. So what can we do?”
I didn’t have much to offer in reply. I said I could send him and his colleagues copies of the growing literature explaining why the CPI was not a good measure. I could write articles for the local press summarizing them or help him and his colleagues write articles and speeches about CPI. I also volunteered to organize in-country seminars and workshops where the many academic experts who have shown why the CPI is fatally flawed could expound on their work. But that was all I could do, and it was clear the agency head and his colleagues didn’t think a public relations campaign to show citizens and government leaders that, despite the enormous attention the CPI garners each year, it was a worthless measure was likely to be of any help.
The agency was created amid much fanfare, enjoying at least the verbal support of the government of the day and surely the genuine support of civil society. During my first visits I was struck by its leaders’ enthusiasm and their sense of mission. But in the several years since our discussion the agency has not lived up to expectations, and while it would be too much to ascribe its disappointing record completely to the CPI, it certainly didn’t help morale when the agency’s senior staff learned that not only was there nothing they could do to improve the country’s score on the chief measure by which their efforts would be judged but that genuine attempts to combat corruption could well worsen the country’s score. Indeed, what could be a worse way to launch an anticorruption agency?
Now you see why I think the UN’s endorsement of the CPI to measure progress fighting corruption is a Major Setback in the Fight Against Corruption?
GAB | The Global Anticorruption Blog 
I fully agree with Rick and Matthew. With TI introducing new scale from 0-10 to 0-100 and claiming new scores to be comparable from one year to the other, I attempted to have comments and views from Prof. John G. Lambsdroff – inventor of CPI. Unfortunately, he declined to make one. If SDG goal 16 is to be measured solely based on CPI then we will be making a gross mistake.
I doubt TI will ever drop the CPI, all experts understand that it’s deeply flawed methodologically but it’s a golden communications tool. TI gets onto the front pages of media worldwide once a year, and nearly every time a journalist discusses corruption anywhere, he or she adds on some free advertising for TI by citing the country’s CPI score.
So, the famous “TI Index” is here to stay. The question is how to change its methodology. One very simple step that would help is to abandon the comparative rankings and just report the score (ratings not rankings). This way, if a country improves from 4.4 to 4.5, it will be seen as an improvement, even if five other countries meanwhile overtake it in the index by improving 0.2 points.
And yes, I do realize that an 0.1 or 0.2 point improvement is meaningless. But most journalists don’t, and that’s the key target audience here :))
I’m actually more sympathetic to the CPI than are many others (I think including you and Rick), at least when it’s used appropriately. As a snapshot cross-country measure, it can be useful, so long as we keep in mind that it’s only measuring perceptions (which TI itself emphasizes) and that there’s often a substantial margin of error associated with the score estimates (which TI is usually also pretty good at pointing out). And as I’ve discussed in an earlier post, there’s actually some decent evidence that the CPI correlates (albeit imperfectly) with other, more objective corruption measures (see https://globalanticorruptionblog.com/2014/04/18/objective-validation-of-subjective-corruption-perceptions/). Also, though Rick points out an important disadvantage of the salience of the CPI, it’s possible that this salience can have good political effects as well, even if the measure is flawed (something else I’ve discussed in an earlier post: https://globalanticorruptionblog.com/2014/03/21/does-the-social-value-of-corruption-indicators-depend-solely-on-their-accuracy/).
But where I have concerns — and where I agree with you entirely — is when people want to use year-to-year, within-country changes to measure changes in corruption or, worse, the success or failure of anticorruption policies. I won’t belabor all the reasons why here, since I’ve outlined my concerns in earlier posts, and you touch on some as well. But that move strikes me as extremely problematic.
One methodological question, though: My impression was that when TI altered its methodology in 2012 (partly in an attempt to make the CPI scores comparable over time), it addressed the specific problem you raise about relative rankings. I think under the current methodology, if the source scores for country X improve by an average of 5 points, country X’s CPI score will improve by 5 points, even if the source scores for every other country in the sample improve by an average of 10 points. (Though to keep everything in the same range, scores are bounded from below at 0 and from above at 100.)
I think Fredrik Galtung in 2006 nicely summed up the problems with the CPI see https://www.academia.edu/2229126/Corruption_definition_and_measurement . At the end of the day the CPI methodology is deeply flawed and should not be used by anyone as a benchmark.
If the methodology of SDG 16 a fatal flaw, how should we approach the problem? I find it hard to believe that the best option is to ignore corruption in the sustainable development goals. I am also uneasy with the concept that it would be good to adopt an anti-corruption goal yet leave it without any way to measure efficacy (one of the most effective parts of the MDGs was the fact that they could be more or less easily measured, though sometimes this led to non-ideal incentives, i.e. countries had a reason to make sure children were in school, but not to ensure they were learning anything while there.)
Corruption is extremely hard to measure. What’s the best way to structure things to achieve a good outcome if countries are willing to make anti-corruption an SDG?
Good question. I wonder if measuring the strength of certain institutions that correlate with lower instances of corruption might be a way to move the ball forward, recognizing that there are flaws with year-to-year CPI as a measurement, and also that there are not a number of apparent alternatives to CPI. Correlated indices are not optimal, but if other elements were controlled for they might paint a rough picture of progress within a country.
How to measure the strength of institutions, and which institutions to choose is a separate question. In terms of institutions, an independent judiciary, anticorruption agency, and whistleblower protections are a few that come to mind. However, each country is different, and “cookie cutter” institutional reforms should not be the goal of SDG 16.
Dear all,
Thanks for the interesting discussion and willingness to look at the more difficult questions about Goal 16. I believe it is good to debate whether the goal has been framed right and what it actually means for us who recognise the impact that governance and anti-corruption have on sustainable development.
Through my work with Transparency International, I have collaborated with many partners to shape the post-2015 agenda for the past three years to ensure that governance was not left off the agenda. Nationally, TI national chapters – from UK to Indonesia, Germany to Mexico, Bangladesh to Liberia – have been critical to the success of seeing Goal 16 included and approved. This is a watershed moment for the anti-corruption movement. We should not undervalue this. It is the first time that corruption and development are formally linked for all countries. This is not a North/South divide.
Now to the questions being asked about the CPI:
1.) Is the CPI being recommended to measure target 16.5: No. Officially, the group tasked with selecting the indicators – the Interagency and Expert Group (IAEG) has published their top list: http://unstats.un.org/sdgs/files/open-consultation-iaeg/Summary%20of%20Comments%20on%20Indicator%20Proposals%2025-9-15.pdf. For target 16.5, the recommended indicator draws on people’s experiences with petty bribery. This question is very much aligned with Transparency International’s Global Corruption Barometer (question 7): http://www.transparency.org/gcb2013/in_detail.
2.) Does Transparency International think the CPI should be used to measure SDG progress on target 16.5?: No. As active contributors to the discussions on indicators, TI has not advocated for the CPI as a measurement for this target. TI recognises the usefulness of the CPI in and of itself, but also it limitations, which we point out in our CPI material (http://www.transparency.org/cpi2014/in_detail). For Target 16.5, it is not a useful measure, first and foremost, because it would hard to measure comparative change (as the countries are indexed against each other). This is one of the reasons why, while the World Governance Indicators are fantastic, they would not be useful here. This same logic applies to the use of any index as a data source for the SDG indicators. In the lead up to the launch of CPI 2015, to be released in early 2016, TI will put out a more detailed analysis of the CPI, and its strengths and weaknesses. So, watch this space ”
While we are at the end of one long road on getting Goal 16 adopted, we are also at the beginning of another on how to implement it.
Let’s work together to get it right.
Thanks very much for this helpful information and commentary. I had not seen the current IAEG-SDG draft list of indicators, reflecting the most recent round of comments, so this is very helpful. (I’d been focused on the SDN report to the IAEG-SDG, which I’d seen over the summer — I guess I was out of date, so thanks very much for the correction!) I’m much less troubled by the use of regular experience surveys as a way to measure progress toward Target 16.5. (That said, I can imagine some might criticize the mismatch between the stated target — reducing corruption and bribery “in all their forms” — and the indicator, which, even if accurate, clearly measures only a certain kind of petty bribery. But to my mind, better a narrow indicator that actually tells us something than a broad indicator that doesn’t.) I’m glad to know that my concerns were, for the most part, off target (thought it remains to be seen what the IEG-SDG will end up doing in the end).
There are, however, a couple of things you said it your comment with which I might have a friendly and respectful disagreement.
First, on the inter-temporable comparability of the CPI: I’m happy to hear you (and perhaps other people at TI) share my view that the CPI is not useful for measuring comparative change within countries. And I’m even happier to hear that the 2015 CPI will be accompanied by a more detailed discussion of this and other issues. But at least at the moment, the official TI position does not appear to be consistent with what you say in your comment. The link to “CPI details” that you included specifically says (wrongly, in my view) that after 2012, CPI values CAN be compared across years within particular countries. And several of your colleages at TI, including senior people in the research unit have specifically told me that they believe not only that the CPI can be used for such inter-temporal comparisons, but that it could conceivably be used (along with other measures) to assess progress on SDG Target 16.5. It’s not clear that TI is really speaking with one voice on this, but perhaps this will all be cleared up when the 2015 CPI is released.
Second, I continue to be mildly puzzed as to why so many people think, as you put it, that the adoption of SDG is a “watershed moment.” I get that many TI chapters and other activists and advocates lobbied hard to get governance and anti-corruption included in the SDGs. And you/we won. But what, exactly, did we win? You say this is the first time that corruption and development have been “formally linked.” If you mean by that, explicitly linked in a UN document, that’s perhaps almost right (the UNCAC preamble, as I’ve pointed out, already mentions how corruption undermines national economies and thwarts progress toward sustainable development, but I’ll concede that those comments are brief in-passing mentions in a longer list of bad consequences of corruption). But why is that important? The idea that corruption is bad for development has been widely recognized and endorsed for at least a decade. All the major development agencies, as well as virtually all national governments in developing countries, have embraced the idea that corruption is bad for development. There hasn’t been a meaningful North/South divide on that issue for a long time (though many divides, along many lines, still exist regarding policy specifics). Smart people whom I respect keep repeating that SDG 16 is this big breakthrough moment, even if we put aside the measurement issues. But for the life of me, I just don’t understand why. I do get the idea that explicit quantitative proogress indicators, for which countries will be held accountable, is a different approach, for better or worse. But that doesn’t seem to be what most people in the anticorruption community seem to bee so excited about. If you or anyone out there can explain exactly what is so valuable about having governance and anticorruption included in the SDGs, I’d really appreciate it. And I don’t mean that in a snarky or sarcastic way at all — I’m genuinely puzzled, and would love to get a fuller explanation.
I thank Craig as well for clarifying the issue about the “official” measurement for goal 16.5. But the current candidate, people’s experience with petty bribery, is not going to be easy to measure. As Art Kraay and Peter Murrell have shown and Clarke, Friesenbichler and Wong recently confirmed (Comparative Economic Studies 57, 103-135 (March 2015)), surveys that ask people about their experience paying bribes produce unreliable results because many respondents are reluctant to admit they paid a bribe. What’s worse, the reluctance to answer truthfully appears to vary systematically across nations. Hence, without correcting for this variance, cross-national comparisons are not possible.
In the latest compilation of comments on how to measure the SDGs the government of Cuba states about the proposed measure for 16.5: “Remove the indicator; not possible to measure and lacks methodological definition.”
President Obama recently lifted the U.S. embargo on Cuba arguing it had not been successful. The Cuban government’s observation about the impossibility of measuring corruption offers further support for the President’s action. Despite the several decade effort by successive American governments to isolate the Cubans, they have been able to keep up with the best thinking in the social sciences.
Pingback: Guest Post: Connecting Health and the Fight Against Corruption | GAB | The Global Anticorruption Blog
Pingback: Performance-Based Lending as an Anticorruption Tool | Anti Corruption Digest
Pingback: Innovative or Ineffective?: Performance-Based Lending as an Anticorruption Tool | GCNI
Pingback: A Reminder: Year-to-Year CPI Comparisons for Individual Countries are Meaningless, Misleading, and Should Be Avoided | GAB | The Global Anticorruption Blog