Paul Heywood and Elizabeth Johnson raise important questions in a recent journal article about Transparency International’s corruption assessment methodology; their article deserves close attention by consumers and producers of any type of corruption assessment. The purpose of a corruption assessment is to determine where a country is falling short in the fight against corruption and what more it needs to do. An assessment is the backbone of any national anticorruption policy, providing both a roadmap for reform and a gauge for measuring progress, and with a wrong map or inaccurate gauge, the chances the policy will curb corruption are slight.
TI calls its corruption assessment method the National Integrity System (NIS). One of the more than 500 different corruption assessment methodologies (or “tools” in anticorruption jargon) now in use, it is among the oldest and most widely used. Since 2001, it has been an input into anticorruption policy in over 100 countries. Heywood and Johnson find it has four weaknesses –
1) Emphasis on institutions in isolation. Heywood and Johnson explain that at its core the NIS was to be a review of “public sector anti-corruption strategies, watchdog agencies, public awareness and participation, accountability of the courts, roles of the media and the private sector, and international cooperation.” While NIS’ originators meant this to be a suggestive rather than a prescriptive list of issues and institutions, in practice the assessments have evolved into an institutional check-list – lengthy discussions of a long list of institutions ranging from the different branches of government to individual agencies like the police and electoral commissions to political parties, the media, and other non-governmental actors. This institution-by-institution focus ignores how these institutions interact, how certain ones reinforce or undermine others and how some take on new missions to compensate for the failure of others. It is as if a structural engineer reported on the strength of each beam in the building without any evaluation of how their interconnections shored up or threatened the structure’s overall integrity.
2) Lack of cultural sensitivity. Johnson led a 2014 NIS assessment of Cambodia, and the Heywood/Johnson critique was prompted by the poor fit between the NIS methodology and the Cambodian reality. One of the world’s poorer nations, governed by an autocrat, still recovering from the effects of a long war and its genocidal aftermath, and highly aid-dependent, the NIS provided an uncertain guide to corruption in Cambodia or what to do about it. Questions like whether judges had a well-developed ethics code or whether there is a judicial services commission seemed beside the point given Cambodia’s recent history. Measured by the Millennium Development Goals, Cambodia (with its donors) has made significant progress in improving human welfare and creating a functioning state. Yet the NIS methodology was unable to reflect these changes. Heywood and Johnson’s complaint:
“If a country is starting from a low base, but building fast, this is surely a more positive picture than a country starting from a high base and declining. The ‘one shot’ nature of the NIS, coupled with the reductionism inherent in the [institutional summary approach] of the NIS evaluation, abstracts away important nuances about country level developments.”
3) Poor definition of “integrity.” Heywood and Johnson laud TI’s methodology for using integrity rather than corruption as the starting point. They argue that an integrity focus better captures the desirable characteristics of any state and the many ways in which corruption can be controlled beyond simply more vigorous enforcement of the anticorruption laws. But in practice, they say, the NIS methodology defines “integrity” too narrowly, to mean simply the opposite of corruption. The result of this overly narrow a definition is that the NIS does nothing more than examine rules that deter citizens from engaging in corruption. A genuine focus on integrity broadly defined would consider issues such as whether the political system engenders trust between rulers and the ruled and other characteristics that political theorists would consider elements of civic virtue.
4) Too focused on compliance-based approaches. According to Heywood and Johnson, the result of all these weaknesses is that an NIS assessment has become nothing more than what auditors term a compliance audit. A compliance audit lists the rules and regulations to which a business is subject and then determines whether the company is obeying each one. Likewise, as Heywood and Johnson report, in an NIS the evaluators first ask questions such as –
“Are there examples of attempted interference by external actors, particularly the government or judiciary, in the activities of the legislature? To what extent is the symptom of the ‘revolving door’ (i.e. executive officials moving back and forth between big business and government positions) a concern? Are there any examples of undue external interference in judicial proceedings?”
The fewer “right” answers, the more corrupt the country. And the remedy in every case is the same: reforms to increase the number of right answers.
The problem with such an approach is that, like a simple compliance audit, it focuses on rules and their violations and treats every wrong answer the same: all must be remedied. No attempt is made to rank the severity of the violations, their consequences, the resources needed to remedy them, or the leadership’s commitment to remedying them.
Those familiar with the other 499 plus corruption assessment methodologies will recognize that Heywood and Johnson’s critique applies to many of them as well. No matter the starting point, my sense is that many assessment methods have evolved into little more than compliance checklists (as they have with money laundering assessments). Given how assessments are commissioned, the current learning on corruption, and the limitations of social science techniques, that may be inevitable. Compliance audits may be the best that can be produced. Nor are they without real value.
But the topic is surely worthy of debate among funders, practitioners, and policymakers. Heywood and Johnson’s article is indeed part of a nascent debate in academic journals on assessment methods. Given the importance assessments play in developing and implementing anticorruption policies, however, it is a debate that deserves a far wider audience.
Very helpful piece. This is something that we at Global Integrity are giving serious thought to – what sorts of corruption, integrity, or governance assessments can actually generate meaningful data, that is useful to reformers in particular countries?
More broadly, we’re exploring along with many others what an anti-corruption agenda which is less about supposed “best practice” and assessing against that benchmark, and more about “best fit” in particular political contexts, might look like. The work of Mushtaq Khan and colleagues at the DFID-funded Anti-Corruption Evidence Centre seems like a very useful way forward. https://ace.soas.ac.uk/
Oh, can you add the link/reference to the Heywood/Johnson piece please.
Thanks for weighing in and glad to know I am not alone in my concerns about assessment methods. I think incorporating some sense of priorities through a risk assessment may be a promising avenue. But as I suggested in my earlier post on risk assessments, many risk assessments are primitive to say the least.
Clicking on “in a recent journal article” in the first line should bring up the Heywood/Johnson article.
Several years ago I was an M&E advisor to INL (US State Department) and witnessed many ‘assessments’ of the criminal justice sector (including corruption) suffer from all the points noted in this article; most pointedly the checklist issue. It was this experience that led to our Fletcher School/CDA effort to develop a systems-centric analysis methodology to understand corruption in the CJS that incorporates an understanding of the connections between factors. Here is a link to the Central Africa Republic analysis (blog post) which was our third pilot of this methodology: http://cdacollaborative.org/blog/pity-man-stands-alone/ We will be coming out shortly with the full methodology with the hopes that others can use, build and improve it.
Pingback: Assessing Corruption Assessments: TI’s National Integrity System | Matthews' Blog
Hmmm… I agree with much of what you (and Heywood & Johnson) have to say about this. But at the same time, I think it’s sometimes a little to easy to launch these critiques without fully appreciating the challenges associated of assessing something this complicated.
More generally, there’s a well-understood trade-off, with respect to designing any assessment tool, between complexity and accuracy. Checklists, as you point out, have all sorts of problems. But they also have some advantages — their simplicity makes them low-cost to implement and less dependent on the subjective judgment calls of the person making the evaluation, and they can call attention to certain problems and deficiencies in a clear, focused way. Those advantages aren’t realized if the checklist is garbage, of course. But sometimes simple checklists might capture, say 40% or 60% or 75% of what we care about, and in those cases the relevant question is not whether they might go wrong in some cases, but whether the marginal benefit of a more refined/complex assessment tool is worth the marginal cost.
Which brings me to the other side of the equation: A lot of the more complex alternative assessment tools I’ve seen bandied about are _really_ complex — so much so that it’s not always clear whether the results would be the same if the analysis were performed by others, and in any event extremely resource intensive. It’s relatively easy to call for more complexity, but one shouldn’t assume that more complex tools are always better.
The question comes down — as it often does — to what exactly the tool is to be used for. Sometimes low-cost simple checklists are the way to go. Sometimes an in-depth, nuanced institutional analysis is necessary. Sometimes we need a combination, or something else. And in this context the issue is even harder because these institutional integrity/corruption assessments have both a managerial/technocratic purpose and a political purpose, and there may sometimes be a trade-off between them.
I don’t think I’ve said anything you’d disagree with, nor do I disagree with anything you say in your post. But I wanted to chime it because the qualifications that you rightly note in your penultimate paragraph might need some additional amplification and elaboration.