Paul Heywood and Elizabeth Johnson raise important questions in a recent journal article about Transparency International’s corruption assessment methodology; their article deserves close attention by consumers and producers of any type of corruption assessment.  The purpose of a corruption assessment is to determine where a country is falling short in the fight against corruption and what more it needs to do.  An assessment is the backbone of any national anticorruption policy, providing both a roadmap for reform and a gauge for measuring progress, and with a wrong map or inaccurate gauge, the chances the policy will curb corruption are slight.

TI calls its corruption assessment method the National Integrity System (NIS).  One of the more than 500 different corruption assessment methodologies (or “tools” in anticorruption jargon) now in use, it is among the oldest and most widely used.  Since 2001, it has been an input into anticorruption policy in over 100 countries.  Heywood and Johnson find it has four weaknesses  –

1) Emphasis on institutions in isolation.  Heywood and Johnson explain that at its core the NIS was to be a review of “public sector anti-corruption strategies, watchdog agencies, public awareness and participation, accountability of the courts, roles of the media and the private sector, and international cooperation.”  While NIS’ originators meant this to be a suggestive rather than a prescriptive list of issues and institutions, in practice the assessments have evolved into an institutional check-list – lengthy discussions of a long list of institutions ranging from the different branches of government to individual agencies like the police and electoral commissions to political parties, the media, and other non-governmental actors.  This institution-by-institution focus ignores how these institutions interact, how certain ones reinforce or undermine others and how some take on new missions to compensate for the failure of others. It is as if a structural engineer reported on the strength of each beam in the building without any evaluation of how their interconnections shored up or threatened the structure’s overall integrity.

2) Lack of cultural sensitivity. Johnson led a 2014 NIS assessment of Cambodia, and the Heywood/Johnson critique was prompted by the poor fit between the NIS methodology and the Cambodian reality.  One of the world’s poorer nations, governed by an autocrat, still recovering from the effects of a long war and its genocidal aftermath, and highly aid-dependent, the NIS provided an uncertain guide to corruption in Cambodia or what to do about it.  Questions like whether judges had a well-developed ethics code or whether there is a judicial services commission seemed beside the point given Cambodia’s recent history.  Measured by the Millennium Development Goals, Cambodia (with its donors) has made significant progress in improving human welfare and creating a functioning state.  Yet the NIS methodology was unable to reflect these changes.  Heywood and Johnson’s complaint:

“If a country is starting from a low base, but building fast, this is surely a more positive picture than a country starting from a high base and declining. The ‘one shot’ nature of the NIS, coupled with the reductionism inherent in the [institutional summary approach] of the NIS evaluation, abstracts away important nuances about country level developments.”

3) Poor definition of “integrity.”  Heywood and Johnson laud TI’s methodology for using integrity rather than corruption as the starting point.  They argue that an integrity focus better captures the desirable characteristics of any state and the many ways in which corruption can be controlled beyond simply more vigorous enforcement of the anticorruption laws.  But in practice, they say, the NIS methodology defines “integrity” too narrowly, to mean simply the opposite of corruption.  The result of this overly narrow a definition is that the NIS does nothing more than examine rules that deter citizens from engaging in corruption.  A genuine focus on integrity broadly defined would consider issues such as whether the political system engenders trust between rulers and the ruled and other characteristics that political theorists would consider elements of civic virtue.

4) Too focused on compliance-based approaches. According to Heywood and Johnson, the result of all these weaknesses is that an NIS assessment has become nothing more than what auditors term a compliance audit.  A compliance audit lists the rules and regulations to which a business is subject and then determines whether the company is obeying each one.  Likewise, as Heywood and Johnson report, in an NIS the evaluators first ask questions such as –

“Are there examples of attempted interference by external actors, particularly the government or judiciary, in the activities of the legislature? To what extent is the symptom of the ‘revolving door’ (i.e. executive officials moving back and forth between big business and government positions) a concern? Are there any examples of undue external interference in judicial proceedings?”

The fewer “right” answers, the more corrupt the country.  And the remedy in every case is the same: reforms to increase the number of right answers.

The problem with such an approach is that, like a simple compliance audit, it focuses on rules and their violations and treats every wrong answer the same:  all must be remedied.  No attempt is made to rank the severity of the violations, their consequences, the resources needed to remedy them, or the leadership’s commitment to remedying them.

Those familiar with the other 499 plus corruption assessment methodologies will recognize that Heywood and Johnson’s critique applies to many of them as well.  No matter the starting point, my sense is that many assessment methods have evolved into little more than compliance checklists (as they have with money laundering assessments).  Given how assessments are commissioned, the current learning on corruption, and the limitations of social science techniques, that may be inevitable.  Compliance audits may be the best that can be produced.  Nor are they without real value.

But the topic is surely worthy of debate among funders, practitioners, and policymakers. Heywood and Johnson’s article is indeed part of a nascent debate in academic journals on assessment methods.  Given the importance assessments play in developing and implementing anticorruption policies, however, it is a debate that deserves a far wider audience.