José-Miguel Bello y Villarino, an official with the Spanish Ministry of Foreign Affairs and doctoral candidate at the University of Sydney, contributes today’s guest post:

There is a wide debate about how to produce and use data to assess and compare countries’ performance, particularly in domains that are, by nature, global such as human rights. In the corruption domain there are some well-known international indexes that purport to express a country’s perceived corruption level in a single number, such as the Corruption Perceptions Index (CPI) published annually by Transparency International (TI). Other diagnostic tools have been developed to assess individual countries’ anticorruption frameworks and policies against some global standard or benchmark. Among the latter, TI produces the National Integrity System (NIS) Country Assessments.

These assessments do not try to determine how much corruption there is in a country, but rather “how well a country tackles the problem.” NIS assessments do not aim to give each country a final “score” that can be compared to the scores of other countries. The assessments’ declared objective is to look into the effectiveness of each country’s anticorruption institutions by focusing on a standard set of “pillars” (things like democratic institutions, the judiciary , the media, and civil society). Consequently, NIS assessments are not meant to provide definitive conclusions, but rather observations within a common framework to supply a starting point for analysis, and to identify risks and possible areas for improvement. Their conclusions are designed to help stakeholders work to develop more concrete and country-specific responses.

The NIS Country Assessments, and similar tools (TI has identifies roughly 500 diagnostic tools used in the anticorruption area), have come in for a fair share of criticism. Much of this criticism centers upon their allegedly formalistic, formulaic, standardized approach to assessing anticorruption institutions. Some of those criticisms have appeared on this blog. A few months ago Richard Messick posted a commentary on a piece by Paul Heywood and Elizabeth Johnson that challenged the relevance and value of NIS reports for developing democracies (using Cambodia as an illustrative example), principally due to insufficient appreciation of cultural distinctiveness and an overemphasis on compliance-based approaches. Last month, Alan Doig’s post continued this conversation. Mr. Doig defended the value of the NIS Country Assessments as they were originally conceived, but argued that TI’s current approach to NIS assessments has become overly formalistic, which limits the utility of NIS country studies as an effective starting point for analysis or platform for progression. Though coming from a different perspective, Mr. Doig’s criticism is very similar to the core argument of Professors Heywood and Johnson. In essence, they share a skepticism that one can usefully apply broad global standards or categories to individual countries, given each country’s unique, particular, idiosyncratic circumstances.

Respectfully, I think these criticisms go too far. Taking individual country circumstances into consideration of course has value. However, standardization of assessment methodologies, the somewhat “formulaic” approach, can have benefits that may outweigh the costs. Continue reading →