José-Miguel Bello y Villarino, an official with the Spanish Ministry of Foreign Affairs and doctoral candidate at the University of Sydney, contributes today’s guest post:

There is a wide debate about how to produce and use data to assess and compare countries’ performance, particularly in domains that are, by nature, global such as human rights. In the corruption domain there are some well-known international indexes that purport to express a country’s perceived corruption level in a single number, such as the Corruption Perceptions Index (CPI) published annually by Transparency International (TI). Other diagnostic tools have been developed to assess individual countries’ anticorruption frameworks and policies against some global standard or benchmark. Among the latter, TI produces the National Integrity System (NIS) Country Assessments.

These assessments do not try to determine how much corruption there is in a country, but rather “how well a country tackles the problem.” NIS assessments do not aim to give each country a final “score” that can be compared to the scores of other countries. The assessments’ declared objective is to look into the effectiveness of each country’s anticorruption institutions by focusing on a standard set of “pillars” (things like democratic institutions, the judiciary , the media, and civil society). Consequently, NIS assessments are not meant to provide definitive conclusions, but rather observations within a common framework to supply a starting point for analysis, and to identify risks and possible areas for improvement. Their conclusions are designed to help stakeholders work to develop more concrete and country-specific responses.

The NIS Country Assessments, and similar tools (TI has identifies roughly 500 diagnostic tools used in the anticorruption area), have come in for a fair share of criticism. Much of this criticism centers upon their allegedly formalistic, formulaic, standardized approach to assessing anticorruption institutions. Some of those criticisms have appeared on this blog. A few months ago Richard Messick posted a commentary on a piece by Paul Heywood and Elizabeth Johnson that challenged the relevance and value of NIS reports for developing democracies (using Cambodia as an illustrative example), principally due to insufficient appreciation of cultural distinctiveness and an overemphasis on compliance-based approaches. Last month, Alan Doig’s post continued this conversation. Mr. Doig defended the value of the NIS Country Assessments as they were originally conceived, but argued that TI’s current approach to NIS assessments has become overly formalistic, which limits the utility of NIS country studies as an effective starting point for analysis or platform for progression. Though coming from a different perspective, Mr. Doig’s criticism is very similar to the core argument of Professors Heywood and Johnson. In essence, they share a skepticism that one can usefully apply broad global standards or categories to individual countries, given each country’s unique, particular, idiosyncratic circumstances.

Respectfully, I think these criticisms go too far. Taking individual country circumstances into consideration of course has value. However, standardization of assessment methodologies, the somewhat “formulaic” approach, can have benefits that may outweigh the costs.

To be clear, I am not trying to defend all diagnostic tools based on standardized assessments. The methodology of some is simply horrendous. Nor am I arguing against qualitative evaluation or nuanced explanations as an essential part of an overall diagnosis. And, of course, I accept that there are some situations that really are exceptional, where the usual evaluation template doesn’t apply. (A country in, or in the immediate aftermath of, a devastating civil war might be an example of such a situation.) But at the same time, for all the defects and limitations of a standardized approach as embodied by the (current) NIS Country Assessments, this approach has many advantages that should not be ignored or casually dismissed.

• First, standardization facilitates comparability. Indeed, comparability almost always comes at the expense of a certain degree of simplification. Such simplification is an acceptable price to pay if it helps make measurement easier, more objective, and more transparent—especially when all the parties involved in the process understand the trade-offs that have been made, and why.
• Second, standardization helps ensure that that certain questions are consistently asked of all countries, and that the answers are considered carefully. After all, how would it help to not assess whether a country has an independent anticorruption agency or a prosecutor’s office, or whether this agency or office is actually performing its mandate? Surely it would be better to make this assessment, irrespective of the country-specific context.
• Third, and related to the above points, an overemphasis on particularities and idiosyncrasies can be used as an excuse to open the door to exceptionalisms that fit quite poorly with our declared objectives of global promotion of universal standards of well-being. True, we don’t want to fall in the trap of endorsing a one-size-fits-all solution for all countries. At the same time, if we go too far in the other direction, and assume that every country is different and that there aren’t any presumptive benchmarks or starting points, then it’s far too easy to rationalize every institutional pattern that we see as somehow “adaptive” to that country’s unique circumstances.

The problem, then, is not the use of a standardized framework as such. Rather, the problem is failure to apply such frameworks correctly, and to be mindful of their strengths and limitations. As my co-author Ramona Vijeyarasa and I detail in a commentary on Sally Engle Merry’s book concerning this issue, the right approach is not to assume that “formalistic” diagnostic tools are inappropriate, but rather to ask, with respect to any of these tools, the following five questions:

1. What are these “formalistic approaches” are trying to do, and is the objective of the exercise clear and useful?
2. Was the tool developed appropriately, with a solid design and appropriate opportunities for participation?
3. What is the tool meant to assess, and is that object of assessment the true outcome of interest or rather a proxy?
4. How is the data collected and processed?
5. How is the data used, who uses it, and for what?

Criticisms of diagnostic tools for being (too) “standardized” or “formulaic” are absolutely legitimate, notably if such standardization does not align with their declared objectives or comes attached to a flawed methodology. Criticisms targeted at a given tool for not delivering something it was never designed to do… well, not so much.