Last month, we had a spirited debate in the anticorruption blogosphere about the conceptualization of corruption, academic approaches to the study of the topic, and the relationship between research and practice. (The debate was prompted by provocative piece by Bo Rothstein, to which I replied; my critical reaction prompted a sur-reply from Professor Rothstein, which was followed by further contributions from Robert Barrington, Paul Heywood, and Michael Johnston.) I’ve been thinking a bit more about one small aspect of that stimulating exchange: How do we, or should we, think about evaluating the success (or lack thereof) of an anticorruption policy or other intervention? I was struck by the very different assessments that several of the participants in last month’s exchange had regarding whether the anticorruption reform movement had been “successful,” and this got me thinking that although part of the divergence of opinion might be due to different interpretations of the evidence, part of what’s going on might be different understandings of what “success” does or should mean in this context.

That observation, in turn, connected to another issue that’s been gnawing at me for a while, that I’ve been having trouble putting into words—but I’m going to take a stab at it in this post. My sense is that when it comes to defining and measuring “success” in the context of anticorruption reform (and probably many other contexts too), there’s a fundamental tension between two conflicting impulses:

On the one hand, anticorruption advocates would like to promote an evidence-based reform strategy, one that prioritizes interventions that can be rigorously assessed—if not quantitatively, then at least through reliable qualitative evidence based on expert or participant observation. Emphasizing rigorous monitoring, evaluation, and testing of “what works” (as practitioners sometimes put it) helps ensure that that those supporting anticorruption reform efforts can invest their resources most effectively and efficiently. This approach is also helpful in building political and institutional support for anticorruption reform efforts, because advocates can point to interventions that have a proven track record.

On the other hand, there is a widespread understanding that, at least in those societies where corruption is a systemic problem, fighting corruption is a long-term process, one that requires fundamental changes in the political, social, and economic order. These changes can take decades, or even generations, to accomplish. This is not to say that policy interventions or institutional reforms are irrelevant, and that the transition from endemic corruption to norms of integrity, when it occurs, is due to “deep” processes of social evolution that policymakers cannot affect. In fact, “good government” reforms of various types may be a critical part of the process. But the process takes so long that the effects of even an extremely important and successful reform might not be felt for years after adoption. Moreover, the process of political and societal transformation is sufficiently complex that it may be impossible to isolate and trace the impact of any individual reform.

But of course this gives rise to a challenge for advocates and reformers, as well as for researchers trying to evaluate the success or failure of various approaches to anticorruption. Contrary to the periodic assertions that all anticorruption interventions have been abject failures (or the more moderate criticism that we still have no idea of “what works” when it comes to fighting corruption), in fact we do have a fair amount of evidence on the short-term impact of specific interventions. (The most obvious example that leaps to mind concerns various forms of financial oversight, such as public expenditure tracking and regular audits. We also have qualitative reports of how certain interventions have changed behavior on the ground, or how legal reforms have altered incentives in ways that discourage bribery.) But this evidence is sometimes dismissed, or at least downplayed, because these incremental changes, which often involve more low-level transactional corruption, do not indicate meaningful changes in the larger order and/or the prevailing norms of governance.

That’s a fair point. But when we turn to that larger question, and ask what sorts of interventions might help—and what can learn from past experience—it turns out that it’s extremely hard to say anything with great confidence. (Side note: Distrust anyone who does say anything with great confidence about how societies and norms of governance transform. Scholars have some educated guesses and intriguing theories, but on this subject extreme intellectual humility is essential.) I can’t stress enough that this does not mean that reformers should not pursue this larger transformative agenda. In my view, it is vitally important that they do so. And researchers (academics and others) have a crucial role to play here, maybe even more so than in the context of shorter-term project assessments, because in the absence of reliable observable evidence about the effects of a given intervention, we must rely even more on theory, by which I mean simplified mental models of how the world works that can provide a coherent and parsimonious account of important known facts, and that can generate plausible hypotheses and predictions that can provide some guidance, however provisional and imperfect, to practitioners. But in this context “success” is virtually impossible to assess rigorously, at least not with our existing tools.

What follows from all this? I’m not quite sure, but let me offer a few concluding thoughts.

• First, I do think that at least some of the disagreement over whether anticorruption interventions have been successful or unsuccessful arises because people have different understandings of “success” in mind, and end up talking past one another.
• Second, I think (returning to my above observation about theory) that different people have different unspoken mental models of the relationship between short-term, measurable, incremental progress and long-term systemic change. My conjecture (and it is only that!) is that some people implicitly assume that every little bit helps, and that small but concrete improvements are both substantively meaningful and will cumulatively contribute to larger transformations in governance norms; others, by contrast, tend to see the sorts of positive changes attributable to discrete interventions as marginal at best, and as having little to do with the bigger-picture struggle for systemic change. So, when presented with a study showing that a new auditing system for local governments in Country X (where corruption is reputed to be rampant) resulted in a 12.6% reduction of missing funds, one person would be inclined to celebrate this as an important success and evidence that meaningful progress is possible in Country X, while another person would think that this is mostly beside the point when we know most high-level politicians in Country X still embezzle substantial sums and elections largely financed with off-the-books illicit donations from wealthy private interests who expect and receive favorable government treatment in return. Which of these views is closer to the truth is itself an empirical question, one to which we do not yet have a definitive answer (and to which the answer may depend on a host of contextual factors).
• Third, I tend to think that both perspectives have something useful to offer, and that whichever way we lean, we should try to make an effort to see things from the other perspective as well. Those who are more practically-minded and concerned with measurable on-the-ground improvements must avoid falling into the trap of thinking that if something isn’t measurable it isn’t important, or of losing sight of the bigger picture, or of neglecting the relevance to anticorruption of changes that are not on their face directly about corruption per se. And those who tend to think in bigger-picture terms must take care not to neglect or dismiss the cumulative importance of seemingly marginal changes, must stay appropriately humble about our collective ignorance regarding macro-systemic change, and must be mindful of the political and institutional imperatives that may drive a focus on measurable short-term outputs and outcomes.
• Fourth, we could all stand to be a bit clearer in explaining what we mean by “success” in this context. It’s a bit funny to me that the anticorruption community spends so much time and energy (an excessive amount, in my view) arguing about the correct conceptualization, definition, and measurement of “corruption,” yet seems to pay comparatively little attention to explicitly discussing the proper conceptualization, definition, and measurement of “success,” as applied to any given anticorruption intervention, or the anticorruption movement as a whole.

Those are just a few preliminary thoughts. I’d welcome further thoughts—and vigorous disagreement—from readers.