Jesper Johnsøn, a Senior Adviser at the the U4 Anti-Corruption Resource Centre who leads the Centre’s “evaluation and measurement” theme, contributes the following guest post:
In development policy jargon, a “program evaluation” is a systematic and objective assessment of an ongoing or completed project or policy, including its design, implementation, and results. Very few aid agencies have incorporated corruption considerations into their standard program evaluations–despite the fact that these same agencies have focused heavily on corruption measurement (as a separate endeavor) since the mid-1990s. This is a mistake: A good evaluation should include consideration of corruption, given that program success can be threatened by waste, leakage, and outright theft of resources, and also that evaluations can be useful tools for corruption risk management, accountability, and learning about how to build better anticorruption mechanisms. Part of the explanation for the failure to integrate corruption considerations into program evaluation may be the difficulty of rigorously measuring corruption levels and impact, yet as I have argued elsewhere, this difficulty should not be used as an excuse for not evaluating anticorruption efforts systematically.
Aid organizations can and should incorporate corruption issues into their standard evaluation policies. But not all program evaluators are anticorruption experts, and so the anticorruption community needs to provide more guidance on how to integrate anticorruption analysis into program evaluation. This is one of the things we are trying to do at U4. Based on our work, and helpful discussions with other experts, here are some suggestions for how this can be achieved:
First, corruption concerns should be explicitly integrated into the terms of reference for program evaluators, in order to encourage the evaluators to explicitly analyze corruption risks and costs. The four questions below are all relevant and can be adapted for different programs:
- What proportion of resources are lost directly to corruption within programs?
- How much is lost to waste and leakage?
- What is the impact of corruption and leakage on attainment of your overall development objective?
- How cost-effective would anticorruption measures be, or how cost-effective have they been?
Given the right designs and adequate resources, these are all answerable questions. Obtaining the information will not only help organizations learn how to reduce corruption costs, but will also provide valuable information to the public and the politicians who support development work. Of course, not all evaluations can answer the above four questions at the same time. Questions three and four in particular need good planning and resources. Nonetheless, organizations need a policy for how to respond to these questions as part of the overall evaluation portfolio.
Second, in addition to revising the terms of reference to include anticorruption considerations, evaluators need guidance on how corruption ought to be considered in the context of a program evaluation. A good starting point is to situate anticorruption considerations in the context of the familiar OECD evaluation criteria: (1) relevance, (2) effectiveness, (3) efficiency, (4) impact, and (5) sustainability. Each of these five standard evaluation criteria can easily be connected to corruption, and doing so will help program evaluators integrate corruption considerations into a familiar framework they already employ. Those connections can be made as follows:
- Relevance has to do with assessing whether interventions pursue the right objectives and activities. Since anticorruption is meant to be mainstreamed in most development cooperation, it would be relevant for many programs to consider corruption issues.
- Effectiveness is a measure of the extent to which an activity reaches its objectives. Here, it is straightforward to show that corruption can often be a major factor in the achievement or non-achievement of objectives.
- Efficiency includes both qualitative and quantitative measures of outputs in relation to inputs (i.e., whether a project could have achieved the same or better results with the same or fewer resources). “Leakage,” or other sorts of resources lost due to corruption, obviously reduces program efficiency, and effective anticorruption measures may therefore result in a net gain of resources, and improvement in overall cost-effectiveness, as I have discussed in more depth elsewhere.
- Impact captures the positive and negative changes produced by an intervention, including both intended and unintended results, taking into account the impact of external factors. Although the “external factors” considered are usually things like terms of trade and financial conditions, a change in corruption levels in a country or sector is also a relevant external factor (or perhaps even an unintended consequence of the program) that should be captured in the evaluation.
- Sustainability involves assessing whether the benefits of an activity are likely to continue after the program ends. If you don’t design and implement effective corruption risk management systems it can undermine sustainability as funds may be diverted from intended use.
Third, we need to recognize that one reason that evaluators may be discouraged from digging too deep into corruption issues is that they are told that writing about corruption is simply too political. Aid agencies should squarely address this concern by promulgating clearer guidelines that make it explicit that corruption must be considered as part of the evaluation process, and that it is not “too political” to take this into account.
One of the issues which has been discussed a few times on his blog, notably in Sarah’s post on sextortion (https://globalanticorruptionblog.com/?s=sextortion) is how to incorporate non-traditional/non-economic corruption into anti corruption strategies. I wonder if the evaluation world faces the same challenges in terms of monitoring and evaluating non-monetary corruption (sextortion, political favors, etc.) associated with development programs? Would development agencies be able to use the questions and terms of reference you describe to incorporate these issues?
At a time when not having an adequate anticorruption measure in a system could be a crime, this commentary on in-built anticorruption lenses in the evaluation system is definitely a commendable work.
Hi Jesper, thanks for this interesting post. As someone who works in a development agency (DFID) on anti-corruption, and with a strong interest in evaluation, this was right up my street! You make some helpful suggestions, and in a perfect world donors should certainly be doing lots more of this kind of work. But we have a long way to go before we can put this into practice:
– Firstly, in order to get any reliable information on the amount of programme funds that are lost to waste or corruption you will have to pore through partner organisation’s (sometimes developing country Governments’) documents including procurement files etc. This will end up looking more like a forensic audit than a conventional evaluation, and this kind of methodology is usually only used when there is a specific cause for concern… doing this tends to cause serious problems in the relationship between the donor and the implementing partner.
– Secondly, it’s not just that writing about corruption is too ‘political’ but that many donors are unwilling to put hard numbers on the amount of resources that are lost to corruption in their programmes. These kind of figures can be used in the media to undermine the case for aid.
I am not trying to say we shouldn’t be doing the kind of corruption evaluations you suggest – in fact I think they would be an important step towards greater accountability to taxpayers and intended beneficiaries of aid. They might also help us to form a better understanding of things like the cost effectiveness of safeguards (we have some safeguards on aid which are probably quite ineffective even though they are costly to implement). But you/we will probably need to deal with quite a bit of resistance before this idea turns into reality.
Thank you all for good comments! Justin, you raise two good reasons why evaluations have had difficulties preventing corruption in the past. Reliable data is almost always a scarcity, not just in the corruption field. You are right that to get the type of data needed to make reasonable estimates on levels of corruption we need tools from the audit trade. I disagree that it has to be as extensive as a forensic audit, but the main point is that the traditional view of audits on one hand and evaluations on the other should be revised. Audits should provide data that can feed into an broader evaluation exercise. Think performance audits where you use evaluation methods such as cost-benefit and cost-effectiveness analysis. On your second point, I agree that hard numbers on waste and leakage can seem scary, but I also think that it is wrong to believe that citizens in the UK or elsewhere currently do not think that a substantial proportion of funds are lost to corruption. An argument against ODA is often that “all the money goes into the wrong pockets anyway.” Well, we know that this is not true. Most funds do reach their intended beneficiaries. I would not be so afraid of an informed debate as some other people. Heather Marquette has recently made this case quite well in her Communications Note for the OECD DAC.
Melanie, I just wanted to thank you for raising one of my favourite points: corruption is more than bribes and fraud, and in our quest to quantify costs we should not disregard those costs that are difficult to quantify but may be much more damaging. You mention sextortion, but the list could go on to include land grapping, nepotism, conflict of interest, etc. As a minimum, appraisals should ensure that initiatives do not promote such practices, and evaluations should document that they have not taken place due to the initiative.