In May 2016, at the London Anticorruption Summit sponsored by then-Prime Minister David Cameron, participating countries issued declarations announcing a variety of commitments—some new, some continuations of existing policies—to further the fight against international corruption. Of course, all too often governments fail to follow through on their grandiose promises, so I was heartened by Transparency International’s announcement, in September 2016, that it had gone through all the country declarations, compiled a spreadsheet identifying each country’s specific promises, and would be monitoring how well each country was following through on its commitments.

Last month, a year after TI published the spreadsheet documenting the list of summit commitments, TI released a report and an interactive website that purport to track whether countries have followed through on those commitments. So what do we learn from this tracking exercise?

Alas, the answer is “almost nothing.” TI’s “Anti-Corruption Pledge Tracker,” in its current form, is a catastrophic failure—a slapdash, amateurish collection of arbitrary, often inconsistent judgments, unsupported by anything that resembles serious research, and (ironically) non-transparent. This is all the more surprising—and disappointing—given the fact that TI has done so much better in producing similar assessment tools in other contexts. Indeed, at least one such recent tool—TI’s Government Defense Anti-Corruption Index—provides a model for what the Pledge Tracker could and should have looked like. Given the importance of tracking countries’ fulfillment of their summit pledges, and TI’s natural position as a leader on that effort, I dearly hope that TI will scrap the Pledge Tracker in its current form, go back to the drawing board, and do a new version.

I know that sounds harsh, and perhaps it seems excessive. But let me explain why I don’t find the Pledge Tracker, in its current form, worthy of credence.

I’ll focus on the assessment of the United States–partly because I have a particular interest in whether my own government is living up to its commitments, and partly because TI’s assessment of US adherence to its summit pledges is so harsh. The report describes the US performance on its pledges as “poor,” asserting that the US is “inactive” on 55% (11 out of 20) of its summit pledges. (The Tracker categorizes country performance of each pledge as “complete,” “ongoing,” “underway,” “overdue,” or “inactive.”) That headline number seemed really bad, so I took a closer look at the pledges for which TI deemed the US “inactive.” Here are a few of them, with what I hope are some useful comparisons:

One of the US pledges at the summit concerned the UN Convention Against Corruption (UNCAC). In particular, the US declared that it would “work with other countries, civil society, business, and international organizations to promote more effective implementation of” UNCAC. TI declares that the US is “inactive” on this commitment. Why? For some of the assessments, TI’s Pledge Tracker provides an “evidence” link, but there is none for the US commitment to UNCAC. To the best of my knowledge, the US is still an UNCAC member in good standing and has participated fully in the UNCAC process. True, the extent to which the US “works with” others to “promote more effective” UNCAC implementation is hard to measure, and maybe there’s no affirmative evidence of such steps. But I don’t get the sense that TI looked very hard.

Perhaps TI just decided to apply a very stringent standard here, insisting not only on UNCAC compliance/participation, but clear evidence of specific additional efforts to promote and improve the UNCAC process? To see whether that’s what’s going on, we can check TI’s assessment of other countries with equivalent UNCAC pledges. Indonesia and Spain—two countries that TI’s report calls “unanticipated standout performers”—both have UNCAC commitments that are worded similarly to the US pledge, and both of those countries are deemed as having achieved “complete” fulfillment of their pledges. Why? Well, this time both Indonesia and Spain’s UNCAC pledges have “evidence” links, so I clicked on them. Indonesia’s goes to the annual report of Indonesia’s anticorruption agency (the KPK), which contains a lot of general information but mentions UNCAC only briefly and in passing. Spain’s “evidence” link just goes to its 2016 UNCAC evaluation report (that is, UNCAC’s peer review of Spain’s compliance). Meanwhile, TI deems Jordan’s fulfillment of its UNCAC commitment “ongoing” (with an “evidence” link that goes to a lengthy Arabic document, apparently from its anticorruption commission), while three other countries with UNCAC commitments—Afghanistan, Russia, and Nigeria—are credited as “underway” with respect to those commitments. (The evidence link for Afghanistan goes to an Arabic document from the Ministry of Justice, which I suspect is the text of an anticorruption law. The Nigeria evidence link goes to a news story about a bill creating a Financial Intelligence Agency; the story doesn’t mention UNCAC. There’s no evidence link for Russia.)

On what basis did TI determine that the US is “inactive” on its pledge to implement and support UNCAC, but that Spain, Indonesia, Russia, Nigeria, Jordan, and Afghanistan all have either already fulfilled equivalent commitments or are currently doing so? Can anyone give a straight-face explanation for how such a judgment could possibly make sense? And did TI even bother to check whether the “evidence” links they included in their online Pledge Tracker actually provided such evidence, or to offer an explanation as to how those pieces of evidence demonstrate fulfillment of the commitment? I expect the answers are none, no, no, and no.

OK, moving on, but sticking for the moment with pledges that concern supporting international anticorruption efforts: TI also declares the US “inactive” on its pledge to “continue to support the review mechanisms of the UNCAC, the InterAmerican Convention against Corruption, OECD Working Group on Bribery, and Council of Europe.” An “inactive” rating on this pledge sounds really bad, doesn’t it? It sounds like perhaps the US has stopped supporting these other organizations. Has it?

No, it hasn’t. To the best of my knowledge, the US is still a member in good standing of the first three of these organizations, and has continued its support for the fourth. On what basis did TI conclude that the US no longer supports, to take one of these examples, the OECD Anti-Bribery Working Group’s review mechanism? Is there a link to the OECD webpage announcing withdrawal of US support? No, there isn’t. There’s no link to anything. Now, admittedly, neither this pledge nor the UNCAC pledge was much of a commitment in the first place. The US basically promised to continue to do what it already had been doing for some time. But to declare that the US is “inactive” on either pledge implies a change for the worse, and that implication seems wholly unjustified.

Here again, to get a sense of whether TI is consistently applying an appropriate standard, it helps to compare TI’s assessment of the US commitment to continue to support UNCAC, the IACAC, and the OECD Convention with TI’s assessment of similar pledges by other countries. I found two such pledges in the Pledge Tracker. First, Indonesia gets credit for its “complete” fulfillment of its pledge to strengthen an entity called the South East Asian Parties Against Corruption (SEA-PAC). But when you click on the “evidence” link, it takes you to a Memorandum of Understanding regarding SEA-PAC’s objectives from 2014–two years before the London Summit even occurred. So, how can TI conclude both that the US has not fulfilled a commitment to continue supporting UNCAC and the OECD—when in fact there’s been no change—but Indonesia gets credit for having fulfilled its commitment to strengthen SEA-PAC, when no evidence of any new initiative from Indonesia on that front?

Oh, and the other country that gets credit from TI for “complete” fulfillment of a pledge to support UNCAC? That would be Russia, which pledged to “support and strengthen central coordinating role of the UN in international anticorruption cooperation and promote the implementation of resolutions and decisions adopted by the sixth Conference of the States Parties to the UN Convention against Corruption, including [the] St. Petersburg statement on promoting public-private partnership in prevention of and fight against corruption.” The evidence link here goes to a document (in Russian) from the 2015 UNCAC Conference of States Parties. What does this tell us about whether Russia has been active in promoting the implementation of the resolutions adopted at that Conference? Nothing. (Russia, by the way, also gets credit for “complete” fulfillment of its pledge to “implement the principles of the Open Data Charter,” but the “evidence” link just goes to the Open Data Charter itself—from a UK government website—and provides no evidence that Russia has actually taken any steps to implement it.)

Here’s another example along similar lines: The US pledged at the London Summit to support the World Bank/UNODC Stolen Asset Recovery Initiative (StAR). So did Indonesia. According to TI, the US is “inactive” on this pledge, while Indonesia has achieved “complete” fulfillment of its similar pledge. Why the difference? Again, it’s not at all clear. There’s no “evidence” link for the US evaluation—which is odd, given that one would think it’s an easily verifiable fact whether the US is providing support to StAR. (UNODC’s StAR information webpage still lists the US as one of the contributors to StAR’s multi-donor trust fund.) For Indonesia, the “evidence” for “complete” fulfillment of its pledge “to strengthen [StAR]” is a link that takes you to the text of a 2008 Mutual Legal Assistance Treaty. (No, the date is not a misprint. So far as I know StAR didn’t even exist in 2008, or if it did it had just gotten started.)

Let’s take something a bit more concrete, and perhaps a bit easier to measure objectively, though not entirely uncontroversially: enforcement of laws against foreign bribery. At the London Summit, the US pledged to “continue to prosecute cases of violation of the Foreign Corrupt Practices Act (FCPA).” Yet again, TI deems the US “inactive” on this pledge. That is, not to put a fine a point on it, one of the most misinformed things I’ve ever read in any publication from a reputable organization working on anticorruption issues.

I recognize that there’s some uncertainty about the future of FCPA enforcement under the Trump Administration, which I’ll get to in a moment. But let’s remember that the relevant time period for assessing US compliance with its pledge to continue to enforce the FCPA begins either in May 2016 (the date of the summit) or September 2016 (when TI complied its list of pledges and announced it would be tracking compliance). In January 2017 alone, the US government resolved six FCPA matters, resulting in a total of $256.5 million in penalties. Three of the ten largest FCPA cases ever, as measured by total penalties, were resolved during the period when, according to TI, the United States was “inactive” in prosecuting FCPA cases: Och-Ziff (September 2016), Teva (December 2016), and Telia (September 2017). Two other big FCPA cases, JP Morgan Chase (November 2016) and Odebrecht (December 2016), were also resolved during this period. (Odebrecht would have cracked the top ten but for a subsequent decision to reduce the penalties in light of the company’s ability to pay.)

So where does TI get the idea that the US is “inactive” in FCPA enforcement? Well, perhaps TI, like many others (including me), is worried that FCPA enforcement might decline under the Trump Administration, and these fears were stoked by the relatively little (but not zero) FCPA action we’ve seen since the inauguration. That’s a reasonable worry, and one I share to some degree, though my worries are more about the longer term. But that doesn’t mean the US is no longer active in this area. Indeed, many have argued that the apparent drop-off in enforcement actions in early 2017 doesn’t actually mean much, and expect FCPA enforcement to continue with little change. So there’s a debate to be had on whether, under Trump, the US will indeed sustain its past level of FCPA enforcement. I suppose it’s not impossible for an organization like TI to make a case for the pessimistic assessment that, whatever else may have happened in the last 12-16 months, at this point US FCPA enforcement is “inactive”–if we’re willing to put aside the fact that, on its face, the Pledge Tracker claims to be a backward-looking evaluation, not a forward-looking prediction. But to make such a case, TI would have to provide some evidence and argumentation.

And what evidence does TI provide for the conclusion that US FCPA enforcement should now be considered “inactive”? Well, here the Pledge Tracker does include an “evidence” link, but embarrassingly (for TI), the link goes to a July 2017 Wall Street Journal blog entry that makes the exact opposite point. The full piece is behind a paywall, but the headline, in big bold letters, is “FCPA Dry Patch Means Little for Enforcement Efforts,” and the first sentence is, “The has been a relative paucity of foreign-bribery cases by the U.S. government since [the] President took office, but the U.S. Justice Department and defense attorneys don’t believe there is any letup in enforcement efforts” (emphasis added). So, TI’s only evidence for the facially absurd claim that the US has been inactive in enforcing the FCPA in the past year is a piece asserting that there has not been any letup in FCPA enforcement efforts since Trump’s inauguration. (And indeed TI’s spreadsheet with more detail on the evaluations behind the Pledge Tracker evaluations states, in a note on this commitment: “The United States continues to prosecute FCPA cases, though at a somewhat reduced pace from recent years. Department of Justice officials have reaffirmed the Department’s commitment to enforcement of the FCPA and research has presented plausible explanations for the slowed pace of enforcement in the first half of 2017. Advocates will continue to observe enforcement activity in the coming months.” Yet this note appears right next to the designation “Inactive”–in bright scary orange.)

Oy.

How did this happen, and what can be done to fix it?

In terms of how it happened, I have a guess. According to TI’s description of its methodology, neither the TI UK office (which oversaw the Pledge Tracker project) nor the TI Secretariat actually did the evaluations themselves. That makes total sense – after all, doing evaluations like these requires sufficient staff, as well as country-specific expertise, often including language skills. But instead of carefully selecting individual country experts, TI outsourced the evaluation of individual countries to TI’s national chapters. (Because TI currently lacks a US chapter, the US monitoring task was assigned to three other organizations, Global Witness, ONE, and Global Financial Integrity.) The national chapters (or surrogates, in the US case) were instructed to contact the responsible government departments, and to search for publicly available information on the commitment. I suspect that the country organizations were not given enough guidance beforehand, and TI didn’t do adequate review and quality control after the fact. I wouldn’t be a bit surprised, for example, if the US organizations never contacted anyone in the US government to request information related to the US government’s summit commitments, even though they were supposed to. I also suspect that the national organizations varied wildly in how they applied the vague standards in TI’s guidance document. TI Indonesia and TI Spain were probably “soft graders,” while the US organizations were either really harsh graders, or just plain lazy or incompetent in gathering information.

There’s also a bit of a problem built into TI’s definition of the “inactive” category. According to TI’s guidance document, the “inactive” label should be assigned if “no activity has taken place on this commitment” OR if “there is no accessible public evidence of activity having taken place on this commitment.” The “inactive” language is therefore highly misleading: Most casual readers probably won’t realize that “we looked and couldn’t find anything” leads to an “inactive” rating (as opposed to a “not enough information” rating). And this problem interacts with the previous problem of different chapters/organizations applying different standards. I wouldn’t be a bit surprised, for instance, if what happened with the US is that the responsible organizations delegated the task of doing the actual evaluations to some 22-year-old unpaid intern, who then did a quick Google search, and if nothing came up that clearly showed recent activity, the intern just assigned an “inactive” rating, and knocked off early to get to the bar in time for Happy Hour. And then nobody at TI bothered to check carefully or insist on some kind of evidence for claims that are facially outlandish.

In terms of what should be done at this point, I have two related recommendations:

• First, TI should immediately take the Pledge Tracker and associated materials down from its website, issue a brief statement apologizing for premature publication of incompletely vetted draft materials, and promise to issue a revised Pledge Tracker as soon as it’s ready.
• Second, TI should then redesign the project from the ground up, using as a model TI’s own Government Defense Anti-Corruption Index, to which I alluded above. This post has already gone on for far too long, so I won’t go into great detail about this index, except to say that, while one can always quibble with this or that, it has all the desirable features that the current version of the Pledge Tracker so conspicuously lacks: The questions put to the outside evaluators, though often subjective, are precise, and the criteria used for making the evaluations are clear. The evaluators use multiple sources (often over a dozen) to assign the scores in each category, and all of them are listed, along with some commentary on the reasoning behind the scores. Each evaluation is subjected to peer review by 2-3 outside experts, and the peer review commentary on each assessment is included and easily accessible through the interactive webpage. (Here, for example, are the assessments for the United States and Indonesia.) It seems to me that a similar model, with appropriate adaptations, can be used for the Pledge Tracker. If this is doable in roughly six months, then TI can time the publication of the new and improved Pledge Tracker to coincide with the two-year anniversary of the London Summit (in May 2018).

The above suggestions may seem extreme. But for an organization like TI, its credibility is its greatest asset. And the project of monitoring countries’ adherence to their summit commitments is important enough that it needs to be done right. So, please, TI, swallow your pride and fix this, as fast as you possibly can. And if the advice to start over from scratch is too bitter a pill to swallow at this stage, at least consider a major review of the assessments and a drastic revision of how the material is presented.