WAGs: What’s the Harm?

GAB is pleased to publish this Guest Post by Maya Forstater, well-known analyst on business and sustainable development, on a topic of continuing concern to scholars and activists working on corruption and development matters.

Are unreliable guesstimates  and made-up statistics mildly irritating, indispensably powerful  or potentially dangerous in the public debates on corruption? The topic comes up so often on the Global Anti-Corruption Blog that it has been given its own own three-letter acronym: WAGs (or Wild Ass Guesses).

Those at the sharp end of advocacy maintain, with some justification, that in the battle for attention, an arrestingly big number makes all the difference. But as Rick has argued, overinflated figures can also cause harm.

Something similar happens on the related topic of tax and illicit flows. One example of this is the widespread belief that ‘developing countries lose three times more to the tax avoidance by multinational companies than they receive in aid’. This much quoted WAG gives the impression of huge potential gains for the poorest countries, but is based on a chain of misunderstandings .  In practice the magnitudes of revenues at stake are likely to be several times smaller than aid  for the countries where that comparison matters.

Similarly, broad estimates of illicit flows or the scale of the black economy (“trillions”) are often presented in ways that suggest that the sums to be gained from tackling corporate tax avoidance are larger than any serious analysis supports.

I have written about these big numbers previously in a paper published by the Centre for Global Development here (or here  for the short version).

But what harm do such numbers do, compared to their power at getting people talking about the issues? Is it really worth pointing out misunderstandings and myths in pursuit of a more rigorous and careful approach to evidence? (Or as I have been asked‘ Do you ever wonder how much you help the tax abusers?’)

I see four key dangers from inflated perceptions of the numbers:  Continue reading

The 2014 CPI Data Demonstrates Why, Even Post-2012, CPI Scores Cannot Be Compared Over Time

A little while back, I expressed some skepticism about whether Transparency International’s Corruption Perceptions Index (CPI) scores can be compared across time, even after TI changed its methodology in 2012 and claimed that its new scores would now be comparable across years.  More recently, I criticized TI’s 2014 CPI for burying the information on the margins of error associated with the CPI values, and for wrongly asserting that changes in the CPI score between 2013 and 2014 for certain countries (most notably China) were substantively meaningful.  (In fact, not only does the change in China’s score between 2013 and 2014 seem not to be statistically significant, but the change was due almost entirely to the dropping of a source in which China did abnormally well in 2013, and an abnormally large movement in a single other source.) I decided to follow up on this by taking a closer look at the other ten countries that TI singled out as having experienced significant CPI changes (in either direction) between 2013 and 2014.

Upon closer examination, I’m even more certain that CPI scores cannot be compared over time. I’m also more confident in my judgment that TI has been unforgivably sloppy — and downright misleading — in how it, and its representatives, have portrayed the substantive significance of these CPI changes. It turns out that the problem I found with the China calculations was not unusual. For almost all of the eleven countries TI identified as big movers, the CPI changes were driven by (1) the addition or elimination of sources from year to year for particular countries, and/or (2) abnormally large (indeed, implausibly large) movements in a single source. Until TI fixes its methodology, the safest thing to do is to ignore year-to-year changes in the CPI. And for the sake of preserving its own integrity and credibility, TI should either (A) persuasively explain why I am wrong in my analysis of the data (in which case I will gladly concede error), or (B) issue some sort of retraction or correction to its earlier press releases, and either drop the claim that post-2012 CPI scores can be compared across time or fix its methodology going forward.

Allow me to elaborate my analysis of the data: Continue reading