Guest Post: Using Big Data to Detect Collusive Bidding in Public Procurement

Bence Tóth and Mihály Fazekas of the Corruption Research Center Budapest (CRCB) contribute the following guest post:

As several earlier posts on this blog have discussed (see, for example, here, here, and here), collusion and corruption in public procurement is a significant problem, one that is extremely difficult to detect and combat. The nature of public procurement markets makes collusion easier to sustain, as pay-offs are higher (demand is often inelastic due to the auction mechanisms used), administrative costs increase entry barriers, and the transparency of procurement contract awards–often intended as an anticorruption device–can actually make it easier for cartel members to monitor one another and punish cheating. Law enforcement agencies have tried various techniques for breaking these cartels, for example by offering leniency to the first company that “defects” on the other cartel members by exposing the collusive arrangement. However, although leniency policies have sometimes proven to be an effective tool to fight coordinated company behavior, the efficacy of this approach is limited given the relative unlikelihood that the government will ever acquire convincing evidence of collusion absent such a defection by an insider. Hence, there is great need for alternative methods to identify collusive rings and guide tradition investigation.

In many markets, using quantitative indicators to detect collusion has not been feasible, as gathering meaningful tender-level data (or even market-level data) is too costly, or simply impossible. However, in the case of public procurement markets, there is a huge amount of publicly available data, which makes the use of “Big Data” techniques to pinpoint collusion-related irregularities more feasible. Indeed, in collaboration with our colleagues at CRCB, we have developed a simple, yet novel approach for detecting collusive behavior. Continue reading

Big Data and Anticorruption: A Great Fit

There is no shortage of buzz about Big Data in the anticorruption world. It’s everywhere — from public efforts like Transparency International’s public procurement analysis to cutting-edge private-sector FCPA compliance programs implemented by Ernst & Young. TI has blogged about Big Data and corruption, with titles like “Can Big Data Solve the World’s Problems, Including Corruption?” and “The Potential of Fighting Corruption Through Data Mining.” Ernst & Young’s conclusion is more definite: “Anti-Corruption Compliance Now Requires Big Data Analytics.”

In previous posts, contributors to this blog have written about how the anticorruption community was excited about social media-style apps (“crowdsourcing”) in anticorruption efforts. Apps like iPaidABribe allow citizens to report their encounters with corrupt officials, generating a fertile data set for anticorruption activists. Big Data is a related effort: activists can mine huge amounts of data for patterns that reveal corrupt activity, making it a powerful tool for transparency. However, as the name suggests, Big Data requires massive amounts of data in order to be useful.The anticorruption community should throw its weight behind proposals to open up data sets for Big Data analysis. As with crowdsourced anticorruption efforts, the excitement surrounding Big Data could quickly turn into disappointment unless this tool can be integrated into the broader anticorruption effort. Continue reading

The U4 Proxy Challenge–some quick reactions

One of the big challenges in anticorruption work, which I suspect we will be discussing quite a bit on this blog, concerns the measurement of corruption. After all, there are a bunch of different theories about the causes and consequences of corruption, and about the best way to combat it. Testing these theories requires some way of measuring the extent of corruption (or different forms of the corruption problem). And for folks actually doing anticorruption work (donors, governments, NGOs, etc.), it would be nice to be able to assess how well programs are working. Yet all of the existing measures have significant problems.

To try to inspire some creative thinking about new ways to measure corruption, the good people at the U4 Anti-Corruption Resources Centre (affiliated with the Christian Michelsen Institute in Bergen, Norway), with the assistance of the UK’S Department for International Development (DFID), recently sponsored a competition (the “Proxy Challenge”) to come up with new proxies that would help track the progress of anticorruption reform initiatives. U4 hosted a one-day workshop last month to let the five finalists present their proxies, to choose a winner, and to promote some general discussion of the challenges of developing useful proxies for corruption in a variety of contexts. I was able to attend. I’ll try to post a some more substantive thoughts in a later post, but here are a few quick reactions. Continue reading