Kevin Davis, the Beller Family Professor of Business Law at New York University School of Law, contributes today’s guest post, based on his recent working paper.

Academics and policymakers enthusiastically endorse “evidence-based” policymaking, for obvious reasons. (After all, what is the alternative? Faith? Popularity contests?) But while evidence—including quantitative evidence—is often helpful, we must be mindful of the limits on what empirical analysis can tell us about important topics. Take the regulation of transnational bribery. Scholars and policymakers would like to know if the current regime—laws like the U.S. Foreign Corrupt Practices Act (FCPA) and U.K. Bribery Act, and international instruments like the OECD Anti-Bribery Convention—has “worked.” That is, have these instruments reduced bribery by the firms that they cover? And did those laws have additional, possibly undesirable collateral consequences, for example reducing investment in countries perceived to be corrupt?

The most sophisticated efforts to answer these questions (see, for example, here and here and here) essentially rely on what social scientists call “natural experiments.” First, the intervention (the law or policy change) of interest, which (in a borrowing from medical terminology) researchers call the “treatment.” Next, one must identify the population of interest—say, firms or countries—and an outcome of interest (such as the frequency of bribery or the level of investment). Then, the researcher identifies the subset of those entities that are affected by the intervention (for example, the firms that fall under the jurisdiction of the new anti-bribery law); this is the “treatment group.” The researcher also identifies another subset of entities—the “control group”—that appears otherwise similar to the treatment group, but did not receive the treatment (for example, a group of firms that are outside the jurisdiction of the new law). The big difference between a “controlled experiment” and a “natural experiment” is that in a controlled experiment the researcher can randomly choose which members of the population receive the treatment (for example by randomly selecting some patients to get a new drug and giving the other patients a placebo), but in a natural experiment, the assignment of the treatment is done not by the researcher, but by some “natural” process in the world. In trying to figure out the effect of an anti-corruption law, it generally is not feasible to conduct a controlled experiment: researchers can’t decide that these firms but not those firms, selected at random, will fall under the jurisdiction of an anti-bribery law. So the best that researchers can do is to rely on natural experiments and try to account as best they can for possible differences between the control group and the treatment group by including additional control variables in a multivariate regression.

Unfortunately, when it comes to studying the effects of transnational anti-bribery laws, these sorts of studies face several fundamental challenges, which are all too often overlooked or understated. Continue reading →