With apologies to those readers who couldn’t care less about methodological issues associated with corruption experience surveys, I’m going to continue the train of thought I began in my last two posts (here and here) with further musings on that theme—in particular what survey researchers refer to as the “social desirability bias” problem (the reluctance of survey respondents to truthfully answer questions about sensitive behaviors like corruption). Last week’s post emphasized the seriousness of this concern and voiced some skepticism about whether one of the most common techniques for addressing it (so-called “indirect questioning,” in which respondents are asked not about their own behavior but about the behavior of people “like them” or “in their line of business”) actually works as well as is commonly assumed.

We professors, especially those of us who like to write blog posts, often get a bad rap for criticizing everything in sight but never offering any constructive solutions. The point is well-taken, and while I can’t promise to lay off the criticism, in today’s post I want to try to be at least a little bit constructive by calling attention to a promising alternative approach to mitigating the social desirability bias problem in corruption experience surveys: the unmatched count technique (UCT), sometimes alternatively called the “item count” or “list” method. This approach has been deployed occasionally by a few academic researchers working on corruption, but it hasn’t seemed to have been picked up by the major organizations that field large-scale corruption experience surveys, such as Transparency International’s Global Corruption Barometer (GCB), the World Bank’s Enterprise Surveys (WBES), or the various regional surveys (like AmericasBarometer or Afrobarometer). So it seemed worthwhile to try to draw more attention to the UCT. It’s by no means a perfect solution, and I’ll say a little bit more about costs and drawbacks near the end of the post. But the UCT is nonetheless worth serious consideration, both by other researchers designing their own surveys for individual research projects, and by more established organizations that regularly field surveys on corruption experience.

The way a UCT question works is roughly as follows:

• First, the survey population is randomly divided into two separate groups. Randomization is important because it ensures that, so long as the sample size is sufficiently large, the distribution of characteristics of the respondents in each group will be approximately the same.
• Next, each group receives a different version of the question related to the sensitive behavior. One group (the control group) gets a question along the lines of: “How many of the following things have you done?”, followed by a list of items which are not particularly sensitive, and which are relatively infrequent but not impossible. The other group (the treatment group) gets the same question, followed by a list that includes all the same items on the control group’s list, plus the sensitive item. The idea is that because respondents only need to say how many of the listed behaviors they’ve engaged in, but don’t need to say which ones, respondents will feel more comfortable answering honestly. Even on a survey that’s already anonymous, the UCT can create a kind of double-anonymity that encourages truthful responses, and may also make it less obvious to the respondent that the researcher is particularly interested in information about the sensitive behavior.
• Then, the researcher can then compare the average number of activities reported by the control group and the treatment group. Because of randomization, the treatment group and control group should engage in the non-sensitive behaviors at approximately equal rates. As a result, the difference in the average response from the control group and the treatment group provides a measure of the frequency of the sensitive behavior.

That all may sound a bit abstract. To give a better sense of how this technique can be used specifically in corruption experience surveys, I’ll take as an illustrative example the excellent paper by Ed Malesky, Dimitar Gueorguiev, and Nathan Jensen. That paper used data from the annual Vietnam Provincial Competitiveness Index survey, which in 2000-2012 employed a UCT to get more accurate data on bribe payments by businesses operating in Vietnam. The control group received a question that said:

Please take a look at the following list of common activities that firms engage in to expedite the steps needed to receive their investment license/registration certificate. How many of the activities did you engage in when fulfilling any of the business registration activities listed previously?

1. Followed procedures for business license on website

2. Hired a local consulting/law firm to obtain the license for the firm for you

3. Looked for a domestic partner who was already registered

The treatment group got exactly the same question, but a different list:

1. Followed procedures for business license on website

2. Hired a local consulting/law firm to obtain the license for the firm for you

3. Paid informal charge to expedite procedures

4. Looked for a domestic partner who was already registered

For the control group, the average number given by respondents was approximately 1.3. For the treatment group, it was approximately 1.5. From this, one can infer that approximately 20% of the businesspeople surveyed paid bribes (or “informal charges”) when investing in Vietnam (1.5-1.3=0.2), so long as we can assume that answers to the list question are honest (and that, due to randomization, the average number of non-sensitive behaviors engaged in by members of the treatment group was also approximately 1.3).

The UCT approach has been widely employed in a range of other research areas, but other than the Malesky et al. paper noted above, and a working paper by two of the same authors, there appears to be very little use of this technique in corruption experience surveys. This seems a shame, given the importance, for so many different research and policy questions, of gathering accurate information about actual corruption experience as opposed to perceptions, and the glaring inadequacies of the most widely-used methods of mitigating social desirability bias in corruption surveys, such as indirect questioning.

Still, it’s worth acknowledging – indeed, emphasizing – that the UCT is no panacea, and has a number of important limitations and drawbacks:

• First, it probably comes as no surprise to learn that there are vigorous debates (most of which are over my head mathematically) about the right way to conduct UCT experiments, the validity of the assumptions necessary to draw appropriate inferences, and so forth (see, for example here, here, here, and here). Because I don’t have the expertise to evaluate, let alone participate in, these debates, I can’t really advocate for the technique, but will instead suggest that it’s worth further exploration, even though I know that’s a bit of a cop-out.
• Second, fielding UCT surveys will often be more complicated and expensive, due to the need to field two different surveys (to the treatment and control groups, respectively). This is particularly relevant for those surveys that ask about a host of sensitive issues, and aren’t specifically or primarily about corruption. The organizations that field these surveys may not think it worthwhile to field a whole second version of the survey just for the sake of getting more accurate answers on the one question regarding corruption, but at the same time they might not think it’s worth the expense to field dozens of versions of the survey in order to use the UCT approach for all the sensitive questions that the survey covers.
• Third, if it is too obvious what the survey is about – if it’s clearly a corruption survey, for example – then it’s possible that the UCT won’t work as well, because respondents in the treatment group will see what the researchers really want to know about, and so social desirability bias might kick in again, even with the double-anonymity provided by the list question. And if the same survey asks multiple questions about corruption, it becomes both less feasible to use the UCT for all of them and less plausible that the UCT would effectively eliminate social desirability bias.
• Note that the second and third points create something of a dilemma: For surveys that include a question about corruption in the context of a much larger set of questions about a broader range of topics (such as the WBES), it may not seem worthwhile to the responsible organization to alter the survey design just to get employ the UCT for the corruption question. But for surveys that are specifically about corruption (such as the GCB), it may not be possible to use the UCT for all the corruption questions, and in any event the technique may become less effective when the subject of the survey is more obvious.

The concerns noted above are important, but in my view shouldn’t be deal-breakers. At the very least, the organizations responsible for fielding corruption surveys should take a good long look at the techniques they currently use to address social desirability bias and consider whether some other approach, like the UCT, might be appropriate.

The UCT is not the only alternative for addressing social desirability bias problems in a manner that’s more effective than simple anonymity coupled with indirect questioning. I hope to discuss another one in an upcoming post.