Moving Beyond Measurement: Adapting Audit Studies to Test Bias-Reducing Interventions

Publication Year
2017

Type

Journal Article
Abstract

This paper discusses how audit studies can be adapted to test the effectiveness of interventions aimed at reducing discrimination. We conducted an adapted audit experiment to test whether making officials aware of bias could reduce levels of racial bias. While the limitations of our design make it difficult to assess where information alone can reduce bias, our study makes two important contributions. First, we replicate prior studies by showing that white, local elected officials are less responsive to black constituents. That local officials exhibit biased behavior is particularly worrisome, as local government is often the level that most directly affects citizens' daily lives. Second, we provide several suggestions for future audit studies that draw from the strengths and weaknesses of our own design. We hope that they will help improve future work on identifying and reducing discrimination. Copyright © The Experimental Research Section of the American Political Science Association 2017.

Journal
Journal of Experimental Political Science
Volume
4
Pages
57–67
Type of Article
Journal Article
Full text

The following is an excerpt of the intervention methodology. For more information, please see the full text of the article on the publisher's website or through your institution's library.

We use an audit study to test whether information can reduce racial bias. In our audit study, we sent elected, municipal officials (i.e., mayors and city councilors) in the U. S. requests for assistance from putative constituents, randomizing whether the request came from someone with a distinctively black name or a distinctively white name. [...] Because we could not identify the race of all city officials, we restricted our sample to the types of cities where, based on the racial make up of the city, the vast majority of elected officials are likely to be non-Latino, whites. [...] We used the 2011 International City/County Management Association (ICMA) city survey [...] to determine which cities to include in our sample. Based on these data, we restricted our sample to cities where 75% or more of the population is white and less than 15% of the population is Latino. [...] We also restricted our sample to officials from cities with at least 50 black individuals and a total population of 3,000 or more. [...] Our final sample included 11,801 city officials from 2,160 cities from across the United States. [...] To measure levels of differential treatment by race, we emailed all of the 11,801 officials in the sample, randomizing whether the email was sent from either a putatively black or a putatively white individual. [...] In total, we used 76 different aliases: 35 black aliases and 41 white aliases [...] The mean city in our sample has 23,949 residents, of whom 4.23% are black. Cities range in size from 3,011 to 583,776 people, and the black population within cities ranges from 50 to 42,188. [...] We carried out the study so that no two officials from the same city received a request from the same putative constituent. In particular, we first assigned officials to receive either the black or white constituent email and then block randomized, by city, aliases from the assigned racial treatment. Similarly, the request found in each email was randomly drawn from a list of simple requests adapted from the “frequently asked questions” sections of various city websites. [...] We also randomized when we sent the emails so that they went out over a 5-day period. We spread the emails out to help ensure that no city in the study received all of the emails on the same day. About two weeks earlier, we sent the elected officials assigned to the treatment group (n = 4,004) a pair of emails with information about prior research on racial bias exhibited by public officials. [...] The email states that research has shown: [Text Stimuli A...] [...] we used a multilevel design where we first randomly assigned the cities into two groups: in one-third of the cities no officials received information; half of the officials in the remaining cities were randomly chosen to receive the information.

Type of Prejudice/Bias
Country
Method
Setting