comment on thornhill (2019)

A few weeks ago, the journal Sociology of Race and Ethnicity published an article by Ted Thornhill called “We Want Black Students, Just Not You: How White Admissions Counselors Screen Black Prospective Students.” It’s a good article and I encourage you to read it. Basically, the author conducts an experiment where he sends college admissions counselors letters of inquiry. Some have indicators of anti-racist activism. The result is that letters with the anti-racist treatment get fewer responses. It’s a solid contribution to the growing field where people use audit studies to identify the effects of race on employment and educational opportunities. This post has three additional comments.

I. I think the word choice in the letter merits more scrutiny. On page 7 (of the online version), Thornhill lists the four letters sent. One has no mention of activism (narrative 1). The second mentions an “environmental awareness group” and a greenhouse planning committee (narrative 2). A third mentions jazz band, gospel choir and a student group promoting cultural understanding (narrative 3). The final one mentions the Black Student Organization and Anti-Racism student alliance  (narrative 4).

The logic of the experiment is that narrative 4 differs in that it reveals the letter writer’s race and participation in anti-racist clubs. But one has to be careful – narrative 4 has two students groups listed and they may have very differing cultural valences. On college campuses, Black Student Unions go back to the 1960s and they have often been at the center of some very contentious political events, like the 1968 Third World Strike at San Francisco State College. Thus, groups that use the label “Black Student Union,” or the very similar “Black Student Organization” may evoke reactions related to radicalism and even violent conflict. In contrast, the “anti-racist student alliance,” I think, has no similar history and the effect might be different. Thus you have to potentially differing effects.

To get a sense of the issue, consider if narrative 2 had instead had contained the sentence: “I belong to the Earth Liberation Front high school chapter and I started a chapter of Be Nice to Puppies.” ELF evokes strong negative feelings due to its radicalism and contentious past while the pro-puppy group does not. Having both in the same letter muddies the issue.

II. The other issue that merits comment is narrative 4. Specifically, the order of themes is simply different in narrative 4 than the other letters. Specifically, narratives 1-3 all end with a version of “would your college be a good fit for me?” Narrative 4 inserts that in sentence 2 after the salutation.

I found this to be odd since we know from decades of survey research that question order, phrasing and other features of survey questions can have a noticeable effect on survey responses. This may have been done to misdirect the respondents and avoid detection of the experiment. But it may be important. Why? Since each admissions officer was only sent 2 of the four narratives and there were 517 people in the sample, narrative 4 was sent only to 275 people (Table 5, page 12 in the online version, top 3 audits). I don’t think word ordering would reverse the effects, but it could reduce them and it’s not too crazy to think that it may affect the statistical significance of audit 3 (narrative 3 v 4/anti-racist v racially conscious).

III. The study focuses only on the responses of white admissions counselors. I found this to be puzzling. Given the research design is low cost (emails, basically), why not do the same for non-whites? Or admissions officers at HBCUs? Such a study would be intrinsically interesting for many reasons. For example, there is a debate about the role of HBCUs in anti-racist politics so it would be good to know if their staff follows the same scripts as staff at HPWIs. Also, in the section on research design (page 9) reports a 100% (!!) agreement between the author and an assistant in assessing the race of admissions counselors. How was that achieved? Getting 100% agreement on racial ascriptions in a larger sample is very challenging.

To be honest, I think the result reported in the paper is correct and I think this is a valuable contribution to the study of race and higher education. However, I think there is some reasonable criticism to be made of the experiment and I hope that future replicators will take those into consideration.


50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)
A theory book you can understand!!! Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)
The rise of Black Studies:  From Black Power to Black Studies 
Did Obama tank the antiwar movement? Party in the Street
Read Contexts Magazine– It’s Awesome!!!


Written by fabiorojas

September 20, 2018 at 4:21 am

Posted in uncategorized

3 Responses

Subscribe to comments with RSS.

  1. I was glad to see this paper, too. One issue I was wondering about is the reference to majors. The first letter mentions majoring in English and Math, the others don’t mention majors specifically. It could be an issue that counselors are more pumped to admit Black students to do math or biology/environment, where they are less well represented. Again, I doubt that’s the whole story but maybe one mechanism.


    Philip N. Cohen

    September 20, 2018 at 5:01 pm

  2. Thanks for pointing that out. The take home for me is that I believe the effect is real, but you have to be super careful about the experimental design. The discussion of college major is another place where I think the PI should have really kept the letter completely identical in all respects minus the treatment.



    September 20, 2018 at 6:22 pm

  3. I’m in the middle of fielding a few audits and was hoping to run one on graduate school admissions as well. I could see if their results hold up at the graduate level. My bigger concern with these studies is glossing over who is replying and how much discretion they have in how they respond.


    Michelangelo Landgrave

    September 21, 2018 at 3:08 am

Comments are closed.

%d bloggers like this: