orgtheory.net

my deep burning hatred of letters of recommendation

Econjeff mentions my long standing critique of letters of recommendation (LoRs). Here, I describe my personal experience with them and then I restate the massive empirical research showing that LoRs are worthless.

Personal experience: In graduate school, I had enormous difficulty extracting three letters from faculty. For example, during my first year, when I was unfunded, I asked an instructor, who was very well known in sociology, for a letter. He flat out refused and told me that he didn’t think I’d succeed in this profession. In the middle of graduate school, I applied for an external fellowship and was informed by the institution that my third letter was missing. Repeatedly, I was told, “I will do it.” Never happened. Even on the job market, I had to go with only two letters. A third professor (different than the first two cases) simply refused to do it. Luckily, a sympathetic professor in another program wrote my third letter so I could be employed. Then, oddly, that recalcitrant member submitted a letter after I had gotten my job.

At that point, I had assumed that I was some sort of defective graduate student. Maybe I was just making people upset so they refused to write letters. When I was on the job, I realized that lots and lots of faculty never submit letters. During job searches at Indiana, I saw lots of files with missing letters, perhaps a third were missing at least one letter. Some were missing all letters. It was clear to me that l was not alone. Lots of faculty simply failed to complete their task of evaluating students due to incompetence, malice, or cowardice.

Research: As I grew older, I slowly realized that there are researchers in psychology, education and management dedicated to studying employment practices. Surely, if we demanded all these letters and we tolerated all these poor LoR practices, then surely there must be research showing the system works.

Wrong. With a few exceptions, LoRs are poor instruments for measuring future performance. Details are here, but here’s the summary: As early as 1962, researchers realized LoRs don’t predict performance. Then, in 1993, Aamondt, Bryan and Whitcomb show that LoRs work – but only if they are written in specific ways. The more recent literature refines this – medical school letters don’t predict performance unless the writer mentions very specific things; letter writers aren’t even reliable – their evaluations are all over the place; and even in educational settings, letters seem to have a very small correlation with a *few* outcomes. Also, recent research suggests that LoRs seem to biased against women in that writers are less likely to use “standout language” for women.

The summary from one researcher in the field: “Put another way, if letters were a new psychological test they would not come close to meeting minimum professional criteria (i.e., Standards) for use in decision making (AERA, APA, & NCME, 1999).”

The bottom line is this: Letters are unreliable (they vary too much in their measurements). They draw attention to the wrong things (people judge the status of the letter writer). They rarely focus on the few items that do predict performance (like explicit comparison). They have low correlations with performance and they used codes that bias against women.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

October 6, 2015 at 12:01 am

13 Responses

Subscribe to comments with RSS.

  1. I couldn’t disagree more. Letters of recommendation from people from high tier universities making invidious distinctions among the handful of top students may well be worthless. Or not. You just let us decide what is bureaucratic propaganda and what is worthwhile information about job candidates. Letters of recommendation are THE MOST important thing that we here down in the trenches rely upon for vetting potential colleagues who might make tenure while teaching and researching under less than adequate situations. If you cannot speak to that, maybe you are not helping your students as well as you should. You know, most of them are not going to top research universities…..Yet, even at second and third and even fourth tier places, we need to know if the children are potty trained. And, you are supposed to be the diaper changer. Suck it up. Don’t ever say shit like this again.

    Like

    sherkat

    October 6, 2015 at 12:20 am

  2. Darren:Show me your data and I will be happy to change my tune.

    Like

    fabiorojas

    October 6, 2015 at 12:44 am

  3. I imagine the same holds true for such entries on LinkedIn. Was that part of the study?

    Sent from my mobile phone.

    Donald Frazier frazierdp@comcast.net 303 817 0201 mobile Mtn US time zone, GMT-7

    Like

    Donald Frazier

    October 6, 2015 at 1:05 am

  4. I would be willing to go along with this argument, but only if there were a reasonable alternative for students like mine. My students apply to graduate school from a college no one out there in R1 land has ever heard of. They have grades that are fantastic for our institution, but look lower than their competitors who apply from colleges with rampant grade inflation (yes, public comprehensive colleges have resisted grade inflation more than other types of institutions, and there is data to support this claim: http://www.gradeinflation.com/). They can’t afford GRE prep courses and never paid much attention to standardized tests growing up, so their GRE scores undersell their potential. The three letters of recommendation they can get from faculty who know them–and their outstanding intellectual and professional promise–is what we out here in the rest of the world can offer. So yes, they are biased, unreliable, and invalid, sure. But until you R1 people come up with a better way for my amazing-as-hell students to demonstrate how very qualified they are for a place at your table, I think I am voting for them.

    Like

    Mikaila

    October 6, 2015 at 2:04 am

  5. Mikalia: Here’s the deal – LoR’s are probably doing a dis-service to your students because readers tend to pick up on status of the letter writer and so forth. I’d humbly suggest that GPA and GRE would be better for your students than letters on the average.

    Like

    fabiorojas

    October 6, 2015 at 2:20 am

  6. Mikaila: letters of reference are more useful for students applying to grad school (or high school students applying to college) exactly in the circumstances you mention: letter writer from No-Name School knows how to write a letter to persuade Self-Important U that they are able to recognize talent and differentiate among their students. I have seen such letters and they are very important in opening doors for a student. BUT: (1) the undergrads at Big State U can’t get letters like that and (2) most of the letter-writers at mo-name schools don’t know how to write the letters that will actually help their students and (3) the students who got into the ivies are presumed to be brilliant no matter how tepid their letters are. That is, the letters are really important only for a small segment of applicants and even for them, the letter-writer’s ability to qualify themselves properly is at least as important as the student’s talent.

    For jobs (what Fabio is talking about) the situation is worse. The most important thing in an academic’s portfolio ought to be the work that they’ve done. A strong and good letter may help persuade a hiring committee that the person is brilliant despite not having anything done yet, but at least as often a strong letter that is at variance with the actual written record persuades a department to hire someone who burns out and fails to perform. And tepid letters are often used as an excuse not to even bother reading an applicant’s actual work. Strong letters from Third Teir U or Second Tier U will not help anyone get a job at First Tier U. The only path of upward mobility at this step is actual published work that impresses people. Sherkat is saying they would like the letter-writers from First- and Second-Tier U to tell them the truth about who can get things done and who can’t. But isn’t a record of actually getting things done in grad school also the best predictor of that?

    Liked by 1 person

    olderwoman

    October 6, 2015 at 2:21 am

  7. @Fabio: I suppose that means you remain in favor of a stratification system in which students from educationally-disadvantaged background will be entirely excluded from earning Ph.D.s, since the GRE/GPA metric is, as I outlined, not to their general advantage. I wish departments would post on their admissions page which perspective they adhere to, so that we could advise our undergraduates not to bother applying where they obviously are not wanted.

    @olderwoman: I agree about job recommendation letters. From my experience on faculty searches, it seems that the primary concern of people reading those letters is who they are from, and thus a one-sentence “I approve this applicant” message would go just as far and be just as useful (useless?). As for the fact that Big State people cannot get the recommendation letters that my students can–Armstrong and Hamilton, in “Paying for the Party,” clearly find that their disadvantaged participant pool is more successful in regional comprehensive colleges, and your point is another piece of evidence to support that perspective.

    Like

    Mikaila

    October 6, 2015 at 12:44 pm

  8. Mikaila: Agreeing that a smaller directional campus can serving disadvantaged students better than Big State U, but I’ll repeat: the letter-writer for a grad program has to know how to qualify themselves, it isn’t just a matter of the student’s talent. For every letter writer like you who knows how to do such a letter, there are 10 who don’t and whose letter don’t help. The successful letter writer has to know how to overcome the prejudice and speak directly to the program by explaining their own credentials for knowing what they are talking about as well as explaining how this student stands out. Very few letter writers from less prestigious schools do this, in fact they are less likely to know (or at least to exhibit) the “code” for writing letters than letter writers at Big State U.

    Like

    olderwoman

    October 6, 2015 at 1:47 pm

  9. So,… can we trade burning hatred for letters (inanimate physical objects) for training/encouraging/vilifying lazy, incompetent, or disinterested letter-writers? The thread implies that a scorched earth policy for recommendation letters for the entire profession solves the wrong problem.

    Liked by 1 person

    Randy

    October 7, 2015 at 2:51 pm

  10. This thread contains many allusions to the secret code for writing effective letters, which we plebs outside of R1-land are apparently ignorant of. Does anyone want to share what the code is?

    Liked by 1 person

    Lucy

    October 8, 2015 at 6:14 pm

  11. Lucy: I teach in an R1 and I am at a loss because nobody explained the code to me. I do know a few things: “hard worker” = not smart and compensates with effort (usually directed at women), “Feel free to call me” = this person is nuts but I can’t say it in print. Sadly, I used both all the time. There are others… I wish there was a handbook.

    Like

    fabiorojas

    October 8, 2015 at 7:49 pm

  12. Re “code” there’s a lot. I was specifically referring to the need to overcome the prejudice against “no name” schools by the letter writer knowing how and going to the trouble to self-qualify: people do things like name particular alumni of their school who earned PhDs from top programs and are now professors at major universities, or talk about their own experiences in a top graduate program or working at a top school and how they know the difference between ordinary students and those who have what it takes to really excel, or acknowledge that the reader may not have heard of the school but giving information about what a great place it is.

    The other parts of the code I think vary by discipline. But the general assumption I work with is that letter writers try to say everything positive they can about a student, and if you don’t say something it is not true. A letter that says someone was a good student in your class and asked questions is a pretty tepid letter. If that is your intention–it is the best you can say without lying–that is good. But if you thought that was a strong endorsement of the student, it isn’t.

    Also if the student has mediocre test scores or grades, it is more helpful if the letter-writer knows that and speaks to the issue.

    Like

    olderwoman

    October 9, 2015 at 9:00 pm

  13. I may be out here in little-people land, but I have been told by grad faculty that I write excellent letters, so I will put in my $0.02:

    I think another thing that matters is how you show that you know the student has what it takes to succeed in the particular kind of program you are writing about. For a PhD program, you cannot say that so-and-so is hardworking, smart, cooperative, etc.–you need to show that this is a student who has developed the capacity to succeed stunningly at independently-motivated research and that he or she understand what PhD programs are and do and how the student will be likely to succeed not only in coursework but in dissertation-writing and in a post-PhD career. Specific examples are good. For students who deserve it, I will explicitly rank them in comparison to the body of students I have taught over my career and I will mention more elite schools I have taught at (as olderwoman suggests). I like to make sure my letters echo the student’s self-presentation in the personal statement so that we are telling a consistent and compelling story about who s/he is and what s/he is driven to accomplish, and I provide enough detail to make very clear how well I know the student’s capabilities and how closely I have worked with him/her over time. As Chambliss and Takacs say, know your students, and show that you know them.

    This code would be quite different for another kind of graduate program. For example, for social work students I emphasize their understanding of diversity, their caring and compassion, their ability to work well with others different from themselves, etc. For law students it is communication skills, leadership experiences, and understanding of/commitment to a career in law. But for all groups, as olderwoman said, you need to contextualize anything that is not so great–did they have a bad semester because they were taking 5 classes while working 40 hours a week on overnight shifts to pay tuition, and they can’t even dream of how amazing it would be to actually get a measly $17k stipend to live on? Etc.

    For those writing reccs for job applicants: please do your students the favor of not writing “they dream of a research university job” in letters that will be sent to teaching colleges. I mean, we love it, because it gives us a good excuse to delete one from the pile. But it is a terrible disservice to the applicant. Furthermore, if they are applying for teaching colleges, it would really help if at least one of you has any clue about their teaching.

    Like

    Mikaila

    October 9, 2015 at 9:40 pm


Comments are closed.

%d bloggers like this: