letter of recommendation theory

There’s a pretty wide spread feeling that letters of recommendation are very inefficient. Scatter has a very good discussion here, check out olderwoman’s comment. Leiter Report, a professional philosophy blog, has a very nice discussion of how people may not know the “code” when writing the letters, and thus unintentionally damage their students. On this blog, I’ve proposed eliminating letters of recommendation in many cases.

So I got to thinking, to have any decent opinion on letters of recommendation, you need at least a theory of how they operate. Here is what you need to know:

1. “Codes”: does the letter writer know the formal and informal standards for writing letters?
2. Incentives I: Does the letter writer have any incentive to write a truthful letter? Is there any reason to belive that letter inflation can be contained?
3. Incentives II: Does the letter writer have any incentive to write a proper letter if they know the “codes?” Even if they are honest, why would people take the time to write letters?
4. Validity: Do letters actually predict performance?
5. Value added: What does the letter add that is not obvious from observed behavior?
6. Costs: What is the abosolute and relative cost of letters in terms of processing and managing them?

Here’s what I think are the answers:

1. Obviously, there seems to be a huge variance in letter writing codes. In my own case, I always thought that “hard working” was a good and rare attribute. Omar points out that the wider academe sees this as a signal of low quality (i.e., they compensate for lack of talent with extra effort, which is somehow a bad thing). It’s clear that simply getting the form and filling it out is insufficient.
2. The folk wisdom is that most fields suffer from letter inflation. I am not sure what incentives people have to write accurate letters. People say “I don’t trust letters from so and so.” But who cares? Will that change so and so’s behavior? Not a chance. It probably punishes the applicant.
3. There is little evidence that people will take the time to write lengthy and carefully crafted letters. Most seem generic. I’ve seen many files that are incomplete. The faculty simply fail to write the letters.
4. There’s concrete evidence here. Letters are a poor predictor of performance in the settings where they have been studied. Here’s a clip from an article in the journal Public Personnel Management : “Even though references are commonly used to screen and select employees, they have not been successful in predicting future employee success (Muchinsky, 1979). In fact, the average validity coefficient for references is only .13 (Browning, 1968; Mosel & Goheen, 1959). This low validity is due mostly to four main problems found with references and letters of recommendation: Leniency, knowledge of the applicant, low reliability, and extraneous factors involved in the writing and reading of letters of recommendation.” There’s a number of studies in psychology and medicine about letters and future perfomance. The studies show letters aren’t very good. They’re glorified junk.
5. In most cases, the value added is small. For example, don’t you know enought about a student from the 30+ college courses reported in the transcript? And their standardized tests, often taken multiple times? And the writing sample? That’s a huge amount of information and better than what almost all letters contain.
6. Costs: It takes time to write letters, ship them, recieve them, sort them, and read them. Lot’s of time. There’s storage and postage. There’s also the time it take students to hound faculty to get the letter done and even then, too many job and grad school applicants don’t have complete files that are often discarded because they could only get 2 of 3 letters. There’s a significant cost.

That’s it in a nutshell: People don’t know what letters are about, they have incentives to write lousy letters, letters don’t predict performance, and there’s a non-trivial cost. Why on earth do we keep doing it?

If you want my “subtle” opinion, mandatory letters should be abolished in most educational settings. Students should be encouraged to submit letters only if they feel that there’s something that’s not clear from the transcript (e.g., a semester of bad grades due to death in the family). Letters might also be required in cases where the committee reading papers truly needs external evaluation. For example, junior job canidadates often have thin publication records and non-specialists might find it hard to understand what’s good in a dissertation. But in most cases, we have a pretty good sense of what’s happening with a student.