orgtheory.net

letter of recommendation theory

There’s a pretty wide spread feeling that letters of recommendation are very inefficient. Scatter has a very good discussion here, check out olderwoman’s comment. Leiter Report, a professional philosophy blog, has a very nice discussion of how people may not know the “code” when writing the letters, and thus unintentionally damage their students. On this blog, I’ve proposed eliminating letters of recommendation in many cases.

So I got to thinking, to have any decent opinion on letters of recommendation, you need at least a theory of how they operate. Here is what you need to know:

  1. “Codes”: does the letter writer know the formal and informal standards for writing letters?
  2. Incentives I: Does the letter writer have any incentive to write a truthful letter? Is there any reason to belive that letter inflation can be contained?
  3. Incentives II: Does the letter writer have any incentive to write a proper letter if they know the “codes?” Even if they are honest, why would people take the time to write letters?
  4. Validity: Do letters actually predict performance?
  5. Value added: What does the letter add that is not obvious from observed behavior?
  6. Costs: What is the abosolute and relative cost of letters in terms of processing and managing them?

Here’s what I think are the answers:

  1. Obviously, there seems to be a huge variance in letter writing codes. In my own case, I always thought that “hard working” was a good and rare attribute. Omar points out that the wider academe sees this as a signal of low quality (i.e., they compensate for lack of talent with extra effort, which is somehow a bad thing). It’s clear that simply getting the form and filling it out is insufficient.
  2. The folk wisdom is that most fields suffer from letter inflation. I am not sure what incentives people have to write accurate letters. People say “I don’t trust letters from so and so.” But who cares? Will that change so and so’s behavior? Not a chance. It probably punishes the applicant.
  3. There is little evidence that people will take the time to write lengthy and carefully crafted letters. Most seem generic. I’ve seen many files that are incomplete. The faculty simply fail to write the letters.
  4. There’s concrete evidence here. Letters are a poor predictor of performance in the settings where they have been studied. Here’s a clip from an article in the journal Public Personnel Management : “Even though references are commonly used to screen and select employees, they have not been successful in predicting future employee success (Muchinsky, 1979). In fact, the average validity coefficient for references is only .13 (Browning, 1968; Mosel & Goheen, 1959). This low validity is due mostly to four main problems found with references and letters of recommendation: Leniency, knowledge of the applicant, low reliability, and extraneous factors involved in the writing and reading of letters of recommendation.” There’s a number of studies in psychology and medicine about letters and future perfomance. The studies show letters aren’t very good. They’re glorified junk.
  5. In most cases, the value added is small. For example, don’t you know enought about a student from the 30+ college courses reported in the transcript? And their standardized tests, often taken multiple times? And the writing sample? That’s a huge amount of information and better than what almost all letters contain.
  6. Costs: It takes time to write letters, ship them, recieve them, sort them, and read them. Lot’s of time. There’s storage and postage. There’s also the time it take students to hound faculty to get the letter done and even then, too many job and grad school applicants don’t have complete files that are often discarded because they could only get 2 of 3 letters. There’s a significant cost.

That’s it in a nutshell: People don’t know what letters are about, they have incentives to write lousy letters, letters don’t predict performance, and there’s a non-trivial cost. Why on earth do we keep doing it?

If you want my “subtle” opinion, mandatory letters should be abolished in most educational settings. Students should be encouraged to submit letters only if they feel that there’s something that’s not clear from the transcript (e.g., a semester of bad grades due to death in the family). Letters might also be required in cases where the committee reading papers truly needs external evaluation. For example, junior job canidadates often have thin publication records and non-specialists might find it hard to understand what’s good in a dissertation. But in most cases, we have a pretty good sense of what’s happening with a student.

Advertisements

Written by fabiorojas

February 11, 2010 at 4:34 am

Posted in academia, fabio

31 Responses

Subscribe to comments with RSS.

  1. As a current student that is required to apply annually for a scholarship I have no chance of winning (60 total provincial scholarships for all disciplines including natural sciences), I have to hound professors for letters every September.

    What I would like to add to your list is something you may not have encountered. I feel embarrassment asking for letters of recommendation. Every time, no matter the person. I have never been denied by a professor but I try desperately never to ask the same person twice. I am not sure if I am normal or abnormal as most grad students are pretty cocksure.

    Any ideas?

    Like

    R. Pointer

    February 11, 2010 at 5:09 am

  2. R. Pointer: I was certainly embarrassed. I was even turned down by a professor in soc grad school who told me flat out that he didn’t think I was very good. That did *not* help my confidence. But once I got a few years into the program, the embarrassment went away because my MA and PhD thesis advisers were extremely good about writing letters and they made me feel better about the experience.

    Your comment only makes the case stronger. You should definitely toughen up a bit and ask for stuff in grad school, but this never ending begging for letters is degrading. I was so happy once I finished grad school and I no longer needed letters for every little thing.

    Like

    fabiorojas

    February 11, 2010 at 5:21 am

  3. Any official ASA/APSA/else publication suggesting that teachers should be actually taught how to write these letters?

    Like

    Fr.

    February 11, 2010 at 7:12 am

  4. When asking for letters of recommendation, I have experienced it several times that the professor or supervisor actually asked *me* to write it (one time I was told to write “a draft”, but in the final version, only little was changed). I don’t know how wide-spread this practice is, but to me it makes the whole game even more pointless.

    Like

    Johann

    February 11, 2010 at 9:13 am

  5. Fabio, I wonder if you missed a signaling aspect, although I don’t know how important it is. Getting three letters of recommendation, at least as an undergrad, requires some work in terms of creating relationships with faculty. For undergrads at liberal arts schools, this seems to be a relatively easy process. For undergrads at a big public school (e.g. Michigan), it can be easy to get through 4 years without having made a single real relationship with a faculty member. However, the skill of being able to sell yourself to faculty, or more generally interact with them, is a not inconsequential one. So I wonder if the LoR barrier does serve some – perhaps unintended – effect of keeping out potential applicants or helping eliminate applicants lacking the confidence or skills to build relationships with faculty.

    I’ve never been on an admissions or search committee though, so I have no idea if letters are seen in such a light (at least, consciously). I do know friends who have struggled to find LoR because they struggled to build relationships with faculty. For law school, this did not seem like a useful barrier (as they did not need to build such relationships to successfully finish), but for a PhD program I think it might have been a good sign that these folks were not cut out for one.

    Like

    Dan Hirschman

    February 11, 2010 at 12:39 pm

  6. Often the letters are simple endorsements. E.g., “and he has a letter from Wallerstein.” If that is the case, maybe simple endorsements would work as well in some cases, and also signal that the student has successfully managed to have at least one communication exchange with the professor.

    Like

    Philip Cohen

    February 11, 2010 at 1:42 pm

  7. R. Pointer’s comment is a direct rebuttal to the premise of Dan Hirschman that in a nutshell demonstrates one of the major problems with the letter system. R. Pointer finds asking for letters so embarrassing that s/he never repeats the request. What you (R. Pointer) do not know is that repeating is better, because most of the work is in writing the letter the first time, and asking someone who has already written once to update the letter and send it to another place is a relatively minor request. What you (Dan Hirschman) are eliding is the extent to which knowing how the system works, both for students like R. Pointer about “getting to know faculty” and for many faculty letter-writers regarding the “codes” etc., involves a high level of cultural capital and is essentially elitist in its foundations.

    Like

    olderwoman

    February 11, 2010 at 2:00 pm

  8. @Olderwoman – I basically agree with your point. But “Knowing the System” or being able to learn it might be at least a plausible indicator of whether or not someone will succeed in grad school – bracketing for a moment Fabio’s evidence that letters don’t do that. That “Knowing the System” is also an indicator that will likely reinforce traditional sorts of discrimination and disparity is one good reason for not taking it into account (to the extent possible). Most of what I was trying to get at was that letters may be doing something, even if it’s not the something we want, or something we realize they are doing (as opposed to Fabio’s subtext of letters doing nothing).

    My last line comparing law school to PhD programs was just the beginning of a thought about whether or not even that trait (interacting with faculty) was a useful data point (if one that reinforces elite advantages) for all LoR requiring endeavors. For PhD programs, it might possibly be. But for law school, I don’t see it adding even that.

    Like

    Dan Hirschman

    February 11, 2010 at 2:15 pm

  9. This probably makes me sound terrible, but I’m in my first year as an assistant prof and I’m not sure I know what the “code” is for letters of recommendation in sociology! I’ve only been asked to write one letter so far, and it was for a small grant, not a job, but still….anyone willing to clue me in? I am still traumatized from years of having to constantly ask people for letters. My feeling is that at least in the initial stages, the process should work more like normal job applications, where the candidate provides the contact information for several references who can be called/emailed if more information is desired.

    Like

    Bedhaya

    February 11, 2010 at 3:01 pm

  10. Bedhaya: My brief comments are with respect to elite sociology departments. There are different codes in different disciplines, which is why it gets hard and is so much a matter of cultural capital. For starters, in academic sociology the letters are always positive and are read for what they don’t say. You say as many specific factual things about the person as you can say, and if the letter ends up short and vague, it is an indication that there isn’t that much to say. If “made good comments in class” is the best you can do, it isn’t much. The letters are also read for how much time/effort the writer is willing to invest in the person. Then there is the smartness axis, which is why “hard working” is treated as a code for not so smart, because smart people are typically elitists who believe that a sign of being intelligent is that you don’t have to work so hard. If you are at a non-prestigious place you have to qualify yourself by saying why you know what the standards are at prestigious places, otherwise you are discounted.

    Like

    olderwoman

    February 11, 2010 at 3:12 pm

  11. Interesting question to me is what can an individual letter writer do to be more effective on behalf of the applicant as well as more helpful to the evaluating institution.

    My thoughts (based more on experience in business):

    (1) If you cannot honestly recommend the student, decline to write the letter. We’re human, so this is hard. And it’s hard to do so without being discouraging. But I think it’s the right thing to do.

    Two easy let-downs: (a) You can tell the person, I can’t recommend you now, but if you did XYZ in the next year, I could recommend you then (a faculty member mentioned to me they had done this), or (b) You can tell the person, I can’t recommend you for XYZ (better school) but I can recommend you to ABC (worse school).

    If you are in the grey area where your sincere recommendation would be positive on balance, you could tell them what you are willing to say and give them the option of whether to have you write a letter.

    (2) Have the applicant give you bullet points of what they would like you to convey and specific experiences you have had with the applicant that illustrate those qualities.

    This puts the hard work on the applicant, discourages them from being unrealistic about what you can say about them, and makes the letter far more effective for the applicant and helpful to the reader.

    (3) Perhaps come up with a preface that you include in all letters that explains the context of how you write letters. E.g. in the past 3 years, 14 students have approached me to write letters, and I have agreed to recommend only 6 of them. Maybe people who have read letters can comment on what it would be helpful to include in that preface and what might make such as preface credible (from the comments on scatter this appears to be a repeat game in many cases, so it seems credibility could quickly be established or not).

    Like

    ThoughtMarkS

    February 11, 2010 at 3:22 pm

  12. I’ve had to decline in a few cases, because I simply did not know the student well enough. It felt awful too, because I had to wonder why I as an instructor from a general education course with 55 students should somehow know the student better than other professors. What was the quality of interaction with the student’s other professors? Profs and students won’t know each other better until upper-level courses, but that doesn’t help the scholarship-seeking student or the student who wants to transfer elsewhere.

    Like

    Mark Stoneman

    February 11, 2010 at 7:35 pm

  13. One other thing, I know nothing about “codes,” which could be a problem, but I only write recommendations when I can be highly specific about one or more qualities with concrete examples. If I don’t know enough to do that, I don’t write a recommendation.

    Like

    Mark Stoneman

    February 11, 2010 at 7:36 pm

  14. It is my understanding that outside of academe it is not considered good practice to use letters of reference during the appointment process.

    References should be used only after a decision has been made. The purpose of references is to confirm what has been discovered by other means. Check facts. Check your impressions of a candidate with someone who knows them well.

    For most hiring decisions in academe this would be an easy switch to make.

    And for fellowships, scholarships, etc, shouldn’t the other evidence weigh more heavily anyway?

    Like

    JoVE

    February 11, 2010 at 7:39 pm

  15. @Mark: You have to read a few hundred – you’ll get the idea. Search and admissions committees are great experience. Oh the humanity.

    Like

    Philip Cohen

    February 12, 2010 at 3:20 am

  16. ThoughtMark’s suggestions are good, especially for undergrads you don’t know well. But declining to write unless you are strongly positive is sometimes just mean. You should certainly tell a student if there isn’t a lot you can say. Like “Um, you got a C in my class, I’m not going to be able to say you were really good in it. Are you sure you can’t do better?” But sometimes the answer is: “You are the only prof I’ve had who is still at this university. I need a letter, I don’t really care what it says.” If you teach at a big university where most classes are taught by stringers who disappear, you should expect to have to write letters for people you don’t know very well.

    Like

    olderwoman

    February 12, 2010 at 3:52 am

  17. I graduated summa cum laude in criminology in 2008 from Eastern Michigan University. I applied to MSU, U of M, and George Mason for graduate school. I withdrew all three when none of the six professors I asked for letters would write one.

    I earned the grade point because I am good at this. Being a political conservative did not win me any friends.

    Already enrolled at EMU, I returned to complete a master’s in social science. I graduate in April. When looking for a doctoral program, I intend to get around the problem of the letters by having publications instead.

    Like

    Michael E. Marotta

    February 16, 2010 at 12:08 pm

  18. I’ve never been on an admissions or search committee though, so I have no idea if letters are seen in such a light . I know friends who have struggled to find LoR because they struggled to build relationships with faculty. For law school, this did not seem like a useful barrier (as they did not need to build such relationships to successfully finish), but for a PhD program I think it might have been a good sign that these folks were not cut out for one.

    Like

    Alan

    April 13, 2010 at 1:30 pm

  19. […] of recommendation: As I have argued many times on this blog, most research, with a few exceptions, shows that letters of reference are poor predictors of future…. I quickly scan letters for red flags to weed applicants who are clearly not suited for graduate […]

    Like

  20. […]  Here are a few links.  First, know that there are differing codes (also some discussion at orgtheory on this).  And, biases (e.g., linked to the attractiveness of the student) may play a role in receiving a […]

    Like

  21. I think students who are not receiving grant money (free money) for college shouldn’t have to be required to submit letters of recommendation because that is like saying I have to get the okay or get a favorable reference from someone to spend my own money to attend your school. Really? How backwards is that? – F.Y.I. from a student who recently graduated with a masters degree with a 3.9 G.P.A.

    Like

    Angela W.

    July 6, 2012 at 6:16 pm

  22. Professor Rojas,

    A majority of your sentiments here are spot on. And I sympathize with your frustration with reading generic tripe in letters of recommendation. But I fail to see that citing public sector employee employment and medical and psychological research support your conclusion about sociology letters. There extraordinary institutional differences between the structure of public and private sector employment, and that of the academy, and no way to control for confounding effects. First on that list would seem to be the weight in-person interviews take in their “admissions” decisions.

    Further, medical and psychological researchers most often work with teams of several graduate students, providing undergraduate research assistants with menial opportunities to clean data and test tubes. Graduate students themselves often vet and train these undergraduates, and a letter from the graduate student supervisor herself would probably be more substantive.

    In contrast professors of social science, save economics possibly, can develop profound relationships with their undergraduate research assistants, involving them in the theory and empirics of the project, in addition to advising and guiding independent research and thinking.

    Of course the picture I pain is probably rare, and reading through sixty letters which list precisely the same adjectives about 60 different people do mean very little. The solution to this situation is to point to the texture of the problem — not to summarily dismiss the entire practice.

    Cheers.

    Like

    Graham Peterson

    December 4, 2012 at 11:03 pm

  23. I disgree Graham. If studies in multiple areas show that the practice is bunk, then the burden of proof is on people to show that letters matter in my area. Until then, I feel free to dismiss letters.

    Like

    fabiorojas

    December 5, 2012 at 12:20 am

  24. Adding up a multiplicity of incomparable fields does not make the case stronger. You made the positive claim — the burden is yours.

    Like

    Graham Peterson

    December 5, 2012 at 12:33 am

  25. No, the burden is on people to prove that academic letters are an exception to the trend. In social science, we can’t prove that something works in every situation. So we conduct experiments and observational studies. Then we come to a rough estimate of a policy that works. So yes, in theory there may be exceptions. But in practice we don’t abandon decades of research. Instead, we ask that people who look for exceptions to take on the burden of proving the exception.

    When it comes to letters of reference, we are now at the stage where we have decades of research which shows a general trend. We don’t toss it all. Instead, we say yes there may be an exception – please show me the data. Until then I’ll side with prior research.

    Like

    fabiorojas

    December 5, 2012 at 12:45 am

  26. I’ll try to avoid honing the argument any sharper than it already is. I’m aware of the general outline of methodology in social science. You didn’t cite decades of research — you cited three papers. But the topic of the debate has by now moved, and I’m not here to attack your lack of citations for a blogroll (particularly obnoxious when comment flame wars turn into two people arguing over who’s more scholarly about their internetting).

    I made a substantive point about the difference of the nature of the relationship between medical and psychological researchers and undergraduate research assistants, which it would be nice for you to address.

    Like

    Graham Peterson

    December 5, 2012 at 1:13 am

  27. Check the citations to those articles. Or even better, use google scholar and show that I am wrong. I am often wrong.

    You ask about the comparability of undergrads applying to grad school and medical school graduates. You are right. They are different. By itself, it doesn’t imply much. But when you have studies of many different types of groups you start capturing variance, though imperfectly. If letters don’t predict performance for medical professionals and they don’t for the types of white collar office jobs in the occupational psychology literature, then that decreases my confidence in the policy in my own domain.

    Also, I add my own experience. I have now seen thousands of letters for graduate school and faculty applicants. Too many say positive things about students with modest records. The information in these letters can’t be high.

    I may be wrong. I have changed my mind on many policy issues. Please show me some data. The best pro-letter evidence is the study by Levity and colleagues about Econ grad school applicants. But that is one study. The issue deserves more study. And if I am wrong, it will justify the effort I expend letters every year, which quite a bit.

    Like

    fabiorojas

    December 5, 2012 at 1:34 am

  28. Well, now we’re on home turf for me, considering I’ve equivocated between sociology and economics applications for the last year and unfortunately know an insane (and unnecessary) amount about economics applications. The GRE quant score is indeed an *excellent* predictor of quant success. In fact, the GRE was redone precisely because there wasn’t enough variance at the top of the quant scale for engineering, economics, and physics programs — everyone had 800s. They made the material harder, and put the old 99th percentile at a conversion of 166/170. Lots of headroom to sort the curve now.

    My data is anecdotal, but large-sample nonetheless. http://www.urch.com/forums/phd-economics/ Thousands of applicant profiles listing maths experience and quant GRE scores. People who have crushed tons of advanced mathematics need only marginal practice to ace the GRE quant. People who have say a mathematics minor will need much more practice. And there is a degree of immersion necessary to really ascertain deductive reasoning — which the quant GRE accurately tests (albeit using “tricky highschool algebra and geometry”) The score thus predicts one’s ability to quickly go from introductory courses to writing one’s own proofs professionally, extraordinarily accurately. T10 trashes scores below the 99th percentile unless . . . you’ve got a good name on one of your recs (maybe you had a bad day, etc.).

    So the predictions among economics applicants should really only suggest to you, I think, that what letter writers say about a students quant abilities doesn’t matter as much as the quant GRE and math grades.

    Further, SOPs do not get read in economics applications except really outside the T20, and maybe even T40. Nobody cares about your qualitative undergraduate thesis, and it’s in fact likely to end up on the wall at the Christmas party if you send it in. It’s a profoundly different admittance procedure than sociology or anywhere else. In fact — LORs mean a ton in economics, because these people are the only ones who can credibly say anything about your ability to think economically, as there is no writing sample and again the SOP is just a is-he-crazy-test. Outside LORs the only thing that distinguishes you are your math grades.

    Professional letters are an incredibly noisy signal when hiring therapists. Somewhat sensibly, I think, I would want to sit down and talk and read someone who was going to work in my practice rather than believe the letter someone fired off about them. Knowing they won’t matter much, why would a LOR put much into then anyway?

    And the point still stands about medical research — you’re applying to a lab — not to a graduate program. Brass tax. You’re applying as an entry level research assistant, who has done next to no substantive work in the past.

    Sociology, poly sci, and anthropology applicants have on the other hand potentially had intimate relationships with their advisers and the projects thereof.

    But like I said — that does not detract from the fact that there are just lots of mediocre candidates who can find a teacher to regurgitate some vapid nonsense using generic adjectives. Which I’m sure you’ve had to suffer through and are right I think in discounting, not dismissing.

    Like

    Graham Peterson

    December 5, 2012 at 1:51 am

  29. You raise a good point about economics. It is a field that requires a lot of math but applicants varyin how much math they know. Also, the main measure of that skill has a compressed high end, so it is not useful. Then, letters may matter as a social signal. Levitt & co’s study found that prestige of the recommender was a surprisingly good predictor of success.

    So I am willing to admit that exception because we have evidence. I do have a hypothesis that it may work in other fields. Perhaps high status profs in other fields are only willing to write letters for students who have a knack for that field. Or, at th least it shows academic street smarts which correlates with success.

    If you believe that story, then the policy should be to not read letters most of the time and go on prestige of the letter writer.

    Overall, the exception of one field does not make me want to invest a lot in letters in other fields. I want to see more work on this.

    Finally, yes medical residents are different. But these letter studies look at residents not lab workers. It’s not completely unrelated to other types of work. You are making a very strong assumption about occupations if you don’t adjust your beliefs at least a little in response to prior work.

    Like

    fabiorojas

    December 5, 2012 at 2:20 am

  30. Oh no — I think you’re absolutely on the money to point out that most letters in sociology and other social sciences are vapid tripe, overwrought from minimal coursework and independent-research advising that can vary from “she read a couple papers and submitted able expository summaries,” to, “she made fundamental contributions even to my own thinking because of sheer academic horsemeat.”

    And it’s clear that professional recommendations in many settings are a bad device. Deirdre McCloskey wrote an article in her series of comments in Eastern Economic Journal about how useless they are in faculty appointments.

    I just think extrapolating from the fields you have provides a solid enough case to dismiss them altogether, that’s all. Otherwise I think you’re on to something and are rightly critical of the process.

    Another note from economics — most economics committees want writers to rank a student and compare their performance directly to graduate students whom they’ve advised — giving hard estimates of promise.

    I think the point generally is that if a letter is going to mean anything anywhere it’s just got to be overwhelmingly effusive, detailed in experience, and convey enough honesty about the candidates flaws (which all have) to be credible. Assuming candidates are normally distributed, and letter writers too, very few letters will meet that criteria. That’s no reason to dismiss the entire institution. Just skim bad letters looking for the eye-poppers.

    Like

    Graham Peterson

    December 5, 2012 at 2:31 am

  31. […] them about whether I am actually helping my students or not. Upon obtaining my job I was not given the code, so I wouldn’t mind abolishing them. As long as they are around (and students can help me out […]

    Like


Comments are closed.

%d bloggers like this: