are you a top or a bottom?
I just encountered a really fantastic little article by Morris Holbrook, entitled “A Note on Sadomasochism in the Review Process: I Hate When That Happens.” Not only is the piece really funny, but is chock-full of really good advice. Even though the article was written in 1986 and it concerns the field of marketing in terms of the quality of the recommendations on both the writing and reviewing sides it could have been written yesterday and applies to all of the social sciences. One caveat: while the advice is good, we should be so lucky to posses the psychological maturity to actually be able follow it, especially on the reviewing side.
The comedy of Willie and Frankie was in the senselessness of their self-abuse: He wacks a meat thermometer into his ear with a ball-pean hammer, and treats it like it was somehow unavoidable.
But the difference with publishing is that there are actual forces that compel us toward the masocistic behaviours described here. Article length gets used as a heuristic for importance or depth or rigour, encouraging researchers to be verbose. Careful reviewing matters little to hiring or promotion-and-tenure committees, meaning that reviewers have an incentive to do the minimum needed to keep “reviewer” on their vita. (And as an aside, I think you guys have talked about the fact that good reviewing doesn’t tend to get you a reputation because the double-blind persists even after publication).
luke
June 30, 2008 at 11:10 pm
For me, the key issue is the editor. A smart editor will do things to minimize the craziness of reviews. For example, if reviewers offer conflicting advice, then the editor should not, as Holbrook recommends, encourage a compromise. Satisfying reviewers with conflicting points of view leads to sprawling, bad papers. Instead, the editor should simply say “the editors value the reviewers, but we think you should focus on A…” or “you are not obliged to satisfy A and B, but you have to offer an explanation of why you revised in the manner that you did.”
Another example of editors reducing craziness: Many editors will send an R&R paper to *completely* new reviewers. Result? Endless revisions and rejections. Basically, an editor should not be a slave to reviewers, they should instead have a level of quality and use reviewers to help them figure out if the paper has reached that level.
Personally, the craziness of reviewers doesn’t bother me, it’s the craziness of editors. More often, the editors should step in and smooth out the review process. Too often, editors will let papers languish for years, or allow contradictory advice to be followed, or allow really off base unhelpful reviews.
fabiorojas
July 1, 2008 at 1:02 am
Here is a classic exchange from Mind!, the centenary supplement to Mind.
Kieran
July 1, 2008 at 1:34 am
Omar – that was really, really helpful. Doing reviews isn’t something they teach you in grad school and so most of us learn solely through experience. Having a few bad reviewers for papers I’ve written has helped teach me a little of what not to do.
I really like his suggestion, “Avoid rejecting the basic premise or purpose of the work.”
brayden
July 1, 2008 at 1:50 am
Doing reviews isn’t something they teach you in grad school and so most of us learn solely through experience.
Interestingly enough, writing a specimen journal review was one of the first exercises I was assigned in my research methods course my first semester in grad school, complete with the journal’s tick-the-relevant-box paperwork. This was a fantastically useful course that combined stuff on the logic of social research with very concrete applications on the practice of social research. I flipped back and forth between merely hating and actively fearing the class, but of course that was because I was getting an education in it.
Kieran
July 1, 2008 at 2:02 am
I agree with Fabio that the editor is key and a hands on editor that offers guidance is infinitely better than a passive editor who simply tells you “follow the reviewers advice” while sending R & Rs to completely new reviewers. I’ve been the victim of that and the result is precisely what Fabio predicts: endless revisions and going around in circles.
Omar
July 1, 2008 at 3:02 am
Lately, it seems like I’ve had a run of MSS to review for which I’m a new reviewer on an R&R. I really dislike this situation, but I try to take account of the situation in writing my review. Ditto if I find a problem in an R&R that I missed the first time, I apologize and explain why I still think it is important.
The one thing I’d add, as a frequent reviewer, is that we have good days and bad days. Sometimes I’ll give a good review and then I’ll read the other reviewers’ comments and realize that they caught a big problem I missed. And sometimes something just hits me the wrong way and I really hate a piece when it just isn’t that bad. Especially in the latter case, I usually write a note to the editor telling him/her that it is OK to override my very negative review if the other reviewers see more promise. In short, I agree with Fabio and Omar that the editor has to take a stand and judge the reviews.
It’s really a lot of relatively thankless work to write reviews. Sometimes you read something really interesting and exciting and that’s cool. But other times . . .
olderwoman
July 1, 2008 at 4:50 am
Reminds me of what one of my senior colleagues once said: “Editors are the only people in this profession who have real power.”
mikemcbride
July 1, 2008 at 7:30 am
I agree that it is great having an editor who is smart and committed.
Of course, as Fabio stresses, editors do not always use this discretion wisely. And it can be particularly infuriating and demotivating when as a reviewer, you point out serious/fatal flaws in a paper, only to see the editor tell the authors to disregard your review. (Recently, this blow was softened by an editor actually including in the decision letter, an apology to me for accepting the paper despite my review!)
It is worth noting in this forum the vast difference between the editorial style of orgs/mgmt journals and soc journals. Editors in orgs journals tend to act as something like super-reviewers, with a several page letter justifying their decision by summarizing and prioritizing the reviewers’ comments and often adding their own 2 cents. There are obvious pathologies that flow from this. Essentially, you face death by a thousand cuts. And in general, it fosters an idiom where all comments are valid and have to be addressed. By contrast, decision letters from soc editors may include nothing other than the decision and some boilerplate. Though they sometimes include the kind of guidance that Fabio describes, which is crucial on an R&R (note that saying which reviewer to pay attention is complementary with having short reviews. If you are not going to say much and the reviews conflict, providing some kind of guidance is hugely important). A downside of this approach is that the editorial decisions are often uninformed. I think that, just as people in the soc world would be shocked to see the length of a decision letter from the orgs world, people in the orgs world would be shocked to learn that soc editors typically do not read the papers they adjudicate.
ezra zuckerman
July 1, 2008 at 11:32 am