orgtheory.net

reviewer draw and paper acceptance

I think who one happens to “draw” as a reviewer has a lot to do with whether a paper gets accepted or not. This might be fairly obvious, but I think its more consequential than we’d like to think. I recently talked to a colleague about this, his perfectly excellent and publishable paper (and, he indeed does first-rate work) simply got a “bad reviewer draw”: two reviewers strongly rejected the paper, one liked it, and the editor sided with the negative reviews and rejected the paper. Had I been a reviewer (replacing one of the negative views), the majority would have liked the paper, the outcome may have been different. My friend got a bad draw.

I certainly don’t think that editorial activity should consist of vote-counting, quite to the contrary, and thus I presume reviewer draw plays into an editor’s decision calculus. Overall, I trust that editors have taste and judgment about whether a piece meets standards of scientific rigor, theoretical contribution, etc, independent of what some reviewers might say. Thus, editors may — from my perspective, even ought to — go completely counter to what the majority of reviewers might say. I am curious as to how often this happens.

So, while blind peer review is still the best system out there, there are also many problems with the system, specifically related to who one happens to get as a reviewer: incommensurable epistemological commitments, judgment and taste, and perhaps, reviewer ineptness or lack of knowledge.

I am sure editors are sensitive to issues of “reviewer draw.” These days, with electronic manuscript review systems (and, quick electronic access to reviewer publishing records and areas of research), its probably easy to track all kinds of things: who reviewed, how often do they accept, to know a reviewer’s epistemological sympathies, theoretical proclivities, etc.

I don’t know that anything needs to necessarily be done about the issue of reviewer draw. Some journals address some these problems by letting you suggest potential reviewers. Other journals are so specialized that you know the audience will be somewhat sympathetic. Some journals have “departments” or special editors for particular topics.

I suppose, in the end, given that most disciplines at least have three or so top outlets, the best papers end up getting published. And, while we wait for that acceptance, reviewer draw provides a nice excuse that might temporarily placate the academic ego after a rejection.

Written by teppo

April 23, 2008 at 8:11 am

Posted in research, teppo

9 Responses

Subscribe to comments with RSS.

  1. I think a bigger issue that does need some form of redress is the issue of blind review in the google age. Blind review has never been fully blind — editors know who the author(s) are and this information undoubtedly shapes the impressions of even those who work very hard to avoid such biases — but now, it is, as a friend over at scatterplot said to me in a recent phone conversation, “almost hard NOT to know” whose paper you are reviewing. Recently, I reviewed a very very weak paper (fatally flawed, actually) and was astonished to find it was accepted by an excellent journal. Able to resist no more, I googled the title and found an explanation — the author is a very prominent person in the field. Did the other reviewers know this? Perhaps, perhaps not — but either way the issue remains that we are kidding ourselves if we pretend that blind review is blind review with the wealth of information available a keystroke away (facilitated largely by conference paper titles being online).

    Solution?

    Like

    mom

    April 23, 2008 at 1:20 pm

  2. Mom: Have you read my post on “triple blind review?”

    https://orgtheory.wordpress.com/2007/01/23/a-modest-proposal-triple-blind-review/

    Like

    Fabio Rojas

    April 23, 2008 at 2:44 pm

  3. I have of course read the triple blind review post, provocative idea. But, in this case, I think it makes the draw issue worse rather than better.

    Like

    tf

    April 23, 2008 at 3:20 pm

  4. Hadn’t seen that fabio — amusing… though we’ll have to have ghosts present our papers at conferences and have them be namelsess, or else the google problem doesn’t go away ;-)

    Like

    mom

    April 23, 2008 at 3:52 pm

  5. It’s things like this that terrify me as a graduate student; especially when it’s likely that my first paper submitted for publication will be challenging the conclusions of a “big name” who may end up as a reviewer on the paper.

    Like

    tcausey

    April 23, 2008 at 4:04 pm

  6. The reviewer draw problem is fairly random though isn’t it? You may get a bad draw at one journal, but it’s highly unlikely that you’ll get a bad draw with multiple, consecutive submissions of the same paper. My guess is that if you feel like your paper is always getting a bad draw, then you’re probably not framing the paper in the right way (not speaking to you directly on this Teppo – just general advice). You need to figure out how to signal to the editor which kinds of reviewers would be preferable for a paper like yours and/or change something else that keeps bogging the paper down in the review process.

    Like

    brayden

    April 23, 2008 at 4:12 pm

  7. Good point, Brayden. I currently have a paper where I’ve had two not so great reviews and I’ve come to the conclusion you might have suggested. I’ve just really badly framed the paper and I’m going to really have to rework it if my target audience will buy it.

    However, I’d also add that if after multiple draws you get fairly random comments, I’d say keep submitting. Unless you really suspect that the paper is flawed, random reviews probably suggest it’s in the ball park and you should wait for an editor to R&R with some firm advice on further revisions. No point in spending all this effort revising in response to random noise. Only revise in response to coherent instructions.

    Like

    Fabio Rojas

    April 23, 2008 at 4:19 pm

  8. “…only revise to coherent instructions.”

    I certainly concur there! There is a certain mystique to reviewers comments sometimes, and, its simply not necessary (famous last words).

    And, yes, I certainly don’t mean to suggest that one should not take reviews seriously (in terms of revisiting framing, etc), just that who one happens to get (however this is done…and, thats important, scarcely random) as a reviewer will impact the fate of a paper. And, yes, luckily there are enough outlets where I think the good papers, in general, get published in the good outlets.

    Like

    tf

    April 23, 2008 at 4:36 pm

  9. […] has included some modest and crazy proposals. Open peer review is one potential solution to the problems inherent in the review process – e.g., getting good reviewers, determining the quality of papers, etc. […]

    Like


Comments are closed.

%d bloggers like this: