orgtheory.net

on peer review and butterflies

The mechanics, dynamics, and vicissitudes of peer review have always been a hot topic around these parts (e.g. here and here).  In fact, one of our most famous posts consists of an indecent proposal to reform the peer review system. Usually clarion calls for radical revolution in the peer review process come from a shared intuition that really good, really innovative papers, can get squashed by the forces of homosocial reproduction, cognitive blinders and plain old inefficiency that are inevitably bred by a system in which producers are their own critics.

I have always been skeptical of the “radical inefficiency” viewpoint, challenging people to show me that brilliant, path-breaking paper that was never published in a top journal due to the recalcitrance of short-sighted reviewers (with the lone Akerloff example predictably thrown in my face).  But I do recognize the validity of the question behind these complaints.

It turns out that PNAS which has three separate submissions tracks (one traditional, one “self-promoting” and the other “friend-promoting”), provides a window with which to answer the following question: would it make sense to lower the average quality of publications if that meant that a few more highly innovative papers–papers that would not have made through the standard peer review channels–would find a home in a high impact journal?  The results of a recent analysis suggests that the answer may be yes (I think registration is required to take a look at the story).

The study finds that the top 10% (in terms of citations) of papers that are published as either “contributed” (where the NAS member gets her own reviews–from presumably friendly peers–and submits those along with the papers) or “communicated” (where an NAS member does the favor for a non-NAS colleague) tend to have more impact than the  top 10% of “regularly submitted” (blind peer review, editors in charge of getting reviewers, etc.).  This is in spite of the fact that regular papers tend to have a greater overall impact (when considered as a whole).  The authors of the study interpret this is a tradeoff of overall quality for the chance to sneak in a truly revolutionary “hit.”  So there might be something to those complaints after all…but of course, there are always those pesky butterflies.

Written by Omar

December 3, 2009 at 12:35 am

5 Responses

Subscribe to comments with RSS.

  1. The thing I’ve never understood about the Akerlof thing is that “Market for Lemons” *was* published in a top-5 journal. Sure it was rejected by 3 of the other top-5 journals in the discipline but–using today’s rankings–it ended up at arguably the highest ranking one.

    http://nobelprize.org/nobel_prizes/economics/articles/akerlof/index.html

    One of the journals that rejected the “Market for Lemons” is the Review of Economic Studies. This is entertaining because Paul Samuelson complains that his paper with Wolfgang Stolper on the adjustment of factor prices to opening to trade (“protection and real wages”) was rejected at all the best journals and ended up at…the Review of Economic Studies (see here: http://www.econ.rochester.edu/Faculty/jones/Protection_and_Real_Wages.pdf). So yes it suffered rejection, but it’s not like the profession threw it out.

    Like

    maybe

    December 3, 2009 at 2:05 am

  2. Maybe: I think the “poster child” in economics is Theory of the Firm by Coase. It ended up in Economica, which is ranked 113 out of 209 econ journals in 2008. The folk lore weas that it was rejected numerous times, but I’ve never verified that.

    Like

    fabiorojas

    December 3, 2009 at 4:56 am

  3. Truthfully, the problem we’ve seen most recently isn’t about individually innovative papers being kept out, it’s about the way peer-review interacts with the tenure and grant application processes to dampen research into particular areas.

    When Brad Delong wonders why we’re in some kind of “dark ages”, when people ask why so few academic economists were asking important, system-scale questions about finance, when people notice that every defence of the economics discipline involves citing the fact that “heterodox economists” exist and publish, without mentioning that few of them manage to publish in the “top” journals… well, that’s where the nexus of peer-review, publish or perish, economics chairs sponsored by “businessmen with firm views” all comes together.

    Peer-review works well for simple empirical disciplines – political processes still exist when there’s a set of data that contradicts someone else’s set of data, but if it’s easy to repeat the experiment then the power of politics is muted – because in extremis you could put a paper with the data in on arxiv, people will see it and try repeating it.

    As we move further and further away from simple, repeatable experiments, politics gets to take more and more of a hand. We’ve all seen it where you submit a paper to a journal that questions the results of an important paper – and they send it to the author of that paper for review. Sometimes you get a gracious response, other times they just play the system to kill it off – especially if there’s any time element to your work.

    Is this damaging careers? Not that much, it happens to us all once and we realise that we need to play the politics too. However, it can go wrong on the large scale – as in economics – when a philosophy (either of method, or of “what’s important”) starts to dominate the journals that hold the prestige that runs the tenure tracks…

    Like

    Metatone

    December 3, 2009 at 9:14 am

  4. Great Post Omar. Good to see you posting again.

    Btw, remember G. Tullock’s “The Concept of Rent Seeking.” This paper is one the landmark works in Public Choice. However, it was rejected many times by top tier econ journals because it was not replete with equations. As Metatone notes, a method or a notion of what is important can, indeed does, dominate the prestigious journals.

    Like

    Brian Pitt

    December 3, 2009 at 12:16 pm

  5. A friend forwarded me a link to this youtube video…

    Thought it might be interesting to the readers of this blog.

    [PS: This is just a comedy, so please take it lightly]

    Like

    Rajiv

    December 6, 2009 at 12:03 pm


Comments are closed.

%d bloggers like this: