orgtheory.net

open peer review

Academia is one step closer to embracing open peer review (hat tip to David Kirsch). The Andrew W. Mellon foundation has given NYU and MediaCommons $50,000 to develop and test an open peer review system for academic journals. We’ve had a lot of debate here about how to improve the review process, which has included some modest and crazy proposals. Open peer review is one potential solution to the problems inherent in the review process – e.g., getting good reviewers, determining the quality of papers, etc. Open peer review would allow authors to post their papers online and anyone could step in and serve as a reviewer, offering public comments and suggestions through multiple iterations of paper revision.  Editors use the feedback posted on the online system and monitor revisions to determine which papers should ultimately make it into their journal.  It uses a crowdsourcing logic to move papers to publication. If  you get more eyes on a paper, the author gets better feedback during the revision process and editors will be better at filtering out lower quality papers.

A couple of journals have already tried open peer review. In 2006 Nature‘s experiment was seen as a failure, but four years later the humanities journal, Shakespeare Quarterly, used it quite successfully, which has prompted the journal to try it again. I like the idea of making peer review more open and competitive. I’m not one of those who thinks the peer review system is currently broken (by and large I’m happy with the quality of articles published in our fields’ top journals), but I think we should be embrace technological opportunities to improve the system. One upside of an open peer review system would be improvement in paper quality, but more importantly I think that open peer review could speed up the peer review process. If you allow more people to quickly gain access to a paper, you wouldn’t have to wait months and months to hear back from the editors’ assigned reviewers. Feedback and revision could occur simultaneously, which is really an ideal model for social science.

There are some serious downsides to consider. Some authors will resist having their work vetted openly.  Public criticism can be hard to take, especially if you’re a junior person seeking tenure. The system might frustrate scholars of all rank and status who don’t want to let the public in to see their half-baked ideas and analysis. In some ways we’re all invested in the illusion that great scholarship just blossoms on its own – we’d rather not let everyone see how the sausage is made, especially when it’s of our own making. There is also the potential for a tragedy of the commons scenario. Currently, the direct incentive to peer review is to maintain one’s good standing with journals we’d like to publish in some day.  If no one is calling on you to review, the system completely relies on professional norms and reviewers’ good will. Open peer review might work well for one or two special issues, but when the novelty wears off and the system is congested with hundreds of submissions, willingness to review might dissipate. Sadly, I also think it’s possible that the system could be overloaded quickly if everyone just starts posting their crap online. Open peer review would require that authors take responsibility in submitting papers selectively.

I think we could overcome these obstacles, but it would requires some innovative solutions (e.g., editors could choose which papers get posted online for review and desk reject the rest). Someone will eventually have to take the risk and volunteer to be the journal to try the model out in our field. Given the risk involved, my guess is that the instigator will have to be one of the well established, high status journals if this is going to have any chance of success. Perhaps a special issue of ASQ or AJS is in order.

Written by brayden king

April 14, 2011 at 3:05 pm

5 Responses

Subscribe to comments with RSS.

  1. I wonder to what extent the idea that the “best” articles will simply come out on top because of their intrinsic qualities, rather than because of social dynamics, will actually hold in this case. As Watts and Salganik show, this is certainly not the case in cultural markets. It is possible that the academic market is similar to cultural markets and there are collective social dynamics at play here too – long tail distributions and all that…

    Like

    orgtheory reader

    April 14, 2011 at 8:47 pm

  2. I’m afraid this could lead to a large drop in the quality of *reviews*. I’ve gotten a few too many reviews back from AJS/ASR over the years, where it’s clear that the referee didn’t read the whole paper or didn’t read it in any depth. I’m afraid that open review will too easily encourage a sort of “hit and run” reviewing, where people skim or read some small bit of the paper then criticize it, without taking the time to try to understand the paper as a whole.

    If we’re going to do experiments, I’d rather try things like triple-blinding before something like this.

    Like

    joe

    April 14, 2011 at 11:55 pm

  3. Joe,
    I think one of the ideas in the open review is that the author can openly comment on the superficial reviews. Every so often review comments raise issues that can be dealt with very easily “oh I didn’t realize that would be confusing, if you read two pages back/fowards I actually address this issue”.

    OT Reader, I agree. It could easily turn into a popularity contest. Even worse, we might have battles over broader programmatic questions quite besides the quality of the individual article (institutional theory is too vague; rational choice is naive; blahblah).

    Like

    Henri

    April 15, 2011 at 11:38 am

  4. […] the merits of an open- peer review process. For an insightful opinion on this, you can check out Brayden King’s post on orgtheorynet and a more detailed commentary, click here. . I see the merits of an open review […]

    Like

  5. Друзья , нужно Ваше мнение , кто знает или сталкивался.

    Есть потребность купить колечко с бриллиантом массой более карата, но понимаю , что это стоит не мало и мне не по карману.

    Но читала , что есть облагороженные бриллианты, которые ничем не отличаются от обычных, но стоят дешевле в

    три раза.

    Кто-нибудь вообще держал такие в руках, они правда прекрасны ?

    Like

    Luna

    June 5, 2011 at 6:06 pm


Comments are closed.

%d bloggers like this: