a crazy proposal: universal peer review

I had a crazy idea while talking to some friends over dinner Saturday night. Why not create a universal peer review system that applies to all journals in a field? The problem with the peer review system, as many see it, is that it takes so darn long to get a paper from first submission to publication stage and it tends to exhaust really good reviewers (who sometimes may be asked to review 3-4 papers a month). The world would be a better place if we could kill these two birds with one stone by shortening the time to publication and using reviewers’ time more efficiently. A universal peer review system would deal with those problems.

The idea stems from the law review system in which people submit their papers to a clearinghouse where it can then be assessed by all of the law review editorial staffs that the author targets for submission. You can submit to as many law reviews as you’d like. Once one law review accepts the paper  a bidding war begins between the reviews. The author almost always takes the offer from the most prestigious law review, which often doesn’t come until a number of lower-tier publications have made their case for why you should publish the paper in their review. It may be easier to convince a higher tier journal to accept your work if you already have 15 acceptances. The system is fast. You submit papers twice a year, which is followed by 2-3 months of negotiation until all of the publishable papers are divvied up among the reviews.

Someone suggested it would be great to have the same process with management (or sociology) journals, but the barrier, of course, is that we rely upon peer review, whereas law review editors rely upon their own intrinsic criteria.  You could make it work though, I think, if you were to create a universal peer review system. Authors would submit their work to a clearinghouse and check the journals to which they wanted to submit. The system would then pick reviewers from an approved list of reviewers from the set of journals that the author submitted to. You would probably pick more reviewers (4 or 5) given the need to ensure an accurate estimation of quality. In addition to the normal feedback and ranking of quality,  reviewers would be asked to rate the journal in terms of its appeal to various audiences (e.g., does this paper appeal to a general management audience? sociological? economics? etc.) The reviewers themselves could be ranked according to reliability and taste so that the editors could assess the fit of the paper to their journal. Once all of the papers are reviewed (you should be able to do this in a month since reviewers will only be doing this twice a year), the editors could start sorting the database and finding papers that fit their journal’s criteria of quality. Once they’ve identified some subset of papers they find worthy of publication the bidding could begin. Once a paper was accepted, of course, the editor and perhaps a subset of the first-round reviewers would commit to work with the author until the paper was in acceptable shape for publication; however, the biggest chunk of work for reviewers would come at the front end, when assessing the quality of papers.

Why would this work? Well, for obvious reasons it would be great to have a system that pushed papers to publication quicker. Many papers, especially those that deal with timely topical issues, lose their relevance unless published immediately. Even theoretical papers are sometimes outdated by the time they are published. I see this as a huge problem with our field. It would also put more incentive on the authors to publicly air and get feedback on their papers before submitting their work to journals. While I’m okay with a paper being submitted that isn’t 100% ready (what paper is?), I’m afraid that too many low quality papers are submitted simply because the author is looking for developmental feedback. With a one-and-out system like this, authors would be more diligent in seeking feedback and polishing their work before submitting. I also like the idea because it would allow innovative ideas to rise to the top. During the bidding process, naturally intriguing papers that might not fit the conventional model of a journal might suddenly appear more publishable if they are vigorously pursued by many other journals in a field.  Perhaps the best reason for doing this is that it would vastly cut down on the amount of time scholars spend reviewing work. Some papers get submitted 5+ times before being accepted. Multiply that by the number of reviewer hours you used to move that paper to publication and you get a sense of the inefficiencies of the current system. (I’d argue that most papers are only marginally improved by going through multiple rounds of review.)

I’m sure that you readers can think of many reasons why this system wouldn’t work, but I’m willing to bet that we could collectively figure out a way to implement the system despite the reservations. It may seem like a crazy idea, but I think it’s a potentially very effective solution to our field’s inefficiencies.

For more crazy ideas on improving the peer review system, see Fabio’s suggestion for a triple-blind peer review process.


Written by brayden king

August 9, 2010 at 1:35 pm

Posted in academia, brayden

10 Responses

Subscribe to comments with RSS.

  1. i recently reviewed a paper for a top-tier journal and although i liked it the paper was rejected after an R+R, i then got a request from a second-tier journal to review the same paper. w/o telling the editor /how/ i had reviewed the paper the first time, i told him that i had already reviewed it and was happy to do so again but wanted the editor to have the option of getting a fresh reviewer, which the editor in fact chose to do.

    i think implicit in the option i gave the editor and the editor’s choice to exercise the option is that the main appeal of the current system is based in an understanding that peer review is a very noisy process. having a universal peer review system would conserve error rather than allowing a fresh and independent draw of error. this would imply some nontrivial number of false negatives of papers that are reasonably good but published at low-ranked journals or never published at all because on the initial draw they get stuck with (and then conserve) hostile reviewers.



    August 9, 2010 at 2:40 pm

  2. Gabriel, true, this could be a problem. That’s why I think you want an initially larger reviewer draw (5 reviewers) so that the editor gets more opinions to look at the first time. You could even imagine adding an option to the system that would allow the editor to drop the worst review to ensure that the mean is not being dragged down by some strange outlier.

    Because editors have quirky reviewer preferences, editors could also weight more heavily reviews given by people from their preferred inner-circle (which happens anyway). Thus, it’s possible that a paper with a median average rating could still make it into a top journal if the editor’s preferred reviewers loved the piece.

    There would definitely be some error in the system, but there is now, but it just comes at a much higher cost to the collective.


    brayden king

    August 9, 2010 at 3:10 pm

  3. You may also want more associate editors in this kind of system given that editors will have more work in the initial selection process.


    brayden king

    August 9, 2010 at 3:18 pm

  4. Hmmm…I think this would have some possibilities. It seems to me, though, that the trickiest part is in defining the boundaries of the field. In law, this is fairly clear–though there are a few outliers, basically if you are a law review from a law school you participate in the system. But in many disciplines the boundary lines are a lot fuzzier. For instance, consider a journal like Mobilization, which publishes work by sociologists, political scientists, etc. Would it be listed in the universal peer review systems of multiple disciplines? Choose only one discipline, and people who wanted to submit from another would have to put it in that system? Opt out entirely, and thereby undermine the entire idea?



    August 9, 2010 at 4:37 pm

  5. mikaila,

    on the “socialism in one country” question, it’s worth noting that econ recently started an approximation of Brayden’s proposal on a small scale by allowing authors rejected by AER to request that the entire review portfolio be forwarded to one of the baby AERs for expedited review.



    August 9, 2010 at 4:47 pm

  6. Interesting. Prüfer and Zetland present a somewhat similar idea in a forthcoming article in Public Choice ( Here is the abstract:

    “An Auction Market for Journal Articles”
    Abstract: We recommend that an auction market replace the current system for submitting academic papers and show a strict Pareto-improvement in equilibrium. Besides the benefit of speed, this mechanism increases the average quality of articles and journals and rewards editors and referees for their effort. The “academic dollar” proceeds from papers sold at auction go to authors, editors and referees of cited articles. This nonpecuniary income indicates the academic impact of an article – facilitating decisions on tenure and promotion. This auction market does not require more work of editors.


    Jon Bingen Sande

    August 10, 2010 at 10:30 am

  7. Mikaila, That is a problem indeed. But I think you could start small and hope that one day you create a massive social science peer review system from which journal-specific reviewers could be drawn. To get started though would probably take the initiative of a small subset of journals. Imagine what would happen should AJS, ASR and the other ASA journals, Social Problems, Theory and Society, Social Forces and the other regional journals all decided to go in on a system like this. The incentives for a journal like Mobilization to join the fun would increase because they wouldn’t want to miss out on the papers in the pool.

    John – thanks for the article link. Very relevant!


    brayden king

    August 10, 2010 at 1:39 pm

  8. teppofelin

    August 25, 2010 at 3:08 am

  9. […] Universal peer review. […]


  10. […] a lot of debate here about how to improve the review process, which has included some modest and crazy proposals. Open peer review is one potential solution to the problems inherent in the review […]


Comments are closed.

%d bloggers like this: