orgtheory.net

improving journal review times

Raj Chetty, Emmanuel Saez, and László Sándor have an NBER paper on an experiment to improve journal review times. The experiment? Pay reviewers:

We evaluate policies to increase prosocial behavior using a field experiment with 1,500 referees at the Journal of Public Economics. We randomly assign referees to four groups: a control group with a six week deadline to submit a referee report, a group with a four week deadline, a cash incentive group rewarded with $100 for meeting the four week deadline, and a social incentive group in which referees were told that their turnaround times would be publicly posted. We obtain four sets of results. First, shorter deadlines reduce the time referees take to submit reports substantially. Second, cash incentives significantly improve speed, especially in the week before the deadline. Cash payments do not crowd out intrinsic motivation: after the cash treatment ends, referees who received cash incentives are no slower than those in the four-week deadline group. Third, social incentives have smaller but significant effects on review times and are especially effective among tenured professors, who are less sensitive to deadlines and cash incentives. Fourth, all the treatments have little or no effect on agreement rates, quality of reports, or review times at other journals. We conclude that small changes in journals’ policies could substantially expedite peer review at little cost. More generally, price incentives, nudges, and social pressure are effective and complementary methods of increasing prosocial behavior.

Love it. I wonder if the ASA Pub committee noticed?

50+ chapters of grad skool advice goodness: Grad Skool Rulz/From Black Power

Written by fabiorojas

July 23, 2014 at 12:01 am

Posted in academia, fabio

11 Responses

Subscribe to comments with RSS.

  1. […] […]

    Like

  2. When I was younger I worried about how long it took reviewers to get back to me. What I worry about now is how long it takes me to get back to them.

    In that sense this rather expensive reform doesn’t address what’s really wrong with peer review, but gets distracted by a side issue.

    Like

    gabrielrossman

    July 23, 2014 at 3:05 am

  3. For those who do worry about review times, or about other people’s submission/review experiences at specific journals, and for those who would like to share their own experiences, check out SciRev (http://scirev.sc/) if you have not done so already. It’s a fast growing data base of submission experiences. Disclaimer: I’m not affiliated with this website, I just think it’s a brilliant initiative.

    Like

    Alex

    July 23, 2014 at 11:03 am

  4. As a related aside from the research, I have a (minor, but simple) suggestion to speed up review times. Most journals will not drop a 2nd review request on a reviewer with one already on their plates, but I’m sure we’ve all experienced that as soon as one review is submitted, the next request to review comes in. The logical response to this incentive structure is to wait until the deadline to submit a review, even if you actually write it days or weeks earlier, in order to buy oneself a little time. The simple solution: if the journal’s policy sets a 6 week deadline for the review, mark the reviewer as “unavailable for additional reviews” for 6 weeks, regardless of when they actually get the review in. This works on both sides — if you submit early, you get a well-earned hiatus; if you submit late, you are subject to possibly getting another review request.

    Liked by 1 person

    Adam K

    July 23, 2014 at 1:31 pm

  5. And paying would create a disincentive for journals (assuming they are the ones paying) to send papers to extra reviewers on the second round, too.

    I assume editors use a four- or six-week deadline because a shorter timeline would result in more “nos,” but I wish the deadlines were two weeks. Either way, I’m going to send the review in at the deadline, give or take a day or two, despite my good intentions.

    Like

    epopp

    July 23, 2014 at 1:36 pm

  6. Well, if journals are still funded by institutional subscriptions, then having the journal divert some of this money back to me seems like a pretty inefficient way of doing things. Effectively it would mean that my school gives Sage $$, and then Sage gives some of it back to me. Instead, I’d rather my place just bump up my paycheck and put journal reviews on my annual merit TPS report.

    Also, I think some in academia would drop everything and just do reviews all day long for $100 a pop. Adjunct pay is pretty miserable. If I were doing that, teaching online, or having to overload, I’d rather write up endless reviews than grade a million papers.

    And one last thing. If I’m reading this right, this experiment gave away $37,500 in honorariums. I’d like that research budget.

    Like

    kenkolb

    July 23, 2014 at 2:13 pm

  7. I wonder how long this NBER working paper (aren’t all NBER papers “working papers”) would take to get reviewed in a peer-reviewed journal?

    For a bit of background on NBER papers see this link:
    http://krugman.blogs.nytimes.com/2013/04/22/understanding-the-nber/?_php=true&_type=blogs&_r=0

    I’ve always liked the idea of NBER as the economist’s Arxiv.org, but most people seem to link to and cite these papers as if they’ve been vetted, which they have not.

    Like

    LKT

    July 23, 2014 at 4:08 pm

  8. Elizabeth,

    I like how you think. Even better, if you changed it to $100 for the reviewer’s first round but $50 for their second round it would be a tax on the dreaded “second round new reviewer.” And then you’d want to up it to $150 for the reviewer’s third round to strongly discourage editors from using the second round R&R as a way to kick the can down the road and/or wait for the emergence of reviewer consensus that ain’t gonna happen.

    (My ideal scenario involves fitting editors with pain collars for this sort of behavior, but I’m an incrementalist and will accept financial incentives as second best).

    Like

    gabrielrossman

    July 23, 2014 at 7:16 pm

  9. This blog needs a “like” button on comments.

    Like

    epopp

    July 23, 2014 at 8:06 pm

  10. “This blog needs a “like” button on comments.”
    I figured out how to turn on like buttons on Scatterplot. Anybody with admin privileges on this blog should check out options in dashboard->settings->discussion.

    Liked by 2 people

    olderwoman

    July 23, 2014 at 8:53 pm

  11. Thank you to whomever did that!

    Like

    epopp

    July 23, 2014 at 11:57 pm


Comments are closed.

%d bloggers like this: