how many journal article reviews should I do?
Blogger emerita Christine Percheski asked how many reviews a person should do. O.w. responded:
cperchesk et al on # you should review. Minimum baseline: you “owe” the system at least 3 reviews for every article you submit. That is the minimum. Anyone has (IMO) a moral obligation to say yes to at least that many requests. But the majority of submissions come from relatively new people who will not be asked to review by most journals until they are published. So people with track records, especially tenured people, have to make up the slack. My next benchmark thus is that anyone with tenure owes at least 6 times as many reviews as s/he submits articles in a given year.
I share this sentiment, but I add a budget constraint. We should definitely “pay back,” especially more established scholars. However, there are real time constraints. I have found that I simply can’t write quality reviews more than once a month or so and still keep teaching, mentoring grad students, grading papers, and so forth.
I say yes to any review request for which I am qualified. I space my reviews to 1 or 2 per month. If it will take more than 3 months to respond, which means that I have already accepted about 5 papers, then I tell the editors: “I’d love to review, but I have already committed to reviewing other papers and it will take at least X months for me to respond.” That seems to keep things reasonable. It also helps that I adopt a triage model of reviewing. Effort is spent on papers on the margins, not clear accepts or rejects. My energy is conserved for papers that might make it up , but need some TLC.
What do you think?
orgtheory.net 
I’d suggest not being too instrumental about reviewing. This is a community good, and unfortunately there just aren’t enough reviewers to go around. Saying yes to reviews demonstrates commitment to our academic ideals and helps split up the work that needs to be done to help make our intellectual fields move forward. Clearly there are times when it may be necessary to say no, especially if the journal or paper are far afield from your own expertise, but those times are probably less common.
If you’re worried about wasting time on too much reviewing, I suggest economizing your reviewing time depending on your closeness to the topic. I put more effort into reviewing when I’m very close to the topic. If the paper is right in my sweet spot – social movements and organizations – I spend more time with the paper and tend to write more notes in the margins, do background research to make sure claims are accurate, etc. When I’m reviewing a paper that is more distant from my research, I spend less time in the nitty gritty details and usually stay at the level of the big picture (e.g., how well do they answer the research question?), soundness of the methods, etc. I’ll spend an hour reading the paper and no more than an hour writing the review, sometimes less if the issues are simpler.
I’d also add that reviewing seems to proceed more quickly the more you do it. You can train yourself to spot the key strengths and weaknesses of a paper.
LikeLike
brayden king
June 8, 2011 at 1:22 am
I agree with Brayden that reviewing can be done in a couple of hours. Sometimes a bit more (particularly in cases of R&Rs that could make a good contribution), sometimes a bit less (particularly in cases of clear rejects).
I do think we all should be regularly reviewing and saying yes to lots of requests. Though I’m not sure the minimum rate should 3 for every paper one submits. I think it could be more like 1.5 or 2 per submission, as the field is (presumably) expanding and not every paper gets 3 reviews. It’s like the seeding torrents debate: some argue for a ratio of 1.5, some for 1, and others for .5.
LikeLike
cwalken
June 8, 2011 at 2:41 am
The minimum rate HAS to be N reviews for each paper you submit, where N is the number of reviews a paper gets, or the people writing articles are free riding on the reviewing work of others. Your own “budget constraint” about how many you can review per month is irrelevant. And, again, the majority of papers circulating for review are coming from people who are not yet established. So I assert that a tenured person owes at least 2*N reviews to the system. I asserted that N was 3. Suppose it is only 2. So, how “productive” are you? If you are submitting 2 articles a year and are tenured, my calculation says you owe the system 4-6 reviews a year. If you are submitting 6 articles a year, my calculation says you owe the system 12-18 reviews a year. That is within Fabio’s budget constraint. But if you are cranking out 12 articles a year then you owe 24-36 reviews, i.e. 2-3 a month.
I personally get asked to review 4-6 articles a month, and so do a lot of other people I know, so somebody out there is not pulling their weight. Or my estimate of the fraction of papers generated by junior scholars is too low, and the multiplier for being senior needs to be 3 or 4, not 2.
LikeLike
olderwoman
June 8, 2011 at 4:06 pm
OW: Of course, the moral math of all this gets more difficult for people who do a lot of co-authoring.
LikeLike
jeremy
June 8, 2011 at 4:28 pm
I have a hypothesis that those of us not in research institutions are not approached often enough to review in the first place. I don’t have evidence to support this in terms of article reviews, given that I am a junior scholar who would probably not be approached so often in the first place; however, my investigation into reviewer selections for book reviews in Contemporary Sociology (about which I commented in the September 2009 issue: http://csx.sagepub.com/content/38/5/498.2.citation) provides clear evidence of a selection bias in favor of those at research institutions. There are of course legitimate reasons for focusing on reviewers with high research productivity, but being at a research institution (especially after tenure) is only a rough proxy for that. When I was an assistant editor at Sociological Forum as a grad student, we were tasked with identifying relevant reviewers through Sociological Abstracts searches, which did, I believe, generate a sample of reviewers dispersed more broadly across the discipline. If memory serves, the research university types were much more likely to excuse themselves from reviewing on account of being too busy–not surprising, given how often you all seem to be asked!
LikeLike
Mikaila
June 8, 2011 at 4:31 pm
Mikaila – I think you’re right about that, although I have no evidence to support it.
Here’s the problem we’re now facing. Non R1 schools are increasingly requiring more publications from their faculty to demonstrate competence/quality (I’m not saying that’s a bad thing, but it’s a change from previous generations), which leads to an increase the overall amount of reviewing that needs to be done. However, scholars from R1 universities still get asked to do most of the reviewing, often not because they are the only qualified people to do it but because they are typically the most salient and visible scholars in their fields, which increases the burden on those scholars and leads to more people declining to review. I think the answer should be that for every article submitted by a scholar to a journal, the editor should ask that person to review 3 articles that year. Quid pro quo.
LikeLike
brayden king
June 8, 2011 at 5:03 pm
BK: editors are not going to trust the reviews of people whose articles they reject.
LikeLike
olderwoman
June 8, 2011 at 5:12 pm
Excellent discussion. A few replies:
1. We are definitely in broad agreement, but here is why I must disagree a little on the margin. Beyond a certain point, my reviews simply drop in quality.
You may respond: “But it’s your responsibility to properly review your paper.” My response is: “Well, yes, but I also have to teach, work with graduate students, write grants, do administrative work, and be the best I can be with my family.” My experience is that after I hit two reviews per month, my quality drops because I simply don’t have the extra time in the budget and I refuse to short change my students or my family. There is no point in handing a poor review to the author. I’ve wasted my time and I’ve wasted their time. Better to send the paper to someone else.
2. @brayden: “The rule of 3” The problem isn’t with journals I submit to. I have only once in my entire life declined a request from a journal I’ve submitted to – and that was because it simply wasn’t my area. However, I get lots of requests from journals that I have little contact with. So in a semester, I may get a request from AJS, ASR, and one of the big management journals. That’s three right there. Then I’ll also get 3-6 requests other journals and book publishers. That’s literally *hundreds* of pages I have to closely read and comment upon. At one point, I think I had 8 (!) requests in a single month. That’s when I started spacing out reviews. I simply can’t do that many and do a decent job.
LikeLike
fabiorojas
June 8, 2011 at 5:51 pm
I was taught never to say no to a review that I was capable of doing. Of course this might, at some point, get out of hand, but I would have a hard time turning a review request down. I see reviewing as a vitally important part of contributing to the discipline – as important as training students, teaching, or publishing my own research. It pays off in a different way, as it doesn’t count toward tenure and is largely anonymous, but I believe it still has tremendous merit.
Maybe it’s naive, but I don’t see the peer review process as a vetting that needs to be done, but as an opportunity to improve the quality of work that ultimately published. I hope that people who review my work feel the same way. I want to benefit from their expertise and insight and for them to give me the opportunity to improve my work. I learn a great deal from the manuscripts I read – even if they’re ultimately rejected – and from the other reviewers’ comments and I enjoy seeing the final results. I don’t think that focusing on the big picture that Brayden described is a cop-out when pressed for time or outside of your area of specialization. It can be a huge help to both editors and authors.
LikeLike
jessica
June 8, 2011 at 6:22 pm
Journals that have sent me rejections (in some cases rather mean-spirited) have then asked me to review papers. Being an administrator has led me to cut way back. I always have too much “urgent” junk to do (only for three more months).
I think it is very important to do only that which you can do well. I find editors are pretty understanding if you explain that doing the job properly in the amount of time allowed seems highly unlikely. Sometimes you get the feeling the reviewer didn’t read the whole paper. That can be disheartening.
LikeLike
David Hoopes
June 8, 2011 at 6:47 pm
I’d echo Mikaila’s guess that the larger pool of potential reviewers is not equitably called upon, which overburdens the most prominent scholars in any field.
Perhaps journals should make more of an effort to reach out to scholars not at R1 universities, graduate students, and even those outside of academia who are still engaged with their fields. We could also think of having a different threshold for being qualified to review–is it publication in that area that makes you an expert? Or could conference presentations, dissertations in progress, etc., count? Given the ever lengthening time of the publication process, waiting to call upon graduate students and early assistant professors until they’ve published extensively could be leaving a large number of potential reviewers aside.
I, too, always say yes to a review if it’s close enough to my field of expertise. But I’m consistently surprised by the quality of what gets submitted. I wish editors were a bit more judicious in suggesting further work before letting papers go to the review process which will be rejected and then have little to no hope of publishing the piece in that journal once it’s improved.
LikeLike
cwalken
June 8, 2011 at 7:36 pm