the editors speak: what makes a good review

One of the most important things we do as members of an intellectual community is assist in peer review.  As important as it is, reviewing papers is one of the tasks that receives the least amount of attention in graduate school training.  We certainly learn how to critique in grad school, but, as you’ll see by the editors’ comments below, critiquing is not the same thing as reviewing. Most of us learn how to be reviewers simply by doing it.  While there will never be a definitive how-to manual for reviewing, I thought it would be nice if our field could identify some of the best practices in reviewing. With that idea in mind, I asked a number of current and former editors at journals in organizational theory and sociology to comment about what they think makes a good review.  This post includes their thoughts.

You’ll notice that the editors seem to agree on several important points (e.g., be constructive!), but there is some variation as well.  Some of the editors make very specific and useful points about what reviewers should and should not be recommending in their reviews. Rather than summarize, I’ll just let you read it for yourself. I’ve put their responses in no particular order.

Diane Burton – senior editor, Organization Science

Before I address the substantive aspects of reviewing, I’d like to make a few comments about the reviewing process.

(1)    “Please respond to review requests!”.  I can’t tell you how frustrating it is to have review requests ignored.  The most important thing I do as an editor is identify the set of reviewers for a given manuscript.  I work hard to identify a panel that includes both methodological and substantive experts, but also includes some variety in terms of background and experience.  I ask you to be a reviewer because I think you will play an important role in the review process for a particular paper.  In other words, it is not an automated process, nor is your name a random draw from a hat!  Your decision to review or not is an response to an editor who has made a specific and deliberate request of you.  While a positive reply is always preferred, we all understand that there are sometimes occasions when you must decline a review request.  But when this is the case, don’t hesitate!  If you stall, you hold up the entire reviewing process.  If you are stalling on the reply so that you can extend the review clock – don’t!  It’s much more considerate to say “yes” and then ask for an extension.   If you must say “no”, then try to follow up with suggestions of other people who might be appropriate.

(2)    “Please don’t use the confidential comment area to make substantive points.”  The confidential comment area is a place to disclose that you have seen earlier versions of the paper and/or know the identity of the author.  It is a place where you can explain your approach to the review and/or your own expertise.  Maybe you don’t feel fully up-to-speed on a particular literature or methodology.  Maybe you found the theory so problematic you didn’t spend time worrying about the method (or vice versa).  This is helpful information for the editor.  The confidential comment area is not an appropriate place for substantive feedback that is not shared with the author.  Please do not use it as the place where you express your true feelings about the manuscript.   You put the editor in an awkward position when you write a constructive review with only mildly critical comments that you share with the author, but couple it with a scathing quantitative assessment and devastating confidential comments.

Ok… now, on to what I think makes a good review…

(1)    Good reviews take the manuscript seriously and attempt to meet the author(s) in the place where they are speaking from.  Reviews that suggest the authors ask a different question, address a different literature, or collect different data are not particularly helpful to anyone.  While it is certainly appropriate to suggest literature that may have been omitted, be respectful of the disciplinary orientation that the author(s) starts with.

(2)    Good reviews help the author(s) to improve the work.  Reviews that highlight what is good about the work and build upon it are much preferable to reviews that tear work apart.  In my view reviewers should be arbiters of “quality” rather than arbiters of “taste”.  Instead of deciding whether research is interesting or novel, instead decide whether it is well done, whether it is an advance over what has already been published, and whether it needs to be improved/clarified in order to be as persuasive as possible.

(3)    Good reviews prioritize feedback.  Reviews that make clear which comments must be addressed and which are friendly suggestions are much better than those with laundry lists of suggestions or those with a smattering of half-baked suggestions.  If you are asking for a revision, what will you insist upon being done, and what will you let go?

Michael Lounsbury – associate editor, Organization Studies; editor, Research in the Sociology of Organizations

What makes a good review?  This is hard to pin down as it is context specific, but there are indeed reviews that are better than others and some that are downright poor.  Most good reviews tend to provide a balance of positive feedback and critical observations and suggestions.  It is important to emphasize that the best reviews follow up critical commentary with concrete suggestions on how to improve a paper—both analytically and theoretically.  Also, while most well trained graduate students are able to identify a paper’s weaknesses (of which there are often many), I think that the best reviewers also look for what is particularly valuable or interesting in a paper, and think carefully about how to make the gems (even if hidden) in a paper flower.  The most rewarding part of being an editor is to help authors develop their contribution into a paper that has a major theoretical contribution that is clear and can resonate across multiple audiences.  Having good reviews, and thoughtful reviewers, that are similarly disposed to help make papers the best they can be are invaluable.  Keep in mind that most papers get rejected upon first and even second and third submissions.  Given this state of the field, it is even more important to have good reviews that not only highlight the fundamental flaws, but also offer encouragement and advice on how to go forward—the more specific the better.

Tim Pollock – associate editor, Academy of Management Journal

There are many things that go into crafting high-quality, developmental reviews. And writing this kind of review not only helps other scholars improve their work, it helps you improve your own research by shaping how you think about and approach a manuscript. I have tried to summarize what I think are the main dos and don’ts below. AMJ also provides substantial resources for reviewers on its web site at http://journals.aomonline.org/amj/reviewer-resources. In addition to describing what goes into a good review, the site provides examples of reviews written by each of the editors. The macro and micro editors all reviewed the same macro or micro manuscript, respectively. Although they vary to some extent in form and points of emphasis, together these reviews provide concrete examples of we would consider the type of high-quality, developmental reviews we like to receive.

Do’s and Don’ts

• Stick to the major issues – Particularly on first-round reviews, it is best to focus on the six to eight major concerns and issues you have with the manuscript. If the paper gets an R&R much of the little stuff is likely to change, so it’s not valuable to spend time on that now. Super-long reviews with forty points are not helpful to either the authors or editors, and they don’t impress the editors with your acumen. Two to four pages, single spaced 12 point font is plenty long enough.
• Be developmental – Good reviews suggest solutions to the problems raised. Just saying, “this is bad, this is wrong, I disagree with this, why should I care…” doesn’t really help the authors. Telling them what would solve the problem for you does. If you can’t come up with a tractable solution, you may want to reconsider whether or not you are making a valid criticism, or being too negative and nit-picky. All research has weaknesses. The question is, are these weaknesses fatal, addressable, or just inherent to the theory or methodological approach employed and thus not something the author can address but should be mindful of when drawing inferences and making claims.
• Organize your review according to the importance of the issues – It’s generally much more helpful to editors and authors if you organize your review according to the importance of the issues than according to the layout of the paper. That way we know what you are most concerned about, and the authors know where to focus their efforts. At a minimum, explicitly note which points are your most major concerns.
• Start from a happy place – If you approach a review assuming you are going to reject the manuscript odds are you will, because you will focus primarily on information that confirms this expectation. As a reviewer, when I start reading a paper I assume that I’m going to give it an R&R unless the authors convince me otherwise. This approach makes me more open to the positive aspects of the paper and the potential nuggets that can be developed, and which might be overlooked if you are only looking for reasons to reject the manuscript. It may not take long for the authors to convince me to reject their paper, but I’ve also seen some real diamonds in the rough polished into gems because I was open to seeing what the paper might be.
• Be conversational – Your review should be a conversation with the authors. Address them directly. Don’t talk about them in the third person. On a related note, you don’t need to summarize what the paper is about in the opening of your review. The authors already know, and so does the editor.
• Be nice – Reviews, especially negative ones, are tough enough to deal with. Don’t rub salt in the wound by being nasty or snarky, or by personalizing your comments. Focus your criticisms on the work, not the authors personally, and avoid unnecessary invective. Also try to find something nice to say about the study.
• Number your points – It is very difficult for editors to note specific reviewer comments if they are not numbered. Please number all of your points.

Greta Hsu – associate editor,  Management Science

A good review has two goals. The primary goal is to help the handling editor make a decision about the manuscript.  The secondary goal is to help the author understand how to improve it.  For both, it’s important to have a clear structure to the review.  Be clear about what you find the major issues/concerns to be and which points are minor by comparison.  Have some kind of organizing logic (as opposed to a more stream of consciousness approach) that will help both the editor and the author properly digest the points that you are trying to make.  And be mindful of which concerns you have that emanate from content versus style (the latter of which should probably be relegated to the “more minor” section of your review).

In terms of the second goal–being helpful to authors–try to be as constructive and professional as possible.  If you think there are potential ways to address a problem that you have raised, offer guidance on how to do
so.  If you think the framing is too limited, offer ideas about other literatures or studies the author might look to.  Even if the paper is rejected, it is more likely to improve through the review process if you provide concrete feedback in a professional way.

Jerry Jacobs – former editor, American Sociological Review

A few quick thoughts. It seems to me that the reviewer’s task really depends on the quality of the manuscript. There are some manuscripts that are really weak. This is sometimes quite clear to everyone involved (except the author of course.) In this case, the review should make clear the main weaknesses of the paper, but gently and constructively. The reviewer should reserve any really harsh words for the section reserved for the editor. The review does not have to be super long. It should not be a list of all of the papers that the reviewer would like to see.

The next step up is a manuscript that probably won’t make the cut at the journal in question but which probably should be published somewhere. Again, the review should zero in on the main weaknesses and limitations. Too often reviewers focus on typos, writing style, manuscript length or other issues that do not always speak to the central issue, namely whether the paper in question makes a contribution, and, if so, how its strengths can be strengthened and its weaknesses minimized.

Really strong papers often elicit long reviews which elaborate on how the manuscript could be improved in a zillion ways. The reviewer can have fun here, but only so long as the editor can put the comments in perspective and not make unreasonable demands on the author. Again, the goal is to help to bring the best out in a manuscript, to make sure that the paper presents a compelling argument with sufficient evidence.

Henrich Greve – associate editor, Administrative Science Quarterly

Researchers generally learn reviewing from others, and perhaps especially from the reviews that they receive. There are some reasons why that would not be the best approach. First, when we conduct science it is not desirable that a central part of our system is not learnt systematically. Second, the reactions that researchers often have when receiving reviews suggest that at least some reviews are not necessarily role models (yet they will later emulate those very reviews). Third, learning reviewing from reviewers cuts out the main consumer of reviews: the editor.

What are the key considerations in writing a review? First, consider the audience. A review is requested by an editor who wants help in deciding on a paper (reject, revise, accept). It will be read by author(s) who want help understanding the decision and deciding what to do next (especially when it is reject or revise). Although the editor wants to accept only the best papers and the authors only care about acceptance of their paper, they actually have a shared interest in a neutral and informative report on the paper. But what should be in the report?

Editors want impactful papers, so they are very sensitive to the contribution relative to past work. For the editor, who is often “near” the research topic but not a specialist, reviewers are very valuable when they provide new information on how a paper relates to other work. Has similar research been done before? (Was the past research credible?) Does the research open up new opportunities for those active in this literature? If it builds on existing work, does it do a good job of it? If it is a radical departure from existing work, does it look exciting? These criteria are equally important for theory and empirical work, though they tend to surface when the reviewer discusses the theory.

Reviewers are especially useful if they have a good critical eye on published work. I often find that to be lacking in reviews. Instead of pointing out that there is related research but has clear flaws, which is very useful information for editors, reviewers can be quite obsequious of work in print. As a reviewer I have sometimes criticized a published paper in order to show how a manuscript can get a sharper cutting edge, but I rarely receive such reviews.

Reviews are useful if they help the editor and authors overcome simple barriers to making a contribution. A paper with too many hypotheses gives an impression of being superficial. Are some of the hypotheses more interesting, so that a stronger paper can be made by improving the reasoning for them and deleting the rest? A paper with a convoluted statement of contribution can discourage readers. Is there advice on emphasis and writing that will clean it up?

The examples above are what we call developmental reviews, and are simple forms of development that stay true to the author’s intentions. Developmental reviews that go beyond these limits and instead seek to reformulate the contribution are controversial among editors and authors. Authors typically had a reason or formulating the contribution the way they did, and may not be happy about alternative proposals. Many editors will not support such attempted reformulation in their decision letter because it puts implicit coercion (the reward of a publication) behind a request to deviate from the original goals of a paper. It seems more reasonable to reject the paper, which leaves the author with a free choice of what to do when submitting for the next journal, or to overrule the reviewer.

An important function of reviews is to make sure that the published findings are correct. Here I am mostly speaking from my experience with quantitative methods, but it also holds for qualitative methods. In order to do that well, it helps to remember that editors generally also are strong methodologists, so the main role of the reviewer is write a brief report that the editor can point to, and to check for details that the editor might miss. Because editors are more interested in specialist knowledge than general knowledge, they need more help on specific data sources, measures, and sampling schemes than on analytical methods. Reviewers tend to suggest alternative regression methods too often and to examine samples or measures too rarely. Also, for the important decision of rejecting a paper versus revising/accepting it, data sources and sampling is much more important because it is more difficult to recover from a mistake. Editors prefer not to be placed in a position of asking an author to collect significant additional data, only to reject the paper, so any problems with the data have to be brought up in the first round of reviews so the editor can make the reject decision then. Because these issues need to be dealt with right away, editors dislike the phrase “Because of these theoretical problems, I will not comment on the empirical study,” which is occasionally found at the end a review. These considerations are also in line with the interests of authors, who would rather have a swift rejection and chance to recover in another journal than a protracted process with the same result.

A basic requirement for making sure the findings are correct is that the reviewer knows the method that he or she is commenting on or recommending. I raise this point because now and then I read papers with advanced methods that are inappropriate for the data at hand. Although this could be a result of authors randomly going through Stata regression commands, a more worrying process may be at work. From my work mentoring junior colleagues I know that these are sometimes requested by reviewers, presumably (I am speaking as a diffusion researcher now) because they saw them in reviews or in papers. It is OK to ask why one method isn’t chosen over another, but it is not OK to assert that an author needs to use a method that you only know through the Stata user manual and a couple of published papers. The published papers could be wrong, and user-manual descriptions of what a method does are often not enough to know when it is appropriate.

Finally, and perhaps unlike this post, a good review is not overly long. An editor is not sure what to do when you write a really long review. Maybe you really did not like the paper (but you could have said so in fewer words); maybe you liked it a lot and have many ideas on how it might change (but those ideas seem incompatible). Although I am perfectly willing to tell an author that some parts of a review can be dismissed in a revision decision, I can guess the level of discomfort that will cause in the author, and I would worry about the length of time in getting the revision back and the coherence of the revised paper. Experienced editors know that the best revisions come back when the authors can be focused on a limited set of key issues, while long reviews often produce worse revisions.

Reviewing is not an art. It is sensible application of expert knowledge and sound judgment. Although starting is difficult, it is easy to improve by paying attention to good reviews of your papers and papers that you review, and to how editorial letters respond to the reviews of your papers and papers you review.

Holly McCammon – editor, American Sociological Review

The peer-review process in many ways is a core arena in which scholarly exchanges take place.  Authors receive feedback on their work from their knowledgeable and skilled colleagues and often are invited to resubmit their revised work for further assessment.  The role of reviewer is critical to the peer-review process, and when new scholars are invited to review a paper for a journal, they should jump at the chance.  It’s not only an opportunity to read and comment on new research; it’s also a chance to think deeply about how social scientists can conduct the most rigorous and thorough research possible.

A good review is typically about a page and a half to two pages (single-spaced) in length.  It’s very helpful to both author and editor if the review is clearly organized and doesn’t contain lots of typos and spelling errors.  There are a number of important components to a review and if a reviewer attends to each of them, this will be of great help to all parties involved.  Reviews typically begin with a very brief summary of the content of the manuscript, usually in just two to three sentences.  Very quickly the review should begin its assessment of the work.  The reviewer might begin with a clear overall evaluation of the paper.  Usually this can be done by checking off the appropriate box on a reviewer form (for example, “accept,” “revise & resubmit,” or “reject”), but it is also useful if the reviewer puts his or her general thoughts about the manuscript in writing.  For instance, the reviewer might concisely state what he or she sees as the major strengths and problems in the manuscript.

Sometimes there is an opportunity to send confidential comments to the editor.  Here the reviewer can give a frank opinion as to whether the paper is ready to be published, whether it can be satisfactorily revised, or whether its flaws will be significantly difficult to overcome.  Comments solely for the editor, however, should not be any more critical than comments that will be shared with the author.  Keep in mind also that in the comments that the author will receive from the reviewer, most editors do not want a reviewer to state whether the paper should be published or not.  In the end, this is a decision the editor must make, and in some instances, the editor’s decision will not be the same one the reviewer recommends.

The reviewer should comment on whether the paper is appropriate for the journal.  Reviewers are chosen because of their expertise in the field, and are likely to represent typical readers of the work appearing in the journal.  Does the reviewer find the scholarship interesting and important?  Does the research make a significant contribution to the field?  Is the journal’s readership likely to find the paper important?

The reviewer should also assess the paper’s theoretical argument.  Has the literature been adequately and fairly reviewed?  Are pertinent citations missing?   Are key concepts clearly developed? Is the author’s theoretical argument logically developed or are important questions left unanswered?  Is the author’s theoretical contribution to the literature clearly stated?    In my experience as editor, a common complaint among reviewers is that authors have not clearly articulated their work’s core contribution to knowledge.  Many readers of articles in scholarly journals may not be reading with an interest in the particular empirical case, but are reading instead for the general ideas or the work’s theoretical argument.  Reviewers will assist both authors and editors greatly if they can comment on whether the paper has succeeded along these lines.

Authors and editors also both need a frank assessment of the paper’s analysis, that is, its data, its methods, and its interpretation and presentation of results.  In this portion of the review, specific descriptions of problems are more useful than general statements.  Is the presentation clear?  If not, what key elements are missing or need further refinement?  Is the appropriate method of analysis used?  Are there significant weaknesses in the data?  Are measurements the best availability?  Is the best available adequate?  Do the author’s conclusions overstate the actual findings?  These and many other questions about the analysis are fair game in this part of the review.

Finally, reviewers might also comment on whether there is a compelling “take-home” point from the study.  Overall, is this scholarly work that readers will find memorable, or is the work entirely too “incremental”?

If the reviewer finds that he or she is not qualified to judge some aspect of the paper (for example, the paper’s methods), then the reviewer should simply acknowledge this.  Typically editors ask multiple reviewers to read a paper and one reviewer may be chosen for her or his methodological expertise while another is chosen for his or her substantive knowledge.

Let me close by saying that, in general, the tone of a review should be constructive and respectful.  Even highly critical comments about the paper’s theory or methods can be stated with a certain gentility.  Civility is a good thing for all involved in this scholarly exchange of ideas.

Ed Zajac – co-editor, Strategic Management Journal

As Co-Editor of the Strategic Management Journal (SMJ), I can provide an more official and a more personal statement regarding what we seek from reviewers.  Our formal invitation to new SMJ editorial board members states the following:

We have three focal goals for reviews by editorial board members – first, to provide open-minded assessment of strategic management research that helps expand the boundaries of the field as well as deepen existing research themes; second, to provide developmental reviews that will help authors advance their work, whether for publication in SMJ or in other venues; and, third, to provide timely reviews, with the goal of returning decision letters to authors within 90 days of receiving a submission.

I like the simplicity of having a small number of analytically distinct reviewing goals, in that it helps a reviewer quickly and easily assess his/her review before submitting it to a journal.  I hesitate to rank the importance of the individual goals (since it is likely to be context-specific), but I will say that  if a reviewer achieves only one of these goals, I’d call that a bad review.  Achieving two makes it a good review.  Reaching all three goals is a beautiful thing, and can transform the process of manuscript evaluation into something pleasurable for all parties involved!

Mario Small – associate editor, American Journal of Sociology

1.  Describes the paper’s argument.  You’d be surprised by (a) how many reviews fail to do so and (b) how many sets of reviews which do so differ on their understanding of a paper’s bottom line.

2.  Focuses on the paper, not on the author.  Writing in the second person (“you lost me when you wrote that…”) almost always makes a paper appear less constructive, and more aggressive, than it might otherwise be.  But independently of the rhetorical issue, the assessment that is most useful to an editorial board is that of what the paper did or failed to do, rather than what the reviewer believes the author was trying to do.

3.  Evaluates the paper that was written, not the one that (a) the reviewer would have written or (b) the reviewer wishes someone had written.

4.  Explains how flaws could be corrected.

5.  Is written as if the author were a graduate student.  In a nontrivial number of cases, cranky evaluations are produced by reviewers who purport to know who the author is (and the targets of these guesses are rarely graduate students).  By contrast, when reviewers assume that the author is a graduate student (they often tell us, in the comments to the editors), they inevitably come across as more helpful, objective, and—frankly—rational.

6.  Remembers that, in a plurality if not majority of cases, a paper’s reviews will disagree strongly on whether it should be published.  The differences in perspective are striking (and even a little depressing—even very senior scholars only rarely agree on whether a paper represents good sociology).

7.  Avoids offering an assessment to the author (“a strong effort that should be improved on several fronts before seeing publication…”) that appears to contradict the private statement to the editors (“might be salvageable after a lot of work, but is still not for this journal…”).  This inconsistency only irritates the authors of rejected papers, who believe that the reviews were stronger than they really were.  Yes, journal editors typically ask reviewers not to divulge their recommendations in the comments to the author.  But the review should be completely consistent with the private statement to the editor.  In fact, a probably worthwhile practice would be to avoid writing private comments to the editor unless they’re absolutely necessary.  Really—-the review is already anonymous: How much more privacy does a reviewer really need?

Rory McVeigh – editor, Mobilization

I think that the best reviewers are those who help the author to become a better scholar in the long run, as well as in the short run.  You can count on just about all reviewers to provide feedback that is intended to improve the paper that is under review.  Yet the great reviewers offer the type of advice that will help reviewers to develop better future projects.  One way to do this is to set a high bar, encouraging the author to be more ambitious in terms of the data collection, research design, formulation of a question, or development of a theoretical argument.  Because there is so much pressure to publish—especially for graduate students and assistant professors—there are strong incentives to cut corners or to submit papers before an author really has anything important to say.   I believe that young sociologists (and even old ones like me) are much better off in the long run if they aim high and take the time that is needed to develop a paper that could have a significant impact on the field.  Reviewers, therefore, provide an important service to authors and to the academy when they consistently maintain high standards.  The great reviewers not only call attention to where the paper falls short, but also provide concrete advice about how things could be done differently (and more effectively).

I also especially appreciate reviewers who are able to imagine the research problem from the perspective of the author, rather than imposing her or his own agenda on the author.  A great reviewer takes the time to help the author to discover what is most promising in the author’s own writing, even when what is most promising may hidden or buried beneath a pile of rubble.  Finally, I think the best reviewers are those who recognize and appreciate originality and creativity.  They encourage the author to take a chance and develop the original idea, rather than forcing authors to play it safe by simply applying old ideas to new problems.

Arne Kalleberg – editor, Social Forces

Reviewing is essential to maintaining and developing a professional social science.  Peer reviews are the main ways by which quality control in journals (as well as in granting agencies, books, etc.) is maintained.  Passing the hurdle of peer review enhances our confidence that articles appearing in journals adhere to high standards and should be taken seriously.  While reviewing is time consuming, writing reviews is an important professional obligation.  It also has individual benefits to reviewers, often teaching them new things and getting them to think about problems and issues in new ways.

Given the importance of reviewing, it is worth considering what constitutes a good review.  In my experience, a good review has the following key elements:

1. A good review provides a balanced evaluation of the paper, summarizing both its strengths and weaknesses.  This helps the editor and author decide whether it makes sense to develop the paper further for the journal or to submit it elsewhere.
2. Good reviews are written for the benefit of both the editor and the author.  For the editor, reviewers serve a crucial screening function by critically evaluating the paper and pointing out flaws and problems as well as strengths.  Good reviewers also provide recommendations to editor about the disposition of a manuscript without revealing it to the author.  For the author, reviewers serve as coaches, suggesting constructive ways that the author may improve the paper.
3. Good reviews are polite and respectful of the author’s efforts.  We have all experienced cases where we have submitted a paper to a journal that is not as developed or as good as it should be.  We thus appreciate constructive criticism that will help to make our paper better, especially if it is delivered in a positive spirit and tone.