orgtheory.net

design-focused review: a guest post by samuel r. lucas

Samuel R. Lucas is professor of sociology at the University of California, Berkeley. He works on education, social mobility, and research methods. This guest post proposes a reform of the journal review process.

On-going discussion about the journal publication process is laudable. I support many of the changes that have been suggested, such as the proposal to move to triple-blind review, and implemented, such as the rise of new journals that reject “dictatorial revi”–oops, I mean “developmental review.” I suggest, however, that part of the problem is that reviewers are encouraged to weigh in on anything–literally anything! I’ve reviewed papers and later received others’ reviews only to find a reviewer ignored almost all of the paper, weighing in on such issues as punctuation and choice of abbreviations for some technical terms. Although such off-point reviews are rare, they indicate that reviewers perceive it legitimate to weigh in on anything and everything. But a system allowing unlimited bases of review is part of the problem with peer review, for it shifts too much power to reviewers while at the same time providing insufficient guidance on what will be helpful in peer review. I contend that we need dispense with our kitchen-sink reviewing system by removing from reviewer consideration two aspects of papers: framing and findings.

Framing is a matter of taste and, as there is no accounting for taste, framing offers fertile ground for years of delay. Framing is an easy way to hold a paper hostage, because most solid papers could be framed in any one of several ways, and often multiple frames are equally valid. Authors should be allowed to frame their work as they see fit, not be forced to alter the frame because a reviewer reads the paper differently than the author. A reviewer who feels a paper should be framed differently should wait for its publication and then submit a paper that notes that the paper addressed Z but missed its connection to Q. Such an approach would make any worthwhile debate on framing public while freeing authors to place their ideas into the dialogue as well.

As for findings, peer review should be built on the following premise: if you accept the methods, then you accept the findings enough for the paper to enter the peer-reviewed literature. Thus, reviewers should assess whether the paper’s (statistical, experimental, qualitative) research design can answer the paper’s research question, but not the findings produced by the solid research design. Allowing reviewers to evaluate findings allows reviewers to (perhaps inadvertantly) scrutinize papers differently depending on the findings. To prevent such possibilities, journals should allow authors to request a findings-embargoed review, for which the journal would remove the findings section of the paper as well as the findings from: 1)the abstract, and, 2)the discussion/conclusion section of the paper before delivering the paper for review. As some reviewers may regard reading soon-to-be-published work early as a benefit of reviewing, reviewers could be sent full manuscripts if the paper is accepted for publication.

A review system in which reviewers do not review framing and findings is a design-focused review system. Once a paper passes a design-focused review, editors can conduct an in-house assessment to assure findings are accurately conveyed and the framing is coherent. The editors, unlike reviewers, see the population of submissions, and thus, unlike reviewers, are well-placed to fairly and consistently assess any other issues. Editors will be even more enabled to make such calls if they can make them only for the papers reviewers have determined satisfy the basic criterion of having a design solid enough to answer the question the paper poses.

The current kitchen-sink review system has become increasingly time-consuming and perhaps capricious, hardly positive features for effective peer review. If findings were embargoed and reviewers were discouraged from treating their preferred frame as essential to a quality paper, review times could be chopped dramatically and revise and resubmit processes would be focused on solidifying design. As a result, design-focused review could lower our collective workload by reducing the number of taste-driven rounds of review we experience as authors and reviewers, while simultaneously reducing authors’ potentially paralyzing concern that mere matters of taste will block their research from timely publication. Design-focused review may thus make peer review work better for everyone.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

April 28, 2016 at 12:02 am

9 Responses

Subscribe to comments with RSS.

  1. Removing framing from the review process is a very interesting proposition, and I agree that it would make journals more dialogue if reviewers were encouraged to publish their framing critiques. Indeed, speeding up the reviewing process would make this all the more realistic.

    Like

    Hpatrick

    April 28, 2016 at 9:46 am

  2. Removing framing from the review process is indeed a great idea. I haven’t counted the hours, days, and weeks I discussed about how to frame the paper with my co-authors, before I received the reviews in which 2 reviews recommended three different ways how to frame the paper. Because framing is highly subjective, I think there is no good argument as to choose one framing over the other (as long as both are related to the data/findings). Researchers invest so much effort and time into framing their paper/developing an interesting “story” so that little time is left to care about data collection and analysis (the fundamentals of science).

    Like

    #removeframing

    April 28, 2016 at 10:20 am

  3. So I’m going to take a contrary position here. Good sociology is not just solid research design and statistical analysis (please read Abbott’s ‘preface to lyrical sociology’ published in Sociological Theory several years back). A good piece of research must justify its importance to the public good, scholarly debate, and so on. All of this is part of the ‘framing’ boogeyman. To be sure, raw empirics are an important element to any good study, but the paper should be intellectually stimulating and a joy to read. Going this road you will inevitably end up with some subjectivity and disagreement over stylistic preferences , but let’s not throw the baby out with the bathwater. Let’s instead emphasize to reviewers that they should evaluate the internal logic and persuasiveness of the author’s framing. Even if the particular framing isn’t the reviewer’s cup of tea, he or she should be able to evaluate it on the author’s terms.

    Liked by 1 person

    Stan

    April 28, 2016 at 2:52 pm

  4. The “framing” discussion is perhaps conflating two or maybe three different things. (1) Does your data answer the research question you posed? Does the front end match the back end? This criterion should be applied to any paper. Is the paper internally consistent. (2) What audience are you speaking to? How is your research important to them? Are they more interested in your specific empirical case or more interested in the broader theoretical implications of your result? This is always an important consideration in any writing, but it probably the point where reviewers go awry as they imagine the only relevant audience is themselves. (2.5 or 3) Comments focused on battling against the author’s theory or agenda and simply refusing to accept the author’s assumptions about audience.

    Like

    olderwoman

    April 28, 2016 at 3:07 pm

  5. I agree with Stan (and probably others) that papers are more than quantitative or qualitative empirics, and I did not mean to suggest otherwise. And I take the implicit point that not all papers have a form that would allow design-focused review; indeed some of my own published papers would not allow such a review. So, the proposal might only be activated (either by author-request or editorial decision) for works whose structure allows such review.

    Further, I certainly agree that reviewers should say whether a claim inside a frame is right or wrong. If someone says “Bourdieu views one’s embodied cultural capital as easily malleable” that would be wrong, and part of reviewers’ job is to identify such (often more subtle) errors. However, deciding whether a frame (as opposed to a claim) is right for a given paper–that, I contend, is not the role of reviewing.

    I come to this view because if I’ve seen it once, I’ve seen it hundreds of times: the reviewer thinks Uggen and Manza is really about the coercive power of the state to categorize people, or about the long reach of poverty (because many/most felons experienced poverty prior to incarceration), not about incarceration and disenfranchisement. Certainly, those two ways of extending or considering Uggen and Manza make sense. But, so does the original framing. To indulge reviewers’ read of the paper’s appropriate frame, the authors would have to take out much of the literature they had included and add in a lot of literature they did not include or, worse, add the alternative literature so that what was a tight, compelling story becomes a bloated mess. (NOTE: That did not happen with U&M, I just want a visible example to make the point).

    When it comes to frames, if reviewers can confine their assessment to whether the frame makes sense, then I have no problem, in fact, that would be a good thing. But, all too often, reviewers can’t show the frame doesn’t make sense, they simply assert that the frame they prefer is THE frame. This is a major problem. In my experience, many editors indulge such reviewer assertions.

    So, perhaps the part of my proposal that advocates removing the framing entirely from the review goes too far. Perhaps on that score all that is needed is to ask reviewers to address whether the frame is coherent, and to tell reviewers to avoid offering alternative frames if the submitted frame is coherent. And, editors should simply ignore, and tell authors to ignore, any reviewer framing assertions unless the reviewer successfully shows the existing frame is invalid. Reviewers should of course continue to note any errors of fact. Such an approach would reign in reviewers by encouraging them to evaluate the veracity of claims, and discouraging them from attempting to replace valid frames with their own.

    Like

    Sam

    April 28, 2016 at 3:38 pm

  6. How in the world does it serve the development of knowledge to exclude the findings from peer review? Yes, obviously reviewers should not just be arguing with the findings, but I have reviewed several papers in which the findings did not support the conclusions the paper made or did not entirely reflect the design outlined in the methods. If we are to abolish this sort of review, then there is no sense bothering to review finished papers at all and only a brief summary of the research methods should be provided.

    As for framing–yes, lots of time is wasted arguing about theoretical perspectives. But some papers have confused frames which could be made more clear. Perhaps the issue here is not what the reviewers say but the extent to which the editor requires authors to make changes–if someone is just disputing the frame you have selected, perhaps the best approach would be for the editor to ask the author to write one paragraph explaining the choice of the frame in the revision memo, and as long as it makes sense, be done with it.

    Like

    Mikaila

    April 28, 2016 at 5:10 pm

  7. This reply is largely driven by my recent experience of a peer reviewer asserting in a review of a paper about prisons and gerrymandering that “racial representation is not a right.” That’s factually untrue and was one of 5 reasons given for rejecting the paper (2 others related to that claim questioning our focus on race instead of age or gender). That same paper was also rejected by another journal in which the editor had incorrectly given us comments for an entirely different paper as reasons for rejecting it (and it’s quite possibly my favorite paper I’ve ever written, I think) and in which the reviewers made multiple false statements about the paper (including not citing a piece we cited 5 times and that by focusing on the 12 legislative districts made either too small or too large to be constitutional in our tables we “obscure how much of a problem any of these issues are for the entire state of Pennsylvania”

    The problem Sam identifies is a frustration for me. But the solution is, to me, obvious: editors should be editors. Reviewers are advisors. Sometimes advisors are wrong. Sometimes advisors nit-pick minor aspects of a paper, like its frame, or demand that the authors add analyses not directly related to the topic. Editor need to see and respond to the imperfection in reviewers as well. I had one editor do so in my entire career (Rob Warren at sociology of ed was great) either as a reviewer nor as a submitter.

    Liked by 1 person

    rorykramer

    April 28, 2016 at 10:48 pm

  8. Relevant to this discussion: Strang and Siler, Sociological Theory 2015. http://stx.sagepub.com/content/33/1/71.full.pdf+html (paywalled).

    Short version, for those without access: according to a survey of the authors of *accepted* articles in ASQ published from 2005-2009, their reviewers most often challenged framing, less often challenged methods or results. 54% characterized the reviewers’ requested changes to conceptual or theoretical issues as “major” or “significant;” none got a complete pass.

    Authors had very disparate opinions about whether the forced reframing improved the paper, and/or whether it led to a more conservative paper. Some thought it added a lot of value, others thought it weakened their papers.

    The average number of citations between the first submission and the final paper increased by 26%, from 75 to 96.

    Like

    krippendorf

    April 29, 2016 at 6:19 pm

  9. […] not bother waiting for a response or dealing with chaotic and contradictory reviews. I also think Sam Lucas is onto something when he suggests that we should not allow reviewers to write open ended …. Bottom line: the journal system allows people to do all kinds of bad things, but simple reforms […]

    Like


Comments are closed.

%d bloggers like this: