orgtheory.net

if you aren’t part of the journal review time solution, you’re part of the problem

One of the most frustrating aspects of social science reviewing is the slow review time. Gabriel Rossman says that we are the problem. Rather than focus on what can be easily fixed or provide up or down decisions, reviewers take too long, offer contradictory recommendations, and encourage bloated papers. If I were to summarize Gabriel’s post, I’d say that:

  1. Keep your review short. Don’t write that 6 page single space commentary. One page or so probably enough in most cases
  2. Don’t whine about what the authors should have written about. Evaluate what they actually wrote about.
  3. Be decisive. Yes or no.
  4. Don’t ask for endless citations, commentaries, extra analyses, etc.
  5. All suggestions should be constructive, not busy work.
  6. Let it go: after a while, it becomes counter productive. If you hate, just say so. If you like it, just say so. No more revisions. It’s done.

I also like Gabriel’s suggestion that reviewers should show some spine. In the summer, I was asked to review a 3rd R&R. My entire response was “Dude, seriously? Three R&R’s? Just accept it.” Result: paper accepted.

The True Word: From Black Power/Grad Skool Rulz

 

Written by fabiorojas

November 22, 2013 at 12:01 am

Posted in academia, fabio

15 Responses

Subscribe to comments with RSS.

  1. Gabe has clearly had some of the same reviewers as me.

    We’ll see if peer review works, but personally I don’t think the system is broken. 3 R&Rs is the editor’s problem, not the reviewers. Once can always read something and notice things that could be fixed – editors are the ones who need to make that decision, in context with the journal mission and editorial goals.

    We might want to have shorter review cycles – I can’t be the only person who waits until close to the given deadline to turn something around. Likewise, better tracking of reviewer “asks” and performance. As with AOM, journals could identify cranky reviewers and then systematically reduce how often they are asked to review. Adding new reviewers after 1st R&R might also be a productive area to address.

    Like

    Erica

    November 22, 2013 at 2:11 am

  2. A new working paper by David Strang suggests that reviewers may be disproportionately focusing on “framing” and “theory” because they’re more accessible to the non-author (the reviewer!) than the research design & analytical methods stuff. (This is my extension of his work and so I don’t want to implicate David in what I say next.)

    My experience & that of others I’ve spoken with is that the empirical observations/results of most papers rarely change much in the reviewing process. Why? Perhaps b/c reviewers are chosen for their topic expertise & thus don’t feel qualified — or are NOT? — to dig deeply into incompletely described designs/methods?

    Big leap: how many unsound empirical observations are floating around out there?!

    Like

    Howard Aldrich

    November 22, 2013 at 2:42 pm

  3. I am regularly surprised that out of four reviewers mine was the only review to critique methods. Usually it takes a bleeding edge or controversial methodology (HAPC models, LCA) to get more than reviewer commenting on method. I find this unfortunate, because whether or not an author is using fancy methods, simple things like variable coding and basic sensitivity analyses can mean the difference between significance and null results. Poking at significance should be job 1 (are these findings representing something “real”, or likely to be due to chance or researcher choices)? Interestingness or complexity of theoretical story should only be relevant if the findings themselves hold up to scrutiny.

    Like

    jmir

    November 22, 2013 at 3:20 pm

  4. @Erica: Everybody has a problem – editors allow reviewers to make insane suggestions; reviewers will pick on absurd issues and make irrelevant suggestions; we sit on reviews forever.

    @jimi – This has to do with the “free style” reviewing. If we directed people, we could get different results.

    Like

    fabiorojas

    November 22, 2013 at 3:52 pm

  5. So changing the questions asked of reviewers for empirical papers: (1) Do the results appear to be correct? Are the research and analysis methods properly executed. (Instruction to authors: you may provide methodological appendices for the reviewers about all the supplementary analyses you did to verify the results.) The trickiest elements here are omitted variable bias or sample selection bias or model specification, in which the reviewer believes that doing the analysis differently would change the results. You can wonder into the weeds of endless hypotheticals at this point, but still asking the reviewers to focus on the issue of correctness ought to help. (2) Is the link between the results and the research questions/conclusions correct and properly substantiated? I.e. do the data answer the question they are purported to answer? (3) Is the relevance of the results in relation to other similar work correctly explained? (4) What is the theoretical significance of the work? (confirming existing theory? contradicting existing theory? developing new theory?) (4) Is the link between data and theory properly explained? Are the data actually relevant to the theory? These are the most important criteria for meriting publication. Clarity of writing will influence the extent to which these criteria are manifest in the article.

    Two other criteria may justly affect WHERE a piece is published: (1) “Importance” of paper — likelihood that it will either lead to major rethinking within a subfield or have a broad impact across subfield. (2) Good writing, esthetics.

    The criteria for a theoretical contribution need to be different, but this would seem to be a start.

    Like

    olderwoman

    November 22, 2013 at 4:47 pm

  6. @o.w.: what you have outline is essentially the review form for PLoS One? Mostly technical. The only difference is that journals would take importance and aesthetics into account. At the very least, you’ll probably get more focused reviews.

    Like

    fabiorojas

    November 22, 2013 at 4:51 pm

  7. FWIW, I’m not nearly as bothered by the time it takes the reviewers to get back to me as I am by the time it takes me to get back to them. At least while I’m waiting to hear back I can be doing something productive, which isn’t true of the time I spend writing response memos or reading tangentially related citations. (btw, to the reviewer who berated me for not citing an article but got the author’s name wrong, this Bud’s for you).

    Also, I think Howard Aldrich is really onto something. There’s a presumption that the reviewer will say something and if you can’t admit that you don’t really understand how ergm works then the only thing you really can say is “Hey, I’ve heard of Ron Burt and know he has something to do with networks, why aren’t you citing him”?

    Like

    gabrielrossman

    November 22, 2013 at 6:01 pm

  8. I think there are a lot of different kinds of problems in the review process. Some editors send just about everything out for review unless the subject matter is totally inappropriate for the journal. I always try to provide constructive criticism, but I get quite a few papers for which it is difficult to figure out what to say, because the paper lacks a clear argument, or doesn’t pose a central issue or question. Right now, I’m trying to figure out what to say about a paper that’s fairly incoherent, but obviously written by a non-native English speaker. This seems like it should be the editor’s job, not mine. An entirely different problem is reviews that are too short and superficial. I’ve gotten these (sometimes one paragraph saying something like, this is a good paper and here’s a minor suggestion). This is not very helpful for revisions.

    Like

    soctraveller

    November 22, 2013 at 7:53 pm

  9. “An entirely different problem is reviews that are too short and superficial. I’ve gotten these (sometimes one paragraph saying something like, this is a good paper and here’s a minor suggestion). This is not very helpful for revisions.”
    Yikes! It is statements like that are exactly what the problem is. Good grief. Here is this whole thread about turn around time and editors not making decisions and you are complaining because reviewers liked your paper and thought it should be published with only minor revisions. You seem to be failing to distinguish between an article reviewer and your advisor or professional colleagues. If you had complained that editors discount positive reviews and refuse to publish articles when reviewers just give short reviews and say the article is ready to go, then I’d be with you. But no, you think somebody volunteering their time as a reviewer owes you several pages of prose about “suggestions for revision” even when they don’t actually have any suggestions.

    Like

    olderwoman

    November 22, 2013 at 9:06 pm

  10. Well, I have gotten these reviews along with at least one other review that suggested very significant revisions (and I knew there were issues with the paper as well). So that makes me think the one paragraph person was simply not paying much attention. Also, I have seen this now from the editorial side — sending out papers that I think need a lot of work but have potential, and getting back reviews suggesting only minor changes.

    Like

    soctraveller

    November 22, 2013 at 9:24 pm

  11. My point exactly. That reviewer thought the paper was fine, but (a) you say you sent a paper out for review knowing it was weak but wanting to get reviewers to tell you how to revise the paper, and (b) you are adhering to the standard that all good reviewers suggest revisions. That’s the problem. If the author thinks the paper still needs revisions, the author ought not to be bothering reviewers with it. It’s the model that EXPECTS reviewer suggestions for revisions and thus EXPECTS that half-baked papers will get cleaned up in the review process that is causing all this trouble. Under this system, authors rationally don’t bother to finish papers before mailing them and view their lousy $10 submission fee as payment for “high quality reviews.” Ugh.

    The people who like this system see themselves as reviewers who mentor papers. I’d rather see this process happen in another way. It is pretty clear that our current reviewer system with its expectation that reviewers will give authors constructive suggestions instead of just saying whether the paper is any good or not is close to the breaking point.

    Like

    olderwoman

    November 22, 2013 at 11:55 pm

  12. “Olderwman” [I know who you really are!] makes a great point — I’ve heard many junior scholars talk about sending a paper to ASR or ASQ first, so that they’ll get “good reviews to help me revise it” Clearly, they’re “using” the review process in the way Olderwoman describes, and in a way that clogs up the process & burns out reviewers.

    Shouldn’t people in need of constructive feedback try their friends & colleagues first?

    Like

    Howard Aldrich

    November 23, 2013 at 3:19 am

  13. I agree with olderwoman and Howard. I have advisors tell me never to send a paper out for review if (a) you think it needs more work, and (b) before anyone else reads it. As a result, I only submit papers for review after I feel that I’ve done all I can. However, reviewers also need to suck it up and state whether the paper is good enough for publication or not. I feel reviewers sometimes use R&R as a way to avoid making decisions – too nice to reject it but not confident enough to recommend acceptance or conditional acceptance. I find this ironic because it is their professional opinion that editors are seeking – do you think this is a manuscript that is publishable? R&R appears to be the norm, not because there is an abundance of excellent papers that require revising before it is ready for publication, but mostly because reviewers prefer this middle ground.

    Like

    SocGrad

    November 23, 2013 at 6:49 am

  14. I don’t think we should submit obviously terrible papers to journals, but I do wonder where this supposed abundance of peer and advisor time to review papers is coming from. Sure, in the best of worlds, we want to get a lot of eyes-on help before submission, but in practice, I’m always really impressed by colleagues who turn around good comments in a timely fashion. That’s because it is rare, in my experience, especially for a longer manuscript.

    My personal workaround is to present at seminars and conferences, maybe schedule a meeting to “discuss” which forces the issue. But I can certainly imagine authors asking themselves whether comments from an advisor or friend are really worth waiting 2-3 months to get, and will I have to nag this person and potentially damage the relationship, and if I keep working on the thing in the interim, will the comments even be relevant, or is this just a waste of everyone’s time? You’re almost better off asking someone to coauthor, because at least then they’d feel obligated.

    In contrast, reviewers are obligated to do something, and with blind review, they will actually point out the holes in your paper.

    Hm, maybe this is the lurking problem? Lack of time for collegial review? Or maybe I’m not in the loop here and there’s some strategy to get feedback on papers that I just do not know about?

    Like

    Erica

    November 25, 2013 at 4:49 am

  15. Erica, good point. I have a strategy I often suggest to grad students & junior colleagues; invoke the norm of reciprocity! How? Locate people who are doing stuff that you’re very interested in — can be local, can be anywhere — and volunteer to read THEIR stuff. Give them the kind of feedback that you want for yourself. In other words, show them what you’re looking for.

    I tell them, “never use ‘track changes,’ that’s for co-authors. Use ‘insert comment’ instead.”

    Having experienced such excellent feedback from you, it would be a churlish person indeed who refused a subsequent request.

    Like

    Howard Aldrich

    November 25, 2013 at 2:10 pm


Comments are closed.

%d bloggers like this: