so, did asr actually change anything?
A few weeks ago, I expressed dismay at the multiple R&R, multi-year revision process that now takes place at our flagship journal. I picked on them in particular, but it’s really a demand for all journals (inc. AJS, SF, SP) in general to stabilize the review process and adopt some concrete rules. You should only R&R if you think there’s a reasonable chance of success. You really shouldn’t assign new reviewers in most cases. And, please, cut the multi-R&Rs unless it is a de-facto admission that a manuscript will almost certainly be published. This is the norm in economics – many R&R’s, but the R&R means that the paper will be published.
So my question is this: is there any sign at all this was taken seriously? I recently was asked to review at ASR and I expressed my concerns. I got a polite email back, but little indication otherwise. I agreed to review the paper (1st R&R and I was an original reviewer) but warned that I will not participate in 2nd or 3rd R&Rs.
I spent relatively little time at ASA, so I don’t know what people thought about this issue, or if our editors are thinking about getting control over the process. Your thoughts? What is the buzz on the street?
Adverts: From Black Power/Grad Skool Rulz
orgtheory.net 
I am not sure of the accuracy of your allegation. It may be so in some cases. I did speak to an editor and described the situation of multiple edits and she stated that it sounded unusual. I also noticed that the article by Andrew Papachristos in the June 2013 ASR and the talk he gave at the ASA conference, or rather, the talk let his colleague give, were related; but it is not evident that there is a significant time gap between submission and publication.
LikeLike
Fred Welfare
August 21, 2013 at 2:56 am
Check the comments from the post. ASR’s own published stats indicate a huge increase in R&Rs, while keeping the same final rate of publication. Not good. Means a lot of R&R papers don’t have a chance.
I’ve also gotten two emails from senior scholars about this issue. One complained about being reviewer number 4 on an R&R. A similar complaint from the 2nd. This is excessive.
LikeLike
fabiorojas
August 21, 2013 at 3:01 am
The brouhaha led to someone from AJS (not ASR) to reach out to me to say that they were taking the issue very seriously.
Nobody spoke to me from ASR. Regarding ASR, I’m actually planning to prepare a complaint about a recent editorial decision that I consider to be outrageous. The short version is that after stalling the paper for four years with four R&Rs and eight different reviewers and receiving positive reviews in the last round, the editor rejected the article on the grounds that in the mean time someone else had published a similar finding in a different — much less prestigious — journal that had not demanded the large number of alternate analyses of the data requested by the eight ASR reviewers. There is no question that the article given the run-around by ASR was in fact the first article to address the question (it actually won quite a few “best paper” awards the year it was first written) and that it is the other now-published papers that are derivative. And there is suggestive evidence that the editor’s decision was prompted by self-serving “cite me” claims by some of the last-round reviewers.
LikeLike
olderwoman
August 21, 2013 at 3:14 am
olderwoman, i am deeply sympathetic. that’s insane and should not have happened!
LikeLike
OS
August 21, 2013 at 4:23 am
Four R&Rs is three too many. 8 reviews is at least 4 too many.
I like the standard of one R&R followed by a conditional accept or a reject. And 2-3 reviewers on the first round, followed by no more than one more on the next with no new ones to follow.
LikeLike
cwalken
August 21, 2013 at 5:40 am
N=1, but I just had an experience @ASR that met cwalken’s standard. But, the second round reviews were all very positive and very short, so maybe the outcome would have been the same in the absence of the recent brouhaha over R&Rs.
Let’s not all lay this on the editors, though. Reviewers are part of the problem, for recommending too many R&Rs and, in some cases, being unreasonably picky. I’ve tried to be more cautious about recommending R&R to the papers I review. So far in August, I’ve recommended 3 rejections and 1 conditional accept; last year, I probably would have recommended 2 rejections, a weak R&R, and a conditional accept to the same papers.)
Maybe I’ll gain a reputation as a cranky reviewer, and editors won’t send me as many papers anymore.
LikeLiked by 1 person
lurker
August 21, 2013 at 12:25 pm
lurker: I disagree. It IS the editor’s fault. There is no such thing as a perfect paper and no such thing as a good reviewer who does not offer suggestions for revisions. Reviewers who say “the paper is good, print as is” are considered to be lazy reviewers, no matter how good the paper is. The fault lies with editors who keep adding reviewers at each iteration of an R&R and with editors who refuse to adjudicate between conflicting reviews, just telling the author to please everyone. I agree that not all reviewers are good: that is part of the editorial justification for asking for additional reviews. But if an editor lets “picky” comments dominate the review process, it is the editor’s fault. Journal editors should be senior scholars who can be trusted to make judgments, not bureaucrats who move paper around.
LikeLiked by 1 person
olderwoman
August 21, 2013 at 2:22 pm
The protracted R&R system only works in economics (among other reasons) because it has robust working paper series in which articlces of consequence have already been read by people who care by the time they appear in print. ASA has actively discouraged this with ASR, including asking people to take down working paper versions of papers after they have been accepted by ASR (I don’t know where they currently stand on the practice).
LikeLike
jeremy
August 21, 2013 at 2:48 pm
Isn’t a better solution just to change our culture in sociology? At the risk of “committing sociology” my reading of this blog and others suggests that the culture of our discipline, at least among some disciplinary elites, seems to basically say that you have to get published in AJS or ASR to be a “good” scholar or to even keep your job. By privileging these journals we are creating a system that gives a great deal of power to a handful of editors. Maybe it would be better to change the system and change our culture. As far as I can tell, our current system/ culture gives the editorial board of major journals no incentive to “do right”. Just a thought…….
LikeLike
Silly Wabbit
August 21, 2013 at 3:18 pm
But I would point out that this issue is not limited to AJS/ASR and the prestige they command. I had 3 R&Rs with a manuscript at Teaching Sociology, on a paper that ultimately was rejected. I am not necessarily opposed to multiple R&Rs if the later stages are basically “Conditional Acceptance pending successful completion of revisions,” but otherwise it seems inappropriate.
LikeLike
socteach
August 21, 2013 at 4:09 pm
Socteach,
Thx for the comment. I can’t speak to the multiple R&R but the long review times are definitely not confined to the top journals in my limited experience…..waiting about 8 months on one paper currently…..
LikeLike
Silly Wabbit
August 21, 2013 at 5:23 pm
I will add my voice of complaint. Too many R&Rs — which, besides slowing down authors and the progress of the discipline, burns out reviewers, making it harder for all journals to get first-round reviews…. And, as far as I can tell, as a reviewer and a reader, this process has NOT improved the average quality of the papers that appear in ASR….. An initial R&R should be a conditional acceptance based on meeting the central concerns of the first reviewers. An additional reviewer should be used just to make sure that no real howler of an error gets through. The end. Surely, some clunkers will get published; I doubt that the number will be greater than are getting published now.
LikeLike
Claude Fischer
August 21, 2013 at 5:29 pm
OW: I don’t think we’re that far apart. I don’t see a conditional accept as saying, “publish as is,” but, “it’s a strong enough paper that if you do X, the paper should be published so that it can be vetted by the scholarly community.” (Recognizing that standards for “strong enough” may differ across outlets.)
What I see, though, is that too many reviewers hold papers to the standard of “perfect” as opposed to “the best that can be done with the available data and better than anything else that’s out there,” but are also reluctant to recommend an outright rejection.
Yes, you could say that it’s the editor’s job to read every paper closely enough to be able to differentiate essential from non-essential suggestions in an R&R, and to squash “serial R&R” reviewers. I agree that editors should do more of this. But, I don’t see that it’s realistic to ask them to carry all of this burden. As a reviewer, if 80%-90% of your reviews over the long haul recommend an R&R, you’re not doing your job of helping editors separate the wheat from the chaff.
LikeLike
lurker
August 21, 2013 at 6:44 pm
I already posted this on older woman’s blog, but here it is for those of you who don’t read that as often as you read orgtheory.net…
I sent this screed to the editors of ASR when I submitted a review to them, and I will send the same message to any other sociology journal for which I review:
********
“I am willing to review a revision of this manuscript – but only on the condition that you do not send the paper to new reviewers on the second round. I’m fed up with the norm in sociology journals of adding new reviewers on subsequent rounds of review.
When you editors bring in new reviewers on subsequent rounds (you are by no means alone in doing this, but you’re my target right now because it’s you I’m reviewing for), you violate the implicit contract that you create between authors and reviewers when you assign reviewers to a paper and transmit the reviews to authors for their consideration and response. I take the reviews of my own papers very seriously, just as I take the reviews that I write for journals very seriously. Bringing in new reviewers on the second round (or third or fourth…) disrespects both parties to the implicit contract you created when you selected reviewers for the first round.
And every time you bring in new reviewers, you doom authors to new critiques of their papers. Sociology is a low-paradigm discipline, so different reviewers are bound to have different perspectives on any paper they read. Consider this analysis of a nearby discipline, psychology. In a 1990 article in American Psychologist, titled “But the reviewers are making different criticisms of my paper!”, Donald Fiske and Louis Fogg analyzed reviews of papers in psych journals and found very, very, very low correspondences between pairs of reviews in terms of the types of criticisms raised; they also found very, very, very low associations between pairs of reviews in terms of the overall evaluation (positive, neutral, or negative) of the article. Given the high probability that different reviewers will find different things to love and hate in a paper, it’s not at all surprising that the new reviewers who are brought on during the second round of review raise entirely novel concerns. This is what leads to the endless treadmill that many of us feel we’ve been on with some journals lately – including the journal you edit.
So, to sum up, I am willing to review this paper on a second round, but only if you do not bring in new reviewers – unless one of the original reviewers dies or refuses to do the review, and you feel you absolutely must have someone with his/her expertise look over the paper.”
*******
I urge all of you to take action with the editors of the journals for which you review. We can try collective action next August — storm the editorial board meetings & demand a change in review process — but that’s a long way off. In the meantime, we can make our voices heard loudly & clearly.
Onward & upward!
h2
LikeLike
Heather A. Haveman
August 21, 2013 at 7:07 pm
Much like Heather, I have also taken to writing rather vituperative notes to the editors demanding a final decision when I am a reviewer on an overly protracted review process.
And I agree with OW that editors tend to ignore brisk “it’s fine, print it” reviews which is why the last time I saw an excellent paper get a second round R&R in the face of intractably split reviewers I wrote a thousand word argument as to why exactly the article was so compelling and the other reviewers were mistaken. I wouldn’t have bothered to spend several hours doing so if I thought the editors would take seriously a simple thumbs up review. And I feel this time was well-spent since the second R&R proved to be contentious but the excellent paper was ultimately published whereas it probably would not have been if I just said “+1.” (Not that I deserve more credit than the authors or anything but I am proud of having helped midwife the paper).
As for Lurker’s point, I have a post I’ve been drafting for my own blog on how to be a serious reviewer without being a “this isn’t the paper I wouldn’t have written so try again” asshole. I’ll probably post it in a few weeks when Sociological Science starts accepting manuscripts.
LikeLike
gabrielrossman
August 21, 2013 at 8:59 pm
It may be more of a problem “top” journals, because reviewers might think, “This is good, but to be ASR material it’s gotta have 2 more data layers and another year of fieldwork.” This is one problem with the prestige system we are mired in – wasting time and effort and feeding inaccurate quality signals for hiring and promotion.
If the associations published high volumes of good-enough papers and let people judge their quality by reading them (and citations), it would be much better.
LikeLike
Philip N. Cohen
August 22, 2013 at 1:13 am
If only there were a high-quality journal friendly to organizational sociology with a mean time to first decision of 39 days, a policy of “no new reviewers, no third-round R&Rs,” a track record of getting excellent papers from initial submission through revisions to print in under a year, and excellent visibility in the press.
[OK, seriously, I’m obligated to say this, but still. Our handling editors are not shy about making decisions.]
LikeLike
jerrydavisumich
August 22, 2013 at 1:13 am
Jerry,
Not to be a pain in the ass or complain about you moving in the right direction, but is there a reason you allow two R&Rs instead of capping it at one R&R only? (Let’s take it as granted that the Sociological Science policy of 0 R&Rs would be radical for an established journal but 1 R&R seems more plausible).
(FWIW, my few review experiences w ASQ have been positive).
LikeLike
gabrielrossman
August 22, 2013 at 2:04 am
Gabriel: ASQ used to be a 1 R&R cap journal. As an author who submitted under that system, I found that my work was R&R’ed and then rejected after a new reviewer added more requirements for publication. Maybe 2 R&Rs are needed for papers with promise but need some work. Perhaps sticking with the same reviewers would allow a 1 R&R system since you wouldn’t add new requirements.
LikeLiked by 1 person
fabiorojas
August 22, 2013 at 2:08 am
Gabriel,
Fabio is correct that ASQ had a long-standing stated policy of “one round up or out.” I looked into the stats carefully when I started, and it turned out that papers essentially never made it in after only one revision. We handling editors would say “I’m going to make a rare exception to our policy in this case…” but in practice two rounds was the norm. (The second round, if there is one, is typically much less extensive than the first, and it almost never entails bringing in new reviewers, which I personally think is silly.)
The papers that end up being the biggest contributions typically come in pretty hairy, and require a couple of rounds to be readable as a stand-alone article. One round of review might work if one is simply reporting the results of a straightforward experiment or regression, which is perhaps what Soc Science or PLOSOne aim for. I can’t see it working for papers at the vanguard, which is how I like to think of ASQ. These require thoughtful reviewers and editors willing to put in some heavy lifting.
LikeLike
jerrydavisumich
August 22, 2013 at 2:28 am
Jerry and Fabio,
Thanks for the clarification (and for barring new reviewers).
LikeLike
gabrielrossman
August 22, 2013 at 2:42 am
Gotta be honest, Jerry. On the one hand, I feel reassured that you are taking things in a new direction. ASQ is in good hands. On the other hand, knowing that other authors routinely got the “.. in your case, we’ll make an exception” treatment makes me, well, feel not so great. I think we’re having a good conversation.
LikeLike
fabiorojas
August 22, 2013 at 3:00 am
Jerry,
what is the empirical basis for your claim that the papers that have the greatest ultimate impact require multiple rounds of revision and heavy lifting by editors? Seems like it would be hard for you to know the counter-factual, and that all kinds of psychological biases would drive such a belief by editors.
At Sociological Science we are open to the possibility that this is not true, and that authors are capable of writing good and impactful papers without the help of the editors. Even papers that do more than report the results of a straightforward experiment or regression. You may wish to check our website http://www.sociologicalscience.com if you are concerned that these are the kinds of papers we are “aiming” for. We are certainly not.
To that end Sociological Science would welcome your submission substantiating your claim that ultimate impact == heavy lifting by editors. :-)
LikeLike
Jesper Sorensen (@sorensenjesperb)
August 22, 2013 at 3:14 am
Fabio, you should feel fine: we clarified the policy years ago, and the “exceptions” were pretty much every paper that got a second R&R. It was not a form of special treatment, just a pragmatic recognition that one round of revision was rarely enough to polish worthy papers into gems. Note that in 2012 50% of the papers published in ASQ were by assistant professor or doctoral students. Junior authors often have the most innovative ideas, but less experience in the review process; therefore, it’s not a surprise that it may take a couple of rounds to achieve nirvana.
LikeLike
jerrydavisumich
August 22, 2013 at 3:18 am
Jerry: If you take me to Nirvana, I’ll eat less meat.
LikeLike
fabiorojas
August 22, 2013 at 3:20 am
Jesper–
Sold! I will submit, one way or another, to Sociological Science. The empirical basis for my claim is the very high proportion of papers that come to be seen as classics that came out of dissertations (e.g., by Morten Hansen, Amy Edmondson, Gautam Ahuja, Jenny Chatman, David Obstfeld, Ezra Zuckerman, and–NPV–Ethan Bernstein, whose recent paper won the best publication from BOTH the OB and OMT divisions).
Your experience my be different from mine, but ambitious dissertations rarely fit with a “yes/no” decision. They benefit from thoughtful guidance. Similarly with early career work, or innovative big projects. But I am happy to be proved wrong, and discover that “revisions” are a giant waste of time.
LikeLike
jerrydavisumich
August 22, 2013 at 3:30 am
For every dissertation or good paper that’s melded into a great paper, how many great papers are turned into merely good papers through the review process (which, as others have noted, is inherently conservative)? How many wind up being less coherent, or at least no more coherent, than the original as authors try to satisfy reviewers with nearly orthogonal tastes and recommendations? How many good papers never see the light of day?
For every superstar-in-the-making whose work is improved by developmental reviews, how many established superstars don’t bother to send their best work to the flagship US journals because they think their time is better spent producing new research than responding to reviewers? (I recognize that this cost::benefit assessment changes if there is a student or junior co-author who needs the imprimatur of AJS/ASR.)
These are not intended to be rhetorical questions, even if we lack the data to answer them.
LikeLike
kweeden
August 22, 2013 at 12:01 pm
Consider also submitting to Social Currents, the new journal of the Southern Sociological Society and published by Sage (first issue February 2014)!!!
Manuscript submissions are being accepted at http://mc.manuscriptcentral.com/scu
The journal’s editor’s and editorial board are very committed to very timely reviews and review process, clear and consistent decisionmaking, and openness to a variety of theoretical and methodological orientations.
The uniqueness of Social Currents lies in its format: the front end of every issue is devoted to short, theoretical agenda-setting contributions and short empirical and policy-related pieces, ranging anywhere from 1,500 to 4,000 words. The back end of every issue includes normal journal length articles (7,500-12,000 words) that branch across subfields, including the many specialties of sociology and the social sciences in general.
LikeLike
V_Roscigno
August 22, 2013 at 1:09 pm
i just had a paper that went through r&r at asr and sf. at both, the reviews were mostly positive with criticisms, of course. we addressed *every* comment the reviewers made, most of which were pretty much spot on. at both, they added new reviewers, and at asr, one of the original reviewers then criticized us for not changing stuff s/he hadn’t even commented on the first time. and in both cases, the editors, instead of saying hey, you addressed everything that’s great, said sorry, but the reviewers… i mean, you’re editors. make executive decisions. say i/we think it’s great or we find it’s just not up to snuff. don’t hide behind what the reviewers say. if it’s not good enough for you now, it’s likely it wasn’t good enough when it was first submitted. three years this paper’s been languishing. it’s a good historical piece now.
maybe it’s better to get rid of the r&r altogether and go with reject/conditional accept.
LikeLike
syed1ali
August 22, 2013 at 2:19 pm
I have to agree with Weeden about conservatism in the review process. My most innovative paper was rejected by ASR, AJS and was later published in a specialty journal and ended up winning an article award. A paper with a similar framework got rejected by Mobilization, Sociological Theory, and now has a favorable R&R from another top specialty journal. On the other side, my straightforward regression model papers have been readily accepted at top specialty places and SF.
So either I’m a better normal scientist than original thinker (in which case the award is an oversight on the committee’s part), or the review process tends to select for thinking inside the box.
LikeLike
cwalken
August 22, 2013 at 10:58 pm
Since my name was taken in vain, a few notes on this:
* I have had good experiences with multiple R&Rs and bad experiences with multiple R&Rs, both as author and as reviewer. So I am willing to believe that there is a role for journals that do multiple rounds. Though I think there is also a role for journals that do away with the R&R and the costs that it imposes. That is one reason I am part of Soc Sci. Let 1,000 flowers bloom (but let it not be shortly followed by a massive purge of intellectuals; see under Mao).
* Jerry: surely as editor of ASQ, you don’t want to suggest that the way we can tell that multiple R&Rs is good for papers is by pointing to papers that did make it through? I know some pretty darned innovative papers that ASQ rejected; those need to be considered too. And while I’d love to think that papers that win multiple prizes are always so fabulous, check out this book review (http://web.mit.edu/ewzucker/www/Review_of_Spillman_Solidarity_in_Strategy.pdf) of a book that won two ASA section book prizes this past summer. (of course, it could be that AOM prize committees are better judges of research quality than ASA committees…) So let’s admit that we have no evidence to support the hunch that multiple rounds is a good policy. But I’m willing to believe that it is good for some journals, including ASQ.
*I have had good experiences and bad experiences with the ASQ review process over the years. My favorite moment was in acceptance letter on a paper for which (as reviewer) I vehemently recommended rejection, when the editor started off the letter [an acceptance letter!] by saying “Not everyone who has been associated with this manuscript will be happy about the outcome.” (I wonder if the authors put an asterisk on their vitas– yes, this paper was accepted, but there is a reviewer out there who knows the paper is crap, and so did the editor actually….)
* Looking back at the 2000 paper to which Jerry refers, it is very hard to say whether:
(a) the paper was really so innovative [Joel Podolny and I once had a conversation in which we agreed that a crucial reason our dissertation papers did so well was not because they were so great but because they told people what they wanted to hear about the importance of identity in markets; I think I have written more innovative papers since, but they have generally not done as well—I think because they were less easy on reviewers’ and readers’ ears. Of course, that is just my opinion. Another possibility is that I peaked ridiculously early…];
(b) the paper was helped by the multiple rounds. For fun, I posted the original submission (see: http://web.mit.edu/ewzucker/www/FCP898.pdf) and the second-round submission (see: http://web.mit.edu/ewzucker/www/FCP799.pdf), which can be compared with the final draft (see your friendly neighborhood jstor). Does the paper evolve in the way that Jerry describes? Maybe. You certainly can see the framing change in interesting ways. And there is some change in the analysis, some of which I think is an improvement. You can also see me overcorrect in my attempt to satisfy reviewers by trying to adopt a feature of the ASQ style that I have never liked (the laundry list of hypotheses that often takes the place of theory in orgs/management journals) and then figure out a way of doing it that worked better, both for me and reviewers. You can also see me overcorrect in trying to answer micro-OB type critics who challenged me on why conglomerates can’t use impression management tactics to address pressure on them to engage in de-diversification (this response shows up in the new framing in the second round and then migrates to the discussion section in the final draft). Is this a process that follows the path that Jerry described? Perhaps to some extent. I think if you look at the first draft though, it’s hard to rule out the possibility that the paper would have had the same level of impact if had never even gone through the review process. And it certainly would have had a more unique voice had that happened. The clearest thing that can be said is that the process made the paper into more of an ASQ-style paper, for better or worse.
LikeLike
ezrazuckerman
August 23, 2013 at 3:31 am
Ezra: Thanks for the note (essay?). The issue for most folks, including myself, isn’t that we oppose multi R&Rs, or even long multi-R&Rs, We oppose multi R&Rs that take years and end up being rejected anyway. Read o.w.’s horror story, where a paper was held up so long that it was rejected because the idea was published by someone else in the mean time. Wrecks careers. Seriously.
LikeLike
fabiorojas
August 23, 2013 at 3:41 am
Andy Abbott had two interesting things to say at the AJS consulting editors’ lunch in NYC that are relevant to this discussion. I am not necessarily endorsing either, but think they’re interesting points:
The idea that a publication in one of the discipline’s top journals is a sine qua non for employment in the discipline is weird. Why should we expect research carried out by a second- or third-year graduate student to be competitive with the pinnacle of the field? The fact that this has become an expectation[1] puts additional pressure on the top journals both because many not-ready-for-prime-time manuscripts are submitted and because they are thrust into the role of gatekeeper for the assistant-professor job market.
The editorial decision to publish a paper should ultimately be based on the answer to the question: “should this paper be published here?”. That’s a very different question from “has the author responded adequately to the reviewers’ concerns?”. It’s a mistake to understand the essential “contract” to be between the journal and the author; the journal has a separate, and probably more important, commitment to the discipline and its readers.[2]
[1] I am not convinced that this expectation is actually true, but many job market participants believe that it is.
[2] I have personally been the victim of this situation (responding to everything, then having the editor say they wouldn’t publish it). It was extremely frustrating and demoralizing. But the practice may nevertheless be good for the journal and the discipline.
LikeLike
andrewperrin
August 23, 2013 at 5:10 pm
on point 2, if it’s a matter if a paper is good/right fit for the journal or not, the editor should sense that when the paper is originally submitted, right? if it make it past the editor to reviewers, the answer should in the vast majority of cases be yes (allowing that sometimes an editor may let a paper go to review then decide s/he was initially mistaken. that’s acceptable to a degree.) but once you’re at r&r, even with the disclaimer that they may not decide to publish it (reasonable), if you revise according to what the reviewers have said to do, which i imagine the bulk of authors do (or maybe they don’t?), then it’s reasonable to expect a favorable decision. i had one paper at a well-known specialty journal where i did everything asked and sent back a detailed memo showing how i addressed everything and the editor immediately wrote back saying they were going to publish it. that’s how it really should be.
LikeLike
syed1ali
August 23, 2013 at 5:31 pm
We’ve jumped the track on the ASR discussion a bit, but I’m game.
Ezra: The editor of ASQ is highly familiar with the hazards of sampling on the dependent variable. Jesper had asked “what is the empirical basis for your claim that the papers that have the greatest ultimate impact require multiple rounds of revision and heavy lifting by editors?” One indicator of ultimate impact is lots of awards and cites, and by this indicator, one sees a greatly disproportionate number of dissertation papers. Cites and awards are imperfect measures (and one of them is getting much worse over time due to unscrupulous journal practices); I’m open to alternatives. I would be hesitant, however, to rely on authors to select which of their own papers was most astounding (and unfairly overlooked), for the same reason that I would not trust parents to judge how adorable their children really were.
It’s reasonable to imagine that authors get better with experience at managing the review process and writing with readers in mind. After 10 or 20 or 30 years in the field, the article format becomes highly familiar, as do the routine objections of reviewers, so one writes with this in mind. This might well have a leveling effect on the kinds of submissions one sees from more senior people. Conversely, those at very early career stages might be full of innovative ideas and intriguing new methods, some of which end up in their dissertation, but have relatively limited experience in turning those into a 40-page journal submission. These papers typically benefit from an involved review process.
A process like this would yield what I described: the most “classic” published papers coming disproportionately from early career work. Note that MY sample is not just the papers that end up in print, but roughly 50 manuscripts per year, going several years. Admittedly, this is only the papers that got submitted to ASQ, but it is hard to think of a sampling frame that could overcome this.
Ezra’s point about different journals for different purposes is an important one. Thomson-Reuters now indexes 172 journals in “Management,” compared to just 60 ten years ago. The barriers to entry for starting a journal are trivial now that no one expects to see an actual physical journal, and within a few years everyone will have their own journal. Some journals publish 300-400 articles per year (e.g., “Journal of Business Ethics,” inexplicably included in the FT45 list, publishes 28 issues per year). There is a journal outlet for every conceivable type of paper; no stray thought need ever go unpublished, and no brilliantly innovative argument shot down by The Man at ASR need stay in that proverbial filing cabinet.
In this context, there is plenty of room for a “pass/fail” journal. (One might add “and it’s called PLOSOne.”) If one imagines papers to have an intrinsic level of merit that skilled editors and reviewers can identify and judge, then this is a good model. On the other hand, if papers are part of a conversation in a field, then interacting with skilled interlocutors (reviewers and editors) might help make them more persuasive and well-grounded in contemporary debates, and therefore more likely to reach their audience. That’s where I think ASQ shines. (And thanks to Ezra for sharing the various iterations of his classic 2000 article, to help readers judge whether the review process was closer to sculpting or foot-binding.)
LikeLike
jerrydavisumich
August 23, 2013 at 5:33 pm
Fabio,
Speak for yourself. Obviously the worst thing career-wise is the multiple R&R followed by a reject. However there’s also the converse problem of editors procrastinating on a reject and the accepting out of guilt. (I was recently willing to accept a paper on the second round even though I fundamentally don’t trust the paper in my gut because I felt guilty about the authors enduring multiple R&Rs and that I was put in the structural asshole position of “new reviewer”).
Aside from the rejected (or accepted) on the third round problem, there’s still lots to hate about R&Rs. Kim and Ezra (and probably a few other people) have raised the “too many cooks spoils the soup” problem of anonymous co-authorship. And then there’s simply the fact that R&Rs are a huge time sink that we could be devoting to new work. I spent the entire 2012-13 academic year on two R&Rs without doing any new work at all. In the case of one of the two papers I feel the process materially improved the paper but in the case of the other not so much and overall I’d rather be doing new work than tweaking my old work.
Anyway, the incident OW describes is especially egregious as kind of inversion of the patricide begging for clemency on grounds of being an orphan, but in the counterfactual world where editors treated R&R2+ as basically a conditional accept on the installment plan, there’d still be a lot of problems with protracted and intrusive R&Rs.
As I’ll be articulating shortly at length on my own blog, the real problem with R&Rs is not “hey, we had a deal!” but “maybe that’s the paper you would have written but it’s not the paper I wrote.”
LikeLike
gabrielrossman
August 23, 2013 at 5:45 pm
The issue for most folks, including myself, isn’t that we oppose multi R&Rs, or even long multi-R&Rs, We oppose multi R&Rs that take years and end up being rejected anyway.
I’m skeptical that “most folks” in sociology feel this way. I will certainly raise my hand of someone who opposes multiple R&Rs. I don’t think there is any reason a journal should ever give a third R&R, whether it eventually ends up as an accept or a reject.
Granted, I have a certain sympathy for the counter-argument that “our first committment is publishing the best content, period, and the author is free to take their paper somewhere else if they don’t like how we do things.” But ASR and AJS have such a position in sociology that once they give the first R&R to a graduate student or assistant professor, the younger person is de facto stuck going down the rabbit hole to wherever the process ends. Multiple R&Rs mess with younger scholars lives and prevent them from writing other papers in what should be the most creative period of their careers.
LikeLike
jeremy
August 23, 2013 at 7:50 pm
Thanks for the thoughtful reply, Jerry. There’s an interesting tension in the model of scholarly development you sketch, whereby scholars start off being more innovative because they are poorly socialized into the standards/norms of publishing, they get help from journals like ASQ in socializing them, and then this process of socialization makes them less innovative over time. To the extent that model is accurate (its plausible to me that this describes some careers; not sure how general it is), it doesn’t sound so inspiring, and it raises questions about the journals’ role- it seems like they are problematic for innovativeness in the long-run.
LikeLike
ezrazuckerman
August 25, 2013 at 1:10 am
Ezra: A good friend once summarized journal publishing as “proving the semi-obvious.” If you make an argument that is truly not-obvious, a random selection of three scholars are never going to approve it. Thus, the peer review system trends toward producing things we already kind of know. The advance is in technique, not basic ideas. And if you need to publish fast, that doubly discourages innovation.
LikeLike
fabiorojas
August 25, 2013 at 2:54 am
Ezra: I’m not persuaded that scholars are only capable of a single type of work, or that journals are that effective at socializing them. A more optimistic account would be that younger scholars have to publish in fancy journals for career reasons, so they have more incentive to turn their insights into standard journal articles. Old guys who are less worried about getting tenure (like us) have a bigger variety of potential venues available, so the normal science works might go to fancy journals, the more innovative stuff that doesn’t include all the methodological accoutrements might go to less-fancy journals, and the intemperate screeds go into edited collections. Perrow wrote something along these lines in 1986 in an ROB chapter called “Journaling careers,” which is brief and wise.
LikeLike
jerrydavisumich
August 25, 2013 at 4:33 pm