so, have things changed at the asr?
Last year, we discussed a specific policy at the American Sociological Review (and me getting booted from the reviewer pool for complaining!). What appears to be happening is the papers are being sent out for 3rd and 4th reviews, to new reviewers, and then getting rejected after years of review. Since I haven’t submitted in about a year and a half, I have no idea – have things have changed? I ask in all seriousness. I’m just a believer in not jerking people around.
50+ chapters of grad skool advice goodness: From Black Power/Grad Skool Rulz
orgtheory.net 
Just went through 3 rounds of rigorous reviews at ASR with Tiantian Yang (who was senior author). The paper was improved, approved, and then published in the April 2014 issue. A new reviewer raised issues that had not come up in the first round, which was somewhat off-putting, but in the end, the reviewer was open to our selective appreciation of the suggestions…
Would we have liked fewer rounds? Yes.
Was the paper improved? Yes.
LikeLike
Howard Aldrich
April 8, 2014 at 1:20 am
still ugly i would say (especially if you are not famous) …
LikeLike
OS
April 8, 2014 at 2:25 am
Within the past few weeks, I had a long and cranky conversation with the editors around these issues. I’m not sure I heard them admitting that they were wrong. It was more “we want to hear your opinions.”
LikeLike
olderwoman
April 8, 2014 at 12:49 pm
@Howard: The issue isn’t that revision improves papers. We all agree that it does. The issue is the justification for rejecting papers after years (literally) of revision. Why bother with revision if you can’t see a reasonable path toward publication? What on earth were editors thinking on the 3rd R&R? Are we damaging scholars by with holding their work for years? In other words, if I were an editor, I would only 2R&R a paper if I felt really confident that the is a an obvious way to revise the paper. Otherwise, reject.
And BTW, congrats on your paper!
@OW: If you believe some chatter at SJMR, ASR is handing out fewer R&Rs, which would go a long way to solving the problem. Perhaps, their pride prohibits them from admitting error.
LikeLike
fabiorojas
April 8, 2014 at 6:19 pm
I was just talking about this to my colleague Myra Ferree last night. Her position is that there is only 1 R&R. After that, it should be a conditional accept if there is something that should be fixed that the editor can evaluate (I’d also accept a conditional accept that a reviewer checks), or a reject. My view is pretty close to this, except I think it is a crime to add new reviewers to an R&R unless there is a special reason to do it, in which case the author should be told about this at the time of the R&R. The editors would not accept my claim that adding new reviewers is just wrong.
What I think is happening at ASR is they’ve gotten the word and are now rejecting more, but often rejecting people who have already had two or even three R&Rs and often rejecting papers that have actually satisfied the reviewers on grounds that seem illegitimate in light of the paper trail of reviews.
LikeLike
olderwoman
April 8, 2014 at 7:53 pm
@fabiorojas. I would have to say that almost all the value-added to the review process occurred in the first round. So, from that point of view, we could have stopped without great loss with the resubmission. The ‘new’ reviewer seemed unaware of much of the literature we had addressed. But reviewers must review, eh?
LikeLike
Howard Aldrich
April 8, 2014 at 8:06 pm
@OW: Wow, that astounds me. I wish there was some way to add some structure here to help authors trapped in the R&R trap.
LikeLike
fabiorojas
April 8, 2014 at 8:14 pm
I suppose we should say that, to their credit, the whole process took about 14 months, from submission to acceptance of the third revision.
LikeLike
Howard Aldrich
April 8, 2014 at 8:31 pm
Well, no one complained about the speed of reviews (yet). But wow, 3 reviews in 14 months? I am on month 6 from a leading health journal … for round 1!! I need to slap your name on my paper to speed things up!
LikeLike
fabiorojas
April 8, 2014 at 8:32 pm
Hey, no grass grew under our feet as we were doing the revisions! And the reviewers were prompt, as well!
LikeLike
Howard Aldrich
April 8, 2014 at 8:33 pm
More power to you, Brother Howard.
LikeLike
fabiorojas
April 8, 2014 at 8:34 pm
Actually I would dispute the point that reviews (inevitably) improve papers. Often they do, especially on the first round, but reviews can also make papers worse by ruining the style with parenthetical hedges and irrelevant citations that make them painful to read. (A big part of this is that many authors are risk averse about “selective appreciation” of suggestions and put a strong presumption in favor of making requested changes). That and the more basic fact that multiple rounds of review delay the paper and so we should have a sort of NPV view of a paper’s quality such that a paper published in three months is better than a paper published in eighteenth months. Otherwise why not age papers in oak barrels like whiskey? Or bury them in time capsules for our grandchildren?
I’m also impressed that Professor Aldrich acknowledges that the process took 14 months and rapidly hit diminishing returns after the first round, but still takes a glass half-full view of the process. It’s like one of those passages from the lives of the saints where people are ripping the guts out of Saint So and So as he prays for his eviscerators.
LikeLike
gabrielrossman
April 9, 2014 at 12:10 am
Small n, but I’ve heard of more people who are getting “reject with an invitation to resubmit.” (I can’t remember the exact wording, but that’s the gist.) One wonders if the primary difference between a “reject & resubmit” and a weak “revise & resubmit is that the traditional “R&R” looks worse on the official stats.
LikeLike
anon
April 9, 2014 at 2:35 am
With all the criticism ASR has been getting, why is there no similar criticism of AJS? AJS has been doing multiple R&Rs (often with rejects after 2), new reviewers at each round, and reject and resubmits for at least 10-15 years. Also, AJS is much slower than ASR at each round. If ASR has problems, AJS invented the problem. It feels ASR editors are being punished because of the transparency of their review statistics. How about some similar criticism of AJS or at least acknowledging the problem is spread across the discipline rather than concentrated at ASR?
LikeLike
anonymous
April 9, 2014 at 5:52 am
My experience with AJS (both as an author and longtime reviewer) is that they make a real effort to be very up front with authors throughout the review process about where the paper stands and how the review will proceed. In the first round, Andy will tell you whether he thinks your R&R is a “long shot,” which leaves it up to you whether to resubmit. I’ve seen maybe 30-40 papers go through the review process at AJS, and only two received second round R&Rs, and two received a “conditional conditional” acceptance. In both cases, Andy was up front with the authors about the probability of acceptance as well as how the review process will unfold (e.g., “the final paper will be evaluated in house,” or “we will seek input from an outside reviewer, but it will be someone who already contributed reviews in previous rounds”). I’m sure there are occasionally situations where new people are brought on in the third round, but I suspect it’s VERY rare. More than that, my sense is that the editorial board is quite concerned with the intellectual and professional value of the review process, and work hard to give each paper a fair evaluation.
AJS *is* slow to turn around reviews, but that’s not the fault of the journal. The problem is that, when they switched to an electronic system (rather than a nice thick envelope arriving in your mailbox), people found it far too easy to say “no” when asked to review. So to get each paper reviewed by three people, they have to ask on average something like 6 people. Now consider that each person asked may hold onto the paper for a week (or a month!), and some of them agree to do the review and never follow through. That time adds up. A key difference between AJS and ASR is that ASR has many more “consulting editors” than AJS. I can’t remember the exact numbers, but it’s at least 3 and possibly 6 times as many. Consulting editors are the workhorses of the review process. They’ve committed to reviewing at least one article a month, and they try to do it in a reasonable amount of time. The more consulting editors you have, the more strong backs you have who will likely say yes.
YMMV. But if my experience is any indication, AJS is doing an amazing job.
LikeLike
muriel
April 9, 2014 at 6:34 am
muriel,
I’m anonymous from prior comment. I appreciate there is a lot of heterogeneity in experiences with any one journal. However, I was on the board at AJS for a term, have submitted there several times, and have also seen >30 review processes. My experience is quite different from yours (though I don’t mean to imply doubt for your experience). At AJS, I’ve seen multiple R&Rs several times, and have seen new reviewers added on second round (though not 3rd) on several occasions.
I think ASR and its board have the same standards of professionalism. And the “up frontness” of AJS is present at ASR as well. In both cases, these are template letters and the editors use the same language over and over again. Maybe ASR should be more selective with reject and resubmits and second R&Rs. I had a paper recently accepted after two R&Rs and as best I can tell nine independent reviewers (though it wasn’t entirely clear if all nine were new, at least 7 of them were). I have no dispute with the claim that 2 R&Rs as the mode is not optimal. That said, my first ASR was published about ten years ago after a reject and resubmit and two R&Rs, so such practices existed in the past as well (though one gets the impression this has increased).
Last, I don’t at all believe AJS’s story that it is the reviewers’ fault for the delays. If so, it would be more widespread at other journals. Rather, AJS is an outlier for delays. Abbott made this claim a few years ago at the board luncheon and it defies credibility (and is frankly rather glaringly self-serving). I’ve seen multiple times where the office has the reviews on their desk (they sometimes tell you if you ask) and they don’t make the decision for months. Slow reviewers cannot explain why ASR can do reviews in >4 months and AJS needs as much as a year. Yes, seriously a year.
Sorry, in my and many colleagues’ experience, ASR is doing at least as good of a job as AJS. Maybe my judgements reflect differential success at the two journals. But, the imbalance in the public critique of ASR relative to AJS seems quite unfair.
LikeLike
anonymous
April 9, 2014 at 6:59 am
I have also noticed the asymmetry in complaints and think it has to do with the fact that AJS is privately managed by one department while ASR is publicly managed by the publications committee of the ASA. This has not seemed fair to me. It is like the general cultural practice of giving corporate actions a pass and complaining about government. It is my impression that AJS is generally slow in the editorial office and has its own biases. At the same time, the recent cases that I’ve encountered that seemed to me to be outrageous editorial actions/decisions were at ASR. This might have just been small-n. The more stable editorship of AJS makes its behavior somewhat more predictable (for better or worse) than the rotating editorship of ASR.
LikeLike
olderwoman
April 9, 2014 at 11:29 am
I do tend to hold ASR to higher standards, but that’s because it professes to be the flagship journal for the field, backed by the power of the ASA brand. It’s also a revenue-generating machine for an organization that ostensibly represents us. And, it is published by a company that is making a handsome profit off of us and our labor. (Olderwoman’s ASR=government, AJS=corporate analogy thus seems incorrect; you could just as easily argue that ASR=corporate, AJS=nonprofit.)
A second issue is the time to publication after acceptance. I’ve waited more than 6 months from acceptance to page proofs — so, basically, final processing at ASR + copy editing + processing at Sage + typesetting. That’s not even to “on-line first.” This is much longer than I’ve experienced at other journals (including AJS), and belies the claim that for-profits’ economies of scale make the production process more efficient and streamlined for authors. And, these delays have consequences for authors’ careers and for the discipline, just as long review times do.
LikeLike
krippendorf
April 9, 2014 at 12:12 pm
If there was one journal that inspired creation of Sociological Science, it was AJS and not ASR. I would not defend ASR practice in recent years, but AJS set the standard for a slow, progeny-biased, and Editor-taste-determined sociology journal. AJS still publishes some great papers because the every best sociologists seem willing to tolerate the pain. Sociology as a discipline would have been better off if AJS had been eliminated in 1995 and all of its papers were redistributed as submissions across all other journals. That also would have prevented ASR from becoming a journal so heavily dominated by grad student papers.
LikeLike
AJS critic
April 9, 2014 at 12:23 pm
On a related note, what explains the extremely slow production time for AJS issues? My calendar reads April 2014, yet the most recent issue available online for AJS is September 2013!!
Perhaps Fabio has some insight here.
LikeLike
Jim H.
April 9, 2014 at 1:54 pm
I am irritated that ASR published a paper with serious statistical flaws but then did not publish my letter pointing out the flaws. I think this is a systematic problem with journals, that the informal rules for publication (that findings be substantively important and statistically significant) bias things toward the publication of exaggerated claims. It goes like this: paper A makes a dramatic claim and is published. It turns out that paper A has methodological problems. But pointing out such problems is less exciting than the original large claim, of course. To say it again: I understand and appreciate the rationale for wanting to publish major papers with major claims. But I do think this introduces what one might call a bias toward drama. Perhaps it’s just my statistics training that gives me a bias toward boringness. I think it’s worth going back and spending the time to criticize things that have already been published.
It may be that in this particular situation my letter did not warrant publication (I think it did, but that’s my perspective), but in any case I think the reluctance to print criticisms is a major problem with lots of journals. Nothing specially bad about ASR here.
Related to this is the idea that journals don’t just spread information and improve communication, they also represent chits for hiring and promotion. From that perspective, I can see the attitude of, “We can’t just publish every critical letter, then people will do nothing but criticism as it’s so cheap compared to original research.” But that attitude irritates me because I wasn’t writing that letter to get a chit; I was writing the letter as a public service. I’d have no problems if critical letters were identified as such in the publication record so the whole chit issue wouldn’t have to come up.
LikeLike
Andrew Gelman
April 9, 2014 at 8:13 pm
anonymous,
thanks for sharing your experience at AJS; it’s useful to hear a different perspective. i should clarify that i have seen new reviewers added in the second round (my sense is that they typically add one new reviewer, and send it back to one previous reviewer). but i’ve never seen a new reviewer added in the third round. it sounds like that’s your experience as well.
not sure what to believe about the AJS’s story re: reviewers being responsible for long wait times, although i do notice that i am often reviewer E, F, or even G (which suggests that somewhere along the line at least two or three people declined). i’m assuming here at reviewers are assigned alphabetically, of course. also, when choosing between ASR and AJS, i think the general belief is that AJS is more willing to publish especially long papers. i’ve never checked the average number of pages in an AJS vs. ASR paper, but i’d bet that AJS papers are significantly longer. even if all other practices are the same at the two journals, that would likely also lead to longer review times (and more people saying no).
i should add, i haven’t had as much experience reviewing for ASR. so i’m not saying that AJS is necessarily better, i am just better informed about their SOP.
LikeLike
muriel
April 9, 2014 at 8:51 pm
Andrew – my sense is that most journals rarely publish letters of response because they don’t like to use the print space. But this could be easily resolved in today’s age of online journals. Most readers of ASR don’t actually read the print version anyway. They read the articles online. It would be really easy to post letters, addendums, appendices, etc. directly underneath the original article in the online journal. There would be some cost, of course, including editorial services, but I think it would be well worth the investment.
LikeLike
brayden king
April 9, 2014 at 8:59 pm
Andrew – Did the folks at ASR give you official reasons why they chose not to publish your response? (I’m guessing this is about Hamilton’s article on income and college performance)
LikeLike
Kyle Siler
April 9, 2014 at 10:50 pm
Brayden, that’s a brilliant idea! Have you proposed it to the people at ASA/ASR?
LikeLike
Howard Aldrich
April 10, 2014 at 2:04 am
Andrew: I had a similar experience at ASR. My comment showed that specific empirical claims in the paper were wrong. However, the reviewers and editor agreed that it was not important enough to warrant publication.(That was a long time ago. I guess I could dig it up and put it online somewhere.)
LikeLike
Philip N. Cohen
April 10, 2014 at 1:05 pm
Is there a session at ASA where the ASR editors (and editors of other major ASA journals) discuss issues coming up in editing the journals and take questions? If not, why not?
LikeLike
gradstudentbyday
April 10, 2014 at 8:01 pm
Brayden:
I completely completely agree with you that online publication should be no problem. But I think the issue is not just physical space in the journal (or even the time it takes for the journal staff to edit and proofread the articles). I think the “chit” issue also comes into play.
Kyle, Philip:
The story of my experience at ASR is in my article, It’s too hard to publish criticisms and obtain data for replication. As I wrote, I don’t fault the author of the original article–we all make mistakes–and in some sense I don’t fault the ASR either, as they’re following their policy. Kyle: The reason they gave for not publishing was that, as with Philip’s example, the reviewers and editor agreed that it was not important enough to warrant publication. They also had some specific criticisms of my letter but I think the non-importance was key. I demonstrated that the article had statistical flaws but I did not demonstrate that the flaws would have serious impact on the article’s major conclusions. I think I could’ve done this but it would’ve required more work, and I was already having lots of problems getting the data (not the fault of the author of the article, it was a problem with the keepers of the dataset).
Here’s what I wrote in my article about the episode:
“The asymmetry is as follows: Hamilton’s paper represents a major research effort, whereas my criticism took very little effort (given my existing understanding of selection bias and causal inference). The journal would have had no problem handling my criticisms, had they appeared in the pre-publication review process. Indeed, I am pretty sure the original paper would have needed serious revision and would have been required to fix the problem. But once the paper has been published, it is placed on a pedestal and criticism is held to a much higher standard.”
Again, the point is that had my comments been in the form of a referee report, they would have to had been addressed, there’s no way the article could’ve been published as is. As a referee, I would not need to offer an independent data analysis and proof that the statistical error would have a major effect on the conclusions. It would’ve been enough just to point out the error. But once the article appears, the burden of proof is reversed. And I think that’s too bad. I think it would be appropriate to publish my letter (and I’d have no problem if, in the review process for my letter, I’d been told to add a paragraph emphasizing that I had not demonstrated that the statistical error had a major effect on the conclusions).
LikeLike
Andrew Gelman
April 10, 2014 at 8:13 pm