orgtheory.net

inside the american journal of sociology

Don’t worry, I won’t give away state secrets.

In the 2000-01 and 2002-03 academic years, I worked at the American Journal of Sociology as a member of the manuscript intake group and later as an associate editor. I also worked for a while, roughly at the same time, as the managing editor of Sociological Methodology, which was then edited by my dissertation advisor, Rafe Stolzenberg. In this post, I want to tell you a little bit about how top academic journals work. This is important because academics reward people based on getting into highly selective journals. There should be a lot of discussion about how the institution works and what does and does not get accepted.

Background: The AJS is the oldest general interest journal in American sociology and has, during its entire existence, been based at the Department of Sociology at the University of Chicago. To my knowledge, it has never rotated to another program. In fact, the relationship between the Department and the journal is so strong that one of Chicago’s faculty, Andy Abbott, has written a very nice monograph just about the AJS called Department and Discipline. It’s a good book and you should read it if you want to either understand the evolution of journals or how Chicago fits in to the broader discipline.

Some time ago, the AJS developed this system where students were strongly involved in the operation of the journal. For example, the AJS usually is run by a full time manager, the incredible Susan Allan, and a few students who run the office. These folks do budgets, office organization, crazy amounts of paper work, and a whole lot more. But it goes beyond administration. Students are deeply involved in the shaping the journal’s content.

At the time I was a doctoral student, the AJS was organized into three major committees: the editorial board, which is always headed by a senior professor; a manuscript intake group, which assigned reviewers to papers; and a book review board, also headed by faculty. The manuscript intake group and the book review board are mainly staffed by students. The editorial board usually has one or two students on it, who have a major voice.

In contrast, Sociological Methodology was run like many specialty journals. You had a manager (me) and the editor (Prof. Stolzenberg) who choose reviewers, read reviews, and made decisions. These two people did about 90% of the work running the journal

Lessons from working at AJS: In many ways, the AJS resembled other major journals that must process hundreds of papers per year. There is a basic intake/review/decision cycle. That process has up and down sides. The up side is that the journal review process is actually pretty decent at weeding out garbage. After a while, you can easily spot bad papers. Unending rants, poor spelling, poor formatting, lack of data. Another upside is that many papers do actually improve once people respond to reviewers.

I also saw some of the downsides of the review process. For example, I discovered that only about a third of people agree to consistently review papers, making the workload highly unequal. Some of the patterns are obvious. A lot of people stop answering the mail post-tenure. People in some sub-fields are simply bad citizens and refuse to write reviews or write bad ones. Finally, like a lot of journals, we could let papers fall between the cracks and go without a decision.

Perhaps the biggest insight that I had was the power of editors and the randomness that goes into a “great paper.” Example: while I was on the editorial board, we had a paper with ok but not great reviews. I read it and disagreed strongly. Right as the chief editor was about to assign it to the reject pile, I interjected. It was published and was covered by the national media. This may sound like a great story, which it is, it also shows the weakness of the journal system. If I had been absent that week,  or if we had another student editor, the paper would have been rejected. Conversely, I am sure that I overlooked some excellent work.

A related lesson is that the chief editor matters a great deal. An editor can doom a paper from a scholar they don’t like, or on a topic they hate, by simply assigning it to known mean reviewers. Editorial influence appears in other ways. While most papers are clear rejects, many are on the border. An interventionist editor can strongly affect what is accepted from these border line cases. One editor I worked with would actually ask the authors for the data and rerun the analysis to see if reviewer 2’s criticism was right. Another is very comfortable with adding a few suggestions and then just tossing it back to the authors. The power of editors, and the Chicago department, also manifests itself in the fact that AJS is way more tolerant of longer, theory driven papers than other peer social science journals.

A second lesson is that there are big structural factors that influence what gets published. The first factor is type of research. Simply put, ethnographers produce papers at a slower rate than demographers. So if you have a small number of papers, it doesn’t make sense to risk it all on the AJS. Instead, you move to the book or more specialized journals.That’s one reason why ethnographic work is rare in top journals  A second factor is culture. There are some sub-fields where the reviewers seem to be really difficult. For example, during the late 1990s, there seemed to be a sort of feud in social psychology. Each side would tank the others in the review process. Ethnography is similar. When people did submit field work papers, it was nearly impossible to get 2 or 3 reviewers to say “this is good enough.” Just endless and endless demands.

The final lesson I took is that we are humans and we are biased. While 95% of decisions really based on reviews, there were definitely times that our biases showed. There were one or two papers I promoted because I was excited about social movement research. At other times, decisions took into account touchy political situations and author prestige. As I said, this is not typical but it does happen and I include myself in this evaluation.

Lessons from Working at Sociological Methodology: This was a totally different experience. Instead of being embedded in a larger group, it was just literally me and a filing cabinet and my advisor. We had a weekly meeting to discuss submissions, I took notes, and he told me what to do.

Probably the biggest take home point from working with Professor Stolzenberg was that editors make or break the journal. The dude was really on top of things and few papers went past 2 or 3 months. Once a paper couldn’t get a single review after six months and the editor wrote a letter to the author explaining the situation and they mutually agreed to release the paper from review.

Stolzenberg was also not afraid of people, a strong trait for an editor. He didn’t mind rejecting people and making the process speed up. Although he didn’t desk reject often, he was good about getting reviews and writing detailed rejection letters. That way, the journal didn’t get clogged with orphaned papers. The lesson is that there really is no excuse for slow reviews. Get reviews, reject the paper, or get the hell out of the editing business.

Final note – authors and reviewers are lame: I conclude with a brief discussion of reviewers and authors. First, authors are quite lame. They are slow at responding to editor. They fail to read reviewer comments or take them seriously. And even more puzzling, they fail to resubmit after the R&R. I was shocked to discover that a fairly large fraction of AJS and SM papers at the R&R stage were not resubmitted. Perhaps a third or so. Second, reviewers are lame. As Pamela Oliver put it so well in the recent American Sociologist, the review process is simply broken. Reviewers ask for endless revisions, the focus on vague issues like framing, or simply write hostile and unhelpful reviews. So I thank the 1/3 of academics who write prompt and professional reviews and I curse the 2/3 of shirkers and complainers to an eternity of reading late reviews that criticize the framing of the paper.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

June 6, 2016 at 12:01 am

Posted in academia, fabio, journals

11 Responses

Subscribe to comments with RSS.

  1. I appreciate this, but I don’t like it. I don’t want to get (or give) lessons about how to publish in private paywalled journals. I want lessons about how to do good work and make it have an impact, and how to get sociology out from under our disfunctional legacy journal system. I don’t review for AJS if I have any requests open from open access or association journals, which means basically never. I want my students to have careers in the new world, not the old. Inertia doesn’t just happen, it’s made – by mentors and senior people coaching their students and junior colleagues on how to get ahead in the old system. The next generation is always ready to do it differently but it’s hard for them to do it alone.

    To the senior people: why not do a couple fewer AJS reviews this year and spend that time getting good young people promoted based on the quality of their work instead of the pedigree of their publications.

    Sorry for the rant!

    Liked by 1 person

    Philip N. Cohen

    June 6, 2016 at 3:29 am

  2. ASR was founded in 1935 in a rebellion against the domination of sociology by Chicago. AJS still has elements of a “house journal.” Lengermann’s 1979 ASR article says it resulted from a combination of personal fights, theoretical crisis fueled by the depression and professional anxieties also fueled by the depression. JSTOR link: https://www.jstor.org/stable/2094504

    Like

    olderwoman

    June 6, 2016 at 4:50 pm

  3. Retiring soon. Glad to be leaving this world behind. But I still enjoy reading these exchanges!

    New gig:

    Like

    william bielby

    June 6, 2016 at 5:31 pm

  4. Fantastic insight.

    In my own field of Economics, there is a scandal brewing about the shady editorial practices at the flagship journal (AER), as laid out in detail:

    http://retractionwatch.com/2016/05/26/economists-go-wild-over-overlooked-citations-in-preprint-on-prenatal-stress/#comment-1040453

    Interestingly, a group of anonymous economists have responded to this perceived injustice:

    https://ideas.repec.org/p/pra/mprapa/71699.html

    Sadly, anonymity is thought to be necessary to prevent professional retribution. But some mechanisms to keep editors in check are ugly, I suppose. Such is life. Such are the current incentives of the peer review process.

    Liked by 1 person

    Nicolas Bourbakirk

    June 6, 2016 at 6:32 pm

  5. Fabio:

    I’ve published a few papers in AJS and that’s fine, but I’m still mad at them for asking me to write an article for them (a discussion of some other article they may have published, I’m not sure) and then telling me they decided they didn’t need it. That’s not cool.

    To say “authors are lame” because they don’t resubmit after the R&R . . . what’s all that about? Who says an author has an obligation to provide free content for the journal? Also it seems odd to complain about the reviewers. You’re asking them for free labor, then you’re complaining they don’t do what you want them to do. Whassup with that?

    Liked by 1 person

    Andrew Gelman

    June 7, 2016 at 3:10 am

  6. Andrew:

    1. If AJS didn’t publish a solicited article, shame on them. I completely agree. You should write up your experience and make it public.

    2. Sorry, but it is quite lame not to resubmit under normal circumstances. A journal isn’t a free advice service. It’s an attempt to publish current research. To do that, two or three of your colleagues spent hours reading your paper. The editor sat down and read it and many editors will spend a lot of time working on letters to authors. And the managing editor (ahem) spent a bit of time hounding people to get the review done and do other administrative work. For some one to simply throw away all that free work is frankly insulting.

    I think the only valid reasons not to resubmit a paper are: (a) a belief that the results are no longer valid; (b) incapacity; (c) an inability to actually do the requested revisions; and (d) exit from the scholarly community.

    3. Also, you are wrong about reviewers. We don’t randomly ask people to review. We ask people to review if they are practicing scholars. That means their books and articles have been reviewed by others. They owe us, big time. I think that the statement made by Academy of Management journals sums it up best – submitting an article to a journal is using a resource and it is expected that you pay back. Period.

    If you want to boycott specific journals for bad behavior, I support you 100%. But if a journal asks you to review a paper in your area, you aren’t already “booked” for reviews, and they behave well, you really do have a moral obligation to help out. Otherwise, you’re just a freeloader … shame on you! Review or get out of the journal system.

    Liked by 1 person

    fabiorojas

    June 7, 2016 at 3:33 am

  7. 2. Another reason not to resubmit is you think the reviews and/or the editor are wrong and they are unlikely to accept your article after the revision. I’m sure Fabio would prefer some kind of small gift acknowledging that instead of just ghosting, but hey, people are busy reviewing for other paywalled journals.

    3. So this is a journal saying you are obligated to do unpaid work for them to produce an article they will literally not let you read without paying for it. (I assume Fabio would also say that as a scholar you are obligated to read AJS). I don’t know what you call a system like rhis, but I think loving it might be called Stockholm Syndrome.

    Liked by 1 person

    Philip N. Cohen

    June 7, 2016 at 12:12 pm

  8. Phil:

    Don’t put words in my mouth. Let’s stick to what I actually said. What I clearly said is this: If you participate in a community, you really should try to play by its rules. If you think that current paywall journals are immoral, then don’t submit to them and you don’t have to play by their rules.

    In terms of resubmission, I do think it would be extremely helpful if authors would tell journals that they won’t resubmit. This is important because if the non-resubmission rate gets too high, then journal production of the paper version is delayed, which has consequences for publishers and authors. This is much less of a problem in the current day, but it is still an issue. That is why some journals assume that you’ve bailed after 6 months or so. These journals will say in their letters that extremely late resubmissions will be treated as new submissions.

    Also, I never said or implied that scholars are “obligated to read AJS.” Instead, I wrote that if you’ve built a career on journals and book that rely on volunteer work, you should probably help out.

    If you want to switch the system to non-peer review, or all open access, or paid review, let’s have that talk. But until then, help out or stop playing the game.

    Like

    fabiorojas

    June 7, 2016 at 12:39 pm

  9. That’s fair, but the “journal system” is not homogeneous. There are better and worse journals. A privately-published, unaccountable, private-school, paywalled journal is at one end of the spectrum – it scores low on all dimensions. So you can participate in “the system” and still oppose AJS.

    (I assumed you think someone who publishes in journals must read AJS, because everyone thinks that. Wouldn’t you be offended or reject them if you were reviewing their work and they didnt cite AJS work that was relevant? That means in effect that if you don’t read AJS you can’t get tenure.)

    Like

    Philip N. Cohen

    June 7, 2016 at 1:04 pm

  10. Philip:

    You write (to Fabio): “I assumed you think someone who publishes in journals must read AJS, because everyone thinks that.” Remember that we’re not all sociologists! I don’t know if I’ve ever read an AJS article. OK, I’ve probably read a couple at some time or another. But I’ve written a few articles for the journal. And, setting AJS aside, I review articles all the time for journals that I never read. In the old days, people read journals. Nowadays my impression is that journal publication is a stamp of approval and . . . I’m not quite sure what else. I mean, sure, publishing your paper in top journal will get it more readers, so publication does something, somehow. But I don’t know how many people read new journals as they come in. I have no idea how things go in sociology, but in statistics it’s certainly not the case that “in effect that if you don’t read [the top journals] you can’t get tenure.”

    Like

    Andrew Gelman

    June 8, 2016 at 7:11 pm

  11. I didn’t mean that you read the journal as a journal — as in subscribing, looking at new issues as they come out, or reading outside your area — I just mean if there is relevant work published there you are responsible for knowing about and citing it (and this is more the case than for work in other journals, because it’s a high-status journal). For most research sociologists, you can’t function well if you decide never to read articles in AJS. It would be like a biologist refusing to read or cite Nature; you effectively can’t opt out even if that’s not where you want to publish. That’s my impression anyway.

    Like

    Philip N. Cohen

    June 8, 2016 at 7:56 pm


Comments are closed.

%d bloggers like this: