orgtheory.net

sociology journal reviewing is dumb (except soc sci and contexts) and computer conference reviewing is the way to go. seriously.

This post is an argument for moving away from the current model of sociology journal reviewing and adopting the computer science model. Before I get into it, I offer some disclaimers:

  1. I do not claim that the CS conference system is more egalitarian or produces better reviews. Rather, my claim is that it is more efficient and better for science.
  2. Philip Cohen will often chime in and argue that journals should be abolished and we should just dispense with peer review. I agree, but I am a believer in intermediary steps.
  3. I do not claim that computer science lacks journals. Rather, that field treats journals as a secondary form of publication and most of the action happens in the conference proceeding format.
  4. Some journals are very well run – Sociological Science does live up to its promise, for example, as a no nonsense place for publication. I am not claiming that every single journal is lame. Just most of them.

Let’s start. How do most sociology journals operate? It goes something like this:

  1. A scholarly organization or press appoints an editor, or a team, to run a journal.
  2. There is a limit on how many articles can be published. Top journals may about only 1 in 20 submitted articles. Many journals desk reject a proportion of the submissions.
  3. When you submit an article, the editors ask people to review the paper. There are  deadlines, but they are routinely broken and people vary wildly in terms of the attention they give to papers.
  4. When the reviews are written, which can take as short as a few days but as long as a year or more, the editors then make a judgment.
  5. Most papers with positive reviews and that the editors like go through massive revisions.
  6. The paper is reviewed again, completely from scratch and often with new reviews.
  7. If the paper is accepted, then this takes as little as a semester but more like a year or two.

This system made sense in a world of limited resources. But it has many, many flaws. Let’s list them:

  1. Way too much power in the hands of editors. For example, I was told a day or two ago that a previous editor of a major journal simply desk rejected all papers using Twitter data. A while ago, another editor a major journal just decided she had enough of health papers and started desk rejecting them as well. Maybe these choices are justified, maybe they aren’t.
  2. Awful, awful reviewer incentives. Basically, we beg cranky over worked people to spend hours reading papers. Some people do a good job, but many are simply bad at it. Even when they try, they may not be the best people to read it.
  3. Massive time wasting. Basically, we have a system where it is normal for papers to bounce around the journal system *for years.*
  4. Bloated papers. Many of the major advances in science, in previous ages, where made in 5 and 10 page papers. Now, to head off reviewers, people write massive papers with tons of appendices.

Ok, if the system is lame, then what is the alternative? It is simple and very easy to do: move to peer reviewed conference system of computer sciecne. How does that work?

  1. Set up a yearly conference.
  2. Like an editorial board, you recruit a pool of peer reviewers and they commit to peer review *before seeing the papers.* Every year, the conference had new “chairs,” who organize the pool.
  3. Set hard page/word limits. The computer will not accept papers that are not in the right range.
  4. Once papers and abstracts are submitted, the reviewers *choose* which papers to review. People can indicate how badly they want a paper and you then allocate.
  5. Each paper had a “guide” who hounds reviewers and guides conversation
  6. Set hard deadlines. These will be followed (mostly) because there serious consequences if it doesn’t.
  7. Papers can then be ranked in terms of reviews and the conference chairs can have final say. Papers are not perfect or make everyone happy. They just have to be in the top X% of papers.
  8. CS proceedings sometimes allow discussion between reviewers, which can clarify issues.
  9. Some conferences allow an “R&R” stage. If the paper’s authors think they can respond to reviews, they can submit a “rebuttal.”
  10. In any case, accepted or revised papers also have to stay under the limit and must be submitted by a hard deadline.
  11. From submission to acceptance might be 3 months, tops. And this applies to all papers. The processes

Let’s review how this system is superior to the traditional journal system:

  1. Speed: a paper that may take 2-3 years to find a home in the sociology system, takes about one or two semesters in this system. The reason is that the process concludes quickly for every single paper and there are usually multiple conferences you can try.
  2. Lack of editorial monopoly: The reviewers and chairs rotate every conference, so if you think you just got a bad draw, just try again next year.
  3. Conversation: In the CS conference software (easychair.org), reviewers can actually talk to each other to clarify what they think.
  4. (Slightly) Better Reviews: People can choose which papers to review, which means you are way more likely to get someone who cares. Unlike the current system, papers don’t get orphaned and you are more likely to get someone invested in the process.
  5. Hard page limits: No bloated papers or response memos. It is tightly controlled.

The system is obviously faster. You get the same variety of good and bad reviews, but it is way, way faster. Papers don’t get orphaned or forgotten at journals and all reviews conclude within about 2 months. Specific editors no longer matter and single gatekeepers don’t bottle neck the system. It is better for science because more papers get out faster.

Rise up – what do you have to lose except your bloated R&Rs?

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)/Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)/From Black Power/Party in the Street / Read Contexts Magazine– It’s Awesome!

 

Advertisements

Written by fabiorojas

December 6, 2017 at 5:01 am

8 Responses

Subscribe to comments with RSS.

  1. Correction: Soc Science is not well run. But otherwise you might be correct.

    Liked by 1 person

    cwalken

    December 6, 2017 at 7:09 am

  2. I’m not familiar with this model, so I’ll look at it.

    Not sure what you mean about me wanting to “dispense” with peer review. I’m committed to making peer review better, not dispensing with it. Here’s something I wrote about it: http://blogs.lse.ac.uk/impactofsocialsciences/2017/10/16/the-next-stage-of-socarxivs-development-bringing-greater-transparency-and-efficiency-to-the-peer-review-process/

    Liked by 1 person

    Philip N. Cohen

    December 6, 2017 at 12:37 pm

  3. Here is an important question about this system: can your paper be accepted even if you are unable to attend the conference? Because otherwise this is a system wildly biased against those of us who do not have access to substantial travel funding. This includes adjuncts, graduate students, and independent scholars, of course, but it also include full-time faculty. At my institution, a new assistant professor might have to spend 5% of their after tax income to attend ASA, for example. We are also unlikely to submit to journals like Sociological Science because we don’t have support for publication costs. But the regular annoying slow journals remain within our financial reach with their $40 submission fee. So, taking this model seriously requires thinking through the implications it would have for people outside the most elite and well-funded sectors of the discipline.

    Liked by 2 people

    Mikaila

    December 6, 2017 at 1:23 pm

  4. Interesting ideas. I worry about the volume of work that is expected and produced in the current system. That is, there is so much specialization and volume that it is difficult to make those “middle range” connections that can inform policy and practice in the world. I wonder if it was possible for Merton to read about 80% of the content in the major journals. That encouraged people to read outside their area and see connections. Again, I think we have to start thinking more about how we assess and reward academic outputs. I worry about whether the CS system would encourage further balkanization and inward focused boundary-maintenance that has little to do with producing good science.

    Like

    ggauchat

    December 6, 2017 at 5:22 pm

  5. Thanks for the comments:

    1. Philip: I apologize if I mis-characterized your view. The link clarifies things.

    2. Mikalia: You are correct. One very big disadvantage is the cost of conferences. I don’t have any “silver bullet” but people deal with it this way:

    a. CS papers are usually team efforts. So usually at least one person can show up.
    b. Conference organizers tolerate a *modest* amount of no-shows. If it passes peer review, it it is published.
    c. If conferences become the norm, then people start budgeting for it.

    None of these solutions are magical, but together they seem to work.

    3. ggauchat: The CS world is huge but fragmented. They still have some super general/high prestige conferences (such as WWW for the social science/CS world) but there is a network of related conferences that link various specialties. This is different than soc, where we have 2-4 general interest journals and everything is way lower in prestige.

    In the end, it depends on what world you want. I personally like a world where more is published and faster. That is not the soc world, where you need to essentially hit two journals to get high impact. I am open to persuasion, though.

    Liked by 1 person

    fabiorojas

    December 6, 2017 at 8:18 pm

  6. Having reviewed in both worlds I would say that in general the reviews in the CS space are worse. They tend to be substantially shorter and often miss the (theoretical) forest for the (methodological) trees. Not to say that this would happen if ASA went to a CS conference model — you may still get very good reviews that are longer and well thought-out. And when the goal isn’t R&R but just yes or no, the reviews have a different purpose. In any case, I’d like to see how this plays out in the social science space. There are some other social science conferences which use easychair — International Conference for Computational Social Science and Union for Democratic Communication are two I’ve reviewed for this year.

    Also echoing Mikalia’s very real concern. CS conferences registration fees — not to mention the travel itself — are $800-1000. They typically allow students to do some in-kind volunteering for a much lower fee, but for profs in social sciences this is generally not tenable.

    Liked by 1 person

  7. Thanks, Alex! I agree with Mikalia and you on costs, but my suspicion is that the travel costs would be built into grants for fields like demography and experimental work and that in other cases, people would transfer funds from regular conferences like ASA to ones that yield peer review products.

    But I will disagree on one point, in my experience, journal reviews vary wildly. I have gotten outstanding and short junky reviews. At this point in my career, I easily prefer a series of short semi-decent CS reviews that I can get over 12 months than to wait a year for maybe, a single thoughtful review. It honestly puzzles me why sociologists are so tolerant of letting journals waste time.

    Liked by 1 person

    fabiorojas

    December 7, 2017 at 7:44 pm

  8. Hooking up with conferences can be a good idea for special issues. But in absolute terms it does not speed up the review process if you can only submit once a year. I think a more Internet adequate way of doing it is to publish on a rolling basis. Apart from that, I like many of the other ideas. With the Internet Policy Review http://policyreview.info we founded a journal that incorporates several of them. It is a fast track, open access, non-blind peer review online journal and quite successful so far.

    Like

    Uta

    December 7, 2017 at 9:53 pm


Comments are closed.

%d bloggers like this: