some proposals for journal reform
Fabio
My attitude toward academic journals is this: they are useful, but imperfect and highly inefficient. For every brilliant cutting edge paper that the system wrongly weeds out, ten truly bad manuscripts are justifiably rejected. And that’s a good thing! Rather than trash the journal system, I’d rather focus on making it better. Here are some serious proposals:
- Triple blind review: A few months ago, I suggested that the editor not know the identity of the author to reduce possible bias. It’s easy to do and would help reduce incumbency advantages.
- Multiple submission: I have never heard any rational argument for not allowing more than one journal to review a paper. Book publishers, law journals, and grant agencies all entertain multiple proposals and things seem to work pretty well. Why not have journals compete to find the best papers?
- Reviewer expiration dates: Journals should simply switch reviewers if they are more than 4 months late. Seriously, how hard can it be to read a 35 page paper with a few regressions in it? If they can’t bother to read a paper in a timely fashion, then why should we believe that they will ever take the time to write a decent review?
- Stronger editorial discretion: Conflicting reviewers? The editor should simply take a stand and say which reviewer’s suggestions should be followed.
- Two strikes and your out: If a paper can’t receive reviews after a few rounds, it’s rejected. If a journal’s reviewers simply can’t take the time to read a paper, then maybe another journal should look at it.
- Semi-public referees: If the author and reviewer agree, the names of the reviewers are revealed after the paper is published. Why? People want to be associated with quality work and having your name associated with quality work might make people submit faster reviews.
Drop your own ideas in the comments.
orgtheory.net 
Your last idea was actually implemented by Paula England when she edited ASR at the University of Arizona. For instance, Bethany’s great paper was reviewed by Lynn and Deena Weenstein among others and Barry Schwartz’s paper on the collective memory of Lincoln was reviewed by Robert Bellah and Charles Tilly. I agree with you, since I always thought that that was a very neat idea. Bring it back! Bring it back! Bring it back!
LikeLike
Omar
June 7, 2007 at 10:06 pm
I wonder if any other journals have done that. ASR should make it standard. Easy, zero cost and might encourage quicker turn around.
LikeLike
Fabio Rojas
June 7, 2007 at 11:41 pm
I think there must be a more systematic way to encourage reviewers to provide timely reviews. I think one of the main reasons reviewers take so long is that they just get too busy and forget. A solution to this problem would be to put a short deadline on the paper (give reviewers a month) and then follow up with reminder emails. Once you’ve agreed to complete the review in a month, you shouldn’t be too bugged by getting a friendly reminder email, right?
LikeLike
brayden
June 8, 2007 at 4:27 am
“A few months ago, I suggested that the editor not know the identity of the author to reduce possible bias. It’s easy to do and would help reduce incumbency advantages”
I remember this post and I still don’t get it. What is the evidence for incumbency adv? One thing that trip blind review would mean is that fewer assistant profs coming up for tenure would have that “key publication” come through “just in time.” I’m willing to give editors the discretion to make this happen.
LikeLike
pubnperish
June 8, 2007 at 11:13 am
ASR has already instituted Brayden’s suggestion. Also Theory Culture and Society which uses a fancy electronic system (to which Social Forces has recently moved), sends an automatic reminder three weeks after, and then once week after that. I think it works pretty well. I was like, jeez, all right, all right, I’ll write the freaking review, just leave me alone!
LikeLike
Omar
June 8, 2007 at 12:27 pm
Nothing like a little pestering to speed up the review process! I know it’s annoying but it’s probably the easiest thing you can do to increase the probability that reviews get done on time.
LikeLike
brayden
June 8, 2007 at 3:41 pm
I’m surprised you’ve never heard a rational argument against multiple submissions. I think maybe if I could get AJS to review one of my papers in under six months, I’d feel better about the idea of them trying to deal with 2-3x as many submissions and papers occupying their pipeline that would be published elsewhere even if AJS accepted them. It would also presumably imply at least 2x (maybe much more) papers being sent to people like you and me to review. I’m not going to review any more papers than I already do.
As far as suggestions go, my first one would be that journals like ASR and AJS greatly increase the proportion of papers they reject without review.
LikeLike
Jeremy
June 8, 2007 at 3:53 pm
Hi, Jeremy. I’m a little skeptical:
1. “I think maybe if I could get AJS to review one of my papers in under six months, I’d feel better about the idea of them trying to deal with 2-3x as many submissions and papers occupying their pipeline that would be published elsewhere even if AJS accepted them.”
In theory, I could imagine multiple submissions clogging journals, but I just don’t believe it be too bad in practice. First, this doesn’t seem to be a problem with law reviews. Second, I think what would happen is that people would submit to a few journals at a time, just like book proposals. You’d see a bump up in submissions, and with a little more weeding, you’d probably get a good response time.
2. “It would also presumably imply at least 2x (maybe much more) papers being sent to people like you and me to review. ”
If the reviewer got multiple copies to review, just print another copy. It’s easy to do.
To Pub: A few studies have suggested incumbency advantage, mainly in analyses of economics journals For example, it’s shown that JPE disproprtionately publishes Chicago profs and QJE published Harvard profs. I have no reason to expect sociology to be an exception to this rule. If we had network data about editors and colleagues, I bet you’d see advantages as well.
LikeLike
Fabio Rojas
June 8, 2007 at 6:16 pm
I don’t think law reviews are a good comparison to our peer-reviewed journals. Law reviews get bombarded with papers, not all of which get serious consideration from the editors. Rather than send them out to peers for review, they get filtered through student editors, who often use acceptances by lower-ranked reviews to indicate whether they should even bother looking at a paper or if they should just throw it in the rejection pile.
LikeLike
brayden
June 8, 2007 at 7:06 pm
B-King: I would never use law reviews as models. The use of students as gatekeepers just seems very, very wrong. However, my point is that other academic institutions (grants, law reviews, books, etc.) use multiple submission and the world has not come to an end. They also seem to be about efficient as single submission journals.
And btw: using other people’s acceptances as signals is actually what we do in the job market. If it’s good enough for us, why not for our papers?
LikeLike
Fabio Rojas
June 8, 2007 at 9:14 pm
Ahh – a topic close to my heart. I have been working on an auction idea for a few years (old version http://tinyurl.com/2xgjol). I am working on a revision RIGHT NOW with a co-author. I shall try to post a link when I get it online.
Your readers are right that the law review version is not feasible for other disciplines. Semi-public referees appear under the heading “open-access” review that others have proposed.
Our solution hinges on the main issue: referee incentives. We do not suggest a flat payment for performance but a piece of the action. The market mechanism we explore has such an incentive system. The gist of its mechanism is that a popular paper will return more points to the author/editor/reviewer(s) who take it from idea to publication. There are some other, useful side effects like proper measurement of value, etc.
LikeLike
David Zetland
June 8, 2007 at 9:35 pm
Another problem with multiple submission is that it might lead to some awkward etiquette issues. Consider this: I send my paper to ASR, and Social Forces. It gets an R & R at ASR but it gets accepted at Social Forces. Do I “decline” my acceptance at SF to take a shot at the ASR revision? Situations like that would create a lot of ill will around. One unintended consequence of the fact that journals don’t “compete” for papers in a market-like sense is that a lot of the negative emotional consequences of overt competition are averted (i.e. pitting journal against journal revealing the explicit and implicit prestige ranking among them).
LikeLike
Omar
June 8, 2007 at 9:38 pm
Omar,
You are right — journals are NOT interested in competition, in the same way professors are not (cf, tenure).
LikeLike
David Zetland
June 8, 2007 at 9:46 pm
David,
Journals are not interested in competition in the very same way that a monopolist is not interested. It’s not good for business.
LikeLike
Omar
June 8, 2007 at 9:50 pm
O-Liz: Ettiquette? I find that to be a strange response. Do we do “single submission” job searches to spare the search committee’s feelings? Do we do “single submisison” book proposals so editors won’t have bad feelings? No – we recognize that there should be competition among purveyors of knowledge. Just as authors are expected to deal with rejection, so should the editors.
And I find your situation of A at SF vs. R&R at ASR to be a very good situation. Within 3 months, the author has found out that (a) the paper is probably decent and (b) two sets of comments. What’s so bad about that?
Just like in the rest of life, the author must make a judgment about risk. And that’s a good thing! Do we consider it bad if you get a job offer and an interview (“accept” and “R&R” in the job market)? We simply say that life has risk and you decide if you want the lower rank job or take a risk with the fancy job. Same with papers. It’s up to you if want journal A for sure or journal B with 50% risk of failure. What’s so bad about that? The rest of life works that way. Why not journals?
LikeLike
Fabio Rojas
June 8, 2007 at 9:51 pm
Fabio,
The scenario that you mention is a very good situation for the author not for the SF editor. The system is arranged to make the life of people who run journals easier not the life of people who publish in journals (there is more of us and very few of them). You just have the wrong analogy. Publishing papers is not a market and a journal editor is not a “buyer.” Instead the situation is closer to that which obtains in artworlds in which there are lots of producers and a few outlets. This immediately creates a context (power-dep 101) of power for the outlets, with various producers competing among themselves for outlets (i.e. galleries). This also means that the gatekeepers who control access to outlets, since they are advantaged in any negotiation with the culture producer, will be able to enforce exclusivity stipulations into any contract to squash competition (in order to get a show at this gallery you may not show in any other ones). You are just thinking like an economist here (normatively: in an unreal world, this is what “should” happen) instead of looking at what is the structural arrangement that allows for the institutional form to be there in the first place and for academics to “accept” it in all its unfairness.
LikeLike
Omar
June 8, 2007 at 10:09 pm
Fabio — I think you underestimate the impact of simultaneous reviews on the reviewers, even “thinking like an economist.” If everyone who submits papers to AJS also submits them to SF and ASR at the same time, there will be 3-n times as many reviews to complete, where n is the number of papers that would eventually be submitted to all three journals in sequential submissions. Assuming the pool of good reviewers is fixed and limited (something editors are always complaining about), each reviewer will see 3-n times as many papers.
Granted, there will be some duplication in what hits the reviewer’s desk, and the reviewer can simply send the manuscript back to editor who was late to the party. In small and well-defined subfields, that may turn out to be common, in which case it becomes a small pool problem for the editor. In other fields, there won’t much duplication, because there are so many submissions that the odds of any one reviewer getting the same paper from two editors are pretty low.
How will reviewers respond? Some will cap the number of papers they review per year and reject all other review requests (the Marwell strategy). I’d guess that most people will fill their quota with higher-tier journal submissions, on the theory that they get more out of reading the average AJS submission. This will make it even harder for second-tier journal editors to find good reviewers.
Other
suckersreviewers will respond by agreeing to do more reviews. I’d guess this would either extend the average time to review, lead to hastier and lower quality reviews, or both.Other costs accrue to “the discipline.” In a simultaneous submission system, there’s a strong incentive for authors to submit to all three journals, wait until all the reviews come back, and then choose the journal with the comments that are most favorable / easiest to address. (Authors can play the wait-for-good reviews game under the current system, but it’s more costly.) If you assume that tough reviewers aren’t tough arbitrarily, this will lower the quality of published papers, on average. And, there’s a likely feedback effect: tough reviewers will see that their efforts are more likely to be ignored and will either stop being as tough or stop agreeing to do as many reviews.
It’d be the end of civilization! Anarchy! Death to sociology!
OK, maybe not.
LikeLike
kim
June 8, 2007 at 10:56 pm
Isn’t the main problem how to increase the quality of timeliness of reviews? If that is the problem then I really don’t think that a multiple submission process will improve it for the same reasons noted by Kim. We want to ease the reviewers’ costs (here I am thinking like an economist) not multiply them. Unless you offer some sort of incentive to offset the costs of doing more reviews, I’m afraid that a multiple submission process would only worsen this and not make it better.
This was my point about law reviews. One reason law reviews use student reviewers is to deal with the inevitable problem of finding so many quality assessors. It would be too costly, both for the journals and for the entire profession, to find that many peer reviewers and so they just have smart students do it. I don’t think the outcome of this is a very good one though. Many papers that have academic value often get overlooked because they don’t fit with what students learn in the classroom (i.e. what’s the prescriptive value of this paper??). Just imagine how weird our journals would become if we had smart grad students serve as the filtering mechanism. I realize you’re not suggesting this as a solution Fabio, but I think that opening up the review process to multiple submissions will force editors to use highly ineffective filtering mechanisms, if not student reviewers then perhaps time-drained associate editors who do nothing but make gut decisions based on abstracts. Not good.
LikeLike
brayden
June 8, 2007 at 11:47 pm
To Omar: I think your analogy to the art world is highly flawed because they use multiple submission. People routinely submit the same portfolio to multiple galleries, curators, and buyers. You can think of a commercial gallery show as a simultaneous submission to thousands of people, who actually haggle over who gets the art if its popular. The art world is in many respects is the ultimate “multiple submission” venue- that’s why it’s so intense… Discussion Q: How would life be like if we put our papers in public and let editors pick them?
Also, Omar suggests that I am acting normatively and not empirically. Au contraire, if you look at the academic world, you see wide variation in evaluative practices: multiple submission (books, law reviews, grant agencies), single submission (journals), double blind review, single blind review, non-blind review, etc. There’s a lot of variation that should make us think our own arrangements are sub optimal. I would suggest that Omar is acting like the economist – you assume existing institutions are the equilibrium state! I say: let’s look around and see if we can do better.
To Kim: I think you are correct to think that reviewers might be overloaded in the mult sub system, but it’s the same story right now, as I learned when I was on staff at two major journals. You would not believe how incredibly hard it is to extract reviews from profs. It is not unheard of to wait a year from places like SF or Gender and Society because a lot of people are lame. So my guess is that it probably couldn’t get much worse in mult sub and we could speed up the process considerably.
To Bryaden: You are right that law reviews farm out work to students and that is bad. I don’t have a great response, but my sense is that the single editor system is to blame. Many editors do everything themselves, and mult sub would kill them. However, I have noticed that faster journals tend to have a strong deputy system/editorial collective in place. I think this could easily take up the slack. Also – and this is a pet peeve from working at journals – a lot of people volunteer for editorial boards but flake. If people new that they had serious obligations, I bet they could easily take up the slack created by multiple review. You wouldn’t believe how many editorial board members do nothing.
LikeLike
Fabio Rojas
June 9, 2007 at 12:36 am
I’ve just used the law review submission process for the first time. I stopped at 30 reviews, even though I got this little message that basically said, “Are you sure? People who submit to 80 are X more likely to get a good answer!”
I think Fabio is waay underestimating the tight link between the manageability of multiple submission and the fact that law reviews run on the backs of an army of student labor with a massive status-competition incentive to work their tails off.
In my field, I’d say that there are four journals among which I’d be basically indifferent at the very top, and then another 6-8 at the next tier down. You’re talking in terms of submitting to 2-3 journals at the same time; what if it were 6 or 8 or 14? I’m with Kim and Brayden– I think multiple submission would take a basically-functioning system and break it. It could only function with MUCH more first-stage editorial screening, and that makes the whole system less peer-review-based.
LikeLike
Jacob T. Levy
June 9, 2007 at 2:56 pm
[…] dynamics, and vicissitudes of peer review have always been a hot topic around these parts (e.g. here and here). In fact, one of our most famous posts consists of an indecent proposal to reform the […]
LikeLike
on peer review and butterflies « orgtheory.net
December 3, 2009 at 12:35 am