orgtheory.net

a modest proposal: triple blind review

Fabio

In academia, we have single or double blind review. In the first case, editors and referees know the identity of the authors, but authors don’t know the referees. This is common in grant proposals, book evaluation and in some sciences. We also have double blind review – authors and referees don’t know each other’s identities. This is supposed to reduce bias. But we all know that the editor could still be biased because of the author’s status. So why not institute a new form of refereeing?

TRIPLE BLIND REVIEW – the authors anonymously upload the paper to the journal website and the editor handles the submission without knowledge of who the author is. The author is identified only by number. Communication happens through the website.

If the editors know the author before hand, or the author is so well known that they can’t hide their identity, then this won’t work. But it would probably maintain confidentiality in about 90% of cases. But what if the author contacts the editor and biases the process? Or the editor and author are friends? Then we institute:

QUADRUPLE BLIND REVIEW – No one knows the editor of the journal. The editor is appointed by a board, like the ASA publications committee, but is kept secret. It’s all done through the journal web site.

Once again, bias could creep in. Maybe readers will only cite works by previously famous scholars. What if we went one step further?

QUINTIPLE BLIND REVIEW – Even after publication, the author’s name is kept secret, at least for a few years. You would cite the journal name and article number.

You would search for good articles, not articles by established scholars. But there is one level of anonymity that we can still use:

SEXTUPLE BLIND REVIEW – The name of journal itself is kept secret. You know nothing about the editor, reviewers, authors or the journal’s sponsoring organization. It’s just a web site with articles and you would read them looking for good articles.

This last proposal is nifty – it’s a direct attack on the monopoly of status possessed by publishing houses, professional associations, and established academics. Anyone could create a website that published articles and only the quality of the articles would matter. If readers have responses to my proposals, you can email me, anonymously, and you’ll get feedback from a super secret panel of high quality reviewers.

Written by fabiorojas

January 23, 2007 at 5:56 pm

Posted in academia, fabio

28 Responses

Subscribe to comments with RSS.

  1. Hmmm…. What some people call a “monopoly of status possessed by publishing houses, professional associations, and established academics,” other people call “editorial judgment.” Super-secret, X-tuple blind review seems suspiciously close to what Leland Yeager calls second-handism.

    Under sextuple-blind review, does the reader get to know the URL of the website? If so, why not try a septuple-blind process in which the reader types a keyword into Google and clicks the “I’m Feeling Lucky” button. Instruct the browser to mask the linked site’s URL, so all the reader sees is pure, unadulterated text, without any distracting external markers. Yummmm!

    Like

    Peter Klein

    January 23, 2007 at 6:40 pm

  2. Somehow it seems that ownership and associated incentives/property rights (Nicolai?) would be important in all this.

    Like

    Teppo

    January 23, 2007 at 6:44 pm

  3. Peter: Maybe septuple blind review is the wave of the future. Dump all research into google and google’s search engines will match editors, reviewers and readers. Since it is mathematical in nature, we just have to monitor for hackers…. Sounds appealing to me… Peter, maybe you’ve had a “peanut butter and chocolate” moment…

    Teppo: On a serious note, I do think that ownership of ideas seems crucial in the academic system. A lot of great research is motivated by prizes, fame and prestige. Not to say that we don’t love knowledge, but rewards sure do matter a lot. Maybe X-tuple blind review is an issue diminishing returns: eliminating bias is important, but too much secrecy undermines the incentives in the system.

    Like

    Fabio Rojas

    January 23, 2007 at 7:17 pm

  4. Of course, it would all come crashing down unless people blinded their CVs as well. At least that would make the end of the year reviews a lot simpler. “I published 14 books and 37 articles this year. I can’t tell you the titles, co-authors, journals, editors, or publishing houses, but it was really great stuff, I swear!”

    Liked by 1 person

    Dan Myers

    January 23, 2007 at 9:25 pm

  5. Dan – You know there’s an easy solution: the Blind Review Reporting Service (BRRS). In the X-tuple blind system, computers would keep track of who submitted what via social security numbers. Then, come promotion time, each person would pay for a BRRS report that states the articles & readership/citation statistics associated with a certain social sec #. Just like the SAT – blind test writers, grades and responses – but you can order verifiable information via BRRS. Dystopian? Maybe. Progress? Definitely!

    Like

    Fabio Rojas

    January 24, 2007 at 2:41 am

  6. Fantasitic Idea! Wanna go into business together doing it? I’m thinking per-report costs could be something like $5000. You do want tenure, don’t you? And, we could provide the service for job candidates too; and Grant/Fellowship Applications! Damn! This thing is a gold mine!

    Like

    Dan Myers

    January 24, 2007 at 4:57 am

  7. Dan, this is exactly why they put you in charge. You’re a man of ideas!

    Liked by 1 person

    Fabio Rojas

    January 24, 2007 at 5:09 am

  8. I think this is a perfectly pragmatic and workable proposal except for the part where you imagine something being known by the ASA Publications Committee and being otherwise a secret.

    Liked by 1 person

    Jeremy

    January 24, 2007 at 7:58 am

  9. People’s writing styles can be quite distinctive, so I think another layer is required, with papers being put through a processor that renders them into Basic English (you know, the one with the 600 word vocabulary) or some equivalent. Maybe just MS Word’s style/grammar checker would be enough to ensure sufficient homogeneity.

    Like

    Kieran

    January 24, 2007 at 11:05 pm

  10. why not a step further: a machine would generate papers. those papes would be run through some n-tuple review process. prestige scores would then be randomly be assigned to the URLs at which the papers are posted and authors would be randomly assigned to papers. everyone wins!

    Like

    blindman

    January 27, 2007 at 12:51 am

  11. Here is a problem with sextuple review: eliminating this much information from the process also does not allow people to easily sort through research. There is a ton of stuff out there, and I don’t have time, nor do most scholars, to do the sorting myself. If the articles are not previously sorted for us by Author/publication, which signals the quality for us, then some other mechanism will take its place quite quickly, with people doing the signaling mechanisms through citation lists or something. Instead of journal prestiege, you would have citation list prestiege. And then you have your bias problem all over again.

    Like

    Steven McMullen

    February 3, 2007 at 10:51 pm

  12. I want to know who wrote the article so I can possibly collaborate with him/her on future work. Scientific work is highly social, much of it presented in person at conferences, discussed over breaks, at poster sessions, and over drinks afterwards. I submit that this social aspect is a far more powerful quality control mechanism than anonymous review. Only authors at the margins of credibility have any serious trouble with the review process. It is in proposal review–the fight for resources before the work has been done–that anonymity is more critical. Here the stakes are higher, the reviewers less well-match to the material, and the risk of personal bias greater.

    Liked by 1 person

    Paul

    February 4, 2007 at 12:10 am

  13. I know this is partially a joke, but even double blind review is impossible these days, at least in Economics. By the time a paper reaches journal stage, it’s already well-known through conferences, talks, SSRN and EconLit. If you’re qualified enough in a field, say evolutionary game theory, to review a paper, you’ll surely know who wrote it by the time it hits your desk. And even if you don’t, I think it’s well known that reviewers routinely Google the paper to see who wrote it.

    Let’s just drop the charade of double-blind altogether.

    Liked by 1 person

    cure

    February 4, 2007 at 1:25 am

  14. […] editor responsible, the journal and eventually the author if it is published. Suffice it to say, this post suggesting how to cover it all up amused me. It presumes that people will produce research for the […]

    Like

  15. The reply I thought most thoughtful was

    ” Of course, it would all come crashing down unless people blinded their CVs as well.”

    So in the new world, all papers will be assigned a number at the Google registry. All web pages containing information about the paper will embedd this number in a special html tag .
    Every person using a browser will also have an identity and this will be used in all reviews.

    Whenever a CV , journal, or citation database is viewed online, the person browsing will be allowed to see only those paper that they have not reviewed or edited. This will be called “blind browsing.” Google may substitute ads for the ommitted papers.

    Like

    Charles Petrie

    February 6, 2007 at 5:29 pm

  16. […] channels” and in offline sources (read: over a beer at a conference). There was a great tongue-in-cheek post about this over at Org Theory a while back, suggesting sextuple-blind review, which the greatest […]

    Like

  17. […] blind review: A few months ago, I suggested that the editor not know the identity of the author to reduced possible bias. It’s easy to do and would help reduce incumbency […]

    Like

  18. […] a triple blind review process would eliminate problems with editors? Or, if all of this doesn’t work, we could still go […]

    Like

  19. Eliminate journals, publish everything on one website, publish multiple reviews, multiple ways to rate and rank articles and the reviews of them, allow anonymous or semi-anonymous reviews

    Like

    Michael Bishop

    April 28, 2008 at 9:10 pm

  20. Come on, if you strip out even the journal name, you remove all sources of accountability. Outright fabrications can stand side by side with true research, and no one can trace the process.

    Liked by 1 person

    bah

    June 13, 2009 at 3:42 am

  21. […] process overall, though I am guessing it introduces some bias.  Even Fabio’s proposal for triple blind review does not solve the problem of potential biases (one way or the other) associated with knowing the […]

    Like

  22. […] around these parts (e.g. here and here).  In fact, one of our most famous posts consists of an indecent proposal to reform the peer review system. Usually clarion calls for radical revolution in the peer review […]

    Like

  23. […] Triple blind peer review. […]

    Like

  24. […] had a lot of debate here about how to improve the review process, which has included some modest and crazy proposals. Open peer review is one potential solution to the problems inherent in the […]

    Like

  25. Each discipline should build a series of computer algorithms to process papers as input, rank them on a scale of 100, and then organize them into subfields so they may be easily searched and easily distributed. Then no more waste of human effort!

    Like

    Tom

    October 17, 2011 at 2:29 am

  26. Valuable information! Looking forward to seeing your post

    Like

    achat kamagra

    November 29, 2011 at 11:47 am

  27. […] ciego en las revistas científicas tiene sus seguidores y detractores, (hay quien propone,  el triple, cuádruple y séxtuple ciego ). No es el objeto de este post hablar de las bondades del sistema, sino todo lo contrario. El […]

    Like

  28. […] ciego en las revistas científicas tiene sus seguidores y detractores, (hay quien propone,  el triple, cuádruple y séxtuple ciego ). No es el objeto de este post hablar de las bondades del sistema, sino todo lo contrario. El […]

    Like


Comments are closed.

%d bloggers like this: