is blind peer review an illusion?
So, I’m reviewing a piece for a journal today. I wondered: is blind peer review an illusion? First, the sub-circles that many of us hang out in often are so small that one is likely to have seen the piece presented previously (probably worth sending the editor a note if you know the authors). Second, with tone, citation patterns and sub-topic one can often pick out, say, a Lizardo from a King. Third, it can be all-too tempting and easy to google the title of the manuscript, and more often than not one is likely to find the authors. I don’t know how widely “googling” is used by reviewers, it would be interesting to find out via an anonymous survey.
I think journals are doing various things to address the above matter; for example, requesting authors to remove their papers from web sites. But with SSRN, conference posting of papers, etc, blind review might now be an illusion. I don’t know how this affects the reviewing process overall, though I am guessing it introduces some bias. Even Fabio’s proposal for triple blind review does not solve the problem of the potential biases (one way or the other) associated with knowing the author(s) of an article that one is reviewing.
orgtheory.net 
I don’t say it’s an illusion. Lots of papers are written by people who don’t bother to post them on websites, or they are rewritten so they don’t have the same title anymore. So maybe it’s 50% of an illusion. Of course, with “big projects,” you can guess who it is.
LikeLike
fabiorojas
July 14, 2009 at 7:32 pm
Here’s one data point: I’ve googled papers I’m reviewing – I’ve googled a lot worse too.
One time I reviewed a paper, but couldn’t find the person on google, however, in the paper he/she used a turn of phrase about a specific book that was also in a review of my paper that I just got back. In other words, I was able to figure out from the paper that I was reviewing someone who just reviewed me!
LikeLike
musa
July 14, 2009 at 7:34 pm
about 50% of the reviewing i’m doing these days seems to be from single-blind journals. and i really think that’s the way to go, because it seems to me, even if only 50% of the time, that yes, it is an illusion.
LikeLike
shrinkingisaac
July 14, 2009 at 8:21 pm
So figure this one out…
I was just asked to review a paper for a journal that uses the web-based editorial manager system. I was very surprised to see that in the system there were action links to do a google scholar title search and a google scholar author search.
It’s like the editors know people do these searches and are OK with them doing them. I pressed the links to see what they did, and sure enough, they are what they say they are. And I never do such searches on my own in an attempt to remain blind. Should I now feel like it’s a part of being a referee to do these?
Also, I don’t understand why there was a google author search when it is supposed to be double-blind. A title search I understand because the referee knows the title, but an author search? When I pressed the author search, I could see the author’s name in the search result line. So they just revealed to me the name, which I wouldn’t have even known, because the google scholar title search revealed that the paper is not online. I wouldn’t have known the author had I not used the journal’s link!
LikeLike
Mike McBride
July 14, 2009 at 8:26 pm
Merton’s theory of the institutionalized imperatives of science has been on my mind watching the Sotomayor sessions in Congress. I take it that his point is not “to what degree are these norms followed” but “to what degree are they binding upon participants as members of the community?”
LikeLike
Jenn Lena
July 14, 2009 at 8:30 pm
I have to confess that I, too, google paper titles that I review. However, I am strict in that I only do it after I have completed my review and am just fulfilling my own curiosity (and gathering useful information, such as who else is working in the field).
Perhaps more interesting, I once learned the identity of another reviewer on a manuscript I reviewed due to the phrasing of a comment on this blog.
LikeLike
cwalken
July 14, 2009 at 9:01 pm
It never occurred to me (before now) to google a manuscript title… maybe that’s a generational thing as well as a question of ethics/norms?
But often I do know who at least one of the authors is, and I’ve both recused myself and been happier to review the paper, depending on whether my bias was negative or positive (respectively). Even so, I don’t think that I’ve consciously been more positive about a paper because I had a hunch who wrote it. Probably like all of those women evaluating women playwrights, I’ve raised my standards (and then tried to be even more helpful?)
Sometimes I ‘salt’ my reviews with tasty vocabulary words (e.g., incipient, polysemous), superclever turns of phrase, and when I can I try to offer clever names for variables— then I get a big silly kick out of seeing any of those turn up in a revision or the final publication.
LikeLike
CV Harquail
July 14, 2009 at 9:10 pm
[…] think we’ve used the handy survey feature here at orgtheory before, so here goes. So, with reference to the previous post — here’s an unscientific and anonymous poll (don’t worry: it’s anonymous, […]
LikeLike
anonymous survey: do you google the papers you review? « orgtheory.net
July 14, 2009 at 9:28 pm
shrinkingisaac: Most organizations/mgt journals are double-blind — Management Science was the exception, though it recently switched from single to double blind.
LikeLike
tf
July 14, 2009 at 9:45 pm
I’m sorry, but I believe the illusion lies in the idea that bias can really be stripped from the flawed peer-review process.
And yes, I admit I’ve Googled…but after the fact just to confirm a suspicion.
LikeLike
Angela
July 15, 2009 at 1:37 am
I feel so idealistic; It never occurred to me to google for the paper title or to try to figure out the author (although at least once I recognized the author from the style, perspective, data and citations).
LikeLike
Rebecca
July 15, 2009 at 4:00 am
I believe that the important part of the process is the anonymity of the reviewer. People will criticize their friends under the cloak of anonymity. If you know your field, you know who is doing what in it, there is no way to avoid that unless you prefer that papers be reviewed only by people with no expertise in the area being reviewed.
I have Googled papers after reviewing them, not before. I’ve done this when the paper was very good and influenced my thinking, and wanted to attribute the ideas to the correct person. Otherwise you are at risk of using that idea without giving proper credit to the author.
LikeLike
olderwoman
July 15, 2009 at 1:23 pm
Blindness can be an illusion on occasion (does that even make sense ;-) but it is still very important. If an author is someone I know well, I always ask the editor if I should proceed. They’ve always said to continue with the review unless I feel uncomfortable doing so. As others have noted, its a small world and hopefully manuscripts come to us because we have the right expertise to help bring out the contribution.
Now, some of my colleagues have had awkward experiences with single-blind journals. One, a Williamson student, got a comment from a reviewer as follows: “Ollie can get away with a statement like that but a rookie like you cannot.”
In other words, the single blind process may encourage reviewers to actively evaluate the manuscript based on who the author is. For that reason, I feel that even if double-blind is partly an illusion, the norms it helps to promote are very real.
LikeLike
Russ Coff
July 16, 2009 at 7:24 pm
[…] and vicissitudes of peer review have always been a hot topic around these parts (e.g. here and here). In fact, one of our most famous posts consists of an indecent proposal to reform the peer […]
LikeLike
on peer review and butterflies « orgtheory.net
December 3, 2009 at 12:41 am
[…] of online conference information (sometimes including papers) and internet search engines. This question came up at Orgtheory a while back, with the definitive follow-up poll suggesting that most people do look […]
LikeLike
“Blind” peer review « Memoirs of a SLACer
February 21, 2011 at 4:00 am