are reviewers too powerful?


Yes, says Nicolai Foss at Organizations & Markets – see his post.

Update: The post also references this interesting paper by Bruno Frey on “intellectual prostitution.” Abstract below:

Survival in academia depends on publications in refereed journals. Authors only get their papers accepted if they intellectually prostitute themselves by slavishly following the demands made by anonymous referees who have no property rights to the journals they advise. Intellectual prostitution is neither beneficial to suppliers nor consumers. But it is avoidable. The editor (with property rights to the journal) should make the basic decision of whether a paper is worth publishing or not. The referees should only offer suggestions for improvement. The author may disregard this advice. This reduces intellectual prostitution and produces more original publications.

Written by teppo

August 27, 2006 at 11:42 pm

Posted in blogs, research, teppo

10 Responses

Subscribe to comments with RSS.

  1. This is the perennial complaint (for lack of a better word) of all academics. It has become somewhat ritualistic to whine about reviewers; of course, the issue is usually put in pseudo-altruistic terms of “damage to the discipline” versus of damage to, let’s say, my career.  It is unclear however whether all-powerful editors vetoing reviewers recommendations according to inscrutable and idiosyncratic criteria is really the answer.  Hayek 101 says that the distributed intelligence of many people should ultimately (in the long run) be a better judge of quality than the narrow judgment of a single person.  In my view the editor should listen to as many people as possible before making a decision:  multiple reviewers, the resident editorial board member who is a expert in the subfield and (for general interest journals) a non-expert who should rate the paper in terms of the possible interest that it may arise among non-specialists.  The editor of course, should serve as a sieve for the many recommendations, highlighting the ones that she thinks the author should follow and the ones that she deeems should be discarded.  When it comes to judging quality and deciding what gets into journals, I think that many cooks improve the broth.



    August 28, 2006 at 12:40 pm

  2. But you are forgetting the moral-hazard problem. Anonymous reviewers have weak incentives to write careful reviews. A critical point of Hayek 101 is that efficient exploitation of distributed knowledge requires property rights (Frey’s point). Nicolai’s claim is not that editors should not seek outside opinions, but that editors should take responsibility for the final decisions. An alternative, of course, is to require reviewers to disclose their names.


    Peter Klein

    August 28, 2006 at 2:40 pm

  3. The problem with most of these critiques of the peer review system is that they pass around the designations of “braindead editors who slavishly follow reviewer recommendations” (everybody has an anecdote of course) like it was a hot potato. We are asked to believe that Nicolai is not one of them, but it is just those other unnamed others. Yet, in my discipline, every introductory editorial by an incoming editor promises editorial autonomy, and endless search for creativity, an attempt to push the boundaries and blah, blah, blah. In other words every editor thinks that she’s independent, it is only the authors that think that she is a empty-headed moron who is making him follow those ridiculous reviewer recommendations.

    Which brings us to the crux of the issue, which is not property rights, but is judgment of creativity. Frey is kidding himself if he really thinks that the worth of a paper is given by what the author thinks of his own work. Thus the distinction between autonomous authors and “other directed” (who knew that the old Riesmann typology was still alive?) prostitutes is tenuous at best (and the suggestion that it might be genetic is risible of course). A lot of “autonomous” people out there are just loony megalomaniacs (ever been to the meetings?) that are plain WRONG. Thank god there are reviewers out there that cut them down to size.

    Further, the recommendation that academics should be more like artists is one of the funniest things that I’ve ever seen written; especially since he gives the Renaissance (the period in which artists had the least control over their work, anecdotal tales about Michelangelo notwithstanding; [Baxandal is the definitive work here]) as the golden era of creativity. Yet, (modern) artworlds are no different from science in this respect (as the studies by Harrison and Cynthia White demonstrate). There, the “reviewers” are the critics, and where morose complaints about “true creativity” being demolished by brainless art critics and powerful dealers are legion. So if Frey thinks that economics should be more like real artworlds (and not his Utopian version of it) then it already is.

    Thus, ultimately in no system of cultural production that has ever been observed, is the creator the ultimate judge and jury of the worth of his own work (The Perelman controversy in Mathematics is a case in point). It is the community of creators, especially in fields such as science and art in which there is no distinction between creators and consumers, since most producers produce for other producers. Thus reviewers have all the right to tailor the product to their own taste since it is they who are the ultimate consumers (which means that they don’t look at reviewing as some exogenous imposition produced by an entity to which they have no attachment [the property rights argument], but as a great opportunity to have a say in a product that they will ultimately consume). There is no separate population of consumers that are not also reviewers or potential reviewers. Reviewers therefore, are closer to those focus groups that watch an early version of a movie and recommend a different ending; if the ultimate end is to satisfy the consumer, then the system is working just as planned. The studio (i.e. editor) makes the director (i.e. author) put in the different ending under serious protests (i.e. Frey). The director, if powerful and successful enough, then gets “final cut” rights put into her next contract. What are final cut rights in academia? It’s called a trade paperback.



    August 28, 2006 at 4:39 pm

  4. Omar, I strongly disagree with your last paragraph. The level of aggregation is too high. Reviewers, producers, and consumers of art or science are not homogeneous groups. I have reviewed many papers that I would never “consume” as a reader. The literature is simply too specialized, and researchers too heterogeneous. Rather than members of a focus group, many reviewers are like voters in New York deciding what tax or immigration policy Californians should have.


    Peter Klein

    August 28, 2006 at 6:05 pm

  5. Then the heart of the issue is not necessarily slavish editors following terrible recommendations from reviewers but editors (or members of editorial boards) who are not picking the right reviewers. I can see how such a mismatch might occur when submitting to a generalist journal (where the audience is the indistinct “mass” of the discipline in general), but I am hard pressed to think of how a paper (especially in economics a field that we usually enviously look at as the prime example of “paradigmatic consensus”) can go to a reviewer that is neither a) a specialist in the subfield in question and/or b) not part of the potential pool of readers that the paper would have if it were published. I bet that a lot of the grief produced by the review process comes precisely from this type of mismatch were reviewers recommend things that “sound crazy” or that violate the author’s most cherished ideas, precisely because there is little cognitive overlap between the reviewer and the author. When reviewing a paper, I have found myself many times writing in those “confidential notes to the editor” a paragraph that includes the sentence “readers of sociologia obscura would not be interested in this paper” implicitly counting myself as one such reader.

    I have always seen the role of editors as much closer to TV “network execs” than anything else. Their job is to put on the “best lineup” possible given supply side constraints, and the role of authors is as creative artists dependent on the culture industry (and ultimately the “public”) to survive. There is a conflict of interest between gatekeepers (such as editors) and creators, but it is not between the creative vanguard and the repressive system, but one between the editor’s judgment of what the audience would like (citation counts are the closest to ratings that we have here so the editor is expected to move the author toward making her paper look more and more like previously succesful ones) and the author’s comittment to her artistic/scientific vision (which might involve breaking with established molds or a complete rejection of audience reaction as in avant garde movements in the arts). However, what is forgotten here is that true creativity is EXTREMELY RARE (and its correlation with “autonomy” may in fact be zero). Authors raving about how incredible their work is are a dime a dozen, but the fact is that when subjected to the scrutiny of the entire community their work is “not all that.”

    In for-profit cultural industries, in non-profit artworlds and in science, the author always loses or at least has to bend to the tyranny of the market (and the audience). Anecdotes of unrecognized genius abound, but the mundane fact is that for the most part (and surprisingly given the allegedly anarchic way in which reputational capital is distributed in science) genius is recognized and provided with the appropriate rewards. Horror stories about everything going to hell in a hand basket, “increasingly” boring papers, “increasingly” boring art, etc. are a constant however, and keep us all entertained.



    August 28, 2006 at 7:03 pm

  6. Peter, this entire discussion has just reminded me that I have a couple of papers sitting on my desk that I must hammer down into gray, indistinct mush.

    One other plug that I have for the current review system is the fact that it is great therapy and very cathartic (for the reviewer)! Without peer review we would all be a little more insane. Reviewing a paper can be good for your health as you get rid of all of that anger and stress and take it out on the paper (each paper is probably worth about one day of a relaxing vacation).

    BTW, one of the best things ever written on peer review was the last editorial by Camic and Wilson in 2003 (former editors of American Sociological Review) entitled Authors and Reviewers: in Appreciation. The highlight:

    Judging by the war stories sociologists tell one another, the author-reviewer relationship looms as a major trouble spot of the academy. The litany of complaints is long and disturbing. Seen from the angle of the author who has submitted a manuscript to a journal, reviewers are criminally slow to complete their evaluations, willfully inattentive to what the manuscript actually says, careless and corrupt in their judgments, and perversely inclined toward frivolous criticisms that distort the manuscript’s great strength: Where did the editor’s ever come up with these clowns? Seen from the angle of reviewers, authors are publication-crazed drones who submit papers still only half-baked, lack a self critical eye, overinflate the significance of their work, chronically aim either too high or too low, and obstinately fail to take good advice, even when it is spoon-fed. Why do the editors ask busy scholars to waste time reading this dreck?

    Since we all have already had the chance to be both clowns and drones, I ask: can’t we all just get along?



    August 28, 2006 at 7:11 pm

  7. I just have to point out the irony of this debate. A sociologist argues that the peer review process is fairly efficient in discriminating between good and bad ideas, and a Hayekian argues that the system is structurally flawed.

    I find myself agreeing with both of you on most of your points. I’d only like to add that many of the grievances created between reviewers-authors exist because:

    a) the journal has a particular theoretical or methodological and editors pick reviewers that favor that bent;

    b) the author fails to frame the argument properly in the first couple of pages and ends up drawing the wrong kind of reviewers;

    or c) the journal is meant for publications of a general disciplinary interest and the paper is better suited for a specialist journal.

    Each of these problems could be avoided more regularly if authors were more careful in framing their papers and targetting the right journals.  But I think that Nicolai’s larger point may be that certain generalist journals may end up having too much of a particular theoretical bent, which means that a great deal of research gets filtered out and ends up in specialist journals.



    August 28, 2006 at 7:46 pm

  8. I thought your readers would enjoy the irony. :-) Seriously, though, the reason I don’t regard the market for academic journal publications as “efficient” is because the major player is government. Higher education is of course massively subsidized by the state. Most of the consumers and producers described by Omar are employees of state-subsidized universities, the major purchasers of journal subscriptions are state university libraries, and so on. I wouldn’t expect efficient outcomes in such a market any more than I would expect them in, say, the Chinese electricity sector.


    Peter Klein

    August 28, 2006 at 10:20 pm

  9. One reason why I don’t buy the massive ineffeciency hypothesis is that most of the empirical implications of such a view are simply not confirmed, at least not for the field that I am expert on (sociology), but I am very skeptical that my wild induction would fail for economics. If the massive inefficiency hypothesis was correct, we should observe an inordinate amount of really good papers (or papers that are judged to be really good after some time by the relevant audience) being published in obscure places. However, (once again speaking for the field that I know well) that is not even close to being the case. I do, however, on the suspicion that the market might indeed fail, regularly scan the issues of less prestigious journals in sociology. Every month I do read one or two articles from such outlets. I find that those articles are, well, not that good. The articles that appear on AJS and ASR or ASQ are on the other hand pretty good. In fact, I cannot think of a ground breaking paper (a paper for instance that I feel that I “have” to cite) that has not been published in the discipline’s top three journals. “The Iron Cage Revisited” appeared on ASR, “Institutionalized Organizations: Formal Structure…” appeared on AJS, “Educational Organizations as Loosely…” appeared on ASQ, and so on, and so on, and so on. In fact I dare any of the sociologists out there to point me to that crucial paper that was relegated to the rinky dink journal of quarterly garbage, but that served to reorient the discipline.

    Frey also makes it sound as if working papers are these amazing fountains of creativity and edginess and published papers are pale shades of their former selves with all of the creativity squeezed out by the evil reviewers. If this is correct, Google Scholar should be teeming with untapped sources of creativity and wisdom. However, this is clearly and exageration, Google scholar is full of plain old papers on plain old topics in economics or sociology and when a paper does appear in published form, it is a rule not dramatically different from its working paper version.



    August 29, 2006 at 1:22 am

  10. Omar, I think your conjecture about the most important articles consistently appearing in the top handful of journals would definitely not hold for economics. Of course, it is hard to define “important” consistently across subfields. But the top general-interest journals in economics publish very few papers in particular subfields (organizaitonal economics, for example).

    Fortunately, it should be possible to address this disagreement empirically, by comparing impact scores for unpublished papers on SSRN, Google Scholar, etc., with the citation counts of those papers once they are subsequently published, weighted by some quality measure of the journals in which they appear. Sounds like a good project for an enterprising graduate student.


    Peter Klein

    August 29, 2006 at 2:38 pm

Comments are closed.

%d bloggers like this: