orgtheory.net

sociology IS a combat sport

Omar

One impression that one gets from the “economic theory of journal reviewers.” Is that since reviewers don’t have “property rights” to the journal that they are reviewers for, not only will they produce inferior recommendations (which may damage the discipline) but also that they don’t give a flying hoot about their job as reviewers. In other words, one empirical implication of this view is that reviewers don’t care nor do they have any emotional attachment or stake on their roles as reviewers. From this point of view, what’s wrong is that there is no true source of (egocentric) motivation for reviewers to do a good job. Not only that, while we should expect authors to be deeply distressed when things don’t go their way (since their careers depend on publications) there is no reason to postulate the existence of “think-skinned reviewers” since these people once again have no egocentric stake (property rights) on the journal for which they are called to perform this service. Their fleeting connection to the journal ends when the review is submitted. Who cares if the editor publishes the paper or not?

Last spring, I remember reading one of the most bizarre things in the world in Footnotes which is the newsletter that every member of the American Sociological Association receives in the mail every couple of months, under the “annual editorial reports” from each of the ASA journal editors. While each report was incredibly boring and humdrum (i.e. “we published 69 papers, rejected 85%, submissions were up, submissions were down, etc.), Ross Stolzenberg‘s report for Sociological Methodology was nothing short of surreal:

This has been a year punctuated by drama. Your editor seems to have encountered once again a small, previously unrecognized, nascent social movement that he calls the Thin-Skinned Scholar Movement (TSSM). TSSM serves the needs of scholars who object to publication of opinions that contradict their own. Your editor believes that the goals of TSSM are misguided, as his own professional fortunes have been advanced by the publication of debates about his own research. More important, disagreement is fundamental to scholarship, making the suppression of disagreements a fundamental violation of the purpose for which Sociological Methodology is published. Indeed; every paper published in Sociological Methodology includes clear statements of dissatisfaction with previous studies; it is this dissatisfaction with previous work that motivates and justifies the production and publication of new contributions.

Your editor is deeply distressed by the style of the TSSM. In particular, consider the following incident: Several weeks ago, I encountered a thin-skinned scholar, who was driving in his car as I walked to my own car in a parking lot. Apparently unimpressed by the writings of Miss Manners, this scholar opened his car window, loudly and repeatedly declared strong views about the composition of my head and the phylum in which I should be classified, and rapidly drove his car so close to me that it did, on the third such maneuver, brush against my pants. I wonder still, is this thin-skinned scholar just a talented and kind-hearted stunt-driver with unusual ideas about parking? Or does he reveal true malice, a will to evoke fear and a willingness to use his car to damage a pedestrian? These are questions that I cannot answer. But answers are suggested by his emailed statement (with copies to others) that he would be pleased to see my body lifeless and in pieces. More to the point, these are questions that no editor should have to consider. This thinskinned scholar has wasted great volumes of an editor’s time and effort, reviled the editor in numerous hostile email letters (with copies sent to a variety of others), delayed publication of Sociological Methodology, wasted hours of time by talented and highly-paid lawyers, and badly strained relations between an editor who sought to uphold the principles under which scholarly journals are published, and the ASA executive officer, who sought to save the ASA the expense and trouble of a lawsuit by an enraged scholar.

Apparently, even without any kind of instrumental stakes on the journal for which they review, some referees take their jobs as reviewers dead serious. How is this attachment to an external entity to be explained (rhetorical question of course, since this would not be a mystery for any sociologist)?

Written by Omar

September 6, 2006 at 5:15 pm

18 Responses

Subscribe to comments with RSS.

  1. [S]ince reviewers don’t have “property rights” to the journal that they are reviewers for, not only will they produce inferior recommendations (which may damage the discipline) but also that they don’t give a flying hoot about their job as reviewers.

    Um, no, that isn’t the claim at all. It’s that reviewers without property rights will exert less care, on the margin, than reviewers with such rights. (If you like, you could even add “for given levels of intrinsic motivation.”) It’s straightforward marginal analysis, nothing more. I don’t see how the claim is even controversial.

    Like

    Peter Klein

    September 6, 2006 at 5:46 pm

  2. I don’t think that the positive claim that the addition of property rights would increase reviewer effort is controversial either. My target was the allied claim that in the the absence of something like property rights reviewer effort would be uniformly subpar. Of course, the standard way of explaining heterogeneity in reviewer effort in the absence of any apparent instrumental incentive would be to postulate something like “intrisic motivation.” But that begs the question of whether high degrees of reviewer effort and care (or psychopathic zeal) are not already being induced by something other than property rights.

    Like

    Omar

    September 6, 2006 at 6:07 pm

  3. I have a different perspective on reviewer effort. As a former intern at AJS and soc meth, I worked a lot in the journal system. My view is that yes, incentives would improve reviewer response time and quality, but not much because most journals could not pay reviewers enough to make slogging through a paper a high priority (compared to teaching, writing your own papers, dealing with family, etc).

    My hypothesis is simply that reviewer response time is primarily a function of (a) professional norms and (b) personality.

    (a) I have noticed that time to review is heavily dependent on discipline, speciality and time period. For example, reviews were very quick (maybe a few months) until the early 1990s or so. Now, reviews in sociology are medium to long (with some reviewers waiting a year). Some specialties (ethnography) seem to be populated with slow, cranky reviewers while others (orgs) seem to be filled with thoughtful, efficient reviewers.

    (b) Some scholars are simply unprofessional. Any experienced editor knows that certain professors are dependable, while others are flakes or they write atrocious reviews, and you try to save the good reviewers for good papers. And there seems to be nothing at all you can do to make people do reviews because many scholars consider reviewing to be an activity not worthy of their time. Smart editors cultivate good editorial boards and dependable “go to” people to help manuscripts that have stalled in the review process.

    Editor’s can’t do much to establish the norms of their field, but they can actively good relations with dependable people and keep a careful eye on the stack of papers at the journal. It’s my opinion that the average time of review for any paper shouldn’t be more than three or four months. Journals that have many papers hitting a year or more in review are simply not working hard enough.

    Like

    Fabio Rojas

    September 6, 2006 at 6:39 pm

  4. I think we do have something akin to property rights in the review process – area-specific commitments and rights. Reviewers typically invest time in a review when it pertains to an area or subfield where they have invested a great deal of their time and resources doing research. In fact, the more invested a reviewer is in that particular domain of research, the more individual effort they should exert to assure that the quality of publications in that area remains high. If the publication quality declines, the area of study would soon lose its prestige and the reviewer’s own work would potentially be assessed negatively in the future.

    Sometimes however that incentive can become perverse, as illustrated by the car brushing incident described above.

    Like

    brayden

    September 6, 2006 at 7:23 pm

  5. And, let me add, that story is just absolutely insane!

    Like

    brayden

    September 6, 2006 at 7:32 pm

  6. My question is: which paper in Sociological Methodology (!!) could possibly have caused such a degree of animus? Could it have been “A New Approach to Estimating Life Tables with Covariates and Constructing Interval Estimates of Life Table Quantities”? Or perhaps, “Improved Regression Estimation of A Multivariate Relationship with Population Data on the Bivariate Relationship”?

    Like

    Omar

    September 6, 2006 at 7:34 pm

  7. In all seriousness, the lead article (which I am planning to read) in the 2005 issue is a piece on casuality written by none other than James Heckman. Perhaps that is the offending article. Sociologists get very heated when arguing about casuality.

    Like

    Omar

    September 6, 2006 at 7:36 pm

  8. Right on second guess, Omar. The alleged driving incident to which Stolzenberg referred is part of a larger drama involving three players: Stolzenberg, Heckman (the driver-author in Stolzenberg’s story), and Michael Sobel. Heckman was asked to contribute a paper on causality to Soc Meth, Sobel wrote a critique, and Heckman wrote a response. Both Heckman and Sobel accuse each other of modifying their papers after reading the other person’s rebuttals (thereby making their critiques obsolete or nonsensical) and accuse Stolzenberg of sharing drafts inappropriately. Heckman also accuses the press of bungling his contribution in the production process, among other sins.

    Heckman has a web site documenting his version of events, including the e-mail correspondence between himself, Stolzenberg, and the press. Judge for yourself.

    Like

    anon

    September 6, 2006 at 8:46 pm

  9. From Heckman’s website:

    “Lurid Fantasies

    I will not comment on the lurid fantasies posted on the symposium website except to say that I deeply respect life and would never destroy it or threaten to destroy it.”

    It’s reassuring to hear that our leading econometricians promise not to hurt anyone…

    Like

    anon2

    September 6, 2006 at 9:32 pm

  10. Academic outbursts – Hmm, luckily nothing similar to this happens in business schools – oh wait – it does, here’s the NYT on an incident at Emory, and here’s the case. After being at Emory for a year, I never got the full scoop on what exactly happened – with well-meaning folks defending both sides of the argument.

    Like

    Teppo

    September 6, 2006 at 9:41 pm

  11. Omar: This would’ve been an appropriate and entertaining place to wax knowledgeable about Bourdieu’s H-word, no?

    Fabio: Do you think your hypotheses apply equally well to predict review quality?

    Brayden: If reviewers are driven by a self-interest to improve the prestige of their subfield (and thus their own work), shouldn’t we also expect them to free ride on other reviewers in their subfield? I think Olsen would have more to say here about subfield size than reviewer investment.

    Heckman: I’m very impressed with the clarity and robustness of your website. In retrospect, do you think your professional (and personal) reputation could have been better maintained and time better spent by writing your next book instead?

    Like

    Jeff

    September 7, 2006 at 7:20 pm

  12. Jeff: Not really. Most people who have enough invested in a subfield to really care about its associated prestige probably don’t trust others to defend it as well as they could. Call it the ego-factor. Anyway, this probably accounts for a smaller proportion of the reviewer population than we’d actually like, which explains why so many reviews seem half-hearted.

    Like

    brayden

    September 7, 2006 at 8:01 pm

  13. Hi, Jeff: “Fabio: Do you think your hypotheses apply equally well to predict review quality?”

    I think so, but I would also add that career stage seems to matter a lot. Emeritus profs and beginner grad students, in my experience, don’t write the best reviews. In both cases, they aren’t up to speed on the most recent research.

    As far as personality and specialty, they probably have similar effects on review quality. Review quality – defined by clear and reasonable suggestions for revisions – also seems to vary by field. My colleague here at IU, Scott Long, tells me that as a Soc Meth editor it was hard for him to publish qualitative research because the reviews were never consistent.

    And also remember personality – people who believe reviewing is an important form of service are more likely to develop the capacity to efficiently write informative reviews. And senior scholars can back me up – how many times have you gotten good reviews after a year? Usually, the better (not necessarily positive) reviews come back in a few months. Extremely late reviews indicate that the person doesn’t consider the review a high priority and thus probably writes junky reviews.

    Like

    Fabio Rojas

    September 7, 2006 at 9:00 pm

  14. Jeff: the H-word scares people! You only use it if you must.

    Like

    Omar

    September 8, 2006 at 2:27 am

  15. […] Of course, my former grad student colleague, Ari Adut, now a professor at UT Austin, would not be surprised. He would say this is yet another example of how scandals necessitate the creation of new norms and practices.  After the Dan Rather blog fiasco, I knew it was only time that blogs would be used to initiate a real crisis. And for loyal readers, orgtheory has already covered one academic brouhaha. […]

    Like

  16. […] Ah, yes, we’ve heard before about the happiness levels of UofC. […]

    Like

    happiness « orgtheory.net

    November 28, 2006 at 3:12 pm

  17. […] Sociology is a combat sport. […]

    Like

  18. if sociology is a combat sport, then all the better. we have a lot of battles left to win.

    maybe this is just my graduate student naivete, but maybe it’s not for the worse that we start taking reviews personally. maybe the time for disinterested debate in passive aggressive op-eds has passed, and now is the era of real-world conflict.

    and doesn’t the principle of public sociology demand that we address conflict wherever we may find it? i say we stop looking for it and start causing it… bring the fight to the fighters, as it were. if cars-brushing-pants action is the most we’re getting right now, maybe we’re not really doing our jobs.

    (and if that is my grad student naivete, please let me down gently… it’s finals time and i’m wound really tightly…)

    Liked by 1 person

    joe

    December 2, 2009 at 6:24 pm


Comments are closed.