orgtheory.net

response to gelman on retractions in science

Last week, I argued that retractions are good for science. Thomas Basbøll correctly points out that retractions are hard. Nobody wants to retract. Good point, but my argument wasn’t about how easy it is to retract. Rather, it’s about the fact that science is exceptional in that it has a built in error correction mechanism.

In reviewing the debate, Andrew Gelman wrote:

One challenge, though, is that uncovering the problem and forcing the retraction is a near-thankless job…. OK, fine, but let’s talk incentives. If retractions are a good thing, and fraudsters and plagiarists are not generally going to retract on their own, then somebody’s going to have to do the hard work of discovering, exposing, and confronting scholarly misconduct. If these discoverers, exposers, and confronters are going to be attacked back by their targets (which would be natural enough) and they’re going to be attacked by the fraudsters’ friends and colleagues (also natural) and even have their work disparaged by outsiders who think they’re going too far, then, hey, they need some incentives in the other direction.

A few thoughts. First, fraud busting should be done by those who have some security – the tenured folks – or folks who don’t care so much (e.g., non-tenure track researchers in industry). Second, data with code should be made available on journal websites, with output files. Already, some journals are doing that. That reduces fraud. Third, we should revive the tradition of the research note. Our journals used to publish short notes. These can be used for replications, verifications, error reporting and so forth. Fourth, we should rely on journal models like PLoS. In other words, the editors will publish any competent piece of research and do so in a low cost and timely way. Fraud busting and error correction will never be easy, but we can make it easier and it’s not hard to do so.

Adverts: From Black Power/Grad Skool Rulz

Advertisements

Written by fabiorojas

July 17, 2012 at 12:01 am

12 Responses

Subscribe to comments with RSS.

  1. Fabio:

    Good point regarding science being different than other fields. That’s an aspect of your original post that I’d not picked up on. Another area in which there is effective fraud-busting is government. Fraud-busting in gov’t often comes from outside (via the news media and political advocacy organizations) and also from the separation of powers (legislative, executive, judicial). At the university (as with most organizations, I think) there is no separation of powers: there is only the executive, with minimal legislative and judicial branches. This creates a problem.

    Regarding your other point, I agree that in practice the fraud-busting is often done by tenured professors, but I disagree with your statement that it “should” be done by them. For one thing, I don’t know if Uri Simonsohn is tenured. For another, Mark Hauser’s improprieties were revealed by students (or maybe it was postdocs?) working in his lab. These are two biggies right there. For another, the vast majority of people at the university are not tenured profs. By restricting fraud-busting to this small fraction of people, you’re restricting the possibilities.

    I understand the practicalities but I am really really uncomfortable with the idea that fraud-busting should carry such a risk of reprisal that it can only be done by people who are not at risk. To me, it would be much better to work toward a world where students, postdocs, and junior faculty can make open statements without fear of reprisal from the Frank Fischers of the world.

    That’s one reason why I am so supportive of Basbøll’s efforts to spread the word on Weick’s plagiarism. If Weick wants to say he’s sorry, that’s fine, but when an ethical violation is revealed and then is shrugged off or even mocked by colleagues within the violator’s subfield, that’s not healthy, and it does not help in the development of a healthier climate for scientific inquiry.

    Like

    Andrew Gelman

    July 17, 2012 at 8:29 am

  2. One of the interesting things I learned by reading Michael Lewis’s The Big Short was that there was, at least initially, no straightforward way for the people who predicted the crash of mortgage bonds to express that prediction as a trade (which, of course, is what is supposed to provide the market with the “information” necessary to ensure that those bond prices and, by extension, house prices reflect their real value). As Lewis puts it: “the market for subprime mortgages simply had no place for people in it who took a dim view of them” (p. 29).

    I feel the same way about the stories that circulate in organization studies. I take a dim view of that story about the soldiers in the Alps, for example (it’s only one example), but there’s not a lot of room to express that view. Like the subprime market, then, we get a storytelling bubble. It’s much easier to introduce a story into the literature than it is to remove it. To put it in Weick’s terms, it’s easier to “affirm” a story than it is to critique it.

    I agree with Andrew that we should not limit critique to tenured professors and outsiders, but I think that’s more or less what is happening. After all, we’ve got mainly a (tenured) political scientist and two “perklempt” “Danish guys what’s his name and watchumacallit” arguing that something is amiss in sensemaking scholarship. Meanwhile, the estbablished org theory community waits silently to see how it will play out. (When not outright censoring the discussion, for fear of litigation!. Sort of like in the art world.)

    That’s pretty similar to what was going on in the mortgage bond market. It was simply an uncritical environment. They believed that “any old balance sheet will do” (so long as house prices keep rising!). As Lewis notes, “far too many people were taking far too many financial statements on faith” (p. 177), they were not practicing “due diligence” in assessing them (p. 130). One of the interesting dots I’m connecting in my research on this, by the way, is that Weick told the map story to a Wall Street exec in the mid-1980s. The two prominent men, a scholar (and editor of ASQ) and leader (treasurer at J.P Morgan), agreed that “any old strategic plan will do” (especially in a crisis). A couple of decades later, another scholar and another leader, this time in the pharmaceutical industry, would affirm each other in the same way.

    In this climate of mutual appreciation, people like me, just like the Big Shorters (Eisman and Burry), not only have to know that something is wrong (i.e., detect an error), we also have to invent, or re-invent, a form of critique through which to present it. Clearly, however, the sensemaking literature has been “affirmative” for way too long. Has there ever been a retraction in that field? Has anyone ever admitted that, say, 20 years ago we believed something about sensemaking that turned out to be untrue?

    Like

    Thomas

    July 17, 2012 at 11:26 am

  3. Holy mother of….someone just compared himself with the guys in the Big Short that made 100s of millions of dollars, risking it all, to win it all? Talk about hubris!!!

    And, once again, I have said it before: Whether or not the story is plagiarized, the theorizing that follows it can be just as as valid than stuff that is not plagiarized. This difference really needs to be acknowledged, I think, if you want to be taken seriously. Otherwise, it will just be way too easy to say: Ah well, it’s not as bad as you claim it is (as e.g. Omar did, basically, back then)

    And, a related problem: If all you (mainly Thomas) talk about is this plagiarism affair, people forget that he also did other stuff that you disapproved of, didn’t he? To be honest, I can’t even remember what it is…

    Like

    Anonymous

    July 17, 2012 at 12:40 pm

  4. Anon:

    My impression is that the damage from Weick’s plagiarism has arisen because he and others have presented the story as if it were true. Holub’s poem clearly identifies it as a “story,” not necessarily “something that happened.” Also, the lack of citation of the story has liberated its retellers to alter it to suit their taste, thus destroying its ability to falsify people’s preconceived notions.

    As you note, a theory based on plagiarized anecdotes or faked data can be valid. For that matter, a theory based on no data at all can be valid. But science is not just about coming up with theories, it’s also about evaluating them and understanding the evidence behind them. Honesty and openness about sources is an important part of this process.

    Regarding your last paragraph, here’s what I think is the ideal scenario, conditional on the what has happened up to this point. Weick’s colleagues acknowledge that he has plagiarized. Weick admits his plagiarism, and his story in future is reported with its proper citations. Basbøll can write about other things that he cares about. Others can study Weick’s theories with a better understanding of their sources.

    Like

    Andrew Gelman

    July 17, 2012 at 1:32 pm

  5. Andrew: The reason that I discourage fraud busting among the unternured is that it can be very time consuming and you don’t get a reward. You are senior faculty – what if some junior faculty member burned up a few semesters of time to fraud bust and had fewer publications? Would you vote for tenure?

    Also, there are some senior profs who spend inordinate amounts of time settling grudges. I actively try to do research and have children, so I don’t know how people find the time for petty squabbles, but they do. A junior faculty who is in the same subfield as the fraudster can find their grants and publications blocked by the fraudster and his/her allies.

    Like

    fabiorojas

    July 17, 2012 at 3:05 pm

  6. Fabio, you keep sliding from “ought” to “is”. I think we all agree about the nature of the problem. But you seem to think that the solution lies simply in helping individuals to cope with their powerlessness. Let’s start a movement instead! The way you describe it, the fraudsters have way too much power. Let’s take it back. The simple solution is to let those “burned up” semesters of work count: let the junior faculty member publish the results of the sleuthing. Everyone’s smarter, and the fraud-buster gets a publication or two. Then give them tenure for their contributions to the field. I always use Ezra Pound’s metaphor: weeding, too, makes a contribution to a garden.

    Like

    Thomas

    July 17, 2012 at 3:34 pm

  7. Andrew – I agree with your points about the scientific process, in general. But in relation to that particular story: It’s not as if he did a proper qualitative sampling and examined 4 different cases and compared. It’s a particular event that offers an image/metaphor and is used as such. To treat the story as “evidence” is just misleading (to be fair I have no idea if Weick does that). And anyways, such an event probably happened at some point in world history (i.e. an event where just any (“wrong”) map was useful). Wether Weick has found such a story or not is not the main thing in his work, the main thing is if his contribution helped improve (our understanding of) organizational behavior. So, basically. It’s just not the best example to keep referring back to (unless of course Thomas hopes to make a billiond dollars as people in Lewis’ story).
    On the other hand: The Lichtenthaler story – Now that is something!

    Like

    Anonymous

    July 17, 2012 at 4:39 pm

  8. For those who want to see the problem, you can start with “Substitutes for Strategy,” from 1987, which is reprinted in Making Sense of the Organization (2001). It makes the classic sensemaking point that it’s not the strategy that matters but the action that “forms and affirms” it, as he says in his introduction to the chapter (p. 305). “Two favorite stories of sensemaking bracket this chapter,” he says. “Soldiers lost in the Alps in a snowstorm and hungry Naskapi Indians searching for game both find what they are looking for by means of just enough strategy, just in time, to get them moving and acting mindfully.” He got the first story from a poem by Miroslav Holub and the second from an anthropologist named Omar K. Moore. He cites neither. (And it actually turns out he probably got Moore from a 1964 article in ASQ by David Braybrooke, also not cited of course). In the case of the soldiers he presents what his source calls “a story from the war” as “an incident that happened”, and in the case of the second he presents a speculative theory (which has never had any currency in the anthropological literature) as an established fact. Finally, both stories actually involve people who stop moving and reflect on their situation before acting. So he just gets them completely wrong, and then covers his tracks. This isn’t just about one act of plagiarism, it’s about a whole “style of using stories” (as he puts it) in a very undisciplined, if perhaps imaginative, way.

    More at my own blog.

    Like

    Thomas

    July 17, 2012 at 6:51 pm

  9. Fabio:

    In your comment above, you ask, “what if some junior faculty member burned up a few semesters of time to fraud bust and had fewer publications? Would you vote for tenure?” My response: it depends how good the work is. Fraud busting is a form of research which can have serious implications for science, as can be seen clearly in the work of Simonsohn and Basbøll. Fraud busting requires its share of grunt work, but then again so does other research in statistics and political science.

    I also want to respond to the second paragraph of your comment. I know what you mean about grudges. When I was an untenured professor at Berkeley, I was the subject of a grudge by some senior faculty who didn’t even really know me. That was the worst thing: getting caught in the middle of someone else’s grudge! And I’ve seen this happen to other people. It’s just horrible.

    But I do not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not think that pursuing a case of scholarly misconduct is a form of “grudge.” I do not, for example, think that Basbøll’s writings on Weick represent a “grudge.” I just don’t see any grudginess there. Basbøll discovered something about Weick that many people didn’t know, that had relevance in that subfield, and he wanted to spread the word. I think this kind of obsessiveness is a good thing for scholars. We burrow in and in and in.

    You write, “I actively try to do research and have children, so I don’t know how people find the time for petty squabbles, but they do.” As a blogger, I am wary of making such arguments. Most people probably think blogging is a waste of time. Other people, even those who have multiple children and are active researchers, find time to go to movies, play sports, watch TV, etc.

    You write about “petty squabbles.” I assume you are referring to disagreements that aren’t very interesting to you, hence the disparaging words “petty” and “squabble.” In that sense, I can understand why you don’t know how people “find the time” to do this. The quick answer, I think, is that things that you find petty are things that other people find important. As Basbøll notes in his comments above, Weick’s misrepresentations have serious implications for the fields known as organization theory and sensemaking scholarship. I am not particularly interested in these academic fields (to clarify: I’m interested in the topics, just not in the academic fields. Before hearing the plagiarism story, I’d never heard of Weick, just as I’d never heard of Marc Hauser before his scandal erupted, etc). But I am interested in academic misconduct, maybe one reason for this is that as a statistician I am acutely aware of the need to model what we call the data-collection process. If data are not sampled at random, or if they are manipulated, it can be important to understand this.

    To me, a case of a prominent academic in his subfield who plagiarizes, is caught at it, is unrepentant and supported by many of his colleagues–that’s interesting to me. Sort of like the story of Marion Barry: even after he was caught, he had defenders who felt that he represented a certain political position that they liked. Then again, I find politics fascinating–I’m a political scientist. I know that other people think of politics as petty squabbles.

    In any case, this exchange is interesting. I am interested in scholarly misconduct and think that, as Basbøll puts it, “weeding the garden” is important. Based on your posts, I think that you value scientific retractions but place less value on the effort that goes into forcing the retraction. That’s fine, but I think you’re in a tough situation. If the effort of chasing down scientific misconduct is belittled, disparaged, and mocked, then you’ll get a lot less retraction than you might like to see. That’s why I wrote in my linked blog post that people like Uri Simonsohn who discover research misconduct should be celebrated.

    Also, I agree with the summary paragraph in your post above, and I think that to the extent that critical work is given publication and respect, we can hope to see more of it from graduate students, postdocs, and junior faculty, as well as from the sorts of senior faculty who happen to have time to do research, take care of children, and blog.

    Like

    Andrew Gelman

    July 17, 2012 at 10:26 pm

  10. Andrew and Thomas: Thanks for your comments. A few responses:

    – First, I want it to be clear that I do not consider fraud busters to be settling grudges. I was just warning readers that that the fraudster may hold a grudge and that it can be destructive for junior faculty. If, for example, I believe that Andrew Gelman is publishing false data, I could probably kiss my Bayesian textbook in political science goodbye. Now that I’m tenured, I don’t care. But this can be devastating for junior faculty.

    – Second, I would agree with you that fraud busting or other error detection can advance science. A much less sensational example is the APSR article on the garbage can model, where replication showed that the original text was deeply flawed. We learned a lot about garbage theory from replication. A more sensational example is the vaccine-autism article. Investigation and retraction resulted in the refutation of a fraudulent article that resulted in real damage to people (ie, not getting vaccines is very dangerous).

    – Third, that doesn’t automatically mean that we should pursue all fraudulent work equally. Just like the police who decide which case to prosecute, scientists must pick and choose fights. You can pick bigger fights once you have a decent track record and some job security.

    – Fourth, Andrew and Thomas have persuaded me that “weeding the garden” is a contribution. I was mistaken. But still, I don’t know how easy it would be convince my colleagues of this point. All the time, academia talks about new ways to earn tenure. The action doesn’t match the rhetoric. If you want a successful career, you will need to go through a handful of high prestige journals. Since journals publish very little replication or re-investigation of original research, I believe that it will remain very, very difficult to get promoted on fraud busting or replication, unless the research is really, really amazing. At best, replication/fraud busting/investigation would be part of a much larger research portfolio..

    – Fifth, Yes, Andrew, blogging may be considered squabbling or a waste of time. But here’s the key difference. Nobody is forced to read this blog. I do not suck up the time of junior faculty, nor do I lodge frivolous complaints, or otherwise waste the energy of my colleagues or ruin their careers. In all walks of life, we find people who perceive some injustice or slight and then actively seek retribution. Professors are no exception.

    Returning to the larger point, there are many faculty who would try to get revenge on fraud busters. They find the time and energy to meddle in the lives of other people. I was merely expressing my shock at this behavior. From my own view, I love holding my baby. I love writing this blog and doing my research. I love teaching my classes. I love taking a walk on the beach and I love reading other people’s research. Yet, there are many people who forgo theses activities and instead spend the little time they have on this earth engaged in activities that strike me as vindictive.

    The reason I raise this point is that the untenured scholar, or the low status scholar, who chooses to pick a fight with another scholar may find themselves tangled up in a nasty fight. Andrew, if you live in a world where people who are criticized always take it in stride and aim to do better, then consider yourself lucky. In my world, fraud busters may find their careers stymied by faculty who are petty and enjoy squabbling. These scholars will even take the time to interfere in other people’s lives.

    Like

    fabiorojas

    July 18, 2012 at 4:19 am

  11. Fabio, this is why we need a social movement. In your last two paragraphs you very clearly describe the stakes. Under the present conditions, if you want to enjoy life as a scholar, don’t point out other people’s mistakes and misdemeanors. But if enough people came out in support of fraud-busters, and even just critical scholars, within the relevant subfield, people who do that sort of work would be able to hold their babies and take walks on the beach too, as well as teach students and read research that interests them. For my part, I would not be able enjoy the company of my children or my students if I thought my right to do so was dependent on keeping my mouth shut.

    Like

    Thomas

    July 18, 2012 at 6:43 am

  12. Fabio:

    Thanks for the clarification. I’d misunderstood your earlier comment. I agree that it is one thing for someone like me to applaud fraudbusters but quite another for me to recommend that everybody be a whistleblower without acknowledging the existing risks.

    Like

    Andrew Gelman

    July 18, 2012 at 8:38 am


Comments are closed.

%d bloggers like this: