orgtheory.net

what is “bad” research?

Last week, we had a fruitful discussion of graduate school and publishing. I think we all agreed that most graduate students should learn how to publish quickly. But we also raised some red flags. For example, we shouldn’t encourage people to publish “bad” articles. Others thought that we shouldn’t publish “too much.”

So let’s begin with a consensus: yes, if you are a graduate student, you should definitely learn the publishing process. No let’s move on to lower consensus issues. First, what counts as “bad” research? A few definitions:

  1. Research that is fraudulent.
  2. Research that is in a technical sense correct, but misleading.
  3. Research that is sloppy or poorly written.
  4. Research that is made in good faith, but in error.
  5. Research that is chopped up into lots of small chunks, in terms of article length/word or page counts.
  6. Research that makes extremely small or incremental arguments.
  7. Research that is in the “wrong” journal – low prestige, niche, online, or in a lower status discipline.

Now, when is it bad to publish work in any of these categories? There is overwhelming consensus that #1 is bad and should never be tolerated. In fact, academia has such a strong norm on #1 that fraudulent articles are almost always retracted and people might lose their job. I think we’d agree that #2 is also bad, though there is disagreement about what should be done with misleading articles, as we found out when discussing He-Who-Shall-Not-Be-Named-in-Texas.

Once we get past fraudulent and misleading research, it’s very unclear that any of the remaining categories can be claimed to be uniformly bad. For example, garbage can paper (March, Cohen, and Olson 1972) was successfully shown to be a very sloppy work (see the Bendor, Moe, & Schott 2000 APSR article). No way around it. But, as olderwoman points out, powerful ideas are often presented in sloppy packages.

Then we get to #4: good faith papers with mistakes. In some cases, #4 is obviously bad. We find out that the answer is different when we correct our code – retraction. But in other cases, it’s ok. For example, among mathematicians, incorrect proofs are sometimes left in the record. The overall idea remains promising, but maybe some future scholar can read the mistake and fix it.

#5, #6 and #7 are clearly not universal. If you look around academia, you see that some fields hate, hate, hate small articles (history) while other fields exist primarily in tiny, tiny articles spread out in big and small journals. Even with one field, like sociology, you see huge variance. Demographers routinely “chop and spread it,” while ethnographers save it all for one big AJS/ASR article.

I’ll finish with how I think about my own publication strategy. My first allegiance is to knowledge. So I have never suppressed any article that I thought had a specific contribution to make, big or small. Also, in my own experience, I have benefited greatly from articles published in some obscure places. “Small” doesn’t mean dumb or useless. Just small – which might be very important to someone out there (including me).

What I have ended up with is a sort of triage: articles that are “big” in some sense are channeled to major journals, while “small” contributions are sent to niche journals. That results in a output stream where the modal is “small” but the stream is punctuated by a few “bigs.” Finally, one thing that I don’t do is rewrite the same article over and over. I make no claim that this is optimal, only that this is what you get if you believe that “small” contributions and niche journals have a place in the academic world.

Adverts: From Black Power/Grad Skool Rulz

Advertisements

Written by fabiorojas

October 7, 2013 at 12:01 am

Posted in academia, fabio, research

3 Responses

Subscribe to comments with RSS.

  1. An interesting list of discussions of bad research is at the Economic Logic blog: http://economiclogic.blogspot.com/search/label/bad%20research

    Like

    M. H.

    October 7, 2013 at 12:43 am

  2. Reblogged this on Undergraduate Research Blog and commented:
    As undergraduates try harder and harder to publish their work, these standards (hidden curriculum, even) seem crucial for undergraduate students, especially given that college students may write what is considered “sloppy” work (#3) by professionals may be “the best” an undergraduate can achieve as they learn how to write like an academic or the contribution made by the student project may only be modest or provide an “incremental” benefit to the field (#6), but that may also be the best that a student can produce … are these contributions not to join the academic communications (even if published in relatively modest journals or undergraduate research journals) OR, and this worries me a great deal, is allowing undergraduates to publish anything but the highest-quality scholarly work part of a broader, and generally negative, culture of entitlement?

    Like

    Nicholas

    October 7, 2013 at 12:38 pm

  3. The author (ironically?) writes:

    “For example, garbage can paper (March, Cohen, and Olson 1972) was successfully shown to be a very sloppy work (see the Bendor, Moe, & Schott 2000 APSR article). No way around it.”

    The paper was published in 2001, and the final author’s name is Schotts. More broadly, “successfully”? No mention of Olsen’s response (“Garbage Cans, New Institutionalism, and the Study of Politics,” also 2001 APSR) and the criticism of the original paper for baldly misunderstanding the point of the simulation in the garbage can?

    To the larger point, this post seems technically accurate, yet the broader point (on which younger scholars are judged) is towards publication strategy, not any particular article. If, for example, you go on the job market with two small, incremental publications and no description of a larger, ambitious research program into which they fit, you will be seen as an incremental scholar and your candidacy likely downgraded accordingly (and your articles retroactively judged as “bad” or more likely, the kiss of death: “just not that interesting.”) Similarly, you will likely be forgiven for *some kinds of* sloppiness if you are judged to be an original thinker working at the cutting edge of your discipline. These two strategies of self-presentation each carry significant risks, but the point is that a significant portion of folks in sociology tend to judge holistically, and not on the basis of aggregating “good” or “bad” individual contributions. (But just to emphasize, any version of #1 will get you put on career-long time out.)

    Like

    Hmm

    October 7, 2013 at 1:14 pm


Comments are closed.

%d bloggers like this: