orgtheory.net

rob warren’s harsh critique of the submissions he got at soe

If you don’t get the Sociology of Education newsletter, or even if you do and just don’t read it, you probably didn’t see Rob Warren’s pretty devastating criticism of the submissions he usually got when he was the editor of Sociology of Education.  As a junior scholar who has sent out my own share of not-quite-formed papers, his points are well taken, and my hunch (and what I’ve heard from editors) is that these complaints extend to other major journals as well.  Read the whole thing at his website, but here’s a sample:

Most of the papers that I read had one or both of two basic problems:

First, a large percentage of papers had fundamental research design flaws. Basic methodological problems—of the sort that ought to earn a graduate student a B- in their first-year research methods course—were fairly common.4 (More surprising to me, by the way, was how frequently reviewers seemed not to notice such problems.) I’m not talking here about trivial errors or minor weaknesses in research designs; no research is perfect. I’m talking about problems that undermined the author’s basic conclusions. Some of these problems were fixable, but many were not.

Second, and more surprising to me: Most papers simply lacked a soul—a compelling and well-articulated reason to exist. The world (including the world of education) faces an extraordinary number of problems, challenges, dilemmas, and even mysteries.  Yet most papers failed to make a good case for why they were necessary. Many analyses were not well motivated or informed by existing theory, evidence, or debates. Many authors took for granted that readers would see the importance of their chosen topic, and failed to connect their work to related issues, ideas, or discussions. Over and over again, I kept asking myself (and reviewers also often asked): So what?

Written by jeffguhin

August 5, 2016 at 3:41 am

Posted in academia

Tagged with , , ,

5 Responses

Subscribe to comments with RSS.

  1. i’m wary about making sweeping pronouncements like “the system is broken” — but, you know, the system might be broken. as a junior scholar myself, i know all too well how strongly we are incentivized to churn stuff out. this has led to a rash of quant papers in sociology that are what i like to call the “but what about this variable paper” in which an enterprising young researcher slightly changes the modeling decisions of other papers in the field and tries to make something out of it. some might say this is the way that incremental science should work, but as someone who is still relatively new to the system it has always struck me as, um, not ideal.

    the way i see it, we are never going to compete with economists in terms of statistical modeling and math sophistication — at least not on a discipline-wide basis. what we can do, though, is collect better social data than they do. to do that, though, we would need to combine resources and provide incentives for people to engage in multi-year, multi-site, multi-method coordinated research projects. only these sorts of projects, i would argue, will yield the kind of data that would allow sociology to increase its profile in policy circles and the more general intellectual climate. imagine moving to opportunity but for multiple topics, coordinated across several research universities and multiple PIs. young scholars would work in a situation akin to what happens in the natural sciences, as “lab assistants” working on the larger project (performing interviews, running analyses, etc.) and would find their own piece to work on for their master’s/dissertation.

    we would need to develop different reward structures in the labor market, but i think it’s doable. in the meantime, i’ll keep trying my best to churn out a bunch of articles (to get a job, then to get tenure) and i’ll do my best to make them really good under the many constraints that i face. i suspect though, that the quality of the social science that i, and others like me, produce will be impoverished relative to what it could be in a different model.

    Liked by 2 people

    anonymous_young_scholar

    August 5, 2016 at 4:44 am

  2. The best authors complain that their manuscripts keep being rejected by the top journals. The editors of the top journals, on the other hand, complain that they struggle to find enough good articles to fill out their issues. Where is the problem and what can be done about it?

    Like

    Guillermo

    August 5, 2016 at 2:57 pm

  3. And as a corollary but very important question, how do graduate programs contribute to the program? Stephen Turner offers a few ideas in one of his books.

    Like

    Guillermo

    August 5, 2016 at 3:00 pm

  4. I wonder how much of these issues come from two virtual norms we have on the editor’s side.

    1st is the norm to almost never accept a new submission right away. I have heard a few very senior, very published folks advise students to hold some robustness checks back when first submitting a paper, so that when it gets the inevitable R&R the author has something ready to show. I don’t know how valid that advice was, but since one of my recent publications ended up with reviewers suggesting more robustness checks after I measured a concept 4 different ways and used 3 different statistical techniques, I could definitely see it happening.

    2nd is sociology reviewers’ almost endless fascination with “framing” arguments (which is related to 1). Meaning that anything controversial will likely have to work on “framing” a whole lot.

    Granted, the examples presented above are a bit more extreme than that. But the two virtual “norms” of publishing within sociology I mention above would certainly provide incentive to submit less than polished stuff, as it is then easier to address the specific reviewers’ concerns. That is, why spend weeks or months writing a paper that anticipates most methodological and framing criticisms if the best you can realistically hope for is an R&R? Might as well do enough to get that R&R and then address those specific concerns.

    This is especially the case for young scholars, as anonymous_young_scholar notes above. Why spend the extra weeks with additional robustness checks and tinkering with the framing of the paper, if it will not lead to publication any faster, and can likely substantially hurt your career? It is strategically better to submit something that just clears the R&R threshold, because then you will get concrete things to “fix” in your paper, then to spend months anticipating those things to be fixed, because it will inevitably lead to an R&R.

    Like

    Jr

    August 5, 2016 at 8:15 pm

  5. […] some very engaging and insightful comments about his time as the editor of Sociology of Education. Jeff Guhin covered this last week. Here, I’ll add my own comments. First, a strong nod of […]

    Like


Comments are closed.

%d bloggers like this: