orgtheory.net

agreements and disagreements with rob warren

Rob Warren, of the University of Minnesota, wrote some very engaging and insightful comments about his time as the editor of Sociology of Education. Jeff Guhin covered this last week. Here, I’ll add my own comments. First, a strong nod of agreement:

First, a large percentage of papers had fundamental research design flaws. Basic methodological problems—of the sort that ought to earn a graduate student a B- in their first-year research methods course—were fairly common.4 (More surprising to me, by the way, was how frequently reviewers seemed not to notice such problems.) I’m not talking here about trivial errors or minor weaknesses in research designs; no research is perfect. I’m talking about problems that undermined the author’s basic conclusions. Some of these problems were fixable, but many were not.

Yes. Professor Warren is correct. Once you are an editor, or simply an older scholar who has read a lot of journal submissions, you quickly realize that there a lot of papers that really, really flub research methods 101. For example, a lot of paper rely on convenience samples, which lead to biased results. Warren has more on this issue.

Now, let me get to where I think Warren is incorrect:

Second, and more surprising to me: Most papers simply lacked a soul—a compelling and well-articulated reason to exist. The world (including the world of education) faces an extraordinary number of problems, challenges, dilemmas, and even mysteries.  Yet most papers failed to make a good case for why they were necessary. Many analyses were not well motivated or informed by existing theory, evidence, or debates. Many authors took for granted that readers would see the importance of their chosen topic, and failed to connect their work to related issues, ideas, or discussions. Over and over again, I kept asking myself (and reviewers also often asked): So what?

About five years ago, I used to think this way. Now, I’ve mellowed and come to a more open minded view. Why? In the past, I have rejected a fair number of papers on “framing” grounds. Later, I will see them published in other journals, often with high impact. Also, in my own career, leading journals have rejected my work on “framing” grounds and when it gets published in another leading journal, the work will get cited. The framing wasn’t that bad. Lesson? A lot of complaints about are framing are actually arbitrary. Instead, let the work get published and let the wider community decide, not the editor and a few peer reviewers.

The evidence on the reliability of the peer review process suggests that there is a lot of randomness in the process. If some of these “soul-less” papers had been resubmitted a few months later, some of them would have been accepted with enthusiastic reviews. Here’s a 2006 review of the literature on journal reliability and here’s the classic 1982 article showing that a lot of journal acceptance is indeed random. Ironically, Peters and Ceci (1982) note that “serious methodological flaws” are a common reason for rejecting papers – that had already been accepted!

This brings me to Warren’s third point – a complaint about people who submit poorly developed papers. He suggests that there are job pressures and a lack of training. On the training point, there is nothing to back up his assertion. Most social science programs have a fairly standard sequence of quantitative methods courses. The basic issues regarding causation v. description, identification, and assessment of instrument quality are all pretty easy to learn. Every year, the ICPSR offers all kinds of training. Training we have, in spades.

On the jobs point, I would like to blame people like Professor Warren and his colleagues on hiring and promotion committees (which includes me!!). The job market for the better positions in sociology (R1 jobs and competitive liberal arts schools) has essentially evolved into whoever gets into the top journals in graduate school plus graduate program reputation.

I’d suggest we simply think about the incentives here. Junior scholars live in a world where a lot of weight is placed on a very small number of journals. They also live in a world where journal acceptance is random. They also live in a world where journals routinely lose papers, reject after multiple R&R rounds and takes years (!) to respond (see my journal horror stories post). How would any rational person respond to this environment? Answer: just send out a lot of stuff until something hits. There is no incentive to develop a paper well if it will be randomly rejected after sitting at the journal for 16 months.

This is why I openly praise and encourage reforms of the journal system. I praise “platform” publishing like PLoS One. I praise “up or down” curated publishing, like Sociological Science. I praise Socius, the open access ASA journal. I praise socArxiv for creating an open pre-print portal. I praise editors who speed the review process and I praise multiple submissions practices. The basic issues that Professor Warren discusses are real. But the problem isn’t training or stressed out junior scholars. The problem is the archaic journal system. Let’s make it better.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 9, 2016 at 12:01 am

4 Responses

Subscribe to comments with RSS.

  1. I like your take on these issues, but I took Warren’s second point a little bit differently. I took it to mean that the articles he saw ended up taking on debates in the literature rather than important or pressing points in ongoing discussions about educational inequality. I saw his argument less about framing, per se, than about the fact that the manuscript lacked a compelling question that could be framed. Perhaps I read too much into his own stated willingness to work to develop papers into outstanding contributions. I think that this could be said of lots of questions in sociology, and could be a second-order consequence of the misplaced importance that our field places on framing (an author saying, “I’m only going to address specific pieces that have been written so that I don’t get called on not addressing X, and Y, and Z along with A in review”). But I took it as a different criticism than framing more generally.

    It seems to me like there would be an opportunity here for an enterprising and aspirational department to build up a department with concerted hires by trying to find the folks who do good research that will ultimately be high-impact but will require time to develop. Really be willing to take some risks on enterprising grad students who are developing new areas that take time to develop cohesive arguments on projects that integrate subfields, develop new data (especially qualitative data), and need help learning to frame good ideas into packages that influential journals recognize as contributions. It would be high risk and would require someone as chair with enough clout to challenge the received wisdom of deans and provosts who like to tout top publications, but I think offers an important opportunity for a department to break into the top tier of grad programs.

    Liked by 1 person

    mike_bader

    August 9, 2016 at 3:59 pm

  2. “I took it to mean that the articles he saw ended up taking on debates in the literature rather than important or pressing points in ongoing discussions about educational inequality.”

    Good point. My response: Isn’t this a cheap shot? Isn’t most of the Soc of Ed discipline already focused on strat issues? The classic complaint about soc of ed (the journal and the subfield) is that every other article is about how race/gender/class plays out in attainment/school experiences.

    Also, peer review is good about building the literature. It is not good at “big ideas” because there will always be one reviewer who has an issue with your paper if you deviate from the literature. I have actually tried to do what Professor Warren asks, but the response is often “this has to be anchored in the literature so we reject it.” That is why books, in my view, are better for the kinds of issues Prof. Warren is thinking of.

    Like

    Fabio Rojas

    August 9, 2016 at 4:21 pm

  3. “The classic complaint about soc of ed (the journal and the subfield) is that every other article is about how race/gender/class plays out in attainment/school experiences.”

    I’m not sure if it’s a cheap shot; simply studying stratification doesn’t necessarily mean that you study the aspects of stratification most relevant to outcomes under study or that one engages with larger questions about how stratification shapes the field. And most scientific research should be somewhat derivative, that is what makes for good, cumulative science.

    But, I would also say that a good portion of what I read and review for journals comparable to Soc of Ed (e.g., Demography) often take as a starting point well-known theoretical positions and attempt to analyze a different dataset or organize the data in a slightly different way to adjudicate whether one is right or wrong. They study important questions of stratification, but lack an imagination of answers by using (in my opinion) previous research as a crutch.

    This approach often leads to contorting data into something resembling a test of the underlying theory, but fails in some major methodological regard (going back to Warren’s first point). It often means not really interrogating the underlying assumptions of existing theories that can often be out of date with the empirical state of the world, that assume some beneficial outcome that might not actually be beneficial, or fill in the gaps where the existing theory leaves off. In other words, they end up not really asking or answering new questions, but attempting to adjudicate old ones.

    You might be completely correct that articles in top field journals, or even major journals for that matter, are the correct place to make completely new arguments. Journals might be the place to develop cumulative science that builds explicitly on past research and theories. But, I can also imagine that it means that much of the research that comes over the transom ends up not being particularly inspired.

    Like

    mike_bader

    August 9, 2016 at 5:21 pm

  4. Your comment raises many issues, but I want to focus on one issue: “contorting data.” Honestly, the reason that people take old/established data and contort it is that when you collect your own data or use new data, people go ballistic.

    What frustrates me, not only with Professor Warren’s comment, but also with the larger idea is that when editors say “I wish people did X” the journal process is actually intolerant of X.If we made journals more tolerant, we’d get better research. Until that happens, we’ll get more of the same.

    Liked by 1 person

    Fabio Rojas

    August 9, 2016 at 6:18 pm


Comments are closed.

%d bloggers like this: