Archive for the ‘academia’ Category
On All Hallow’s Eve, it’s time to flip off the lights, take out the flashlight to illuminate our faces from below, and swap scary stories. For some of us, the scary stuff lurks constantly, just around the corner where we work: the unfinished thesis, courtesy of Academia Obscura:
But, far worse horrors exist – namely, researcher misconduct. While the most obvious cases of falsified data and plagiarized writings are caught and retracted, more subtle cases are unfortunately harder to pursue, prove, and punish.
Are these cases of sociopathic scholars or scholars made sociopathic by the system? At least two civil engineers worry that the current academic system of incentives could possibly tip scholars towards misconduct. In their abstract, authors Marc A. Edwards and Siddhartha Roy warn about how the increasingly competitive race for funding in STEM could potentially ruin scholarship.
Over the last 50 years, we argue that incentives for academic scientists have become increasingly perverse in terms of competition for research funding, development of quantitative metrics to measure performance, and a changing business model for higher education itself. Furthermore, decreased discretionary funding at the federal and state level is creating a hypercompetitive environment between government agencies (e.g., EPA, NIH, CDC), for scientists in these agencies, and for academics seeking funding from all sources—the combination of perverse incentives and decreased funding increases pressures that can lead to unethical behavior. If a critical mass of scientists become untrustworthy, a tipping point is possible in which the scientific enterprise itself becomes inherently corrupt and public trust is lost, risking a new dark age with devastating consequences to humanity. Academia and federal agencies should better support science as a public good, and incentivize altruistic and ethical outcomes, while de-emphasizing output.
In particular, Edwards and Roy single out three conditions as problematic in sites where much of research is conducted: the university:
Recently, however, an emphasis on quantitative performance metrics (Van Noorden, 2010), increased competition for static or reduced federal research funding (e.g., NIH, NSF, and EPA), and a steady shift toward operating public universities on a private business model (Plerou, et al., 1999; Brownlee, 2014; Kasperkevic, 2014) are creating an increasingly perverse academic culture.
They note how academics fear that misconduct is becoming widespread as opposed to isolated instances:
Ultimately, the well-intentioned use of quantitative metrics may create inequities and outcomes worse than the systems they replaced. Specifically, if rewards are disproportionally given to individuals manipulating their metrics, problems of the old subjective paradigms (e.g., old-boys’ networks) may be tame by comparison. In a 2010 survey, 71% of respondents stated that they feared colleagues can “game” or “cheat” their way into better evaluations at their institutions (Abbott, 2010), demonstrating that scientists are acutely attuned to the possibility of abuses in the current system.
They also worry that people attracted to academia for altruistic reasons will be turned off by the perverse incentives and exit the system for careers and workplaces that are more consistent with their values and goals:
While there is virtually no research exploring the impact of perverse incentives on scientific productivity, most in academia would acknowledge a collective shift in our behavior over the years (Table 1), emphasizing quantity at the expense of quality. This issue may be especially troubling for attracting and retaining altruistically minded students, particularly women and underrepresented minorities (WURM), in STEM research careers. Because modern scientific careers are perceived as focusing on “the individual scientist and individual achievement” rather than altruistic goals (Thoman et al., 2014), and WURM students tend to be attracted toward STEM fields for altruistic motives, including serving society and one’s community (Diekman et al., 2010, Thoman et al., 2014), many leave STEM to seek careers and work that is more in keeping with their values (e.g., Diekman et al., 2010; Gibbs and Griffin, 2013; Campbell, et al., 2014).
Under the subheading “If nothing is done, we will create a corrupt academic culture,” the authors warn about the collapse of the academic commons. At the end of their paper, they offer several possibilities for starting to address these issues, including research into the dimensions of research misconduct, a more explicit discussion of values, and a reconfiguration of incentives:
(1) The scope of the problem must be better understood, by systematically mining the experiences and perceptions held by academics in STEM fields, through a comprehensive survey of high-achieving graduate students and researchers.
(2) The National Science Foundation should commission a panel of economists and social scientists with expertise in perverse incentives, to collect and review input from all levels of academia, including retired National Academy members and distinguished STEM scholars. The panel could also develop a list of “best practices” to guide evaluation of candidates for hiring and promotion, from a long-term perspective of promoting science in the public interest and for the public good, and maintain academia as a desirable career path for altruistic ethical actors.
(3) Rather than pretending that the problem of research misconduct does not exist, science and engineering students should receive instruction on these subjects at both the undergraduate and graduate levels. Instruction should include a review of real world pressures, incentives, and stresses that can increase the likelihood of research misconduct.
(4) Beyond conventional goals of achieving quantitative metrics, a PhD program should also be viewed as an exercise in building character, with some emphasis on the ideal of practicing science as service to humanity (Huber, 2014).
(5) Universities need to reduce perverse incentives and uphold research misconduct policies that discourage unethical behavior.
My response to this question on Facebook:
- Do not publish in PLoS if you need a status boost for the job market or promotion.
- Do publish if journal prestige is not a factor. My case: good result but I was in a race against other computer scientists. Simply could not wait for a four year journal process.
- Reviewer quality: It is run mainly by physical and life scientists. Reviews for my paper were similar in quality to what CS people gave me on a similar paper submitted to CS conferences/journals.
- Personally, I was satisfied. Review process fair, fast publication, high citation count. Would not try to get promoted on the paper by itself, though.
- A lot of people at strong programs have PLoS One pubs but usually as part of a larger portfolio of work.
- A typical good paper in PLoS is from a strong line of work but the paper just bounced around or too idiosyncratic.
- PLoS One publishes some garbage.
- Summary: right tool for the right job. Use wisely.
Another person noted that many elite scientists use the “Science, Nature, or PLoS One model.” In other words, you want high impact or just get it out there. No sense wasting years of time with lesser journals.
The Society for the Advancement of Socio-Economics (SASE) website has made Neil Fligstein‘s powerpoint slides on the history of economic sociology available for general viewing as a PDF. (Here are the slides in powerpoint form: 1469704310_imagining_economic_sociology_-socio-economics-fligstein) It’s a fascinating read of the development of a sub-field across continents, and it also includes discussion of a challenge that some believes plagues the sociology discipline:
Both Max Weber and Thomas Kuhn recognized that Sociology as a discipline might be doomed to never cumulate knowledge.
- Sociology would proceed as a set of research projects which reflected the current concerns and interests of a small set of scholars
- When the group hit a dead end in producing novel results, the research program would die out only to be replaced by another one
- Progress in economic sociology is likely to be made by putting our research programs into dialogue with one another to make sense of how the various mechanisms that structure markets interact
- Failure to do so risks the field fragmenting of the field into ever smaller pieces and remaining subject to fashion and fad
Fligstein’s claim for these field-fragmenting tendencies stems from the current structure of the academic field. He depicts sociology as rewarding scholars for applying ideas from one area to another area where current theorizing is insufficient, rather than expanding existing research:
- … the idea is not to work on the edge of some mature existing research program with the goal of expanding it
- But instead, one should be on the lookout for new ideas from different research programs to borrow to make sense for what should be done next
In short, scholars tend to form intellectual islands where they can commune with other like-minded scholars. Bridging paths to other islands can generate rewards, but the efforts needed to disseminate knowledge more widely – even within a discipline – can exceed any one person’s capacity.
Today, among the various administrative tasks of scheduling meetings with students and other responsibilities, I decided to RSVP yes for an upcoming evening talk. I didn’t make this decision lightly, as it involved coordinating schedules with another party (i.e., fellow dual career parent).
With the use of technology such as email, increasing job precarity, and belief in facetime as signalling productivity and commitment, the workday in the US has elongated, blurring boundaries to the point that work can crowd out other responsibilities to family, community, hobbies, and self-care. However, one Ivy institution is exhorting its members to rethink making evening events and meetings the norm.
In this nuanced statement issued to department chairs, Brown University’s provost outlines the stakes and consequences of an elongated workday:
This burden [of juggling work and family commitments] may disproportionately affect female faculty members. Although data on Brown’s faculty is not available, national statistics indicate that male faculty members (of every rank) are more likely than female faculty members (of every rank) to have a spouse or partner whose comparably flexible work schedule allows that spouse or partner to handle the bulk of evening-time household responsibilities. Put differently, male faculty members are more likely than female faculty members to have the household support to attend campus events after 5:30. We must be attuned to issues of gender equity when we think about program scheduling. We must also take into consideration the particular challenges faced by single parents on the faculty when required to attend events outside the regular hours of childcare.
This Guardian piece about bad incentives in science was getting a lot of Twitter mileage yesterday. “Cut-throat academia leads to natural selection of bad science,” the headline screams.
The article is reporting on a new paper by Paul Smaldino and Richard McElreath, and features quotes from the authors like, “As long as the incentives are in place that reward publishing novel, surprising results, often and in high-visibility journals above other, more nuanced aspects of science, shoddy practices that maximise one’s ability to do so will run rampant.”
Well. Can’t disagree with that.
But when I clicked through to read the journal article, the case didn’t seem nearly so strong. The article has two parts. The first is a review of review pieces published between 1962 and 2013 that examined the levels of statistical power reported in studies in a variety of academic fields. The second is a formal model of an evolutionary process through which incentives for publication quantity will drive the spread of low-quality methods (such as underpowered studies) that increase both productivity as well as the likelihood of false positives.
The formal model is kind of interesting, but just shows that the dynamics are plausible — something I (and everyone else in academia) was already pretty much convinced of. The headlines are really based on the first part of the paper, which purports to show that statistical power in the social and behavioral sciences hasn’t increased over the last fifty-plus years, despite repeated calls for it to do so.
Well, that part of the paper basically looks at all the papers that reviewed levels of statistical power in studies in a particular field, focusing especially on papers that reported small effect sizes. (The logic is that such small effects are not only most common in these fields, but also more likely to be false positives resulting from inadequate power.) There were 44 such reviews. The key point is that average reported statistical power has stayed stubbornly flat. The conclusion the authors draw is that bad methods are crowding out good ones, even though we know better, through some combination of poor incentives and selection that rewards researcher ignorance.
The problem is that the evidence presented in the paper is hardly strong support for this claim. This is not a random sample of papers in these fields, or anything like it. Nor is there other evidence to show that the reviewed papers are representative of papers in their fields more generally.
More damningly, though, the fields that are reviewed change rather dramatically over time. Nine of the first eleven studies (those before 1975) review papers from education or communications. The last eleven (those after 1995) include four from aviation, two from neuroscience, and one each from health psychology, software engineering, behavioral ecology, international business, and social and personality psychology. Why would we think that underpowering in the latter fields at all reflects what’s going on in the former fields in the last two decades? Maybe they’ve remained underpowered, maybe they haven’t. But statistical cultures across disciplines are wildly different. You just can’t generalize like that.
The news article goes on to paraphrase one of the authors as saying that “[s]ociology, economics, climate science and ecology” (in addition to psychology and biomedical science) are “other areas likely to be vulnerable to the propagation of bad practice.” But while these fields are singled out as particularly bad news, not one of the reviews covers the latter three fields (perhaps that’s why the phrasing is “other areas likely”?). And sociology, which had a single review in 1974, looks, ironically, surprisingly good — it’s that positive outlier in the graph above at 0.55. Guess that’s one benefit of using lots of secondary data and few experiments.
The killer is, I think the authors are pointing to a real and important problem here. I absolutely buy that the incentives are there to publish more — and equally important, cheaply — and that this undermines the quality of academic work. And I think that reviewing the reviews of statistical power, as this paper does, is worth doing, even if the fields being reviewed aren’t consistent over time. It’s also hard to untangle whether the authors actually said things that oversold the research or if the Guardian just reported it that way.
But at least in the way it’s covered here, this looks like a model of bad scientific practice, all right. Just not the kind of model that was intended.
[Edited: Smaldino points on Twitter to another paper that offers additional support for the claim that power hasn’t increased in psychology and cognitive neuroscience, at least.]
“Who did you work with?” It’s a question applicants get all the time on the job market, and for good reason: if you know someone’s dissertation advisor, you probably know a bit more about how that person works, the kinds of questions they study, the sorts of methods and theory they use. Of course, the American job market doesn’t want to hire disciples, so a student can’t be too close to the teacher: there has to be some difference, theoretical or methodological, creative or substantive. Yet there needs to be some commonality too, if nothing more than a commonality of competence. Knowing someone’s mentor has chops is supposed to translate into knowing that person has chops as well. Those of us who have grad students have them because we believe that being a good sociologist can be taught, and that skills—even excellence—can be reproduced. And so status moves forward, down the genealogical line: begat, begat, begat.
There are a lot of problems with this focus on individual-level mentoring, with an advisor’s status functioning as a high-level credential (eg: “Oh X is solid. She worked with Y”). First off, the sociology of it is not at all obvious to me. It’s simply an empirical question how much a mentor matters in the formation of good scholars: we are, after all, a big wide community, and why can’t it take a village to raise a sociologist? That “village” might extend beyond a graduate school to the discipline as a whole: think of a student at a low-prestige grad program whose mentor is not well-known nor super invested in the discipline (or grad students). Yet this grad student reads widely, attends conferences, and networks assertively, finding other people to read her work. She doesn’t have the currency of a high status mentor, but she might well have some good publications. It’s interesting how much the status of the mentor still matters when I think about that student’s chances on the market (and the status of her university).
I’m not saying anything new to say that it’s deeply ironic how a discipline so committed to fighting inequality in the world at large maintains such deep inequalities in its own house. And a focus on mentors as tokens of worth, while important and understandable, can have a significant role in maintaining those inequalities. What if, instead of thinking of ourselves as a guild of masters and apprentices, we thought of ourselves as a community of practitioners, eager to help everyone get better at what they do? I have no idea how that would work out practically, but it’s worth thinking about, and I’ll write more about this. If you have any thoughts, do please let me know in the comments or over e-mail.