Cristobal Young is an assistant professor at Stanford’s Department of Sociology. He works on quantitative methods, stratification, and economic sociology. In this post co-authored with Aaron Horvath, he reports on the attempt to replicate 53 sociological studies. Spoiler: we need to do better.
Do Sociologists Release Their Data and Code? Disappointing Results from a Field Experiment on Replication.
Replication packages – releasing the complete data and code for a published article – are a growing currency in 21st century social science, and for good reasons. Replication packages help to spread methodological innovations, facilitate understanding of methods, and show confidence in findings. Yet, we found that few sociologists are willing or able to share the exact details of their analysis.
We conducted a small field experiment as part of a graduate course in statistical analysis. Students selected sociological articles that they admired and wanted to learn from, and asked the authors for a replication package.
Out of the 53 sociologists contacted, only 15 of the authors (28 percent) provided a replication package. This is a missed opportunity for the learning and development of new sociologists, as well as an unfortunate marker of the state of open science within our field.
Some 19 percent of authors never replied to repeated requests, or first replied but never provided a package. More than half (56 percent) directly refused to release their data and code. Sometimes there were good reasons. Twelve authors (23 percent) cited legal or IRB limitations on their ability to share their data. But only one of these authors provided the statistical code to show how the confidential data were analyzed.
Why So Little Response?
A common reason for not releasing a replication package was because the author had lost the data – often due to reported computer/hard drive malfunctions. As well, many authors said they were too busy or felt that providing a replication package would be too complicated. One author said they had never heard of a replication package. The solutions here are simple: compiling a replication package should be part of a journal article’s final copy-editing and page-proofing process.
More troubling is that a few authors openly rejected the principle of replication, saying in effect, “read the paper and figure it out yourself.” One articulated a deep opposition, on the grounds that replication packages break down the “barriers to entry” that protect researchers from scrutiny and intellectual competition from others.
The Case for Higher Standards
Methodology sections of research articles are, by necessity, broad and abstract descriptions of their procedures. However, in most quantitative analyses, the exact methods and code are on the author’s computer. Readers should be able to download and run replication packages as easily as they can download and read published articles. The methodology section should not be a “barrier to entry,” but rather an on-ramp to an open and shared scholarly enterprise.
When authors released replication packages, it was enlightening for students to look “under the hood” on research they admired, and see exactly how results were produced. Students finished the process with deeper understanding of – and greater confidence in – the research. Replication packages also serve as a research accelerator: their transparency instills practical insight and confidence – bridging the gap between chalkboard statistics and actual cutting-edge research – and invites younger scholars to build on the shoulders of success. As Gary King has emphasized, replications have become first publications for many students, and helped launched many careers – all while ramping up citations to the original articles.
In our small sample, little more than a quarter of sociologists released their data and code. Top journals in political science and economics now require on-line replication packages. Transparency is no less crucial in sociology for the accumulation of knowledge, methods, and capabilities among young scholars. Sociologists – and ultimately, sociology journals – should embrace replication packages as part of the lasting contribution of their research.
Table 1. Response to Replication Request
|Yes: Released data and code for paper||15||28%|
|No: Did not release||38||72%|
|Reasons for “No”|
|IRB / legal / confidentiality issue||12||23%|
|No response / no follow up||10||19%|
|Don’t have data||6||11%|
|Don’t have time / too complicated||6||11%|
|Still using the data||2||4%|
|‘See the article and figure it out’||2||4%|
Note: For replication and transparency, a blinded copy of the data is available on-line. Each author’s identity is blinded, but the journal name, year of publication, and response code is available. Half of the requests addressed articles in the top three journals, and more than half were published in the last three years.
Figure 1: Illustrative Quotes from Student Correspondence with Authors:
- “Here is the data file and Stata .do file to reproduce [the] Tables…. Let me know if you have any questions.”
- “[Attached are] data and R code that does all regression models in the paper. Assuming that you know R, you could literally redo the entire paper in a few minutes.”
- “While I applaud your efforts to replicate my research, the best guidance I can offer
is that the details about the data and analysis strategies are in the paper.”
- “I don’t keep or produce ‘replication packages’… Data takes a significant amount of human capital and financial resources, and serves as a barrier-to-entry against other researchers… they can do it themselves.”
You can read Caroline’s self-introduction, and her past posts about her and others’ research on deliberative democracy here, here, and here, as well as her observations about what it’s like to be a professor at a small liberal arts college (SLAC).
To read Ellen’s self-introduction about her research on diversity, go here and here. See her past posts about conducting ethnographic research on organizations, including gaining access and the use of pseudonyms.
Please join me in welcoming guest blogger Victor Tan Chen,* a sociologist at Virginia Commonwealth University. He is the author of newly published book based on his comparative study of autoworkers and families in Canada and the US: Cut Loose: Jobless and Hopeless in an Unfair Economy (UCPress 2015). He is also a founder and editor of IntheFray, and co-author with Katherine S. Newman of The Missing Class: Portraits of the New Poor in America.
* No relation to me, despite our shared surname!
Victoria Reyes is an assistant professor at Bryn Mawr in the Growth and Structure of Cities Department. Her research is about specific urban sites in the global system. This post addresses her recent article in Theory and Society.
Thanks to Fabio for allowing me to post about my work. I study global inequality through a cultural and relational lens, and am particularly interested in places of foreign-control, by which I mean places that are either foreign-owned or are heavily influenced by foreigners. I have two recent articles about this (see below for citations and abstracts).
In one that was recently published in Theory and Society, I draw and extend work on global cities and cities along geopolitical borders to develop a concept I call “global borderlands”—semi-autonomous, foreign-controlled, geographic locations geared toward international exchange. These are places like overseas military bases, embassies, tourist resorts, international branch campuses (e.g. NYU Abu Dhabi), and special economic zones, where tariff barriers are relaxed. When I speak of global borderlands, I do not necessarily assume negative connotations. Indeed, some people may enjoy or prefer working, visiting, and/or living within global borderlands, while others are excluded from these places.
I argue that these places have three features in common. First, semi-autonomy and foreign-control. These are places where the notion of “who rules?” is fluid and negotiated, and where regulation depends on nationality. Second, like many other places, global borderlands are defined by geographic and symbolic boundaries. Third, these places are built on unequal relations, by which I refer to structural inequality that again, does not necessarily come bundled with negative connotations. For example, in my work, I examine the Harbor Point mall within the Subic Bay Freeport Zone, Philippines, and compare it to both the SM mall in Olongapo City—which is 30 feet away, on the other side of the Freeport’s gate—and Hanjin Shipping, a Korean-owned shipping and manufacturing company within the Freeport that is known for human rights violations. Although most Harbor Point mall employees cannot afford to purchase lunch within the mall, they prefer working within it because of the higher wages they earn and the relatively more stable employment, when compared to similar work outside the Freeport.
The Subic Bay Freeport Zone, Philippines was originally the location of the former U.S. Subic Bay Naval Base. In another article, in City & Community, I examine how the legacies of the U.S. military continue to influence present day practices and discourses, and Filipino elites’ role in institutionalizing these legacies.
As the job market begins in earnest, I thought I would post a few things to consider for those seeking a tenure-track job at a small liberal arts college, somewhat mythic places that can seem like romantic idylls or claustrophobic hellholes depending on the novel or film.
First, for newer readers of orgtheory, the comments on this post from 2012 are a fantastic guideline for framing your application in terms of the balance between teaching and research:
Second, for those not so sure about whether or not to apply, here are some pros (and associated/implied cons) based on my own personal experience of a decade or so:
PRO 1: We’re hiring and investing in faculty. My own institution is advertising for t-t social science jobs in IA, Anthro, Econ, and American politics this year. I count about 15 t-t liberal arts positions advertised on the ASA job board as of early August, many with teaching loads of 2-2 or 3-2 (sometimes 2-2 in the first year). Endowments of private liberal arts colleges have rebounded since 2008. Teaching loads may vary quite a bit, but leave policies, internal research grants, funding for undergrad research assistants, and externally-funded course development initiatives can be generous as well. Grant for integrating choreography into your classroom, anyone? Do your homework on the health of a school’s endowment, its teaching load and leave policies, and its relationships with foundations like Mellon, Teagle, etc. While the research support does not approach an R1 and the grants office may be small, that also means it is not burdened with a lot of red tape. For this reason, my two collaborators at R1s and I ran our joint ASA-NSF grant through my SLAC’s grants office (thanks Nancy!). I can apply for internal funding for small research expenses quickly; this works a lot better for my not-very-expensive research program than spending oodles of time applying for external grants I may or may not get.
PRO 2: If you like teaching, have a short attention span and eclectic interests, you will have lots of opportunities to indulge your curiosity about millennials’ interests in commune birthing practices, ecotourism, etc. I’ve received course development grants for a sociology of biography course, one on environmental utopias, and one on “Democracy 2.0”, and taken students ziplining, to plays, to museums, Occupy Wall Street, etc. You will most certainly end up attending lectures and dinners on topics way afield of your specialty. Seriously consider if that is something that sounds appealing or burdensome to you. But if you want to collaborate or co-teach with an engineering professor or a GIS expert? Live abroad with students in London or Costa Rica for a semester? Have at it. I took this to the limit by living in a faculty apartment in a “scholars’ house” on campus for 3 years (free room and board and bat abatement!)– but you don’t have to take it that far to enjoy the occasional dinner with a playwright. The flipside of this is, of course, service demands way beyond your department before tenure– on anything from the campus master plan to student conduct.
PRO 3: Small college towns in out of the way places can have great quality of life. As in, two-minute walk to the office. Every day is bring your dog to work day. Cheap rent because you aren’t competing with grad students for housing. Affordable home prices and child care. Free, plentiful faculty parking. For those considering rural or suburban liberal arts colleges but thinking, “I’ll live in Nearest City!”, I strongly urge you to consider living close to campus, at least for the first year or two. For a lot of reasons, this is often impossible, but if that’s the case, you will want to consider carefully before applying whether your home/personal/family life will be sustainable or tolerable with a long commute and commitments on campus at night or on weekends– see PRO 2. Get advice from people who have made it work if this is what you are planning.
Finally, in terms of updating the 2012 advice above, candidates interviewing at SLACs this season should prepare to ask/talk about hot topics like: residential life and how much faculty are expected to participate (this is increasing at a lot of places), assessment practices and attendant demands on faculty (social science fac in particular), interdisciplinarity and any advising or teaching expectations for other programs, and community engagement (ditto on social science faculty demands). Most importantly, don’t assume what the students are like based on school reputation. Beyond the outstanding departmental groupies you will be invited to meet, ask as many faculty as you can what the range of students at a particular school are REALLY like in terms of privilege, politics, partying, writing ability, interests, etc. For all their similar marketing, many liberal arts colleges have very distinctive cultures– and a community of 2000 students may be deeply influenced in great ways by small groups of D1 athletes, nontraditional students, or theater buffs. If you are running out of questions or need to catch your breath, ask a senior faculty member how the student body has changed over time.
There’s plenty more I could say here but I’d love to get comments from other faculty on the liberal arts job market and the lived experience of liberal arts jobs, which perhaps more than others, require a love of the absurd ritual, or at least a healthy sense of humor about collegiate pretensions. So commenters, what did I get wrong or right? Best or worst novels or films about the liberal arts experience? Fire away!
Question: Someone asked me – good soc PhD programs for crim/deviance? Student is interested not only in elite programs (e.g., as long as Rob Sampson is at Harvard, it will produce good crim students) but also lower ranked programs that have solid placement records in crim/deviance. Please use the comment section.