Archive for the ‘guest bloggers’ Category
Cristobal Young is an assistant professor at Stanford’s Department of Sociology. He works on quantitative methods, stratification, and economic sociology. In this post co-authored with Aaron Horvath, he reports on the attempt to replicate 53 sociological studies. Spoiler: we need to do better.
Do Sociologists Release Their Data and Code? Disappointing Results from a Field Experiment on Replication.
Replication packages – releasing the complete data and code for a published article – are a growing currency in 21st century social science, and for good reasons. Replication packages help to spread methodological innovations, facilitate understanding of methods, and show confidence in findings. Yet, we found that few sociologists are willing or able to share the exact details of their analysis.
We conducted a small field experiment as part of a graduate course in statistical analysis. Students selected sociological articles that they admired and wanted to learn from, and asked the authors for a replication package.
Out of the 53 sociologists contacted, only 15 of the authors (28 percent) provided a replication package. This is a missed opportunity for the learning and development of new sociologists, as well as an unfortunate marker of the state of open science within our field.
Some 19 percent of authors never replied to repeated requests, or first replied but never provided a package. More than half (56 percent) directly refused to release their data and code. Sometimes there were good reasons. Twelve authors (23 percent) cited legal or IRB limitations on their ability to share their data. But only one of these authors provided the statistical code to show how the confidential data were analyzed.
Why So Little Response?
A common reason for not releasing a replication package was because the author had lost the data – often due to reported computer/hard drive malfunctions. As well, many authors said they were too busy or felt that providing a replication package would be too complicated. One author said they had never heard of a replication package. The solutions here are simple: compiling a replication package should be part of a journal article’s final copy-editing and page-proofing process.
More troubling is that a few authors openly rejected the principle of replication, saying in effect, “read the paper and figure it out yourself.” One articulated a deep opposition, on the grounds that replication packages break down the “barriers to entry” that protect researchers from scrutiny and intellectual competition from others.
The Case for Higher Standards
Methodology sections of research articles are, by necessity, broad and abstract descriptions of their procedures. However, in most quantitative analyses, the exact methods and code are on the author’s computer. Readers should be able to download and run replication packages as easily as they can download and read published articles. The methodology section should not be a “barrier to entry,” but rather an on-ramp to an open and shared scholarly enterprise.
When authors released replication packages, it was enlightening for students to look “under the hood” on research they admired, and see exactly how results were produced. Students finished the process with deeper understanding of – and greater confidence in – the research. Replication packages also serve as a research accelerator: their transparency instills practical insight and confidence – bridging the gap between chalkboard statistics and actual cutting-edge research – and invites younger scholars to build on the shoulders of success. As Gary King has emphasized, replications have become first publications for many students, and helped launched many careers – all while ramping up citations to the original articles.
In our small sample, little more than a quarter of sociologists released their data and code. Top journals in political science and economics now require on-line replication packages. Transparency is no less crucial in sociology for the accumulation of knowledge, methods, and capabilities among young scholars. Sociologists – and ultimately, sociology journals – should embrace replication packages as part of the lasting contribution of their research.
Table 1. Response to Replication Request
|Yes: Released data and code for paper||15||28%|
|No: Did not release||38||72%|
|Reasons for “No”|
|IRB / legal / confidentiality issue||12||23%|
|No response / no follow up||10||19%|
|Don’t have data||6||11%|
|Don’t have time / too complicated||6||11%|
|Still using the data||2||4%|
|‘See the article and figure it out’||2||4%|
Note: For replication and transparency, a blinded copy of the data is available on-line. Each author’s identity is blinded, but the journal name, year of publication, and response code is available. Half of the requests addressed articles in the top three journals, and more than half were published in the last three years.
Figure 1: Illustrative Quotes from Student Correspondence with Authors:
- “Here is the data file and Stata .do file to reproduce [the] Tables…. Let me know if you have any questions.”
- “[Attached are] data and R code that does all regression models in the paper. Assuming that you know R, you could literally redo the entire paper in a few minutes.”
- “While I applaud your efforts to replicate my research, the best guidance I can offer
is that the details about the data and analysis strategies are in the paper.”
- “I don’t keep or produce ‘replication packages’… Data takes a significant amount of human capital and financial resources, and serves as a barrier-to-entry against other researchers… they can do it themselves.”
Victoria Reyes is an assistant professor at Bryn Mawr in the Growth and Structure of Cities Department. Her research is about specific urban sites in the global system. This post addresses her recent article in Theory and Society.
Thanks to Fabio for allowing me to post about my work. I study global inequality through a cultural and relational lens, and am particularly interested in places of foreign-control, by which I mean places that are either foreign-owned or are heavily influenced by foreigners. I have two recent articles about this (see below for citations and abstracts).
In one that was recently published in Theory and Society, I draw and extend work on global cities and cities along geopolitical borders to develop a concept I call “global borderlands”—semi-autonomous, foreign-controlled, geographic locations geared toward international exchange. These are places like overseas military bases, embassies, tourist resorts, international branch campuses (e.g. NYU Abu Dhabi), and special economic zones, where tariff barriers are relaxed. When I speak of global borderlands, I do not necessarily assume negative connotations. Indeed, some people may enjoy or prefer working, visiting, and/or living within global borderlands, while others are excluded from these places.
I argue that these places have three features in common. First, semi-autonomy and foreign-control. These are places where the notion of “who rules?” is fluid and negotiated, and where regulation depends on nationality. Second, like many other places, global borderlands are defined by geographic and symbolic boundaries. Third, these places are built on unequal relations, by which I refer to structural inequality that again, does not necessarily come bundled with negative connotations. For example, in my work, I examine the Harbor Point mall within the Subic Bay Freeport Zone, Philippines, and compare it to both the SM mall in Olongapo City—which is 30 feet away, on the other side of the Freeport’s gate—and Hanjin Shipping, a Korean-owned shipping and manufacturing company within the Freeport that is known for human rights violations. Although most Harbor Point mall employees cannot afford to purchase lunch within the mall, they prefer working within it because of the higher wages they earn and the relatively more stable employment, when compared to similar work outside the Freeport.
The Subic Bay Freeport Zone, Philippines was originally the location of the former U.S. Subic Bay Naval Base. In another article, in City & Community, I examine how the legacies of the U.S. military continue to influence present day practices and discourses, and Filipino elites’ role in institutionalizing these legacies.
Is management research a folly? If not, whose interests does it serve? And whose interests should it serve?
The questions of good for what and good for whom are worth revisiting. There is reason to worry that the reward system in our field, particularly in the publication process, is misaligned with the goals of good science.
There can be little doubt that a lot of activity goes into management research: according to the Web of Knowledge, over 8,000 articles are published every year in the 170+ journals in the field of “Management,” adding more and more new rooms. But how do we evaluate this research? How do we know what a contribution is or how individual articles add up? In some sciences, progress can be measured by finding answers to questions, not merely reporting significant effects. In many social sciences, however, including organization studies, progress is harder to judge, and the kinds of questions we ask may not yield firm answers (e.g., do nice guys finish last?). Instead we seek to measure the contribution of research by its impact.
Management of humans by other humans may be increasingly anachronistic. If managers are not our primary constituency, then who is? Perhaps it is each other. But this might lead us back into the Winchester Mystery House, where novelty rules. Alternatively, if our ultimate constituency is the broader public that is meant to benefit from the activities of business, then this suggests a different set of standards for evaluation.
Businesses and governments are making decisions now that will shape the life chances of workers, consumers, and citizens for decades to come. If we want to shape those decisions for public benefit, on the basis of rigorous research, we need to make sure we know the constituency that research is serving.
Hi all, I’m Ellen Berrey. I’ll be guest blogging over the next few weeks about inequality, culture, race, organizations, law, and multi-case ethnography. Thanks for the invite, Katherine, and the warm welcomes! Here’s what I’m all about: I’m an assistant professor of sociology at the University at Buffalo-SUNY and an affiliated scholar of the American Bar Foundation. I received my PhD from Northwestern in 2008. This fall, I jet off from the Midwest to join the faculty of the University of Denver (well, I’m actually going to drive the fading 2003 Toyota I inherited from my mom).
As a critical cultural sociologist, I study organizational, political, and legal efforts to address inequality. My new book, The Enigma of Diversity: The Language of Race and the Limits of Racial Justice (University of Chicago Press), is officially out next Monday (yay!). I’ll dive into that in future posts, for sure. I’m writing up another book on employment discrimination litigation with Robert Nelson and Laura Beth Nielsen, Rights on Trial: Employment Civil Rights in Work and in Court. These and my articles and other projects explore organizational symbolic politics, affirmative action in college admissions (also here and here), affirmative action activism (and here), corporate diversity management, fairness in discrimination litigation, discrimination law and inequality (and here), gentrification politics, and benefit corporations.
I’ll kick off today with some thoughts about a theme that I’ve been exploring for many years:
How can powerful, elite-led organizations advance broad progressive causes like social justice or environmental protection? I’m not just referring to self-identified activists but also corporations, universities, community agencies, foundations, churches, and the like. Various arms of the state, too, are supposed to forward social causes by, say, ending discrimination at work or alleviating poverty. To what extent can organizational decision-makers create positive social change through discrete initiatives and policies—or do they mostly just create the appearance of effective action? Time and again, perhaps inevitably, top-down efforts to address social problems end up creating new problems for those they supposedly serve.
To the point: Have you come across great research that examines how organizations can bring about greater equality and engages organizational theory?
I think this topic is especially important for those of us who study organizations and inequality. We typically focus on the harms that organizations cause. We know, for example, that employers perpetuate racial, class, and gender hierarchies within their own ranks through their hiring and promotion strategies. I believe we could move the field forward if we also could point to effective, even inspiring ways in which organizations mitigate inequities. I have in mind here research that goes beyond applied evaluations and that resists the Polly Anna-ish temptation to sing the praises of corporations. Critical research sometimes asks these questions, but it often seems to primarily look for (and find) wrongdoing. Simplistically, I think of this imperative in terms of looking, at once, at the good and bad of what organizations are achieving. Alexandra Kalev, Frank Dobbin, and Erin Kelly’s much-cited American Sociological Review article on diversity management programs is one exemplar. There is room for other approaches, as well, including those that foreground power and meaning making. Together with the relational turn in the study of organizational inequality, this is a promising frontier to explore.
More soon. Looking forward to the conversation.
Raj Ghoshal (www.rajghoshal.com) is an assistant professor of sociology at Goucher College. Claire Whitlinger (www.clairewhitlinger.com) is a graduate student in sociology at Michigan, starting as an assistant professor at Furman University in August. This guest post discusses racial violence and the sociology of collective memory.
In a new special issue of Race & Justice (http://raj.sagepub.com/content/current) on the legacies of past racial violence, we each consider how commemorative projects can impact present-day views and events. As we discuss broad cultural processes, our articles may be of interest to cultural sociologists generally.
In “What Does Remembering Racial Violence Do?” (http://raj.sagepub.com/content/5/2/168.abstract), I (Raj) consider the Greensboro, North Carolina’s Truth and Reconciliation Commission, a private group which in the early 2000s revisited the 1979 killings of five communist demonstrators by the Ku Klux Klan. I draw on a survey of 716 North Carolinians and find that the Commission’s efforts had some, albeit modest, impact on stated support for state redress of past racial injustice toward African Americans. But despite this overall effect, two factors shaped the Commission’s effectiveness in surprising ways.