orgtheory.net

Archive for the ‘fabio’ Category

sociologists need to be better at replication – a guest post by cristobal young

with 36 comments

Cristobal Young is an assistant professor at Stanford’s Department of Sociology. He works on quantitative methods, stratification, and economic sociology. In this post co-authored with Aaron Horvath, he reports on the attempt to replicate 53 sociological studies. Spoiler: we need to do better.

Do Sociologists Release Their Data and Code? Disappointing Results from a Field Experiment on Replication.

 

Replication packages – releasing the complete data and code for a published article – are a growing currency in 21st century social science, and for good reasons. Replication packages help to spread methodological innovations, facilitate understanding of methods, and show confidence in findings. Yet, we found that few sociologists are willing or able to share the exact details of their analysis.

We conducted a small field experiment as part of a graduate course in statistical analysis. Students selected sociological articles that they admired and wanted to learn from, and asked the authors for a replication package.

Out of the 53 sociologists contacted, only 15 of the authors (28 percent) provided a replication package. This is a missed opportunity for the learning and development of new sociologists, as well as an unfortunate marker of the state of open science within our field.

Some 19 percent of authors never replied to repeated requests, or first replied but never provided a package. More than half (56 percent) directly refused to release their data and code. Sometimes there were good reasons. Twelve authors (23 percent) cited legal or IRB limitations on their ability to share their data. But only one of these authors provided the statistical code to show how the confidential data were analyzed.

Why So Little Response?

A common reason for not releasing a replication package was because the author had lost the data – often due to reported computer/hard drive malfunctions. As well, many authors said they were too busy or felt that providing a replication package would be too complicated. One author said they had never heard of a replication package. The solutions here are simple: compiling a replication package should be part of a journal article’s final copy-editing and page-proofing process.

More troubling is that a few authors openly rejected the principle of replication, saying in effect, “read the paper and figure it out yourself.” One articulated a deep opposition, on the grounds that replication packages break down the “barriers to entry” that protect researchers from scrutiny and intellectual competition from others.

The Case for Higher Standards

Methodology sections of research articles are, by necessity, broad and abstract descriptions of their procedures. However, in most quantitative analyses, the exact methods and code are on the author’s computer. Readers should be able to download and run replication packages as easily as they can download and read published articles. The methodology section should not be a “barrier to entry,” but rather an on-ramp to an open and shared scholarly enterprise.

When authors released replication packages, it was enlightening for students to look “under the hood” on research they admired, and see exactly how results were produced. Students finished the process with deeper understanding of – and greater confidence in – the research. Replication packages also serve as a research accelerator: their transparency instills practical insight and confidence – bridging the gap between chalkboard statistics and actual cutting-edge research – and invites younger scholars to build on the shoulders of success. As Gary King has emphasized, replications have become first publications for many students, and helped launched many careers – all while ramping up citations to the original articles.

In our small sample, little more than a quarter of sociologists released their data and code. Top journals in political science and economics now require on-line replication packages. Transparency is no less crucial in sociology for the accumulation of knowledge, methods, and capabilities among young scholars. Sociologists – and ultimately, sociology journals – should embrace replication packages as part of the lasting contribution of their research.

Table 1. Response to Replication Request

Response Frequency Percent
Yes:   Released data and code for paper 15 28%
No: Did not release 38 72%
Reasons for “No”
    IRB / legal / confidentiality issue 12 23%
    No response / no follow up 10 19%
    Don’t have data 6 11%
    Don’t have time / too complicated 6 11%
    Still using the data 2 4%
    ‘See the article and figure it out’ 2 4%
Total 53 100%

Note: For replication and transparency, a blinded copy of the data is available on-line. Each author’s identity is blinded, but the journal name, year of publication, and response code is available. Half of the requests addressed articles in the top three journals, and more than half were published in the last three years.

Figure 1: Illustrative Quotes from Student Correspondence with Authors:

Positive:

  1. “Here is the data file and Stata .do file to reproduce [the] Tables….  Let me know if you have any questions.”
  2. “[Attached are] data and R code that does all regression models in the paper. Assuming that you know R, you could literally redo the entire paper in a few minutes.”

Negative:

  1. “While I applaud your efforts to replicate my research, the best guidance I can offer
    is that the details about the data and analysis strategies are in the paper.”
  2. “I don’t keep or produce ‘replication packages’… Data takes a significant amount of human capital and financial resources, and serves as a barrier-to-entry against other researchers… they can do it themselves.”

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 11, 2015 at 12:01 am

global borderlands – a guest post by victoria reyes

Victoria Reyes is an assistant professor at Bryn Mawr in the Growth and Structure of Cities Department. Her research is about specific urban sites in the global system. This post addresses her recent article in Theory and Society.

Thanks to Fabio for allowing me to post about my work. I study global inequality through a cultural and relational lens, and am particularly interested in places of foreign-control, by which I mean places that are either foreign-owned or are heavily influenced by foreigners. I have two recent articles about this (see below for citations and abstracts).

In one that was recently published in Theory and Society, I draw and extend work on global cities and cities along geopolitical borders to develop a concept I call “global borderlands”—semi-autonomous, foreign-controlled, geographic locations geared toward international exchange. These are places like overseas military bases, embassies, tourist resorts, international branch campuses (e.g. NYU Abu Dhabi), and special economic zones, where tariff barriers are relaxed. When I speak of global borderlands, I do not necessarily assume negative connotations. Indeed, some people may enjoy or prefer working, visiting, and/or living within global borderlands, while others are excluded from these places.

I argue that these places have three features in common. First, semi-autonomy and foreign-control. These are places where the notion of “who rules?” is fluid and negotiated, and where regulation depends on nationality. Second, like many other places, global borderlands are defined by geographic and symbolic boundaries. Third, these places are built on unequal relations, by which I refer to structural inequality that again, does not necessarily come bundled with negative connotations. For example, in my work, I examine the Harbor Point mall within the Subic Bay Freeport Zone, Philippines, and compare it to both the SM mall in Olongapo City—which is 30 feet away, on the other side of the Freeport’s gate—and Hanjin Shipping, a Korean-owned shipping and manufacturing company within the Freeport that is known for human rights violations. Although most Harbor Point mall employees cannot afford to purchase lunch within the mall, they prefer working within it because of the higher wages they earn and the relatively more stable employment, when compared to similar work outside the Freeport.

The Subic Bay Freeport Zone, Philippines was originally the location of the former U.S. Subic Bay Naval Base. In another article, in City & Community, I examine how the legacies of the U.S. military continue to influence present day practices and discourses, and Filipino elites’ role in institutionalizing these legacies.

Read the rest of this entry »

Written by fabiorojas

August 10, 2015 at 12:01 am

janel and anthony, big sur (2007)

Yup, we’re going into an alt.folk phase. Deal with it.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 9, 2015 at 12:01 am

crim/deviance PhD programs?

Question: Someone asked me – good soc PhD programs for crim/deviance? Student is interested not only in elite programs (e.g., as long as Rob Sampson is at Harvard, it will produce good crim students) but also lower ranked programs that have solid placement records in crim/deviance. Please use the comment section.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 7, 2015 at 12:01 am

Posted in fabio, grad school rulz

blogging will not ruin your career

A few years ago, a friendly senior scholar sat me down and warned me: “Fabio, you really have to stop this blogging thing. It’s not good for you.” This was a pretty serious conversation. He meant it as kind advice from a more seasoned friend and colleague. I smiled, mumbled, and said, “Um, ok, thank you.” Then, I proceeded to blog as usual.

I figured that blogging wasn’t bad as long as (a) you did not let it displace your actual job and (b) your blog posts were professionally written and not shrill in tone. I only seriously blogged once my book on Black Studies was published and had a solid list of articles under my belt. Also, I made sure that the bulk of my blogging was about sociology, org studies, and professional issues (e.g., the Grad Skool Rulz). I have never regretted my choice to continue writing in this format. People still like reading it and I get lots of positive feedback.

As I look down the orgtheory crew list, I see that I’m not alone. Other orgtheory crew members are doing quite well despite their association with this blog. Omar, most famously, is now an editor of ASR, our flagship journal. Brayden is doing well at the leading b-school in the US. Kieran not only blogs here on occasion, but also his personal blog and at Crooked Timber. He’s doing quite well at Duke and his data visualizations are often picked up my major news media. Katherine has written an award winning book and Elizabeth is graduate chair at Albany. Sean directs the MA program at Sciences Po in France and Teppo has just become chair of his department at Oxford.

This is obviously a selection effect. I think it’s silly to think that these folks weren’t already top notch when they participated in this blog. It speaks well, though, of Teppo and Brayden, who founded this blog and reached out to so many people who have excelled in the profession.

The lesson I have for early career readers is this: When there is something new, something that doesn’t fit the mold, you shouldn’t run away from it. Don’t be scared to reach out and develop your voice. Surround yourself with good people. As long as you write from a position of integrity and respect for the reader, it will be ok.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 6, 2015 at 12:01 am

Posted in blogs, fabio

the parsons taboo

As readers already know, I am hard at work on a book that reviews contemporary sociology. In writing the book, I ran into two taboos: rational choice and Parsons (ironic, since Parsons was opposed to utilitarianism). The reviewers were very touchy about these two topics. The first makes sense. Sociology has always been allergic to anything “econ-y” or “math-y” from the beginning. I do understand why people might want to expunge a book of rational choice. I still don’t think it’s wise since the profession still has people working in related areas like Granovetter style embeddness research, social capital, Harrison White micro-network hybrid work, and applied game theory. Also, the rational choice tradition (including social capital) is the major link between sociology and the poli sci/economics axis.

The Parsons taboo really surprised me since (a) the book only had a total of about five paragraphs about Parsons, (b) I am definitely not a functionalist and I present it as background for more modern stuff like cultural sociology and institutionalism, and (c) Parsons’ descendants still have big followings, like Jeffrey Alexander and Niklas Luhmann. Also, another weird thing is that the reviewers asked me to incorporate Swidler’s recent work (Talk of Love), a discussion of Poggi’s theory of power and Vaisey’s work, which all explicitly speak of Parsons.

So what is up with this weird allergy to even *mentioning* Parsons? In 2015, are people still fighting the battles of 1975? Here’s my theory. Parsons’ did two things, one bad and one good. The bad thing is that he created a highly visible and rigid orthodoxy, complete with “religious” texts (i.e., his books). That is what the sociologists of the 1970s revolted against and that is what made Parsons the devil in our profession. And I can’t blame people. Reading classic structural functionalist texts is really taxing and frequently unhelpful.

The good thing is that he created, by accident, the kernel of a lot of modern sociology. Inside those big, nasty books, there were a lot of important insights that are now standard. For example, his 1959 ASQ article on organizations made the crucial distinction between the technical and “institutional” components of organizations, a core idea in modern organizational research. The functionalist approach to schools is still a standard reading. The distinction between achieved and ascribed status is “strat 101.” Even his much maligned theory of norm driven action lives on, even if we admit that norms are constructed situationally rather than ex ante.

The “good” and “bad” Parsons explains my situation. You don’t have to be a functionalist to appreciate some of his good ideas, nor do you need to be a hard core follower to understand the historical importance of Parsons. For example, you simply can’t understand why Swidler’s (1983) toolkit argument was such a big deal unless you understand how Parsons’ theory of norms and his interpretation of the Protestant Ethic was dominant at the time. The Swidler critique set the agenda for cultural sociology for decades.  So you need to address Parsons and point out the contribution. If you do that, however, people get angry because they remember (or their advisers told them) about the bad Parsons.

This also helps explain when and where you can get away with it. If the whole text is about critiquing work like Parsons and developing alternatives (e.g., Swidler or Vaisey), you can do it. If you are very senior scholar who is writing “big think” work (e.g. Gianfranco Poggi), you can do it. But not a synthetic and pedagogical overview – people will think that even including him (or the rational choicers) is a horrendous rear-guard action that puts discredited work back into the canon.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

Written by fabiorojas

August 5, 2015 at 12:01 am

no duty to obey migration law: a kingian analysis

How do we know if restrictionism is unjust? Is it ethically good or bad to prevent migration between countries? In this post, I draw on Martin Luther King Jr.’s Letter from a Birmingham Jail to argue that restriction laws are unjust. King sets out the problem and offers a solution: “How does one determine whether a law is just or unjust? A just law is a man made code that squares with the moral law or the law of God. An unjust law is a code that is out of harmony with the moral law.” In other words, there is no intrinsic demand that the law be followed. You don’t have to follow the law just because it is a law.

But there arises a problem, how do we know if a law is in accordance with “moral law?” King begins by pointing out that just laws try to help people, but unjust laws degrade people and create privilege and superiority: “Any law that uplifts human personality is just. Any law that degrades human personality is unjust. All segregation statutes are unjust because segregation distorts the soul and damages the personality.”

King makes a behavioral argument. If a law were indeed just, the group that passed the law would apply it to themselves. “An unjust law is a code that a numerical or power majority group compels a minority group to obey but does not make binding on itself. This is difference made legal. By the same token, a just law is a code that a majority compels a minority to follow and that it is willing to follow itself. This is sameness made legal. Let me give another explanation. A law is unjust if it is inflicted on a minority that, as a result of being denied the right to vote, had no part in enacting or devising the law.”

Let us now turn to border controls and deportation. How do these laws “uplift human personality?” Restrictionist laws clearly do not “uplift” the people who are banned from entry. People who migrate may want jobs, or they want to be with family, or they simply want to be in another place that they deem safe. By preventing people from being with family, they clearly degrade people. By preventing people from earning a living and peacefully enjoying property, they degrade people.

What about the native citizen? How do restrictionist laws “uplift” her personality? They can’t because they are aimed at others. One’s moral standing is based on their action, not the action of others. Simply living in a nation that excludes others does nothing for one’s moral worth. To the contrary, the active approval of laws that degrade others decreases one’s moral standing. Supporting migration restrictions and deportations gives, in King’s words, the restrictionist a “false sense of superiority and the segregated a false sense of inferiority.”

King’s behavioral criteria implies that migration restrictions are unjust. If these laws are so wise and proper, then why do we not see border controls between the states of the Union? Or between different cities? If the restrictionist is truly concerned about the dangers of outsiders, why shouldn’t Northerners build a fence to keep Southerners out because of their different values? Should Catholics and Protestants dig a moat around Utah to stop Mormons from entering their territory? If the restrictionist is truly concerned about outsiders exploiting public assistance, why doesn’t New York City build a wall to prevent New Hampshire’s citizens from exploiting that state’s more generous government services? The fact that such walls do not exist, and the restrictionists do not ask for them, says to me that these laws can’t truly be just.

From the founding of the United States to the Chinese Exclusion Act of 1882, this nation had 106 years of free movement across its borders. Millions of Jews, Poles, Chinese, Mexican, German, and Russian people came to this shore, relieved that they could live their lives as they saw fit, free from deportation, exile, and murder. Since then, wave after wave of anti-immigration law has been passed by this nation’s citizens. It is time to recognize that these laws are unwise and unjust and have same moral standing as the laws of Jim Crow. They do not command respect or honor and should be seen for what they are: attempts to harass people who, by chance, were born in another nation. Ignore them at will.

50+ chapters of grad skool advice goodness: Grad Skool Rulz ($2!!!!)/From Black Power/Party in the Street

 

Written by fabiorojas

August 4, 2015 at 12:01 am

Posted in current events, fabio

Follow

Get every new post delivered to your Inbox.

Join 3,253 other followers