Archive for the ‘research’ Category
In a previous post’s discussion, Graham Peterson kindly shared a great link of videos with Howie Becker’s thoughts about conducting research, specifically at the graduate level. I found the last video clip of particular interest. In “10. Savoir finir : Comment achever une thèse alors que les données de terrain ne cessent d’affluer ? [Knowing how to end: how to finish a thesis when field data keeps arising?],” Becker discusses the issue of knowing when to end data collection. Some signals make this clear – the funding runs out, the return ticket’s date shows up, or less frequently, the field site closes. (For historians, the re-closure of an archive is the equivalent.)
I would add another possibility – sometimes the data collection doesn’t end, and a researcher continues to analyze, write, and publish along the way, particularly if the phenomena under study changes and sparks additional areas of inquiry. New research questions may arise, or the researcher may add other field sites for comparison. As I tell grad students who are deciding among research projects, it’s likely that a researcher will live with an ethnographic research project for years beyond the dissertation’s completion, particularly if s/he writes a book and needs to publicize it. Of course, out of expedience or boredom, some researchers will quickly move onto another research project. However, with ethnographic research, the researcher who can return to the same field site faces the dilemma of sunk costs – forming relations with informants and developing expertise and local knowledge all take time, and it may be difficult to give all that up, especially when the passage of time starts to reveal dynamics not readily apparent before.
I keep hearing about the coming big data revolution. Data scientists are now using huge data sets, many produced through online interactions and media, that shed light on basic social processes. Big data data sets, from sources like Twitter, Facebook, or mobile phones, give social scientists ways to tap into interactions and cultural output at a scale that has never been seen before in social science. The way we analyze data in sociology and organizational theory are bound to change due to this influx of new data.
Unfortunately, the big data revolution has yet to happen. When I see job candidates or new scholars present their research, they are mostly using the same methods that their predecessors did, although with incremental improvements to study design. I see more field experiments for sure, and scholars seem more attuned to identification issues, but the data sources are fairly similar to what you would have seen in 2003. With a few notable exceptions, big data have yet to change the way we do our work. Why is that?
Last week Fabio had a really interesting post about brain drain in academia. One reason we might see less big data than we’d like is because the skills needed to handle this type of analysis are rare and much of the talent in this area is finding that research jobs in the for-profit world are more lucrative and rewarding than what they’re being offered in academia. I believe that’s true, especially for the kinds of people who are attracted to data mining techniques. The other problem though, I think, is that social scientists are having a hard time figuring out how to fit big data techniques into the traditional milieu of social science. Sociologists, for example, want studies to be framed in a theoretically compelling way. Organizational theorist would like scholars to use data that map on to the conceptual problems of the field. It’s not always clear in many of the studies that I’ve read and reviewed that big data analyses are doing anything new other than using big data. If big data studies are going to take over the field they need to address pressing theoretical problems.
With that in mind, you should really read a new paper by Chris Bail (forthcoming in Theory and Society) about using big data in cultural sociology. Chris makes the case that cultural sociology, a subfield that is obsessed with understanding the origins of and practical uses of meaning, is prime for a big data surge. Cultural sociology has the theoretical questions, and big data research offers the methods.
More data were accumulated in 2002 than all previous years of human history combined. By 2011, the amount of data collected prior to 2002 was being collected very two days. This dramatic growth in data spans nearly every part of our lives from gene sequencing to consumer behavior. While much of these data are binary and quantitative, text-based data is also being accumulated on an unprecedented scale. In an era of social science research plagued by declining survey response rates and concerns about the generalizability of qualitative research, these data hold considerable potential. Yet social scientists – and cultural sociologists in particular – have ignored the promise of so-called ‘big data.’ Instead, cultural sociologists have left this wellspring of information about the arguments, worldviews, or values of hundreds of millions of people from internet sites and other digitized texts to computer scientists who possess the technological expertise to extract and manage such data but lack the theoretical direction to interpret their meaning in situ….[C]ultural sociologists have made very few ventures into the universe of big data. In this article, I argue inattention to big data among cultural sociologists is particularly surprising since it is naturally occurring – unlike survey research or cross-sectional qualitative interviews – and therefore critical to understanding the evolution of meaning structures in situ. That is, many archived texts are the product of conversations between individuals, groups, or organizations instead of responses to questions created by researchers who usually have only post-hoc intuition about the relevant factors in meaning-making – much less how culture evolves in ‘real time’ (note: footnotes and references removed).
Chris goes on to offer suggestions about how cultural sociology might use big data to address big theoretical questions. For example, he believes that scholars studying discursive fields would be wise to use big data methods to evaluate the content of such fields, the relationships between actors and ideas, and the relationships between different fields. Of course, much of the paper is about how to use big data analysis to enhance or replace traditional methods used in cultural sociology. He discusses how Twitter and Facebook data might supplement newspaper analysis, a fairly common method in cultural and political sociology. Although he doesn’t go into great detail about how you would do it, an implicit argument he makes is that big data analysis might replace some survey methods as ways to explore public opinion.
I continue to think there is enormous potential for using big data in the social sciences. The key for having it accepted more broadly is for data scientists to figure out how to use big data to address important theoretical questions. If you can do that, you’re gold.
Two uncomfortable, if not disconcerting, realizations of academic publishing is that (1) people don’t read, or (2) if they do read and write, they don’t cite relevant or appropriate work. A hard-working academic’s day can be quickly sent down the dark hole of despair or rage face when reading a manuscript or publication that doesn’t properly cite relevant work or (ahem) one’s own seminal work. Worse, if cited, the work may be cited wrongly, or the minor points override the major takeaways in subsequent research and citations. In these situations, the imperative that one should contribute to standing on the shoulders of giants calls to mind the bleak image of a whale fruitlessly calling out for colleagues in an endless sea.
More whale-calling, channeling Andy Abbott, below the fold…
Read the rest of this entry »
For our budding orgheads, a new doctoral research award of possible interest:
“Emerald and the European Foundation for Management Development (EFMD) announce the launch of the 2013 Emerald/EFMD Outstanding Doctoral Research Awards.
These Awards reward the best doctoral research projects in 12 different categories, each sponsored by an Emerald journal. For a list of all categories and details of how to apply please visit the webpage below:
The Awards are open to those who have completed and satisfied examination requirements for a Doctoral award, or will do so, between 1 October 2010 and 1 October 2013, and have not applied previously for one of these Awards. The closing date for receipt of applications is 1 October 2013.
Award-winning entries will receive a cash prize of €1,500 (or currency equivalent), a certificate and a winners’ logo to attach to correspondence.”
Last Sat., NYU’s Institute for Public Knowledge hosted the conference “LOVE AND LET DIE:
An all-day consideration of Ballooning Longevity, the Quality of Life, and the Coming Generational Smash-up.” Conference speakers, whose ranks included doctors, anthropologists, and caregivers, described the difficulties of dying a “dignified” death in the US. When contemplating one of the few sureties of life, people wish to die painlessly and effortlessly, preferably in their sleep. However, technology and medical interventions have made this less likely a prospect for some, particularly for hospitalized older adults.
Among some conference attendees, the American medical care system raised concerns about conflicts of interest. For example, one audience member worried that as he ages, he will become a “cash cow” for the medical industry, in which institutions order possibly unnecessary tests to secure reimbursements from Medicare. Speakers noted how medical professionals go through heroic efforts in providing medical care for some conditions, but these efforts may both diminish the patients’ quality of life and even curtail their lifespan compared to palliative care. Interestingly, some doctors do not want certain kinds of medical intervention such as CPR or ventilator use for themselves.
Medical anthropologist and author of And a Time to Die: How Hospitals Shape the End of Life (UChicago 2006) Sharon Kaufman described how the “upstream” work of pharmaceutical companies, medical device manufacturers, and biotech shapes the kinds of medical care people now receive. Kaufman detailed a fascinating system in which corporate-driven research trials and Medicare reimbursements for experimental medical devices can push patients to choose medical care that may not really reflect their preferences. Usually patients and family members do not have the expertise or knowledge to fully comprehend the consequences of installing, say, a defibrillator. (Hint: dying peacefully in one’s sleep may not be possible when a device can automatically shock the heart.) In addition, increasingly effective immunosuppressants have allowed medical interventions that were not possible before, such as an adult child donating a kidney or other organ to a parent, raising additional dilemmas in decision-making. Such issues highlight the difficult decisions in store for some.
If we are “lucky,” many of us will face some form of these questions about expectations for the end of life for ourselves, friends, and loved ones. The presenters recommended having conversations about one’s wishes with doctors, who order care, and loved ones. However, not all medical personnel and institutions will abide by documented wishes. Thus, presenters urged larger societal discussions about these issues.
A sample scene from Wild Strawberries:
A joke shared by a conference speaker:
“There are 3 kinds of people in the world: 1 that can count and 1 that can’t.”
For seventy five years, Harvard University has conducted a longitudinal study of 269 men who graduated in 1938. It’s an attempt to learn, in detail, about the factors that might contribute to a good life. Business Insider has a nice summary of a new book, Triumphs of Experience, that presents the results of the study. A few take home points:
- Alcoholism is a disorder of great destructive power.” Alcoholism was the main cause of divorce between the Grant Study men and their wives; it was strongly correlated with neurosis and depression (which tended to follow alcohol abuse, rather than precede it); and—together with associated cigarette smoking—it was the single greatest contributor to their early morbidity and death.
- Above a certain level, intelligence doesn’t matter. I assume that it doesn’t matter for the types of life course outcomes social scientists measure (employment, health, happiness, marriage).
- Relationships matter, a lot: “Men who had “warm” childhood relationships with their mothers earned an average of $87,000 more a year than men whose mothers were uncaring.” and “Late in their professional lives, the men’s boyhood relationships with their mothers—but not with their fathers—were associated with effectiveness at work.”
- Dad matters as well: “warm childhood relations with fathers correlated with lower rates of adult anxiety, greater enjoyment of vacations, and increased “life satisfaction” at age 75—whereas the warmth of childhood relationships with mothers had no significant bearing on life satisfaction at 75.”
The formula for a good life: no alcohol or smoking; be nice too people, especially your kids; and you’re probably good enough to get what you want out of life.
Last Fri., I attended a talk by Sarah Babb of Boston College. In her talk, titled “Beyond the Horror Stories: Non-Experimental Social Researchers’ Encounters with Institutional Review Boards (IRB),” Babb revealed findings that included misconceptions about federal guidelines for human subjects. Contrary to what some IRB review boards demand from principal investigators (PIs) undertaking qualitative research, the federal guidelines do not require:
– signed consent from a low risk population
– an institutional research permission slip
To repeat, the above two are “not in federal regulations at all.”
Babb noted that at larger institutions, IRB boards often involve nonprofessionals – that is, those who don’t have appropriate professional expertise – in the decision-making processes about proposals. Moreover, qualitative research don’t fit well into the one-size-fits-all medical template often used to vet research proposals. Compounding these challenges is the lack of accountability in terms of IRB review boards’ responsibilities to PIs. Only 20% of IRBs that Babb examined had an appeals procedure that would allow PIs to contest decisions.
Not surprisingly, this talk evoked spirited discussion of the myriad problems encountered by researchers going through the IRB process at their institutions, as well as the unintended consequences of a review process ostensibly intended to protect human subjects. The audience noted the following unintended and undesired consequences: (1) normalized deviance,* (2) chilling effect upon the types of research undertaken, and (3) mission creep in which IRB review boards critique the suitability or worth of the research design, rather than evaluating risk to human subjects. In particular, senior researchers worried that tenure-track faculty and graduate students face great uncertainty about whether their project proposals will successfully navigate the IRB process in a timely fashion.
Audience members asked whether the sociologists’ professional association, the American Sociological Association (ASA), had taken an official position on IRB guidelines. None present were aware of any such activities (if you know of anything brewing from this or other associations, do write them in the comments). Attendees noted that because a tenured faculty member may be more able to surmount IRB issues on his/her own (or not need to go through the IRB process because of the type of research conducted), fashioning IRB standards that are more appropriate for a wider variety of research methods is a collective action problem.
I opined that these identified problems need to be considered a commons issue. Those with more power should consider it a professional responsibility to help budding researchers – undergraduate students, graduate students, junior faculty – go through an IRB process that is appropriate to their research methods and questions, especially if researchers hope to have future generations of audiences and colleagues. Unfortunately, dark humor may not be sufficient to get the point across – when a psychology colleague sent his IRB board a proposal to reproduce the Stanley Milgram experiment on April Fool’s Day, an IRB staffer called to inquire if the proposal was serious.
* One of my past posts discussing the IRB draws a steady stream of traffic from those searching for the answer to one of the quiz questions on the online Collaborative Institutional Training Initiative (CITI), a certification program mandatory for researchers and students at some institiutions.
Jenn Lena broke the news before I could. I’ll add my excitement and say that creating an open source sociology journal with a fast and limited review process that allows online comments and community engagement is something that needed to happen. And it IS happening. In Fall 2013 you can submit your papers to Sociological Science and, if you get through the evaluation process, you can see your paper published within months of submission. One of the most exciting aspects of the journal is how reviews work. Rather than forcing authors to go through months (or years) of agonizing back-and-forth with reviewers, the editors will make an up-or-down decision based on an initial review. The reviews will be evaluative, not developmental. Once published, readers can respond to articles and “challenge or extend other people’s work.” Publication will be continuous, and so as soon as your article has been accepted and edited, it will go online as a published article.
I think the journal is going to fill an important niche in sociology. I hope that one consequence of the journal will be to pressure other journals to speed up the process and to make publications be more interactive. It’s still too early to tell how the journal will fare in attracting high quality papers. I sincerely hope that people will send some of their best stuff to the journal. If they do, then I wonder what consequence this will have for the vast set of secondary/specialist journals in our field. Journals like Social Forces and Social Problems will be those most likely to take hits.
I have a bleg. What do you think are the best organizational theory papers published in a sociology or management journal in 2012? I’m on a nominations committee and I don’t want to miss anything. Let me know what you think in the comments.
Unit of analysis: US House elections in 2010 and 2012. X-Axis: (# of tweets mentioning the GOP candidate)/(# of tweets mentioning either major party candidate). Y-axis: GOP margin of victory.
I have a new working paper with Joe DiGrazia*, Karissa McKelvey and Johan Bollen asking if social media data actually forecasts offline behavior. The abstract:
Is social media a valid indicator of political behavior? We answer this question using a random sample of 537,231,508 tweets from August 1 to November 1, 2010 and data from 406 competitive U.S. congressional elections provided by the Federal Election Commission. Our results show that the percentage of Republican-candidate name mentions correlates with the Republican vote margin in the subsequent election. This finding persists even when controlling for incumbency, district partisanship, media coverage of the race, time, and demographic variables such as the district’s racial and gender composition. With over 500 million active users in 2012, Twitter now represents a new frontier for the study of human behavior. This research provides a framework for incorporating this emerging medium into the computational social science toolkit.
The working paper (short!) is here. I’d appreciate your comments.
* Yes, he’ll be in the market in the Fall.
In this last post, I’ll discuss why I fundamentally disagree with the argument presented in Reinventing Evidence. There are two reasons. First, I agree with Andrew Perrin that Biernacki wants us to embrace a textual holism. One of Biernacki’s major arguments is that by isolating a single word, or passage, we are losing the entire meaning of the text. Thus, interpretation is the only valid approach to text. Coding and quantification is invalid. Perrin points out that lots of things be isolated. For example, if I see the n-word, I can say that, on the average, the text is employing racist language.
Second, Biernacki does not seem to consider cultural competence. In other words, human beings are creatures that can often reliably capture the meaning of utterances made by other humans from the same cultural group. Of course, I am talking about things like every day speech or short and simple writings like newspaper articles. More complex texts, like novels, will have networks or dense layering of meaning that go beyond a human’s native capacity for communication. These probably could be coded, but it would require intense training and an elaborate theory of text, which sadly we don’t have in sociology. But my major point remains. There’s a lot of fairly simple text that can be coded. If you believe that people can accurately convey the meaning of a text or label some aspect of it because they are “native speakers” of the culture, then coding is a valid thing to do. To believe otherwise, is to assume a world of solipsistic culture where every act of utterance requires a stupendous level of interpretation on the part of the audience.
So to wrap things up. I give credit to Biernacki for making us think hard about the quality of coding which is lacking. The fact that science is presented in ritual is fair, but doesn’t address whether a particular procedure produces valid measurement or inference. And I think that the view that texts are essentially uncodable is in error.
I am not sure what this accomplishes, but some journalists are trying to get further records from the University of Central Florida, where the editor of Social Science Research works. From the website of activist and author John Becker:
Despite the wide reach of the New Family Structures Study, much about the process by which it was peer-reviewed and published by the journal Social Science Research remains unknown. We know that the timetable was extraordinarily compressed — according to data from the University of Texas and SSR, Regnerus submitted his paper 20 days before the end of the data collection period and 23 days before the data file was delivered to the university. Sounds fishy, doesn’t it? And the entire process, including the paper’s initial submission, review, revision, and acceptance, took place within six weeks. But why? What are the reasons for moving so quickly? Did Regnerus just catch a lucky break, or is there more to the story? We already know that his funders had an anti-gay agenda and the study itself was plagued by troubling conflicts of interest; were the peer review and publication processes similarly compromised?
Last month, I filed a Freedom of Information Act request with the University of Central Florida, which houses Social Science Research, seeking public records relating to the peer review and publication of the New Family Structures Study. My goal is simply to discover the truth: whether everything was above board and best practices and ethical standards were followed, or whether something more sinister occurred. The documents I requested from UCF may help to answer these important questions.
This Spring, our book forum will address Richard Biernacki’s Reinventing Evidence in Social Inquiry: Decoding Facts and Variables. In this initial post, I’ll describe the book and give you my summary judgment. Reinventing Evidence, roughly speaking, claims that numerically coding extended texts is a very, very bad idea. How bad? It is soooo bad that sociologists should just stop coding text and abandon any hope of providing a quantitative or numerical coding of texts or speech. It’s all about interpretation. This is an argument that prevents a much needed integration of the different approaches to sociology, and it deserves a serious hearing.
In support of this point, Biernacki does a few things. He makes an argument about how coding text lacks validity (i.e., associating a number to a text does correctly measure what we want it to measure). Then he spends three chapters going back to well known studies that use content analysis and argues, at varying points, that the coding is misleading, obviously incorrect, or that there were no consistent standard for handling the text or the data.
As a proponent of mixed methods, I was rather dismayed to read this argument. I do not agree that coding of text is a hopeless task and that we should retreat into the interpretive framework of the humanities. There seem to be regularities in speech, and other text, that makes us want to group them together. If you accept that statement, then it follows that a code can be developed. So, on one level I don’t buy into the main argument of the book.
At a more surface level, I think the book does some things rather well. For example, the meat of the book is in replication, which many of us, like Jeremy Freese, have advocated. Biernacki goes back and examines a number of high profile publications that rely on coding texts and finds a lot to be desired.
Next week, we’ll get into some details of the argument. Also, please check out our little buddy blog, Scatterplot. Andrew Perrin will discussing the book and offering his own views.
As I posted earlier, I’ll be presiding over a conversation between George Ritzer and Carmen Sirianni from 3:30-5pm on Fri., March 22, 2013 at ESS in the Whittier Room (4th Flr) of the Boston Park Plaza hotel.
In the past several years, disasters like Hurricane Sandy and Katrina have sparked growing interest in what both conventional and innovative organizations can (and cannot) do given conditions of uncertainty vs. certainty. Both featured scholars’ work cover the limits of particular organizing practices (i.e., Ritzer’s work on McDonaldization), as well as the potential of organized action (i.e., Sirianni’s work on collaborative governance). Thus, I’ve given this particular conversation the broad title “Organizations and Societal Resilience: How Organizing Practices Can Either Inhibit or Enable Sustainable Communities.”
What would you be interested in hearing Ritzer and Sirianni discuss about organizations and society? Please put your qs or comments in the discussion thread.
For those unfamiliar with Ritzer and Sirianni, here is some background about their work:
George Ritzer is best known for his work on McDonaldization and more recently, the spread of prosumption in which people are both producers and consumers.
J. Mike Ryan‘s interview of Ritzer about his McDonaldization work:
J. Mike Ryan’s interview of Ritzer about why we should learn about McDonaldization (corrected link):
Carmen Sirianni is known for his work on democratic governance.
A brief video of Sirianni arguing that citizens should be “co-producers” in building society.
A more extensive video of Sirianni presenting on his book Investing in Democracy: Engaging Citizens in Collaborative Governance (Brookings Press, 2009).
Over a week ago, a colleague called to let me know that our advisor, Harvard Prof. J. Richard Hackman, had passed. For months, I knew that this news would eventually come, but it’s still painful to accept. I will miss hearing Richard’s booming voice, having my eyeglasses crushed to my face from a bear hug (Richard was well over 6 feet tall), or being gleefully gifted with a funny hand-written note imparting his sage advice on a matter.
Richard was a greatly respected work redesign and teams researcher. At Harvard, his classes included a highly regular and popular (despite its “early” morning time slot) course on teamwork. For those undergraduate and graduate students who have been lucky enough to take Richard’s course on teams, the course interweaves concept and practice as students must work in teams, something that most of us get very little practice with outside of organized sports or music.
In July 2012, Richard emailed several of his former teaching fellows asking us to join him in Cambridge and help him rework this course. On short notice, we assembled at the top floor of William James Hall and went over the materials, with Richard expertly leading us as a team, with clearly designated boundaries (those of us assembled for the task), a compelling direction (revising the material to attract students across disciplines), enabling structure (norms that valued contributions of team members, no matter their place in the academic hierarchy), and a supportive context (reward = tasty food, an incentive that always works on former graduate students, and good fellowship).
During this last meeting, Richard asked us about how we thought his course on teamwork could most impact individuals. I opined that his biggest impact wouldn’t be through just the students who took his course, but via those of us who would continue to teach teamwork and conduct research in other settings. This question may have been Richard’s gentle way of telling us that he was passing on the baton.
Here are several ways that I think Richard’s legacy lives on.
Read the rest of this entry »
Recently, at a faculty meeting of professors and graduate students from several disciplines, discussion turned to the IRB’s interpretation of human subjects guidelines and the implications for students’ efforts to document phenomena for class assignments. Participants pointed out a variety of problems, including changes over the years in IRB decisions about whether results of projects could be publicly shared – in this case, whether students’ videorecorded interview of a retired elected official could be publicly shared under today’s IRB guidelines. Faculty and graduate students also described delays in getting feedback from their IRBs, raising concerns about how the lack of accountability on the part of some IRBs increases the uncertainty of planning class research, students’ timely graduation, and faculty productivity.
At orgtheory, we’ve discussed how researchers face challenges concerning the IRB here and here. Although the IRB offers detailed guidelines that can protect human subjects in medical research, how the IRB and human subjects concerns can contribute to the conduct of qualitative research, particularly organizational ethnography, is less clear.
Several recent publications offer researchers’ experiences with these issues.
Is there any relationship between accusations of corporate deviance and the diffusion of new practices? My coauthor, Ed Carberry, and I think so. In a new paper that just came out in the Journal of Management Studies we show that firms began using stock option expensing, a practice that used to be seen as quite problematic and undesirable by executives and boards, after a series of scandals rocked the corporate world in the early 2000s, causing firms to look for new ways to restore their credibility. Stock option expensing became a tool that companies could use to distance themselves from the stigma associated with corporate scandal. Our analyses show that firms facing media scrutiny around claims of corporate fraud and firms that were targets of shareholder activism around corporate governance were much more likely adopt stock option expensing. Firms that faced both intense media scrutiny and shareholder activism were especially likely to adopt the practice. We argue that in the period directly following the Enron scandal stock option expensing became seen as an impression management tactic that firms could use to restore confidence in their accountability to the public.
The title of the paper is “Defensive Practice Adoption in the Face of Organizational Stigma: Impression Management and the Diffusion of Stock Option Expensing.” You can download the paper on my website. Here is the abstract.
Although most diffusion research focuses on firms adopting new practices to maintain their legitimacy, this paper examines a setting in which firms adopted a controversial practice to defend themselves against relating to corporate deviance. We argue that understanding defensive adoption requires attending to both the dynamics of organizational stigma and impression management, and test our theoretical claims by analysing the diffusion of an accounting practice, stock option expensing (SOPEX), following the Enron scandal. We first provide evidence that the media and shareholder activists transformed the practice into a defensive device by theorizing it as a solution to problems relating to corporate fraud and corporate governance. Using event history analysis, we then show that corporations that became targets of stigma- inducing threats were more likely to adopt SOPEX and that the media were a key force channeling these threats.
Academy of Management Review, AMR, has a call out for a special issue on “communication, cognition and institutions.” Be sure to check it out.
Former orgtheory guester Eero Vaara is involved as an editor, along with John Lammers, Joep Cornelissen, Rodolphe Durand and Peer Fiss. Submissions are due July, 2013.
The most recent Nature features an article by a team of political scientists and network scholars who did an experiment using Facebook to show that strong ties influenced voting behavior in the last election. You may say, so what? We’ve known for a long time that social influence operates through strong ties in interpersonal networks. That’s not a new insight. But I think the study is innovative for a couple of reasons. The first is that the impact of of using direct messaging through Facebook was substantively significant – that is, just messaging people reminders to go out and vote increased the likelihood that the person would vote – but that the larger effect was transmitted indirectly via social contagion. Consider the setup of the experiment.
What makes a study interesting? Is it the empirical phenomena that we study or is it the theoretical contribution? For those of who are really paying attention (and I applaud you if you are), you’ll notice that I’ve asked this question before. It’s become a sort of obsession of mine. For the field of organizational theory, it’s an important discussion to have, although it’s not one that will likely yield any consensus. Scholars tend to have very strong opinions about this. Some people feel that as a field we’ve fetishized theory to the point of making our research inapplicable to the bigger world we live in. Others claim that by making “theoretical contribution” such a key component of any paper’s value, we ignore really important empirical problems. But in contrast, some scholars maintain that what makes our field lively and essential is that we are linked to one another (and across generations) via a stream of ideas that constitute theory. What makes an empirical problem worthy of study is that it can be boiled down to a crucial theoretical problem that makes it generalizable to a class of phenomena and puzzles.
At this year’s Academy of Management meetings, I was involved in a couple of panels where this issue came up. It was posed as a question, should we be interested in problems or theory? If we are interested in studying problems, we shouldn’t let theoretical trends bog us down. We should just study whatever real world problems are most compelling to us. If we’re interested primarily in theory, we need to let theory deductively guide us to those problems that help us solve a particular theoretical puzzle. Some very senior scholars in the field threw their weight behind the former view. I don’t want to name any names here, but one of the scholars who suggested we should be more interested in real-world problems is now the editor of a major journal of our field. He offered several examples of papers recently published in that journal that were primarily driven by interesting observations about empirical phenomena.
One of the new assistant professors in the crowd threw a pointed objection to the editor. And I paraphrase, “This all sounds great. I’d love to study empirical problems, but reviewers won’t let me! They keep asking me to identify the theoretical gap I’m addressing. They demand that I make a theoretical contribution.” Good point young scholar. Reviewers do that a lot. We’ve had it drilled into us from our grad school days that this is what makes a study interesting. If the paper lacks a theoretical contribution, reject it (no matter how interesting the empirical contribution may be)! This is a major obstacle, and I don’t think the esteemed editor could offer a strong counter-argument to the objection. Editors, after all, are somewhat constrained by the reviews they get. I think what we need is a new way to think about what makes a study valuable. We need new language to talk about research quality.
Finally got around to reading the Regnerus SSR article. A few comments:
1. Up front – my politics: I am against laws that distinguish between people based on sexual orientation. I also believe that people should be tolerant of many sexual orientations, not just heterosexuality.
2. My prior scientific belief: I believe that it may be possible that sexual orientation may be correlated with family outcomes in positive or negative ways, just as there might be differences between other groups (e.g., Latinos may be better or worse parents according to some measure).
3. My prior legal belief: The presence of group differences doesn’t entail policy change unless the differences are extreme. For example, we might discover that alcoholics are worse parents, but I would be against a law that banned alcoholics from having children. In other words, I can believe that some group (e.g., gays) may be better or worse parents than other groups, but that doesn’t mean we should discriminate.
4. What I’ve been told: I am not a family sociologist, but multiple people have told me that prior research tends to find little or no difference between children of straight parents and gay parents. They could be wrong and I’d be willing to update my belief if a sufficiently strong study came out.
5. The actual Regnerus study contains a modest, but interesting result. According to multiple measures, people seem to be worse off if they had a parent who had same gender sexual relations. This isn’t surprising given that most reported two person families were mixed gender. That suggests that same sex contacts were outside the family. In other words, the study measured the “Larry Craigs” of the world. I am not shocked that their children may be worse off in some way.
6. The issue, to me, seems to be the claim that the data provide evidence against same sex marriage. First, even the author admits, there are very few people who reported two parents of the same gender (17, to be exact). Second, there is a severe selection effect. Most of the survey respondents grew up when same sex marriage was illegal, thus preventing what might the equilibrium state in an environment where same gender marriage is legal. To be blunt, the gay people who set up families a decade or more ago are not the same people who might set up families in the current environment.
7. There was petition asking the Social Science Research editor to explain how this paper went through the review process. As an author whose papers have gotten stuck for *years* at a time, I was shocked to learn that it went through in a matter of weeks.
8. Critics claim that the outcry was a “witch hunt” (see the Scatter discussion). That’s a vague and charged term, so I will ignore it. But a few things are safe to say. Science is built on skepticism. If a paper comes out claiming that all previous work in the topic was flawed and produces a controversial result, it would be normal for people to ask questions. The proper response is to provide an explanation of how the research was conducted, not accuse people of a “witch hunt.” No one is asking that anyone be fired or banned from doing sociology. The petitioners are merely asking, “why was this published?”
9. There is some truth to the charge that the outcry is political. Consider a thought experiment, what if a researcher produced a flawed article that supported a liberal policy? Has there been an equivalent level of outcry against bad research that supports liberal points of view? This doesn’t mean that the Regnerus critics should stop. They were right to ask questions. Rather, it means that we should bring the same skepticism to all research, regardless of policy implications. Liberals and conservatives should be equally fearful of sociology’s methodology police.
10. Darren “BMX” Sherkat was asked the the SSR editor to do an audit of the paper and its review process. Personally, I think this is excessive. The editor, James Wright, is an accomplished scholar and likely knew that the paper would be controversial, even used as ammunition in a political dispute. We give editors great leeway. They may agree with reviewers or override them. He chose to publish this paper after getting some feedback, which is normal. Darren found that the reviewers had some connection with the author. This isn’t always bad. I’m sure that the reviewers of some of my rejected papers know me personally, and that I’ve rejected the papers of people who I like and admire. The bottom line is that the James Wright is an adult and sociology is a contact sport.
11. Bottom line: I think this a modest paper that presents an intuitive result. If one of your parents is gay but is with a different gender partner, then kids may be worse off. A family where there a sever mismatch in orientation between parents is likely to be stressful, to say the least. At best, the paper would have to be severely rewritten to match the text and the results. At worse, as Darren notes, the paper should just be rejected along with other papers where the claims don’t match the data. The extremely fast publication process suggests that these options were not seriously considered.
Last week, I argued that retractions are good for science. Thomas Basbøll correctly points out that retractions are hard. Nobody wants to retract. Good point, but my argument wasn’t about how easy it is to retract. Rather, it’s about the fact that science is exceptional in that it has a built in error correction mechanism.
In reviewing the debate, Andrew Gelman wrote:
One challenge, though, is that uncovering the problem and forcing the retraction is a near-thankless job…. OK, fine, but let’s talk incentives. If retractions are a good thing, and fraudsters and plagiarists are not generally going to retract on their own, then somebody’s going to have to do the hard work of discovering, exposing, and confronting scholarly misconduct. If these discoverers, exposers, and confronters are going to be attacked back by their targets (which would be natural enough) and they’re going to be attacked by the fraudsters’ friends and colleagues (also natural) and even have their work disparaged by outsiders who think they’re going too far, then, hey, they need some incentives in the other direction.
A few thoughts. First, fraud busting should be done by those who have some security – the tenured folks – or folks who don’t care so much (e.g., non-tenure track researchers in industry). Second, data with code should be made available on journal websites, with output files. Already, some journals are doing that. That reduces fraud. Third, we should revive the tradition of the research note. Our journals used to publish short notes. These can be used for replications, verifications, error reporting and so forth. Fourth, we should rely on journal models like PLoS. In other words, the editors will publish any competent piece of research and do so in a low cost and timely way. Fraud busting and error correction will never be easy, but we can make it easier and it’s not hard to do so.
Thanks to those who suggested additional examples of self-managing organizations on my previous post about self-managing organizations! In the comments, Usman has also kindly provided a link to a documentary, The Take. Such examples show how people use self-managing organizations to reverse economic decline or stagnation, as well as defend their community, dignity, and livelihoods. For more examples of how grassroots organizations and democratic organizations can underpin economic revitalization, Orgheads might be interested in Jeremy Brecher‘s Banded Together: Economic Democratization in the Brass Valley (2011, University of IL Press). Drawing on archival research, participant-observations of meetings, and interviews conducted about efforts to revitalize Western Connecticut’s Naugatuck Valley in residents and workers’ interests using Alinskyite methods, Brecher delves into several case studies of reorganizing the workplace, from factory to home-care. (See my review of Brecher’s book in Contemporary Sociology for a more detailed synopsis.)
Participatory practices are also spreading to local governance in the US. Last fall, with the help of local organization Community Voices Heard, the Participatory Budgeting Project, and scholars and other groups, and trained volunteers such as myself, four districts in NYC experimented with participatory budgeting. Those who live, work, or attend school in these districts could propose and then prioritize projects on how to allocate several million dollars of city funds to improve community life. Volunteer budget delegates then developed proposals selected at the neighborhood assemblies, which they presented to the public. Residents aged 18 years and older voted for their top choices. Elected officials then allocated funding to these choices; some allocated additional funds for proposals that hadn’t won the popular vote. For more info on this experiment, see a PBS segment, which includes an interview with Celina Su, one of the advisers to this experiment. (Su published Streetwise for Street Smarts: Grassroots Organizing and Education Reform in the Bronx, which compares Frierian and Alinskyite organizing tactics.) See also my op-ed about this experiment and its implications for otherwise underrepresented voices in a local paper.
Think these practices might work in your hometown or organizations? Add your comments and recommendations below.
Hello fellow orgtheory readers! Orgtheory was kind enough to invite me back for another stint of guest blogging. For those of you who missed my original posts, you can read my 2009 series of posts on analyzing “unusual” cases, gaining research access to organizations, research, the IRB and risk, conducting ethnographic research, ethnography – what is it good for?, and writing up ethnography.
Those of you are familiar with my research know that I have studied an organization that mixed democratic or collectivist practices with bureaucratic practices. Here’s a puzzle: although we operate in a democracy, most of our organizations, including voluntary associations, rely upon topdown bureaucracy. However, this doesn’t mean that alternative ways of organizing can’t thrive.
Valve, the game developer behind Portal, has attracted much buzz (for example, see this article in the WSJ and various tech blogs entries, such as here and here) about its self-managing processes. The company prides itself on having no bosses, and their employee handbook details their unusual workplace practices. For example, instead of waiting for orders from above, workers literally vote with their feet by moving their desks to join projects that they deem worthy of their time and effort. Similarly, anthropologist Thomas Malaby describes how Linden Lab workers, who developed the virtual reality Second Life, vote how to allocate efforts among projects proposed by workers. Sociologist David Stark has described how workers mixed socialist and capitalist practices in a factory in post-Communist Hungary to get work jobs done, dubbing these heterarchies.
Interestingly, several of the conditions specified by Joyce Rothschild and J. Allen Whitt as allowing collectivist organizations to survive may also apply to these workplace organizations – for example, recruiting those like existing members and staying small in size. However, my research on Burning Man suggests that these are not always necessary or desirable conditions, particularly if members value diversity.
Although these self-managing practices may seem revolutionary to contemporary workers, orgtheory readers might recall that prior to the rise of management and managerial theories such as Taylor’s scientific management, workers could self-determine the pacing of projects. Could we make a full circle?
Any thoughts? Know of other interesting organizations or have recommendations for research that we can learn from? Put them in the comments.
Several people have pointed out Neal Caren lists of most cited works. I appreciate how hard it is to do something like this and I appreciate the work Neal Caren has done. So my criticism is intended more to get us closer to the truth here and to caution against this list getting reified. I also have some suggestions for Neal Caren’s next foray here.
The idea, as I understand it, is to try and create a list of the 100 most cited sociology books and papers in the period 2005-2010. Leaving aside the fact that the SSCI under counts sociology cites by a wide margin, (maybe a factor of 400-500% if you believe what comes out of Google Scholar), the basic problem with the list is that it is not based on a good sample of all of the works in sociology. Because the journals were chosen on an ad hoc basis, one has no idea as to what the bias is in making that choice. The theory Neal Caren is working with, is that these journals are somehow a sample of all sociology journals and that their citation patterns reflect the discipline at large. The only way to make this kind of assertion is to randomly sample from all sociology journals.
The idea here is that if Bourdieu’s Distinctions is really the most cited work in sociology (an inference people are drawing from the list), then it should be equally likely to appear in all sociology articles and all sociology journals at a similar rate. The only way to know if this is true, is to sample all journals or all articles, not some subset chosen purposively. Adding ASQ to this, does not matter because it only adds one more arbitrary choice in a nonrandom sampling scheme. .
I note that the Social Science Citation Index follows 139 Sociology journals. A random sample of 20% would yield 28 journals and looking at those papers across a random sample of journals is going to get us a better idea at finding out which works are the most cited in sociology.
Is there any evidence that the nonrandom sample chosen by Neal Caren is nonrandom? The last three cites on his list include one by Latour (49 cites), Byrk (49 cites) and Blair Loy (49 cites). If one goes to the SSCI and looks up all of the cites to these works from 2005-2010, not just the ones that appear in these journals, one comes to a startling result: Latour has 1266 cites, Bryk, 124, and Blair Loy 152. At the top of the list, Bourdieu’s Distinctions has 218 on Neal Caren’s list but the SSCI shows Distinctions as having 865 cites overall. Latour’s book should put him at the top of the list, but the way the journals are chosen here puts him at the bottom. It ought to make anyone who looks at this list nervous, that Latour’s real citation count is 25 times larger than reported and it puts him ahead of Bourdieu’s Distinctions.
The list is also clearly nonrandom for what is left off. Brayden King mentioned that the list is light on organizational and economic sociology. So, I did some checking. Woody Powell’s 1990 paper called “Neither markets nor hierarchies” has 464 cites from 2005-2010 and his paper with three other colleagues that appeared in the AJS in 2005, “Networks dynamics and field evolution” has 267 cites. In my own work, my 1996 ASR paper “Markets as politics” has 363 cites and my 2001 book “The Architecture of Markets” has 454 from 2005-2010. If without much work, I can find four articles or books that have more cites than two of the three bottom cites on the list (i.e. Byrk’s 124 and Blair Loy’s 152 done the same way), there must be lots more missing.
This suggests that if we really want to understand what are the most cited and core works in sociology in any time period, we cannot use purposive samples of journals. What is required is a substantial number of journals being sampled, and then all of the cites to the papers or books tallied for those books and papers from the SSCI in order to see which works really are the most cited. I assume that many of the books and papers on the list will still be there, i.e. things like Bourdieu, Granovetter, DiMaggio and Powell, Meyer and Rowan, Swidler, and Sewell. But because of the nonrandom sampling, lots of things that appear to be missing are probably, well, missing.
Neal Caren has compiled a list of the 102 most cited works in sociology journals over the last five years. There are a lot of familiar faces at the top of the list. Bourdieu’s Distinction, Raudenbush’s and Bryk’s Hierarchical Linear Models, Putnam’s Bowling Alone, Wilson’s The Truly Disadvantaged, and Grannovetter’s “Strength of Weak Ties” make up the top 5. It’s notable that Grannovetter’s 1973 piece is the only article in the top 5. The rest are books. I was also interested to see that people are still citing Coleman. He has three works on the list, including his 1990 book at the number 6 spot. Sadly, Selznick is nowhere to be found on the list (but then neither is Stinchcombe). Much of the work is highly theoretical and abstract. There is a smaller, but still prominent, set of work dedicated to methods (e.g., Raudenbush and Bryk). I’m glad to see there is still a place for big theory.
It’s striking, however, how little organizational theory there is on the list. Not counting Granovetter, whose work is really about networks and the economy broadly, no organizational theory appears on the list until 15 and 16, where Hochschild’s The Managed Heart (which might be there due to the number of citations it gets from gender scholars) and Dimaggio’s and Powell’s 1983 paper show up. There are several highly influential papers in organizational theory that I was surprised were not on the list. One could deduce from the list that sociology and organizational theory have parted ways.
I don’t think this is really true, but I think it speaks to some trends in sociology. The first is that most organizational sociology, excluding research on work and occupations, no longer appears in generalist sociology journals outside of the American Sociological Review and the American Journal of Sociology. Journals like Social Forces or Social Problems just don’t publish a lot of organizational theory. Now, there are a lot of great organizational papers that get published in ASR and AJS, but that is a very small subset of the entire population of sociology articles. The second is that Administrative Science Quarterly no longer seems to count in most sociologists’ minds as a sociology journal anymore. Perhaps its omission leads to some significant pieces of organizational sociology being underrepresented (or perhaps not since ASQ publishes fewer articles than many of the sociology journals). To be fair to Neal, I don’t think he’s unique among sociologists as failing to recognize ASQ as an important source of sociology.* One reason for this, I’m guessing, is because a lot of non-sociologists publish in it. But a lot of non-sociologists publish in other journals that are on the list as well, including Social Psychological Quarterly, Mobilization, and Social Science Research. Another reason may just be that it’s because a lot of organizational sociology is no longer taking place in sociology departments, making the subfield invisible to our peer sociologists. Although I have no data to support this, my intuition is that fewer organizational theory classes are taught in sociology Phd programs today than were taught twenty years ago. Because of this, younger sociologists are not coming into contact with organizational theory, and so they are not citing it. Again, I have no evidence that this is the case.
I don’t think organizational research is waning in quality. A lot of organizational research still gets published in ASR and AJS. But a lot of it is probably not read or consumed by most sociologists.
UPDATE: Neal has updated the analysis to include ASQ. The major effect has been to boost DiMaggio and Powell to number 10.
*And yes, I’m lobbying Neal to include ASQ in future citation analyses.
Here’s an interesting piece extending Tajfel et al by studying 5-year-olds and intergroup bias: “Consequences of ‘‘Minimal’’ Group Afﬁliations in Children” Child Development. So, do 5-year-olds have a bias toward members of their in-group, even if they are arbitrarily assigned to these groups? They do.
Interesting paper. The paper also raises questions about whether in-group bias is learned (“enculturation,” Spielman, 2000), or whether it perhaps is an evolutionary-survival-type thing, or something driven by expectations of reciprocity or competition. Or something else.
Here’s the abstract:
Three experiments (total N = 140) tested the hypothesis that 5-year-old children’s membership in randomly assigned ‘‘minimal’’ groups would be sufﬁcient to induce intergroup bias. Children were randomly assigned to groups and engaged in tasks involving judgments of unfamiliar in-group or out-group children. Despite an absence of information regarding the relative status of groups or any competitive context, in-group preferences were observed on explicit and implicit measures of attitude and resource allocation (Experiment 1), behavioral attribution, and expectations of reciprocity, with preferences persisting when groups were not described via a noun label (Experiment 2). In addition, children systematically distorted incoming information by preferentially encoding positive information about in-group members (Experiment 3). Implications for the developmental origins of intergroup bias are discussed.