orgtheory.net

samples and social movement research

This post is both intellectual and self-interested. Throughout my career, I have chosen to focus on specific movements and treat them like a “laboratory” for social change. My work on Black Studies uses Black Power as starting point for studying institutionalization. My more current work on the anti-Iraq War movement examines how parties interact with movements.

One problem that persistently comes up during peer review is that people have problems with the selection of the movement. The complains often say (a) “Why study one movement?” or (b) “Your sample is too limited/has issues.” For example, I am currently having a paper reviewed that has a sample of nearly 700 activists drawn from the largest convention of social movement activists in recent years. The reaction from prior reviews? “Meh.” Here, I refer only to the sample, not criticism of the paper’s theoretical argument, which has been revised in subsequent versions.

The touchiness about the sample puzzles me because many of our most enduring works on social movements rely on samples of activists from a single organization or movement. For example, many of McAdam’s papers on Civil Rights movement participation often rely on extremely specific samples, like the paper with Paulsen that uses data from the Freedom Rides – a very specific phase of the Civil Rights movement. Snow et al. 1986’s article on framing uses a hand full of interview snippets from Christian activists to illustrate the function of framing within movements. Meyer and Whittier’s spill over paper uses illustrative cases from the women’s movement.

So, why the push back on the sample? A few hypotheses:

  • That’s just sociology. We have inconsistent standards.
  • The field has shifted. We no longer tolerate samples from one organization or movement. The norm is just different.
  • Outsiderness. Perhaps, papers that analyze a sample of activists is just not what is expected from a movement paper, so people are hyper-critical.

Aside from being personally frustrated on this issue, I find it puzzling. If one really wants to understand activists and movements participation, then it would be normal to generate data from a sample drawn from, say, protests, or an organizational roster, or participants in a movement convention or conference. As long as your question is about internal comparisons, and not comparing participators with non-participators, then this seems obvious – at least to me.

Adverts: From Black Power/Grad Skool Rulz

Written by fabiorojas

July 31, 2013 at 12:00 am

16 Responses

Subscribe to comments with RSS.

  1. There is an international trend towards ever-expanding samples, also in qualitative research. For me in Norway, where influential organisations can have about ten key stakeholders, publishing internationally with such small samples can be difficult. What to do about it, though…

    Like

    henrikkarlstrom

    July 31, 2013 at 7:39 am

  2. How 700 activists comprise a “small sample” is a surprise to me. Just because they come from a single case of a movement…? I just don’t get it. The way I understand and teach qualitative methods would say that (given the info you give here) you’re doing it right. Sorry I can’t be of much help ’cause I’m flummoxed too.

    Like

    Vilna Bashi Treitler

    July 31, 2013 at 9:27 am

  3. I have had similar experiences. I have a survey sample with 100<N<150 and a response rate just under 30 percent. The population it is drawn from, is notoriously difficult to get to respond (very busy professionals that are very reluctant to reveal the type of information I’m interested in). The funny thing is, it is probably one of the largest most representative samples out there in the particular field, and it addresses a research question for which this type of data would be a great step forward.
    But for editors and peer reviewers, small N studies are just an easy target. Which is understandable, given the number of submissions that good journals and popular reviewers have to decide on. It just makes it easier to quickly put something on the ‘no’ pile without spending too much time on it.
    Unfortunately, that, no matter how well thought out, executed, and how potentially valuable to progress in the field, it is generally much less risky to just fit your research question to be answerable with well establish secondary data then to go out and collect the date yourself. Unless you’re a big name, which does seem to help with getting small N studies published.

    Like

    Alex

    July 31, 2013 at 9:57 am

  4. It certainly strikes me as a lazy way to critique a paper. Plenty of qualitative studies have very small samples. Sometimes only 20 people — as long as there is something interesting to say about them! However, I agree with the first poster that in some quarters of the discipline does seem to be a trend toward very large samples and more macro perspectives in qualitative research. This has been evident to me from reading AJS/ASR in the last few years. I see less of it in journals like Gender & Society, Qualitative Sociology, and others that publish more “traditional” qualitative work.

    Like

    soctraveller

    July 31, 2013 at 1:43 pm

  5. I think this is a particular issue in social movement research where the single case-study approach has been overdone and where it’s often the case that authors don’t attempt to link to other contexts in a comparative manner. The case analysis approach established by McAdam, Snow, and others created a great model for doing research that was imitated by junior scholars and dissertation writers. The problem with that approach is that every student/junior scholar simply imitated the model without thinking about how their case was different from or similar to what came before it. It turned social movement research into the study of “my favorite movement.” It still seems to be a major problem in the section. If you ask a random person at a CBSM section event what they study, they will likely tell you what movement they’re interested in – e.g., I study the abortion rights movement in Guatemala (apologies to anyone if this happens to be your dissertation – rather than give you a conceptual description of what they’re doing.

    With the sudden availability of new sorts of protest/online data, CBSM seems to have polarized among those who prefer multi-movement, large N studies versus those who are still focused on particular movements. I think both sides have the potential to make good contributions. The multi-movement folks (and I’ve been known to be among this group) say that if we want to make generalizable statements we need to examine trends that exist across movements, whereas the latter group values contextualization above everything else. I think there is definitely room for both types of research, and i agree with soctraveller that the critique made about your sampling seems like a lazy criticism to me. The important thing is that we can communicate how/why our data address a theoretical problem in the area.

    Like

    brayden king

    July 31, 2013 at 2:33 pm

  6. I agree with Vilna. N=700 is only small from the perspective of someone doing research with massive datasets. I would consider than N sufficiently large unless you’re studying, say, four-way interactions. Some as yet unpublished research by Sanjay Srivastava shows that an N of 200 is adequate for most social psychological research in terms of statistical power.

    On the other hand, it’s important to have a comparison (or control) group, because without such a group you cannot find out if your findings are distinctive or not. It may turn out that the members of the group have some of the same attributes as people outside the group, so one needs to control for spurious distinctiveness.

    Like

    Chris M

    July 31, 2013 at 4:07 pm

  7. I think the issue is less about the size of N and more to do with what sorts of observations are in the data set. The reviewers are likely hung up that all of the observations come from the same social movement and don’t span multiple contexts.

    Like

    brayden king

    July 31, 2013 at 4:10 pm

  8. 700? Ha. Your project sounds huge to me. All my qual stuff has about an N of twenty.

    I’m somewhat new to the social movement stuff (working on my ASA pres now), and I guess my biggest concern about samples is that most tend to focus on movement wins than movement failures. That is, a group accomplishes something and then we retroactively apply a narrative to show how it was inevitable. To be fair, this is not true of all socmvmt stuff, but it is my early impression. If others think I’m wrong on this front, I’m happy to reconsider.

    Like

    KenKolb

    July 31, 2013 at 5:09 pm

  9. I think professor King is right. Additionally I think it’s contradictory to think about “context” as “very particular and local situations represented in all theory shining glory.” To claim that context is important is precisely to claim that one must look outside particular actors and behaviors, casting a wider umbrella, and looking at the interdependencies among the units of analysis.

    There’s no reason that we can’t cast a wide contextual net and derive generalities across several populations and events. And these results can even be thick in description. Any modeling, describing, measuring, or metaphoring, of any situation, with any method, anywhere, necessarily implements some degree of parsimony. Knowing the general mechanics of movements across cultural and organizational boundaries would be thick in relevance, if not in its description of what its leaders ate for lunch.

    Like

    Graham Peterson

    July 31, 2013 at 5:11 pm

  10. Thanks for the comments. The irony is that we sampled at an activist convention to be able to compare activists from different movements…

    Like

    fabiorojas

    July 31, 2013 at 7:45 pm

  11. i think your reviewer might be cribbing not about 700 “number” per se, but perhaps raising an issue that the variables between which you’re computing correlations from the survey are also ones that determine selection into the sample, viz convention attendees… that said, I don’t really know what research question you’re asking, but it would strange if you reviewer is merely saying “700 low, 700 bad, 700 no good”

    Like

    sd

    August 1, 2013 at 3:00 am

  12. As a young scholar that has a similar type of sample – a national sample of several hundred individuals in a single emerging occupation influenced by a social movement I have been thinking about this problem. It seems to me that there appears to be an “uncanny valley” for “data quality” in top journals. That is, I was taught that perfect data hardly ever exist and as long as you improve significantly on what came before and can carefully articulate the strengths and limitations of your data then you have data that has some value. However, I don’t think this is as true as I believed. Using past published studies (especially many of the iconic, foundational studies, in an area) as a standard is misleading as more reviewers seem to be anchoring their expectations on “the perfect data” and asking how close you are. Even if your data is *signficantly* better than previously *published* studies you get no credit unless it deviates only marginally from the hypothetical perfect data. In doing so I think there are significant number of papers that are rejected and only found in in the “drawer” based on rapid change in data standards, which acts to leave a discontinuity the data quality in published studies which can naively be seen by young scholars as an “opportunity” because we continue to judge/model based on the studies we see, not those that are rejected. Sometimes it seems like the closer you approach gold standard data the harsher its deviance from perfection is treated. Hence the “uncanny valley”. Its like good but not great data creeps many reviewers out no matter how carefully and articulately you recognize its limitations.

    Like

    Ryan

    August 5, 2013 at 6:55 am

  13. Ryan, I think Brayden King adressed your point at July 31, 2013 at 2:33 pm

    Like

    Anonymous

    August 5, 2013 at 9:35 am

  14. Brayden’s example is definitely within the set, specific to the SM literature. I think that it is part of a broader class of issues that have a similar dynamic. I was trying to generalize it a bit :)

    Like

    Ryan

    August 5, 2013 at 4:56 pm

  15. This issue of samples and sampling technique, quality, validity and reliability has also come up in occupy work…and my sense is that some of the claims that are made in some studies about “randomness” and representation are problematic…if you are trying to study a march then a (hopefully really) random sample of that march may be appropriate but if you are trying to study a (national or even global) movement, one sample at one rally (as random as it may be) in one city (of many) on a given day does not give the researcher a random sample of a “movement”…but, of course, what do reporters really know about any of this…http://www.nytimes.com/2012/05/01/books/academia-becomes-occupied-with-occupy-movement.html?pagewanted=all&_r=0

    Like

    Hector

    August 11, 2013 at 7:11 am

  16. Studies on activism among sexual minorities routinely runs into sampling concerns by reviewers. Reviewers often dismiss samples of around 400. Finding gays and lesbians in the United State is not easy, getting any measures on sexual identities and political behavior is even harder (especially since activism is often related to the disclosure practices of sexual minorities). From my perspective, some reviewers prefer no articles on a topic until the perfect samples to come along. It is a little alarming that Fabio goes to antiwar protests to gather data and still gets slammed.

    Like

    Eric

    August 14, 2013 at 1:01 am


Comments are closed.