take the pill
Today, I’ll directly address Sam Lucas’ article, “Beyond the Existence Proof: Ontological conditions, epistemological implications, and in-depth interview research,” published in 2012 in Quantity and Quality. In it, he argues that there is no basis at all for generalizing conclusions from the types of unrepresentative samples that are used by interview researchers. The best you can do is use the sample to document some fact (“an existence proof”), not make any out of sample generalizations. You can read Andrew Perrin’s commentary here.
To illustrate his argument, let’s return to yesterday’s hypothetical about unrepresentative samples. I said: “What if Professor Lucas suddenly found out that his heart medication was tested with an unrepresentative sample of white people from Utah? Should he continue taking the medication?” I’ll outline two answers to this question.
1. According to Professor Lucas, the reason that you can’t generalize from unrepresentative samples is that the world is “lumpy.” What does this mean? It means this:
All analysts confront a social world that is lumpy. By “lumpy” I mean that in the large-dimensioned social space there are concentrations of entities, and sparse locales; some constellations of characteristics are common, others rare; hills and mountains rise from some spots on the social terrain, valleys and ravines mark others.
In other words, individual cases are not linearly, or normally distributed, in the vector space that describes the variables we care about. The density map of the social world looks like this:
Thus, a sample that comes from one “island” (or near a critical point in the surface, to use our sophomore calculus terminology) would yield data that would have a large error term when extrapolated. Professor Lucas would tell his doctor: “Thanks for the medication, but African Americans might be in a cluster that is very far away from the sample of White men in Utah. Since this is expensive and I might have serious side effects, I’ll discontinue the treatment. “
2. My response is that Professor Lucas has an unstated assumption, which may or may not be supported. The assumption is that all degrees of lumpiness are equally likely. In other words, the world I showed you is just as likely as this one:
In other words, Professor Lucas correctly points out that it is possible that you live in a lumpy world (an obviously correct theoretical point), but then tacitly assumes that there a good chance that you actually live in a lumpy world. The second point is empirical and can only be determined via study. If it’s true, then you are, in general, allowed to dismiss unrepresentative samples. If not, then you can start thinking about how good or bad the sample is.
So, here’s what I would say, which is similar to some of the commenters: “I recognize that the study testing the efficacy of Berkleyflaxin is flawed in an important way. But I also recognize that it has some information. What I’ll do is then look for evidence that that I live in ‘lumpy’ world where there is huge variation the correlation between ethnicity and response to other similar medications. If I find it, I’ll discontinue. If I find evidence that I live in a less lumpy ‘linear’ world, then I’ll take my chances. If I find no evidence, I’ll probably continue because there aren’t *that* many drugs where there is huge variation in effectiveness across ethnic groups.”
Now, I want to make a few things clear. In many ways, I strongly agree with the gist of Lucas’ argument. It is often the case that in-depth interview researchers pretend to limit their claims and then jump to broad conclusions. I can easily imagine the Berkeley cultural sociology student who wants to make claims about schools based on their in depth interviews of twelve immigrant kids in the Mission district. But still, that doesn’t mean that unrepresentative samples have no value for inferential social science. Instead, we have to start figuring the different types of processes that produce data and build systematic theories of when the data is useful or not.