biernacki book forum – part 2: the joys of replication

Part 1, Scatterplot review by Andrew Perrin.

To summarize: Richard Biernacki claims that coding textual materials (books, speech, etc) is tantamount to committing gross logical errors that mislead social scientists. Overall, I think this point is wrong but I think that Reinventing Evidence does a great service to qualitative research by showing how coding of texts might be critiqued and evaluated. In other words, ironically, by critiquing prior work on text coding, Biernacki draws our attention to the fact that qualitative research can be subjected to the same standards as quantitative research.

What do I mean? Well, a big problem with qualitative research is that it is very hard to verify and replicate. It is rare when ethographers go to the same field site, or informants are re-interviewed by others. A lot of the strength of quantitative research lies in the fact that other researchers can replicate prior results. For example, if I claim that party ID is correlated with gay marriage attitudes in the GSS, another researcher can download the same data and check the work. If they think the GSS made a  mistake in collecting the data, a second survey can be conducted.

Biernacki, in trying to prove that coding qualitative data is pointless, follows a similar strategy by choosing a few articles of note and then he tries to reproduce the results. For example, he chooses Bearman and Stovel’s “Becoming a Nazi: A Model for Narrative Networks” which appeared in Poetics. The article creates a network out of ideas and themes mentioned from the memoir of a Nazi. Assuming that Biernacki reports his results correctly, he’s persuaded me that we need better standards for coding text. For example, he finds that Bearman and Stovel use an abbreviated version of the memoir – not the whole thing. Big problem. Another issue is how the network of text is interpreted. In traditional social network analysis, centrality is often thought to be a good measure of importance. Biernacki makes the reasonable argument that this assumption is flawed for texts. Very important ideas can become “background,” which means they are coded in a way that results in a low centrality score. This leads to substantive problems. For example, the Nazi mentions anti-semitism briefly, but in important ways. Qualitatively we know it is important, but the coding misses this issue.

Next week, I’ll get to my views on Biernacki’s attack on coding. But for now, I’ll give him credit for drawing my attention to these issues. The problems with the coding of the Nazi memoir point to me that there is more work to be done. We need to first start with a theory of text and then build techniques. If you want to use network analysis, you may have to take into consideration that standard network ideas may not be suitable. That will help us address problems like how to judge a text and the way we code data. That may not be the lesson Biernacki intended, but it’s a good one.

Adverts: From Black Power/Grad Skool Rulz

Written by fabiorojas

April 15, 2013 at 4:38 am

9 Responses

Subscribe to comments with RSS.

  1. Perrin’s post in full at Scatterplot is *well* worth the time to read. For some more perspectives, the digital humanities people are dealing smartly with the concerns about interpretation, pragmatic context, etc. Particularly good work coming out of the Stanford Literary Lab.


    Graham Peterson

    April 15, 2013 at 6:06 am

  2. There is coding and there is coding, and I would like to hear what you think about the applicability of the critique to organization theory (the topic of this blog, anyhow)?
    I always thought the point of Gioia-like coding schemes is to organize the evidence and help provide support for the interpretation of the author who thoroughly knows the case. It matters little that coded material is cherry-picked and open to alternative codings if the goal is to provide support to a set of claims inferred ex-ante from the data. I don’t think anyone who actually does qualitative research in AMJ style believes that results emerge from the _coding_ (even if they do emerge from reading the data).
    What I mean is that it is not unlike like regression models, which are not designed to reveal the truth about the data (the world does not pick the dependent variable) but to justify/support theoretical claims.



    April 15, 2013 at 8:15 am

  3. I suspect that many org theorists today use coding as a substitute for “thoroughly knowing the case”. In the Nazi memoir example, Biernacki’s argument is that it used as a substitute for reading (in any recognizable sense) the source.

    I’m working on a critique of a couple cases myself these days. It’s frustrating to engage with an analysis that is based on coding and ignores (and even contradicts) basic, obvious conclusions that would emerge from a “mere” reading of the sources. The coding ritual here feels like a way of preemptively dismissing all corrective readings of sources as “unscientific”. There is an important difference between reading someone’s source and “replicating” their analysis. It’s the rather successful marginalization of that first simple activity that I lament.



    April 15, 2013 at 9:32 am

  4. I haven’t read the book or Poetics article, so perhaps this issue has already been addressed… However, as an ethnographer, I am not at all surprised to see a critique of coding of textual or interview data. In fact, the very strength of ethnography is that it gives the researcher a chance to examine people’s statements in both their social context and the context of their actions. I find that I have a much deeper, more intuitive understanding of my interview data when I have had time to see how my informants interact with others, find out about their home and family life, hear what they say at public events, etc.



    April 15, 2013 at 2:03 pm

  5. The Bearman and Stovel article is based not on one memoir but on 600 life stories submitted for a competition for “the best life story of an adherent of the Nazi movement. The competition
    was organized by Theodore Abel (a sociologist at Columbia University), but sponsored
    by the NSDAP and SA. A cash award was to be given to the best essayist.” (from pg. 72 in the B&S article) A footnote included in this little bit of text makes note that the life histories had been previously analyzed by other scholars (Abel, 1938; Merkl, 1977; Rhodes, 1980) including, as you see, the organizer of the contest.


    Jenn Lena

    April 15, 2013 at 3:08 pm

  6. The Bearman and Stovel article is based on one memoir drawn from a collection of 600 life stories. In the abstract, “We illustrate our approach to narrative networks by analyzing a single story that, in conjunction with the analysis of other stories, yields a ste of insights into becoming and being.” The “in conjuction with the analysis of other stories” is a suggestion of the paper rather than an integral part of its demonstration of narrative analysis. All of the figures represent network analysis of the single story and all of the discussion of linking network structure to content refers to this same story.



    April 15, 2013 at 4:27 pm

  7. I guess I’m the person who’s wrong on the internet today.


    Jenn Lena

    April 15, 2013 at 5:43 pm

  8. […] is Fabio Rojas. His is the more moderate of the two. At Orgtheory, he […]


  9. I was sitting at home tonight watching the NBA playoffs and performing my whiteness when I came across a blog post ( ) from Peter Klein about this paper ( . Like Peter, I was not taken by the paper. Then I wondered if the nice people at orgtheory were actually speaking about this kind of research in Biernacki book forum. So I would pose these questions. Could this research be replicated — in the sense that Biernacki uses? If one does not have the same particular critical perspective of the authors, is there any likelihood that the coding could be replicated? Is it typical of this kind of research to not have data triangulation, particularly given the claims about ethnicity of vendors, laborers, and customers.

    BTW, should it be a surprise that the Davis Farmers Market looks white, given that Davis is the whitest city in the Central Valley? Would the farmers markets in Modesto or Stockton be bastions of white farm imaginary?



    April 21, 2013 at 3:58 am

Comments are closed.

%d bloggers like this: