standardizing ethnography
On the Soc Job Rumor Board, there was a discussion of the non-replicability of ethnography. I think this is mistaken. Ethnography is easily replicable, it’s just that ethnographers don’t want to do it. For example, ethnographers could:
- Stop making everything anonymous so others can verify and check. Mitch Duinier is right about this.
- Group ethnography. Have multiple observers and do inter-coder reliability.
- Standardize data collection – how field codes are done and recorded.
- Encourage others to revisit the same population (which is actually done in anthropological ethnography)
Of course, no single study can strive for replication in the same way and some folks do a good job addressing these issues. But still, the anti-positivist framing of much ethnography probably prevents ethnographers from developing intuitive and sensible things to create standards that would move the field away from the solo practitioner model of unique and non-replicable studies.
50+ chapters of grad skool advice goodness: From Black Power/Grad Skool Rulz
orgtheory.net 
I’m not sure it’s an anti-positivist framing that is responsible for the lack of replication in ethnographic studies (that’s not to say that such a strain does not exist). I think there is an issue of replication in sociology as a discipline more generally. What is the incentive to reproduce any sociological study (except for revisiting historical analyses/conclusions)? In psychology, folks can actually publish validation studies (and build careers on such publications). Where’s the outlet in sociology for anything that can’t be presented as “original” research? So I would argue that it’s not just the ethnographers that don’t want to do it. No one wants to do it.
LikeLike
Paul-Brian
April 10, 2014 at 1:14 am
I am totally in agreement with Fabio on points 1 and 4–the fake places and names is out of control, often done without even token justification, and offers no way for folks to even compare what the ethnographer reports with census data, newspaper articles, or their own knowledge of a place [let alone replicate the study]. Some of the great ethnographies are revisits [Burawoy, of course; but Duneier also posits a kind of revisit of Jacobs’ Greenwich Village in Sidewalk] but how many more might be done if places were disclosed? When an ethnographer decides in particular to anonymize a whole neighborhood they have made a choice that will affect any future researcher who may want to see, say 20-30 years later, how the neighborhood has changed. Paul-brian is right that few if anyone would try to replicate an ethnography right afterwards, although I think it is important that we set up studies so that they can at least try, but having snapshots of the lived experience of a place over long time scales, with future ethnographers going back to a former fieldsite of someone else and doing the same study to see how things have changed, is very useful.
LikeLike
Colin
April 10, 2014 at 2:06 am
The last three are doable and desirable, but the first may be very very very very tricky.
LikeLike
Luke Zheng
April 10, 2014 at 2:15 am
I recommend reading or rereading Harold Garfinkel’s ‘Ethnomethodology’ in order to come to grips with some of the contingencies in constructing a description of an event or episode. Levi-Strauss said upon reflection of his methods which were largely a matter of receiving information from a local informant, “Why did he tell me that?” which throws a kink into the reasoning process related to the ethnographer’s determination of why an actor decided to do what s/he did.
LikeLike
Fredrick Welfare
April 10, 2014 at 3:27 am
Reblogged this on Installing (Social) Order.
LikeLike
Nicholas
April 10, 2014 at 3:56 am
It seems to me that the disciplinary disincentives are far and away the biggest bar to replication. But I suspect that a far more important problem is the reluctance of ethnographers to test the findings of recent ethnographies, by adopting the same methodological strategies to study “similar” populations in “similar” settings (I’m no ethnographer so please accept my apologies if I am mistaken). The emphasis on novelty and addressing empirical gaps prevents us from refining (or discarding) many existing theories that could probably benefit from more rigorous reevaluation.
LikeLike
Kevin .
April 10, 2014 at 6:09 am
Paul-Brian: Sociological Science is explicitly open to replication studies. (Other journals may be as well, this is just the journal with which I am most familiar.)
It’s a bit of a chicken-and-egg issue, though. SocSci hasn’t received a submission that reports on a replication study, out of 75-ish papers. This isn’t a huge n, but if our early experience generalizes to other journals, it’s not a surprise that replication studies are scarce or even absent from sociology journals. And, if prospective authors interpret the lack of published replication studies as evidence that such studies can’t be published, authors won’t bother to conduct replications or write them up.
LikeLike
Kim Weeden
April 10, 2014 at 11:18 am
There are two main obstacles to replication in corporate ethnography specifically:
(1) Not anonymizing the data is most often not an option. Most corporate executives will not grant ethnographic access at all, but those that do almost always require that they and their company not be identified. Also, employees must be anonymized in order to collect accurate reports of their experiences otherwise their jobs can be put at risk. To require ethnographers to stop anonymizing would ground most corporate ethnographic work to a halt.
(2) It’s unlikely that any given corporation would allow a second ethnographer in shortly after the first was there (and if too much time has lagged then the “replication” is not a “replication”). It is seen by most executives and employees as a major commitment to allow someone the sort of deep access that makes for good corporate ethnography. Also, in my experience, executives and employees do read the work that is written about them so to the extent that its implications are not as favorable as they may like, that further dampens their interest in opening their doors again.
(Sometimes, of course, companies do let multiple ethnographers in and are not as concerned with anonymity so that we know, many years later, that Kanter, Kunda and several others all studied DEC in the 1980s)
LikeLike
anonymous
April 10, 2014 at 3:00 pm
Agree with above poster. I’ve studied economic development planners and the sometimes shady financial tricks they engage in to market municipal bonds and otherwise leverage outside capital. There is NO WAY I’d ever consider making them not anonymous; there is a big difference between somebody theoretically being able to find out who you are and having your name come up when googled in relation to a piece of academic work. People have been doing ethnography for a long time and most of the conventions are there for a good reason. It is a problem when we take standards that make sense in one field site and make big claims about what all ethnographers should do. Even Duinier justifies naming his informants partially on the basis that he focus on a population that has had much written about it rather than speaking for itself. Presumably, that is not the case for everyone who ethnographers study.
LikeLike
Josh P
April 10, 2014 at 3:32 pm
The most important findings in ethnography transcend the single word, sentence, or event of speech. The important thing is that we get right the recurring, stable, and persistent forms of interaction that comprise social structures. Their persistence is what makes them worth studying in the first instance as a feature of social life. As Duneier and others have noted, individuals are inconsistent in their behavior, and forgetful of their utterances. Under the very best circumstances having the names may allow people to say whether they were in a particular place at a point, but it will rarely allow you to confirm why they said what they did, or even what they said. Given the importance of circumstances, one could go so far as to say evidence in quotes and individual passages represents hearsay. How many of us remember the verbatim text from what we said on a conference panel, much less the emotions and other compulsions leading us to speak? Ultimately, specific individuals in our work should be substitutable. One could say that it is more important to get at people in the same position as respondents—given that people may move from their location in a social structure by the time a book is published, then what we think we get from names is in fact not the most important. For that reason, confirming our results doesn’t require accessing individuals, but, rather, documenting situations. The key, then, is that one needs to show challenges to the definition of the situation, and their repair. Individuals can’t tell us what we need for replication. Rather, we need the specific features of the social situation.
LikeLike
mench
April 10, 2014 at 4:13 pm
I am an ethnographer who studies alternative-to-incarceration and drug treatment programs. Because this research deals with people who are considered prisoners and with information that is potentially stigmatizing, it would be unethical not reveal the names of the individuals or the institutions. Ethnographies of institutions are different than the classic ethnographies of groups or neighborhoods. Access is often dependent on confidentiality, in addition it is typically risky for participants (whether they are workers or clients) to be identified to the authorities in charge. I aim provide enough information about the types of institutions that others can study similar places, but this could never be a perfect replication. I debate whether to conceal the state/city of the places I study. Context and history is important to the analysis, but anyone with knowledge of the treatment field would probably be able to identify some of my sites and from that, possibly identify individuals. There is no easy way out of this dilemma.
IRBs are another reason why pseudonyms are ubiquitous. My IRB would never, ever have approved the research without this confidentiality agreement. This is something that critics of ethnography should always keep in mind.
LikeLike
Allison
April 10, 2014 at 8:03 pm
small correction to the above post: … it would be unethical TO reveal the names of the individuals or institutions.
LikeLike
Allison
April 10, 2014 at 8:05 pm
I want to register my concern with suggesting–even obliquely–that any of us read the Soc Job Rumor Board.
LikeLiked by 1 person
Jenn Lena
April 10, 2014 at 10:34 pm
I am an ethnographer and I agree with many of the points made by Allison. In my own work, I use pseudonyms for most people, but a few of my informants are fairly well known public figures in their country, so I use their real names (and that way I can also sometimes use their quotes from media sources). I also use real names for the organizations I study. In fact, I didn’t plan to do so, but they requested it. I use the real locations as well, because the specificity and context of real places is important to me. However, for many ethnographers there are very important confidentiality and ethical issues related to revealing identities, so I don’t think it would be appropriate to expect all ethnographers to do what I have done. I also want to underline what others have said about the disciplinary issues involved here. While some journals might be open to replication studies, I doubt most book publishers are (unless, perhaps it involves replication is of a study that is not at all recent), and most ethnographers aim to publish books. I also can’t imagine how having done a replication study would work out for someone on the job market.
LikeLike
soctraveller
April 10, 2014 at 10:40 pm
Like Allison, I perceive the push to anonymize data as coming primarily from IRBs. Hopefully if OHRP ever gets around to real reforms on human subjects research, we can let participants decide for themselves if they want to be identified or not.
LikeLike
Ken
April 11, 2014 at 5:48 pm
Of course, for vulnerable populations, etc., real names/places may not be possible/ethical. But I think IRB is sometimes used as a cop-out, meaning that because the IRB form asks you “how will you protect the confidentiality of your participants” people take that to mean they have to do so; but in fact they never tried to simply say in the IRB form that they will use real names with permission and then list how to protect those who don’t want real names used. I have sometimes asked people who presented innocuous findings using fake places [and of course names] why they did so, and it is a knee-jerk reaction to say IRB would not let them do otherwise. I don’t think it is true. I have gotten through real places/names on 3 projects at 3 different IRBs and have my grad students do the same in instances where it is appropriate.
LikeLike
Colin
April 14, 2014 at 3:36 am
There may well be knee jerk reactions to IRBs. In my experience, though, when I’ve tried to suggest alternatives to conventional IRB approaches it’s simply meant more paperwork, longer delays in approval, and ultimately the requirement to use the conventional approaches or some version thereof.
Also, that misses the bigger point of many of the above comments – even if some people are over-reacting to IRBs in certain instances, that doesn’t mean one could get access to other sites without promising anonymity to the organization and its members.
LikeLike
anonymous
April 14, 2014 at 11:54 am
Colin, I’m glad to hear you’ve found IRBs to be reasonable on this. I find they vary in how rigidly they interpret their job. What traits must a study have to make IRBs ok with using real names/places? If some individuals permit use of their real name and others do not, wouldn’t naming some people typically enable identification of the unnamed people? In these cases, it would be all or nothing…
LikeLike
Allison
April 14, 2014 at 2:20 pm
I’ve always wondered if a grad student reverse-engineered every aspect of Sidewalk into an IRB application how it would fare today. “As a way to ensure homeless participants fully consent to being identified in my study, I will offer them the prospect of malt beverages contingent upon their signing of the release form.” Colin says it is possible. I’m less optimistic.
LikeLike
kenkolb
April 14, 2014 at 2:58 pm
Assuming that an ethnographer is able to navigate IRB requirements in such a way as to make his or her work revisitable, what is it that others would replicate? If the people about whom an ethnographer has written at time x do not say the same thing or do the same thing at time y, does that mean that the work has been shown to be inaccurate? It seems to me that what we would want to replicate are the relations, rules, and mechanisms operative in situations and occasions rather than specific behavior or beliefs of identified people. So maybe there is a distinction that we need to make between verification and replication. Goffman wrote, “Not men and their moments, but moments and their men,” and that’s a pretty good for one kind of ethnographic work. But even Burawoy didn’t write about the same factory workers who had been written about before but about different workers in the same factory 30 years later. Maybe replicability is just really different in ethnography.
LikeLike
art
April 14, 2014 at 7:48 pm
Hm. I dealt with high-prestige and media-savvy subjects for the art market research, and gave them the choice of whether to be named or unnamed. Chicago’s IRB was OK with that, due to the main risk being reputational harm. I should note that even with this media-savvy population, where being cited in a research thesis could be seen as positive for credibility, even there, only a few actually wanted to be named. Apart from artists, who are trying to be public figures, not only were they generally the most media-savvy and advanced in their careers, they also tended to be a bit careful in their answers. (I gave named interviewees the ability to review transcripts and chapter drafts, for the reasons below. With less savvy people, I think it might be better to not let them make bad choices, e.g. if studying marginalized populations.)
Aside from the issue of not getting that much useful data with named interviewees, I always worry about whether people have carefully thought through things when my findings are effectively riding on their backs. It’s not like we’re back in Malinowski’s time where you could go over and do a study in the Pacific Islands and come back and nobody would have even the remote possibility of crossing paths with the people from the ethnography, unless they were anthropologists.
Put yourself in their shoes: imagine a reporter called you just now and was on deadline, so they’d love to interview you, but won’t have time to let you see the article before it goes to press in both online and print. How many of you would allow this hypothetical reporter to use your name? Let’s say you want to be anonymous and they print your name anyway. Do you feel exploited?
My point is that everything is so easily searchable these days. Someone who is senior and not trying to deep six their career is not going to want to go on the record over something controversial. It can also be intimidating for your everyday informant to want to put their name out there. This leads me to suspect that anonymity is a way to ensure better data. Australia actually has a policy that working on indigenous projects means that the tribal elders get to see the results before they are released. I think it is a good policy, even if it might slow things down.
As for the other points:
#2, good luck with that. Ethnography is already expensive. If someone would like to pay for additional RA’s, I’ll totally take them. My RAs are costing about $50/hr here once I pay for fringe and on-costs.
#3, I can’t see how that would work. There are some codebooks in cultural anthropology, if that’s what you mean. But they tend to speak to standard structures/roles/functions in village life. Not necessarily new processes or innovations. Bernard’s research methods in cultural anthropology book has some examples.
#4, part of my dissertation replicated Olav Velthuis’ Talking Prices, which was a good check on making sure I was doing a good enough job. But one has to go well beyond a replication in order to make a publishable contribution. Maybe we should have a journal for replication studies?
LikeLike
Erica
April 15, 2014 at 8:52 am
“Maybe we should have a journal for replication studies?” We should certainly have a something for replication studies. It is a serious problem that we don’t.
“Put yourself in their shoes: imagine a reporter called you just now and was on deadline, so they’d love to interview you, but won’t have time to let you see the article before it goes to press in both online and print. How many of you would allow this hypothetical reporter to use your name?”
This happens all the time. I’ve done it. It’s how talking to reporters actually works in the real world. You don’t get to see the story before it is printed. The reporter does not need my consent to quote me correctly or incorrectly, there is no IRB for journalism and experts have no expectation of privacy when speaking to the media. I’m an expert and expect to get quoted and expect that the quotations will be misleading a good fraction of the time. Yes I know reporters are supposed to try to get it right, and the good ones do, but they are taking notes, not recording, and they often get it wrong, or quote you out of context. The same rules of the game apply if you are an ordinary person and a reporter asks you a question at the scene of an accident where your child has just been killed (although I understand there is an ethical debate among reporters about whether it is right to ask question under those circumstances). The point is: not a good example for your point about ethnography.
LikeLiked by 1 person
olderwoman
April 16, 2014 at 2:53 am
[…] as an organizational ethnographer seems pertinent to the on-going orgtheory discussion of ethnography. Kunda not only describes how he became drawn to organizational studies (hint: questioning a […]
LikeLiked by 1 person
high risk, high reward? kunda’s reflections on ethnography and organizational studies | orgtheory.net
April 16, 2014 at 9:14 pm
Why is replication an issue? Is it because we want to be like the natural sciences? But sociologists of science have spent forty years nailing the myth that natural scientists are interested or involved in replication. It’s not just about replication studies not getting credit or not getting published – it’s clear (a la Harry Collins’ experimenter’s regress) that replication doesn’t resolve scientific controversy.
So why would ethnographers get het up around replication?
LikeLike
Adam Hedgecoe
April 17, 2014 at 12:02 pm
Many of the things in the original post are wrong probably due to a mix of misinformation, stereotype or simply scarce familiarity of the author with what actually ethnographers do.
Firstly, the idea of an “anti-positivist framing of much ethnography” is trite and sounds so XX century. It is based on a bankrupt way of framing sciences according to a hierarchy of evidence which was itself dreamed in the XIX century (and which will make quantum mechanics unscientific). Ethnography is simply a non-positivist way of doing research. It uses methods and procedures that are different from inductive social science (see the very clear recent explanation by Mantere and Ketoki 2013). If you want a hint that this hierarchy is bankrupt and needs to be revised just start to look for how many neo-institutionalists are employed in industry. Then check how many ethnographers work in companies around the USA.
Second, many of the things mentioned in the post are already common practice –although not in the form prescribed by positivist social science. Many ethnographies are “revisitations” of old studies and many ethnographers return to the original field over and over. Of course, while you can do the same ethnography many times you cannot not replicate it as this concepts belongs do an inductive way of doing social science. Also, you can do ethnography in teams but you end up with several ethnographies — as ethnography is the account, not the observation. Again this is done on a regular basis, and is very useful. Finally joint analysis is also very common ( it even involves users, which, of course, for a positivist is anathema) but people do not standardise codes as this is a daft way of proceeding which would kill what ethnography has offer in alternative to other methods. So what seems to be wrong here is the stereotype of the “solo practitioner model of unique and non-replicable studies” .
There is more than one way of doing social science. Just live with it.
LikeLike
Davide nicolini
April 21, 2014 at 9:36 am