why the identification movement?
I made three claims in my previous posts. First that economists were, as a group, much less wedded to a single model of human behavior than they used to be. Second, that “modeling for the sake of modeling” is a much less prevalent phenomenon. Third, there is a growing place in the literature for research that is only loosely motivated by theory but makes convincing causal arguments (and sometimes informs theory as a result, see here). Now, clearly, there is wide variation in the extent to which these developments have taken hold, across institutions, across subfields, and across journals. But the fact that there exists significantly more diversity in research styles and approaches today than in a previous era seems true as a factual matter.
[As an aside, I do not have a problem with “neoclassical research” per se, maybe in contrast with many orgtheory readers. The paper by Acemoglu & Pischke I mentioned previously is a beautiful example of applied economic theory that yields testable empirical implications that one probably would not have been able to anticipate before the modeling exercise.]
I am most interested in discussing the cross-disciplinary appeal of the identification movement, because I see no particular reason why it should remain confined to applied economics. I also want to be careful and separate normative from positive claims. It’s not that I don’t make value judgments on these things, I certainly do, as many of my colleagues will attest. But these seem to me to be conversation stoppers. orgtheory readers care about social movements and diffusion phenomena, and it is in that spirit that I would like to proceed. What are the barriers to adoption?
Ezra mentions in his comments that experiments (both lab and field) have a storied history in Sociology, one that probably predates the wide adoption of experimental methods in economics. He is of course correct. But I will maintain that sociologists/org theorists and economists tend to place their emphasis on different features of an empirical research project. I’d like to discuss this through examples in the next few posts.
One comes from the personal experience of presenting the same paper to Sean’s group at Chicago GSB, and to various seminars in econ departments. I don’t want to provide a lot of detail, but the basic point of the paper is to look at the effects of collaboration with a superstar scientist by examining what happens upon the sudden and unexpected death of said superstar. After establishing the basic facts (research output goes down, gradually and permanently), we try to elucidate the mechanisms that could explain this, and we have some success (I think!) demonstrating that it is driven, at least in part, by a genuine loss of ideas. I had great fun presenting to both audiences. But the great majority of the questions I got from Sean’s colleagues were in the form of provocative/intriguing suggestions on how to dig deeper on the mechanisms. I could not necessarily act on these suggestions in this particular paper, but these discussions certainly planted the seeds of future projects using the same basic design (e.g., how does the network heal after the [plausibly exogenous] loss of central actor?)
My econ colleagues, on the other hand, did not buy the basic research design, at least not right away (maybe one hour and 15 minutes into the seminar!). They made the reasonable point that it is problematic to use colleagues of stars who have already died as controls for those who experience the loss of a superstar currently. They pointed out that there could be unobserved, collaboration-specific life cycle effects that might bias the results downwards. And much time was spent on discussing the construction of control group that would allay these fears. It’s not that economists did not care about mechanisms. They did, but – dare I say it – in a relatively less sophisticated way than Sean, Matt, Damon and Co. Mostly, they wanted to know if superstars mattered because of their connections or because of their ideas. And that was that.
This contrast encapsulates what I mean by “two empirical cultures.” Many orgtheory readers teach at business schools, and therefore are used to present their work to “mixed audiences.” But these two audiences were as homogenous as it gets, and it may explain why these differences are so salient in my mind.
orgtheory.net 
I think I am with Ezra on this: lots of people already do identification (e.g., experiments, searching for natural experiments, IV, etc.) So the question is actually something like: why don’t non-economists adopt the identification tools of economists? Hypothesis:
– Low tech field, you don’t have enough people who can quickly use and adopt new math intensive techniques
– Aversion to “cute-o-nomics.” The most well known recent id studies are “cute,” which can turn people off.
– Good instead of perfect. Economists put first priority on math model building. Clean identification fits nicely with that general framework. If sociologists required clean models, entire fields might disappear since so many topics have limited data (e.g., social movements). It’s really hard, though not impossible, to build big stories on a series of clever identification papers.
LikeLike
fabiorojas
July 30, 2009 at 7:29 pm
For those of you who aren’t sure what Pierre means by “identification” (since for many org theorists and sociologists identification is a theoretical concept, not a statistical term), check out this paper by Steven Levitt about the contribution of James Heckman to economics – http://pricetheory.uchicago.edu/levitt/Papers/LevittHonoringHeckman2002.pdf.
I think part of the problem lies in the value and belief differences between org. scholars/economic sociologists and economists. Most of org. scholars don’t really believe that statistical models are ever perfect and that perfect identification is something worthily sought after but that is rarely obtained. Therefore, we may be more interested in a finding based on an imperfect model and an interesting theoretical contribution, especially if the finding has high external validity, than we would in a model that is properly identified but doesn’t offer a new theoretical twist. That is one reason that qualitative research is so important to sociologists – it provides the robustness check that no statistical diagnostic can. Mixed methods are big right now for that reason. I would add that org. scholars certainly work hard to make sure there regression models are robust (e.g., many of the tools that Heckman developed are now routinely applied in any top sociology or management journal), but we are ultimately most interested in the theoretical contribution.
I would also add that there are a group of sociologists who are primarily interested in identification and care a lot less about theory – demographers. Giving a talk to a group of demographers is fun because you can nerd out as much as you’d like on the specifics of the model and not worry so much about what the big contribution is. Demographers, in that sense, are culturally more similar to applied economists than they are to economic sociologists.
LikeLike
brayden
July 30, 2009 at 9:38 pm
I wonder how much of this comes from the various sources of academic insecurity on both sides of the discussion. I’ll agree with Brayden and Ezra that there is maybe more attention paid to identification questions among economic sociologists than you’re implying. But agree with you too that the lucky bastard of an economic sociologist who comes up with a really great instrumental variable for pure status effects or for trust or for reciprocity or for tolerance (or maybe even something not currently central but nevertheless mentioned in passing by Durkheim or Simmel), will likely become a star of the field.
But as it stands, many of us seem locked into this mental complex about economists and how they see our work. As Fabio and Henry Farrel’s back and forth suggested, we see economics as our “other”. What economics has on us for now is a set of mechanisms that are not only quantifiable and “provable” but which are readily understood by sophisticated laymen. (Or in the absence of understanding, at least widely accepted. Perhaps economists’ cocky attitude has something to do with that?). Or maybe we don’t really need to bring the economists into it; econ soc is looking for the next generation of really interesting ideas to emerge; its been a long time since 1977.
Just as we’ve got our insecurities, so do applied economists. As far as I can tell, they have less to do with the theory of action (that’s basically assumed to have been proven for some time…). To Fabio’s point, the applied economists I see on a regular basis do seem worried about the question of cuteness. Whats the implication of the findings? Is it all just a series of debaters’ points? Freakonomics might be cool. But price theory (and increasingly behavioral economics) have really moved policy. At least thats my read on where the ideological faultlines that might produce anxiety lay. Would be interested to hear your take.
The upshot is: one hypothesis would be that it has to do with the boundaries of the fields and where the “other” is located. The other, I guess, would simply be a matter of cultural drift. And its only at places like MIT and Chicago where there’s enough of a cross-fertilization between the two worlds that there’s any angst at all about it. If one buys the idea of structural holes and good ideas and all that, it would suggest that those are the places where productive convergence or mutation will eventually emerge.
LikeLike
Sean Safford
July 30, 2009 at 10:16 pm
As someone who used to be an economist (ABD in economics from MIT, and worked as a policy economist for many years) and is now an Org Theorists, I think one big difference is that economists want to make explicit policy predictions, while org theorists are more interested in explaining how the world works, most often ex post.
In the policy world, where I used to work for, providing the right number is how you have impact. This is what is now going on in the health care reform debate and why the CBO is so influential in whether and how reform will happen.
If you play the policy game, then the identification game is critical as you want to make sure you have the right causal model to predict the “right” number.
As Brayden’s comments implies, org theorists are skeptical of there being a true, number to predict any effect. In my view, and I believe implicit in many if not most org theorists, any estimation is predicated on existing networks and existing cultures, and if these are not stable, the estimated coefficients may also not be stable. So even if you somehow solved the identification problem you would still have the problem of structural stability. Now I think this perespective is correct but at the same time less useful to policy practitioners who want an answer, and an answer given in numbers–e.g., what are the fiscal consequences of a public option?
When I was a public policy economist I once got the CBO to change an estimate of a provision in a bill from a cost of $80 million a year to $1 billion a year. I did have a statistical model to back up my numbers but I can’t really say that I totally believed what I was doing. And I must say that I don’t really believe a better identified model would give us the truth either. But it would sure sell better in Washington.
After my CBO experience, I went back to graduate school and got a Ph.D. in organizational behavior. It works better for me.
LikeLike
Willie
July 31, 2009 at 6:28 pm
[…] Brayden and Willie expressed doubt with the idea that there might be a “correct” treatment effect that social scientists could uncover given our methods and the availability of data. In their view, the perfect could be the ennemy of the “good enough,” especially if obsession with clean identification leads us to look for our keys under a limited set of lampposts. I have sympathy for this view. A bit of intellectual history. Twenty five years ago, econometric studies were viewed with a great deal of cynicism in the profession. Lalonde (1986) showed that our non-experimental techniques performed dreadfully (in terms of replicating an experimental benchmark) when applied without a great deal of forethought. More than any other development, the identification movement can be understood as a reaction to Lalonde’s findings. From then on, it became clear that technique would not be enough to make convincing causal claims. To give just an example, it is of course possible to use Heckman’s (1977) famous two-step sample selection technique to estimate a treatment effect free of sample selection bias. But without a convincing “exclusion restriction,” the exercise becomes part of an elaborate ceremonial that authors, editors, and reviewers engage in. Such estimates are not much more convincing than plain vanilla estimates that ignore sample selection bias. […]
LikeLike
Yet more on experiments, in and out of the field « orgtheory.net
August 2, 2009 at 10:38 pm
[…] isolated one from another. They are not mutually exclusive. This gets to the question of the empirical cultures of economics and sociology. My provocative retort would be that identification-oriented […]
LikeLike
azoulay bait: russian dolls and the identification movement « orgtheory.net
August 18, 2009 at 3:00 pm