orgtheory.net

why freakonomics bothers me

Fabio

The New Republic recently ran a very critical article  by Noam Scheiber on Steve Levitt’s freakonomics, which uses natural experiments in unexpected data to clearly identify cause and effect relationships, or illustrate economic principles. Ideally, freakonomics isn’t a bad thing – it can be wonderful. If you can exploit a funky data set to address a thorny problem, that’s great. It can also be fun. However, Noam Scheiber raises an issue that deserves more discussion. He argues that freakonomics has lead to “cute-o-nomics.” That is, Scheiber accuses Levitt of encouraging parlor games at the expense of hard thinking. This article struck a chord with me because my model of scientific work assumes that progress comes from thinking long about hard and potentially unsolvable problems, and collecting and analyzing insane amounts of data on real world topics. Freakonomics seems to challenge these ideas.

Taken to an extreme, freakonomics departs from these two modes of scientific inquiry. If your method is to find natural experiments, then you don’t have a strong incentive to deal with tough theoretical problems. You meticulously stick to what can be proven with your peculiar data sets. For example, in the sumo paper, Duggan and Levitt find evidence of cheating among sumo wrestlers. The article doesn’t offer any overall theory of cheating or the institutions regulating/encouraging cheating, although the authors tell us cheating and corruption are important economic topics. The contribution of the paper is to show how you can detect cheating in this sort of environment. Levitt’s defense is usually that these papers address big topics with interesting data. I believe it for his papers on political campaigns or crime, but it’s a hard argument to accept for the sumo paper or the weakest link paper, which examines racial discrimination on a game show. I’m more inclined to believe that the idiosyncratic nature of the social situation (like a game show or wrestling tournaments) is too far removed from normal interactions to provide general lessons. I need a general theory to tell me why game shows/sumo/penalty kicks are typical or atypical events. None is provided.

Furthermore, searching for data quirks means that you let the data drive your choice of topics, rather than let your topic drive the data. Scheiber quotes economist Raj Chetty on this point: “People think about the question less than the method…They’re not thinking: ‘What important question should I answer?’ So you get weird papers, like sanitation facilities in Native American reservations.” You won’t do the meticulous, and often difficult, data collection that is often needed for progress and instead focus on what you can easily extract from existing data. You are a data opportunist, which is ok for a few folks, but is a risky strategy for most people.

Levitt is quoted as saying that “I’ve always been someone who thought it better to answer a small question well than to fail to answer a big question.” There’s wisdom in this – it is important to be able to actually accomplish something in our short time on earth. Taken on its face, however, the quote is misleading. For example, a failed answer to a big question can yield valuable answers to important small and medium sized questions. You might never find these if you stick to interesting toy situations. Another problem is that people actually do solve big problems once in a while. It would have been a shame if Andrew Wiles had stopped working on Fermat’s Last Theorem just to get an easier, but more clever, result. Levitt is also quoted, though not directly, as saying that if you don’t have what it takes to make a dent in a traditional topic, you can try his style. Perhaps, but I would add that you could also try neglected but important topics. There’s more of them than you might expect.

In the end, I’m not anti-freakonomics.  I actually like a lot of Levitt’s papers, especially the ones on politics and crime. A vibrant profession like economics has space for productive and wildly creative people like Levitt. Further, analysis of quirky data has its place, but in the absence of a compelling theoretical program, it has the potential to run into problems. Ultimately, the Levitt I would like to see is hidden inside the papers of the Levitt we now have. I would love to see him mine the insights that seem to be poking out of his papers. For example, in general, what sorts of institutions encourage or mitigate cheating or corruption? What general theory of crime might emerge from his analyses of crime waves in the 1990s? If we got extended answers to these questions, we would have a book that could proudly stand next to Friedman’s Monetary History of the United States, or Becker’s Treatise on the Family, an achievement ensuring his place among the great Chicago economists. In the short term, I suspect we’ll keep seeing papers on game shows and penalty kicks.

Advertisement

Written by fabiorojas

April 26, 2007 at 5:39 am

Posted in academia, economics, fabio

22 Responses

Subscribe to comments with RSS.

  1. Freakonomics is hard to judge–meaning to form a strong opinion one way or the other–as a coherent entity for a variety of reasons, the main one of which is that it is not a coherent entity, and in that manner referring to the heterogeneous strands of research that have recently been grouped under that banner with a single term is somewhat deceptive.

    For the Freakonomics stuff that is most directly related to economics (I don’t think the crime and abortion paper is, although Levitt is a master front-end framer making it look as if everything is related to economics), especially the stuff on cheating, I think you are asking too much of Mr. Levitt. The idea of “innovation” in his slice of the intellectual community is Beckerian, which means that the term innovative scholarship is usually an oxymoron. It is so elementary that it can be written as an algorithm (recipe):

    (1) assume that some version of the standard rational actor model is correct and unquestionably true; preferably the Stigler-Becker version in which preferences are exogenous, constant across time and possibly common to all actors and in which the opportunity structure of incentives (reward systems in sociologese) do all of the explanatory work.

    (2) find a natural experiment in which the incentive structure of a set of actors has shifted in a a manner that was unanticipated by the actors or in which the very process of some established way of doing things (i.e. a game or tournament) provides such a shift without it having been foreseen by the people who established the rules of the process. This allows us to address the thorny issue of actors self-selecting into environments with certain forms of incentive structures.

    (3) demonstrate that under condition (2) actors will behave according to Williamson’s maxim of acting with “self interest” and with “guile” (actors will cheat, take advantage of the new loopholes, etc.) when given the opportunity.

    (4) abduct that the evidence found in step (3) provides support for (1), the metatheoretical structure with which you started. Everything is alive an well in Chicago econ-land. People behave according to the predictions of the theory. New theory? New models? Pushing the boundaries of traditional economic theory? no hablo Español.

    In all fairness, I think that what makes Mr. Levitt’s version of this strategy fresh and appealing is that he begins with a fairly vague and loose verbal model in step (1) as opposed to the mathematical straitjacket that some of Becker’s descendants tied themselves up in. This gives him breathing room to be more creative and think “outside the mathematical box.” The metatheoretical looseness in step (1) makes Levitt seem like he’s just proposing a soft version of what Hechter and Kanazawa referred to as “sociological rational choice theory.” A conception of the rational actor so underspecified and open to suggestions as to make anybody who doesn’t accept it a certifiable loon (since it boils down to such platitudes as “people will take advantage of opportunities if these are given to them”). This I think accounts for the “sociological” flavor of Levitt.

    Like

    Omar

    April 26, 2007 at 11:57 am

  2. I agree…and then I don’t. I think both Fabio and Omar ask a little too much. What was refreshing about Freakonomics is that it suggested economists could answer neat questions in a fashion comprehensible to the lay public. I don’t think there was ever any thought of “general theory” beyond the implict recipe that Omar so nicely lays out. It was precisely the emphaiss on General Theory that has led economists into irrelevance in the first place – lots of math, divoced from data, to produce the barest tweak on an already wholly unrealistic edifice. In short, economics needs a Leavitt. His sucess has definitely generated a sub-grenre of cute-o-nomics, and I can see how this would irriate serious economists (who are likely more than a little green with envy as well…BTW, is that ‘rational?’), but, hey, give the guy a break.

    Like

    freak

    April 26, 2007 at 3:27 pm

  3. …forgot to add:

    Think of Leavitt as sort of the J.K. Galbraith of the neo-liberals. Why shouldn’t they have one? And in a much nicer, less-scary package than Friedman (whose popular appeal never stretched all that far beyond the true-beliver/Ayn Rand crowd).

    Like

    freak

    April 26, 2007 at 3:42 pm

  4. […] Steven Levitt in Freakonomics blog –don’t miss the piece–it is too good to be passed–especially the last three paragraphs; via Tyler Cowen. Take a look at this piece in orgthery too! […]

    Like

  5. I think you hit a major point, Omar. By employing very lose language, the work becomes vague and hard to pin down, aside from the concrete empirical analysis. For example, in the sumo paper, D&L suggest that corruption is important because of the role it plays in development. True, but one is left to piece together the link between corruption among athletes and broader issues. The focus of the paper then quickly shifts to solid empirical work. That style leaves a lot of folks like me confused.

    What Levitt has run into is the problem that qualitative case study researchers have: awesome case, difficult transition to general theory. Except in the crime papers, Levitt side steps the issue by appealing to tightly argued econometrics and the legitimacy of the broadly implied broader issue (corruption, racism, game theory, etc.)

    In sociology, we are used to this and can spot the problem quickly. But this is rare in economics, so I can see how reviewers can be dazzled by the case and care less about the transition to general theory.

    Of course, other economists don’t necessarily have this problem. The drive for “clean identification” in field experiments has much promise because the projects require that the research team have a pretty well developed theory ready to go before you even do the experiment. Thus, you get a clear jump from case to theory. Opportunistic data crunching however means that your nice data doesn’t always have that connection. And that’s the critique of Levitt in a nutshell.

    Like

    Fabio Rojas

    April 26, 2007 at 8:43 pm

  6. The freak and Omar say it best this time Fabio!

    However, something was left from both their arguments (Their humility halted them from saying it). That is, sociologists MUST embrace the form of methodological individualism (i.e., purposive action)engaged in by Levitt; and bequeathed by Max Weber. And with that, sociologists can obtain “thick” descriptions of human phenomena, yet have an analytical, not to mention persuasive, focus.

    Hey sociologists engage in a slew of interesting questions. However, sociology remains crippled by its dearth of analytical structure. Once sociology circumscribes its analytical focus to the type made known by Weber, sociology will have its Levitt.

    By the way Fabio, if the qualitative sociologist is working from the assumption of purposive action, will the case study have to make a huge leap into “general theory?”

    Like

    Brian Pitt

    April 26, 2007 at 11:13 pm

  7. Brian, it is not very often that I get called humble, but I’ll take it.

    The TNR article raises a number of very difficult questions that I think should be separated and unpacked. First, there is the question of the sources of theoretical creativity and what is the best way to go about doing empirical research; relatedly there is the question of what’s best for a discipline: to have lots of people applying for grants that will tackle less glamorous but “meat and potatoes” type of problems, or trying to engage in clever ways to answer interesting questions of possibly marginal “practical” value?

    According Scheiber, it is those who do boring, tedious but “meticulous and careful” work that are the true vanguards of any discipline, and those who do “clever” and “flashy” research should be silenced. I disagree with this stance. First, I think there is certainly room for theoretically motivated empirical research that does not depart from a carefully formulated mathematical (or verbal) model ex ante but that relies on statistical exploration of the data to discover inductive patterns that might then be explained by some theory. This type of “data-first” approach can lead to the discovery of interesting results; as much as a “theory first” approach can. Most really creative research in sociology (think Roger Gould or Peter Bearman) is obviously of this type.

    In my experience doing quantitative empirical research, I have had both experiences, one sitting in my chair and getting the flash of inspiration that says: “if theory X is correct then you should expect that variable Z has an effect on Y and not W” going to my computer, running the models and finding out precisely that (more often than not of course, finding out that my flash of inspiration was just a flash). Other times, I have discovered interesting patterns in quantitative data-sets that cry out for a theoretical explanation. I have developed such explanations and found further patterns as a result of reasoning what other empirical patterns we should observe if my initial inductive guess is correct. This has actually resulted in pretty interesting papers (if I can say so myself) that have no practical value, but that I think make some theoretical contribution. One of my advisers, Ron Breiger, was a big fan of exploratory data analysis a la Tukey, and I imbibed a ton of techniques from him on how to do that in Grad School, stuff that to this day I am still trying to get a handle on (I spent the whole day today writing a Stata program to implement this cool stuff, for instance)

    My sense is that these “two cultures” of quantitative research separate those theoretically creative people that don’t have much patience to write a grant, collect data (sometimes under difficult conditions) and maybe come up with something that is worth reporting, from those careful researches who do enjoy writing the grants, collecting the data, etc. It would be the pot calling the kettle black if I were to condemn Levitt for following the data first strategy and of coming up with interesting theoretical explanations for empirical patterns in data that is already available or that can easily be collected. I think the limitations of freakonomics is that it is not willing to push theoretical boundaries. I think the difference between an economist and a sociologist is that in sociology you are rewarded for theoretical integration, theoretical elaboration and sometimes outright theoretical demolition and reconstruction. We don’t have a sacred set of precepts that we hold dear. In my own case I benefited from questioning a long standing “sociological” presupposition regarding the primacy of social networks in determining culture for instance.

    This means that theoretical bricoleurs can do well in this system, benefiting from the theoretical heterogeneity of sociology and for brokering between all of the structural holes around the various paradigms and schools. In economics, on the other hand the incentive structure promotes theoretical conservatism, because there is such a strong allegiance toward keeping the basic metatheoretical core of the discipline intact. This means that the brightest people in the discipline such as Levitt, tend to gravitate toward creative ways of answering some offbeats questions given the dominant theoretical paradigm. Most of the creativity and cleverness are moved from theoretical integration (possibly involving other disciplinary theoretical streams) and elaboration (which is where the true cutting edge lies) to identifying tricky ways to establish causation given the usual limits of statistical methods. Hence the Levitt quip about economists being technically equipped to answer lots of interesting questions, but being unable to come up with any.

    I think that the attitude of people like James-look out in the parking lot-Hekman, is pure rearguard conservatism. Yes, education, poverty, etc. are important issues, but all of those things are just brute facts that require new forms of theoretical development to be put in the form of puzzles that spark some sort of creative way to address them. Thus, neither Hekman nor Levitt nor Scheiber are asking the real tough question-regarding economics as a discipline that is-here, which is directly related to the theoretical poverty and stagnation of economics (I think that you need somebody like my neighbor Philip Mirowski to begin to do that), and the bizarre reward structure that punishes young scholars if they push the theoretical envelope (labeling them as heterodox, etc.) and rewards those who keep the theoretical faith. Pace Scheiber, no amount of “careful” empirical research is going to solve those issues for economics.

    Mind you, that this is the discipline that pound per pound probably attracts the most intellectually able young people in the social sciences and then fries their brain with morose mathematical formalisms that go nowhere (none of this applies to our Austrian friends of course, who have been making this same criticism of mainstream econ in a much more effective and eloquent way for a long time). No wonder an “anti-charismatic” hero like Levitt has captured the imagination of the younger crowd by telling them that they can have fun doing their research, something that we in sociology take for granted (since we ain’t certainly doing it for the money, which falsifies at least some of the non-question-begging metatheoretical presumptions of freakonomics). If you can’t be intellectually excited by the ideas and theoretical vision of your discipline, at least you can get titillated by imagining the marginal utility of unsafe drunken sex in a run-down brothel in Puebla.

    Like

    Omar

    April 27, 2007 at 1:56 am

  8. There’s a lot in that comment. I’ll add this – progress is cyclical. Sometimes, you need deep thinkers, other times you need craftsmen who do “meat and potatoes.” My concern about freakonomics is that in its worst form, it is neither meat and potatoes enough, nor does it offer theoretical insights. It’s just too darned concerned about its own cuteness!!

    Like

    Fabio Rojas

    April 27, 2007 at 2:18 am

  9. If freakonomics is neither meat nor potatoes, perhaps it is just the appetizer! If Omar is right, then a new generation of young economists will soon emerge that take Levitt’s example seriously. Get enough people working on the problem and you’ll soon build theoretical substance out of these quirky datasets.

    Like

    brayden

    April 27, 2007 at 5:07 am

  10. By the way, Levitt is taking the offensive and calling out Scheiber forcefully.

    Like

    brayden

    April 27, 2007 at 5:12 am

  11. Brayden: Geez, I can now imagine a future world where people read Freakonomics with the same reverence that people used to read Samuelson’s Economics. Out with the axiom of revealed preference and in with the Weakest Link! Samuelson’s turning over in his grave… and he’s not even dead yet.

    Like

    Fabio Rojas

    April 27, 2007 at 5:14 am

  12. […] a recent article in the New Republic by Noam Schreiber criticizes quite harshly Stephen Levitt’s Freakonomics and the whole buzz around Economics applied to unusual, quirky subject. You can actually read the full article is not necessarily aimed at Levitt himself rather than at the negative influence his type of research might have on the profession. By making Economics entertaining, Levitt runs the risk of making the profession more like Cute-onomics. Schreiber is actually quite pertinent when he differentiates between Geeks (data crunchers, lots of hard work, blood, sweat and tears…) and Freaks (on the lookout for a funny theory…), and pointing out that more valuable work can actually be done by researching the (incorrect) answer to a big question, rather than finding the right one to a small question. Levitt answers but i’m a bit unconvinced by his defence, attacking Schreiber inaccuracies is fair, attacking his lack of a PHD isn’t in my opinion. see also opinions from Joshua Angrist,Alex Tabarrok, Greg Mankiw and Fabio Rojas. […]

    Like

  13. I agree with Fabio that progress is cyclical; I agree with Brayden that Freakonomics may be the appetizer; I also agree with Omar that the question of what is best for a discipline must be unpacked. However, cyclical results, whetting the appetites of future scholars, and promoting the discipline is difficult without THEORETICAL COHERENCE. PURPOSIVE ACTION IS THE DIRECTION SOCIOLOGY MUST GO IN. HETEROGENEITY DOES NOT SEEM THE THEORETICAL ANSWER!

    Question:

    Have any of you guys been asked what is sociological theory? If so, what was your answer? Levitt and the economists have little problem answering these questions.

    Listen, I studied econ in undergraduate school. Luckily, I was not inunated with the ersatz mathematical and statistical formalism that many undergrad economics programs see as requisite. But my undergrad advisor made known to me that I was “mathematically ill-prepared” for graduate economics. But luckily for me, I enjoyed sociology, particularly economic sociology and the sociology of law, just as much.

    My overall point in this reply is that from the introduction of Freakonomics, readers are aware of what theoretical position Levitt is coming from. But I am often left guessing about what theoretical position sociologists are coming from.

    Like

    Brian Pitt

    April 27, 2007 at 6:14 pm

  14. Thanks for posting, Brian. Two notes:

    1. If you were to ask me what “sociological theory” is, I would point out that it depends on which kind of theory. For example, rational choice sociology uses the neo-classical Becker-Stigler framework. Marxist sociologists use the ideas of exploitation and class conflict. Institutionalists employ ideas about socialization and diffusion of practices. Social psychologists have an entire theory of roles and attitudes that they have developed.

    2. I would argue that Levitt often does *not* tell me the theoretical position he’s coming from (except for the crime papers). Let’s take the example in my post – the sumo paper. Aside from “people respond to incentives,” what broad economic insight is supported/rejected by the sumo paper? He tells us corruption is important, but what general lesson am I to take on corruption, other than this sport exhibits occasional cheating? The link is never made.

    Similarly, if I read the penalty kicks paper, what broad point am I to learn, other than if you take professionals and put them in a highly constrained environment, they will actually do the optimal behavior? What mind blowing insight does that imply?

    My favorite “no theory” move is in his paper on Senatorial voting. He literally claims he’s rejected the median voter theorem, and then leaves us with… that’s right, nothing. Senators just vote according to their preferences but there is no theory offered of how they do vote.

    The exception to this tendency are the papers on the crime waves of the 1990s. In the criminology literature, there is a huge debate about what causes fluctuation in crime rates. Levitt’s papers on this topic are truly outstanding examples of how to take a tough problem and answer it with thoughtful empirical analysis. I know other criminologists dispute his conclusions, but I feel confident that Levitt has made a strong case that certain types of crime do follow national economic trends and not other social indicators. These papers combine big data with well motivated hypotheses and careful analysis. They are, in many ways, the opposite of the other papers, which use weird toy data to answer weakly motivated questions.

    But don’t get me wrong, I am not against fun papers, or against “clean identification.” I do actually like much of what Levitt writes. And if Freakonomics brings people to social science, I am all for that as well. What I would prefer some sort of cumulative effort to make stronger links between his quirky data sets and some sort of theoretical point.

    Like

    fabiorojas

    April 27, 2007 at 7:00 pm

  15. fabio,

    what’s wrong with (as in the senator paper) just testing a hypothesis and knocking it down? i thought hypothesis-testing (and not hypothesis-replacement) was the core of normal science. saying that levitt should have proposed an alternate theory of voting is expanding the scope of what the average paper should accomplish.

    furthermore an atheoretical finding of “well it’s kind of messy” is actually a pretty good bit of humility given that as a social science we deal with messier data than celestial orbits or the trajectories of artillery shells.

    Like

    Gabriel

    April 27, 2007 at 7:14 pm

  16. Gabriel, there is nothing wrong with a-theoretical hypothesis testing, but too much draws you away from work that integrates what we know based on prior hypothesis testing. That’s why I truly admire Levitt’s crime papers. Taken together, they constitute an admirable example of testing various theories of crime and then providing a comprehensive picture of what you learn from the exercise.

    Perhaps the senatorial voting paper is not the best example – it is motivated by the Median Voter theorem and he uses data that is clearly important. It can stand on its own. However, the other papers really need more motivation. E.g., the cheating papers (sumo, teachers) – what hypothesis was tested there? That cheating sometimes happens? Or in other cases, like Weakest Link, that in some idiosyncratic situations like game shows you don’t see much discrimination? He does appeal to various theories of racial discrimination (beckerian tastes vs. statistical discrimination) but how game show data relates to real world situations is a dicey jump.

    Like

    Fabio Rojas

    April 27, 2007 at 7:29 pm

  17. fabio,

    fair enough. i think we agree that a paper should at a minimum test a well-framed hypothesis and at best provide a theoretical synthesis. i agree that papers that just vaguely look at kind of cool stuff are insufficient (fun as they may be to read and to write).

    Like

    Gabriel

    April 27, 2007 at 9:27 pm

  18. The two closing paragraphs of Joel Levine‘s Exceptions are the Rule:

    If you assume that political identity is a polarity, strong Republican versus strong Democrat, and then attend to the correlates of this form of political identity, you are building your results on an untested foundation. If you assume that real-world friendship and antagonism are comparable to mathematical relations, an untested model is adopted. When you try to mark off the time of human events by the pace of an astronomical clock you adopt a model from another science whose applicability to our own is an open question. When you set up equations to predict individual social mobility as a linear function of prior causal events — the more events, the less the variance — you assume a contestable theory of social reality.

    Can a social scientist “do” theory? Obviously the question accepts only one answer: “Yes.” But the word “theory” is corrupt in current social science. To some social scientists the word “theory” has no meaning, only a connotation of high status. For them, whatever they are doing — that’s theory. By contrast, to some, “theory” connotes low status. For them “theory” connotes verbal speculation by ponderous thinkers who know nothing of science. To others, “theory” is a term in contrast to “reality.” For them, the further the research can be removed from data the better — I don’t do windows. I don’t do data. I’m a theoretician. To others, “theory” means “not proven” as in: “Why should I believe you? That’s only a theory.” Let’s not fight over the definition. We are trying to describe the social world. We are trying for description that has generality, precision, parsimony, falsifiability, and matches the data.

    Like

    Omar

    April 28, 2007 at 6:39 pm

  19. […] why freakonomics bothers me […]

    Like

  20. […] up on the ongoing discussion about Freakonomics, my review of the book just came out in Sociological Forum, and I think it […]

    Like

  21. […] rise of conservative media like Fox, increasing tenure requirements in the post-baby boom era, freakonomics is fun, poor wardrobe choices, reading too much Jeremy […]

    Like

  22. […] Freakonomist supreme Steve Levitt studies crime, fertility, and other topics without formal models and vanilla statistics. A legion of imitators follows. […]

    Like


Comments are closed.

%d bloggers like this: