A detour through health care reform

Lots of good stuff in the comments about my post on the identification movement. I’ll take that up next week, but today, I’d like to further illustrate what I mean by talking about health care reform. It’s fair to say that Atul Gawande’s (justly) celebrated piece in the New Yorker became exhibit A in the administration’s effort to reform the health care system. Gawande’s article focuses on the delivery of health care services in McAllen, TX, one of the most expensive market in the US for medical care.

Though Gawande is not a social scientist (he is a surgeon by training), one senses that he approaches the “cost conundrum” with an open mind. And he usefully contrasts his observations of medical practices in McAllen with what’s happening in El Paso, another Texas county with reasonably similar demographics, but no worse average health outcomes. His explanation for variation in health expenditures not explained by “fundamentals” (cost of living, patient heterogeneity, etc.) is appealing to orgheads, regardless of their disciplinary persuasion. What Gawande finds in McAllen is a referral system between generalists and specialists that looks and smells like medical payola, and high-powered market incentives leading to a general erosion of professional norms. The inescapable conclusion: it is possible to cut health care costs without affecting the quality of care, but it will require a mix of legal and “cultural” interventions (most notably in medical schools).

Its audience may not be social scientists, but the piece nonetheless puts forth a perfectly plausible and interesting hypothesis, with first order policy implications to boot. My question: how do we know if it’s right? And how would an applied economist obsessed with causality approach the cost conundrum issue that motivated Gawande’s piece?

Enter my MIT colleague Joe Doyle. In what I will call his “vacation from hell” paper, Joe does something very clever. One problem with Gawande’s analysis is the presumption that patient populations are similar across geographic areas. As Joe puts it:

“Estimates of returns to healthcare spending can be confounded by the fact that individuals in worse health receive more care. In fact, there is a strong positive correlation between spending and mortality at the individual level, even after controlling for observable characteristics such as age and comorbidity levels. Regional measures of spending intensity aggregate the choices made at the individual level, which can again confound comparisons.”

To deal with this issue, he looks at mortality outcomes among tourists in Florida who have a heart attack, and how that varies depending on whether the area in which they are vacationing are low- or high-spending areas. The point is that these patients end up in medical systems that were not designed with them in mind. Furthermore, since we are looking at an emergency condition, they are unlikely to have engaged in a lot of comparison shopping. I’ll spare you many of the details (read the paper!), but I will point out that (1) areas that receive a lot of tourists can be found among high and low-spending Florida HSAs; and (2) Joe is careful to compare areas that are close demand substitutes among tourists.

Variation in Health Care Expenditures Across 97 Florida HSAsWhat does he find? In short, mortality from AMI is much lower in higher-spending areas for tourists, but not for locals. In other words, selection matters a lot when making these geographic comparisons of expenditures and health care outcomes.

What should we make of this result?

At the very least, the study should give pause to reformers who believe that it will be easy to slash health care costs without affecting outcomes. Health care reform, the public option, etc. might be good ideas, but they are unlikely to be a free lunch. This does not mean that Gawande is wrong. As Joe will be the first to recognize, there is probably a lot of waste in the health care system. But this situation reminds me of the marketing director’s dictum: “I know that half of my advertising budget is pure waste. I just don’t know which half.” Similarly, we may need Gawande’s scalpel, rather than Perter Orszag’s hatchet, to cut into health care spending without affecting health care outcomes.

That’s the policy lesson. But what does this have to do with “two empirical cultures”?

First (this will answer the first of Fabio’s comments), there is nothing “high tech” about the paper. It’s all totally transparent, and for the most part just uses OLS. In fact, the whole movement is predicated upon the idea that a lot of effort will go in the upfront design of the study (setting up the right comparison), so that one can tell the whole (or at least most of the) story with a simple table of means. There is nothing here that I could not explain to my mum.

Second, I don’t know whether Joe’s paper is “cute.” I’m pretty sure it’s clever. And I am absolutely sure it is important. Although lots of people accuse Steve Levitt of practicing “cute-o-nonics,” perhaps his most well known study dealt with abortion and crime. It might well have been wrong, but it certainly was not trivial.

Third, quasi-experimental papers are often criticized on the ground that they strike the wrong trade off between internal and external validity (I hear echoes of this criticism in Brayden’s comment, but I may have misunderstood him). Generalizing a result such as Joe’s is certainly fraught with hazards. What about other states? What about other, non-emergency, diagnostics? etc. To this I would retort that making the McAllen vs. El Paso comparison the ideological lynchpin of health care reform is also dangerous. But more importantly, it may be more useful to try to replicate Joe’s findings using a different, but equally convincing quasi-experimental approach, rather than cutting corners on establishing a clear causal effect using a much larger dataset.

Finally, Joe’s paper has fairly little to say about mechanisms. How do high spending areas do it? The paper is just not set up to answer this in detail. In contrast, Gawande’s piece was all about mechanisms. My sense is that this is precisely what org theorists find unappealing about the identification movement. And I will agree there is a relative paucity of “whodunnit?” in papers exploiting natural experiments. There are two counter arguments. First, social scientists should have no business talking about the mechanisms driving a causal relationship before they make sure the relationship is indeed causal (the “polishing brass on a sinking ship” metaphor comes to mind). Second, what is true of quasi-experimental papers is not true of papers that implement field experiments (as in here). And though Ezra reminded us that economists did not pioneer this methodology in the social sciences, they are making up for lost time. Up until the beginning of the summer, the E52 building at MIT was teeming with grad students working on field experimenst for their dissertation, very often with topics that are eminently soc-friendly, such as networks, discrimination, peer pressure, etc. What are econ soc grad students up to these days? I sense a very different order of priorities, but I’d be happy to be proven wrong.

More next week!

Written by Pierre

July 31, 2009 at 9:35 pm

5 Responses

Subscribe to comments with RSS.

  1. Once again, I like this post. A few comments:

    – In the context of “two empirical cultures,” this example strikes me as odd. I don’t think any sociologist would dispute the importance of such exercises, and a basic tool of nearly all research (even qualitative) is the strategic comparison.

    – In a discipline with ethnographers and historicists, OLS *is* high tech. If you do more sophisticated things, like I-V or discontinuity, then you’re really talking about a specific slice of the profession, not general knowledge. Also, just doing an OLS probably wouldn’t hack it in most econ journals. You have to create a deep and strong case for a particular instrument/comparison and that can get sophisticated very quickly. 1 semester of OLS wouldn’t prepare most people to produce original JPE quality papers.

    – You ask about the priority of econ soc grad students, there’s a healthy group of people doing audits, as well as other interesting stuff that has nothing to do with identification. For example, Deva Pager has had a successful career doing them (tenure at Princeton), as have others. My colleague Steve Benard (1 year past PhD) has published on experiments in the field. Here’s his description:

    “In a recent paper (with Shelley Correll and In Paik) in the American Journal of Sociology, we find that mothers are disadvantaged on every measure of evaluation in a controlled experimental setting. Evaluators see mothers as less competent and committed than other workers, hold them to higher standards, see them as less worthy of hire, promotion and management training, and offer them lower salaries, while frequently rewarding fathers for being parents. In the same paper, an audit study of real employers shows that employers call back applicants who are mothers half as often as women without children”

    So I am truly puzzled by your view that sociologists are openly resisting identification. Perhaps the real issue is this:

    There is identification the technique vs. identification the fad.

    Most social scientists are on board with identification and have been for years, but, from an outsider’s view, seems as economists are in the middle of a fad. Nothing unique about that, soc has its own fads as well. And fads can be good – the best will stand the test of time and will become part of the standard tool kit. But the mark of a fad is that everyone is doing it to the exclusion of hard foundational work.

    When I think of great economics, I think of questions like: “Is the laissez-faire American model the best?” or “What causes business cycles? or “What will help India pull out of poverty?” I don’t think of: “Will variation in the size of streams and rivers help me identity an effect of competition on school performance?” Maybe I’m misguided, but the tool has to serve the question, not the other way around.



    August 1, 2009 at 1:40 am

  2. Pierre – I really can’t agree with your assessment that “this is precisely what org theorists find unappealing about the identification movement.” I don’t think that org theorists really lose much sleep over the identification cause, and in fact, most of us are in complete agreement with the general principles. There isn’t a comparable counter-movement within org. theory or sociology….

    Honestly, I don’t think there’s much controversy here. I think we can all agree that, all else being equal, better control is a good thing. My earlier comment was meant to just point out where there are slight differences in priority for the two groups. The org. theorists may be willing to sacrifice some internal validity for the sake of external (to use those terms).

    Most criticisms levied against the identification economists come from within economics. The “cute-o-nomics” jibe isn’t something you hear thrown around the halls of the Academy or ASA. In fact, most sociologists I know who do this type of work are really intrigued by and interested in the freakonomics literature. It’s fun! And it’s also more relevant to our research than the rest of economics.

    One point that I’d add to Fabio’s comment – there’s a great deal of heterogeneity (methodological and theoretical) within sociology. In fact, perhaps one reason the identification movement has caught on more in economics than it has in sociology is because of that heterogeneity. We have very high-tech sociologists in the same department as extremely qualitative sociologists. They all get along because they’re happy to be fragmented and work in their own silos. In most top departments these days, though, grad students end up getting exposed to all of these methods. (I’d be surprised if OLS is considered high-tech in any of the top 20 departments.) As a result the newer cohorts of sociology profs are more likely to appreciate and use a diversity of methods, but the generalism of their methodological repertoire may inhibit them a bit from becoming entrenched in identification as a primary interest.



    August 1, 2009 at 4:57 pm

  3. So, I’ve been kind of following the thread through the previous few posts (and attendant discussion), and I feel a little like Brayden – don’t see much controversy, and don’t really see much appreciation or disparagement of the identification cause. At least not in econ soc/culture/orgs that I’m involved in. Perhaps management schools are different. Or perhaps you end up arguing with the people who are down the hall in your own dept.

    I think it’s a generally good think to consider productive ways to envision better health care, social welfare, etc. As the resident hater on these kind of meta-disciplinary discussions, I’m much more interested in your findings, interests, challenges, directions, etc. than I am invested in fishing yet again for a bridge over the soc-econ divide. I think perhaps orgtheory as a collective has a stake in this, per the many discussions of it, but it’s a tough row to how.

    And that last bit about your grad students – it looks a lot like you’re making a claim that identification is important simply because you’ve got a lot of graduate students working on it. I don’t think that’s what you want to be saying, but if that is indeed your claim, I call BS.



    August 1, 2009 at 8:16 pm

  4. Pierre:

    I think Willie’s comment on your last post is relevant here. Joe is writing for a policy audience and there is relatively little demand for such papers by sociologists. Given that, it might be worth looking at the July 2008 issue of AJS, which had a fabulous symposium (including sociologists and economists) on the ‘Move to Opportunity’ experiments. Since it focused on an area where both sociologists and economists speak to policy issues, it is something of a fairer test of how the disciplines differ in their orientation towards experiments and related matters. I personally found Sampson’s article to be a very thoughtful and balanced approach to the issues, with appreciation for what experiments can do but useful warnings about their narrowness.



    August 2, 2009 at 2:28 am

  5. Peter: it certainly was not my intention to imply that field experiments are important simply because a lot of graduate students working on it. All I have been triyng to do in these posts is to describe difference in attitudes and empirical styles across disciplines, because I feel these are more important than other distinctions that are often drawn between the different social sciences in general, and economics and sociology in particular. Also, I think I have been careful not to be normative about these choices.



    August 2, 2009 at 2:22 pm

Comments are closed.

%d bloggers like this: