Wal-Mart and Beyond: Can Social Science be Itself in Court? (Response to Professor Winship)

Amy Myrick is a doctoral student in sociology at Northwestern University. Amy completed her JD from Northwestern University Law in 2009. Amy helped draft the Amicus Brief filed by the ASA about the Wal-Mart case and is a coauthor with Laura Beth Nielsen on related papers.  The following is a response to Chris Winship’s earlier post about the Amicus Brief.

Wal-Mart attacked Sociology.  The ASA responded. 

An important overlooked point for those who think the ASA should have abstained:  the ASA decided to file a brief in this case to defend sociology at large, not to defend Bill Bielby or his conclusions.  Wal-Mart’s Supreme Court brief – widely read and reported in both academic and non-academic circles – attacked not just the expert in this case, but sociology’s basic legitimacy in a way that demanded a response.  Wal-Mart claimed that because women employees could not identify a specific discriminatory policy, they had to rely on “statistics, sociology, and anecdote.”  Wal-Mart then derided each of these sources of evidence, devoting an entire section of its brief to the discipline that ASA helms and to which we all belong.  At minimum, Wal-Mart picked the fight.

Wal-Mart then used an article from a law review (not peer-reviewed) to summarize what sociology “does” and is incapable of doing, even adopting a term – social framework analysis – that sociologists do not own, and that fails to capture sociology’s actual capabilities.  According to Wal-Mart, “Dr. Bielby’s social frame-work analysis fails because it lacks a reliable, scientific basis for linking general research to the corporate setting.”  This assertion does two things:  it labels sociology with legal jargon and claims that, per methodological shortcomings, its cumulative research has no “scientific” value in court.

Had ASA not filed a brief, Wal-Mart would have been allowed to redefine sociology as part of a sham triumvirate that has nothing “scientific” to say about corporate practice in cases like this.  Walking away would have been an embarrassing surrender.  The ASA brief is clear that it aims to show how sociology can make supported claims about how particular cases are likely to work based on cumulative research, and that reliable methods govern this process – in other words, this is science.  Bielby comes in only in reference to whether or not he used those methods.  In the ASA’s own words:

“While we offer no opinion on the substance of Dr. Bielby’s testimony or conclusions, we stress that the methods through which he reached these conclusions are widely accepted and are the bases for research published in the top peer-reviewed social science research journals.”

The ASA defended Bielby’s Methods.  Some Disagree.

            If only sociology’s reputation were at stake, the ASA could have filed a brief that did nothing more that explain what sociology is and try to differentiate itself from (bad, in Wal-Mart’s view) statistics and anecdote, thereby defending its relevance.  But the ASA brief went farther – it placed a disciplinary stamp of approval on Bielby’s specific methods.  To review, that method was to: (1) review aggregate literature on employment practices and their effects, (2) make qualified (“may”/ “might”) claims about the likely effects of Wal-Mart’s practices on the organization’s vulnerability to discrimination; (3) using discovery materials provided by Wal-Mart, based on how they relate to the cumulative findings.

This is where Professor Winship says he diverges.  While he agrees that sociology “obviously” should play a role in court cases, he rejects Bill Bielby’s form of participation (delivery of expert testimony in support of plaintiffs) as generally “problematic” because it falls short of sociology’s standards for research, which, from the legal side, are also what Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993) requires for admissibility.  Professor Winship believes, contrary to the ASA, that Bill Bielby and other sociologists who testify for plaintiffs do not meet scientific standards.

Professor Winship offers three ways that scientific rigor is compromised by experts like Bielby:  1) Their “appropriate participation” is hampered by litigation dynamics that select for scientists with extreme views, conceal their testimony from peers, give judges wide power to choose what to admit, and preclude original research; 2) The logic of law requires a scientist to ignore conflicting evidence, whereas the logic of science requires her to weigh and address it; and 3) Sociology’s public image will be tarnished if people see it as adversarial, partisan, or changing over time.

Professor Winship presents this broad swath of concerns as collectively threatening to scientific standards, and therefore relevant to how (although he seems to mean “whether”) social scientists can participate in litigation.  But, unpacked, they seem to threaten in different ways that suggest different responses.  While some deal with methods and veracity of knowledge – perhaps the most central issue for ASA – others deal with format, and a third set relate to extra-scientific issues of reputation.

1. The Epistemic Critique:  Unscientific Because It’s Wrong or Unsupported?            

Some of Professor Winship’s concerns revolve around whether the epistemic content of expert testimony can meet rigorous scientific standards – in other words, whether experts can know what they claim to know.  This concern echoes Wal-Mart by suggesting that the kind of sociological conclusions that experts reach are simply not supportable by methods that draw on cumulative research.  Winship, too, worries that because experts “cannot conduct the types of research that they might otherwise,” their conclusions in cases are scientifically suspect.  He notes that Barbara Reskin, an experienced plaintiff’s expert, has refused to draw “scientific conclusions” and admitted that her testimony would not be published in peer reviewed journals.

Winship’s concerns about the quality of knowledge suggest that scientific “rigor” only attaches at the point of publication.  In reality, producing research involves lots of steps.  The first step – forming a non-arbitrary, testable hypothesis based on what cumulative research suggests about the case at hand – is closest to what social science experts do when litigation constraints prevent them from conducting new research.  As Barbara Reskin notes, her role “is to use my scientific expertise and to draw on social science knowledge to try …to understand what might be going on.”  This is something that sociologists do all the time in research, most often at the hypothesis-making stage.  Unless one thinks that there is no “rigorous” method at work in hypothesis formation, or that this task is unscientific, Winship’s concern is overblown.

From the legal side, the correct Daubert standard is not whether the expert opinion is publishable as scientific research.  Again, this is impossible given what can be delivered in the courtroom context.  Instead, Daubert requires that the methods be legitimate within the discipline for the purpose they are being used.   There are lots of different purposes in science, everything from conducting original causal research, to reviewing aggregate findings, to forming a preliminary hypothesis.  In litigation, the expert’s purpose is to offer an informed (not arbitrary or subjective) opinion about a particular case that she cannot access first-hand, drawing on her background knowledge.  Daubert then requires this to be done in a way that meets disciplinary standards.  It is a fallacy to say that Daubert requires all expert testimony to be the equivalent of academic publication.  Publishing is only one of many things that social scientists do.  Expert testimony is much more akin to making an informed hypothesis based on currently available information and cumulative past findings.

The difference, of course, is that expert hypotheses delivered in court normally cannot be tested at a later time.  So, they remain forever preliminary.  But, preliminary doesn’t mean “unscientific” or “un-rigorous”; it simply invokes a different stage of the process.

2. The Formatting Critique: Unscientific Because It’s One-Sided?

A major theme for Professor Winship is that expert testimony cannot be scientific if it presents only one of multiple possible views.  This concern underwrites his point about the different logics of law and science, the latter of which he claims is consensual, and therefore requires scientists to address conflicting evidence.  Here Professor Winship seems to conflate the presentation stage of research with the conclusion-forming stage.  The latter does, indeed, require scientists not to ignore conflicting evidence when reaching their conclusions about data.  But, the reason for this mandate is accuracy (see above on epistemic content) – not a logic of consensus.  In Wal-Mart, the question is whether the expert had information that he believed would contradict his conclusion (that Wal-Mart’s employment practices may be discriminatory because they have been linked to discrimination) but chose not to consider it.  That would be unscientific.  Professor Winship has not made this claim (yet!).

Luckily, the “adversarial logic” of law doesn’t require experts to downplay conflicting evidence when reaching their conclusions.  They are, of course, under an oath to speak truthfully, and Daubert requires them to speak as good disciplinarians.  The law requires experts to state a conclusion that they believe is valid according to generally accepted scientific standards.

The adversarial logic of law, however, allows them to make a one-sided presentation after conclusions have been reached.  But here there is no conflict with science.  It is perfectly acceptable, and standard, for sociologists to present work that offers their interpretations alone.  Sociologists who find that culture drives vote choice need not present conflicting research that the economy drives vote choice in order to satisfy disciplinary logics, consensual or otherwise.  At the presentation stage, format is negotiable.  So, there is nothing “unscientific” about providing a one-sided presentation.  Selectively forming conclusions, in disregard of data, is unethical, but that allegation about experts remains to be made.  Barring that, each adversary is responsible for presenting work that supports its position – not too different from an ASA conference panel.

3.  The Reputational Critique: Unscientific Because of Popular Perception? 

Finally, Professor Winship is concerned that by participating in litigation, sociologists will sully their reputation for scientific objectivity.  He worries that sociology will lose its “authoritative” voice if people see it as 1) open for adversarial retention, 2) partisan, or 3) evolving in its views.

These concerns suggest that sociologists should chill useful knowledge in the interests of reputation management.  We find this idea especially appalling.  Professor Winship says he chose his field because he wanted to use science to help “make[] a better world.”  In the area of employment discrimination, social science has moved toward that goal by generating reams of knowledge about how actual practices work, and has the tools to frame valid knowledge for use in the courts.  But, worried that it will not reflect well on its creators, Professor Winship urges sociology  to think about ourselves instead of the world.  Yes, discrimination has become a partisan issue.  But in truth, the overwhelming bulk of principled research shows that it continues to happen in organizations.  So, should sociologists step away from these findings because politics has given them an additional significance?  We find his idea that the evolving nature of science threatens its credibility to be especially puzzling.  A science that doesn’t evolve – see, e.g., creationism – is totally discredited in the eyes of most people.

In addition, Professor Winship seems to think that non-participation promotes an objective reputation. Why?  Does a social scientist who spends a career researching discrimination, then calls repeatedly for sociology to step away from Wal-Mart and other large corporations when they get sued, appear objective?  Non-participation is clearly political.  Witness people who say the courts should “stay out” of everything – does anyone think they are objective?

Finally, as a trained lawyer, I dispute Professor’s Winship’s idea that sociology seems most legitimate outside the discipline when it is most removed.  To the contrary, many lawyers feel that social science is irrelevant and has nothing to say to the real world.  Actual participation is the way to change that sad reputational liability.

Professor Winship maintains that social science “obviously” should play a role in court cases.  But what’s left after his multi-pronged critique?  It is difficult to imagine an institutional design that would satisfy not only his concerns about methods and validity, but also his insistence on format, and fears about the reputational consequences of participation.  If these inescapable features of litigation are thought to block scientific insight, his road seems to lead only to the academic bunker.