Walmart and the ASA (a guest post by Chris Winship)

Note: Chris is a professor of sociology at Harvard University and the Harvard Kennedy School of Government and, since 1995, he has edited Sociological Methods and Research, which is a peer-reviewed scholarly methodology journal. SMR content is also available on the SMRblog.

The current employment discrimination case against Walmart raises the important question of whether social science, and sociology in particular, can effectively participate in court cases and at the same time maintain its scientific integrity. If the answer is yes, there is then the further question of what criteria need to be met for scientific integrity to be maintained. These are important questions requiring discussion, even debate. But first some history.

By early fall, if not sooner, the Supreme Court will make a key decision in the largest employment discrimination suit in history: Dukes v. Walmart.  Oral arguments in the case were heard on March 29. The suit itself, involving a class of as many as 1.5 million women, alleges that Walmart has systematically discriminated against women in its salary and promotion decisions. Potentially, billions of dollars in damages are at stake. The question before the Court, however, is not whether Walmart in fact discriminated against its employees but rather whether such a large case, involving women working in varied circumstances in thousands of different stores and involving different supervisors can be thought to constitute a single class and thus whether the class should be certified.

As reported in a story in the March 27, 2011 issue of The New York Times, the American Sociological Association, as well as many other organizations, has filed an Amicus Brief in this case. (Additional briefs). The primary authors of the ASA brief are Laura Beth Nielsen and Barbara Reskin. The ASA brief is written in support of an expert report submitted for the plaintiffs by sociologist and former ASA President, William Bielby. In a story in the March Footnotes, ASA Executive Director, Sally Hillsman discusses the ASA Amicus Brief at length, pointing out that it is an important example of the type of activity ASA dues pay for. The ASA Amicus brief along with a critique written by three law school faculty members, two of whom are psychologists and all of whom are experienced empirical researchers, will be published in a forthcoming special issue of the journal I edit, Sociological Methods and Research.

A potential question that the Supreme Court may or may not choose to address in Walmart is the admissibility of Bielby’s testimony. Prior to coming to the Supreme Court, the Ninth Circuit Federal Appeals Court (San Francisco), considered by many to be the most liberal court in the country, granted class certification in a 6-5 en banc decision. The minority opinion (page 6255 and following) was quite critical of the scientific status of Bielby’s report. The majority did not comment on this.

In her Footnotes article, Hillsman states that it was critical that the ASA submit an Amicus brief since “whether social science, and sociology in particular, is authoritative and provides valid scientific evidence for helping to define a “class” in class-action cases, and for supporting the contention that social phenomena such as “corporate culture” can and do exist. The implications of this case, therefore, are significant for the discipline.”  She goes on to state: “If the Supreme Court rules in favor of Wal-Mart, the validity of social science research in legal opinions could be significantly diminished.”

The issues that Walmart raises, however, are much broader than Hillsman suggests. It is not whether social science or science more generally should play a role in court cases. Obviously they should. Rather the issue is what are appropriate and effective forms of  participation. In the case of Walmart, the question is whether Bielby’s participation is exemplary as the ASA brief argues or highly problematic.

An expert report is considered admissible if it meets the criteria determined by the Supreme Court in Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993). The basic standard that the court lays out is that expert reports must meet the same standards of scientific methodological rigor as that expected of any published scientific research. (The Federal Judicial Center’s Reference Manual on Scientific Evidence provides a fuller discussion of Daubert.

Although I am not an expert in the Walmart case, I have served as an expert for the defense in more than a half a dozen cases in which Bielby was the plaintiffs’ expert as well as cases in which Barbara Reskin and Donald Tomaskovic-Devey have been experts. In none of these cases have I offered an opinion as to whether a company has discriminated. Rather my sole responsibility has been to evaluate whether in reaching its empirical conclusions the plaintiffs’ expert report meets the standards of scientific methodology. In general, I have found these reports lacking with respect to this standard.

I note that in deposition testimony where Barbara Reskin has been an expert for plaintiffs, she has taken the position that her reports do not meet the standard of science.  For example, quoting from my report in Puffer v. Allstate where we were both experts:

           With regard to whether her opinion is the product of a scientific method, Dr. Reskin stated that her expert report “is not a scientific research product.” (p 383)  Dr. Reskin stated that role “is not that I’m doing or could do what I would see as a scientific analysis of Allstate.  My role is to use my scientific expertise and to draw on social science knowledge to try …to understand what might be going on.   I’m not drawing a scientific conclusion.  I can’t do that in this case.”  (p 77-78)  Also, Dr. Reskin expressed her view that “the scientific method is essentially precluded in a litigation context.” (p 75-76)

                     With regard to whether she would subject her opinion in this case to scrutiny by peer review, Dr. Reskin  stated that she would not hold her causation opinion expressed in Paragraph 2.5 of her expert report up in  front of colleagues and say “this is good science.”  (p 84-85)  Dr. Reskin stated that she “would not dream of  trying to publish Exhibits 2 or 3 or opinions in those reports in an article.”  Exhibits 2 and 3 are Dr. Reskin’s original expert report and her revised expert report.  (Reskin 80, 81)  Dr. Reskin stated that she “would not publish” her causation opinion in Paragraph 2.5 of her report in a refereed journal.  She explained that her causation opinion would not be accepted. (p 84-85)  Also, Dr. Reskin stated that a year from now “I might well have that opinion or I might not.”  (p 85)

(As an aside, I should note that I am well aware that both gender and race discrimination are still a serious problem in this country. In the nearly 20 cases in which I have served as an expert for the defense, my impression, which is qualified since I did no empirical research of my own in these cases, is that in a number of instances the cases had merit, others simply involved extraordinarily difficult individuals on both sides, and others still were without merit at all. I am concerned, however, that law firms, at times, seem more interested in suing companies with deep pockets than in ferreting out egregious cases of discrimination. As social scientists, I would argue that our objective should be to identify places were discrimination is a serious problem and to understand the mechanisms that generate it. More on this in a later post.)

The obvious question that Walmart raises is whether expert reports such as Bielby’s should be admitted in court. I will discuss my position on this particular question on a later post. At a more general and broad level, however, Walmart raises three questions:

1. How can social scientists appropriately participate in court cases, when it is the courts and parties to a case, not social science that controls the process?
2. In participating in a court case, how can a social scientist adjudicate between the competing logics of science, built on a notion of consensus, and our courts, which is built on an adversarial process?
3. Finally and most generally, how then can social science find an appropriate balance between its commitment to objective science and to supporting positive social change?

Appropriate Participation.  For a variety of reasons, it is difficult for a social scientist to act like a scientist in a legal case. There are at least four issues here.

1. Because of the adversarial logic of the legal process, parties to a case have the incentive to hire experts who will take the most extreme position that is supportive of their side. If an expert is unwilling to do this for a client, the client can simply hire an expert who will.
2. Court proceedings are often confidential. The vast majority of class action employment discrimination cases never make it to trial, but rather are settled often with a stipulation that all documents be sealed. In the past, Bill Bielby indicated to me that in the 60 plus cases he has been involved in, in only one instance has he testified at trial. Because cases are often sealed, an expert’s scientific colleagues may be totally unaware of the position they are taking. Bielby should be commended for publishing his expert report in Walmart.
3. Judges are given considerable discretion to decide whether an expert’s opinion meets the standards of Daubert, i.e., essentially whether it is consistent with the standards of scientific methodology. As Bill Bielby has pointed out to me in an email, this can result in wildly different criteria for acceptance. On the one hand, a court may exclude an expert entirely because of problems with one particular data source, say an employee survey. On the other hand, a court may accept a report that provides no description of the methodology used, much less any justification for it. In general, an expert has no way of knowing what specific requirements a judge will have for the admissibility of an expert’s report. Given this indeterminacy, it is very difficult to know what is appropriate.
4. Because of the legal context, experts typically cannot conduct the types of research that they might otherwise. Companies, for obvious reasons, are not going to allow plaintiffs’ experts to carry out direct research on their firm by say doing employee surveys or participation observation. Even if they did, given that the company was being sued, the validity of any information collected would be questionable.

The above four dimensions create a context which is vastly different in which science is typically done. The parties to a case decide which scientists will be heard, not one’s scientific colleagues as say in journal publication. There is often no opportunity for expert’s views to be know much less criticized in the larger scientific community. The criteria by which a judge will assess any given report is basically unknown. Finally, an expert often cannot collect the type of information needed to formulate a well supported opinion.

Consensual versus Adversarial Logics. Science is by construction a community project in which advancement depends on its members coming to consensus about various theories and facts. As part of this process science requires a practitioner to present evidence that both supports and undermines a position and to carefully weigh the support the evidence provides. In general, “truth” is considered provisional, to potentially be replaced by new “truths” with the acquisition of new data and the development of new and more adequate theories.

Law proceeds on an entirely different logic. It is adversarial, not consensual. Each side is expected to make the strongest possible case for their position without regard to conflicting evidence. It is then up to a jury or judge to then definitely decide, subject to possible appeal, guilt or innocence.

The logics of science and the law are thus in deep tension with each other. As I will discuss in greater depth in a future post, this raises the question of whether experts should play by the rules of science or the adversarial rules found in the law. Daubert is quite clear that expert’s should play by the rules of sciences. However, the de facto practice is that experts generally pursue an adversarial strategy stating their conclusions in the strongest terms and making little if any attempt to qualify them much less to consider inconsistent evidence or alternative explanations.

Objectivity and Progress. Many scientists, and certainly many sociologists, have chosen the careers they have in the belief that scientific knowledge can contribute in critical ways to the making of a better world. This is certainly true in my case. The world, however, seldom changes as a function of science alone. Politics is also often involved.

For science to be persuasive or “authoritative” in Hillsman’s words, the public must believe that its conclusions have been reached by objective means and that they are unlikely to change with time. Otherwise science is simply just one more opinion in the crowded square of public discourse.

Science is faced with three formidable challenges in this context:

1. The adversarial nature of the court system makes it highly unlikely that scientific experts will be seen as objective. Rather, the public is likely to believe, often rightly so, that experts have been hired because of the positions they take, not because of the quality of their science.
2. Academics, social scientists in particular, especially sociologists, are much more likely to have liberal political attitudes. (To my mind, Neil Gross is doing the best work documenting this currently.) As such, it can be exceedingly difficult to convince the public that scientific knowledge is driven by objective research as opposed to political motivations. The controversy over climate change is a most obvious example.
3. Scientific knowledge often changes. In the past, social science has often taken positions it claims as authoritative that have not stood the test of time. As Bill Wilson discusses at length in the first chapter of The Truly Disadvantaged, social science did society no favor when it attacked the Moynihan report (1965) and fully supported the position that a 30% out of wedlock birthrate in the African-American community (nearly 70% by Wilson’s 1987 writing) was unproblematic. Similarly, I have written about how the widely held position in the 1970’s and ‘80’s that mental ability was unimportant to social and economic success resulted in social scientists being ill prepared to address the overblown claims made in The Bell Curve, despite the importance of the issues the book is dealing with.

Society is as uninterested in one-handed sociologists as it is in one-handed economists (As economists are known to say “on the one hand, this seems to be true, but on the other hand perhaps…”). But qualification and tempered assessment are at the core of science. Unfortunately, the courts, as currently structured, undermine this central value in science as well. I would be the first to argue that social science research should play a role in court cases and in public policy making more generally.  In a society where there are value conflicts as divisive as ours, it is critical that there be institutional structures such as science that allow for collective decisions to be informed by fact and not simply swayed by opinion alone. The key institutional design question is how to do this appropriately and effectively.