Goals and a Few Answers

I spent last night reading through all the comments on orgtheory and scatterplot. My key goal in writing my initial post was to get a discussion going about the role of sociology in the courts and the particular problems involved. I guess I succeeded! My interest in the Walmart case was only secondary and I discussed it, the ASA Amicus Brief, and Bill’s expert report because it was current, was potentially important, and exemplified many of the issues that I thought needed to be discussed. I did not write it to attack the ASA as Sally Hillsman has accused me of in an email to the Council. Truthfully, I do not know enough about what was done to know whether I would believe it to be unproblematic or not. If the Council, the ASA members’ elected representatives, had the time to seriously consider the matter, read the materials involved, appreciated the issues, and voted to submit an Amicus Brief to the Supreme Court, then I think I and others should not complain. Of course the Mitchell et al. paper does attack the ASA brief, but on scientific, not procedural grounds. [I should also note that Sally’s claim that I offered Laura Beth the opportunity to publish her reply to Mitchell et al. in SMR and withdrew that offer is factually incorrect. I withdrew the offer for her to write a quite different paper, for quite defensible reasons. All that said, what will go in the SMR special issue is still evolving.]

In reading through all the comments last night I was amazed by the number times various people said I said particular things (using their words, not mine), and claimed that I thought various things (with no access that I am aware of to my mind). Amy’s post is perhaps the extreme example of this. In an actual court proceeding this may be appropriate. I don’t think it is appropriate for blogging, assuming the goal should be to try to understand each others’ thinking–why they believe what they think is reasonable–and that by hearing what each other thinks, we might improve and deepen our own thinking. Let’s not put words in people’s mouths or thoughts in their heads. If a position someone has taken is important for a point you want to make then quote the person. If you believe someone thinks a particular thing and that is why they are taking the position they do, then ask them whether that is what they think. More generally, as Laura Beth has asked, let’s keep it as diplomatic as possible. In doing so, this will vastly increase the likelihood of having a constructive dialogue.

To try to respond to all the all claims, accusations and allegations that have made would take me hours and would result in a posting that was thousand of words long. Simply responding to all of Amy’s points would result in this. I do not think this is fair. In the context of a civil dialogue as opposed to a courtroom proceeding, we shouldn’t have to be spending time refuting claims that others have made about what we have said or what we supposedly think. That said, let me respond to some of them. If people think that there are key comments I need to respond to and haven’t, let me know and I will.

1. Science in the courts. I read Amy’s post as saying that she thinks the current way in which science is used in the currents is okay. There is a vast literature, involving probably a couple of dozens books and many articles that argues otherwise. Most of this literature focuses on the use of physical and biological sciences, not the social sciences. There are two complaints: (1) that the quality of the science that is presented is poor; (2) that generally a “war of the experts” occurs, they cancel each other’s opinion out, and science makes no contribution at all. I would be happy to provide references when I am back in Cambridge. I also suspect that googling “science in the courts” or “junk science” will turn up quite a few citations. I don’t know whether it worth discussing this further.

2. No solutions? A number of people seem to believe that my position is that sociology should stay out of the courts because there is no way we can do so and maintain our scientific integrity – somehow that I am arguing that an “objective” position is to stay on the sidelines. This is not my position at all. I was asking us to think about different institutional designs that would improve the situation. Let me suggest a few:

a. A few judges are requiring that expert reports be posted on the web. This is what happened in Puffer v. Allstate. I think this is excellent. If experts know that all the world will have the opportunity to see what they have argued, they are likely to be much more restrained in the claims they make. I could imagine the ASA making this recommendation and perhaps saying that sociologists should only participate in cases where this is not done.

b. Judges can legally hire their own experts. The ASA might make a recommendation to the appropriate associations of judges that they do this.

c. An idea I like a lot is that a judge would require both parties to hire an agreed upon expert who would provide a written evaluation to the judge of the expert reports from each side. This could well create the same type of social dynamic as occurs in binding arbitration where each side has the incentive to be reasonable in their claims so that those claims are not dismissed out of hand.

d. The ASA could appoint a panel to study the matter and make recommendations. We are bright people. Maybe such a panel could come up with some very creative ideas.

My broader concern is that the sciences of all types contribute far too little to policy making or societal decision making more generally. I have taught courses and published on this (see 1, 2); I see this as least in part a problem of having inadequate institutions. Let’s try to figure out how the legal system and society more generally could do it better.

3. The Primacy of Academic Integrity? Related to the last section, various people seem to think I believe that unless we are absolutely sure about the science, we shouldn’t be in the courts. Nothing could be further from the truth. I do think we need to be forthright, but conservative in claiming what we know. I believe this for two reasons: (1) as I discussed in my original post, whether it is the Moynihan report or claims about the lack of importance of mental ability to success, we have taken strong positions in the past that have proven to be wrong; (2) we now have several decades of psychological research showing that people are subject to important cognitive biases with one class of those biases being that people are often certain that they right to a far greater extent than is warranted. (Citations if requested when I am back in Cambridge. One book I like a lot is On Being Certain by Robert Burton, a neuro-scientist who argues that certainty is something that we experience as an emotion as opposed to being a logical property of our thinking.)  I see no reason not to believe that we are subject to the same biases. I want sociology to be cautious in what it claims so that when we are certain about something and the stakes are high, we will be believed. I don’t think either our track record or public reputation is what it should be. To be effective, one has to choose one’s battles. (In saying this, I am not making any claim about whether the ASA should have submitted an Amicus Brief.)

4. Daubert. Laura Beth has argued that all that the Amicus Brief argues is that the methods Bielby uses are the methods used by social science. As I will suggest in my next post, in providing the literature review that it does, it does much more. However, what also seems implicit in this position is that if one uses the methods of science then one has met the standard for admissibility under Daubert. I can’t see how Daubert can be understood this way. It wants methods to be used reliably. To assume this is the case is to say if an expert uses regression, but not appropriately, then they have met the Daubert standard even though they have not met the scientific standard of appropriate use. A quote from Washington Legal Foundation from a comment by Ezra:

“expert opinion is not admissible simply because it is based on reliable principles and methodology. The application of that methodology to the facts of the case must be reliable. See Fed. R. Evid. 702(2)-(3). For this reason, whether an expert’s opinions were deemed admissible in one case has little bearing on whether such testimony should be admitted in a subsequent case where the expert seeks to apply his methodology in a new, untested way. Although the expert in Price Waterhouse v. Hopkins used a methodology similar to Dr. Bielby’s, she did not seek to apply her opinions to thousands of employees spread across the country in a nationwide class.”

I would be interested to hear from the lawyers about the argument for the lower standard.

5. Flaws in Bielby’s Report. My methodological evaluations of a plaintiff’s expert report typically run anywhere from 15 to 70 pages. I will not do that here. What are the methodological standards I believe Bill’s report should have met to meet the standards of science? To be brief:

a. A detailed enough description of the methodology he used so that someone who had access to the same documents and used his methods could determine how he arrived at the conclusions he did. Saying that someone else could analyze the documents is insufficient. If they and Bill arrive at different conclusions, we will be at a loss to know why. Part of science is being clear about what you did. Econ journals now require authors to both post the data and code they used in producing their statistical analyses. To know whether someone has done something right one needs to be able to “examine the code”, i.e. how they reached the conclusions they did.

b. As both Ezra and I have pointed out there are no caveats in Bill’s report. Science is all about stating the confidence one should have in one’s conclusions. This is missing from Bill’s report.

c. One might argue that it was the other side’s responsibility, but in general in science we require researchers to consider other explanations for empirical results and why they are not as plausible or more plausible than the explanation that one is offering. Bill’s report does not do this.

d. I see no evidence that Bill examined the defense’s statistical experts report. Perhaps he did and just didn’t say so. Science requires us to look at all the relevant evidence.

6. There are many more points that I probably should address. I will address some of them in my next post. I would like to finish by making an observation with regards to something Amy says. To quote:

“In Wal-Mart, the question is whether the expert had information that he believed would contradict his conclusion (that Wal-Mart’s employment practices may be discriminatory because they have been linked to discrimination) but chose not to consider it.  That would be unscientific.  Professor Winship has not made this claim (yet!).”

I am not an expert in Walmart and have not examined the documentary evidence in the case. In the nearly twenty cases in which I have critiqued expert reports, I can think of not a single one in which a plaintiff’s expert did not ignore important empirical evidence that was inconsistent with his conclusions. Many of my reports have spent 10-20 pages documenting all the evidence the plaintiff’s expert has ignored. Unfortunately, the courts consider most of these reports confidential. Many social scientists’ expert reports do not meet Amy’s standard.

I look forward to your considered, penetrating, and diplomatic responses.


Written by gbutler

May 25, 2011 at 12:55 am

18 Responses

Subscribe to comments with RSS.

  1. I enter this conversation as a sociologist with legal training, who has been researching the role of social science in American regulatory law since my 1986 dissertation on the politics of economics at the National Labor Relations Board in the 1930s (Stryker ASR 1989, Stryker AJS 1990, Stryker, Politics & Society 1990). I will do my best to be “considered, penetrating and diplomatic” as Chris has asked. I also will lead with the full disclosure that I am someone who spends many hours of my social scientific life teaching, researching and publishing about law, politics and inequality (see e.g. “Half Empty, Half Full or Neither: Law, Inequality & Social Change in Capitalist Democracies.” Annual Rev. Law & Social Sci. 3: 69-97 (2007) at and Rights and Their Translation into Practice: Toward a Synthetic Framework, at

    Substantial empirical research as well as socio-legal theory instructs on how and why an individual minority or female plaintiff who files an employment discrimination lawsuit in federal court is very likely to lose. Class actions are one of a tiny number of effective tools that such employment discrimination plaintiffs currently have to help level the playing field when they take on large, financially and technically resource rich employers in court. I am very worried that the Supreme Court will indeed use the Wal-Mart litigation as a vehicle to undercut the availability of class action lawsuits precisely because, although they are a small percentage of the employment discrimination docket, they are more likely to succeed in court than would be an individual plaintiff. As well, they represent the sort of highly visible litigation threat that can, at least sometimes, stimulate broader social change for the better.

    As both sides in the current blog debate have pointed out, US courts are adversary institutions and they do not follow the “rationality” or “institutional logic” of science (Stryker ASR 1989; AJS 1994, Res. in Sociology of Organizations 2000, Law & Social Inquiry In press). For good or ill, the legal system has been the “senior partner” in the uneasy hybrid institutional logic that has been created, over time, at the intersection of American regulatory law and science, including social science.

    The debate we are having on this blog becomes relevant to legal decision-making, including the forthcoming decision by the Supreme Court, because judges tend to use everything they can to rationalize what they believe is the appropriate decision. Thus, if a majority of the current 9 members of the US Supreme Court want to undercut the class action, they can take notice of, and point to, our intra-disciplinary debates as a way to help rationalize and legitimize their decision. This would be true whether or not there was – or was not – any intention of lawyers for Wal-Mart to attack the scientific status of sociology, or any part of sociology, including case study methods.

    My in press article in Law & Social Inquiry ( examines possibilities for what I term the “co-production” of law and science generated by a politics (broadly speaking) of competing institutional logics. Such a politics tends to be generated when legal and scientific actors and institutions intersect. The article situates the current discussion of the legal term “social frameworks” and of issues of social science methods as they pertain to expert witness testimony in a much longer history and broader set of issues and questions. The article reiterates and extends my longstanding general hypothesis that, for good or ill, once scientific actors become involved in and invested in their positions in highly visible legal disputes, there is an intensified politicization of science. As well, once scientists participate systematically in adversary-based legal contention and development, contention and developments in social science shape the law, while at the same time, contention and developments in the law shape the science. The fact that Chris Winship commissioned a special issue of Sociological Methods & Research, the peer-reviewed journal that he edits, to focus on “social framework” testimony would seem consistent with my hypotheses.

    Finally, however one reads the legal record of the Wal-Mart case with respect to whether and how it does, or does not characterize sociology, my distinct impression when I originally read Adam Liptak’s recent piece about the Wal-Mart case in the New York Times was that Liptak had in fact confused legal issues pertaining to social framework testimony with issues of what courts for sociologists as good sociological science.

    While adjudicating a dispute in US case law requires the judge or jury to decide what happened in that particular case, we social scientists often are busy assessing how things usually happen. More specifically, social scientists are good at assessing general patterns of association or cause and effect in the behavioral world. It’s the difference between asking and answering the question: Are smoking and lung cancer typically related and do we know the ways the former causes the latter? Vs. “Did smoking cause this patient’s lung cancer in this particular case?

    But social scientists do more than research general patterns. As we all know, sociologists also often study particular events, organizations and social environments. Case studies and case-oriented comparative research are part of the foundation of our scientific field. When we do such work, we gather data systematically from ,for example, field observations, interviews, official and informal documents, and so on, and we refract the data through our theoretical perspectives and our accumulated knowledge of the general principles and behavioral patterns found in prior social science research. We do this to understand and explain what happened – or at least what is more or less likely to have happened — in specific events and organizational settings.

    There is considerable literature (full disclosure: I have contributed to this literature – see Stryker, Sociological Methods & Research 1996) on how to do case study research attending to issues of sampling and selection bias, and to ensure that key aspects of the scientific method, including validity and reliability, are maintained. In addition – and stimulated especially by Charles Ragin – the formal qualitative comparative analysis community has done substantial work honing techniques for classifying cases into analytic categories. I do think it important to defend the scientific status of systematic case studies and case-oriented comparative work, and to point to how such lines of work contribute to cumulating social science knowledge.


    Robin Stryker

    May 25, 2011 at 8:53 am

  2. Robin: Thanks for providing your general perspective on these issues, and relevant cites; and thanks indeed for your diplomatic tone. I wonder if perhaps you are being *too* diplomatic, however. I cannot discern from your post what your stance is on any of Chris’s points. Nor can i tell what you think of the substantive and process criticisms that have been raised. The best I can make out is that you think that we critics are playing with fire. And since you mention no words of caution to those we are criticizing, I guess we can infer that you think that those concerns ro raise relative to the risks involved. Is that your position? If so, it is hard to find it persuasive because you have not spoken to the merit of those concerns.

    BTW, it would be useful to have your reaction to the kinds of alternative institutional designs that Chris described. While in the short-term, we must cope with the system we have, it would be great to find our way to a better system.



    May 25, 2011 at 9:47 am

  3. correction of weird typo “concerns ro raise” should be “concerns are minor”



    May 25, 2011 at 9:48 am

  4. I’ve avoided this for a few days because it seemed to be devolving into name calling. As such, I appreciate Chris’s effort to get it back “on track.”

    However, checking in again, I see that the major protagonists on the “pro” ASA filling side have more than tipped their hand at this point. Initially, the “pro” argument seemed to be something like, “Walmart said something impolite about sociology so you surly see that ASA HAD to respond.” Then the “pro” argument moved to something like, “those who oppose the idea of the ASA filing Amicus Curiae in this case are guilty (at the very least) of quietism on gender discrimination.” (See, for instance, “mike” in post below: “Staying out of a debate is also a decision, one that maintains complicity with the status quo”). Finally, “olderwoman” over at SP suggests that those of us who are troubled by the filing are actively serving Walmart and the defense in similar cases. Whew! Add in the brief “Facebookgate” kerfuffle discussed over at SP and this has long passed the point of reasoned discussion and debate.

    I am aware that Chris disagrees with me on the key general issue, but just for the record: Some of us oppose the filing simply because we don’t believe that ASA should be doing such things. That is really all there is to it. Moreover, some of us who oppose the filing are actively involved – as individuals, as participants in SMOs, and as members of political parties – in the fight against gender discrimination and against the practices of Walmart. The fact that – apparently – so many fellow sociologists are incapable of holding these two ideas in their heads simultaneously is the part of this that truly reflects poorly on the field.


    Als Beruf

    May 25, 2011 at 6:23 pm

  5. Chris –

    There’s one position you’re staking out that I don’t quite understand.

    I do think we need to be forthright, but conservative in claiming what we know. I believe this for two reasons: (1) as I discussed in my original post, whether it is the Moynihan report or claims about the lack of importance of mental ability to success, we have taken strong positions in the past that have proven to be wrong; (2) we now have several decades of psychological research showing that people are subject to important cognitive biases with one class of those biases being that people are often certain that they right to a far greater extent than is warranted.

    The first claim is, “we need to be modest, because strong positions have been proven false.” But haven’t modest positions also been proven false? Indeed, aren’t most positions proven false? Perhaps I sound like a hardcore Kuhnian here — but what is the a priori justification for modest vs. strong claims, if both, in the long run and upon continued examination, almost always turn out the be false? The second point seems to simply be an extension of the first — people often think they’re right when they’re not. That applies to you and me, and everyone else, regardless of their side.

    It strikes me that the extension of these points — that positions are proven false & even though people are likely to be sure in them — suggests that scholars shouldn’t speak to court cases, as they usually end up being wrong.

    As a side-note: I’m for the push to keep things civil. And though you give lip-service to that call, Als Beruf, writing things like, “The fact that – apparently – so many fellow sociologists are incapable of holding these two ideas in their heads simultaneously is the part of this that truly reflects poorly on the field,” hardly seeks to push forward such a dialogue.



    May 25, 2011 at 7:25 pm

  6. Shaka,

    I was writing on the fly (between meetings) and I am sorry if you found what I wrote uncivil. I thought that I was pointing out that it is possible to share all of the political commitments of the “pro” crowd and still believe that it is was wholly inappropriate for ASA to act as it has in this case. To assume otherwise, as so many have done in this debate – or, even worse, to feel entitled to saddle our professional association with those political commitments, as the principles in this matter have done – is irresponsible. I do think that it reflects poorly on the quality of thinking in the field, but fail to see how that is an uncivil thought. In any event, I’ll bow out of this now. I don’t want to hijack Chris’s thoughtful post.


    Als Beruf

    May 25, 2011 at 8:19 pm

  7. Shakha, it is hard to see how you are genuinely for ‘diplomacy’ when you lace your claim with a snide remark about ‘lip-service’. Those of us who know Chris quite well know that he doesn’t play those sorts of games.


    Steve Morgan

    May 26, 2011 at 2:00 am

  8. Shakha,

    Rereading at the suggestion of someone, I see that a more plausible interpretation of your sentence is that you are referring to Als Beruf’s lip-service. I now see that more clearly in your syntax and apologize, in pursuit of diplomacy, for my hurried reading.

    Also, rereading your substantive point, I think Chris’ point is that the consequences of others relying on a strong but false claim can be substantial. Less so if the claim is modest. Then again, I am not entirely clear on what strong vs. modest claims actually are, and so I should probably butt out now.


    Steve Morgan

    May 26, 2011 at 2:26 am

  9. Robin:

    Thanks for sharing your in-press Law & Social Inquiry paper you cite above. It is must-reading for anyone who wants to understand the historical context for the issues on social scientists as expert witnesses. It also allows me to refine my question above (uner “BTW…”). In particular, in your piece, you and your coauthors discuss “five types of mismatch” between social science and the law, one of which is the mismatch “between advocacy toward predetermined ends within a purposely adversarial framework and non-adversary institutions encouraging non-predetermined ends and considering other views.” Chris’s first and second points can be read as follows (forgive me Chris, if this mischaracterizes): Yes, there is such a mismatch and this leads social scientists to make akward compromises when we serve as expert witnesses. But do we need to take the current institutions as given? Alternatives have been proposed that promise to allow science to enter the courts on terms that requre fewer compromises (see Chris’s post). What do you think of alternative institutional designs?



    May 26, 2011 at 3:05 am

  10. I am a sociologist and I have a close relationship with one of the attorneys for the plaintiffs against Walmart. I have been reluctant to get into the fray of the discussions on scatterplot and orgtheory because I have never commented on a blog, and one that has stirred significant debate is an intimidating place to start. However, since one common thread throughout most of these discussions has been people’s admitted ignorance about the details of the case, I decided for readers and commenters, it might be worth enriching this discussion with some of the evidence relied upon by the lower court when it certified the nation-wide class of women for their equal pay and promotion claims based on gender discrimination by Walmart. While I understand that this debate is centering not on these details and more about the role of the ASA in writing the amicus brief, I am also troubled by the level of misinformation about the actual case that has been implied in these discussions. I feel it is helpful to situate the ASA’s incentive to respond given the facts of this case. I asked one of the plaintiff’s attorneys if he would be willing to summarize the evidence in favor of class certification. It is important to emphasize that while we can’t know how much Bielby’s report influenced the court’s determining of the facts, the reality is that the court had an enormous amount of data to consider, and Bielby’s report is only one document of millions (yes, that is the actual order of magnitude) the trial judge waded through. Below is his very long response (note that he has not read any of the blog posts):

    “I think it is important for people to understand that the factual allegations made by Walmart in its Supreme Court brief are not supported by the record. The same is true with respect to the statements of the facts of the case made by its counsel in his oral argument at the Supreme Court. Walmart lost in all of the lower courts and the facts they allege in their brief are contrary to the findings of fact the trial court (the trial court is the U.S. District Court) made to support its certification of the class. The record on appeal is based on the findings of fact which were made by the trial court and affirmed by the Ninth Circuit Court of Appeals. The district court’s decision to certify the class is subject to review for abuse of discretion only and its findings of fact are entitled to deference unless clearly erroneous. The district court’s findings of fact were supported by substantial evidence and far from clearly erroneous.
    Below is a summary of some of the findings the District Court made to certify the class. The first statement below is a summary of the evidence relied upon by the District Court in certifying the class.

    The district court issued a detailed 84 page class certification opinion as well as a lengthy opinion addressing challenges to the parties’ expert evidence based on over 200 depositions, production of more than a million pages of documents, 120 anecdotal declarations of women who worked at Walmart, electronic personnel data of the entire workforce of Walmart from December 1998 through March 2003 (the end of discovery in the case prior to certification), the competing statistical studies and opinions of both sides’ statistical experts, the extensive briefs of both sides and the full day of oral argument from both sides.

    The trial court found:

    1.Walmart stores are operated ‘with a high degree of store-to-store uniformity” and centralized control.

    2.”The personnel structure within each store operates in a basically similar fashion…”

    3.”Each individual store is subject to oversight from the company’s Home Office” that includes “a very advanced information technology system which allows managers in the Home Office to monitor the operations in each of its retail stores on a close and constant basis.”

    4. Walmart’s unique culture “promotes and sustains uniformity of operational and personnel practices,” and guides managers in the exercise of their discretion.

    5. The court found “no genuine dispute that Walmart has carefully constructed and actively fosters a strong and distinctive, centrally controlled, corporate culture.”

    6. Walmart’s practice of “promoting from within” means that “the culture lessons learned by junior-level employees contribute to building a foundation of common understanding and practice among the management team.”

    7. The company regularly moves store-level managers across stores and districts, and thereby “ensures that a uniform Walmart Way culture operates consistently throughout all stores.”

    8. Plaintiffs offered evidence that Walmart “cultivates and maintains a strong corporate culture which includes gender stereotyping”.

    9. The court found that stores had similar jobs categories and descriptions and that most hourly workers fall within five job positions.

    10. Employees are assigned and moved among departments frequently and pay policies make no distinction by department.

    11. There are no minimum or preferred education or experience requirements for any hourly job.

    12. Regional and district managers, who are mostly male, make all promotion decisions for all store management positions.

    13. The court found that “the subjectivity in promotion decisions occurs in two fundamental ways: (a) a largely subjective selection practice hindered by only minimal objective criteria combined with (b) a failure to post a large proportion of promotional opportunities.”

    14. “Class members had no ability to apply for, or otherwise formally express their interest in openings as they arose,” and that “managers did not have to consider all interested and qualified candidates, thus further intensifying the subjective nature of the promotion process.”

    15. “All hourly employees at every Walmart store are compensated pursuant to the same general pay structure.” The Home Office establishes minimum starting rates of each hourly job in the retail stores, and while store managers are granted substantial discretion in making hourly pay decisions, any pay increase above a certain percentage is automatically reported to higher management and requires special approval.

    16. The district court found “significant evidence of gender stereotyping.”

    17. The trial court found that “roughly 65% of hourly employees are women, while roughly 33% of management employees are women.”

    18. Women hold only 14% of Store Manager positions yet disproportionately occupy 80 to 90% of the hourly supervisory positions (which are not management positions).

    19. Walmart’s proportion of women in management was compared to that of its 20 largest competitors, and 80% of Walmart’s stores had significantly fewer female managers.

    20. The district court credited Plaintiff’s proof that, after controlling for relevant factors, “a statistically significant shortfall of women [were] being promoted into each of the in-store management classifications over the entire class period.” This shortfall was “consistent in nearly every geographic region at Walmart.”

    21. Women also consistently took longer than men to advance to management positions. (average 4.4 years for women versus 2.9 years for men). These differences existed even though female employees at Walmart generally have more seniority and better performance ratings than men employees.

    22. Plaintiffs’ statistical regressions for hourly and salaried employees showed that in every one of Walmart’s 41 regions women were paid significantly less than men, and this pay gap increased each year. This pattern was consistent for all store classifications even when seniority, turnover, store, job performance, job position, part-time or full-time status, and other relevant factors are taken into account.

    23. The district court concluded that plaintiffs’ statistical analysis raises “an inference of company-wide discrimination in both pay and promotions.”

    And finally, one other note he just sent me in an email:

    With respect to the competing statistical studies of each side, the District Court’s class certification opinion devoted 18 pages to an analysis of the competing analyses and explained why plaintiff’s “statistical method best reflected the alleged discrimination.” The court determined that plaintiffs have “shown” reasons to accept their statistics and the defendant’s “assertion that its approach is necessarily superior does not withstand scrutiny.” In that regard it should be noted that the District Court struck the central justification for Walmart’s expert’s method of analyzing pay by store – a survey conducted by defense counsel – after Walmart’s expert admitted she could not vouch for its reliability. Even the scientifically flawed analysis of Walmart’s expert showed an overall adverse impact.



    May 26, 2011 at 3:07 am

  11. Jem,

    Thanks for posting this, it’s a very useful outline of the findings of facts for a rather voluminous case.

    However, I disagree with your characterization of the OT+SC threads as having “misinformation” since I don’t really see anything in your forwarded summary that contradicts the discussion (aside from a few isolated comments from people who aren’t OT regulars). That is, the summary is broadly consistent with (but much more detailed than) the understanding of the findings of facts that most of us already had. Still, it’s very good to clarify and flesh this out, so thanks.

    I’m not sure how any of that changes the discussion here though, for two reasons. First, like I said, most of us already basically understood and accepted the plaintiff’s (and the lower court’s) version of the facts, albeit in not as much detail as you’ve been good enough to provide. Second, I think it’s fair to say that the debate has not really been about the role of gender in Wal-Mart’s internal labor markets but the role of sociology and ASA in the Wal-Mart litigation. For instance, I have been mostly interested in questions like whether Wal-Mart’s brief was really a challenge to sociology’s scientific validity. Likewise, a lot of people have been interested in discussing issues of epistemology and logic.



    May 26, 2011 at 4:30 am

  12. @steve morgan: your second reading is my intent. And perhaps I was too quick to pull the trigger with Als Beruf — I read that sentence as suggesting that those who disagree aren’t that bright. Perhaps too uncharitable.

    Also, I’m with you on not know the difference between a modest and strong claims (or in Chris’ language, “conservative”). I think “conservative” claims are often subject to the same two conditions outlined, and so I wonder if even they can be made.

    I also think there’s a tension between the logic of law (adversarial and absolute) and the logic of science (negation and tempered) which is at the core of much of this debate. Such things are well beyond what I know, and so like you, I’ll be quiet from now on.


    Shamus Khan

    May 26, 2011 at 6:03 am

  13. I agree with what Gabriel wrote. And Jem: I don’t know what it means for “misinformation” to be “implied.” By definition, “misinformation” implies an active attempt to mislead. I think you mean to say that the dicussion has been based on less than complete information. (In fact, one of my main missions has been to get sociologists, in whose name the brief was written, to read for themselves what the various briefs say. Everything that I (and other critics of the brief, as I can recall– certainly Chris!) has been based on the information we have been able to get. Insofar as you have added some more, thank you.

    Meanwhile, it would be great to get this thread back to the issues that Chris raised (and Robin seemed to join)



    May 26, 2011 at 7:41 am

  14. Shamus, I don’t think this is a debate that either of us is very invested in, and I don’t know enough about law versus science, as you may be thinking about them in your own mind.

    I do know Chris, though, and we have talked a lot about warrants for conclusions in sociology. My own position, and I think it is Chris’, is that sociology (both collectively and perhaps all of us individually) overvalues strong answers to new questions (or at least questions that we can frame so that they seem new). We then collectively undervalue research that integrates and tries to convey the best possible answer that lies between the extremes that first movers have staked out. This scheme of valuation is often justified by appeal to the necessity of vibrancy in the field, but I don’t think it serves us as well as a more balanced orientation would. (I write this only as a sociologist thinking about the good and bad of sociological research. I leave it to Chris and others to discuss how sociology and the law are related and should be related.)


    Steve Morgan

    May 26, 2011 at 1:40 pm

  15. — Jem,

    Thank you for your description. I, like Gabriel, don’t quite see where the contradictions are. I also find your description which seems to imply that the case was unproblematic until it reached the SC difficult to reconcile with the 6-5 en banc (meaning all judges were involved) decision in the Ninth Circuit. This is the most liberal circuit in the country. It is where all lawyer’s filing employment discrimination cases want to file if they can and yet the plaintiff’s barely prevailed.

    A lawyer friend who has been following the discussion suggests that people ought to read the actual court opinions as opposed to relying on summaries. For example, he states:

    it is important to understand that the district court wrote the opinion indicating that it felt constrained not to resolve any disputes among the expert evidence. See, e.g., Dukes v. Wal-Mart Stores, Inc., 222 F.R.D. at 144 n.5 (“Rather than resolving the ‘battle of the experts,’ and without conclusively ruling on admissibility, the Court’s role at the class certification stage is to determine whether the expert evidence adds probative value to plaintiffs’ claims.”); Dukes v. Wal-Mart, Inc., 222 F.R.D. at 191 (“[I]t is clear to the Court that a lower Daubert standard should be employed at this [class certification] stage of the proceedings”).

    I will see if I can come up with links.

    — Als

    I respect your position that professional organizations should not be filing amicus brief’s and other actions that have a political dimension. That is not my position, at least at this time. I think when a group has specialized knowledge that can contribute to a policy or legal decision then one way for that knowledge to be transmitted may be through the relevant professional organization taking a stand. Of course, if there are other institutional structures that can do this effectively, this may not be necessary. There are obviously a host of practical risks as has been suggested in previous comments and in comments over on Scatterplot. As such, it may well be the case that the risks outweigh any benefits and thus for purely practical reasons professional organizations should avoid such activities. I am just not against them in principle as long as the vast majority of the membership supports the position and due process has been followed.

    The question of “blame” is of course a normative issue. In a multicausal situation where changes in any number of factors would change the outcome, who is to blame is a question of who has done something that they shouldn’t have. Not too surprisingly, I find the idea that people shouldn’t raise questions about the ASA Amicus Brief because it might assist Walmart highly troubling. Perhaps if a different brief had been written or process followed, there might be little to no disagreement now. Who is to blame?

    All that said, I seriously doubt the Supreme Court’s decision in Walmart would be affected by either the ASA submitting a brief or not, submitting a different brief, or whether or not there was public disagreement in the discipline about the brief. As Robin notes, the Court is likely to determine the decision they want to make and then look for a way to justify their decision. One of my grad students recently gave me a very nice article (LoPucki, “Legal Culture, Legal Strategy, and the Law in Lawyer’s Heads” Northwestern Law Review Summer 1996) that makes this argument more generally about judges. If this is correct, then there is no blame to assign as none of our actions will have had a causal effect on the outcome in Walmart.

    — Shakha,

    You will have to explain your claim that “most positions are proven false.” I don’t understand:

    1a. Single motherhood is an important contributor to poverty in the US.
    1b. It is not important.

    2a. Currently, intelligence as measured by IQ tests is an important factor in determining an individuals economic success in the US.
    2b. It is not important.

    3a. The World is getting warmer over the long run.
    3b. It is not.

    In each of these cases, assuming that we pin down exactly what the claim is either (a) or (b) is right. In what sense can they both be false.

    I see sciences, and certainly sociology, as having privileged knowledge about how the world works. This knowledge should be shared with the broader world with the goal of improving it. When I propose a modest stance above, or in other words saying that epistemological modesty is a virtue, all I am saying is that we should careful not to be overconfident in claiming what we know. Psychologists have long talked about the fundamental attribution error – the tendency to see agency in outcomes for others as opposed to the importance of context and the reverse for our own outcomes. It now, however, turns out that the fundamental attribution error does seem to apply in Asian cultures. The December 13, 2010 issue of The New Yorker has a scary but important article called “The Truth Wears Off” that discusses multiple cases where science thought it knew something with considerable confidence, as time went on these “truths” no longer were supported by research. Good read.

    Now if you mean by “position” a theory, I would agree that all theories are false in that they are only approximately correct in describing how the world works. But we can certainly argue about good an approximation they are within a particular context. Or if we want to talk about it in the way a pragmatist would, we can talk about how “useful” a theory is. Thus the theory of Newtonian mechanics is wrong, but in most real world day to day situations in which a typical person finds themselves in, it does a good job of explaining how the world works. Say more. It might lead to an interesting discussion.

    One person has emailed me about my decision not to publish Laura Beth’s reply to Mitchell et al. in SMR. Figuring out how to manage the special issue has been an evolving and challenging process. What Laura Beth, Greg Mitchell and I have agreed is to have two special issues, the first focusing on the methodological issues in Walmart and the second focusing on the methodological issues involved in making scientific inferences from general science knowledge to specific cases. Probably neither of them will participate in the second issue. As of this point, the first issue will have the following structure:

    1. Introduction by Winship
    2. ASA Amicus Brief
    3. Mitchell et al. critique
    4. Short comments (5-7 pages) from four individuals
    5. Response from Laura Beth
    6. Response from Mitchell

    Laura Beth and I will pick two of individuals together. Greg and I will pick two together.

    If you have suggestions for commenters, please email me (

    As the process around the ASA Amicus Brief demonstrates, trying to figure out how to do things fairly and ethically is not always easy. I am pleased that Greg, Laura Beth, and I have figured out a way to move forward. (I should say that my history with Laura Beth is that she proposes something that I initially find totally unreasonable and then after thinking about it, I decide is a good idea.)


    Chris Winship

    May 26, 2011 at 2:33 pm

  16. […] Orgtheory and Scatterplot have lively and productive discussions on the Amicus brief filed by the ASA in the Wal-Mart case. Discussions include the nature of causation, science and law, and politics at the ASA. GA_googleAddAttr("AdOpt", "1"); GA_googleAddAttr("Origin", "other"); GA_googleAddAttr("theme_bg", "ffffff"); GA_googleAddAttr("theme_text", "000000"); GA_googleAddAttr("theme_link", "0060ff"); GA_googleAddAttr("LangId", "1"); GA_googleAddAttr("Autotag", "science"); GA_googleAddAttr("Autotag", "books"); GA_googleAddAttr("Autotag", "technology"); GA_googleFillSlot("wpcom_below_post"); This entry was posted in Uncategorized. Bookmark the permalink. ← Get’em while they’re hot! LikeBe the first to like this post. […]


  17. […] case and ASA Amicus Brief in the postings and comments on the orgtheory [with subsequent posts 1, 2] and scatterplot blogs. Little, however, has been said about the literature review in the ASA […]


  18. […] has sided with Wal-Mart in the class action case. As regular readers of this blog will be well aware, sociologists have been more than usually involved in the case and the debate surrounding it. The […]


Comments are closed.

%d bloggers like this: