What is at stake for Sociology in Walmart?

Much has been discussed about the Walmart case and ASA Amicus Brief in the postings and comments on the orgtheory [with subsequent posts 1, 2] and scatterplot blogs. Little, however, has been said about the literature review in the ASA Amicus Brief, though it spans a little more than half the main body of the Brief. Some have even suggested that the only thing the Brief does is take the position that the methods that Bill uses are those of science and sociology in particular. Clearly it does much more. [In providing the analysis below, I want to be quite clear that I am not making any claims about what people’s motives were in writing and submitting the ASA Brief.  Laura Beth has been quite clear about hers and I believe her.]

Any reading of the Brief indicates that in carrying out a literature review, the Brief lays out a theory of how gender discrimination can occur in organizations. It provides cites to various scholarly works, but as far as I can determine there is no discussion of, or cites to, any research that is inconsistent with the theory (though it does cite to papers that are critical to the use of social framework analysis). This leaves the reader with the impression that all aspects of the theory are settled and accepted by the social sciences and sociology in particular.

The theory that the ASA Brief lays out is critical to Dukes v. Walmart in that it broadly overlaps with the theory in Bill’s expert report. As such, it provides support for his theory. This is a theory in various versions that Bill has propounded in all but one of the cases in which I have opposed him. The first case was in 1995. Different variants have also been put forth by Barbara Reskin, Donald Tomaskic-Devey, Susan Fiske, Eugene Borgida among others. The theory has been used by a small number of plaintiffs’ lawyers in probably as many as 100 large class action cases as part of their petition for class certification. In particular, the Bielby/ASA theory is used as the basis for demonstrating commonality in the situation of class members, a critical requirement for class certification. (I should note that in the original case where a theory similar to Bill’s was used, Hopkins v. Price Waterhouse, where Susan Fiske was the expert, APA issued an Amicus Brief, generating much controversy.)

I am sure Bill and others would argue that his theory has been accepted by many courts (and explicitly rejected by a few). I would say that this is only sort of true. In some cases, there is a settlement before class certification. In some cases, a judge will either grant or deny class certification without referring to the Bielby/ASA theory. Of course, in some cases they do. In many cases, however, judges would prefer to be mute, and avoid creating grounds for appeal by the side that loses. The most likely situation is that the case settles and there is no trial. In general both sides find a settlement attractive in that it is far less risky than going to trial before a jury. In short, in the vast majority of cases, the “trial” takes place at the class certification case. Thus, Bill’s theory is generally being used at the critical stage in these cases.

Reasons why might Walmart win?

I can think of at least five reasons:

1. The Supreme Court makes the decision based on grounds that have nothing to do with Bielby. If you read the oral arguments a number of the justices are quite concerned about how damages could be fairly awarded across class members. I am sure that there are other non-Bielby related possibilities for how Walmart might prevail.

2. The Court could decide Bill’s report does not meet the methodological standards of good science, in particular sociology. Note that historically the courts have deferred to particular disciplines to define what their field’s methodological standards are. In this case, all that is being said is that sociology has higher methodological standards than Bill has employed. No disrespect to sociology.

3. The Court decides that Bill’s theory is logically incoherent and rejects it. As Ezra has pointed out if one reads the oral arguments this seems to be a concern for several of the justices. If the court rejects Bill’s theory then they may well deny class certification as a result. Ezra has said that he thinks the theory is coherent, but is very subtle and with unclear scope such that it is difficult to know when it applies empirically.

4. The Court might decide that Bill’s report fails to show that his theory is likely to be applicable to Walmart with a sufficient degree of certainty, or similarly that he can’t estimate the degree to which the processes stipulated by the theory are the reason for observed gender differences. I believe in deposition testimony Bill has indicated that he can’t specify even within a broad range how important the factors in his theory are in Walmart. Haven’t been able to find exactly what he says. The Court could deny class certification for this reason.

5. The Court decides that Bill’s theory does not support commonality.

I am not a lawyer, but I see no chance that the Court will dismiss Bielby on the argument that sociology is junk science. The lawyers out there will have to show me the precedence for this kind of finding. As such, I don’t see the worry that Walmart winning will result in sociology writ large being given a bad name. If (1) occurs, we could complain that we have been ignored. If (2) or (4) occurs then Bill and others will just have to meet a higher standard in the future. If (5) occurs, I see no consequence for sociology, but significant consequences for plaintiffs’ employment lawyers and their efforts to bring employment discrimination cases to trial. The class action suits that they file may well need to be smaller in the scope of people included and/or they will need to find a new theory of commonality. The same for (3). Now that ASA has in essence endorsed Bill’s theory in its Brief, if (3) does occur, it is a “black-eye” for the discipline deserved or not.

What is the Bill/ASA theory? I will try to summarize it briefly. Any of you can read his expert report or the Amicus Brief.

1. A strong corporate culture at Walmart induces uniformity in employment practices at Walmart.

2. Stereo-typing is automatic, and implicit, and as such much bias is unconscious.

3. Because of implicit bias, subjective decision-making leads to discrimination.

4. Because managers are allowed considerable discretion in their decision-making, subjectivity is allowed to have a major influence resulting prejudice and as a consequence discrimination.

5. That there are effective policies that organizations can use that will substantially reduce if not eliminate discrimination as a result of bias.

I don’t believe that in any of the cases where I have opposed Bill that he has argued that I mischaracterized his theory, but I could be misremembering. I am happy to have Bill or anyone else indicate where the above mischaracterizes his or the ASA theory in significant and consequential ways.  I should note that point (1) with regards to corporate culture has not been part of Bill’s theory in a number of cases.

Is Bill’s theory fully supported by social science research?

In evaluating Bill’s reports over the years, on almost every point I have been able to find published peer reviewed research by reputable scholars that is inconsistent with specific points in the theory. I will admit that I am not currently up to date on this literature. I also do not want to argue that this literature obviously undermines Bill’s theory. People, however, I believe, should be aware of it. There is also the question of why it is not discussed in either Bill’s report or the Amicus Brief.

Providing a full review the literature would take would many pages. An example can be found in my report in Puffer v. Allstate where I opposed Barbara.  To be brief, let me point to four literatures.

1. On the research on implicit bias there is a highly critical review in the 2008 Annual Review of Sociology by two psychologists, Hart Blanton and James Jaccard “Unconscious Racism: A Concept in Pursuit of a Measure.” Although this article is about racial bias, much of the research is directly relevant to gender bias. With regards to applicability of implicit bias theories to employment discrimination cases, the key issue is that vast majority of research to date has involved “stranger to stranger” interactions (seconds or minutes in length), precisely where one would expect stereotyping to be relevant. As such, it would seem quite applicable in hiring cases. Its relevance for salary and promotion cases where individuals have know each other for weeks, months, or years, has not been demonstrated, at least to my mind.

2. On the effectiveness of policies to combat racism there is a recent article by Paluck, E.L., & Green, D.P. (2009). “Prejudice reduction: What works? A review and assessment of research and practice.” Annual Review of Psychology, which argues that we know very little about what works. It argues that the effectiveness of most interventions remains largely unknown. I should note that Bill and others have argued since at least the mid-1990’s that we know what employment practices are needed to reduce, if not eliminate, employment discrimination. However, Kalev and Dobbin in their justly famously 2006 ASR article  “Best Practices or Best Guesses? Assessing the Efficacy of Corporate Affirmative Action and Diversity Policies” describe their effort as: “This is the first systematic analysis of their efficacy.”

3. Subjectivity as a mechanism that allows discrimination to occur plays a key role in Bill’s report and in the ASA brief. Below is what I believe is the most recent discussion I have written about this. It is taken from my expert report in Puffer v. Allstate where Barbara Reskin was an expert. The report is public. Note that the literature cited is from entirely outside of sociology.

As already noted, Dr. Reskin cites to no scientific evidence in asserting that subjectivity leads to disparities in gender (and racial differences) in employment outcomes.  A review article by Arvey and Murphy (1998) states (p. 163):

There is increased recognition that subjectivity does not automatically translate into rater error or bias and that ratings are most likely valid reflections of true performance and represent a low-cost mechanism for evaluating employees. The notion that performance evaluations and particularly supervisory ratings of performance are biased against racial and gender groups is simply not supported by the empirical data.

An earlier review article by Bernardin  et al.  (1995) directly examines this issue in the context of expert testimony in EEO cases. The abstract to the article summarizes the findings (p. 63):

We examined a common expert witness theme in EEO cases that rating bias in the form of ethnic, age, or gender differences in personnel decisions based on performance appraisals is moderated by criterion specificity or rating scale format.  Few studies have investigated this issue and results do not support the position that the more objective or specific criteria for assessment will result in smaller differences between groups based on age, gender or ethnic classification.

The Bernardin et al. article (1995) is a comprehensive review of the literature on performance appraisal bias.  The primary focus of Bernardin et al.  is on criterion specificity.  In addition, they provide a more general review of research on race bias in supervisors’ evaluations.  They state (p. 71-72):

The research indicates that all types of appraisal systems, which have been studied to date, are subject to rating distortion, deliberate or otherwise.  The conclusions derived from meta-analytic studies and the equivocal findings in four studies which investigated criterion specificity, and the lack of research comparing different formats for their susceptibility to bias as defined in the case law, supports the argument that positions regarding criterion specificity espoused by some expert witnesses in litigation are not supported by the research. We therefore argue that it is incorrect to assume that the specificity of the criteria in performance appraisal system will necessarily reduce or eliminate bias.

Simply, what is being said here is that the existing research literature has failed to support the claim made by expert witnesses, like Dr. Reskin, that the objectivity of an appraisal system is critical to whether biased evaluations will occur.  More generally at the end of their article they state:

We must conclude that the burden is certainly on those experts who maintain that there is some causal connection between a particular deleterious outcome for some protected class member(s) and a particular type of performance appraisal format or system.  The research does not seem to support this general theme. (p. 73)

These authors have reviewed at some length the research literature relevant to expert witness “opinions” similar to those offered by Dr. Reskin here.  They concluded that the research simply does not support the claim that the use of objective criteria reduces or eliminates race differences in evaluated performance.

Further, more recent research directly contradicts Dr. Reskin’s position.  Roth et al. (2003), in an article on racial/ethnic differences, carried out a meta-analysis of 36 studies which compared individuals’ ratings using objective or subjective measures.  This is the most extensive meta-analysis in the literature to date focusing on ethnic/racial discrimination (Black-White and Hispanic-White).  It is a direct test of the hypothesis that the use of subjective measures leads to discrimination.  They found that “objective measures are associated with very similar, if not somewhat larger, standardized ethnic group differences than subjective measures across a variety of indicators” (p. 694).   Put in other words, they found that racial minorities if anything did worse relative to majority individuals when objective, not subjective, measures were used.  This directly contradicts Dr. Reskin’s assertions about the use of subjective measures.

In another recent study, Hennessey and Bernardin (2003) also examined the consequences of using objective versus subjective evaluation measures, in this case in the context of gender.  They examined the effect of appraisal criterion specificity on the outcomes of minorities and women.  They examined over 248,000 performance appraisals for state employees in a southern state. They “found in general that African-Americans were rated lower than Whites and that greater criterion specificity made little difference in this result.”  They also “found no important difference in adverse impact against females as a function of the appraisal system using any definition of prima faciediscrimination.  Females actually fared better with the more subjective system .…” (p. 155).  These findings in peer reviewed literature provide strong evidence refuting Dr. Reskin’s claim that the use of subjective criteria leads to discrimination against women and minorities.

4. Certainly one of the most important sociologists doing work on employment in the last decade or so is Trond Petersen, Professor and former chair of Sociology at Berkeley. Trond’s research has bearing on Bill’s and the ASA’s theory. [The following is adopted from one of my expert reports.]

A number of within firm studies of Trond’s and others have shown that after controlling for human capital variables, firm and job category, race and gender effects, both for wages and promotion, are eliminated or nearly so (Tomaskovic-Devey 1993, Petersen and Morgan 1995, Barnett et al. 2000, Petersen et al. 2000, Petersen and Saporta, 2004).

An underlying premise of Bill’s and the ASA’s argument is that stereotyping and ingroup bias by managers manifest in discrimination in pay and promotion practices is widespread at workplaces because of the proclivity of people to engage in such behaviors.  If this were so, we would expect to see evidence of gender and racial disparities in pay and promotion of employees at workplaces on a regular basis when social scientists examine such data. However, these studies are evidence that stereotyping and ingroup bias may not be widespread among managers at workplaces in terms of “interior” policies and practices, i.e., wage and promotion practices.

Most persuasive because it contains a large sample of firms from four different cities is the work of Hertz, Tilly and Massagli (2001) which analyzed pairs of individuals holding the same job in 317 different firms.  They found that after controlling for human capital variables (education, experience, etc.) and the firm each pair of individuals worked in, there were no significant race or gender effects.

In a similarly large sample of firms, Tomaskovic-Devey’s (1993) examines the importance of discrimination with jobs on gender and racial inequality.  His study is based on a random sample of 795 employees in North Carolina.  In Table 6.2 he reports a regression analysis of the effects of firm and job characteristics, human capital, and gender and race on hourly earnings.  In his overall model, he finds that neither gender nor race have a significant effect on an individual’s hourly wages after controlling for job characteristics and other individual characteristics.

Of course, it would be important to know that these results held across a large, representative sample of firms. These results do, however, suggest that we and/or the lawyers may generally be looking for discrimination in the wrong places (salary and promotion). In particular, as Trond has argued in his work, it may be at the hiring stage and consequent job segregation that results that gender and race discrimination are most important. The vast majority of employment class action discrimination suits have not focused on hiring as hiring suits are difficult to manage and prosecute successfully.

Again, I am not arguing that the above research demonstrates that Bill’s or the ASA theory is wrong. I think it does suggest that the validity of the theory is not settled to nearly the degree that Bill’s report or the ASA brief suggests. As such, I see both as not considering evidence that is inconsistent with the theory that is being propounded and as such not meeting the standards required by science.

Where do we go from here?

It has been suggested in one of the ASA Council email exchanges that perhaps the ASA needs a set of guidelines when it is considering doing things like submitting an Amicus Brief. I could not agree more. Let me tell a story that occurred in one of the two departments I have been a member of. (I am being purposely vague about which, when it occurred, whether the position was for a junior or senior faculty member or whether the individual accepted the offer.)

A particular person was being considered for a position in the department. A senior faculty member (not me) strongly felt that this person’s scholarship was quite weak and that he shouldn’t be hired. The department listened to his arguments at considerable length. He was given a full hearing. The department then voted X to 1 to hire this individual. At this point the dissenting individual announced that given the Department’s decision he would work hard to recruit this individual. To my mind, that is true professional integrity. It also illustrates how dissent is appropriately handled.

(I should say that if a number of faculty had been against hiring this individual, the individual would not have been made a job offer. In the two departments I have been consensus, not unanimity or majority rule, has been the standard.)

If the process that to lead to the ASA Amicus Brief had been analogous to that in the story above I would have publicly supported the Brief as representing the position of the discipline as a whole and not raised questions. I don’t think I could have found it in myself to actually champion it, though I certainly would have publicly praised the fairness of the process.

In ending, let me reiterate that the above analysis is not meant to question the motives of anyone’s motives. I am willing,and think it is important in situations such as this, that we take people’s claims about intention as being truthful.

Although I am sure I will find myself writing more comments on Walmart and the ASA, my intention is that this be my last full post on the subject. As important as the subject is, there is other work to be done. I have found the discussion provocative and learned much from it. I appreciate all the time so many people have put into responding and the thoughtfulness of their comments.

Postscript (I can’t help myself): I have to say that I am ambivalent about the position that large observed gender differences in outcomes in and of them selves creates a prima facie case for discrimination. There are important examples where neither society nor sociology takes this position. For example, more than 90% of the people in prison are men. Is this gender discrimination? As many commentators have noted, unemployment during this last recession has quite disproportionately affected men? Is this gender discrimination? Women are now significantly more likely to go and graduate from college. Is this discrimination?

My guess is that these are going to become increasingly important questions as our society appears to be evolving into a bifurcated stratification in that at the upper portion of stratification system men are advantaged relative to women, whereas at the bottom women are advantaged relative to men. Of course this takes us to the long-standing debate about the difference between equality of opportunity and equality of outcome and all the problems with the later being used as the standard for judging whether a system is fair.

References from sections (3) and (4) above.

Arvey, R.D., and Murphy, K.R. 1998. “Performance evaluation in worksettings.” Annual Review of Psychology, 49, 141–168.

Barnett, W.P., Baron, J.A., and Stuart, T.E. 2000. “Avenues of Attainment: Occupational Demography and Organizational Careers in the California Civil Service.” The American Journal of Sociology, 106, 1, 88-144.

Bernardin, H.J., Hennessey, Jr., H.W., and Peyrefitte, J. 1995.  “Age, Racial, and Gender Bias as a Function Criterion Specificity: A Test of Expert Testimony.” Human Resources Management Review

Hennessey Jr., H.W., and Bernardin, H. J. 2003. “The Relationship between Performance Appraisal Criterion Specificity and Statistical Evidence of Discrimination.” Human Resource Management, 42, 2, 143-158.

Hertz, T., Tilly, C., and Massagli, M. 2001. “Linking the Multi-City Study’s Household and Employer Surveys to Test for Race and Gender Effects in Hiring and Wage Setting.” in A. O’Connor, C. Tilly, and L. Bobo (eds.), Urban Inequality: Evidence from Four Cities. 407-43 NY: Russell Sage

Petersen, T., and Morgan, L.A. 1995. “Separate and Unequal: Occupation-Establishment Sex Segregation and the Gender Wage Gap.” American Journal of Sociology, 101, 2, 329-365.

Petersen, T., Saporta, I., and Seidel, M.L. 2000. “Offering a Job: Meritocracy and Social Networks.” The American Journal of Sociology, 106, 3, 763-816.

Petersen, T., and Saporta, I. 2004. “The Opportunity Structure for Discrimination.” American Journal of Sociology, 109, 4, 852-901.

Roth, P., Huffcutt, A., and Bobko, P. 2003. “Ethnic Group Differences in Measures of Job Performance: A New Meta-Analysis.” Journal of Applied Psychology, 88, 4, 694-706.