regnerus follow up
Think Progress has been digging further into the back story behind the Regnerus/gay parents paper. The news site got one of the study’s funders to admit that the conclusion was predetermined:
Tellez confirmed to The American Independent that he was referring to same-sex marriage cases. In April 2011 — a year before the study was complete — Tellez wrote in a letter that “we are confident that the traditional understanding of marriage will be vindicated by this study as long as it is done honestly and well.” He also suggested that no prior study had properly compared children raised by a mother and father and those “headed by gay and lesbian couples, but of course the Regnerus study doesn’t even do that.
The study was submitted for publication in February 2012 before Regnerus had even completed all of the data collection and accepted just six weeks later, while many other articles published in the same issue took a year between submission and acceptance. Peer review was similarly hurried, with one social demographer admitting that he only had two weeks to review the study and offer a commentary — without even having access to all the data.
Previous Regnerus discussion on orgtheory.
Adverts: From Black Power/Grad Skool Rulz
orgtheory.net 
The reason why many believe this could be a witch-hunt is possibly because we all know that some null findings or findings that are associated with certain segments of the political spectrum are simply unpublishable regardless of the methodology. As scholars, some supported Regnerus simply because “Between a high, solid wall and an egg that breaks against it, I will always stand on the side of the egg.” It is not about whether the study is done correctly or not; rather it is about the way this case was pursued. Is there any paper out there that reaches the opposite conclusion of Regnerus’s and also gets published like that? Anti-discrimination is righteous in and of itself, but shouldn’t we have the basic etiquette to allow others to have different views? Think about a world that is either all for-something or against-something. I don’t see any difference between the two. It’s all hegemony.
LikeLike
john
March 30, 2013 at 4:11 am
A witch hunt implies that the accused did, in fact, not engage in sociology as sorcery. Unfortunately, the methods of the paper, attenuated review process, and sponsoring group call all that into question.
I’m looking forward to having a newer ethical controversy than the Tea Room Trade to teach in my methods course. Ironic that they both touch on the same issue.
LikeLike
cwalken
March 30, 2013 at 6:46 am
Then the issue may boil down to what counts as sorcery? If I am to single out any article out there, I am pretty much sure I can find mistakes or errors under magnifiers, and the issue at question becomes what is the intention of those errors. Since we all have some preconceived notions/political orientations, it is probably hard to claim that there is ABSOLUTELY no bias involved in their scientific studies. A tough job for all of us is to balance such bias with our conscience and scientific neutrality. Sometimes, I am not sure whether I would submit a paper with findings that completely go against my hunches and/or the consensus in my field. My point is what would happen if regnerus’s paper finds otherwise. That’s probably the starting point of this debate. We are establishing a precedent, and to some extent, a bad precedent, I believe.
LikeLike
john
March 30, 2013 at 12:40 pm
Oh sure! Just let science journal editors subvert the peer review process and knowingly publish researchers lies! Why not?
LikeLike
Scott Rose
March 30, 2013 at 3:30 pm
There are three kinds of research in this area. 1. Good research which finds no difference between children of gay parents and children of straight parents. 2. Bad research that finds no difference between children of gay parents and the children of straight parents. 3. Bad research that finds children of gay parents do worse.
LikeLike
Philip N. Cohen
March 30, 2013 at 4:00 pm
Amusing, Philip, how you assume from the outset what the conclusion of any “good” study should be. I am hesitant to side with John, but you give reasonable people incentive to do so.
LikeLike
Tony
March 30, 2013 at 6:06 pm
@Scott Rose
I can’t agree with you more. But let’s not only single out that paper, but also every paper on gay issues that gets published in major sociology (and related fields) journals during the past decade (or three decades) and have a thorough audit. At the same time, let’s make the data (and codes) from those published work publicly available and have open-source audit. How about that?
LikeLike
john
March 30, 2013 at 8:24 pm
Tony: you misunderstand. “Good” and “bad” in my comment refer to the quality of the research, not my preference for the outcome. This is really all detailed in the ASA brief – the closest thing we have to a consensus view of experts.
LikeLike
Philip N. Cohen
March 30, 2013 at 8:33 pm
It looks stupid to have many addenda on a blog, but I want to ask whether it is true that unlike the common practice in natural sciences, some null findings or non-consensual findings get dropped at the very beginning of our publish-or-perish livelihood, especially in social sciences. That is a bad practice, I believe.
LikeLike
john
March 30, 2013 at 8:40 pm
John: my understanding is that null findings don’t make for sexy careers in hard science either, but the publication rate of them may be demonstrably higher — are they?
I don’t think digging to support one’s priors with positive results is altogether bad methodology, actually. The positivist program asks researchers to self-regulate. It’s a high-minded ethic, and surely works on some margin.
But a much better solution to the incentive-incompatibility problem in social science is academic competition — that is — the traditional of rhetorical argument in the liberal arts. I’d much prefer a world where researchers were actually transparent about their biases and priors, and desire to find positive results, where there was open and vigorous debate to check and balance them — than I would this posturing dance where researchers do elaborate dances to posture as if they’re disinterested hypothesis testers.
My point expands to the group as well — group level confirmation bias is not scientifically toxic — in fact it forces the dissenting opinion (me, for instance), to amass an enormous amount of evidence to support new claims.
LikeLike
Graham Peterson
March 30, 2013 at 9:06 pm
Oops — just to be clear — the above is a general opinion on methodology of mine, not a veiled defense of Regnarus’ paper. I haven’t read it.
One day in a graduate lecture a student said something incoherent about how “these capitalists are against gay marriage, you know,” to which D. McCloskey replied: “Against gay marriage? Hell — I’m taking applications.”
LikeLike
Graham Peterson
March 30, 2013 at 9:15 pm
I think the funder’s comment is pretty clearly not saying “we have deliberately rigged this and soon our conspiracy will come to fruition” but rather “I am confident my ideological priors will be validated by the data.” I give this charitable interpretaton even though I am fully aware that the Regnerus study is often mischaracterized by the anti-SSM camp (including its funders) as “gay households are bad for kids” when it really only showed that (closeted) gay attempts at heterosexual unions -> subsequent household disruption -> bad outcomes.
Basically, I’m always willing to believe in motivated reasoning but I’m skeptical of outright bad faith.
LikeLike
gabrielrossman
March 30, 2013 at 9:46 pm
For those who are wondering if this is a case of PC policing or political hegemony, I encourage them to take the time to really read the Regnerus piece. Pay particular attention to how he disguises a family instability effect as a “gay parent” effect (by using the IBF reference group, instead of comparing to all non-“gay” families). This isn’t about differences of opinion, its outright deception.
LikeLike
Outright Deception is not Legitimate Social Science
March 30, 2013 at 9:55 pm
I had the exact same interpretation, Gabriel.
While I’m on board with calling the study shoddy and questioning the motives of a number of the key players, I didn’t interpret the Tellez quote above as evidence that he in any way suggested that the study should be finessed to make that claim. Instead, it seems like he was 100% confident given his worldview that the results couldn’t possibly suggest anything different. I might not have had as charitable an interpretation if I hadn’t seen Maggie Gallagher at a campus event a while back. These folks are certain that same-sex marriage is bad for kids and just want the liberally biased scientists doing crappy work (their characterization, not my own) to get out of their way so they can prove it.
LikeLike
jessica
March 30, 2013 at 9:58 pm
A really interesting contrast, I think, is the way that economists reacted to the Regnerus event, and then a couple few months later to their own little doosie — the Ashraf and Galor genetic diversity paper.
Apparently when economists use awful methods to make bold claims that could potentially legitimate active genocide in the pursuit of economic growth — it’s just good ‘ol positivists following the data where they take them.
When sociologists use awful methods to make bold claims that could potentially legitimate (er continue) active legal discrimination against gays, and then duly self-correct amongst their own by calling out said poor methods, they’re just a bunch of lefty ideologues.
Thankfully, what came out of both of these debates in both fields, was the settling of the argument based on the quality of the data for asserting X conclusion. As long as these arguments flourish, science works.
LikeLike
Graham Peterson
March 30, 2013 at 10:00 pm
Maybe pointless to chime in this late, but in response to “john” re null findings, null findings (i.e. no effect, no difference) are exactly what the “pro-gay” side is arguing: Once you control for family stability and economic standing, it does not matter whether the parents’ sexes are the same or different.
I agree with Gabriel and Jessica that I interpret the quotation as being sure you are right, not that you will rig the data.
LikeLike
olderwoman
March 30, 2013 at 10:13 pm
Note that, despite the questions about the funders’ motives, the resulting data set seems to be perfectly fine social science data. Its the analysis in the SSR paper that was intentionally misleading.
LikeLike
Outright Deception is not Legitimate Social Science
March 30, 2013 at 10:23 pm
Recent email reports show that plenty of lies and cover-ups led to the eventual publication of this article. Perhaps there are alternative readings. But in accordance with Phil Cohen’s detailed post of email communication prior to the completion of the study, it doesn’t look good for Regnerus or Wilcox. For anyone who has not yet read the post, it’s definitely worth reading: http://familyinequality.wordpress.com/2013/03/11/more-managerial-than-intellectual/
LikeLike
Coqui21
March 30, 2013 at 11:32 pm
Professional Ethics Dilemma: I feel like I shouldn’t review for SSR again while Wright controls it. I don’t want to legitimize or support a sketchy journal unless there’s convincing, serious reform (the audit by Sherkat was a good step, but the leadership and structure of the journal remains unchanged). But are such silent boycotts a useful penalty for malfeasance, or counter-productive for the discipline?
LikeLike
Dilemma
March 31, 2013 at 12:12 am
I guess you could apply that same charitable (naive) view to the decision to hold a meeting with Focus on the Family to plan the publicity rollout before the data was even collected.
LikeLike
Philip N. Cohen
March 31, 2013 at 12:42 am
Dilemma: I have decided not to review for, or submit to, the journal until the editorship changes as well.
LikeLike
Philip N. Cohen
March 31, 2013 at 12:44 am
I don’t understand the “boycott SSR” reaction. If you believe Sherkat’s report, Wright listened to his reviewers– their enthusiasm for the paper, their disclosures of prior relationships (nearly inevitable, in the small world of family structure-child outcome research), and, where there was a prior relationship, their assurances that they could give an unbiased assessment. As I understand it, Wright was putting together a special issue on a topic, and submissions to special issues often don’t get reviewed on the same timeline as regular submissions.
Yes, in this case there were flaws in the paper, and maybe because of these flaws it shouldn’t have been published even though all the reviewers thought it was above the bar. But science advances when work gets out into the scientific community, its flaws get discussed, and better scholarship emerges in reaction to those flaws. Science does not advance when research gets buried in the review process and the only parties to the debate are the author and reviewers, or when an editor ignores reviewers and rejects a paper because it doesn’t conform to his or her idiosyncratic tastes or politics.
I’d much rather that our journals err on the part of publishing the occasional paper that is so flawed that it shouldn’t be published than err on the side of rejecting good papers (after 18 months) because some reviewer thinks there weren’t enough stars after the regression coefficients; or that some omitted variable out there might reduce the number of stars after the key coefficient; or there isn’t enough “theory”; or it only speaks to one arbitrarily defined subfield in sociology rather than two.
Wright has, over the years, done an excellent job with SSR, following something closer to the first model than the second, heavy-handed editorial model. This time he got burned — apparently because he trusted his reviewers — but that’s no reason to boycott SSR unless you privilege your own politics over the scientific process. And that would be, you know, ironic.
LikeLike
krippendorf
March 31, 2013 at 7:29 pm
I’m not organizing a boycott. I decline request to review all the time, and I have no paper that must be published in SSR, a general interest journal. I am disappointed that Wright’s response to the audit and letters included no policy or practice reforms to help avoid this in the future. However, this is not an association journal, with an accountability to an editorial board or an association membership; it is a commercial enterprise run by a for-profit company. If it were an ASA or section journal I would vote for publication committee candidates who advocated reform. In the absence of that, I’m simply applying my efforts elsewhere. I could have kept this decision to myself, but Dilemma asked so I offered my decision.
LikeLike
Philip N. Cohen
March 31, 2013 at 10:11 pm
To start with, I am against any form of discrimination. But I don’t like discourse hegemony and the deterrence strategy that some used in this case. I think it is to set up a precedent such that no one can ever publish a paper that goes against consensual/dominant views. Basically, I don’t like dog fights in academia. Rather, we prevail with reason and etiquette. If one is to be really convinced, then please audit every single article published in major sociology journals on this topic about their methodological limitations, funding sources, time from submission to acceptance, how the review is done, reviewers’ political orientation, etc. in the past decade, and have a summary published by a ASA brief. Until then, I am not fully convinced. By the way, I am irrelevant. So let’s bring down Regnerus. Right, we will see if he or anyone else dares to ever publish things like that.
LikeLike
john
March 31, 2013 at 11:01 pm
We should indeed audit every single article that so blatantly engages in deception. Remember that no audit was needed to find malfeasance here: a decently trained 1st year graduate student should be able to spot the purposive methodology flaws in a single reading of the article. The topic, and its aim of influencing the supreme court, is what prevented the discipline from just ignoring a bad article.
LikeLike
Outright Deception is not Legitimate Social Science
April 1, 2013 at 12:27 am
Also note that Regnerus could have done the same thing to any minority group. What if he had compared all Black parents to only Intact White Families, to argue that Black parents are bad for children, at the behest of a group arguing in court for legal discrimination against Black families? Same methodological deception, just a different category of people to unfairly disparage. Would people defend Regnerus if he was engaging in racism, and not homophobia? Would others still be willing to assume no bad faith?
LikeLike
Outright Deception is not Legitimate Social Science
April 1, 2013 at 12:38 am
@Outright: To turn this into a productive instead of dogfight “enterprise,” I’d like to call for an audit of all articles on this topic published in top generalist and specialty journals in sociology, and form an ad-hoc ASA committee that consists of members from all political spectrum and write up a report. How is that?
By the way, I don’t believe first-year graduate students would be able to do the audit except probably the best ones in the best programs; sometimes, not even well-respected profs or renowned scholars if they don’t know the stuff, unfortunately.
LikeLike
john
April 1, 2013 at 12:49 am
@Outright: Actually sometimes I find it quite mind-boggling that what if people do find differences, differences that do exist between groups after controlling for everything major for any kind of group comparison. Statistically and probabilistically, it is more likely to occur than not if we deal with statistical data long enough and if we buy into the current methodological practices. What are we supposed to do as social scientists? This is a real question, not a rhetorical one.
LikeLike
john
April 1, 2013 at 12:59 am
@john: even undergraduates can see what is wrong with his “Gay families” vs “Totally stable hetero families” comparison, and how this trick could be used to make any group look bad (even white heteros). That’s not statistics, just basic logic. That he could have easily made a “Gay” vs “straight” family comparison from his data, without conflating straight with stability, is what makes his article despicable, not the behind the scenes business. While I encourage ASA to shine as much light as possible on this, I wonder if you would likewise encourage ASA to form an ad hoc panel investigating the inferiority of racial minorities based on a similar deception, perhaps with a balance of racist and anti-racist sociologists on it? Why should we be more lenient on rigged research to disparage gay families than ethnic minority families?
LikeLike
Outright Deception is not Legitimate Social Science
April 1, 2013 at 1:21 am
@Outright: so anti-racist sociologists will never publish any works in which they do find difference (hypothetically) say the inferiority (of course there are many different ways to paraphrase this word, but let me use yours) of whites/minorities because that’s bad to even think about it, let alone publish.
LikeLike
john
April 1, 2013 at 1:29 am
@Outright: we can stop doing research on group differences, because if we find any difference, there got to be some unobserved heterogeneity/confounding factors and the default, no difference.
LikeLike
john
April 1, 2013 at 1:33 am
@Outright: I was not saying first-years graduates were not able to read some part of that paper and identify potential problems. But for a paper that involves intermediate to advanced (for sociology major of course) level , I am not sure if the editor would feel confident allowing an undergraduate to spot any mistake. We all know something about almost every subject if we’ve gone through formal education. But I dare not say I can review papers in areas outside some social science areas or even out of my specialty areas. I don’t buy that. Again, I am not relevant. But one is allowed to disagree, right?
LikeLike
john
April 1, 2013 at 1:46 am
@john: this is an insultingly absurd argument: if people get mad at someone who uses deceptive tricks to make a minority group look bad, then they are saying there should be no research on group differences? Obviously, social scientists should study group differences, but its just as obvious that we have a duty not to purposely deceive the public about those differences, particularly not to disparage minorities with false comparisons.
I believe this discussion has stopped being productive.
LikeLike
Outright Deception is not Legitimate Social Science
April 1, 2013 at 1:52 am
@john: but do take the time to read the Regnerus article. No specialty required, the bogus comparison is obvious. He did at least give us a valuable tool for showing our undergrads what deceptive research looks like.
LikeLike
Outright Deception is not Legitimate Social Science
April 1, 2013 at 1:55 am
@john, with all due respect, your argument is absurd. For the millionth time, the problem with the Regnerus article is not the valence of the conclusion, but the carefully, deceptively flawed science that is used to justify that conclusion. No audit was necessary for that–Regnerus and company explicitly worked to put the research in the public sphere and to argue based on it, thereby inviting substantive rebuttal. He got the rebuttal, in spades, to the point that the study is now completely and utterly discredited. If you have candidates for other studies that deserve similar treatment, please feel free to rebut them. If not, you’re arguing against a straw man, since nobody has argued that the article is wrong because of its political conclusion.
LikeLike
andrewperrin
April 1, 2013 at 6:03 pm
@andrewperrin, I agree that it is a paper with many overly defensive arguments and couched in an unusually elaborate and somewhat pompous manner. But to be fair, it has many qualifiers too. Simply by reading the parts relevant to the criticisms of this paper, my sense is that it won’t be taken to task if it were not about such politically charged issues, and I can feel the author was walking a tightrope. Either he was deliberate or he was torn apart between scientific neutrality and possibly his personal beliefs.
LikeLike
john
April 2, 2013 at 12:35 am
@john, I’m sorry, but the evidence simply doesn’t support your charitable reading. It’s not the qualifiers that are the problem, it’s the central claim that the “no-differences” hypothesis is rejected. The only way one can reach that conclusion from the evidence provided is via personal beliefs, certainly not strong science. And the evidence presented as to the planning process for the study demonstrates unequivocally that he was not “torn apart” by twin demands of neutrality and personal beliefs, but rather that he happily sacrificed scientific neutrality to the desire to make political hay.
As to the question of whether the conclusions of the study explain the degree of the rebuttal, of course they do. That’s been discussed here, and doesn’t in any way undermine scientific method or neutrality.
LikeLike
andrewperrin
April 2, 2013 at 12:58 am
@andrewperrin, regarding the audit, actually every single published paper in sociology invites audit because their data or codes are not made publicly available like what’s been done in economics or political science. As I argued before, for such political charged issues, it is a responsible act to audit/review all important papers published on this topic to arrive at some consensus, even that means inconclusive. The criticisms made in Regnerus paper about previous studies are not pointless.
LikeLike
john
April 2, 2013 at 1:04 am
@john: please perform the audit. We’ll be waiting.
LikeLike
andrewperrin
April 2, 2013 at 1:10 am
@andrewperrin, with a fuller view of available information, especially the fact that this paper takes only a month from submission to acceptance and the very sensitive timing, there is no doubt that this is a clear case of violation of academic honesty/integrity. I just want to add this as my final view on this, and of course it is trivial and irrelevant. Shouldn’t Wright resign then? It is obvious that it is the review process that failed us. I was not playing vengeance of absurdity against those critical of Regnerus paper. I was being critical of the whole review process in the discipline while talking about audit, for example:
1) authors don’t need to submit the data they use for writing up their papers
2) authors don’t need to submit codes used to analyze data and produce results
3) reviewers in general don’t look at the data or codes authors have
4) in some cases and in some sub-areas, the review process is really run by “camaraderie.”
Blind review? I’ve seen people google authors after receiving an invitation for reviewing manuscripts. That would reduce “blindness” probably by half unless you are careful retitling your paper. It is a broken process in many ways.
LikeLike
john
April 2, 2013 at 6:32 pm
FYI, Social Science Research is the only sociology journal I know that practices single-blind review. The reviewers know who the authors are without having to Google them.
LikeLike
Philip N. Cohen
April 2, 2013 at 9:11 pm
[…] Bad science + activism = public harm. The peer review process should have prevented the study from ever being published. And, in being responsible scholars, greater effort should have been made to balance supposed mixed findings: 50 studies say X, but, there is one that says not X; here’s why we the latter study is important (or not). (The ASA brief did this, and further stressed why Regnerus’s study is flawed and irrelevant to LGBT families. Regnerus et al. did not do this in their brief to the Supreme Court.) […]
LikeLike
My Kind of Sociology! Toward A Self-Defined Activist-Academic Career | my sociology
April 17, 2013 at 9:18 pm
James Wright knows that Regnerus lied about his associations with Witherspoon deliberately to mislead the public into believing that Regnerus carried out his “research” with complete autonomy from his anti-gay-rights funders, in an attempt to enhance his credibility, and then additionally Wright published Regnerus’s lie a second time after others made sure he was aware it was a lie. Moreover, Dr. Gary Gates submitted an essay to Wright for inclusion in the November 2012 issue of SSR, explicating that there was no valid peer review of the Regnerus, and suggesting that the paper be withdrawn and put through valid peer review prior to any eventual re-publication. According to Gates, Wright passed that essay on to Elsevier officials,who then blackballed it against being published. Meanwhile, though, Wright and Elsevier continue misleading the public about company policy by saying that Wright has exclusive editorial say-so at SSR. Wright is dishonest and not trustworthy. Because Wilcox is on his editorial board, the two of them have an ethical obligation to disclose everything about Wilcox’s involvement in the study and in getting it published. The Committee on Publication Ethics’ Code of Conduct for Journal Editors, Section 2.1 says that Wright should be making full disclosure of Wilcox’s role. But instead of making full disclosure, both Wright and Wilcox are involved in covering up their misdeeds and preventing the public from having a full understanding of what happened. John M. Becker should not have had to sue UCF for release of relevant documentation. Wilcox’s University of Virginia also is involved in hiding Wilcox’s NFSS-related communications from the public. These are publicly-funded institutions with employees involved in deliberately misleading the public, undermining the trust on which science is based.
LikeLike
Scott Rose
April 27, 2013 at 8:11 pm