orgtheory.net

y’know, i’m kind of proud of science right now, even social science

In age of climate denialism and other chicanery, it’s easy to be a science pessimist. But when I stand back, I become a little more confident about things. Science, as an institution, has not buckled under pressure. For example, I think about vaccine skeptics. Truly bad science that has lead to some deaths. However, science did not abandon vaccines and instead went in search of confirmatory evidence and found nil. This was before the retraction of the infamous article in Lancet.

People may sneer at the social sciences, but they hold up as well. Recently, a well known study in economics was found to be in error. People may laugh because it was an Excel error, but there’s a deeper point. There was data, it could be obtained, and it could be replicated. Fixing errors and looking for mistakes is the hallmark of science. In sociology, we often shy away from the mantle of science, but our recent treatment of the Regnerus paper makes me proud. My fellow sociologists obtained the data, analyzed it, and showed that the new data support the long standing finding of no differences between same sex and different sex parents in terms of childhood outcomes.

If you watch the news, the Coburns of the world claim the attention. But when you think about it, the science haters are really standing in the shadow of a much larger enterprise.

Adverts: From Black Power/Grad Skool Rulz

Written by fabiorojas

May 3, 2013 at 12:13 am

18 Responses

Subscribe to comments with RSS.

  1. I think people’s concern about the state of science isn’t so much the internal workings of science, which as you note seem to be doing fine, but the public’s declining trust in science as an institution and as a method for understanding the world. This is the flip side of the proliferation of news sources and easy access to information online. People can select into whatever kind of information source they want and people tend to choose sources that reinforce their existing biases. So, if you don’t want to believe that global warming is real, it’s very easy to select into a media bubble that will confirm those beliefs regardless of what the actual scientific consensus is on the issue.

    Like

    JD

    May 3, 2013 at 12:37 am

  2. What is really getting sneered at right now is psychological social psychology, between the Stapel stuff and the priming stuff and the precognition stuff, etc.. Like all correctives, some of it has gone too far and I suspect it’s not the most pleasant time to be a social psychologist, but the field had gotten totally out of hand with fetishizing fragile and otherwise dubious results. Proverbs about roosting chickens and unclothed emperors apply, and that people are working now to make it more accountable to the scientific character it’s always claimed is another cause for cheer about science.

    Like

    jeremy

    May 3, 2013 at 12:50 am

  3. Fabio:

    I’d be proud of social science too, except that:

    1. It was over two years before those economists you mentioned above shared the data that allowed people to find the problems in their study. If the system really worked, people wouldn’t have had to struggle for years to try to replicate an unreplicable analysis.

    2. In an unrelated story, when I sent a letter to the editor to a major social science journal pointing out a problem in an article they’d published, they refused to publish my letter, not because of any argument that I was incorrect, but because they judged my letter to not be in the top 10% of submissions to the journal. I’m sure my letter was indeed not in the top 10% of submissions, but the journal’s attitude presents a serious problem, if the bar to publication of a correction is so high. That’s a disincentive for the journal to publish corrections, a disincentive for outsiders such as myself to write corrections, and a disincentive for researchers to be careful in the first place. Just to be clear: I’m not complaining how I was treated here; rather, I’m griping about the system in which a known error can stand uncorrected in a top journal, just because nobody managed to send in a correction that’s in the top 10% of journal submissions.

    Like

    Andrew Gelman

    May 3, 2013 at 12:53 am

  4. Jeremy:

    I’d prefer not to use the word “sneer,” which to me has a pejorative sound. Could we instead say that many specific studies and much of the general practice of statistics in social psychology and medicine is being criticized, and some extreme examples are being mocked?

    Like

    Andrew Gelman

    May 3, 2013 at 12:56 am

  5. Andrew: Sure, I was just tying my comment to Fabio’s use of “sneer” in his post. Criticized and/or mocked are fine and probably more accurate. I do feel like the “hey, look at how this little thing I did has a giant effect size” paradigm had gotten out of control.

    Like

    jeremy

    May 3, 2013 at 1:05 am

  6. I agree with JD above. The problem isn’t (mostly) “science” or “social science.”* It’s the interface between politics, media, public opinion and science. The Reinhart and Rogoff story wasn’t a big deal because a working paper had a spreadsheet error and a questionable weighting scheme. It was a big deal because two big name economists managed to turn a shoddy working paper into a robust, intellectual, science-y sounding defense of policies that were implemented in the year or so after the paper was published (and thus before the discrediting spreadsheet errors could be discovered). They did this in spite of the insistence of many other public-facing economists that what they were selling was, at best, correlations whose causality was likely backwards from what was claimed. (For a longer version of this argument, see here.) So, something’s wrong with the way social science (and probably natural science) is interfacing with the world, since our consensus doesn’t seem to be especially predictive of policy influence, and in fact (for economists at least) economists’ consensus is predictive of the public believing the opposite! (See Sapienza and Zingales’ working paper here.)

    * Scare quotes added because science studies, disunity of science, there’s no such thing as Science!, etc.

    Like

    Dan Hirschman

    May 3, 2013 at 2:46 am

  7. Jeremy + Andrew, as a grad student with an interest in social psychology, can you provide some links to the critiques referenced?

    Like

    socpsy

    May 3, 2013 at 8:13 am

  8. Except your fellow Sociologists did not “obtain the data” as you are reporting. This from the report,

    “Despite numerous requests to Regnerus and SSR editors, neither supplementary analysis that Regnerus stated was already available, diagnostic statistics (e.g., standard errors) nor the raw data had been provided as of this writing, so we were not able to evaluate the underlying quality of the data or assess these alternative mechanisms and hypotheses.”

    Like

    str8grandmother

    May 3, 2013 at 1:47 pm

  9. @socpsy: here are some things to google: “bem precognition”, “stapel fraud”, “smeesters”, “bargh priming debate” See also the graph in this article: http://www.nature.com/news/replication-studies-bad-copy-1.10634

    Like

    jeremy

    May 3, 2013 at 2:10 pm

  10. Thank you, Fabio. I agree. While there is always room for improvement, I think that we should also not act as if the sky is falling either.

    Like

    mike3550

    May 3, 2013 at 6:22 pm

  11. http://retractionwatch.wordpress.com/

    As long as the lights stay on and funding keeps flowing, you will likely feel proud, but this foolishness will fall eventually. The errors are getting too big, and there are too many of us paying attention.

    Like

    August

    May 3, 2013 at 7:17 pm

  12. August, I think you are missing the point. The fact that faulty/fraudulent work is retracted is a good thing. That’s how science works. In any large group (e.g. researchers) there will be some incompetence and misconduct. There will also be honest mistakes. These things happen. However, over time, bad work is debunked (and in some cases retracted) in favor of better work and better evidence. That’s healthy.

    Like

    JD

    May 3, 2013 at 8:34 pm

  13. August – I think you left out the part about it being a dark and stormy night. You’re supposed to lead with that.

    My question: to what extent do we believe that requiring authors to submit their data and code along with their manuscripts will reduce journals’ propensity for publishing results that are in error or, to a lesser extent, entirely fraudulent? (e.g., Simonsohn 2013). I think this would be a good move overall. The tricky part, at least from where I stand on the macro side of the field, is in implementation. Namely, the data we obtain is often proprietary to some degree, and the security of reverse-hashing algorithms may not be enough to placate data providers. And, even if the data is not proprietary, it may be too large to handily “turn in” along with the article. But these are issues of process, not purpose.

    Like

    haastalavista

    May 3, 2013 at 8:40 pm

  14. We can’t really have these conversations because you are locked into this idea that the people ‘on the other side’ are morons. Of course the retractions are good things, as is the vanishingly small attempt of anyone, anywhere, actually trying to replicate a study (who makes a career out of replicating studies?). The other good things are people who get interested in a subject and start reading studies about it. Or programmers getting their hands on code being used to create some of this stuff. Meanwhile, people also learn they can go out and try things themselves: http://blog.sethroberts.net/. This is decentralization, competition.

    Like

    August

    May 3, 2013 at 9:28 pm

  15. I like Hirschman’s comment. The important issue is not science. The important issue is the interaction between science and sociological interests like politics and economics. And there are empirically demonstrable problems associated with this interaction. Which I think is Hirschman’s point. A point not mentioned in the initial post above.

    Like

    Austen

    May 4, 2013 at 2:00 am

  16. @socpsy and @jeremy: See also Blanton et al. (2009 Journal of Applied Psychology) “Strong Claims and Weak Evidence: Reassessing the Predictive Validity of the IAT” https://www.law.upenn.edu/fac/jklick/94JAP567.pdf

    Like

    joshtk76

    May 4, 2013 at 6:44 am

  17. As a scientist Fabio, your sense of pride, or is it surprise, is understandable.

    “Politicians, real-estate agents, used-car salesmen, and advertising copy-writers are expected to stretch facts in self-serving directions, but scientists who falsify their results are regarded by their peers as committing an inexcusable crime [without authentic punishment]. Yet the sad fact is that the history of science swarms with cases of outright fakery and instances of scientists who unconsciously distorted their work by seeing it through lenses of passionately held beliefs.”

    — Martin Gardner
    Science Good, Bad and Bogus (1981), 123.

    Given the availability of (re)iterated scientific findings and myriad opinion-based “facts” which scientists and others use to proclaim the laws of our natural/social systems, are there factors which predict long-term acceptance of one or the other view? Did the majority non-scientist community, conspiracy theorists aside, finally accept a Kepler/Newton view of the physical world when humans succeeded in landing on the moon?

    Like

    Dan

    May 6, 2013 at 6:34 pm

  18. Please keep us updated if you’re still proud of science in cases where it disagrees with your preferences. “Yay science! It confirms all my preconceptions” is not actually a defense of science.

    Like

    bjrob

    May 11, 2013 at 12:07 pm


Comments are closed.

%d bloggers like this: