Archive for the ‘fabio’ Category
Science just published a piece showing that only a third of articles from major psychology journals can be replicated. That is, if you reran the experiments, only a third of experiments will have statistically significant results. The details of the studies matter as well. The higher the p-value, the less like you were to replicate and “flashy” results were less likely to replicate.
Insider Education spoke to me and other sociologists about the replication issue in our discipline. A major issue is that there is no incentive to actually assess research since it seems to be nearly impossible to publish replications and statistical criticisms in our major journals:
Recent research controversies in sociology also have brought replication concerns to the fore. Andrew Gelman, a professor of statistics and political science at Columbia University, for example, recently published a paper about the difficulty of pointing out possible statistical errors in a study published in the American Sociological Review. A field experiment at Stanford University suggested that only 15 of 53 authors contacted were able or willing to provide a replication package for their research. And the recent controversy over the star sociologist Alice Goffman, now an assistant professor at the University of Wisconsin at Madison, regarding the validity of her research studying youths in inner-city Philadelphia lingers — in part because she said she destroyed some of her research to protect her subjects.
Philip Cohen, a professor of sociology at the University of Maryland, recently wrote a personal blog post similar to Gelman’s, saying how hard it is to publish articles that question other research. (Cohen was trying to respond to Goffman’s work in the American Sociological Review.)
“Goffman included a survey with her ethnographic study, which in theory could have been replicable,” Cohen said via email. “If we could compare her research site to other populations by using her survey data, we could have learned something more about how common the problems and situations she discussed actually are. That would help evaluate the veracity of her research. But the survey was not reported in such a way as to permit a meaningful interpretation or replication. As a result, her research has much less reach or generalizability, because we don’t know how unique her experience was.”
Readers can judge whether Gelman’s or Cohen’s critiques are correct. But the broader issue is serious. Sociology journals simply aren’t publishing error correction or replication, with the honorable exception of Sociological Science which published a replication/critique of the Brooks/Manza (2006) ASR article. For now, debate on the technical merits of particular research seems to be the purview of blog posts and book reviews that are quickly forgotten. That’s not good.
I no longer tell people that I teach “critical thinking” in my classes. My view is that “critical thinking” is a poorly defined buzzword that people use when they can’t articulate what they are actually teaching. For example, look at the wiki entry for “critical thinking:”
Critical thinking is clear, reasoned thinking involving critique. Its details vary amongst those who define it. According to Beyer (1995), critical thinking means making clear, reasoned judgments. During the process of critical thinking, ideas should be reasoned and well thought out/judged.
Notice that the first sentence is literally circular. The third sentence actually adds some content – clarity and reason. If you read the rest of the wiki, the definitions vary wildly from tautological to begging the question. E.g., don’t you first need critical thinking to discover if “participatory democracy” is a prerequisite for critical thinking?
If I don’t teach critical thinking, then what do I teach? Turns out that there is a simple answer: sociology. Other people teach stuff like physics or philosophy. Very concrete. The Critical Thinker might ask: don’t you teach a version of critical thinking? Not quite. My courses do not promise to teach vaguely defined analytical strategies. I teach specific forms of critique. For example, if I am teaching statistics for social science students, I don’t teach “clarity,” rather I teach about sampling, Type 1 and Type 2 errors, and related issues. Similarly, my colleagues in other courses teach specific arguments and ideas. The philosopher might teach about syllogism, and the economist might teach about opportunity costs, which people may not appreciate.
Obtaining truth is hard and there is no magical form of thinking called “critical thinking” that can be separated from specific domains. Aside from a very simple general rules of thumb, such as “don’t be emotional in arguing” or “show my your evidence,” the best way to be improve your thinking is to learn from those who have spent a lifetime actually trying to figure out specific problems.
I am a big believer in social science. For example, I believe there is a lot of evidence supporting the view that elite endorsements do predict party nominations, as documented in The Party Decides. So how does one explain Donald Trump’s current popularity?
The answer, I think, is simple. Normally, politicians need party elites because they don’t have the money, name recognition, organization, or media presence to run for office. Trump has all of these:
- A billion dollar fortune he is willing spend from.
- Decades of media presence.
- His own business organization.
- Name recognition from books, tv, and even a board game.
Add to this that Trump is charismatic, then it is easy to see what the issue is. The Party Decides model is mainly about people who need parties for help. If you need a party, and it doesn’t like you, you’ll loose. Trump has his own resources and he’s great at projecting himself on tv. Thus, he has a chance at bucking the system.
This doesn’t mean that he’s a shoe-in. He could easily turn out to be one of the many also-rans in presidential races. But this reasoning does increase my small belief that he could win a state, or run a Ross Perot style campaign and get 10% or 20% of the popular vote. The deeper lesson here is that politicians, relatively speaking, are poor and need parties. That is why most people have to play by the party’s rules. If you have your own bank account, and you’re good on tv, you can write your own rules.
“Dungeons & Dragons is some of the most crazy, deep, deep, deep nerd shit ever invented.”
But beyond all that, the reasons that D&D is still worth playing are the people you play it with. As opposed to online RPGs where players interact through screens or headphones, when you sit down for a game of Dungeons & Dragons you do it with your people. In the same room. With snacks. Without the rest of the bar watching. There’s a story about three witches and a pack mule, which you all not only watched but invented, and then the witch threw a Dorito at you and drank your scotch.
My games are alcohol free, but I digress:
You learn things about your friends during these times, too. Who are these people when the stakes are low and wagers are little and no one is cool? Poker night gives you permission to get into your friends’ wallet; D&D night gives you permission to get into their heads. Sometimes it’s no surprise: Patton Oswalt played a drunken dwarf, Marilyn Manson says he was a dark elf, VICE international atrocity expert Molly Crabappleplayed a thief—but would you have pegged our porn correspondent, Stoya, for a druid with a dog named George? It’s important to know when there are hippies in your house.
The game is meant to reflect the people playing. D&D came out of the mimeographed, amateur-press wargame scene and reached the height of its popularity in the mid-80s, when zines had staples in them, Metallica didn’t suck, and computers had not yet quite eaten the world—and it still carries a heavy debt to the handmade and the DIY. Every rule in the game has been crossed out and rewritten thousands of times by thousands of pencils in thousands of ways by thousands of Brads, Steves, and Marcys for tens of thousands of tables who wanted to do it this way instead of that way, and none of them needed to learn code to do it.
Yes! People coming together and making an absorbing world with each other. Read the whole thing.
PLoS One has a fun article, with good advice like:
5. Work in the Laboratory of a Previous Nobel Prize Winner
Many Prize recipients have benefitted greatly from the inspiration that this approach can bring. Sometimes just working at an institution with a previous Prize winner can be helpful. One prime example is the Medical Research Council (MRC) Laboratory in Cambridge, United Kingdom, where no less than nine staff members have won Nobel Prizes in either Chemistry or Physiology and Medicine, including my own personal hero Fred Sanger, who won the Chemistry Prize twice (1958, 1980), once for inventing protein sequencing and once for pioneering DNA sequencing. In between, he also invented RNA sequencing, but perhaps three Prizes was more than the Nobel Committee could stomach.
6. Even Better Than Rule 5, Try to Work in the Laboratory of a Future Nobel Prize WinnerThis can be very beneficial, especially if you can be a part of the Prize-winning discovery. That has proven to be a very good strategy, but it is not always easy to spot the right mentor, one who will bring you that sort of success and then share the glory with you. The corollary of this strategy is not to work in the laboratory of someone who has already won but whom you think will win again with you on the ticket. This has yet to prove successful based on the previous double recipients named in Rule 5! It is much better to make sure that any big discoveries come from you after you leave the lab and are out on your own.
Check it out.