sign-up to predict/discover the replication rate for social science claims!

This guest post is by Michael Bishop

If your favorite (or least favorite) study was replicated by independent researchers, would they get “the same” result?

We may find out soon because the largest-ever replication effort is under way… but first we need your best research judgment.

Please consider signing up for our project before you miss the chance to evaluate articles from your favorite journals and win prizes for accuracy. Starting this week we are considering studies from: American Sociological Review, American Journal of Sociology, Criminology, and 17 other journals.

Read our recruitment letter below and if you have questions hit me up in the comments or on Twitter. Thanks,


@ReplicationMkts for lots of chatter about the project

@thatMikeBishop will tweet the project at lower volume

Join Replication Markets (RM) for Social Science

With your help we are predicting outcomes of 3,000 social and behavioral science experiments that might be undertaken over the next year. We seek scholars with different voices and perspectives to create a wise and diverse crowd, and hope you will join us.

Sign up!

We invite you – your students, and any other interested parties – to join our crowdsourced prediction platform. By mid-2020 we will rate the replicability of claims from more than 60 academic journals. The claims were selected by an independent team that will also randomly choose about 200 for testing (replication).

RM’s forecasters bet on the chance that a claim will replicate and may adjust their assessment after reading the original paper and discussing results with other players. Previous replication studies have demonstrated prediction accuracy of about 75% with these methods.
RM’s findings will contribute to the wider body of scientific knowledge with a high-quality dataset of claim reliabilities, comparisons of several crowd aggregation methods, and insights about predicting replication. Anonymized data from RM will be open-sourced to train artificial intelligence models and speed future ratings of research claims.
RM’s citizen scientists predict experimental results in a play-money market with real payouts totaling over $100K*. Payouts will be distributed among the most accurate of its anticipated 500 forecasters. There is no cost to play the Replication Markets.

Our project needs forecasters like you with knowledge, insight, and expertise in fields across the social and behavioral sciences. Please share this invitation with colleagues, students, and others who might be interested in participating. We hope to see you in the markets!

On behalf of the Team at Replication Markets,

Yiling Chen, Harvard University
Thomas Pfeiffer, Massey University
Charles Twardy, KeyW, a wholly owned subsidiary of Jacobs
To learn more about the project, visit our homepage and sign up for our newsletter or follow @ReplicationMkts

Ready to join Replication Markets?

Sign up!

*Funding provided by DARPA’s Systematizing Confidence in Open Research and Evidence (SCORE) project.

Camerer, Colin F. et al., 2018, “Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015,” Nature Human Behaviour, 2:637–644.
Danvers, Alexander, Ph.D., 2019, June 2, Psychology Today, “The Gutsy Move Psychology Needs to Make.”
Gelman, Andrew, 2018, July 10, The New York Times, “Essay: The Experiments Are Fascinating. But Nobody Can Repeat Them.”
Rogers, Adam, 2017, July 30, Wired, “DARPA Wants to Build a BS Detector for Science”
Rogers, Adam, 2019, February 15, Wired, “DARPA Wants to Solve Science’s Reproducibility Crisis with AI.”
Science Prediction Market Project,
Tetlock, Philip E, “Superforecasting,” 2015, Broadway Books, New York, NY.



50+ chapters of grad skool advice goodness: Grad Skool Rulz ($4.44 – cheap!!!!)
Intro to sociology for just $1 per chapter – INSANE BARGAIN!!!!!
A theory book you can understand!!! Theory for the Working Sociologist (discount code: ROJAS – 30% off!!)
The rise of Black Studies:  From Black Power to Black Studies 
Did Obama tank the antiwar movement? Party in the Street
Read Contexts Magazine– It’s Awesome!!!!!

Written by fabiorojas

December 3, 2019 at 3:28 pm

Posted in uncategorized

%d bloggers like this: