crowdsourcing sociology department rankings
I think we can all agree that the NRC rankings were a disaster. So, Steve Vaisey and I think that we can generate a new list from scratch. Using Matt Salganik’s excellent “All Our Ideas” site, we’ve set up a tool for pairwise comparison of Sociology Departments. The goal is to get as many head-to-head snap judgments as possible. You can vote as many times as you like — in fact, it’s encouraged. The implicit ranking is generated from the whole collection of pairwise comparisions. Just go to
http://www.allourideas.org/socrankings/
and get voting!
To learn more about All Our Ideas see here.
Update: So, based on publicity from (a) this post, (b) a tweet, and (c) some Facebook status updates, we’re pushing towards 20,000 pairwise votes after about five hours. Not bad. And the results so far look pretty good, certainly if you think of them less as perfectly differentiated ranks and more as banded estimates.
Update 2: Incidentally, several people have taken advantage of the “suggest your own idea” option. Suggestions fall into three categories. First, universities outside North America, which I haven’t been including. Second, schools that are already on the list: San Diego, Riverside, Penn — if you haven’t come across them in your voting, it’s because you haven’t waited long enough to see them. All of the schools ranked in Sociology by the NRC are included. Third, options such as “Harvard Social Relations c. 1955”, “People Who Have Left the University of Wisconsin-Madison”, and (the cleverest one so far) “Tri-Valley Center for Human Potential”. These aren’t going to make it on the list.
orgtheory.net 
Hadn’t seen the technology behind this before — love, love the idea. Very cool.
LikeLike
teppo
January 7, 2011 at 8:11 pm
Well, I am glad someone finally did this. And head to head rankings are easier than the clunky DIY scheme I proposed last year.
Kudos to you and Steve.
LikeLike
fabiorojas
January 7, 2011 at 8:35 pm
How many votes/voters do you need before the results are credible?
LikeLike
brayden king
January 7, 2011 at 8:55 pm
I saw this movie, and I think after the inevitable blowback from insulted departments, you both end up creating Facebook. The question is, will Steve fire Kieran this summer when he is looking for investors in NYC? And who is playing Sean Parker?
LikeLike
carolinewlee
January 7, 2011 at 9:03 pm
Caroline: I think the natural next step for Healy-Vaisey is a similar ranking of journals, editors, key articles, hot soc topics, and individual sociologists/org theorists.
LikeLike
teppo
January 7, 2011 at 9:11 pm
How many votes/voters do you need before the results are credible?
Well, comparatively, it looks like the face-validity:cost ratio is very much in our favor. Compare the results from one blog post + Matt’s website + 45 minutes to NRC Behemoth + Federal Funds + Seven years.
LikeLike
Kieran
January 7, 2011 at 9:15 pm
Genius Teppo! We could have a little catwalk show for the hottest org theorists in Vegas. With bottle service for Kieran and Steve, of course.
LikeLike
carolinewlee
January 7, 2011 at 9:17 pm
There is no question in my mind that the results and method are superior to the NRC. Whether or not they beat out my random assignment technique remains to be seen, but honestly it’s looking pretty good at the moment!
LikeLike
blue monster
January 7, 2011 at 9:19 pm
The bar was fairly low, Kieran.
One way you might assess the validity of the results is when they are stable. When the addition of another 500 votes wouldn’t change the hierarchy much, you know your ranking is pretty robust.
LikeLike
brayden king
January 7, 2011 at 9:50 pm
Kieran, I love that you, Steve, and Matt are responding to the awful NRC rankings, and I love experimentation with new methods of data collection, but I do have a concern. It seems to me that this method will mostly capture prestige and size of the school. Now those are useful things to measure, but they aren’t the things I’m most excited to learn about departments. What do I want to learn?
Which departments do a good job of supporting research? Which do a good job of supporting undergraduate/graduate teaching? Which are downright fun? Which ones have the least/most postmodernists/people studying the influence of genetics on social behavior?
LikeLike
Michael Bishop
January 7, 2011 at 9:58 pm
Teppo: I like the idea of ranking journals and social theorists. Maybe that’ll be next… :-)
Michael: the nice part about “better” is that is means whatever people think it means. Go go ahead and vote pomo. ;-) I also think the influence of size of department will be smaller here than most reputation surveys because you have to judge many, many comparisons for every time you judge your own institution.
LikeLike
Steve Vaisey
January 7, 2011 at 10:18 pm
I’ve heard the best sociology department is actually the hypothetical department comprised of people who were once faculty at the University of Wisconsin. I submitted this as an idea, but, well, spoilsport moderators.
LikeLike
jeremy
January 7, 2011 at 10:24 pm
jeremy, are you suggesting some kind of fantasy sociology league?
LikeLike
gabrielrossman
January 7, 2011 at 10:29 pm
Also, where is the poll to crowdsource what Fabio teaches in his theory course?
LikeLike
jeremy
January 7, 2011 at 10:35 pm
Actually, crowdsourcing a syllabus would be pretty easy with this. Just add every reading you are considering as an item and then let the people decide. You can even add suggestions. I might do this next time I teach Soc of Culture…
LikeLike
Steve Vaisey
January 7, 2011 at 10:39 pm
I’ve heard the best sociology department is actually the hypothetical department comprised of people who were once faculty at the University of Wisconsin
I’d pit Fantasy Arizona against Fantasy Wisconsin any day.
LikeLike
Kieran
January 7, 2011 at 11:00 pm
Why don’t you do an experiment here?
Randomly assign visitors to the ranking page to one of two versions:
Version one is the current one (school names only)
Version two provides both school names and a list of the core faculty members.
I presume you’ll get exactly the same result for the very top — but that the extra-information version may reshape the bottom half of the rankings.
LikeLike
bork
January 8, 2011 at 12:35 am
Bork: Interesting idea. I suspect you’re right. But that’s not something I plan to do. It took about 5 minutes to set this up. What you propose would take a lot longer and this is (mostly) just for fun.
BTW, does anyone have a copy of the 2001 U.S. News Rankings? I can’t find them anywhere online…
LikeLike
Steve Vaisey
January 8, 2011 at 12:46 am
[…] This post was mentioned on Twitter by chokha, Social Psych. Social Psych said: SocialPsych.org news crowdsourcing sociology department rankings: I think we can all agree that the NRC rankings… http://bit.ly/f8ltjR […]
LikeLike
Tweets that mention crowdsourcing sociology department rankings « orgtheory.net -- Topsy.com
January 8, 2011 at 8:02 am
I’m hoping “fantasy sociology” joins “astrosociology” among the new orgtheory-inspired sections of ASA.
LikeLike
Jerry Davis
January 8, 2011 at 3:30 pm
So I spent some time making my picks over the morning coffee, and looked down to see how many I had made. I ended up with something like 200 because I was waiting for my school to pop up at least twice. So this made me wonder: Who has made the most picks on here?
LikeLike
Hillbilly
January 8, 2011 at 4:26 pm
on the “tri-valley center for human potential” thing, wasn’t there some survey experiment once asking if people thought we should invade some totally made-up but vaguely communist sounding country? could be some mileage in throwing a decoy in there to see where it lands in the rankings.
LikeLike
gabrielrossman
January 8, 2011 at 6:19 pm
This is pretty great. UI-wise, I wish skipping a vote didn’t take so many clicks, since this makes it hard to state that you really don’t know what some departments are. Or is this intentional, to make it harder to vote up a certain school without influencing other rankings? I can see how this would be a reasonable tradeoff, but this again seems to benefit precision near the top at the cost of noise near the bottom. Throwing in a few decoy options (like Gabriel suggests) might be a decent way of drawing a bottom line below which the rankings are more noise than content.
LikeLike
Andrei Boutyline
January 9, 2011 at 1:38 am
Hi everyone,
Wow, people seem to love ranking sociology departments; this is one of our most active idea marketplaces. (If you are curious, the most active—in terms of votes and ideas uploaded—was this one by the Washington Post: http://wapo.st/ib4yO3)
I’m really glad that Kieran mentioned the uploaded ideas because that is one of the main reasons I wanted to create allourideas.org. With a regular survey the researcher has to write all the questions and all the answers ahead of time. That makes it very hard to learn about new things. The main alternative would be something like an interview where people can suggest new ideas, but interviews are slow, expensive, and hard to quantify. allourideas.org develops a hybrid approach that I’m calling a “wiki survey” that combines the quantifiability of a survey with the openness of an interview.
You can see how the uploaded ideas scored compared to the seed ideas in the first data visualization on this page: http://www.allourideas.org/socrankings/results. In this case, none of the uploaded ideas scored in the top 10 (not surprising given that most of the alternatives are known ahead of time in this case). However, in most of our idea marketplaces, many of the top 10 ideas are uploaded by users. That means that new, good ideas are bubbling-up. For example, the New York City Mayor’s Office created an idea marketplace to discover ideas that would make NYC “greener and greater” (http://www.allourideas.org/planyc/) in which 9 of the top 10 ideas are uploaded by visitors.
Also, several people suggested tweaks such as adding more information (bork) and adding “decoy” departments (gabriel and andrei). These both seem interesting, and even if Kieran and Steve don’t do this, you can. Anyone can make their own idea marketplace here: http://www.allourideas.org/questions/new. Like Steve said, it takes just a minute to set one up. You could also try using our API to create your own pairwise comparison website (http://blog.allourideas.org/post/847415684/api-released-open-source).
LikeLike
Matt Salganik
January 9, 2011 at 9:56 pm
I set up a Philosophy one yesterday afternoon. I sent it along to Brian Leiter, who linked it on his blog and it’s already surpassed the Soc rankings in activity — it’s pushing 60k votes at this stage.
LikeLike
Kieran
January 9, 2011 at 10:19 pm
[…] where should you go to find sociology rankings? The NRC? No. U.S. News and World Report? Perhaps, if you also need to catch up on AARP-related […]
LikeLike
sociology department rankings: now with added legitimacy « orgtheory.net
January 12, 2011 at 9:15 pm
Given an initial base of rankings, wouldn’t it be more efficient to allow rank similarity to drive subsequent pairings (e.g., there’s no real need to compare Chicago to Sioux Falls Community College, but more ratings of Chicago vs. Wisconsin would be useful)?
LikeLike
Bob
January 12, 2011 at 10:53 pm
@Bob: Yeah, there’s a lot to praise in this system, but that’s one of the major downsides of this approach. Say you have 100 schools. If you were willing to assume no intransitive rankings (which would be a big assumption behaviorally, but not so much from the standpoint of rankingsly), then there are 5050 paired comparisons implicit in the data. I’m not sure how many paired comparisons you’d need to reveal one individual’s rank ordering of 100 schools, but it seems on the order of 400 or so. In other words, if the comparisons worked adaptively and filled in pairs based on information an individual had already provided, I think you’d get something like 10 times the information for the same # of clicks.
LikeLike
Jeremy
January 12, 2011 at 11:04 pm
I talked to Matt about this issue today, as it happens (and he may have more to say about this himself here) — he’s thought about these issues a lot, and there’s already a lot of work behind the system already. It turns out that there are some difficulties (not trivial but not insurmountable, either) to presenting more rank-similar paired comparisions on a moment-to-moment basis. More interestingly, people also dislike making strings those sorts of comparisions — they’re harder to do, cognitively.
LikeLike
Kieran
January 12, 2011 at 11:11 pm
@Bob and @Jeremy: Yes, adaptive approaches for choosing pairs could be far more efficient. This is something that I have thought a lot about and that we are planning on doing. However, being able to implement certain adaptive strategies in real-time and at scale is computationally challenging.
Another issue to consider is that the adaptive approach that @Bob suggested, having people vote on ideas with similar scores (e.g., Michigan vs. Wisconsin), might make the voting process much less fun. Just imagine if each choice you had to make was hard. More generally, the amount of information that a voter contributes is related to the amount of information contributed per vote and the number of votes cast. Thus, increasing the information per vote might actually decrease the amount of information collected, if it causes people to cast fewer votes. Ultimately this is an empirical question and one that we are in a position to answer because we can run experiments on the site where we randomize respondents to different pair selection algorithms (another thing on the to do list).
One last thing about adaptivity is that we are actually doing something adaptive right now: we show ideas that were recently uploaded with higher frequency. That allows us to collect information about the uploaded ideas more quickly. In this particular case the uploaded ideas were not of interest to most people, but in most cases the uploaded ideas are the most interesting (remember, these are the ones that the survey creator didn’t know about ahead of time). Also, the fact that we are doing something adaptive and nobody had to know about it highlights a characteristic of the way that allourideas.org is designed: we can do lots of complicated stuff on the back-end—in terms of how we choose pairs and how we calculate the score—without any need to increase complexity on the front-end. The plan is that these back-end operations will continue to improve over time. So, if you create your own idea marketplace a few months from now, the site might look the same, but it will probably be more efficient in many invisible ways.
Finally, thank you everyone for your interesting comments. It has been neat to “listen in” on the conversations that this has started. If you have any more feedback or questions about the site, please let me know.
LikeLike
Matt Salganik
January 13, 2011 at 4:22 am
matt,
regarding your point of not loading too many similar pairs, see this interview with Nintendo’s Shigeru Miyamoto where he notes that a good game should have some easy parts rather than constantly pushing the player to do the most difficult thing possible.
http://www.newyorker.com/reporting/2010/12/20/101220fa_fact_paumgarten
LikeLike
gabrielrossman
January 13, 2011 at 7:46 pm
A little point: ‘The University of Wales’ is on the comparison list. There is no sociology department at the ‘university of wales’ which is a system like the university of london (i.e. lots of seperate universities, each with its own departments) and _not_ like Oxbridge.
Did you mean the School of Social Sciences at _Cardiff_ University? (Which, unhelpfully, isn’t actually part of the University of Wales. Even tho’ it’s the biggest university in Wales. Go figure)
LikeLike
Adam Hedgecoe
January 14, 2011 at 11:04 am
Hm yes, that snuck in there by mistake — it was suggested by a user. I was monitoring the suggestions, but that one snuck by me.
LikeLike
Kieran
January 14, 2011 at 11:11 am
Would you be willing, after a certain number of votes have been cast, to make public the votes files (including for the Political Science rankings), as per http://goo.gl/4GEH2? It would be interesting to try to identify whether there are multiple dimensions governing evaluation. Thanks!
LikeLike
dsparks
January 14, 2011 at 1:29 pm
@gabrielrossman: Cool, thanks for that link.
LikeLike
Matt Salganik
January 14, 2011 at 3:48 pm
[…] by Kieran and Steve’s ranking initiative (of sociology departments, see here), here’s an effort to crowdsource management journal […]
LikeLike
management journal rankings, crowdsourced « orgtheory.net
January 15, 2011 at 12:20 am
Is this a bug, or did someone add Northeastern a second time?
Needless to say, I could not decide which idea was better!
Also, I really like the idea of using adaptive approaches for choosing pairs. The majority of decisions offered are trivially easy and don’t really reveal new information.
LikeLike
ksiler
January 15, 2011 at 6:09 am
Fixed. It wasn’t there long, I think.
LikeLike
Kieran
January 15, 2011 at 1:42 pm
@ksiler: Glad that wasn’t a bug :). Actually, I’m amazed that duplicate ideas have not appeared more. Lots of ideas have been uploaded and right now our interface for managing them is a bit clunky. I guess Kieran can keep all this stuff in his head.
Also, I’m actually a bit disappointed that Kieran fixed the problem so quickly because duplicate ideas are an interesting test for the system. In other situations I’ve seen similar ideas uploaded (e.g., “fewer students” and “smaller student body”) and they end up scoring about the same. That’s a nice check that nothing crazy is happening.
LikeLike
Matt Salganik
January 15, 2011 at 5:54 pm
Because it is ridiculously easy to set this up for one’s pet discipline, here’s one for neuroscience department rankings: http://www.allourideas.org/neurorank
Thanks for the fun toy.
LikeLike
John Flournoy
January 17, 2011 at 10:33 pm
I thought about trying to set it up to crowdsource sociology journal rankings, since people’s ideas about those vary quite a bit from indicators like impact factor, but I’ve refrained.
LikeLike
Jeremy
January 17, 2011 at 10:47 pm
Jeremy: You should. The management journals one (despite flaws: already discussed at various places) is interesting. Not perfect, not definitive — but certainly interesting.
LikeLike
teppo
January 17, 2011 at 11:27 pm
It would be nice to have a universal directory of the different rankings (would also have saved me the embarrassment of setting up a law school duplicate for the conglomerate). I’m sure the list of rankings is long (well plus private ones).
Of course, the question is whether you want everyone potentially voting, in some cases yes, others not (you could opt into a directory or not) – who sees the rankings, through what sources (listservs, blogs) of course materially affects outcomes.
A side note to ‘all’ our readers in Israel, orgtheory reader Amit set up a ranking of the Knesset: http://www.allyourideas.org/knesset
LikeLike
teppo
January 18, 2011 at 4:04 am
A sociology journal crowdsource can be found here: http://www.allourideas.org/socjournals
LikeLike
countrytrash
January 18, 2011 at 4:47 am
@countrytrash: as far as I know, there is no journal titled “Sociological Methods”–though there is “Sociological Methodology” and “Sociological Methods and Research”. I’m guessing it was supposed to be the latter, since the former is already included on the list.
LikeLike
Andrei Boutyline
January 19, 2011 at 3:29 am
This is a pretty cool effort and thanks for highlighting All Your Ideas. It’s very very cool.
Another interface would be using something like Selectricity (http://selectricity.org) to split the pairwise comparisons out of a full ranking so you’d have one set of comparisons between everyone and everyone else (FULL DISCLOSURE: I helped build Selectricity in a previous life!). With a good drag and drop ranking interface, you’d be surprised how quickly folks can rank even a pretty long list.
But my real question: Do people that don’t study social structure spend as much effort talking about, critiquing, and creating academic program rankings? Am I the only one that has been surprised at how much ink has been spilled about the NRC in these circles?
LikeLike
Benjamin Mako Hill
February 5, 2011 at 11:53 pm
Selectricity looks very interesting — i’ll have to take a closer look at that.
As for
Am I the only one that has been surprised at how much ink has been spilled about the NRC in these circles?
The topic of ranking and quantified categorization in general has become independently popular in the past decade in the areas you mention, so that may have something to do with it. At a minimum you have a lot of social survey/research design people who are suddenly on the sharp end of a survey instrument whose output, the rankings, may have real and possibly quite negative consequences for them.
LikeLike
Kieran
February 6, 2011 at 2:11 pm
Gladwell on rankings – http://www.newyorker.com/reporting/2011/02/14/110214fa_fact_gladwell
LikeLike
teppo
February 9, 2011 at 4:19 am
[…] an estimate of what the de-averaged results probably looked like. Back in 20011, Steve Vaisey and I ran a poll using Matt Salganik’s excellent All Our Ideas website, which creates rankings from multiple […]
LikeLike
Crowdsourcing Sociology Department Rankings: 2013 Edition | orgtheory.net
March 13, 2013 at 1:08 pm