orgtheory.net

don’t give up on aggregation yet peter

When he’s posting consistently, Peter Levin may be my favorite sociologist blogger. He’s witty, informed, opinionated, appropriately snarky, and smart. He also tends to think about a lot of the same issues that I do, although we usually come at the issues from different directions and often draw different conclusions. But that’s one of the reasons the blog is so great. It’s no fun reading something you always agree with, right?

So here is a post that Peter wrote about the problems of aggregating knowledge/opinions/information. He’s less than enamored with so-called efficient attempts to aggregate knowledge as a way to influence opinions and enhance decision-making.

[T]here are some things to think about here that make this ‘new’ system quite problematic. And I ain’t sayin’ so just because I’m an expert (after all, the policy people really don’t come talking to sociologists, despite my preferences). There are one specific and one theoretical.The first specific is that some people are just crazy, and aside from creating a tail-end of a distribution curve, it’s not at all clear what these folks contribute to the crowd. Old but still hilarious is Andy Baio’s Amazon Knee-jerk Contrarian Game. Personally, I like the ratings game at Yelp, an often-loved but massively crowd-sourced guide….

More theoretically, it has never really be adequately explained why a ‘market-like’ information crowd-sourcing should work. I understand why markets might produce a price that incorporates most public and private information about a commodity. But the widespread substitution of expertise with data mining and crowd-sourcing is a market metaphor more than a market. Why should a metaphor work? This is at the heart of someone like Daniel Davies’ criticism. And I get that sometimes aggregation does work. But there’s no good reason why.

My own feeling is that, using March’s metaphor of ‘exploitation’ and ‘exploration’ (where the first is the plumbing of existing knowledge/arenas, and the second is the seeking out of new opportunities), aggregation mechanisms are better at exploitation than exploration. They do better with existing standards of knowledge, of tastes, of commodities, than they do with something that is new.

Go read the whole post to get a better sense of why Peter thinks that people are sometimes just plain wrong. He quotes a review of MoMA that is pretty funny. While I agree with Peter that the reviewer doesn’t have good taste, something about Peter’s take didn’t feel right to me. After sleeping on it, I think I understand why. The problem is that the reviewer’s belief that modern art is over-rated will be consistent with the view of a good percentage of the population. And so for a certain group of people out there, that opinion will be valid and a useful piece of information when deciding what to do in NYC. The problem isn’t that the reviewer is a moron, it’s just that his tastes are completely different than Peter’s or mine.

The key to channeling knowledge usefully is to find the “right” aggregation method. Sorting through opinions in a discussion board is probably the least effective way to aggregate opinions. Averages, as long as opinions are independent from one another and representative of the population, are pretty useful in figuring out what the general perception of something is. If you want to know if a movie is any good or not, check out Rotten Tomatoes or Metacritic and pay attention to the average rating. Pure averages though, while probably useful for the average consumer, can also be less than useful if your tastes happen to deviate in non-random ways from the rest of the population. And so what we’d like, ideally, is an average opinion or rating from someone who has the same taste set as we do. Peter’s frustration with aggregation probably reflects to an extent the fact that his tastes deviate quite a bit from the average consumer.

This is where aggregation gets tricky, but it’s also where aggregation can be incredibly useful. What we want is a system that can identify clusters of tastes and then figure out “what people like you” would like. While search engines like Google still haven’t developed the right aggregation mechanisms for searches like this, there are some websites that are getting fairly good at it. For example, the other day I was looking at my Netflix account and on the front page they listed a bunch of really unique categories of movies that apparently fit my taste profile (based on my past ratings of movies).  It turns out that my ratings reveal that I’m a person who likes “critically acclaimed dark movies based on real life” and “dramas about marriage based on contemporary literature.” It’s true! I do like movies in those categories. How did Netflix know this? Well, it has a very sophisticated aggregation algorithm that identifies taste clusters and it has accurately positioned my profile within that taste space. Netflix isn’t the only website that has figured out how to do this. Check out Pandora or Slacker if you’re into music. iTunes is starting to improve its aggregation mechanism with the Genius sidebar, although I still don’t think it’s as accurate as Slacker.

To Peter’s next problem – are aggregation mechanisms better at helping us exploit rather than explore?   I don’t think the answer is clearly affirmative. First of all, if your taste preferences deviate from the mean, and yet you’re always following the average based on Rotten Tomatoes recommendations, you’ll likely find a lot of stuff that you wouldn’t typically enjoy. That’s exploration of a sort because you’re forcing yourself to watch stuff that doesn’t fit in your taste set. If, on the other hand, you latch on to one of these more sophisticated algorithms, you can begin to exploit your taste set quite thoroughly (e.g., I can watch every dark movie based on real life that’s ever been produced!); but what’s interesting about this kind of exploitation is that you’re actually finding products that you never would have discovered without the aggregation. You no longer have to go to the local record store to find out what the other nerds like you think is awesome. You can do it from the comfort of your own office chair! So while this IS exploitation, it allows you to exploit much more intensively than you could otherwise and discover new tastes that you never realized you had. Exploitation turns into exploration of your own tastes.

The last reason I’d say aggregation mechanisms work better than Peter thinks is because people actually do tend to like popular forms of music, movies, etc. As Omar famously said in an orgtheory post last year:

Heavy culture consumers are not niche “highbrows” but are “omnivores”; the fact that they do have strong, expert tastes for niche cultures does not imply that they stay away from popular culture (tautologously defined as “that culture which is popular”).

Not only does the average consumer tend to like what is popular but even the cultural snobs like popular stuff! And so, aggregation mechanisms that help you locate the most popular restaurant in NYC or the most highly rated movie of 2008 are going to be fairly accurate representations of what you will like as well. Clearly this is not always the case, especially when you get into the art market, but when you’re dealing with products around which there is more agreement, you’ll find that aggregation mechanisms work pretty darn well.

Written by brayden king

April 23, 2009 at 2:11 pm

Posted in brayden, culture, networks

9 Responses

Subscribe to comments with RSS.

  1. I like your distinction between exploitation and exploration.

    Exploration, for me, conjures to mind the central limit theorem. Where a large group of independent minds encounters the same information and produces a judgment, the central limit theorem suggests that in many circumstances the judgments will converge on a certain mean answer. Note that from a social perspective this way of processing information may be less efficient than asking a trained expert. For example, people keep trying to crowd source prior art searching for patent claims. When it works at all, it’s because the experts are too expensive to be worth hiring to address obviously invalid claims. More often, it’s a close question and the standard deviations around the mean of a crowdsourced answer make it useless in comparison to the narrower standard deviations around expert opinions.

    But one has to be very sure that the opinions offered all begin with direct access to the same information. Even a short chain of heresay can completely destroy a measurement. See here.

    Exploitation, by contrast, requires a novel approach to information that has already been exploited in at least one way. You don’t want to converge on any mean. You want to map out the entire space of possible answers in order to see whether there are any that are, in fact, better.

    Twitter and blogs are interesting to me because they make exploitation possible, not because they make it easy.

    Like

    Michael F. Martin

    April 23, 2009 at 5:09 pm

  2. Michael – I wish I could take credit for the exploitation/exploration distinction. Some guy named March came up with it.

    Like

    brayden

    April 23, 2009 at 7:44 pm

  3. LOL. I googled him. I have seen Cyert and March cited elsewhere, but now have a better context for their work. Thanks for the tip. There are embarrassing gaps in my education.

    Like

    Michael F. Martin

    April 23, 2009 at 8:51 pm

  4. What I find interesting from this, Brayden, is that the two of the ‘best’ examples you cite, Netflix and Pandora, both use some combination of experts with crowd-sourcing. This also seems akin to what Peter was saying in the last bit of his post about Five-Thirty-Eight. Experts were used to come up with the slices that form “critically acclaimed” + “dark” + “based on real life” that can then be combined, via crowd-sourcing, to give you the recommendation. My guess is that the best systems are iterative — if the gurus at Netflix can look for systematic patterns where they think that people might like something and raters pan it, they will probably be able to figure out where they are missing other slices. But, I think that it would take someone familiar with movies to be able to figure out which slices might be missing even based on the data.

    I think that this logic also works with my experiences using wiki technology. Despite the more “democratic” nature of wikis, without someone with a vested interest in making sure that projects move who has some knowledge on the subject, my experience is that they generally fail. Of course, they might get to a point where someone from “the crowd” organically emerges to take over this role after initial shepherding, but it still requires the eye of someone with a more than average knowledge to move the project along.

    Finally, there is also the obvious problem that certain opinions are entirely absent from the internet so that some people’s opinions/views are systematically ignored in that way, too.

    Like

    mike3550

    April 23, 2009 at 8:53 pm

  5. […] I saw the title of Brayden’s post, “Don’t Give Up on Aggregation Yet, Peter,” I thought he’d been reading my macroeconomics posts. Alas, Brayden, prefers meatier fare, […]

    Like

  6. In some sense, this exploitation vs. exploration modes seems analogous to systolic and diastolic pressure — both a focused push and an open flow are necessary to keep the process moving. If that’s the case, then it might explain the need for a balance between experts and the crowd. The experts provide the systolic, the crowd the diastolic.

    A convoluting influence on group decisionmaking might arise from an atmosphere of systolic pressure traceable to precognitive models wired into our brains, which emerge in interpreting ambiguous stimuli. See here. Experts are probably crucial to correcting for the systematic errors that might otherwise result.

    Netflix and Google probably work so well because they offer the diastolic pressure of many options within a broad category of interest, but rely on an expert (you) to make the final determination as to what information is the most relevant. Similar design principles probably apply to organizational theory.

    Like

    Michael F. Martin

    April 23, 2009 at 10:16 pm

  7. […] King of orgtheory.net cites a helpful passage from Peter Levin on aggregation and crowdsourcing: More theoretically, it has never really be adequately explained why a ‘market-like’ […]

    Like

  8. Omar’s observation about high culture omnivores is an important one in breaking our assumptions about strictly exclusionary tastes, but is incorrectly interpreted if we think elite omnivores have no preferences within the pop culture universe. Peter will have to step in to help with the specific examples, but I think the Netflix data, released for the algorithm contest, suggested some incredibly successful films were strongly disliked by many (e.g., Little Miss Sunshine). We still have a ways to go in predicting dispositions within popular culture.

    I also think that treating expert opinions as if they were generated following the same logic (specifically: generated following the same organizational or professional procedures) is a category mistake. Jarl Ahlkvist’s article on radio programmers, and Gabriel Rossman’s work on radio suggest that different programmers, and different stations, adopt distinct strategies toward selecting music for promotion.

    Like

    Jenn Lena

    April 24, 2009 at 12:14 pm

  9. Ha, I step out for an evening and realize I’m being called out by name, even praised, after I was so unkind to the orgtheory. Brayden you’re trying to make me reconsider..

    A few things. First, I’m lumping together into ‘aggregation’ and ‘wisdom of crowds’ a number of different types of activities. Recommendation engines and the Hollywood Stock Exchange (where I’m currently ranked 47943th, with a lifetime ROI of +1,164.95%) are very different, and rather than go with the it’s complicated routine, I lumped a bunch together. I’d go a step further than Lena and say it’s a categorical error to suggest either expert opinion or crowd-sourced outcomes are generated with same logics. I’d like to hear more about the relative value of expert-vs-the crowd across these differences.

    Second, design-wise, I might be wrong about exploitation v. exploration. I had in mind that if you put a movie on HSX and try to value it, you will be way off on movies that don’t already conform to existing kinds of movies. Or, if you base your decisions on what kinds of things people like/want, you end up with Playstation 3 and Xbox 360, and miss the Wii – because people didn’t ‘want’ it before it existed. Ugh, I’m getting muddy.

    But I might be wrong – I mean, Paul DePodesta (of Moneyball, soon a movie with Brad Pitt and ironically undervalued at $31.75 on HSX) explicitly noted that he used data and no assumptions about existing scouting experts to come up with a new way to assess player value. Clearly exploration through data mining.

    (and a propos Mike’s comment above, why 538 and Netflix does well, it seems as though it is something about the difference between regression analysis and Singular Value Decomposition/factor analysis; but that requires more explanation)

    But my problems are theoretical, and here I enlist Hahn and Tetlock’s definitive Information Markets: A New Way of Making Decisions. Which for a theoretical basis starts with (on p2) “Why do information markets work as well as they do?” And then references….The Wisdom of Crowds. And then moves on to design. And Surowiecki’s theoretical answer?

    “At heart, the answer rests on a mathematical truism. If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out. Each person’s guess, you might say, has two components: information and error. Subtract the error, and you’re left with information…With most things the average is mediocrity. With decision making, it’s often excellence. You could say it’s as if we’ve been programmed to be collectively smart” (p10-11).

    Not really obvious at all. There is no answer why a market metaphor would result in something better than experts. But we have the technology to do it, and it seems experimentally to work, and in web 2.0 we can get users to do this all for free! And so screw it – off with the design team and on with the user testing.

    Like

    Peter

    April 24, 2009 at 1:18 pm


Comments are closed.

%d bloggers like this: