Aug 12, 2013

The difference between a herd and a crowd

I recently came across this paper¹ that carries out an experiment to examine how early voting on a comment in an HN/Reddit-style website affects its final score. They randomly up- or down-voted comments when they were created, leaving some untouched to serve as the control group. They found:

• comments that were initially down-voted were corrected, so that their final ratings were not statistically different from that of the control group.
• comments that were initially up-voted saw more benefits: their final ratings were on average 25% higher than the control group.

This reminded me of a couple of papers²³ by Duncan Watts and Matthew Salganik⁴ a few years ago that look at a similar question in the context of songs: how do social signals (like ratings and prior downloads) affect the success of a song in an online market?

They used a slightly different metric, the Gini coefficient, to measure inequality in the resulting market. In the control case, people downloaded songs without being shown how often they had been downloaded in the past. In the social influence case, prior downloads were presented. Not surprisingly, “influence worlds exhibit greater inequality—meaning popular songs are more popular and unpopular songs are less popular—than the world in which individuals make decisions independently.”

But more interestingly, Watts and Salganik found that social signals increased the unpredictability of success. They created “parallel worlds” of users for each case. The variability of the success of a given song across worlds was much greater in the social influence worlds than in the independent ones.

And their final insight was the relationship between “quality” and success. They measured quality of a song by its success in the independent worlds. They report that “in general, the best songs never do very badly, and the worst songs never do extremely well, but almost any other result is possible.”

Ever since James Surowiecki published “The Wisdom of Crowds”, I’ve often heard the glib “rule” that “many minds are smarter than one.” But as Surowiecki constantly stresses in the book, crowds are wise only if each individual judgment is uncorrelated with the others. When the judgements are correlated, you don’t have a wise crowd, just a stupid herd.