Friday, February 8, 2013

Title: Scaling Crowdsourcing with Decision Theory
Speaker: Dan Weld (University of Washington)
Date: Wednesday, Feb 13th
Time: 12-1pm
Room: GHC 8102 

To ensure quality results from unreliable crowdsourced workers, task designers often construct complex workflows and aggregate worker responses from redundant runs. Unfortunately, this need to organize and manage crowd workers forms a bottleneck limiting the magnitude of tasks that can be handled.  We argue that machine learning and decision-theoretic control are key tools for scaling crowdsourced systems. As an illustration, we present Cascade, a novel workflow that creates a globally-coherent taxonomy over a large dataset of items (eg pictures or text passages) from the collective efforts of crowd workers, each of whom have only a very local glimpse into the data, perhaps performing only 20 seconds of work each. Our evaluation shows that the quality of Cascade’s taxonomy is 80-90% that of experts, can complete much more quickly, and by using decision-theoretic optimization is competitively priced.

Daniel S. Weld is Thomas J. Cable / WRF Professor of Computer Science and Engineering at the University of Washington. Weld received a BS from yale in 1982, a PhD from MIT in 1988, a Presidential Young Investigator's award in 1989, an Office of Naval Research Young Investigator's award in 1990, was named AAAI Fellow in 1999, and ACM Fellow in 2006.