Saturday, April 22, 2017

Solving Photo Mysteries with Expert-Led Crowdsourcing

Title: Solving Photo Mysteries with Expert-Led Crowdsourcing
Speaker: Kurt Luther, Department of Computer Science, Virginia Tech
Time: 12:30 - 1:30pm
Room: Gates-Hillman Complex 6501


Despite the old adage that a picture is worth a thousand words, images often need context to be meaningful to their viewers. In this talk, I show how expert-led crowdsourcing, a novel approach that combines the relative strengths of experts and amateur crowds, can be used to solve photo mysteries. In one example, I conducted a qualitative study of image verification experts in journalism, national security, and human rights organizations to understand how they perform geolocation, the process of mapping the precise location where a photo or video was taken. This research informed the design of GroundTruth, a system where experts collaborate with crowds to geolocate unknown images. In another example, I partnered with a historical photography magazine to develop Civil War Photo Sleuth, a system that leverages crowdsourcing and computer vision techniques to help experts identify unknown soldier portraits from the 19th century. I also discuss broader challenges and opportunities in crowdsourced investigations, open-source intelligence, and collaborative sensemaking illustrated by these examples.


Kurt Luther is an assistant professor of computer science at Virginia Tech, where he is also affiliated with the Center for Human-Computer Interaction, the Department of History, and the Hume Center for National Security and Technology. He directs the Crowd Intelligence Lab (, an interdisciplinary research group exploring how crowdsourcing systems can support creativity and discovery. He is principal investigator for over $1.5M in sponsored research, including an NSF CAREER Award. Previously, Dr. Luther was a postdoctoral fellow in the HCI Institute at Carnegie Mellon University. He received his Ph.D. in human-centered computing from Georgia Tech, where he was a Foley Scholar, and his B.S. in computer graphics technology from Purdue University. He has also worked at IBM Research, Microsoft Research, and YouTube/Google.

Saturday, April 15, 2017

Supporting Collective Ideation at Scale

A growing number of online collective ideation platforms, such as OpenIDEO or Quirky, have demonstrated the potential of large-scale collaborative innovation in various domains. However, these platforms also introduce new challenges. People have to wade through a sea of possibly mundane and redundant ideas before encountering genuinely inspiring ones. Further, once all ideas are collected, the communities have to spend a lot of time and effort to synthesize the ideas into a few solutions. Alternatively, an intelligent system can select and present ideas for its users instead of leaving them to look for inspirations in a haphazard way.

In this talk, I will show how a system can decide which ideas to present to the users and when to do so. I will introduce a computational model of an idea space, two crowdsourcing methods to generate this model and the model's application for creativity-enhancing interventions. I will also present an empirical study on the effects of timing of example delivery on people's idea generation.

Pao is a Ph.D. candidate in Computer Science focusing on Human-Computer Interaction (HCI) research at Harvard University. She works with Prof. Krzysztof Gajos in the Intelligent Interactive Systems Group. Her research explores how we can apply intelligent technologies and crowdsourcing to enable novel ways for people to come up with creative ideas together. Pao received her B.S. in Electrical Engineering and M.S. in Computer Science from Stanford University where she worked in Stanford HCI group.

Tuesday, April 4, 2017

The Collaboration and Communication Networks within the Crowd

Title: The Collaboration and Communication Networks within the Crowd
Speaker: Siddharth Suri, Microsoft Research, New York City
Time: 12:30-1:30pm
Room: Newell-Simon Hall 1507

Title: The Collaboration and Communication Networks within the Crowd
Since its inception, crowdsourcing has been considered a black-box approach to solicit labor from a crowd of workers. Furthermore, the crowd has been viewed as a group of independent workers dispersed all over the world. One goal of this work is to show that crowdworkers collaborate to fulfill technical and social needs left by the platform they work on. That is, crowdworkers are not the independent, autonomous workers they are often assumed to be, but instead work within a social network of other crowdworkers. Crowdworkers collaborate with members of their networks to 1) manage the administrative overhead associated with crowdwork, 2) find lucrative tasks and reputable employers and 3) recreate the social connections and support often associated with brick and mortar-work environments. We also build on and extend these discoveries by mapping the entire communication network of workers on Amazon Mechanical Turk, a leading crowdsourcing platform. We execute a task in which over 10,000 workers from across the globe self-report their communication links to other workers, thereby mapping the communication network among workers. Our results suggest that while a large percentage of workers indeed appear to be independent, there is a rich network topology over the rest of the population. That is, there is a substantial communication network within the crowd. The existence of these networks could have implications for the burgeoning literature that involves conducting behavioral experiments and research on crowdsourcing sites. Overall, our evidence combines ethnography, interviews, survey data and larger scale data analysis from four crowdsourcing platforms. This paper draws from an ongoing, longitudinal study of crowdwork that uses a mixed methods approach to understand the cultural meaning, political implications, and ethical demands of crowdsourcing. 

Siddharth “Sid” Suri is a computational social scientist.  His research lies at the intersection of computer science, behavioral economics and crowdsourcing.  Sid is currently writing a book with Mary Gray titled “On-Demand: Crowds, Platform Economies, and the Future of Work in Precarious Times” that combines ethnography and computer science to understand the future of work.

Sid earned his Ph.D. in computer and information science from the University of Pennsylvania in 2007 under the supervision of Michael Kearns. After that he was a postdoctoral associate working with Jon Kleinberg in the computer science department at Cornell University.  Then he moved to the Human & Social Dynamics group at Yahoo! Research led by Duncan Watts.  Currently, Sid is one of the founding members of Microsoft Research, New York City.