Quizz: Targeted Crowdsourcing with a Billion (Potential) Users

Title: Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Speaker: Panos Ipeirotis (NYU Stern)
Date: Tuesday, Dec 15th
Time: 12-1pm
Room: GHC 6501

We describe Quizz, a gamified crowdsourcing system that simultaneously
assesses the knowledge of users and acquires new knowledge from them.
Quizz operates by asking users to complete short quizzes on specific
topics; as a user answers the quiz questions, Quizz estimates the
user’s competence. To acquire new knowledge, Quizz also incorporates
questions for which we do not have a known answer; the answers given
by competent users provide useful signals for selecting the correct
answers for these questions. Quizz actively tries to identify
knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging “for free” the targeting capabilities of
existing, publicly available, ad placement services. Quizz quantifies
the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback
allows the ad targeting mechanism to further optimize ad placement.

Our experiments, which involve over ten thousand users, confirm that
we can crowdsource knowledge curation for niche and specialized
topics, as the advertising network can automatically identify users
with the desired expertise and interest in the given topic. We present
controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as
goals, which incentivize the users to contribute. Finally, our
cost-quality analysis indicates that the cost of our approach is below
that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of
potential users all over the planet, and being able to reach users
with specialized expertise that is not typically available through
existing labor marketplaces.

Panos Ipeirotis is an Associate Professor and George A. Kellner
Faculty Fellow at the Department of Information, Operations, and
Management Sciences at Leonard N. Stern School of Business of New York
University.  He received his Ph.D. degree in Computer Science from
Columbia University in 2004. He has received nine “Best Paper” awards
and nominations, a CAREER award from the National Science Foundation,
and is the  recipient of the 2015 Lagrange Prize in Complex Systems, for his
contributions in the field of social media, user-generated content and

Title: Learnersourcing: Improving Learning with Collective Learner Activity
Speaker: Juho Kim (Stanford CS)
Date: Tuesday, Dec 1st
Time: 12-1pm
Room: GHC 6501

Millions of learners today are watching videos on online platforms, such as Khan Academy, YouTube, Coursera, and edX, to take courses and master new skills. But existing video interfaces are not designed to support learning, with limited interactivity and lack of information about learners' engagement and content. Making these improvements requires deep semantic information about video that even state-of-the-art AI techniques cannot fully extract. I take a data-driven approach to address this challenge, using large-scale learning interaction data to dynamically improve video content and interfaces. Specifically, my research introduces learnersourcing, a form of crowdsourcing in which learners collectively contribute novel content for future learners while engaging in a meaningful learning experience themselves. In this talk, I will present learnersourcing applications designed for massive open online course videos and how-to tutorial videos, where learners' collective activities (1) highlight points of confusion or importance in a video, (2) extract a solution structure from a tutorial, and (3) improve the navigation experience for future learners. I will then discuss how the idea of learnersourcing can generalize to broader educational and social contexts. My research demonstrates that learnersourcing can enable more interactive, collaborative, and data-driven learning.

Juho Kim is a Visiting Assistant Professor of Computer Science and a Brown Fellow at Stanford University. He will be an Assistant Professor in the School of Computing at KAIST starting fall 2016. His research interests lie in human-computer interaction, learning at scale, video interfaces, and crowdsourcing. He builds interactive systems powered by large-scale data from users, in which users’ natural and incentivized activities dynamically improve content, interaction, and experience. He earned his Ph.D. from MIT, M.S. from Stanford University, and B.S. from Seoul National University. He is a recipient of six paper awards from CHI and HCOMP, and the Samsung Fellowship.

Title: Crowdsourcing a Meeting of Minds
Speaker: Michael Bernstein (Stanford HCI)
Date: Tuesday, October 27th
Time: 12-1pm
Room: GHC 6501

Crowdsourcing is an increasingly powerful method where computation guides many amateurs' efforts in order to recreate an expert's abilities. However, across domains from design to engineering to art, few goals are truly the effort of just one person — even one expert. If we can now crowdsource simple tasks such as image labeling, how might computation coordinate many peoples' abilities toward far more complex and interdependent goals? In this talk, I present computational systems for gathering and guiding crowds of experts, including professional programmers, designers, singers and artists. The resulting collectives tackle problems modularly and at scale, dynamically grow and shrink depending on task demands, and combine into larger organizations. I'll demonstrate how computationally-enabled expert crowds can pursue goals such as designing new user experiences overnight, producing animated shorts in two days, and even pursuing novel research.

Michael Bernstein is an Assistant Professor of Computer Science at Stanford University, where he co-directs the Human-Computer Interaction group and is a Robert N. Noyce Family Faculty Scholar. His research focuses on the design of crowdsourcing and social computing systems. This work has received Best Paper awards and nominations at premier venues in human-computer interaction and social computing (ACM UIST, ACM CHI, ACM CSCW, AAAI ISWSM). Michael has been recognized with the NSF CAREER award, as well as the George M. Sprowls Award for best doctoral thesis in Computer Science at MIT. He holds Ph.D. and M.S. degrees in Computer Science from MIT, and a B.S. in Symbolic Systems from Stanford University.

Title: Ethical Use of Amazon Mechanical Turk (and Best Practices You Shouldn't Use mTurk Without)
Speaker: Kristy Milland (TurkerNation.com)
Date: Tuesday, October 13th
Time: 12-1pm
Room: GHC 6501

As a Turker, Requester and researcher, Kristy Milland offers insight into use of the AMT platform from all perspectives. If you are looking to use AMT more efficiently and effectively, she will present tips and tricks that can help you become a better Requester. She will also provide insight into who the Turkers are how they work, indispensable if you want to create attractive HITs that get done quickly and correctly, and also if you want to ensure your data isn't corrupted by cheating or bias.

Kristy is a community manager on the oldest forum for Amazon Mechanical Turk workers, TurkerNation.com. She helps researchers, journalists and businesses learn more about "mTurk" from the eyes of a worker, and is an invaluable source of insight for how to enable more productive partnerships between requesters and workers (e.g., better HIT design, tips and resources for finding the right workers, understanding what makes “Turkers” tick). Kristy is also a Psychology major at Ryerson University. Her interests are varied, but she is especially intrigued by studies investigating the psychology of mTurk (motivation, linguistics, bias, etc.), or ethnographic work with Turkers.

Title: Life, the Universe, and Information Processing
Speaker: Pietro Michelucci (Human Computation Institute)
Date: Tuesday, October 6th
Time: 12-1pm
Room: GHC 6501

Despite what some believe to be a looming technological “singularity,” when machines will purportedly become super-intelligent and hopefully save the world, humans today continue to be the most effective integrators and producers of information in the known universe.  We accomplish this directly and through the use of information-processing inventions, such as computing and network technologies. As these inventions become increasingly sophisticated, the human factor will draw upon our most complex cognitive abilities, such as abstract thinking, creativity, and working knowledge of the world. Examining what makes the human contribution unique and how it complements emerging technologies will help us anticipate human labor in a future filled with increasingly automated systems.  Additionally, through the advancement of human computation – novel methods that combine the respective strengths of humans and machines, we can expect to see human and machine based processing become more tightly integrated with each other, resulting in new capabilities. These collective human/machine systems will be able to model and predict physical, biological, and social processes with unprecedented accuracy, enabling more effective problem solving and decision making. Indeed, these systems may ultimately coalesce into a global organism with the capacity to address societies most wicked problems and achieve planetary stability.

Dr. Michelucci received a joint-PhD from Indiana University in Cognitive Science and Mathematical Psychology and has been a science advisor to federal research agencies since 2006. He has actively supported the advancement of Human Computation through a recent Springer handbook, a new open-access scholarly journal, various speaking engagements and workshops, interagencyinitiatives in Social Computing, and Citizen Science working groups. Recently, he led the Human Computation Roadmap Summit, a CRA-funded visioning activity at the Wilson Center, which included White House OSTP participation, toward a national initiative in Human Computation. As an Organismic Computing pioneer, he is interested in developing new methods for augmenting group intelligence and efficacy, and developing high-impact applications of the resultant capabilities that will benefit humanity.

Title:Incentives in Online Contribution: A game-theoretic framework
Speaker: Arpita Ghosh (School of Computing and Information Science, Cornell University)
Date: Tuesday, May 5th
Time: 12-1pm
Room: NSH 3305

User contribution---whether explicit, as in online crowdsourcing and user-generated content systems, or implicit, via utilization of user data---is central to the Web. The efficacy of systems based on user contribution, however, hinges on users actually participating and behaving as intended by the designer. How does one design various aspects of online contribution platforms---algorithms, reward allocation mechanisms, interfaces---to elicit `good’ outcomes, given that the potential participants in these systems are economic agents with their own costs and benefits to contribution? In this talk, we describe a game-theoretic framework for incentive design for online contribution. To illustrate this framework, we investigate widely-used reward mechanisms in user-generated content and crowdsourcing platforms on the Web---such as badges and leaderboards---in the context of equilibrium outcomes that arise when potential users derive benefit from these social-psychological rewards but incur a cost to contribution. Motivated by a growing literature suggesting that user behavior in online environments might deviate from standard economic models, we explore the idea of `behavioral’ design---theoretical analysis with abstract models based on `real’ behavior---in the context of two problems: the optimal design of contests, widely used in crowdsourcing and user-generated content platforms, and the optimal structure of contracts for crowdwork. Our analysis of equilibrium outcomes in these environments translates to design guidelines in the presence of strategic behavior, and illustrates the idea that formal analysis can inform wide-ranging aspects of the design of online environments for user contribution.

Arpita Ghosh is an Associate Professor of Information Science in the School of Computing and Information Science at Cornell University. She received her B.Tech from IIT Bombay in 2001, and her PhD from Stanford in 2006. Prior to joining Cornell, she spent 6 years (2006-2012) in the Microeconomics and Social Sciences group at Yahoo! Research.

Title:Crowdsourcing Translation
Speaker: Chris Callison Burch (Computer and Information Science Department, University of Pennsylvania)
Date: Tuesday, April 7th
Time: 1:30-2:30 pm
Room: GHC 6501

Modern approaches to machine translation are data-driven. Statistical translation models are trained using parallel text, which consist of sentences in one language paired with their translation into another language. One advantage of statistical translation models is that they are language independent, meaning that they can be applied to any language that we have training data for. Unfortunately, most of the world's languages do not have sufficient amounts of training data to achieve reasonable translation quality. In this talk, I will detail my experiments using Amazon Mechanical Turk to create crowd-sourced translations for "low resource" languages that we do not have training data for. I will discuss the following topics: * Quality control: Can non-expert translators produce translations approaching the level of professional translators? * Cost: How much do crowdsourced translations cost compared to professional translations? * Impact of quality on training: When training a statistical model, What is the appropriate trade-off between small amounts of high quality data v. larger amounts of lower quality data? * Languages: Which low resource languages is it possible to translate on Mechanical Turk? What volumes of data can we collect, and how fast? * Implications: What implications does this have for national defense, disaster response, computational linguistics research, and companies like Google?

Chris Callison-Burch is the Aravind K Joshi term assistant professor in the Computer and Information Science Department at the University of Pennsylvania. Before joining Penn, he was a research faculty member at the Center for Language and Speech Processing at Johns Hopkins University for 6 years. He was the Chair of the Executive Board of the North American chapter of the Association for Computational Linguistics (NAACL) from 2011-2013, and he has served on the editorial boards of the journals Transactions of the ACL (TACL) and Computational Linguistics. He has more than 80 publications, which have been cited more than 5000 times. He is a Sloan Research Fellow, and he has received faculty research awards from Google, Microsoft and Facebook in addition to funding from DARPA and the NSF.

Title:The Search for Truth in Objective and Subjective Crowdsourcing
Speaker: Matthew Lease (School of Information, University of Texas at Austin)
Date: Tuesday, March 3rd
Time: 12-1pm
Room: NSH 1507

One of the most studied problems to date in human computation is how to effectively aggregate responses from different people to infer a "correct", consensus answer. In the first part of my talk, I will discuss how it is that we know so little about progress on this problem despite the enormous research investment that has been made in it. To better gauge field progress, my lab has developed an open source benchmark called SQUARE (ir.ischool.utexas.edu/square) for evaluating relative performance of alternative statistical techniques across diverse use cases. Our findings suggest surprising lack of generality of existing techniques and progress toward a general solution. I argue that if we truly wish to “solve” this problem, current evaluation practices must change. A more fundamental limitation of the consensus-based approaches is the underlying assumption of task objectivity: that a correct answer exists to be found. With subjective tasks, response diversity is valid and desirable, suggesting we cannot merely test for agreement with ground truth or peer responses. As a specific, motivating example, I will discuss the thorny phenomenon of search relevance: what makes search results more or less relevant to different users? Borrowing from psychology, I will describe how psychometrics methodology provides a useful avenue toward both ensuring data quality with subjective tasks in general and gaining new traction on the specific, long-entrenched problem of understanding latent factors underlying search relevance.

Matthew Lease is an Assistant Professor in the School of Information at the University of Texas at Austin, with promotion to Associate Professor effective in Fall 2015. Since receiving his Ph.D. in Computer Science from Brown University in 2010, his research in information retrieval, human computation, and crowdsourcing has been recognized by early career awards from NSF, IMLS, and DARPA. Lease has presented crowdsourcing tutorials at ACM SIGIR, ACM WSDM, CrowdConf, and SIAM Data Mining. From 2011-2013, he co-organized the Crowdsourcing Track for the U.S. National Institute of Standards & Technology (NIST) Text REtrieval Conference (TREC). In 2012, Lease spent a summer sabbatical at CrowdFlower tackling crowdsourcing challenge problems at industry-scale. His research has also been popularly featured in WIRED magazine's "Danger Room".

Title: "Reputation Inflation"
Speaker: John Horton (Stern School of Business, New York University)
Date: Tuesday, February 3rd
Time: 12-1pm
Room: GHC 6501

We document reputation "inflation"---a strong, positive trend in average feedback levels---in a large online labor market. Exploiting the matched employer-employee nature of the data, we rule out the possibility that inflation is driven solely by market compositional effects. We argue that this inflation is caused by two factors: (1) negative feedback is more costly to give than positive feedback, and (2) the cost to the rating employer from giving bad feedback is increasing in the market penalty of bad feedback to the rated worker. In an adverse selection model with these assumptions, we show that there is a stable equilibrium in which all rating employers give good feedback regardless of worker performance. We describe a marketplace intervention designed to address reputation inflation: employers were asked to give private feedback about workers in addition to the status quo public feedback. This private feedback differed substantially from the public feedback and was more predictive of future worker performance than public feedback. After this private feedback was collected for a year, aggregated measures of this feedback were experimentally revealed to new employers who were screening applicants. We find that when buyers can condition their screening and hiring choices on this aggregated private feedback, they do so, suggesting they view it as a valuable source of information about workers.

John Horton joined New York University Stern School of Business in September 2013 as an Assistant Professor of Information, Operations and Management Sciences. Professor Horton's research focuses on the intersection of labor economics, market design and information systems. He is particularly interested in improving the efficiency and equity of matching markets. After completing his Ph.D. and prior to joining Stern, Professor Horton served for two years as the staff economist for oDesk, an online labor market. Professor Horton received a B.S. in Mathematics from the United States Military Academy at West Point and a Ph.D. in Public Policy from Harvard University.