Wednesday, April 29, 2015

Title:Incentives in Online Contribution: A game-theoretic framework
Speaker: Arpita Ghosh (School of Computing and Information Science, Cornell University)
Date: Tuesday, May 5th
Time: 12-1pm
Room: NSH 3305

Abstract:
User contribution---whether explicit, as in online crowdsourcing and user-generated content systems, or implicit, via utilization of user data---is central to the Web. The efficacy of systems based on user contribution, however, hinges on users actually participating and behaving as intended by the designer. How does one design various aspects of online contribution platforms---algorithms, reward allocation mechanisms, interfaces---to elicit `good’ outcomes, given that the potential participants in these systems are economic agents with their own costs and benefits to contribution? In this talk, we describe a game-theoretic framework for incentive design for online contribution. To illustrate this framework, we investigate widely-used reward mechanisms in user-generated content and crowdsourcing platforms on the Web---such as badges and leaderboards---in the context of equilibrium outcomes that arise when potential users derive benefit from these social-psychological rewards but incur a cost to contribution. Motivated by a growing literature suggesting that user behavior in online environments might deviate from standard economic models, we explore the idea of `behavioral’ design---theoretical analysis with abstract models based on `real’ behavior---in the context of two problems: the optimal design of contests, widely used in crowdsourcing and user-generated content platforms, and the optimal structure of contracts for crowdwork. Our analysis of equilibrium outcomes in these environments translates to design guidelines in the presence of strategic behavior, and illustrates the idea that formal analysis can inform wide-ranging aspects of the design of online environments for user contribution.

Bio:
Arpita Ghosh is an Associate Professor of Information Science in the School of Computing and Information Science at Cornell University. She received her B.Tech from IIT Bombay in 2001, and her PhD from Stanford in 2006. Prior to joining Cornell, she spent 6 years (2006-2012) in the Microeconomics and Social Sciences group at Yahoo! Research.































Monday, April 6, 2015

Title:Crowdsourcing Translation
Speaker: Chris Callison Burch (Computer and Information Science Department, University of Pennsylvania)
Date: Tuesday, April 7th
Time: 1:30-2:30 pm
Room: GHC 6501

Abstract:
Modern approaches to machine translation are data-driven. Statistical translation models are trained using parallel text, which consist of sentences in one language paired with their translation into another language. One advantage of statistical translation models is that they are language independent, meaning that they can be applied to any language that we have training data for. Unfortunately, most of the world's languages do not have sufficient amounts of training data to achieve reasonable translation quality. In this talk, I will detail my experiments using Amazon Mechanical Turk to create crowd-sourced translations for "low resource" languages that we do not have training data for. I will discuss the following topics: * Quality control: Can non-expert translators produce translations approaching the level of professional translators? * Cost: How much do crowdsourced translations cost compared to professional translations? * Impact of quality on training: When training a statistical model, What is the appropriate trade-off between small amounts of high quality data v. larger amounts of lower quality data? * Languages: Which low resource languages is it possible to translate on Mechanical Turk? What volumes of data can we collect, and how fast? * Implications: What implications does this have for national defense, disaster response, computational linguistics research, and companies like Google?

Bio:
Chris Callison-Burch is the Aravind K Joshi term assistant professor in the Computer and Information Science Department at the University of Pennsylvania. Before joining Penn, he was a research faculty member at the Center for Language and Speech Processing at Johns Hopkins University for 6 years. He was the Chair of the Executive Board of the North American chapter of the Association for Computational Linguistics (NAACL) from 2011-2013, and he has served on the editorial boards of the journals Transactions of the ACL (TACL) and Computational Linguistics. He has more than 80 publications, which have been cited more than 5000 times. He is a Sloan Research Fellow, and he has received faculty research awards from Google, Microsoft and Facebook in addition to funding from DARPA and the NSF.
































Thursday, February 26, 2015

Title:The Search for Truth in Objective and Subjective Crowdsourcing
Speaker: Matthew Lease (School of Information, University of Texas at Austin)
Date: Tuesday, March 3rd
Time: 12-1pm
Room: NSH 1507

Abstract:
One of the most studied problems to date in human computation is how to effectively aggregate responses from different people to infer a "correct", consensus answer. In the first part of my talk, I will discuss how it is that we know so little about progress on this problem despite the enormous research investment that has been made in it. To better gauge field progress, my lab has developed an open source benchmark called SQUARE (ir.ischool.utexas.edu/square) for evaluating relative performance of alternative statistical techniques across diverse use cases. Our findings suggest surprising lack of generality of existing techniques and progress toward a general solution. I argue that if we truly wish to “solve” this problem, current evaluation practices must change. A more fundamental limitation of the consensus-based approaches is the underlying assumption of task objectivity: that a correct answer exists to be found. With subjective tasks, response diversity is valid and desirable, suggesting we cannot merely test for agreement with ground truth or peer responses. As a specific, motivating example, I will discuss the thorny phenomenon of search relevance: what makes search results more or less relevant to different users? Borrowing from psychology, I will describe how psychometrics methodology provides a useful avenue toward both ensuring data quality with subjective tasks in general and gaining new traction on the specific, long-entrenched problem of understanding latent factors underlying search relevance.

Bio:
Matthew Lease is an Assistant Professor in the School of Information at the University of Texas at Austin, with promotion to Associate Professor effective in Fall 2015. Since receiving his Ph.D. in Computer Science from Brown University in 2010, his research in information retrieval, human computation, and crowdsourcing has been recognized by early career awards from NSF, IMLS, and DARPA. Lease has presented crowdsourcing tutorials at ACM SIGIR, ACM WSDM, CrowdConf, and SIAM Data Mining. From 2011-2013, he co-organized the Crowdsourcing Track for the U.S. National Institute of Standards & Technology (NIST) Text REtrieval Conference (TREC). In 2012, Lease spent a summer sabbatical at CrowdFlower tackling crowdsourcing challenge problems at industry-scale. His research has also been popularly featured in WIRED magazine's "Danger Room".

































Wednesday, January 21, 2015

Title: "Reputation Inflation"
Speaker: John Horton (Stern School of Business, New York University)
Date: Tuesday, February 3rd
Time: 12-1pm
Room: GHC 6501

Abstract:
We document reputation "inflation"---a strong, positive trend in average feedback levels---in a large online labor market. Exploiting the matched employer-employee nature of the data, we rule out the possibility that inflation is driven solely by market compositional effects. We argue that this inflation is caused by two factors: (1) negative feedback is more costly to give than positive feedback, and (2) the cost to the rating employer from giving bad feedback is increasing in the market penalty of bad feedback to the rated worker. In an adverse selection model with these assumptions, we show that there is a stable equilibrium in which all rating employers give good feedback regardless of worker performance. We describe a marketplace intervention designed to address reputation inflation: employers were asked to give private feedback about workers in addition to the status quo public feedback. This private feedback differed substantially from the public feedback and was more predictive of future worker performance than public feedback. After this private feedback was collected for a year, aggregated measures of this feedback were experimentally revealed to new employers who were screening applicants. We find that when buyers can condition their screening and hiring choices on this aggregated private feedback, they do so, suggesting they view it as a valuable source of information about workers.

Bio:
John Horton joined New York University Stern School of Business in September 2013 as an Assistant Professor of Information, Operations and Management Sciences. Professor Horton's research focuses on the intersection of labor economics, market design and information systems. He is particularly interested in improving the efficiency and equity of matching markets. After completing his Ph.D. and prior to joining Stern, Professor Horton served for two years as the staff economist for oDesk, an online labor market. Professor Horton received a B.S. in Mathematics from the United States Military Academy at West Point and a Ph.D. in Public Policy from Harvard University.
































Monday, December 1, 2014

Title: Centering the Humans in Human Computation
Speaker: Lilly Irani (Communication & Science Studies, University of California, San Diego)
Date: Tuesday, December 2
Time: 12-1pm
Room: NSH 3305

Abstract:
As HCI researchers explore the possibilities of human computation, their focus has been on the humans enabled by HComp rather than the human workers who power it. Based on five years of design and ethnographic engagement, this talk analyzes the labor inequalities designed into Amazon Mechanical Turk, one popular human computation system. With Six Silberman (UC Irvine) and thousands of Turk workers, I have designed and maintained a system called Turkopticon that both addresses and draws attention to those inequalities. Turkopticon is an activist system that allows workers to publicize and evaluate their relationships with employers. As a common infrastructure, Turkopticon also enables workers to engage in mutual aid. I'll discuss lessons learned from designing and running the system, offer implications for how activists approach design interventions, and conclude with other projects that attempt to center humans in human computation.

Bio:
Lilly Irani is an Assistant Professor of Communication & Science Studies at University of California, San Diego. Her work examines and intervenes in high-technology work practices in situ to understand their relationships with broader cultural, political, and social processes. Her work has appeared at CSCW, CHI, New Media & Society, Science, Technology & Human Values and other venues. Her work has also been covered in The Nation, The Huffington Post, and NPR. Previously, she spent four years as a User Experience Designer at Google. She has a B.S. and M.S. in Computer Science, both from Stanford University and a Ph.D. in Informatics from UC Irvine.































Thursday, November 6, 2014

Title: Emergent Crowdwork during Disaster Events
Speaker: Kate Starbird (Department of Human Centered Design & Engineering, University of Washington)
Date: Tuesday, November 11
Time: 12-1pm
Room: NSH 3305

Abstract:
Crisis events in the physical world are now precipitating mass convergence events online, where thousands and in some cases millions of people turn to social media to seek and share information. This activity includes a new form of spontaneous volunteerism—digital volunteerism—where individuals and organizations come together in online spaces to provide assistance, both to those affected and to emergency responders. Often this takes the form of informational assistance, as volunteers help to process, filter, categorize, map and route information. My research has focused on ways in which remote volunteers contribute to these efforts. In this talk, I will cover some of this previous research, and discuss as well more recent studies examining how members of affected communities, including emergency responders and volunteers, come together with remote volunteers to participate in "emergent crowdwork" after disaster events.

Bio:
Kate Starbird is an Assistant Professor at the Department of Human Centered Design & Engineering (HCDE) at the University of Washington. Dr. Starbird examines online interaction and collaboration in the context of crisis events, specifically looking at how people appropriate social media to help inform others and coordinate response efforts during natural disasters and other large-scale crises. One focal area of her research is digital volunteerism, where remote individuals attempt to provide assistance to those affected. In a related project, she is investigating the flow of misinformation with particular focus on the "work" of the crowd to challenge and correct false rumors. Dr. Starbird's research, which incorporates theory and methods from social science and computer science, is situated within the fields of human-computer interaction (HCI) and computer supported cooperative work (CSCW) as well as the emerging research areas of crowdsourcing and crisis informatics.
































Thursday, October 23, 2014

Title: LabintheWild: Conducting Large-Scale, Uncompensated Online Experiments Around the World
Speaker: Katharina Reinecke (School of Information, University of Michigan)
Date: Tuesday, October 28
Time: 12-1pm
Room: NSH 3305

Abstract:
HAn estimated 95% of research findings in psychology, behavioral economics, and related fields are based on studies with student samples from Western and industrialized countries. As a consequence, some of the knowledge derived from these usually small and locally-recruited student sample populations has been found to be non-generalizable — a finding that led researchers to call these participant samples ``WEIRD'', an acronym for Western, Educated, Industrialized, Rich, and Democratic (Henrich et al. 2010). In this talk, I will show how web-based experimentation with uncompensated samples has the potential to support the generation, replication, verification, and extension of results with larger and more diverse sample populations than previously seen. I will introduce the experimental online platform LabintheWild, which provides participants with personalized feedback in exchange for participation in behavioral studies. In comparison to conventional in-lab studies, LabintheWild enables the recruitment of participants at larger scale and from more diverse demographic and geographic backgrounds: In the last two years, LabintheWild was visited more than 2.5 million times, and nearly 850,000 visitors from more than 200 countries completed an experiment (an average of about 1,000 participants/day). I will discuss how participants hear about the platform, and why they choose to participate. Presenting the results of several experiments, I will additionally show that LabintheWild is a feasible tool to make science less WEIRD.

Bio:
Katharina Reinecke is an Assistant Professor at the University of Michigan School of Information. She received her Ph.D. in Computer Science from the University of Zurich, Switzerland, in 2010, and spent her postdoctoral years at Harvard School of Engineering and Applied Sciences. Her research focuses on cultural differences in users’ interaction with technology with the goal to create culturally intelligent user interfaces that automatically adapt to people’s preferences and perception. To achieve large-scale comparisons of various cultures, she employs datasets from companies such as Doodle and from her own experimental website LabintheWild.org.