Thursday, February 26, 2015

Title:The Search for Truth in Objective and Subjective Crowdsourcing
Speaker: Matthew Lease (School of Information, University of Texas at Austin)
Date: Tuesday, March 3rd
Time: 12-1pm
Room: NSH 1507

Abstract:
One of the most studied problems to date in human computation is how to effectively aggregate responses from different people to infer a "correct", consensus answer. In the first part of my talk, I will discuss how it is that we know so little about progress on this problem despite the enormous research investment that has been made in it. To better gauge field progress, my lab has developed an open source benchmark called SQUARE (ir.ischool.utexas.edu/square) for evaluating relative performance of alternative statistical techniques across diverse use cases. Our findings suggest surprising lack of generality of existing techniques and progress toward a general solution. I argue that if we truly wish to “solve” this problem, current evaluation practices must change. A more fundamental limitation of the consensus-based approaches is the underlying assumption of task objectivity: that a correct answer exists to be found. With subjective tasks, response diversity is valid and desirable, suggesting we cannot merely test for agreement with ground truth or peer responses. As a specific, motivating example, I will discuss the thorny phenomenon of search relevance: what makes search results more or less relevant to different users? Borrowing from psychology, I will describe how psychometrics methodology provides a useful avenue toward both ensuring data quality with subjective tasks in general and gaining new traction on the specific, long-entrenched problem of understanding latent factors underlying search relevance.

Bio:
Matthew Lease is an Assistant Professor in the School of Information at the University of Texas at Austin, with promotion to Associate Professor effective in Fall 2015. Since receiving his Ph.D. in Computer Science from Brown University in 2010, his research in information retrieval, human computation, and crowdsourcing has been recognized by early career awards from NSF, IMLS, and DARPA. Lease has presented crowdsourcing tutorials at ACM SIGIR, ACM WSDM, CrowdConf, and SIAM Data Mining. From 2011-2013, he co-organized the Crowdsourcing Track for the U.S. National Institute of Standards & Technology (NIST) Text REtrieval Conference (TREC). In 2012, Lease spent a summer sabbatical at CrowdFlower tackling crowdsourcing challenge problems at industry-scale. His research has also been popularly featured in WIRED magazine's "Danger Room".

































Wednesday, January 21, 2015

Title: "Reputation Inflation"
Speaker: John Horton (Stern School of Business, New York University)
Date: Tuesday, February 3rd
Time: 12-1pm
Room: GHC 6501

Abstract:
We document reputation "inflation"---a strong, positive trend in average feedback levels---in a large online labor market. Exploiting the matched employer-employee nature of the data, we rule out the possibility that inflation is driven solely by market compositional effects. We argue that this inflation is caused by two factors: (1) negative feedback is more costly to give than positive feedback, and (2) the cost to the rating employer from giving bad feedback is increasing in the market penalty of bad feedback to the rated worker. In an adverse selection model with these assumptions, we show that there is a stable equilibrium in which all rating employers give good feedback regardless of worker performance. We describe a marketplace intervention designed to address reputation inflation: employers were asked to give private feedback about workers in addition to the status quo public feedback. This private feedback differed substantially from the public feedback and was more predictive of future worker performance than public feedback. After this private feedback was collected for a year, aggregated measures of this feedback were experimentally revealed to new employers who were screening applicants. We find that when buyers can condition their screening and hiring choices on this aggregated private feedback, they do so, suggesting they view it as a valuable source of information about workers.

Bio:
John Horton joined New York University Stern School of Business in September 2013 as an Assistant Professor of Information, Operations and Management Sciences. Professor Horton's research focuses on the intersection of labor economics, market design and information systems. He is particularly interested in improving the efficiency and equity of matching markets. After completing his Ph.D. and prior to joining Stern, Professor Horton served for two years as the staff economist for oDesk, an online labor market. Professor Horton received a B.S. in Mathematics from the United States Military Academy at West Point and a Ph.D. in Public Policy from Harvard University.
































Monday, December 1, 2014

Title: Centering the Humans in Human Computation
Speaker: Lilly Irani (Communication & Science Studies, University of California, San Diego)
Date: Tuesday, December 2
Time: 12-1pm
Room: NSH 3305

Abstract:
As HCI researchers explore the possibilities of human computation, their focus has been on the humans enabled by HComp rather than the human workers who power it. Based on five years of design and ethnographic engagement, this talk analyzes the labor inequalities designed into Amazon Mechanical Turk, one popular human computation system. With Six Silberman (UC Irvine) and thousands of Turk workers, I have designed and maintained a system called Turkopticon that both addresses and draws attention to those inequalities. Turkopticon is an activist system that allows workers to publicize and evaluate their relationships with employers. As a common infrastructure, Turkopticon also enables workers to engage in mutual aid. I'll discuss lessons learned from designing and running the system, offer implications for how activists approach design interventions, and conclude with other projects that attempt to center humans in human computation.

Bio:
Lilly Irani is an Assistant Professor of Communication & Science Studies at University of California, San Diego. Her work examines and intervenes in high-technology work practices in situ to understand their relationships with broader cultural, political, and social processes. Her work has appeared at CSCW, CHI, New Media & Society, Science, Technology & Human Values and other venues. Her work has also been covered in The Nation, The Huffington Post, and NPR. Previously, she spent four years as a User Experience Designer at Google. She has a B.S. and M.S. in Computer Science, both from Stanford University and a Ph.D. in Informatics from UC Irvine.































Thursday, November 6, 2014

Title: Emergent Crowdwork during Disaster Events
Speaker: Kate Starbird (Department of Human Centered Design & Engineering, University of Washington)
Date: Tuesday, November 11
Time: 12-1pm
Room: NSH 3305

Abstract:
Crisis events in the physical world are now precipitating mass convergence events online, where thousands and in some cases millions of people turn to social media to seek and share information. This activity includes a new form of spontaneous volunteerism—digital volunteerism—where individuals and organizations come together in online spaces to provide assistance, both to those affected and to emergency responders. Often this takes the form of informational assistance, as volunteers help to process, filter, categorize, map and route information. My research has focused on ways in which remote volunteers contribute to these efforts. In this talk, I will cover some of this previous research, and discuss as well more recent studies examining how members of affected communities, including emergency responders and volunteers, come together with remote volunteers to participate in "emergent crowdwork" after disaster events.

Bio:
Kate Starbird is an Assistant Professor at the Department of Human Centered Design & Engineering (HCDE) at the University of Washington. Dr. Starbird examines online interaction and collaboration in the context of crisis events, specifically looking at how people appropriate social media to help inform others and coordinate response efforts during natural disasters and other large-scale crises. One focal area of her research is digital volunteerism, where remote individuals attempt to provide assistance to those affected. In a related project, she is investigating the flow of misinformation with particular focus on the "work" of the crowd to challenge and correct false rumors. Dr. Starbird's research, which incorporates theory and methods from social science and computer science, is situated within the fields of human-computer interaction (HCI) and computer supported cooperative work (CSCW) as well as the emerging research areas of crowdsourcing and crisis informatics.
































Thursday, October 23, 2014

Title: LabintheWild: Conducting Large-Scale, Uncompensated Online Experiments Around the World
Speaker: Katharina Reinecke (School of Information, University of Michigan)
Date: Tuesday, October 28
Time: 12-1pm
Room: NSH 3305

Abstract:
HAn estimated 95% of research findings in psychology, behavioral economics, and related fields are based on studies with student samples from Western and industrialized countries. As a consequence, some of the knowledge derived from these usually small and locally-recruited student sample populations has been found to be non-generalizable — a finding that led researchers to call these participant samples ``WEIRD'', an acronym for Western, Educated, Industrialized, Rich, and Democratic (Henrich et al. 2010). In this talk, I will show how web-based experimentation with uncompensated samples has the potential to support the generation, replication, verification, and extension of results with larger and more diverse sample populations than previously seen. I will introduce the experimental online platform LabintheWild, which provides participants with personalized feedback in exchange for participation in behavioral studies. In comparison to conventional in-lab studies, LabintheWild enables the recruitment of participants at larger scale and from more diverse demographic and geographic backgrounds: In the last two years, LabintheWild was visited more than 2.5 million times, and nearly 850,000 visitors from more than 200 countries completed an experiment (an average of about 1,000 participants/day). I will discuss how participants hear about the platform, and why they choose to participate. Presenting the results of several experiments, I will additionally show that LabintheWild is a feasible tool to make science less WEIRD.

Bio:
Katharina Reinecke is an Assistant Professor at the University of Michigan School of Information. She received her Ph.D. in Computer Science from the University of Zurich, Switzerland, in 2010, and spent her postdoctoral years at Harvard School of Engineering and Applied Sciences. Her research focuses on cultural differences in users’ interaction with technology with the goal to create culturally intelligent user interfaces that automatically adapt to people’s preferences and perception. To achieve large-scale comparisons of various cultures, she employs datasets from companies such as Doodle and from her own experimental website LabintheWild.org.
































Wednesday, October 15, 2014

Title: Machine Learning for Personalized Clustering
Speaker: Yisong Yue (Computing and Mathematical Sciences, California Institute of Technology)
Date: Tuesday, October 21
Time: 12-1pm
Room: NSH 3305

Abstract:
How would you browse and organize attractions for a potential trip to Paris? How would you organize research articles while conducting a literature review? Such sensemaking tasks are often facilitated via rich user interfaces that enable users to interact more meaningfully with the data. In this talk, I will focus on the problem of learning personalized user models from rich user interactions, and in particular from clustering interactions (i.e., grouping recommended items into clusters). Clustering interactions enable users to express similarity or redundancy between different items, and are a natural way for users to organize and group information. I will describe a new machine learning setting, called collaborative clustering. Analogous to collaborative filtering, in collaborative clustering the goal is to leverage how existing users cluster or group items in order to predict similarity models for other users’ clustering tasks. I will also describe some (very) preliminary work on modeling the full closed-loop interactive clustering setting.

Bio:
Yisong Yue is an assistant professor in the Computing and Mathematical Sciences Department at the California Institute of Technology. He was previously a research scientist at Disney Research. Before that, he was a postdoctoral researcher in the Machine Learning Department and the iLab at Carnegie Mellon University. He received a Ph.D. from Cornell University and a B.S. from the University of Illinois at Urbana-Champaign. Yisong's research interests lie primarily in the theory and application of statistical machine learning. He is particularly interested in developing novel methods for spatiotemporal reasoning, structured prediction, interactive learning systems, and learning with humans in the loop. In the past, his research has been applied to information retrieval, recommender systems, text classification, learning from rich user interfaces, analyzing implicit human feedback, data-driven animation, sports analytics, policy learning in robotics, and adaptive routing & allocation problems.


































Thursday, September 25, 2014

Title: Crowd-Powered Interactive Systems
Speaker: Walter S. Lasecki (Computer Science, University of Rochester)
Date: Tuesday, October 14
Time: 12-1pm
Room: NSH 3305

Abstract:
I create and deploy interactive systems that use a combination of human and machine intelligence to operate robustly in real-world settings. Unlike prior work in human computation, our “Crowd Agent” model allows crowds of people to support real-time interactive systems. For example, Scribe allows non-experts to caption speech in real-time for deaf and hard of hearing users, where prior approaches were either not accurate enough, or required professionals with years of training; Chorus allows multi-session conversations with a virtual personal assistant; and Apparition allows designers to rapidly prototype new interactive interfaces from sketches in real-time. In this talk, I will describe how computationally-mediated groups of people can solve problems that neither people nor computers can solve alone, and scaffold AI systems using the real-world data they collect.

Bio:
Walter S. Lasecki is a Computer Science Ph.D. candidate at the University of Rochester advised by Jeffrey Bigham (CMU). He creates interactive intelligent systems that are robust enough to be used in real-world settings by combining both human and machine intelligence to exceed the capabilities of either. Mr. Lasecki received a B.S. in Computer Science and Mathematics from Virginia Tech in 2010, and an M.S. from the University of Rochester in 2011. He was named a Microsoft Research Ph.D. Fellow in 2013, has held visiting positions at Stanford and Google[x], and is currently a visiting Ph.D. Student at CMU.