Katharina Reinecke

Title: Volunteer-Based Online Experiments With Diverse Samples: Lessons Learned from Six Years of LabintheWild
Speaker: Katharina Reinecke, Assistant Professor of Computer Science & Engineering, University of Washington
Date: May 1, 2018
Time: 12:00-1:00pm
Room: Gates-Hillman Complex 6501

An estimated 95% of our scientific knowledge about people, their behavior, perception, and preferences is based on studies with “WEIRD” samples, an acronym for participants who are Western, Educated, Industrialized, Rich, and Democratic. My work in Human-Computer Interaction has shown that technology developed for WEIRD users---based on knowledge that is derived from studies with mostly US-based student populations---is often misaligned with the preferences, behaviors, and abilities of a large proportion of the world's population. Users differ in their goals, how they perceive information,  and what they can work with most efficiently.

In this talk, I report on six years of experience running the volunteer-based online experiment platform LabintheWild.org, which has enabled behavioral experiments at larger scale and with less WEIRD participants than feasible in laboratory studies or on Mechanical Turk. LabintheWild enables participants to compare themselves to others in exchange for study participation; a feedback mechanism that has attracted an average of more than 1,000 participants a day from 230 countries. I present ten main lessons learned from this experience and show how LabintheWild experiments have enabled us to build volunteer-powered, self-sustaining design support tools in various domains.    

Katharina Reinecke is an Assistant Professor in the Paul G. Allen School of Computer Science and Engineering at the University of Washington. Prior to joining the University of Washington, she was an Assistant Professor in the School of Information at the University of Michigan. She received her Ph.D. in Computer Science from the University of Zurich and spent her postdoctoral years at Harvard School of Engineering and Applied Sciences. Her research explores how people from various demographic and geographic backgrounds vary in their use of technology with the goal to create user interfaces that automatically adapt to people's abilities, preferences, and perception. To find out how people differ, she co-founded the volunteer-based online experiment site LabintheWild.org, which has enabled her to study several million participants from 230 countries. Katharina's work has been recognized with several Best Paper awards and nominations at premier venues in Human-Computer Interaction (ACM CHI, ACM CSCW) and an NSF CAREER Award.

Find out more about Katharina Reinecke at https://homes.cs.washington.edu/~reinecke/.

Edith Law

Title: Socio-technical Challenges in Scientific and Medical Crowdsourcing
Speaker: Edith Law, Assistant Professor in Computer Science, University of Waterloo
Date: April 17, 2018
Time: 12:00-1:00pm
Room: Gates-Hillman Complex 6501

Science is increasingly data-intensive; yet, many research tasks involving the collection, annotation and analysis of data are too complex to be fully automated. The idea of research-oriented crowdsourcing is to engage people without formal academic training to contribute or process data towards answering questions. In this talk, I will discuss the variety of socio-technical challenges that arise when designing scientific and medical crowdsourcing systems, and demonstrate through examples various situations where conventional approaches to crowdsourcing fall short.

Dr. Edith Law is an assistant professor at the David R. Cheriton School of Computer Science at University of Waterloo, where she co-directs the Human Computer Interaction (HCI) Lab. Her research focuses on studying how humans can augment and make sense of intelligent systems, as well as developing new curiosity-based strategies for engaging users and encouraging long-term interactions between humans and machines.  Previously, she was a postdoctoral fellow at the School of Engineering and Applied Sciences at Harvard University. She graduated from Carnegie Mellon University in 2012 with Ph.D. in Machine Learning, and holds a M.Sc. in Computer Science from McGill University, and B.Sc. in Computer Science from University of British Columbia. She co-authored the book “Human Computation” and helped create the first AAAI Conference on Human Computation and Crowdsourcing (HCOMP). Her work on games with a purpose, large-scale collaborative planning and curiosity as an incentive mechanism have won best paper honourable mentions at the ACM SIGCHI conference.

Find out more about Edith Law at http://edithlaw.ca/.

Panos Ipeirotis

Title: Targeted Crowdsourcing with a Billion (Potential) Users
Speaker: Panos Ipeirotis, Professor in Department of Information, Operations,
and Management Sciences at NYU Stern School of Business
Date: April 10, 2018
Time: 12:00-1:00pm
Room: Gates-Hillman Complex 6501

We describe Quizz, a gamified crowdsourcing system that simultaneously
assesses the knowledge of users and acquires new knowledge from them.
Quizz operates by asking users to complete short quizzes on specific
topics; as a user answers the quiz questions, Quizz estimates the
user’s competence. To acquire new knowledge, Quizz also incorporates
questions for which we do not have a known answer; the answers given
by competent users provide useful signals for selecting the correct
answers for these questions. Quizz actively tries to identify
knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging “for free” the targeting capabilities of
existing, publicly available, ad placement services. Quizz quantifies
the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback
allows the ad targeting mechanism to further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that
we can crowdsource knowledge curation for niche and specialized
topics, as the advertising network can automatically identify users
with the desired expertise and interest in the given topic. We present
controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as
goals, which incentivize the users to contribute. Finally, our
cost-quality analysis indicates that the cost of our approach is below
that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of
potential users all over the planet, and being able to reach users
with specialized expertise that is not typically available through
existing labor marketplaces.

Panos Ipeirotis is a Professor and George A. Kellner Faculty Fellow at
the Department of Information, Operations, and Management Sciences at
Leonard N. Stern School of Business of New York University. He
received his Ph.D. degree in Computer Science from Columbia University
in 2004. He has received nine “Best Paper” awards and nominations and
is the recipient of the 2015 Lagrange Prize, for his contributions in the
field of social media, user-generated content, and crowdsourcing.

Find out more about Panos Ipeirotis at http://www.ipeirotis.com/.

Kate Starbird

Title: Muddied Waters: Online Disinformation during Crisis Events 
Speaker: Kate Starbird, Assistant Professor in Human-Centered Design and Engineering, University of Washington
Date: March 20, 2018
Time: 12:00-1:00pm
Room: Newell-Simon Hall 1305

Since 2013, my collaborators and I have conducted research on how rumors and misinformation spread through social media during crisis events. Recently, our work has revealed how a subsection of the alternative media ecosystem facilitates the spread of disinformation—in the form of conspiracy theories or “alternative narratives” about man-made crisis events. This disinformation is often employed as part of political agendas and poses information security risks, in part through reduced trust in information systems. In this talk, I’ll present highlights from our research on alternative narratives of crisis events, describing some of the specific tactics and emergent effects of disinformation. I’ll also share some preliminary findings on more recent work examining the structure and dynamics of the media ecosystem that has taken shape around the ongoing crisis in Syria—an ongoing “site” of information warfare. Finally, I’ll discuss some of the broader implications of online disinformation for humanitarian responders, platform designers, and society at large.

Kate Starbird is an Assistant Professor at the Department of Human Centered Design & Engineering (HCDE) at the University of Washington (UW). Kate's research is situated within human-computer interaction (HCI) and the emerging field of crisis informatics—the study of the how information-communication technologies (ICTs) are used during crisis events. One aspect of her research focuses on how online rumors spread—and how online rumors are corrected—during natural disasters and man-made crisis events. More recently, she has begun exploring the propagation of disinformation and political propaganda through online spaces. Kate earned her PhD from the University of Colorado at Boulder in Technology, Media and Society and holds a BS in Computer Science from Stanford University. 

Find out more about Kate Starbird at http://faculty.washington.edu/kstarbi/.

Extreme Democracy

Title: Extreme Democracy 
Speaker: Ariel Procaccia, Associate Professor in Computer Science, Carnegie Mellon University
Date: December 12, 2017
Time: 12:00-1:00pm
Room: Gates-Hillman Center 6501

I will present several forms of democratic decision making that go far beyond your run-of-the-mill election. Specifically, I will discuss (i) liquid democracy, which allows voters to transitively delegate their votes; (ii) participatory budgeting, in which residents of a city or country vote on how its budget should be divided; and (iii) virtual democracy, which automates ethical decisions by holding elections among models of real voters. I will focus on the computational challenges that these new paradigms give rise to. 

Ariel Procaccia is an Associate Professor in the Computer Science Department at Carnegie Mellon University. He usually works on problems at the interface of computer science and economics. His distinctions include the IJCAI Computers and Thought Award (2015), the Sloan Research Fellowship (2015), the NSF Faculty Early Career Development Award (2014), and the IFAAMAS Victor Lesser Distinguished Dissertation Award (2009); as well as half a dozen paper awards, including Best Paper (2016) and Best Student Paper (2014) at the ACM Conference on Economics and Computation (EC). He is co-editor of the Handbook of Computational Social Choice (Cambridge University Press, 2016).