Monday, February 20, 2017

Real-Time Crowdsourcing for Complex Tasks

Title: Real-Time Crowdsourcing for Complex Tasks
Speaker: Walter Lasecki, Assistant Professor, University of Michigan
Date: Tues, Feb 21
Time: 12:30-1:30
Room: NSH 1507
Creating robust intelligent systems that can operate in real-world settings at super-human performance levels requires a combination of human and machine contributions. Crowdsourcing has allowed us to scale the ubiquity of these human computation systems, but the challenges in mixing human and machine effort remain a limiting factor of these systems. My lab’s work on modeling crowds as collective agents  has helped alleviate some of these challenges at a system level, but how we can create cohesive ecosystems of crowd-powered tools that together solve more complex and diverse needs remains an open question. In this talk, I will discuss some initial and ongoing work that aims to create complex crowdsourcing systems for applications that cannot be solved using only a single tool. 

Walter S. Lasecki is an Assistant Professor of Computer Science and Engineering at the University of Michigan, Ann Arbor, where he directs the Crowds+Machines (CROMA) Lab. He and his students create interactive intelligent systems that are robust enough to be used in real-world settings by combining both human and machine intelligence to exceed the capabilities of either. These systems let people be more productive, and improve access to the world for people with disabilities. Dr. Lasecki received his Ph.D and M.S. from the University of Rochester in 2015 and a B.S. in Computer Science and Mathematics from Virginia Tech in 2010. He has previously held visiting research positions at CMU, Stanford, Microsoft Research, and Google[x].​

Tuesday, February 7, 2017

Perpetually enhancing user experience through collaborative, dynamic, personalized experimentation

Title: Perpetually enhancing user experience through collaborative, dynamic, personalized experimentation
Speaker: Joseph Jay Williams, Harvard University
Date: Tues, Feb 14
Time: 12:30-1:30pm

There is a proliferation of websites and mobile apps for helping people learn new concepts (e.g. online courses), and learn how to change health habits and behavior (e.g. websites for reducing depression, apps for quitting smoking). How can we use data from real-world users to rapidly enhance and personalize these technologies? I show how we can build self-improving systems by reimagining randomized A/B experimentation as an engine for collaboration, dynamic enhancement, and personalization. I present a novel system that enhanced learning from math problems, through crowdsourcing explanations and automatically experimenting to discover the best. My second application boosted responses to an email campaign, by experimentally discovering how to personalize motivational messages to a user's activity level. These self-improving systems use experiments as a bridge between designers, social-behavioral scientists and researchers in statistical machine learning.

Joseph Jay Williams is a Research Fellow at Harvard's Office of the Vice Provost for Advances in Learning, and a member of the Intelligent Interactive Systems Group in Computer Science. He completed a postdoc at Stanford University in the Graduate School of Education in Summer 2014, working with the Office of the Vice Provost for Online Learning and the Open Learning Initiative. He received his PhD from UC Berkeley in Computational Cognitive Science, where he applied Bayesian statistics and machine learning to model how people learn and reason. He received his B.Sc. from University of Toronto in Cognitive Science, Artificial Intelligence and Mathematics, and is originally from Trinidad and Tobago. More information about his research and papers is at

Sunday, February 5, 2017

Crowdbotics: Writing Software with Crowds

Title: Crowdbotics: Writing Software with Crowds
Speaker: Anand Kulkarni, Founder of Crowdbotics, previous Chief Scientist of LeadGenius
Date: Tues, Feb 7
Time: 12:30 - 1:30pm
Room: NSH 1507

Crowd computing systems are now capable of doing more sophisticated work than ever before, accelerating the pace of creative work such as creating movies, carrying out research, and writing stories.  How can crowds be used to automate software creation? This is a complex problem, requiring good methods for expert selection, program synthesis and collaboration.  We’ll discuss strategies for accelerating software creation using crowds of geographically-diverse communities of software engineers, as well as the cultural and technical challenges that emerge.

We'll also discuss several open problems in crowd computing, including efforts by our team and others to create a crowdsourcing compiler, a hypothetical system that can optimally divide an open-ended task between artificial intelligence systems and teams of humans.  These systems blur the line between crowd-powered software and real-world organizations, raising important questions about the future of work.

Anand is founder of Crowdbotics, a startup using crowdsourcing and machine intelligence to accelerate the process of software development. Crowdbotics brings developers from new-to-coding communities worldwide into a global community of software engineers that collaborate 1-1 with technology companies.  Prior to that, Anand was Chief Scientist of LeadGenius, a Y Combinator, Sierra Ventures, and Lumia Capital-backed startup using human computation and deep learning to automate account-based marketing (ABM). LeadGenius has raised over $20M in venture funding and developed best-in-class marketing automation technology used by Fortune 500 customers like Google, eBay, and Box. In conjunction with nonprofits like the World Bank, LeadGenius generates fairly-paid digital employment for over 500 individuals in 40 countries.  Anand has been named as one of Forbes Magazine's "30 under 30" Top 30 Entrepreneurs Under 30. Anand has published over a dozen papers in ACM, AAAI and IEEE magazines, journals, and conferences. Anand previously held a National Science Foundation graduate research fellowship in mathematics. He holds degrees in Industrial Engineering and Operations Research, Mathematics, and Physics from UC Berkeley.

Sunday, January 29, 2017

Collective Creativity Systems: Design families and remix landscapes

Title: Collective Creativity Systems: Design families and remix landscapes
Speaker: Jeffrey Nickerson, Professor, School of Business, Stevens Institute of Technology
Date: Tues, Jan 31
Time: 12:30-1:30pm
Room: NSH 1507

Collective creativity systems often include modification and recombination mechanisms, forms of remixing. In addition, metasystems can accelerate the exploration of design space. For example, customizers create design families by defining parameter ranges. These customizers can in turn be modified. The effect of this reuse for customization is analyzed in the 3D printing community Thingiverse. The study focuses on the design artifacts and their inheritance history, as well as their shape and semantic distance from each other. Some ways of using coordination and computation to catalyze design space exploration are discussed, with examples from several online communities.

Jeffrey Nickerson is a professor in the School of Business at Stevens Institute of Technology. His current research focuses on collective creativity: through observation and experiment, he seeks to understand how systems composed of humans and machines can explore design space. He has a Ph.D. from New York University in Computer Science, as well as an M.F.A. in Graphic Design from Rhode Island School of Design. He is now working on two NSF-funded projects related to online community-based design and problem solving.

Tuesday, January 17, 2017

Speaker: Daniela Braga (CEO of DefinedCrowd)
Date: Tuesday, Jan 24
Time: 12:30-1:30pm
Room: NSH 1507

DefinedCrowd was born out of the founder’s personal frustration around data quality and slowness when dealing with data providers. Artificial Intelligence is next big technology milestone which replaces humans in menial tasks, giving us more time to enjoy more human-human interactions and focus on the really creative tasks. But to achieve this technology leap, two things are required: high quality training data and state-of-the-art machine learning algorithms. In this talk, I will walk through the reasons behind the creation of DefinedCrowd and how we are solving the data quality problem faster and at scale, by combining the latest methodologies on crowdsourcing with machine learning. DefinedCrowd is a global company headquartered in Seattle, WA, with an R&D center in Lisbon, Portugal, graduated from the Microsoft Seattle Accelerator in Machine Learning (with less than 1%  of acceptance rate and recognized by Microsoft as one of the fastest growing companies in the space) and which raised $1.1 M seed round in September 2016 from Amazon, Sony and Portugal Ventures. DefinedCrowd was also featured in as one of the 7 AI companies to watch in 2017.

Founder and CEO of DefinedCrowd, one of the fastest growing startups in the AI space. With seventeen years working in Speech Technology both in academia and industry in Portugal, Spain, China and the US, Daniela Braga has deep expertise in Speech Science and is one the world leaders of Crowdsourcing adoption in large enterprises. Previously at Microsoft worked in pretty much all stacks of Speech Technology and shipped 26 languages for Exchange 14, 10 TTS voices in Windows 8 and was involved in Cortana. At Voicebox, created the Data Science and Crowdsourcing team, where she introduced Crowdsourcing for big data solutions and re-structured the Engineering infrastructure around data collection, processing, ingestion, instrumentation, storage, browsing and discoverability. Her effort has resulted in reducing data collection and processing costs by 80%; her approach has been adopted in multiple organizations. Dr. Braga is oftentimes guest lecturer in the University of Washington, USA, is the author of more than 90 scientific papers and several patents.

Tuesday, December 8, 2015

Title: Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Speaker: Panos Ipeirotis (NYU Stern)
Date: Tuesday, Dec 15th
Time: 12-1pm
Room: GHC 6501

We describe Quizz, a gamified crowdsourcing system that simultaneously
assesses the knowledge of users and acquires new knowledge from them.
Quizz operates by asking users to complete short quizzes on specific
topics; as a user answers the quiz questions, Quizz estimates the
user’s competence. To acquire new knowledge, Quizz also incorporates
questions for which we do not have a known answer; the answers given
by competent users provide useful signals for selecting the correct
answers for these questions. Quizz actively tries to identify
knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging “for free” the targeting capabilities of
existing, publicly available, ad placement services. Quizz quantifies
the contributions of the users using information theory and sends
feedback to the advertising system about each user. The feedback
allows the ad targeting mechanism to further optimize ad placement.

Our experiments, which involve over ten thousand users, confirm that
we can crowdsource knowledge curation for niche and specialized
topics, as the advertising network can automatically identify users
with the desired expertise and interest in the given topic. We present
controlled experiments that examine the effect of various incentive
mechanisms, highlighting the need for having short-term rewards as
goals, which incentivize the users to contribute. Finally, our
cost-quality analysis indicates that the cost of our approach is below
that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of
potential users all over the planet, and being able to reach users
with specialized expertise that is not typically available through
existing labor marketplaces.

Panos Ipeirotis is an Associate Professor and George A. Kellner
Faculty Fellow at the Department of Information, Operations, and
Management Sciences at Leonard N. Stern School of Business of New York
University.  He received his Ph.D. degree in Computer Science from
Columbia University in 2004. He has received nine “Best Paper” awards
and nominations, a CAREER award from the National Science Foundation,
and is the  recipient of the 2015 Lagrange Prize in Complex Systems, for his
contributions in the field of social media, user-generated content and


Sunday, November 29, 2015

Title: Learnersourcing: Improving Learning with Collective Learner Activity
Speaker: Juho Kim (Stanford CS)
Date: Tuesday, Dec 1st
Time: 12-1pm
Room: GHC 6501

Millions of learners today are watching videos on online platforms, such as Khan Academy, YouTube, Coursera, and edX, to take courses and master new skills. But existing video interfaces are not designed to support learning, with limited interactivity and lack of information about learners' engagement and content. Making these improvements requires deep semantic information about video that even state-of-the-art AI techniques cannot fully extract. I take a data-driven approach to address this challenge, using large-scale learning interaction data to dynamically improve video content and interfaces. Specifically, my research introduces learnersourcing, a form of crowdsourcing in which learners collectively contribute novel content for future learners while engaging in a meaningful learning experience themselves. In this talk, I will present learnersourcing applications designed for massive open online course videos and how-to tutorial videos, where learners' collective activities (1) highlight points of confusion or importance in a video, (2) extract a solution structure from a tutorial, and (3) improve the navigation experience for future learners. I will then discuss how the idea of learnersourcing can generalize to broader educational and social contexts. My research demonstrates that learnersourcing can enable more interactive, collaborative, and data-driven learning.

Juho Kim is a Visiting Assistant Professor of Computer Science and a Brown Fellow at Stanford University. He will be an Assistant Professor in the School of Computing at KAIST starting fall 2016. His research interests lie in human-computer interaction, learning at scale, video interfaces, and crowdsourcing. He builds interactive systems powered by large-scale data from users, in which users’ natural and incentivized activities dynamically improve content, interaction, and experience. He earned his Ph.D. from MIT, M.S. from Stanford University, and B.S. from Seoul National University. He is a recipient of six paper awards from CHI and HCOMP, and the Samsung Fellowship.