DrawAFriend: Crowdsourcing through Social Gaming

Title: DrawAFriend: Crowdsourcing through Social Gaming

Speaker: Alex Limpaecher (CSD, Carnegie Mellon)

Date: Wednesday, May 2nd

Time: 12-1pm

Room: GHC 6501

Abstract:

DrawAFriend explores how social game mechanics can be applied to crowdsourcing. DrawAFriend is a socially integrated drawing game that allows users to easily create drawings of their friends and share those drawings on the Facebook social graph. The project has two primary goals: First to create a unique, fun, and artistic experience for professionals and non professionals alike. Secondly to elicit a large database of human-created line drawings that we can later analyze. In this talk I will discuss the game design of DrawAFriend and the results that have come from a limited release.

Organizing Online Production without Formal Organization

Title: Organizing Online Production without Formal Organization
Speaker: Haiyi Zhu, HCII, CMU
Date: Wednesday, April 11th
Time: 12p-1p
Room: GHC 6501

Abstract:

Online production communities have successfully aggregated the efforts of millions of volunteers to produce complex artifacts such as GNU/Linux and Wikipedia. Currently most online production communities rely on a paradigm of self-direction in which people work primarily on the tasks they are interested in. However, this approach breaks down when there are conflicts between the interests of the individuals and the goal of the community as a whole. Many people may want to work on the same popular areas while ignoring less popular areas that require work. People may not want to perform cooperative behaviors (e.g., performing maintenance tasks or socializing newcomers), even though these behaviors are important for the healthy functioning of the community. Therefore, the challenge has become how to motivate people to achieve the community goal that transcends individual interest in an environment which lacks hierarchical structure and monetary incentives. I identified particular mechanisms, including group identification and shared leadership, which intrinsically influence people’s actions to achieve a common goal. I empirically examined their effectiveness in the context of Wikipedia. My research has implications for designing more effective, efficient and successful online production communities.

A Review of Tiramisu - Extending Transportation Information Systems with

Title: A Review of Tiramisu - Extending Transportation Information Systems with Crowdsourcing
Speaker: Anthony Tomasic, Institute for Software Research, CMU
Date: Wednesday, Feb 8th
Time: 12p-1p
Room: GHC 6501

Abstract:

Tiramisu ("pick me up" in Italian) is a prototype implementation of a transportation information system that learns via crowdsourcing information from users. Through a smart phone interface, transit riders search for transit information. Once on a bus, riders contribute new information to Tiramisu through the same interface. The system also leverages the instrumentation of the smart phone to create an automatic vehicle location (AVL) service. Tiramisu is currently deployed in the Pittsburgh region. The system serves as a test bed for a variety of research areas: crowdsourcing system design, universal (accessibility) design, applied machine learning, and service design. In this talk, we will discuss our initial design rational, review research results to date, cry mea culpa over design mistakes and present some new preliminary results. If time permits, we will discuss some future directions.

Joint work with Charlie Garrod (CSD/Swarthmore), Yun Huang (RI), Aaron Steinfeld (RI), John Zimmerman (HCII & Design) and many others.

Shepherding the Crowd Yields Better Work

Title: Shepherding the Crowd Yields Better Work
Speaker: Steven Dow, HCII, CMU
Date: Wednesday, Jan 25th
Time: 12p-1p
Room: NSH 3305

Abstract:

Micro-task platforms provide massively parallel, on-demand labor. However, it can be difficult to reliably achieve high-quality work because online workers may behave irresponsibly, misunderstand the task, or lack necessary skills. This paper investigates whether timely, task-specific feedback helps crowd workers learn, persevere, and produce better results. We investigate this question through Shepherd, a feedback system for crowdsourced work. In a between-subjects study with three conditions, crowd workers wrote consumer reviews for six products they own. Participants in the None condition received no immediate feedback, consistent with most current crowdsourcing practices. Participants in the Self-assessment condition judged their own work. Participants in the External assessment condition received expert feedback. Self-assessment alone yielded better overall work than the None condition and helped workers improve over time. External assessment also yielded these benefits. Participants who received external assessment also revised their work more. We conclude by discussing interaction and infrastructure approaches for integrating real-time assessment into online work.

EteRNA - solving RNA design problem

Title: EteRNA - solving RNA design problem with 30,000 people http://eterna.cmu.edu
Speaker: Jeehyung Lee, CS, CMU
Date: Wednesday, Dec 14th
Room: GHC 4405

Abstract:
We introduce EteRNA, an Internet-based RNA design competition where
players design RNA sequence to match given target shapes, and receive
information-rich wet-lab feedback from high-throughput RNA synthesis and
chemical mapping. We show that players were able to uncover rules for
robust RNA design from continuous wet-lab feedback.

Judging Text-to-Speech by the Wisdom of the Crowd

Title: Judging Text-to-Speech by the Wisdom of the Crowd
Speaker: Prof. Alan Black (http://www.cs.cmu.edu/~awb/)
Date: Wednesday, Nov 30th @Noon
Room: GHC 6501

Abstract:
One of the many hard issues in generating good synthetic speech is the difficulty in evaluating the quality. Objective measures are always useful when optimizing various machine learning algorithms, but in speech generation it is ultimately what the end user actually thinks about the speech that is important. Running human listening tests is expensive, and not very reliable.

This talk lays out the techniques we've used to try to find robust subjective evaluation techniques for speech synthesis. These have been implemented in the annual Blizzard Challenge where teams build synthetic voices from a common dataset and then we have many people judge the quality by listening to them. The results are robust (different subsets of listeners correlate) and there have been interesting results found about the orthogonality of naturalness and intelligibility. However as we go further into using end users are an evaluation system we note a number of issues that must be addressed. People prefer voices they've listened to before (for good speech perceptual reasons). People are not good at judging subtle differences such as voice quality, intonation, timing etc; naturalness and intelligibility are not the only goals.

This talk will present existing crowd sourcing techniques used to evaluate speech synthesis and propose new techniques that might help use evaluate future directions in speech synthesis.

Towards Large-Scale Collaborative Planning using Humans and Machines

Title: Towards Large-Scale Collaborative Planning using Humans and Machines
Speaker: Edith Law
Room: GHC 4405 (@Noon)


Human computation is the study of systems where humans perform a major part of the computation or are an integral part of the overall computational process. There exist several genres of human computation systems -- Games With a Purpose (e.g., the ESP Game) collect data from humans as a by-product of game play; crowdsourcing marketplaces (e.g. Amazon Mechanical Turk) enable algorithmic operations to be outsourced to paid workers in the form of micro-tasks; identity verification tasks (e.g., reCAPTCHA) leverage the help of billions of users who, in the process of gaining access to online content, are engaged in meaningful activities (e.g., digitizing books).

To date, most human computation systems have simple output requirements (e.g., accuracy). In this talk, I will discuss two of my recent work exploring human computation tasks with complex output requirements. In the first case study, I present a human computation algorithm called CrowdPlan that, given a high-level search query (e.g., "I want to ...", "I need to ..."), generates a simple plan consisted of a set of goals and web resources that help support each goal. In the second case study, I introduce Mobi, a collaborative itinerary planning environment, that allows workers to asynchronously put together a complex plan (consisted of a sequence of ordered actions) that satisfies the given qualitative and quantitative constraints. These case studies demonstrate two contrasting solutions - using an explicit algorithm versus a social computing platform - for tackling problems with complex output requirements, and reveal the importance of communication between workers and the end users of the system (i.e., requesters) during the computational process.

Speaker Bio:
Edith Law is a Ph.D. candidate at Carnegie Mellon University, working
with Luis von Ahn and Tom Mitchell on human computation systems that
harness the joint efforts of machines and humans. She co-organized the
Human Computation (HCOMP) Workshops Series (co-located with KDD 2009,
2010 and AAAI 2011), co-authored the book "Human Computation"
published by Morgan & Claypool Synthesis Lectures in Artificial
Intelligence and Machine Learning, as well as presented a tutorial
entitled "Human Computation: Core Research Questions and State of the
Art" at AAAI 2011. Her work is generously supported by a Microsoft
Graduate Research Fellowship.