Judging Text-to-Speech by the Wisdom of the Crowd

Title: Judging Text-to-Speech by the Wisdom of the Crowd
Speaker: Prof. Alan Black (http://www.cs.cmu.edu/~awb/)
Date: Wednesday, Nov 30th @Noon
Room: GHC 6501

One of the many hard issues in generating good synthetic speech is the difficulty in evaluating the quality. Objective measures are always useful when optimizing various machine learning algorithms, but in speech generation it is ultimately what the end user actually thinks about the speech that is important. Running human listening tests is expensive, and not very reliable.

This talk lays out the techniques we've used to try to find robust subjective evaluation techniques for speech synthesis. These have been implemented in the annual Blizzard Challenge where teams build synthetic voices from a common dataset and then we have many people judge the quality by listening to them. The results are robust (different subsets of listeners correlate) and there have been interesting results found about the orthogonality of naturalness and intelligibility. However as we go further into using end users are an evaluation system we note a number of issues that must be addressed. People prefer voices they've listened to before (for good speech perceptual reasons). People are not good at judging subtle differences such as voice quality, intonation, timing etc; naturalness and intelligibility are not the only goals.

This talk will present existing crowd sourcing techniques used to evaluate speech synthesis and propose new techniques that might help use evaluate future directions in speech synthesis.

Towards Large-Scale Collaborative Planning using Humans and Machines

Title: Towards Large-Scale Collaborative Planning using Humans and Machines
Speaker: Edith Law
Room: GHC 4405 (@Noon)

Human computation is the study of systems where humans perform a major part of the computation or are an integral part of the overall computational process. There exist several genres of human computation systems -- Games With a Purpose (e.g., the ESP Game) collect data from humans as a by-product of game play; crowdsourcing marketplaces (e.g. Amazon Mechanical Turk) enable algorithmic operations to be outsourced to paid workers in the form of micro-tasks; identity verification tasks (e.g., reCAPTCHA) leverage the help of billions of users who, in the process of gaining access to online content, are engaged in meaningful activities (e.g., digitizing books).

To date, most human computation systems have simple output requirements (e.g., accuracy). In this talk, I will discuss two of my recent work exploring human computation tasks with complex output requirements. In the first case study, I present a human computation algorithm called CrowdPlan that, given a high-level search query (e.g., "I want to ...", "I need to ..."), generates a simple plan consisted of a set of goals and web resources that help support each goal. In the second case study, I introduce Mobi, a collaborative itinerary planning environment, that allows workers to asynchronously put together a complex plan (consisted of a sequence of ordered actions) that satisfies the given qualitative and quantitative constraints. These case studies demonstrate two contrasting solutions - using an explicit algorithm versus a social computing platform - for tackling problems with complex output requirements, and reveal the importance of communication between workers and the end users of the system (i.e., requesters) during the computational process.

Speaker Bio:
Edith Law is a Ph.D. candidate at Carnegie Mellon University, working
with Luis von Ahn and Tom Mitchell on human computation systems that
harness the joint efforts of machines and humans. She co-organized the
Human Computation (HCOMP) Workshops Series (co-located with KDD 2009,
2010 and AAAI 2011), co-authored the book "Human Computation"
published by Morgan & Claypool Synthesis Lectures in Artificial
Intelligence and Machine Learning, as well as presented a tutorial
entitled "Human Computation: Core Research Questions and State of the
Art" at AAAI 2011. Her work is generously supported by a Microsoft
Graduate Research Fellowship.