Introduction
For my final project, I worked with Professor Jack Mostow and the Project LISTEN team (www.cs.cmu.edu/~listen/) to do some exploratory work for their product, the Reading Tutor.

Background
Project LISTEN is a research project at CMU developing a new software, the Reading Tutor, for improving literacy in children. The Reading Tutor intelligently generates stories and listens to children read, providing helpful feedback.

My Involvement
I was asked to create sketches for a possible new component of the Reading Tutor, which would explore using visual feedback to help children read sentences more fluently (not currently a feature of Reading Tutor). This involved using canned speech and analysis couples to experiment with the intent of becoming “live” at a later date.

The Challenges
The most challenging parts of this project was working with the raw data from speech analysis software making partial hypotheses (and later correcting them) as to what was being said, and doing signal processing on the data. Also it was really fun and stimulating to work with experts in a subject to develop ideas.

Prosodic Data
Prosody is the pattern of rhythm, stresses and intonations of a spoken sentence. Project LISTEN developed speech analysis software that understands what words have been said and measures pitch & intensity over time.
Game Mechanics

Game Mechanics
I emphasized an approach that would incorporate game mechanics to the visualization of prosody to engage and connect with children. Game mechanics provide incentives for improvement and reinforcement through rewards.

The Games

Other preliminary sketches

The Games
My sketches developed into a flexible system of “leveled” gameplay, which grows with the child’s abilities to provide a steady challenge. The framework provides a consistent objective (mimic the correct shape of the sentence) while slightly and intuitively mapping different game mechanisms to different visual scenarios.

Next Steps
I worked with the Project LISTEN team in the last few days of the semester by walking through my code together, so they can continue developing my sketches to hopefully be user tested this summer with local schools.

Thanks to Jack Mostow and Project LISTEN team for the great opportunity, essential guidance and accommodation, and Golan and Patrick for their help along the semester and for a fantastic class.

-David Yen