Blog #21: Human model evaluation in interactive supervised learning

 Human model evaluation in interactive supervised learning

Authors : Rebecca Fiebrink, Perry R. Cook, and Daniel Truema


Authors Bios:

• Rebecca Fiebrink:  assistant professor at Princeton University in the department of computer science and music and has a PhD from Princeton.

•  Perry R. Cook : professor emeritus at Princeton University and has a PhD from Stanford. 

• Daniel Truema : music courses at Princeton University and is an accomplished composer and performer on the fiddle and laptop.

Venue : This paper was presented at the CHI '11 Proceedings of the 2011 annual conference on Human factors in computing systems.

Summary

Hypothesis: 

Knowing why machines have certain outputs that are generate are one of many things that one may not get to understand. The researchers proposed a system that allows trainers to supervise machine learning because current ones provide little or no feedback. Which will allow them to find out why some problems may happen and they could fix those problems. They hypothesized that a system with the capabilities of knowing why certain output is generated, will allow us to make better machines. 

Methods: 

The researchers made a tool called, Wekinator, which would record physical gesters using musical intrements and label them. Then the researchers conducted three studies of people applying supervised learning to their work in computer music.

• The first study consisted of 7 composers working to refine the Wekinator to control new instruments that existed on the computer only and responded to gesture input. The participants trained the system once a week and made suggestions that were acted upon in between sessions.

• The second study focused on the supervised learning aspect of the system and observed 21 students in their use of the Wekinator to produce new instruments controlled by certain gestures (one continuously controlled adjusting to changes in gesture in real-time). The students' actions were recorded by the software and they filled out questionnaires.

• The final study consisted of a professional cellist working with the researchers to classify several gestures that capture many properties of a bow as it is used to play the cello. This bow, called the “K-Bow". After the system captures this data, it should be able to capture the notes being played and add them to a composition on a computer.

Results:

In the first study, participants found that the algorithms used to control the sound were difficult to control in a musically satisfying way using either a GUI or an explicitly controlled sequence.

The second study, users in the second study indicated that they considered high levels of cross validation accuracy to be indicative of good performance, and made use of it.

In the Third study, also used high levels of Cross validation. but direct evaluation was used more frequently in all 3 studies and allowed for quick validation that a system was performing as expected.

Concussion:
The researchers presented there work and concluded that supervised learning models intended to be ultimately used in interactive contexts and should be evaluated with attention to the model qualities important to users in those contexts. So basically, users must evaluate what qualities they are looking for beforehand because there simply isn't enough tools to know all.

Discussion:
The researchers did accomplish their goal of gathering useful data regarding evaluation of supervised learning systems, and I think that this work will be very beneficial in the future. Imagine writing a program or operating system that has no bugs!