GoKawiilLAWRENCE — New research from the University of Kansas uses network science to determine why people make mistakes when lip reading.
Michael Vitevitch, professor of speech-language-hearing at KU, and his co-authors created a visual map of around 20,000 words in English, hoping to better grasp why some words are more difficult to lip-read than others.
The results appear in the Journal of the Acoustical Society of America. Findings could improve training for lip readers and boost the capacity for artificial intelligence to read lips and provide transcription and other digital services.
“What we looked at in this study is how people basically read lips, how accurate they are and, more specifically, what kinds of mistakes they make,” Vitevitch said. “A lot of previous work looked at how accurate people were and didn’t necessarily look at the characteristics of the errors themselves. There’s a lot to be learned from the mistakes you make, and that was the approach we took.”
While previous work on lip reading examined errors, much of that research was done by spoken-language researchers who focused on phonemes — the sounds in a language — and on how close participants were to the word as it sounds.
Vitevitch took a different approach.
“We focused on the visual characteristics,” he said. “Instead of looking at how many sounds of the word people got, we looked at how many of the visual characteristics, which we call ‘visemes’ (the visual equivalent of a phoneme), they got. We focused on what you’re getting from the lips, jaw and mouth without using auditory sound. You’re just trying to get the information from what you’re seeing.”
“How does that sound look when it’s spoken? We don’t care what it sounds like; we care about how it looks when it’s spoken,” he said. “Sometimes words sound similar and look similar, such as ‘kit,’ ‘cat’ and ‘cut.’ Other times words don’t sound alike but still look similar like ‘vet,’ ‘fit’ and ‘fuzz.’ In both cases if you’re just looking at my face, you couldn’t tell one word from the other.”
Through analysis of the word map, researchers determined:
People are more likely to mistake a word for another word used more commonly.
... continue reading