GoKawiilVadims Casecnikovs✉, Gimran Abdullin, Raffi Hotter, Lev Chizhov
July 7, 2026
We trained a model to predict speech from ultrasound recordings of the tongue while the speaker remains silent. On open-vocabulary speech, our system achieves a 15.6% word error rate.
For comparison, lip-reading achieves 12.5% word error rate on a 1M hour dataset. We're excited that our system approaches existing methods despite being an early investigation trained on a 50-hour dataset and done in just a month.
We place an ultrasound probe behind the chin and capture videos like this of the tongue:
Your browser does not support the video tag.
And then we turn them into words. Here's a quick demo:
Your browser does not support the video tag.
A few days ago, we found out that the model also generalizes to new people (as long as they speak with an American accent 🇺🇸🦅11Our Eastern European friends were less thrilled.). Our friends were able to walk in, pick up a probe, and start using the system right away. We did not expect this to work with so little data.
Your browser does not support the video tag.
... continue reading