Joe Maring / Android Authority
TL;DR YouTube plans to launch AI-generated lip sync to automatically dubbed videos.
It may soon start testing the feature in English, French, German, Portuguese, and Spanish.
The feature may incur additional fees, which will likely have to be borne by users.
Whether you like it or not, Google isn’t backing down on its attempts to increase engagement by automatically translating audio into your native tongue. Now, following continuous and unwavering stabs at simply translating audio with autodubbing, YouTube may take it a notch higher with realistic-looking audio dubs with accurate lip sync in an attempt to make videos more appealing with AI.
YouTube’s product lead for Auto-dubbing, Buddhika Kottahachchi, recently detailed the technology behind lip-synced autodubs in a conversation with Digital Trends. Kottahachchi informed the publication that the system makes intricate pixel-level changes to modify the speaker’s mouth, ensuring the dub looks realistic. It is powered by a custom AI built to understand the nuances of facial structure, including 3D perception of the lips, teeth, and even different facial expressions. Looking at Google’s success with its Veo3 text-to-video model, we believe it should be able to accomplish the feat with convincing levels of realism.
At the moment, the lip sync feature only supports 1080p resolution and cannot process 4K videos. Language support will also be limited to English, French, German, Portuguese, and Spanish, but the company plans to expand it to over 20 languages — all of those currently supported by YouTube’s auto-dubbing feature.
Don’t want to miss the best from Android Authority? Set us as a favorite source in Google Discover to never miss our latest exclusive reports, expert analysis, and much more.
to never miss our latest exclusive reports, expert analysis, and much more. You can also set us as a preferred source in Google Search by clicking the button below.
YouTube doesn’t make any concrete promises on he feature’s availability, but it may begin this as a pilot with a small set of creators, as it originally did with the auto-dubbing functionality. Considering that it was rolled out to a broader set of users only last month, lip sync may take its own sweet time to become available across more channels. And while it is only expected to broaden reach, we can expect creators to have the option to turn the feature off for their channel or particular videos.
The feature might reportedly also come at an additional cost, though the exact figure hasn’t been finalized. Whether it’s the consumer or the channel that bears the price of the feature is also worth seeing, though it’s likely to be the former.
Naturally, concerns about automatic lip-syncs being misused are bound to arise, especially when other channels repost videos without proper authorization. In an attempt to combat that, YouTube intends to add a descriptive disclosure, along with an invisible fingerprint, similar to SynthID.
Notably, YouTube isn’t the only video platform with such plans in progress. Last year, Meta launched a pilot program with Instagram creators to dub and lip-sync their Reels into other languages. We have yet to learn more about its success, except that it was recently expanded to four languages, including English, Hindi, Portuguese, and Spanish.
Follow