Microsoft patents AI technology to modify speech and faces in live video

Microsoft has filed a patent for an artificial intelligence system designed to modify video and audio of live performances in near real-time. The described technology goes far beyond simple audio track editing: AI agents will be expected to alter the speaker's image, including lip movements and facial expressions, to match the newly generated speech version.

• Microsoft patented an AI system to alter live video and audio in real-time.
• The technology synchronizes lip movements and facial expressions with new speech.
• Potential uses include translation, filler word removal, and content filtering.
• The patent includes saving original reference recordings for verification.

How the AI modification technology works

The system will be able to filter specific words and phrases, remove them, or replace them with others using predefined parameters or custom user profiles. To ensure seamless viewing, the algorithm must also adjust the video and audio immediately before and after the modified fragment so transitions look and sound natural.

Among the potential applications, Microsoft highlights the translation of live broadcasts into the viewer's preferred language. The system can also eliminate filler words, correct stuttering, and address other speech peculiarities. More controversial options listed in the patent include removing regional slang, incorrectly pronounced words, and inappropriate or offensive statements, with replacements potentially found via web search.

Application formats and safety measures

The technology is intended for various formats such as lectures, presentations, television programs, ceremonies, shows, and sports broadcasts. Viewers could theoretically receive not just a translation, but a personally edited version of a performance tailored to their specific settings.

The patent provides for storing original video and audio as a reference recording to help distinguish between original and altered broadcasts. However, the mere presence of such functionality does not completely eliminate the risk of confusion when modified recordings are distributed.