Research model for editing video and audio jointly from a text instruction, e.g. changing speech with lip sync or voice.
Updated EU status checked Last shown
Sources: the vendor's own pages (linked above) and creator videos at the linked timestamps. EU status is information, not legal advice; see how we check.
InstructAV2AV is a research project on instruction-guided joint audio-video editing, from Haojie Zheng, Yixin Yang, Siqi Yang, Shuchen Weng and Boxin Shi at the Beijing Academy of Artificial Intelligence and Peking University. It takes an input video with its audio and a text instruction, and produces an edited video and audio together rather than editing each track separately.
According to a YouTube creator summary, it can change what a person says while keeping lip sync consistent, and change voice or gender. The project page shows demo videos across application scenarios and comparisons with other methods. The paper is cited as arXiv:2605.18467 (2026).
Code and weights are reportedly listed, but the GitHub page was said to be empty at the time of the video, so no public release is confirmed. Access is currently limited to viewing the demos on the project page.
Watch the moment: the exact second top creators show it. Videos load only when you press play, via youtube-nocookie.com.
System that edits video and audio together from a prompt, changing what a person says along with their lip sync, or changing voice and gender. Code and weights are listed but the GitHub page is still empty.
Closest neighbours by meaning, not just by category.
App · ElevenLabs
ElevenLabs' AI dubbing model that translates video and audio while carrying over the original speaker's voice and emotion.
Model
Video dubbing method that adapts LTX 2.3 with a LoRA to translate speech and re-sync lips in a single model.
Model · Baidu
Baidu ERNIE team's 6.3B joint audio-video generation model with native audio-visual alignment and timbre-controlled speech.
Model · IndexTeam
Open-weights zero-shot text-to-speech model from Bilibili's IndexTeam with voice cloning and emotion control in 5 languages.
App · HeyGen
AI video platform for creating avatar-led videos from text, with voice cloning and lip-synced translation in 175+ languages.
Model · Kandinsky Lab
Open-source diffusion models for synchronized 5-second video and audio generation, with a Full HD super-resolution add-on.
