ByteDance Develops OmniHuman, an AI Framework That Can Generate Reasonable Movies of People

ByteDance, the corporate behind TikTok, just lately shared its analysis on a brand new synthetic intelligence (AI) framework. Dubbed OmniHuman, it’s a video-generation framework that may create reasonable human movies with full-body motion and lip-syncing. The researchers said that it requires a human picture together with movement alerts resembling video or audio to generate output. A number of demonstration movies generated utilizing the AI mannequin have additionally been shared, showcasing the realism of the ultimate output. Notably, the corporate said that the AI mannequin is on the market within the public area.

OmniHuman Can Generate Reasonable Human Movies

The researchers shared a number of demonstrations and detailed the framework on its web site. It’s an end-to-end system that was constructed utilizing a novel multimodality movement conditioning blended coaching technique, the publish claimed. Whereas the researchers didn’t share any benchmark metrics, they claimed that the AI mannequin “considerably outperforms current strategies.”

OmniHuman can generate movies utilizing a picture of the individual and a movement sign. Movement alerts may be audio solely, video solely or a mixture of audio and video. The AI mannequin can generate reasonable movies based mostly on textual content prompts. These movies may be full-body the place the limbs, facial expressions, and lip motion may be synced with the audio or music taking part in within the background. OmniHuman can generate movies in several side ratios, permitting flexibility to customers.

OmniHuman output instance
Picture Credit score: OmniHuman

Using movement alerts is a novel method, which the corporate is asking omni-conditions coaching. With this, the AI mannequin is educated on totally different modalities, together with textual content, picture, audio, and video. Researchers mentioned this allowed the mannequin to study blended conditioning which overcame the shortage of high-quality information.

Notably, the mannequin was educated on 18,700 hours of human video information. The small print in regards to the coaching course of have been documented in a paper revealed within the on-line pre-print journal arXiv.

The corporate additionally shared a number of demonstrations of movies generated utilizing the mannequin, and the outcomes seem like extremely reasonable with pure physique actions, hand gestures, and lip actions. Such realism has additionally raised issues about deepfakes. Nonetheless, the corporate has specified that the AI mannequin is presently not accessible to be downloaded, and there’s no service individuals can use to entry its capabilities.