VASA-1: Lifelike Audio-Driven Talking Faces
Generated in Real Time

Abstract

We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only producing lip movements that are exquisitely synchronized with the audio, but also capturing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a diffusion-based holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos. Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method delivers high video quality with realistic facial and head dynamics and also supports the online generation of 512$\times$512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors. Project webpage: https://www.microsoft.com/en-us/research/project/vasa-1/

1 Introduction

In the realm of multimedia and communication, the human face is not just a visage but a dynamic canvas, where every subtle movement and expression can articulate emotions, convey unspoken messages, and foster empathetic connections. The emergence of AI-generated talking faces offers a window into a future where technology amplifies the richness of human-human and human-AI interactions. Such technology holds the promise of enriching digital communication wang2021one ; ma2021pixel , increasing accessibility for those with communicative impairments johnson2018assessing ; prnewswire2024deepbrain , transforming education methods with interactive AI tutoring bozkurt2023speculative ; kessler2018technology , and providing therapeutic support and social interaction in healthcare rehm2016role ; leff2014avatar .

As one step towards achieving such capabilities, our work introduces VASA-1, a new method that can produce audio-generated talking faces with a high level of realism and liveliness. Given a static face image of an arbitrary individual, alongside a speech audio clip from any person, our approach is capable of generating a hyper-realistic talking face video efficiently. This video not only features lip movements that are meticulously synchronized with the audio input but also exhibits a wide range of natural, human-like facial dynamics and head movements.

Creating talking faces from audio has attracted significant attention in recent years with numerous approaches proposed zhou2020makelttalk ; prajwal2020lip ; zhang2021flow ; sun2021speech2talking ; guo2021ad ; wang2021audio2head ; wang2022one ; wang2023lipformer ; yu2023talking ; zhang2023sadtalker ; ma2023styletalk ; he2024gaia . However, existing techniques are still far from achieving the authenticity of natural talking faces. Current research has predominantly focused on the precision of lip synchronization with promising accuracy obtained prajwal2020lip ; wang2023lipformer . The creation of expressive facial dynamics and the subtle nuances of lifelike facial behavior remain largely neglected. This results in generated faces that seem rigid and unconvincing. Additionally, natural head movements also play a vital role in enhancing the perception of realism. Although recent studies have attempted to simulate realistic head motions wang2021audio2head ; yu2023talking ; zhang2023sadtalker , there remains a sizable gap between the generated animations and the genuine human movement patterns.

Figure 1: Given a single portrait image, a speech audio clip, and optionally a set of other control signals, our approach produces a high-quality lifelike talking face video of 512$\times$ 512 resolution at up to 40 FPS. The method is generic and robust, and the generated talking faces can faithfully mimic human facial expressions and head movements, reaching a high level of realism and liveliness. (All the photorealistic portrait images in this paper are virtual, non-existing identities generated by karras2020analyzing ; betker2023improving . See our project page for the generated video samples with audios.)

Another important factor is the efficiency of generation, which plays a pivotal role in real-time applications such as live communication. While image and video diffusion techniques have brought remarkable advancements in talking face generation du2023dae ; stypulkowski2024diffused ; tian2024emo as well as the broader video generation field blattmann2023stable ; videoworldsimulators2024 , the substantial computation demands have limited their practicality for interactive systems. A critical need exists for optimized algorithms that can bridge the gap between high-quality video synthesis and the low-latency requirements of real-time applications.

Given the limitations of existing methods, this work develops an efficient yet powerful audio-conditioned generative model that works in the latent space of head and facial movements. Different from prior works, we train a Diffusion Transformer model on the latent space of holistic facial dynamics as well as head movements. We consider all possible facial dynamics – including lip motion, (non-lip) expression, eye gaze and blinking, among others – as a single latent variable and model its probabilistic distribution in a unified manner. By contrast, existing methods often apply separate models for different factors, even with interleaved regressive and generative formulations for them wang2021audio2head ; zhou2021pose ; yu2023talking ; wang2023progressive ; zhang2023sadtalker . Our holistic facial dynamics modeling, together with the jointly learned head motion patterns, leads to the generation of a diverse array of lifelike and emotive talking behaviors. Furthermore, we incorporate a set of optional conditioning signals such as main gaze direction, head distance, and emotion offset into the learning process. This makes the generative modeling of complex distribution more tractable and increases the generation controllability.

To achieve our goal, another challenge lies in constructing the latent space for the aforementioned holistic facial dynamics and gathering the data for the diffusion model training. Beyond facial and head movements, a human face image contains other factors such as identity and appearance. In this work, we seek to build a proper latent space for human face using a large volume of face videos . Our aim is for the face latent space to possess both a total state of disentanglement between facial dynamics and other factors, as well as a high degree of expressiveness to model rich facial appearance details and dynamic nuances. We base our method on the 3D-aided representation wang2021one ; drobyshev2022megaportraits and equip it with a collection of carefully-designed loss functions. Trained on face videos in an self-supervised or weakly-supervised manner, our encoder can produce well-disentangled factors including 3D appearance, identity, head pose and holistic facial dynamics, and the decoder can generate high quality faces following the given latent codes.

VASA-1 has collectively advanced the realism of lip-audio synchronization, facial dynamics, and head movement to new heights. Coupled with high image generation quality and efficient running speed, we achieved real-time talking faces that are realistic and lifelike. Through detailed evaluations, we show that our method significantly outperforms existing methods. We believe VASA-1 brings us closer to a future where digital AI avatars can engage with us in ways that are as natural and intuitive as interactions with real humans, demonstrating appealing visual affective skills for more dynamic and empathetic information exchange.

2 Related Work

Disentangled face representation learning.

The representation of facial images through disentangled variables has been extensively studied by previous works. Some methods utilize sparse keypoints siarohin2019first ; zakharov2020fast or 3D face models ren2021pirenderer ; gao2023high ; zhang2023metaportrait to explicitly characterize facial dynamics and other properties, but these can suffer from issues such as inaccurate reconstructions or limited expressive capabilities. There are also many works dedicated to learning disentangled representations within a latent space. A common approach involves separating faces into identity and non-identity components, then recombining them across different frames, either in a 2D burkov2020neural ; zhou2021pose ; liang2022expressive ; yin2022styleheat ; pang2023dpe ; wang2023progressive or 3D context wang2021one ; drobyshev2022megaportraits . The main challenge faced by these methods is the effective disentanglement of various factors while still achieving expressive representations of all static and dynamic facial attributes, which is addressed in this work.

Audio-driven talking face generation.

Talking face video generation from audio inputs has been a long-standing task in computer vision and graphics. Early works have focused on synthesizing only the lips, achieved by mapping audio signals directly to lip movements while leaving other facial attributes unchanged suwajanakorn2017synthesizing ; chen2018lip ; prajwal2020lip ; yin2022styleheat ; cheng2022videoretalking . More recent efforts have expanded the scope to include a broader array of facial expressions and head movements derived from audio inputs. For instance, the method of zhang2023sadtalker separates the generation targets into different categories, including lip-only 3DMM coefficients, eye blinks, and head poses. yu2023talking proposed to decompose lip and non-lip features on the top of the expression latent from zhou2021pose . Both zhang2023sadtalker and yu2023talking regress lip-related representations directly from audio features and model other attributes in a probabilistic manner. In contrast to these approaches, our method generates comprehensive facial dynamics and head poses from audio along with other control signals. This approach differs from the trend of further disentanglement, seeking instead to create more holistic and integrated outputs.

Video generation.

Recent advances in generative models brown2020language ; ho2020denoising ; song2020score ; song2020denoising have led to significant progress in video generation. Earlier video generation approaches vondrick2016generating ; tulyakov2018mocogan ; skorokhodov2022stylegan employed the adversarial learning goodfellow2014generative framework, while more recent methods yan2021videogpt ; blattmann2023align ; girdhar2023emu ; kondratyuk2023videopoet ; bar2024lumiere ; videoworldsimulators2024 have leveraged diffusion or auto-regressive models to capture diverse video distributions. Recently, several works concurrent to us tian2024emo ; wei2024aniportrait have adapted video diffusion techniques to audio-driven talking face generation, achieving promising results despite the slow training and inference speeds. In contrast, our method can deliver both efficiency and high-quality results in the generation of talking face videos.

3 Method

Task definition.

As illustrated in Fig. 1, The input to our method consists of a single face image $𝐈$ of an arbitrary identity and a speech audio clip $𝐚$ from, again, an arbitrary person. The goal is to generate a synthesized video of the input face image speaking with the given audio in a realistic and coherent manner. A successfully generated video should exhibit high fidelity in several key aspects: the clarity and authenticity of the image frames, precise synchronization between the audio and lip movements, expressive and emotive facial dynamics, and naturalistic head poses.

Our generation process can also accept a set of optional control signals to guide the generation, which include the main eye gaze direction $𝐠$, head-to-camera distance $d$, and emotion offset $𝐞$. More details will be provided in the later sections.

Overall framework.

Instead of generating video frames directly, we generate holistic facial dynamics and head motion in the latent space conditioned on audio and other signals. Given these motion latent codes, our method then produces video frames by a face decoder, which also takes the appearance and identity features extracted using a face encoder from the input image as input.

To achieve this, we start by constructing a face latent space and training the face encoder and decoder. An expressive and disentangled face latent learning framework is crafted and trained on real-life face videos. Then we train a simple yet powerful Diffusion Transformer to model the motion distribution and generate the motion latent codes in the test time given audio and other conditions.

3.1 Expressive and Disentangled Face Latent Space Construction

Given a corpus of unlabeled talking face videos, we aim to build a latent space for human face with high degrees of disentanglement and expressiveness. The disentanglement enables effective generative modeling of the human head and holistic facial behaviors on massive videos, irrespective of the subject identities. It also enables disentangled factor control of the output which is desirable in many applications. Existing methods fall short of either expressiveness burkov2020neural ; ren2021pirenderer ; yu2023talking ; wang2023progressive or disentanglement wang2021one ; drobyshev2022megaportraits ; zhang2023metaportrait or both. The expressiveness of facial appearance and dynamic movements, on the other hand, ensures that the decoder can output high quality videos with rich facial details and the latent generator is able to capture nuanced facial dynamics.

To achieve this, we base our model on the 3D-aid face reenactment framework from wang2021one ; drobyshev2022megaportraits . The 3D appearance feature volume can better characterize the appearance details in 3D compared to 2D feature maps. The explicit 3D feature warping is also powerful in modeling 3D head and facial movements. Specifically, we decompose a facial image into a canonical 3D appearance volume ${𝐕}^{app}$, an identity code ${𝐳}^{id}$, a 3D head pose ${𝐳}^{pose}$, and a facial dynamics code ${𝐳}^{dyn}$. Each of them is extracted from a face image by an independent encoder, except that ${𝐕}^{app}$ is constructed by first extracting a posed 3D volume followed by rigid and non-rigid 3D warping to the canonical volume, as done in drobyshev2022megaportraits . A single decoder $𝒟$ takes these latent variables as input and reconstructs the face image, where similar warping fields in the inverse direction are first applied to ${𝐕}^{app}$ to get the posed appearance volume. Readers are referred to drobyshev2022megaportraits for more details of this architecture.

To learn the disentangled latent space, the core idea is to construct image reconstruction loss by swapping latent variables between different images in videos. Our basic loss functions are adapted from drobyshev2022megaportraits . However, we identified the poor disentanglement between facial dynamics and head pose using the original losses. The disentanglement between identity and motions is also imperfect. Therefore, we introduce several additional losses crucial to achieve our goal. For instance, inspired by pang2023dpe , we add a pairwise head pose and facial dynamics transfer loss to improve their disentanglement. Let ${𝐈}_i$ and ${𝐈}_j$ be two frames randomly sampled from the same video of a subject. We extract their latent variables using the encoders, and transfer ${𝐈}_i$’s head pose onto ${𝐈}_j$ as ${\widehat{𝐈}}_{j,{𝐳}_i^{pose}}=𝒟({𝐕}_j^{app},{𝐳}_j^{id},{𝐳}_i^{pose},{𝐳}_j^{dyn})$ and ${𝐈}_j$’s facial motion onto ${𝐈}_i$ as ${\widehat{𝐈}}_{i,{𝐳}_j^{dyn}}=𝒟({𝐕}_i^{app},{𝐳}_i^{id},{𝐳}_i^{pose},{𝐳}_j^{dyn})$. The discrepancy loss between ${\widehat{𝐈}}_{j,{𝐳}_i^{pose}}$ and ${\widehat{𝐈}}_{i,{𝐳}_j^{dyn}}$ is subsequently minimized. To reinforce the disentanglement between identity and motions, we add a face identity similarity loss for the cross-identity pose and facial motion transfer results. Let ${𝐈}_s$ and ${𝐈}_d$ be the video frames of two different subjects, we can transfer the motions of ${𝐈}_d$ onto ${𝐈}_s$ and obtain ${\widehat{𝐈}}_{s,{𝐳}_d^{pose},{𝐳}_d^{dyn}}=𝒟({𝐕}_s^{app},{𝐳}_s^{id},{𝐳}_d^{pose},{𝐳}_d^{dyn})$. Then, a cosine similarity loss between the deep face identity features deng2019arcface extracted from ${𝐈}_s$ and ${\widehat{𝐈}}_{s,{𝐳}_d^{pose},{𝐳}_d^{dyn}}$ is applied.

Figure 2: Our holistic facial dynamics and head pose generation framework with diffusion transformer.

3.2 Holistic Facial Dynamics Generation with Diffusion Transformer

Given the constructed face latent space and trained encoders, we can extract the facial dynamics and head movements from real-life talking face videos and train a generative model. Crucially, we consider identity-agnostic holistic facial dynamics generation (HFDG), where our learned latent codes represent all facial movements such as lip motion, (non-lip) expression, and eye gaze and blinking. This is in contrast to existing methods that apply separate models for different factors with interleaved regression and generative formulations wang2021audio2head ; zhou2021pose ; yu2023talking ; wang2023progressive ; zhang2023sadtalker . Furthermore, previous methods often train on a limited number of identities zhang2023sadtalker ; xing2023codetalker ; fan2022faceformer and cannot model the wide range of motion patterns of different humans, especially given an expressive motion latent space.

In this work, we utilize diffusion models for audio-conditioned HFDG and train on massive talking face videos from a large number of identities. In particular, we apply a transformer architecture vaswani2017attention ; peebles2023scalable ; sun2023diffposetalk for our sequence generation task. Figure 2 shows an overview of our HFDG framework.

Formally, a motion sequence extracted from a video clip is defined as $𝐗=\{[{𝐳}_i^{pose},{𝐳}_i^{dyn}]\},i=1,\mathrm{\dots },W$. Given its accompanying audio clip $𝐚$, we extract the synchronized audio features $𝐀=\{{𝐟}_i^{audio}\}$, for which we use a pretrained feature extractor Wav2Vec2 baevski2020wav2vec .

Diffusion formulation.

Diffusion models define two Markov chains ho2020denoising ; song2020denoising ; song2020score , the forward chain progressively adds Gaussian noise to the target data, while the reverse chain iteratively restores the raw signal from noise. Following the denoising score matching objective song2020score , we define the simplified loss function as

$${𝔼}_{t\sim 𝒰[1,T],{𝐗}^0,𝐂\sim q({𝐗}^0,𝒞)}({\Vert {𝐗}^0-\mathscr{H}({𝐗}^t,t,𝐂)\Vert }^2),$$ (1)

where $t$ denotes the time step, ${𝐗}^0=𝐗$ is the raw motion latent sequence, and ${𝐗}^t$ is the noisy inputs generated by the diffusion forward process $q({𝐗}^t|{𝐗}^{t-1})=𝒩({𝐗}^t;\sqrt{1-{\beta }_t}{𝐗}^{t-1},{\beta }_t\mathrm{I})$. $\mathscr{H}$ is our transformer network which predicts the raw signal itself instead of noise. $𝐂$ is the condition signal, to be described next.

Conditioning signals.

The primary condition signal for our audio-driven motion generation task is the audio feature sequence $𝐀$. We also incorporate several additional signals, which not only make the generative modeling more tractable but also increase the generation controllability.

Specifically, we consider the main eye gaze direction $𝐠$, head-to-camera distance $d$, and emotion offset $𝐞$. The main gaze direction, $𝐠=(\theta ,\varphi )$, is defined by a vector in spherical coordinates. It specifies the focused direction of the generated talking face. We extract $𝐠$ for the training video clips using zemblys2019gazenet on each frame followed by a simple histogram-based clustering algorithm. The head distance $d$ is a normalized scalar controlling the distance between the face and the virtual camera, which affects the face scale in the generated face video. We obtain this scale label for the training videos using deng2019accurate . The emotion offset $𝐞$ modulates the depicted emotion on the talking face. Note that emotion is often intrinsically linked to and can be largely inferred from audio; hence, $𝐞$ serves only as a global offset added to enhance or moderately alter the emotion when required. It is not designed to achieve a total emotion shift during inference or produce emotions incongruent with the input audio. In practice, we use the averaged emotion coefficients extracted by savchenko2022hsemotion as our emotion signal.

In order to achieve a seamless transition between adjacent windows, we incorporate the last $K$ frames of the audio feature and generated motions from the previous window as the condition of the current one. To summarize, our input condition can be denoted as $𝐂=[{𝐗}^{pre},{𝐀}^{pre};𝐀,𝐠,d,𝐞]$

Classifier-free guidance (CFG) ho2022classifier .

In the training stage, we randomly drop each of the input conditions. During inference, we apply

${\widehat{𝐗}}^0=(1+{\displaystyle \sum _{𝐜\in 𝐂}}{\lambda }_{𝐜})\cdot \mathscr{H}({𝐗}^t,t,𝐂)-{\displaystyle \sum _{𝐜\in 𝐂}}{\lambda }_c\cdot \mathscr{H}({𝐗}^t,t,{𝐂|}_{𝐜=\mathrm{\varnothing }})$ (2)

where ${\lambda }_{𝐜}$ is the CFG scale for condition $𝐜$. ${𝐂|}_{𝐜=\mathrm{\varnothing }}$ denotes that the condition $𝐜$ is replaced with $\mathrm{\varnothing }$.

During training, we use a drop probability of $0.1$ for each condition except for ${𝐗}^{pre}$ and ${𝐀}^{pre}$ for which we use $0.5$. This is to ensure the model can well handle the first window with no preceding audio and motions (i.e., set to $\mathrm{\varnothing }$). We also randomly drop the last few frames of $𝐀$ to ensure robust motion generation for audio sequences shorter than the window length.

3.3 Talking Face Video Generation

At inference time, given an arbitrary face image and an audio clip, we first extract the 3D appearance volume ${𝐕}^{app}$ and identity code ${𝐳}^{id}$ using our trained face encoders. Then, we extract the audio features, split them into segments of length $W$, and generate the head and facial motion sequences $\{𝐗=\{[{𝐳}_i^{pose},{𝐳}_i^{dyn}]\}\}$ one by one in a sliding-window manner using our trained diffusion transformer $\mathscr{H}$. The final video can be generated subsequently using our trained decoder.

4 Experiments

Implementation details.

For face latent space learning, we use the public VoxCeleb2 dataset from chung2018voxceleb2 which contains talking face videos from about 6K subjects. We reprocess the dataset and discard the clips with multiple individuals and those of low quality using the method of su2020blindly . For motion latent generation, we use an 8-layer transformer encoder with an embedding dim $512$ and head number $8$ as our diffusion network. The model is trained on VoxCeleb2 chung2018voxceleb2 and another high-resolution talk video dataset collected by us, which contains about 3.5K subjects. In our default setup, the model uses a forward-facing main gaze condition, an average head distance of all training videos, and an empty emotion offset condition. The CFG parameters are set to ${\lambda }_{𝐀}=0.5$ and ${\lambda }_{𝐠}=1.0$, and $50$ sampling steps are used.

Evaluation benchmarks.

We evaluate our method using two datasets. The first is a subset of VoxCeleb2 chung2018voxceleb2 . We randomly selected 46 subjects from the test split of VoxCeleb2 and randomly sampled 10 video clips for each subject, resulting in a total of 460 clips. These video clips are about 5$\sim$15 seconds long (80% are less than 10 seconds), with most of the content being interviews and news reports. To further evaluate our method under long speech generation with a wider range of vocal variations, we further collected 32 one-minute clips of 17 individuals. These videos are predominantly sourced from online coaching sessions and educational lectures and the talking styles are considerably more diverse than VoxCeleb2. We refer to this dataset as OneMin-32.

Figure 3: Generated talking faces under different control signals. Top row: results under different main gaze direction condition (forward-facing, leftwards, rightwards, and upwards, respectively). Middle row: results under different head distances (from far to near). Bottom row: results under different emotion offset (neutral, happy, angry and surprised, respectively).

Figure 4: Disentanglement between identity and motion. In these examples, the same generated head and facial motion sequences are applied onto three different face images.

Figure 5: Disentanglement between head pose and facial dynamics. From top to bottom: the raw generated sequence, applying generated poses with fixed initial facial dynamics, and applying generated facial dynamics with fixed initial head pose and pre-defined spinning poses, respectively.

4.1 Qualitative Evaluation

Visual results.

Figure 1 presents some representative audio-driven talking face generation results of our method. Visually inspected, our method can generate high-quality video frames with vivid facial emotions. Moreover, it can generate human-like conversational behaviors, including sporadic shifts in eye gaze during speech and contemplation, as well as the natural and variable rhythm of eye blinking, among other nuances. We highly recommend that readers view our video results online to fully perceive the capabilities and output quality of our method.

Generation controllability.

Figure 3 shows our generated results under different control signals including main eye gaze, head distance, and emotion offset. It is evident that our generation model can well interpret these signals and produce talking face results that closely adhere to these specified parameters.

Disentanglement of face latents.

Figure 4 shows that when applying the same motion latent sequences onto different subjects, our method effectively maintains both the distinct facial movements and the unique facial identities. This indicates the efficacy of our method in disentangling identity and motion. Figure 5 further illustrates the effective disentanglement between head pose and facial dynamics. By holding one aspect constant and changing the other, the resulting images faithfully reflect the intended head and facial motions without interference.

Out-of-distribution generation.

Our method exhibits the capability to handle photo and audio inputs that fall outside the training distribution. For instance, as shown in Figure 6, it can handle artistic photos, singing audio clips (top two rows), and non-English speech (the last row). Notably, these data variants were not present in the training dataset.

Figure 6: Generation results with out-of-distribution images (non-photorealistic) and audios (singing audios for the first two rows and non-English speech for the last row). Our method can still generate high quality videos well-aligned with the audios, although it was not trained on such data variations.

4.2 Quantitative Evaluation

Evaluation metrics.

We use the following metrics for quantitative evaluation of our generated lip movement, head pose and overall video quality, including a new data-driven audio-pose synchronization metric trained in a way similar to CLIP radford2021learning :

• Audio-lip synchronization. We use a pretrained audio-lip synchronization network, i.e., SyncNet chung2017out , to assess the alignment of the input audio with the generated lip movements in videos. Specifically, we compute the confidence score and feature distance as $S_C$ and $S_D$ respectively. Higher $S_C$ and lower $S_D$ indicate better audio-lip synchronization quality in general.

• Audio-pose alignment. Measuring the alignment between the generated head poses and input audio is not trivial and there are no well-established metrics. A few recent studies zhang2023sadtalker ; sun2023diffposetalk employed the Beat Align Score siyao2022bailando to evaluate audio-pose alignment. However, this metric is not optimal because the concept of a “beat” in the context of natural speech and human head motion is ambiguous. In this work, we introduce a new data-driven metric called Contrastive Audio and Pose Pretraining (CAPP) score. Inspired by CLIP radford2021learning , we jointly train a pose sequence encoder and an audio sequence encoder and predict whether the input pose sequence and audio are paired. The audio encoder is initialized from a pretrained Wav2Vec2 network baevski2020wav2vec and the pose encoder is a randomly initialized 6-layer transformer network. The input window size is 3 seconds. Our CAPP model is trained on 2K hours of real-life audio and pose sequences, and demonstrates a robust capability to assess the degree of synchronization between audio inputs and generate poses (see Sec. 4.3).

• Pose variation intensity. We further define a pose variation intensity score $\mathrm{\Delta }P$ which is the average of the pose angle differences between adjacent frames. Averaged over all the frames of all generated videos, $\mathrm{\Delta }P$ provides an indication of the overall head motion intensity generated by a method.

• Video quality. Following previous video generation works yan2021videogpt ; skorokhodov2022stylegan , we use the Fréchet Video Distance (FVD) unterthiner2019fvd to evaluate the generated video quality. We compute the FVD metric using sequences of 25 consecutive frames.

Compared methods.

We compare our method with there existing audio-driven talking face generation methods: MakeItTalk zhou2020makelttalk , Audio2Head wang2021audio2head ; zhang2023sadtalker , and SadTalker zhang2023sadtalker . MakeItTalk zhou2020makelttalk employs an LSTM to convert audio into dynamic facial landmarks, then use the landmarks to animate a source image into a video sequence through either image warping or neural network based image translation. Audio2Head wang2021audio2head uses a motion-aware recurrent network to translate audio into head poses, which, along with the original audio, are used to generate dense motion fields. The motion fields are further applied to the image features and the final output is then generated by a neural network. SadTalker zhang2023sadtalker employs a VAE network to generate pose offsets from audio and a regression network to predict lip-only coefficients from audio features. A random variable is used for eye blink generation. This method can generate varied poses and eye blinks from identical audio inputs but only regress a deterministic pattern for other motions such as eyebrow, gaze and facial expressions.

Main results.

For each audio input, we generate a single video for deterministic approaches, i.e., MakeItTalk and Audio2Head. For SadTalker zhang2023sadtalker and our method, we sample three videos for each audio and average the computed metrics. Since different pose representations are used by these methods, we re-extract the head poses from the generated frames to compute the pose-related metrics (i.e., CAPP and $\mathrm{\Delta }P$). For the FVD metric, we use 2K 25-frame video clips of both the real videos and generated ones. For reference purpose, we also report the evaluated metrics of real videos.

Table 1 and Table 2 present the results on the VoxCeleb2 and OneMin-32 benchmarks. Note that we did not evaluate the FVD on VoxCeleb2 as its video quality is varied and often low. On both benchmarks, our method achieves the best results among all methods on all evaluated metrics. In terms of audio-lip synchronization scores ($S_C$ and $S_D$), our method outperforms all others by a wide margin. Note that our method yields better scores than real videos, which is due to effect of the audio CFG (see Sec. 4.3). Our generated poses are better aligned with the audios especially on the OneMin-32 benchmark, as reflected by the CAPP scores. The head movements also exhibit the highest intensity according to $\mathrm{\Delta }P$, although there’s still a gap to the intensity of real videos. Our FVD score is significantly lower than others, demonstrating the much higher video quality and realism of our results.

Table 1: Comparison with previous methods on the VoxCeleb2 benchmark.

	$S_C$ $\uparrow$	$S_D$ $\downarrow$	CAPP$\uparrow$	$\mathrm{\Delta }$P
MakeItTalk zhou2020makelttalk	4.176	15.513	-0.051	0.210
Audio2Head wang2021audio2head	6.172	8.470	0.246	0.260
SadTalker zhang2023sadtalker	5.843	8.813	0.441	0.275
Ours	$\mathbb{8.841}$	$\mathbb{6.312}$	$\mathbb{0.468}$	$\mathbb{0.304}$
Real video	7.640	7.189	0.588	0.505

Table 2: Comparison with previous methods on the OneMin-32 benchmark.

	$S_C$ $\uparrow$	$S_D$ $\downarrow$	CAPP$\uparrow$	$\mathrm{\Delta }$P	${\text{FVD}}_{25}\downarrow$
MakeItTalk zhou2020makelttalk	-0.123	14.340	0.002	0.190	304.833
Audio2Head wang2021audio2head	5.992	8.211	0.205	0.239	209.772
SadTalker zhang2023sadtalker	5.501	8.850	0.383	0.252	214.507
Ours	$\mathbb{7.957}$	$\mathbb{6.635}$	$\mathbb{0.465}$	$\mathbb{0.316}$	$\mathbb{105.884}$
Real video	7.192	7.254	0.559	0.405	29.245

4.3 Analysis and Ablation Study

CAPP metric.

We analyze the effectiveness of our proposed CAPP metric in measuring the alignment between audio and head pose.

Table 3: CAPP under frame shifting

0	$\pm$1	$\pm$2	$\pm$3	$\pm$4
​0.608	​0.462	​0.206	​0.069	​0.082

First, we study its sensitivity to temporal shifting by manually introducing frame offsets to ground-truth audio-pose pairs. We extract 3-second clip segments from the VoxCeleb2 test split, yielding approximately 2.1K audio-pose pairs. The average CAPP score for these pairs is $0.608$, as shown in Table 3. Manual frame shifts lead to a rapid decline in CAPP scores, approaching zero for shifts larger than two frames. This indicates a robust correlation between CAPP scores and audio-head pose alignment.

Table 4: CAPP under pose variation scaling

$\times$0.2	$\times$0.5	$\times$1.0	$\times$1.5	$\times$3.0
​0.368	​0.584	​0.608	​0.587	​0.505

We further investigate the effect of head movement intensity on CAPP by manually scaling the pose differences between consecutive frames using various factors. Table 4 shows that altering movement intensity negatively impacts the CAPP scores, demonstrating CAPP can assess the alignment of audio and pose in terms of their intensity. However, this sensitivity to intensity appears less pronounced than that to temporal misalignment.

CFG scales.

The CFG strategy ho2022classifier for diffusion models can attain a trade-off between sample quality and diversity. Here we evaluate the choice of the CFG scales for the audio and main gaze conditions (i.e., ${\lambda }_{𝐀}$ and ${\lambda }_{𝐠}$ in Eq. 2) in our model.

As shown in Table 5, as we increase the value of ${\lambda }_{𝐠}$, the accuracy of gaze control improves. Increasing the audio CFG scale to ${\lambda }_{𝐀}=0.5$ significantly enhances the performance of lip-audio alignment ($S_C$ and $S_D$), pose-audio alignment (CAPP), and pose variation intensity ($\mathrm{\Delta }P$). With positive audio CFG, the lip-audio alignment scores even surpass those evaluated on real videos (the results without audio CFG, i.e., ${\lambda }_{𝐀}=0$, were slightly worse than or comparable to them). Moreover, the FVD score shows a slight drop which indicates slightly better video quality.

Table 5: Ablation study of the audio and main gaze CFG scales as well as the sampling steps. ${ℰ}_g$ denotes the average angular error of main gaze directions and ${ℰ}_s$ is the average head distance error.

	$S_C$ $\uparrow$	$S_D$ $\downarrow$	CAPP$\uparrow$	$\mathrm{\Delta }$P	${\text{FVD}}_{25}\downarrow$	${ℰ}_{{\theta }_g}\downarrow$	${ℰ}_s\downarrow$
${\lambda }_{𝐀}=0.0,{\lambda }_{𝐠}=0.0$	7.087	7.391	0.414	0.291	117.425	5.730	0.004
${\lambda }_{𝐀}=0.0,{\lambda }_{𝐠}=1.0$	7.134	7.345	0.421	0.290	116.547	5.329	0.004
${\lambda }_{𝐀}=0.0,{\lambda }_{𝐠}=2.0$	7.108	7.386	0.414	0.298	117.784	5.064	0.005
${\lambda }_{𝔸}=\mathbb{0.5},{\lambda }_{𝐠}=\mathbb{1.0}$	7.957	6.635	0.465	0.316	105.884	5.253	0.005
${\lambda }_{𝐀}=1.0,{\lambda }_{𝐠}=1.0$	8.218	6.437	0.474	0.342	104.886	5.333	0.005
${\lambda }_{𝐀}=2.0,{\lambda }_{𝐠}=1.0$	8.295	6.397	0.455	0.395	104.293	5.531	0.005
${\lambda }_{𝐀}=\mathbb{0.5},{\lambda }_{𝐠}=\mathbb{1.0}$ (steps$=10$)	8.293	6.363	0.523	0.243	117.060	5.469	0.006
Real video	7.192	7.254	0.559	0.405	29.244	–	–

Further increasing ${\lambda }_{𝐀}$ marginally improves lip-audio synchronization and reduces ${\text{FVD}}_{25}$, but at the cost of slightly degrading audio-pose synchronization and gaze controllability. In addition, observations from the generated videos indicate that a higher ${\lambda }_{𝐀}$ significantly amplifies mouth movements for strong vocals and causes head pose jitter during rapid speech. For balanced performance and overall generation quality, we set ${\lambda }_{𝐀}=0.5$ and ${\lambda }_{𝐠}=1.0$ as our standard configuration.

We also evaluated the influence of sampling steps on performance. Table 5 illustrates that decreasing the steps from $50$ to $10$ improves audio-lip and audio-pose alignment while compromising pose variation intensity and overall video quality. This step reduction could accelerate the inference process by a factor of 5 for this latent motion generation module.

5 Conclusion

In summary, our work presents VASA-1, an audio-driven talking face generation model that stands out for its efficient generation of realistic lip synchronization, vivid facial expressions, and naturalistic head movements from a single image and audio input. It significantly outperforms existing methods in delivering video quality and performance efficiency, demonstrating promising visual affective skills in the generated face videos. The technical cornerstone is an innovative holistic facial dynamics and head movement generation model that works in an expressive and disentangled face latent space.

The advancements made by VASA-1 have the potential to reshape human-human and human-AI interactions across various domains, including communication, education, and healthcare. The integration of controllable conditioning signals further enhances the model’s adaptability for personalized user experiences.

Limitations and future work.

There are still several limitations with our method. Currently, it processes human regions only up to the torso. Extending to the full upper body could offer additional capabilities. While utilizing 3D latent representations, the absence of a more explicit 3D face model such as wu2022anifacegan ; wu2023aniportraitgan may result in artifacts like texture sticking due to the neural rendering. Additionally, our approach does not account for non-rigid elements like hair and clothing, which could be addressed with a stronger video prior. In the future, we also plan to incorporate more diverse talking styles and emotions to improve expressiveness and control.

6 Social Impact and Responsible AI Considerations

Our research focuses on generating audio-driven visual affective skills for virtual AI avatars, aiming for positive applications. It is not intended to create content that is used to mislead or deceive. However, like other related content generation techniques, it could still potentially be misused for impersonating humans. We are opposed to any behavior to create misleading or harmful contents of real persons, and are interested in applying our technique for advancing forgery detection. Currently, the videos generated by this method still contain identifiable artifacts, and the numerical study shows that there’s still a gap to achieve the authenticity of real videos.

While acknowledging the possibility of misuse, it’s imperative to recognize the substantial positive potential of our technique. The benefits – ranging from enhancing educational equity, improving accessibility for individuals with communication challenges, and offering companionship or therapeutic support to those in need – underscore the importance of our research and other related explorations. We are dedicated to developing AI responsibly, with the goal of advancing human well-being.

Contribution statement

Sicheng Xu, Guojun Chen, Yu-Xiao Guo were the core contributors to the implementation, training, and experimentation of various algorithm modules, as well as the data processing and management. Jiaolong Yang initiated the project idea, led the project, designed the overall framework, and provided detailed technical advice to each component. Chong Li, Zhengyu Zang and Yizhong Zhang contributed to enhancing the system quality, conducting evaluations, and demonstrating results. Xin Tong provided technical advice throughout the project and helped with project coordination. Baining Guo offered strategic research direction guidance, scientific advising, and other project supports. Paper written by Jiaolong Yang and Sicheng Xu.