EmbedTalk: Talking Head Synthesis using Gaussian Embeddings
Arpita Saggar1, Jonathan C. Darling2, Duygu Sarikaya1, David C. Hogg1
1School of Computer Science, University of Leeds
2Leeds Institute of Medical Education, School of Medicine, University of Leeds
Preprint

Paper Code Supplementary Video

EmbedTalk models speech deformations in talking head synthesis with learnable per-Gaussian embeddings

Introduction

Synthesising audio-driven talking heads in real-time is an important task for film production, teleconferencing, and virtual assistants. Generating talking heads with 3D Gaussian Splatting involves deforming a canonical Gaussian representation using the speech signal. Prior work uses 2D planes to encode the 3D Gaussians and compress the associated deformation field. However, tri-plane representations suffer from mirroring artefacts caused by feature entanglement between subspaces. Tri-planes also introduce approximation errors that hamper audio-visual alignment, as shown above. Prior work has demonstrated the superiority of learnable embeddings for modelling Gaussian deformations in 4D (3D + time) scene reconstruction (E-D3DGS; ECCV’24). In this work, we extend the embedding-based deformation paradigm to talking head synthesis. We introduce EmbedTalk, which leverages per-Gaussian embeddings to generate stable talking heads with high audio-visual alignment. Quantitive and qualitative assessments establish EmbedTalk’s superiority in facial fidelity, lip-synchronisation, and motion consistency compared to previous 3DGS-based works. Furthermore, by generating mouth movements consistent with an identity’s style, our method achieves higher realism than state-of-the-art generative models that often exaggerate motion.

Method

EmbedTalk begins with a talking portrait video. The video frames enable a dense reconstruction of the head that initialises the 3D Gaussians. Each Gaussian is also associated with a learnable embedding \(z_g\). For each frame, the corresponding speech signal \(a\) and upper-face movements \(e\) are fed into the deformation MLP, along with a positional encoding of \(z_g\) to predict the Gaussian deformations (\(\Delta\mu, \Delta\alpha\)). The deformed Gaussians are passed to the rasteriser, along with the viewing direction (camera), to render the head onto the combined torso and scene background.

An overview of EmbedTalk

Results

We compare our approach with recent 3DGS-based methods: TalkingGaussian (ECCV’24), GaussianTalker (ACM MM’24) and DEGSTalk (ICASSP’25). To contextualise our work beyond 3DGS-based synthesis, we also compare EmbedTalk with state-of-the-art image-based generative methods: AniTalker(ACM MM’24), FLOAT (ICCV’25), KDTalker (IJCV’25) and Sonic (CVPR’25).
Videos are rendered under two different settings. The first is the self-driven setting, where videos generated for an identity are driven by unseen speech sourced from the same identity. The second is the cross-driven setting, where videos are generated using: (1) speech from other identities, and (2) synthetic speech created using a text-to-speech (TTS) model. For this, we source human speech from the HDTF (High-Definition Talking Face) dataset, and generate AI samples using the voices from ElevenLabs TTS. The text for the TTS model is taken from the works of William Shakespeare.

Quantitative Comparison

Results for the self-driven setting, with the top three results in red (first), orange (second) and yellow (third). EmbedTalk achieves the best rendering quality, (personalised) lip-sync, and motion consistency, along with a high inference speed.

Qualitative Comparison

Comparison with other 3DGS-based methods

Comparison with image-based generative models. Some methods redirect the speaker’s gaze (blue arrow) away from the camera

Consecutive differences accumulated over 20 frames. White pixels in the upper head region indicate temporal flickering, with higher intensities corresponding to larger instabilities

Acknowledgements

AS is supported by a PhD studentship that is funded by UK Research and Innovation (CDT Grant Reference: EP/S024336/1). This work was undertaken on the Aire HPC system at the University of Leeds, UK. We are grateful to Professor Vania Dimitrova for helpful comments on the presentation of results.

BibTeX

@misc{saggar2026embedtalktalkingheadsynthesis,
      title={EmbedTalk: Talking Head Synthesis using Gaussian Embeddings}, 
      author={Arpita Saggar and Jonathan C. Darling and Duygu Sarikaya and David C. Hogg},
      year={2026},
      eprint={2603.07604},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.07604}, 
}
Back to top