VarGes: Improving variation in co-speech 3D gesture generation via StyleCLIPS

Generating expressive and diverse human gestures from audio is crucial in fields like human–computer interaction, virtual reality, and animation. While existing methods have achieved remarkable performance, they often exhibit limitations due to constrained dataset diversity and the restricted amount of information derived from audio inputs. To address these challenges, we present VarGes, a novel variation-driven framework designed to enhance co-speech gesture generation by integrating visual stylistic cues while maintaining naturalness. Our approach begins with a variation-enhanced feature extraction module, which seamlessly incorporates style-reference video data into a 3D human pose estimation network to extract StyleCLIPS, thereby enriching the input with stylistic information. Subsequently, we employ a variation-compensation style encoder, a transformer-style encoder equipped with an additive attention mechanism pooling layer, to robustly encode diverse StyleCLIPS representations and effectively manage stylistic variations. Finally, a variation-driven gesture predictor module fuses MFCC audio features with StyleCLIPS encodings via cross-attention, injecting this fused data into a cross-conditional autoregressive model to modulate 3D human gesture generation based on audio input and stylistic clues. The efficacy of our approach is validated on benchmark datasets, on which it outperforms existing methods in terms of gesture diversity and naturalness. Our code and video results are publicly available at https://github.com/mookerr/VarGES/.