DanceDiT: towards efficient and consistent human image animation with diffusion transformer

Pose-driven human image animation aims to synthesize a human video from a reference image and a target pose sequence while preserving identity consistency, motion consistency, and temporal coherence. Existing methods often improve controllability by introducing duplicated reference branches or dedicated pose encoders. However, such designs incur substantial architectural redundancy, memory overhead, and adaptation cost, especially when transferred to large diffusion transformer (DiT) video models. In this paper, we introduce DanceDiT, a lightweight and practical framework for pose-driven human image animation. DanceDiT adapts pretrained DiT video models while explicitly balancing identity consistency, motion consistency, and efficiency. DanceDiT treats the reference image as a unified frame in the latent video sequence, enabling direct interaction between reference and video tokens within a single DiT backbone and eliminating the need for a duplicated reference network. For motion control, DanceDiT encodes the target pose sequence in the shared 3D variational autoencoder (VAE) latent space and injects pose information through an adaptive normalization fusion design, which dynamically modulates pose guidance across denoising steps without relying on a heavy pose encoder. To preserve identity under this lightweight design, we further incorporate complementary facial and semantic cues using ArcFace and DINOv2. In addition, we adopt a capability-preserving fine-tuning strategy that updates only attention-related layers and task-specific conditioning modules, reducing adaptation overhead while retaining the pretrained model’s generative capability. Experiments demonstrate that DanceDiT achieves strong quantitative and qualitative results, providing a better balance among identity consistency, motion consistency, and efficiency than prior heavier pose-driven human animation systems.