Learning multi-grained interpretable latent representation for 3D face manipulation

Representing 3D faces using generative models has been investigated for several years for its numerous applications in computer vision and graphics. However, general 3D face manipulation is often limited by the lack of multi-level interpretability of the latent space in 3D generative models. To address this problem, we propose a novel generative approach dubbed hierarchically semantic regularized variational auto-encoders (HSR-VAE), which explicitly endows latent variables with multi-grained semantics of the synthesized 3D face shapes. Specifically, to accommodate the hierarchical structure of the human face, we decompose the latent space to represent variations in facial features at different scales, from local facial segments to fine-grained attributes. Moreover, part-aware and attribute-aware semantic regularizers are introduced to establish a linkage between hierarchically organized latent variables and multi-grained facial semantics, allowing more interpretable and meaningful representations of the 3D face. Extensive quantitative and qualitative experiments show the effectiveness of HSR-VAE and demonstrate that it can provide a more interpretable, manipulable, and generalizable latent representation than current approaches, facilitating a wide range of 3D face shape manipulation tasks.