Creating 3D textured meshes using generative artificial intelligence has garnered significant attention recently. While existing methods support text-based texture generation and editing 3D meshes, they often struggle to precisely control pixels of texture images through intuitive interaction. Meanwhile, editing texture images by direct image deformation with precise point-to-point control does not necessarily guarantee that the generated mesh textures are a good match to the interactive intent. Existing work supports generative editing of 2D images via dragging interactions, but applying such methods directly to 3D mesh textures still leads to problems such as lack of local consistency between multiple views, error accumulation, and long training times. To address these challenges, we propose a generative point-based 3D mesh texture editing method, DragTex. It utilizes a diffusion model to blend locally inconsistent textures in the region near the deformed silhouette between different views, enabling locally consistent texture editing. We further fine-tune a LoRA decoder to reduce reconstruction errors in the non-dragged region, thereby mitigating overall error accumulation. Moreover, we train LoRA using multi-view images instead of training each view individually, which significantly shortens the training time. Our experimental results show that our method can effectively drag textures on 3D meshes and generate plausible textures that meet the user’s intent.

Talking head generation based on neural radiance fields (NeRF) has gained prominence, primarily owing to its implicit 3D representation capability within neural networks. However, most NeRF-based methods often intertwine audio-to-video conversion in a joint training process, resulting in challenges such as inadequate lip synchronization, limited learning efficiency, large memory requirement, and lack of editability. In response to these issues, this paper introduces a fully decoupled NeRF-based method for generating talking heads. This method separates audio-to-video conversion into two stages through the use of facial landmarks. Notably, the Transformer network is used to effectively establish the cross-modal connection between audio and landmarks and to generate landmarks conforming to the distribution of training data. We also explore formant features of the audio as additional conditions to guide landmark generation. Then, these landmarks are combined with Gaussian relative position coding to refine the sampling points on the rays, thereby constructing a dynamic NeRF conditioned on these landmarks and audio features for rendering the generated head. This decoupled setup enhances both the fidelity and flexibility of mapping audio to video with two independent small-scale networks. Additionally, it supports the generation of the torso from the head-only image with improved StyleUnet, further enhancing the realism of the generated talking head. Our experimental results demonstrate that our method excels in producing lifelike talking heads, and that the lightweight neural network models also exhibit superior speed and learning efficiency with lower memory requirements.

Image and video processing based on geometric principles typically changes the rectangular shape of video frames to an irregular shape. This paper presents a warping based approach for rectangling such irregular frame boundaries in space and time, i.e., making them rectangular again. To reduce geometric distortion in the rectangling process, we employ content-preserving deformation of a mesh grid with line structures as constraints to warp the frames. To conform to the original inter-frame motion, we keep feature trajectory distribution as constraints during motion compensation to ensure stability after warping the frames. Such spatially and temporally optimized warps enable the output of regular rectangular boundaries for the video frames withlow geometric distortion and jitter. Our experiments demonstrate that our approach can generate plausible video rectangling results in a variety of applications.