Translating real portrait video into anime is an application of interest to both consumers and researchers. However, anime differs considerably from portraits, making portrait-to-anime translation challenging. Existing StyleGAN-based portrait stylization works assume that the portrait and stylized generators share the same latent space, but this assumption fails in the style of anime due to the large domain gap. Moreover, directly applying them to each video frame often leads to undesirable temporal inconsistencies. In this paper, we argue that two latent spaces with a large domain gap cannot be shared but can be related by a transformation, and develop a cyclic transformation network to connect the two spaces with two cycle constraints. This provides high-quality translation for each frame. We extend our framework to video transformation by proposing a novel frame interpolation constraint which ensures that in-between frames can be interpolated from their neighboring frames, guaranteeing temporal coherence across translated frames. Together with latent code smoothing regularization, this provides temporally coherent video-to-anime translation. Extensive experiments demonstrate that our framework outperforms state-of-the-art methods both qualitatively and quantitatively.

This paper presents a novel approach for camera pose refinement based on neural radiance fields (NeRF) by introducing semantic feature consistency to enhance robustness. NeRF has been successfully applied to camera pose estimation by inverting the rendering process given an observed RGB image and an initial pose estimate. However, previous methods only adopted photometric consistency for pose optimization, which is prone to be trapped in local minima. To address this problem, we introduce semantic feature consistency into the existing framework. Specifically, we utilize highlevel features extracted from a convolutional neural network (CNN) pre-trained for image recognition, and maintain consistency of such features between observed and rendered images during the optimization procedure. Unlike the color values at each pixel, these features contain rich semantic information shared within local regions and can be more robust to appearance changes from different viewpoints. Since it is computationally expensive to render a full image with NeRF for feature extraction from CNN, we propose an efficient way to estimate the features of individually rendered pixels by projecting them to a nearby reference image and interpolating its feature maps. Extensive experiments show that our method greatly outperforms the baseline method on both synthetic objects and real-world large indoor scenes, increasing the accuracy of pose estimation by over 6.4%.

Transformers have recently lead to encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs: (i) a linear complexity attention layer, (ii) an overlapping patch embedding, and (iii) a convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linearity and provides significant improvements on fundamental vision tasks such as classification, detection, and segmentation. In particular, PVT v2 achieves comparable or better performance than recent work such as the Swin transformer. We hope this work will facilitate state-of-the-art transformer research in computer vision. Code is available at https://github.com/whai362/PVT.