StyTips: Towards high-quality, efficient, and controllable style transfer via transformer filtering prompts

Image style transfer aims to merge the artistic characteristics of a style image with the spatial structure of a content image, maintaining the content identity and showing the desired style. Previous works based on convolutional neural networks lack the ability to capture global features, leading to the content leakage issue. In this work, we propose a method StyTips, short for style via transformer filter prompts, a novel approach for style transfer that distills essential content and injects style information via learned prompts. To achieve efficient and high-quality stylization, we adopt a hierarchical vision transformer encoder to extract multi-scale features of the content and style images, learning disentangled style features of color and strokes. Inspired by the prompt learning technique, we have designed a fine-grained style alignment block in the decoder, consisting of a filter-using self-attention module to distill necessary content information and a style-aware cross-attention module to inject style features. StyTips takes full advantage of the attention mechanism to align color and strokes with the purified content features in appropriate proportions. Qualitative and quantitative experiments demonstrate that StyTips can prevent content leakage and generate high-quality stylized images. StyTips provides controllable style transfer with diverse ways for users to manipulate styles in a fine-grained manner and to determine the strength of style, giving it more practical significance to real-world applications than traditional approaches. The code is available at https://github.com/I2-Multimedia-Lab/StyTips.