Spatiotemporal fusion transformer for video demoiréing

When using digital cameras to capture video from a display screen, the occurrence of moiré patterns can lead to color distortions, significantly degrading the quality of both images and video. Given the escalating demand for video acquisition, designing algorithms for video demoiréing is a significant topic. In this paper, we introduce a novel attention-based network for this task, the spatiotemporal fusion transformer (STFT). By introducing temporal and spatial attention encoders and a multi-scale feature fusion method, STFT can learn dynamic spatial and temporal variations in moiré patterns. In the decoding phase, a self-attention mechanism is employed to learn temporal dependencies at both image-level and video-level, enhancing model moiré removal performance. Experimental results demonstrate a significant improvement in the performance of the proposed model over existing methods on public datasets. Furthermore, STFT can output visual attention maps for analyzing the distribution of moiré and the focus of model learning. STFT’s outstanding performance on the video rain removal task also demonstrates the robustness of our model, highlighting its potential for application to other restoration tasks.