Tackling Long-Tail Video Recognition via Hierarchical Memory Banks

In the real world, long-tailed data distributions are common and natural. This paper focuses on the long-tailed problem in video recognition, which consists of two aspects. First, the inter-video long-tailed distribution affects video samples. The tail video classes have fewer samples and lack within-class diversity, leading to weakened recognition accuracy. Second, the intra-video long-tailed distribution involves background frames that degrade the video representation by dominating the majority of frames unrelated to the video theme. To address these challenges, this paper proposes the long-short memory bank. This approach involves building two feature banks for each video class: the long-term bank and the short-term bank. The long-term bank uses a dictionary to store current and past video-level representations, enhancing the competitiveness of tail classes and mitigating the impact of insufficient samples on video classifier training. The short-term bank stores the most discriminative frame-level information, weakening background information and improving the robustness of video representation. During training, the current batch features are combined with the memory bank features to promote intra-class compactness and inter-class discrepancy. Experimental results on the VideoLT dataset demonstrate that our proposed Long-short Memory Bank improves recognition accuracy for tail video classes by 6.4%, without sacrificing overall recognition accuracy.