Videos contain multi-modal content, and exploring multi-branch cross-modal interactions with natural language queries can be of benefit to the text–video retrieval task (TVR). However, recent methods applying the large-scale pre-trained CLIP model for TVR only focus on visual cues in videos. Furthermore, traditional methods of simply concatenating multi-modal features do not exploit fine-grained cross-modal information in videos. In this paper, we propose a multi-branch multi-modal hybrid fusion (M2HF) network to hierarchically explore interaction between text queries and other modality content in videos. Specifically, M2HF first fuses visual features extracted by CLIP with audio and motion features extracted from videos to obtain fused audio–visual features and motion–visual features respectively. The multi-modal completion problem is also considered and solved in this process. Then, visual features, audio–visual features, motion–visual features, and text extracted from the video are used to establish cross-modal relationships with caption text queries using a multi-branch approach. The retrieval outputs from all branches are then fused to obtain the final text–video retrieval results. Our framework provides two kinds of training strategies, using an ensemble approach and an end-to-end approach. Moreover, a novel multi-modal loss function is proposed to balance the contributions of each modality for efficient end-to-end training. M2HF allows us to obtain state-of-the-art results on various benchmarks: Rank@1 of 66.0%, 68.6%, 33.9%, 57.4%, and 57.3% on MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet, respectively.

Existing Low-Rank Adaptation (LoRA) methods face challenges on sparse Large Language Models (LLMs) due to the inability to maintain sparsity. Recent works introduce methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects the efficiency of LoRA methods. In response to this limitation, we introduce Low Rank adaptation method for Sparse LLM (LoRS), an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recomputing and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, while achieving performance levels that outperform existing LoRA approaches.