A multimodal sentiment analysis based on audio and video features optimization and cross-modal Transformer

To solve problems including low-quality audio and video modal features and inadequate interaction between various modalities, a multimodal sentiment analysis approach based on cross-modal Transformer (CMT) and audio and video feature optimization is suggested. Firstly, we propose a audio and video features optimizing mechanism (AVFOM), which increases the density of sentiment information in audio and video features through synergistic interaction with textual features, thereby improving the quality of audio and video features. Secondly, in order to accomplish full interaction between text-audio and text-video modalities and learn consistent knowledge across various modalities, we construct a cross-modal Transformer structure with text as the dominant modality. Additionally, a label generation method based on the self-supervised learning strategy is introduced to perform single-modality sentiment prediction tasks, learning the characteristics of each modality separately. The proposed method is extensively validated and tested on two public datasets, CMU-MOSI and CMU-MOSEI, which surpass many currently advanced methods in terms of performance and effectively improve the accuracy of multimodal sentiment analysis.