Multi-subspace multimodal sentiment analysis method based on Transformer

Multimodal sentiment analysis refers to recognizing the emotions expressed by characters in a video through textual, visual and acoustic information. Most of the existing methods learn multimodal coherence information by designing complex fusion schemes, while ignoring inter-and intra-modal differentiation information, resulting in a lack of information complementary to multimodal fusion representations. To this end, we propose a multi-subspace Transformer fusion network for multimodal sentiment analysis (MSTFN) method. The method maps different modalities to private and shared subspaces to obtain private and shared representations of different modalities, learning differentiated and unified information for each modality. Specifically, the initial feature representations of each modality are first mapped to their respective private and shared subspaces to learn the private representation containing unique information and the shared representation containing unified information in each modality. Second, under the premise of strengthening the roles of textual and audio modalities, a binary collaborative attention cross-modal Transformer module is designed to obtain textual and audio-based tri-modal representations. Then, the final representation of each modality is generated using modal private and shared representations and fused two by two to obtain a bimodal representation to further complement the information of the multimodal fusion representation. Finally, the unimodal representation, bimodal representation, and trimodal representation are stitched together as the final multimodal feature for sentiment prediction. Experimental results on two benchmark multimodal sentiment analysis datasets show that the present method improves on the binary classification accuracy metrics by 0.025 6/0.014 3 and 0.000 7/0.002 3, respectively, compared to the best benchmark method.