Emotion recognition is vital for mental and physical health, but current methods often rely on active user participation (e.g., questionnaires) or wearable sensors, limiting their real-world adoption. This paper proposes a non-contact ballistocardiogram (BCG) based cross-subject emotion recognition framework. Subject-specific data distribution variations challenge model generalizability across domains. For this reason, we propose an LSCS-BDA method, extending Balanced Distribution Adaptation (BDA) with a Label Similarity Clustering for Selection (LSCS) module to optimize source domain selection. After feature extraction and baseline normalization, only the source domain individuals that are more similar to the target are included in the final source domain set through our method’s LSCS module (reducing training data to 9.78%), and finally the transfer is completed by the BDA algorithm. The experimental results show that the classification accuracy of our proposed method for three emotions of positive, negative, and neutral is 70.38%, which achieves higher accuracy and faster model training than classical transfer algorithms. The LSCS module in our method can be extended to other transfer learning methods. Our approach offers a new idea for contactless emotion recognition based on BCG and contributes to advancing its applications.