With the development of computers, artificial intelligence, and cognitive science, engagement in deep communication between humans and computers has become increasingly important. Therefore, affective computing is a current hot research topic. Thus, this study constructs a Physiological signal-based, Mean-threshold, and Decision-level fusion algorithm (PMD) to identify human emotional states. First, we select key features from electroencephalogram and peripheral physiological signals, and use the mean-value method to obtain the classification threshold of each participant and distinguish individual differences. Then, we employ Gaussian Naive Bayes (GNB), Linear Regression (LR), Support Vector Machine (SVM), and other classification methods to perform emotion recognition. Finally, we improve the classification accuracy by developing an ensemble model. The experimental results reveal that physiological signals are more suitable for emotion recognition than classical facial and speech signals. Our proposed mean-threshold method can solve the problem of individual differences to a certain extent, and the ensemble learning model we developed significantly outperforms other classification models, such as GNB and LR.