Severe cardiovascular diseases can rapidly lead to death. At present, most studies in the deep learning field using electrocardiogram (ECG) are performed on intra-patient experiments for the classification of coronary artery disease (CAD), myocardial infarction, and congestive heart failure (CHF). By contrast, actual conditions are inter-patient experiments. In this study, we proposed a deep learning network, namely, CResFormer, with dual feature extraction to improve accuracy in classifying such diseases. First, fixed segmentation of dual-lead ECG signals without preprocessing was used as input data. Second, one-dimensional convolutional layers performed moderate dimensionality reduction to accommodate subsequent feature extraction. Then, ResNet residual network block layers and transformer encoder layers sequentially performed feature extraction to obtain key associated abstract features. Finally, the Softmax function was used for classifications. Notably, the focal loss function is used when dealing with unbalanced datasets. The average accuracy, sensitivity, positive predictive value, and specificity of four classifications of severe cardiovascular diseases are 99.84%, 99.68%, 99.71%, and 99.90% in intra-patient experiments, respectively, and 97.48%, 93.54%, 96.30%, and 97.89% in inter-patient experiments, respectively. In addition, the model performs well in unbalanced datasets and shows good noise robustness. Therefore, the model has great application potential in diagnosing CAD, MI, and CHF in the actual clinical environment.