Multi-modal corpus is one of the important tools for medical research, clinical diagnosis and teaching. Most of the existing medical corpus only have text data, lacking matching images, which leads to inadequate information. Additionally, a large number of medical images don’t have clear semantic labels, which makes them difficult to be used in corpus construction. To solve the above problems, a novel dermoscopic image classification method for multi-modal medical corpus construction is proposed. Through accurate classification of dermoscopic images, we obtained semantic labels, and then matching them with relevant text by natural language processing method. In this way, a multi-modal corpus with images and texts was established. First, traditional image classification methods based on machine learning are weak in feature extraction of lesions and susceptible to background noise, resulting in poor classification accuracy. Considering the above, in this paper, a two-stream network was constructed to enhance the feature representation by fusing lesions’ shapes and texture features. Second, in order to reduce information redundancy caused by feature fusion, a feature selection method based on channel attention mechanism was introduced to focus on the key features and suppress the influence of noise. In addition, aiming at the problem of model optimization difficulty caused by the imbalance in the number of benign and malignant samples in dermatoscopic images, asymmetric loss function was introduced to improve the robustness of model. Experimental results on ISIC dermatoscopic image dataset proved that the proposed method can accurately and efficiently categorize dermatoscopic images and match images with corresponding medical records to construct a multi-modal medical corpus.