Skin defect inspection is one of the most significant tasks in the conventional process of aircraft inspection. This paper proposes a vision-based method of pixel-level defect detection, which is based on the Mask Scoring R-CNN. First, an attention mechanism and a feature fusion module are introduced, to improve feature representation. Second, a new classifier head—consisting of four convolutional layers and a fully connected layer—is proposed, to reduce the influence of information around the area of the defect. Third, to evaluate the proposed method, a dataset of aircraft skin defects was constructed, containing 276 images with a resolution of 960 × 720 pixels. Experimental results show that the proposed classifier head improves the detection and segmentation accuracy, for aircraft skin defect inspection, more effectively than the attention mechanism and feature fusion module. Compared with the Mask R-CNN and Mask Scoring R-CNN, the proposed method increased the segmentation precision by approximately 21% and 19.59%, respectively. These results demonstrate that the proposed method performs favorably against the other two methods of pixel-level aircraft skin defect detection.