Walnut is one of the most favorite nuts worldwide. However, existing production lines cannot capture the tiny defects and narrow cracks from the natural growth textures of walnuts at the high rolling speed. It is often required for high-precision detection and real-time operation in industrial scenarios. In this study, a lightweight YOLO11n model, named YOLO-EBM, was proposed to detect the external defects of walnuts. Three optimization strategies were designed to balance between detection accuracy and a lightweight model in the actual application needs of walnut production lines. 1) The original backbone network of the baseline YOLO11n model was replaced by the lightweight EfficientNetV2 network. Feature structure was used to efficiently extract the key features of walnut defects. Thereby, the overall accuracy of detection was also improved to effectively reduce the redundant parameters for the lightweight model. 2) Bidirectional Feature Pyramid Network (BiFPN) was introduced into the feature fusion module for the high efficiency of multi-scale feature fusion. The parameters were substantially reduced to further promote the lightweight level of the model. While the performance of detection fully met the actual production requirements. 3) A Multi-Scale Dilated Attention (MSDA) mechanism was added to the small-object detection layer. Fine-grained features of tiny defects and narrow cracks were effectively captured to identify the real defects and walnut natural growth textures. Thus, there was an ideal balance between precision and model size. Experimental tests were carried out to verify the performance of the YOLO-EBM model. The results show that the improved model achieved a mean average precision (mAP) of 93.3%, the number of parameters was reduced to 0.77 M, and the size of the model weight file was compressed to 2.18 MB. Compared with the baseline YOLO11n model before improvement, the mean average precision of the YOLO-EBM model increased by 2.1 percentage points, whereas the number of parameters and the size of the weight file were reduced by 70.2% and 65.4%, respectively. Additionally, compared with YOLOv5n, YOLOv8n, and YOLOv10n, the mAP was improved by 5.2, 4.3, and 2.5 percentage points, respectively. Although the precision of RT-DETR is slightly higher than that of the proposed model, its mAP is only 78.2% and its parameter count reaches 31.99 M, making it unsuitable for mobile deployment. Both high accuracy and lightweight contributed to the deployment and transplantation of the model on resource-constrained industrial production equipment, fully meeting the real-time detection requirements of actual walnut production lines. In summary, this finding can provide a reliable technical reference and practical solution for the intelligent detection of walnut external defects in large-scale production. Theoretical and technical bases for lightweight models can also be obtained to detect the defects of food products.