Chinese herbal medicine (CHM) classification is an important and emerging topic in intelligent medicine. However, due to the limited local receptive field, existing Convolutional Neural Network-based methods are susceptible to background interference, which fails to capture discriminative visual cues and hampers the accurate recognition of visually similar herbal categories. To address these limitations, we propose a dual attention enhancement network for Chinese herbal medicine classification. Specifically, we introduce an object localization module to suppress background interference by accurately localizing the target region, thus guiding the network to focus on discriminative regions. Subsequently, we introduce a fused attention module to integrate horizontal and vertical directional information at different feature scales to capture long-range spatial dependencies and enhance the global contextual perception. Moreover, we propose a dual attention module composed of self-attention and cross-attention mechanisms to achieve explicit fusion and the semantic alignment of multi-scale features. Finally, we build a semantic feature enhancement module to further strengthen the inter-layer complementarity through adaptive semantic fusion, thereby improving the discriminative ability and robustness of feature representations. Extensive experimental results on two CHM datasets demonstrate that the proposed method outperforms existing state-of-the-art methods.