Existing methods always neglect the value of channel features in processing multi-modal agricultural aerial images with Near-Infrared (NIR) characteristics, as well as ignore the importance of channel features in fusion features that contain both downsampled low-level features and upsampled high-level features. Two modules are proposed in this paper to address this issue. For fusion features composed of spatial information, semantic information, and multi-modal information, a transformer-based channel feature enhancement module is first constructed to facilitate the recognition of fusion features located in different channels. The second module is Dual Cross-Entropy Dice (Dual-CE-Dice) loss, which can improve the phenomenon of class imbalance while helping the model to better learn channel features. Extensive experiments have been conducted on the Agriculture-Vision-2021 dataset and the Tianchi suichang-round1 dataset, proving that the proposed method Channel Transformer (CFormer) is superior to the previous methods.