A multi-source ship image fusion recognition method is proposed for unmanned aerial vehicle aerial photography of multi-source ship images. In the face of many interferences in real scenes, pixel level fusion is adopted to fuse infrared and visible light ship images, and then perform target recognition. It can improve the algorithm's interpretability and lessen the network's reliance on samples as compared to feature level recognition techniques. This article focuses on solving the pixel offset caused by different sensor parameters. To eliminate the distortion and artifacts that traditional picture registration can readily cause, image registration has been turned into end-to-end feature alignment. A multi-source ship image fusion recognition network is proposed, which consists of a cross modulation feature extraction module, a feature dynamic alignment module, a multi granularity feature refinement module, and a pyramid feature fusion module. It can fully integrate the features and texture details of different modal images, effectively improving the recognition performance of ship targets. The approach described in this study has demonstrated great interpretability for multi-source images, good fusion performance, and high accuracy and robustness in recognizing ship targets through experimental verification.