Optical-SAR image matching is a fundamental task for remote sensing applications. While existing methods perform well on some popular datasets such as SEN1-2 and WHU-SEN-City, their generalizability across diverse data sources such as satellites, spatial resolutions, and scenes remains insufficiently investigated, hindering the practical implementation of optical-SAR matching in various downstream tasks. Thus, 3MOS, the first multi-source, multi-resolution, and multi-scene optical-SAR dataset, was proposed in our study to address this gap. This dataset consists of 113k optical-SAR image pairs, with the SAR data collected from five satellites and resolutions ranging from 3.5 m to 12.5 m, further categorized into eight scenes, such as urban, rural, and plains through a simple but practical classification strategy. Based on this dataset, the performance of optical-SAR matching methods was evaluated through the data with diverse characteristics. Additionally, extensive experiments were conducted, and the following two findings were obtained. 1) None of the state-of-the-art methods achieved consistently superior performance across different sources, resolutions, and scenes, specifying significant generalization challenges for diverse downstream task data. 2) Training data distribution significantly impacted the matching performance of deep-learning models, highlighting the domain adaptation challenge in optical-SAR image matching. Furthermore, the practical utility of the dataset was comprehensively validated through multimodal change detection experiments, demonstrating its substantial value for a wide range of downstream applications.

Vision-based relative pose estimation plays a pivotal role in various space missions. Deep learning enhances monocular spacecraft pose estimation, but high computational demands necessitate model simplification for onboard systems. In this paper, we aim to achieve an optimal balance between accuracy and computational efficiency. We present a Perspective-n-Point (PnP) based method for spacecraft pose estimation, leveraging lightweight neural networks to localize semantic keypoints and reduce computational load. Since the accuracy of keypoint localization is closely related to the heatmap resolution, we devise an efficient upsampling module to increase the resolution of heatmaps with minimal overhead. Furthermore, the heatmaps predicted by the lightweight models tend to show high-level noise. To tackle this issue, we propose a weighting strategy by analyzing the statistical characteristics of predicted semantic keypoints and substantially improve the pose estimation accuracy. The experiments carried out on the SPEED dataset underscore the prospect of our method in engineering applications. We dramatically reduce the model parameters to 0.7 M, merely 2.5% of that required by the top-performing method, and achieve lower pose estimation error and better real-time performance.

Estimating the pose of non-cooperative targets from monocular images stands as a pivotal technology for future space missions. Recent advancements in deep neural networks have surpassed traditional methods in pose measurement accuracy. However, these networks often entail a high number of parameters and significant computational complexity. This poses a challenge for deployment in on-orbit applications where real-time measurement is crucial, as computational resources are limited. Reducing the number of network parameters compromises the ability to extract representative features, leading to degraded pose estimation performance. To tackle this problem, we present an approach using lightweight neural networks that maintains high accuracy in pose estimation, which is a task far from trivial. Our solution involves a novel semantic keypoint localization method. We develop a lightweighted neural network model with a mere 1.1 ×10⁶ parameters. To enhance the precision of semantic keypoint localization and subsequent pose estimation, we introduce a heatmap decoding technique that allows for sub-pixel level accuracy, while enabling end-to-end supervision of semantic keypoint localization. Moreover, we develop an auxiliary layer supervised training method to further refine the accuracy of semantic keypoint localization. Experiments on public datasets demonstrate that our method not only achieves the highest pose measurement accuracy among all lightweight models with fewer than 10⁷ parameters, but also sets a new benchmark. Additionally, tests on embedded development boards reveal that our method attains measurement frequencies of 5 Hz and 11 Hz in 10 W and 30 W power modes, respectively.

Deep learning significantly improves the accuracy of remote sensing image scene classification, benefiting from the large-scale datasets. However, annotating the remote sensing images is time-consuming and even tough for experts. Deep neural networks trained using a few labeled samples usually generalize less to new unseen images. In this paper, we propose a semi-supervised approach for remote sensing image scene classification based on the prototype-based consistency, by exploring massive unlabeled images. To this end, we, first, propose a feature enhancement module to extract discriminative features. This is achieved by focusing the model on the foreground areas. Then, the prototype-based classifier is introduced to the framework, which is used to acquire consistent feature representations. We conduct a series of experiments on NWPU-RESISC45 and Aerial Image Dataset (AID). Our method improves the State-Of-The-Art (SOTA) method on NWPU-RESISC45 from 92.03% to 93.08% and on AID from 94.25% to 95.24% in terms of accuracy.

Multimodal image matching between optical and SAR images is critical for visual navigation of the aircraft flying over complex areas. Due to different imaging mechanisms, accurate optical-SAR image matching faces several challenges, such as imaging geometry differences between optical and SAR, nonlinear radiation distortion of SAR, and noise interference. Although deep learning-based image matching methods have shown better adaptability than traditional methods, they still cannot fully solve the above-mentioned challenges and their matching accuracy may be not enough for visual navigation tasks. This paper proposes a new image matching method called Multi-point Matching Constraint MatchNet (MMC-MatchNet) based on the typical MatchNet framework, so as to improve the matching accuracy of the single template matching using MatchNet. The method consists of two main steps: sub-templates sampling and matching using MatchNet, and mismatched sub-templates removal based on multi-point geometric constraint. The original input image size of the MatchNet is maintained, while the matching accuracy for optical-SAR image matching is improved. This paper also proposes a multi-level training dataset generation method to train the MMC-MatchNet. Compared to the random sampling method, our method focuses on the difficult samples in image matching, and can be seen as a special data augmentation method for image matching. MMC-MatchNet is tested on two multi-modal image matching datasets, outperforming NCC, NMI, HOPC and the single-template MatchNet. With similar matching accuracy to that of HOPC, MMC-MatchNet can improve the Correctly Matching Rate (CMR). The method based on multi-point matching constraint and multi-level dataset generation can be easily extend to other patch-based image matching models.