To enhance the robustness and real-time performance of robotic arm visual servoing in complex environments such as space, this study proposes DiFA-DETR, a modified object detection framework based on the DETR architecture. The proposed model incorporates an improved ResNet50 Bottleneck structure with spatial dimensionality reduction and sparse interaction mechanisms, alongside a redesigned self-attention module featuring downsampling optimization and adaptive feature enhancement. A custom-annotated satellite component dataset was constructed to train and evaluate the system. Experimental results demonstrate that DiFA-DETR achieves an AP50 of 79.9 %, outperforming existing DETR variants while reducing computational complexity by 31.9 % and nearly doubling the inference speed. The method was further validated in a ground-based visual servoing system using an industrial robotic arm and camera setup. The system successfully tracked satellite targets under dynamic motion scenarios, maintaining millimeter-level positioning accuracy. These results confirm the feasibility and effectiveness of the proposed method in supporting future space robotic applications requiring precision tracking and fast response.