The formation control of unmanned aerial vehicle (UAV) swarms is of significant importance in various fields such as transportation, emergency management, and environmental monitoring. However, the complex dynamics, nonlinearity, uncertainty, and interaction among agents make it a challenging problem. In this paper, we propose a distributed robust control strategy that uses only local information of UAVs to improve the stability and robustness of the formation system in uncertain environments. We establish a nominal control strategy based on position relations and a semi-definite programming model to obtain control gains. Additionally, we propose a robust control strategy under the rotation set $\Omega$ to address the noise and disturbance in the system, ensuring that even when the rotation angles of the UAVs change, they still form a stable formation. Finally, we extend the proposed strategy to a quadrotor UAV system with high-order kinematic models and conduct simulation experiments to validate its effectiveness in resisting uncertain disturbances and achieving formation control.