In this paper, we present a distributed framework for the lidar-based relative state estimator which achieves highly accurate, real-time trajectory estimation of multiple Unmanned Aerial Vehicles (UAVs) in GPS-denied environments. The system builds atop a factor graph, and only on-board sensors and computing power are utilized. Benefiting from the keyframe strategy, each UAV performs relative state estimation individually and broadcasts very partial information without exchanging raw data. The complete system runs in real-time and is evaluated with three experiments in different environments. Experimental results show that the proposed distributed approach offers comparable performance with a centralized method in terms of accuracy and real-time performance. The flight test demonstrates that the proposed relative state estimation framework is able to be used for aggressive flights over 5 m/s.