Transformer has performed good performance when used for object tracking. By introducing the selfattention mechanism, the object which is partially occluded can also be tracked since the transformer can prioritize the most relevant parts in the image. However, when prolonged occlusion occurs, the transformer tends to lose the objects. Kalman filter is always introduced to the neural network to predict the position of the objects. Limited by the assumption that the objects should move linearly, it is still hard to track the objects with irregular motions under prolonged occlusion. To address this problem, this paper proposes a novel approach for robust object tracking. The transformer framework is initially constructed for detection. When prolonged occlusion occurs, featured optical flow points are extracted and the optical flow motion is estimated. The predicted values are compared with the detected ground truth to compute the association probability, meanwhile the covariance matrix of the Kalman filter is updated. Hungarian algorithm is finally employed for association matching. Experimental validation on the datasets demonstrates the effectiveness of the proposed method. Comparative analyses with the state-of-the-art algorithms prove the robustness of the proposed approach in complex scenarios.