In this study, we address a value function reconstruction problem in optimal output tracking of continuous-time linear time-invariant systems. The objective is to determine the state weight matrix within the value function that results in the specified optimal control law. First, by augmenting the system state and the desired tracking dynamic variable, the optimal tracking problem with a discount value function is turned into a linear regulator problem. Inverse optimal output tracking is thus simplified as an inverse optimal control of the augmented system. Second, a model-based inverse reinforcement learning algorithm is suggested to calculate the state weight matrix in the augmented value function. This algorithm updates the cost matrix via gradient descent and calculates the weight matrix through inverse optimal control. After continuous iterations, the weight matrix converges to a steady state. Third, astringency of the algorithm is rigorously analyzed, and the stability of the corresponding system is confirmed. Finally, the proposed algorithm's effectiveness is confirmed through simulation, illustrating that the system output asymptotically tracks a predetermined reference trajectory.