Review of online reinforcement learning control for systems with unknown models: theory, methods, and challenges

Designing controllers without accurate models remains a core challenge in control engineering. While methods such as adaptive or model predictive control have been successful, they depend heavily on precise system knowledge. In practice, modern industrial systems—from chemical reactors to robotic manipulators—are often nonlinear, uncertain, and subject to unmodeled disturbances. These conditions can render model-based controllers ineffective or unsafe. RL provides a promising alternative. By learning directly from interaction, it sidesteps the need for complete models and adapts to unknown dynamics. More importantly, online RL, which updates policies during real-time operation, aligns well with the requirements of safety-critical and time-sensitive applications. A systematic review of this direction is both timely and essential, offering a roadmap for translating RL research into industrial practice.

Research in online reinforcement learning for control has progressed along three major directions. First, theoretical work has established clear links between RL and optimal control, with the Hamilton–Jacobi–Bellman equation providing a unifying principle. This connection has allowed RL to extend classical dynamic programming tools into data-driven settings.

Second, distinct categories of online RL have emerged. Model-based RL learns an environment model (often via neural networks) to generate simulated trajectories, thereby reducing real-world interactions and achieving high sample efficiency. Off-policy integral RL leverages data collected by behavior policies to update the agent's policy, balancing stability and efficiency. Q-learning, in contrast, directly updates the action-value function through interaction, offering simplicity and low computational cost but suffering from stability challenges. These families reflect different trade-offs between data efficiency, theoretical guarantees, and deployment feasibility.

Third, theoretical analyses based on Lyapunov stability have provided convergence guarantees. However, these results often rely on restrictive assumptions such as full state observability or persistent excitation, which limit real-world applicability. Current research is therefore shifting toward relaxing these assumptions and addressing practical concerns. Emerging efforts include sequence-model-based RL for partial observability, safe RL to mitigate exploration risks, and robust RL to maintain performance under disturbances. Together, these advances illustrate both the achievements to date and the challenges that remain for deploying RL in real-world control systems.

Online reinforcement learning is rapidly becoming a central paradigm for intelligent control of systems with unknown dynamics. Existing methods, including model-based RL, off-policy integral RL, and Q-learning, offer complementary strengths but face limitations in terms of sample efficiency, stability, and computational scalability. While Lyapunov-based theory provides a foundation for convergence analysis, its assumptions restrict applicability in industrial contexts.

This review established theoretical foundations, categorizes current online RL methods, and identified the challenges that impede their deployment. Looking ahead, progress will depend on improving sample efficiency by reusing historical data and using transfer learning, enhancing safety during exploration, and designing robust policies that can generalize across uncertainties. Hybrid approaches combining RL with classical control principles may be the most viable path to industrial adoption.

By highlighting these opportunities and open problems, this review aims to provide researchers with a structured understanding of the field and to guide future innovations toward reliable and practical reinforcement learning-based control.