Reinforcement learning-guided Animated Oat Optimization Algorithm with dynamic niching for high-dimensional optimization problems

High-dimensional optimization problems face challenges from exponentially expanding search spaces and deceptive local optima resisting metaheuristics. In order to address these issues, a reinforcement learning-guided Animated Oat Optimization Algorithm with a dynamic niching strategy, called RLDN-AOO, is proposed in this research. RLDN-AOO offers the following major novelties: i) a mathematically formulated three-state dynamic niching mechanism that adaptively partitions the population, preserves diversity, and enhances the algorithm's ability to escape local optima, and ii) a reinforcement learning strategy selection mechanism is proposed to address the issue of the algorithm's inadequate dynamic adaptability. We compared it with state-of-the-art algorithms (CEC2017, Dim = 50, 100, 200, 500), including LSHADE-SPACMA, CMA-ES variants, and RL-based optimizers. In addition, we applied it in the optimization of BP neural networks. Experimental results showed that RLDN-AOO achieves competitive performance across most benchmarks and, in some cases, performs comparably to LSHADE-SPACMA variants. The source code of RLDN-AOO is openly accessible via https://github.com/robingit77/RLDN-AOO.