Bi-level Bayesian and reinforcement learning optimization for integrated energy system design and operation

The high penetration of renewable energy in user-side integrated energy systems (IES), combined with random consumption patterns, renders optimizations based on deterministic supply–demand scenarios inapplicable to dynamic real-world conditions, leading to a disconnect between design and operation. A bi-level optimization framework coupling Bayesian optimization with deep reinforcement learning is proposed. At upper level, a tree-structured Parzen estimator (TPE) searches for optimal equipment configurations; at lower level, a soft actor-critic (SAC) agent learns adaptive operational policies under uncertainty. By embedding equipment configuration information into the agent’s state space, the framework enables configuration-aware strategy learning. An agent reuse mechanism based on configuration similarity accelerates training by transferring knowledge across configurations, reducing the average training episodes from 184.7 to 68.0—a 63.2% improvement in computational efficiency. Validated on a multi-energy office building with 100 Monte Carlo scenarios, the framework achieves annual total cost savings of 6.2% over deterministic optimization and 2.0% over two-stage stochastic programming. Cooling penalty costs are reduced by 96.3% and heating penalty costs by 99.7% compared to deterministic baseline. Additional benchmarks against genetic algorithm-SAC and particle swarm optimization-SAC confirm that TPE outperforms population-based meta-heuristics by 1.3%–1.8% under identical evaluation budgets. Sensitivity analysis across four uncertainty levels (±10% to ±40%) and 30 extreme stress-test scenarios confirms the advantage holds across all tested levels. Overall, the framework offers a practical path toward uncertainty-aware IES co-optimization.