Interpretable Multi-agent Reinforcement Learning for Continuous Double Auction-based P2P Energy Trading

Peer-to-peer (P2P) energy trading in a multiple-microgrid (MMG) community is emerging with the proliferation of distributed energy resources in the distribution network. To achieve maximum social welfare and privacy preservation, continuous double auction (CDA) serves as an effective mechanism. However, bidding decision-making in the P2P market is challenging owing to intense market dynamics and a lack of a coordinated bidding strategy. Despite enabling intelligent bidding, existing multi-agent reinforcement learning (MARL) methods often lack a solid theoretical understanding of market behavior. In this context, this paper proposes a theoretically interpretable Trading Policy Monotonic Improved Learning (TPMIL) template featuring rigorous proof and sequential updates to the agents. It facilitates the interpretation of bidding strategies, understanding of market status through preceding agents' policies, and coordinated joint policy updates. Furthermore, a practical algorithm, i.e., Monotonic-Improved Trust Region Policy Optimization (MI-TRPO), is cast onto the template to optimize the bidding strategy, which eliminates potential bidding conflicts among peers and stabilizes the training process. Case studies on the modified IEEE 33-bus system show the proposed method can yield stable P2P outcomes and reduce community costs.