An Improved Soft Actor–Critic Framework for Cooperative Energy Management in the Building Cluster

Abstract

Buildings are significant contributors to global energy consumption and greenhouse gas emissions, with air conditioning systems representing a large share of this demand. Multi-building cooperative energy management is a promising solution for improving energy efficiency, but traditional control methods often struggle with dynamic environments and complex interactions. This study proposes an enhanced Soft Actor–Critic (SAC) algorithm, termed ORAR-SAC, to address these challenges in building cluster energy management. The ORAR-SAC integrates an Ordered Reward-based Experience Replay mechanism to prioritize high-value samples, improving data utilization and accelerating policy convergence. Additionally, an adaptive temperature parameter regularization strategy is implemented to balance exploration and exploitation dynamically, enhancing training stability and policy robustness. Using the CityLearn simulation platform, the proposed method is evaluated on a cluster of three commercial buildings in Beijing under time-of-use electricity pricing. Results demonstrate that ORAR-SAC outperforms conventional rule-based and standard SAC strategies, achieving reductions of up to 11% in electricity costs, 7% in peak demand, and 3.5% in carbon emissions while smoothing load profiles and improving grid compatibility. These findings highlight the potential of ORAR-SAC to support intelligent, low-carbon building energy systems and advance sustainable urban energy management.

Keywords

• multi-building energy management
• Deep Reinforcement Learning
• Soft Actor–Critic (SAC)