Local integrated energy system operational optimization considering multi‐type uncertainties: A reinforcement learning approach based on improved TD3 algorithm

Abstract

Abstract Local integrated energy system (IES), usually a multi‐building heating and cooling system incorporating cogeneration systems and distributed energy resources (DERs), is becoming an efficient energy infrastructure for energy system decarbonization. However, for small and medium‐scale local IESs, the fluctuation of user loads seriously influences energy system operation, and the increasing DER penetration also enlarges the influence. Given the necessity of exploring an efficient way to handle the uncertainties in local IES, this paper proposes a reinforcement learning (RL) approach based on the improved TD3 algorithm. The mathematical model of local IES is first established considering supply‐ and load‐side flexible resources. The local IES dispatch problem is formulated as a Markov decision process (MDP), in which multi‐type uncertainties of renewable generation, electric load and heat load are considered. For solving the MDP, an improved twin delayed deep deterministic policy gradient (TD3) algorithm is proposed with a dynamic balance mechanism of exploration noise. Based on a local IES testbed in the Nantong Central Innovation District, China, a comparison analysis is conducted to verify the promoting effect of flexible resources on the operation economy and renewable energy consumption. The system operating cost reduces by 18.46%, and surplus renewable energy can all be accommodated considering flexible resources. The dispatch policies obtained by the deep deterministic policy gradient (DDPG), the improved TD3, the original TD3 and traditional optimization algorithms are also compared. The results reveal that the convergence stability and solving accuracy of the improved TD3 outperform the other two RL algorithms. Specifically, the system operating cost of the improved TD3 reduces by 2.76% compared with the DDPG, and the energy supply imbalance decreases by around 88%. Meanwhile, the improved TD3 exhibits better operation economy and adaptability to the uncertain environment than the deterministic optimization and intraday rolling algorithms.

Keywords

• flexible resource
• local integrated energy system (IES)
• optimal dispatch
• reinforcement learning (RL)
• twin delayed deep deterministic policy gradient (TD3)
• uncertainty