Empowering Dynamic Active and Reactive Power Control: A Deep Reinforcement Learning Controller for Three-Phase Grid-Connected Electric Vehicles

Abstract

Advancing bidirectional voltage source inverters in the on-board chargers of electric vehicles is vital for smart grid electric power management. These electric vehicle inverters play a crucial role in actively contributing to active/reactive power flow control in vehicle-to-grid and grid-to-vehicle applications. In this research, a proof-of-concept, model-free reinforcement learning-based agent/controller is introduced that eliminates the need for system identification or linearization techniques for control design. The goal is to dynamically control user-specified active and reactive electric vehicle on-board charger power outputs in all four quadrants for three-phase grid-connected electric vehicle voltage source inverters. A Twin-Delayed Deep Deterministic Policy Gradient deep-reinforcement learning-based controller was designed (1) to smoothly learn continuous control variables and to generate deterministic actions from the learned control policy, (2) to avoid exhaustive system identification techniques associated with nonlinearities involved in the on-board charger of electric vehicles, and (3) to ultimately track continuous active and reactive powers across all four-quadrant operation modes, ensuring safe and minimal steady-state errors. In this paper, a Markov Decision Process framework was established, encompassing the environment, actions, states, and a unique reward function. The minimal steady state error achieved in active and reactive power reference tracking across all four quadrant operation modes substantiates the trained agent’s capacity to sustain tracking accuracy and stability across a variety of operational settings. This research could potentially pave the way for any type of electric vehicle to actively play a role in smart grid management, especially through vehicle-to-grid and grid-to-vehicle capabilities.

Keywords

• Electric vehicle (EV)
• on-board charger (OBC)
• voltage source inverter (VSI)
• reinforcement learning (RL)