Coordinated Charging Strategy Applicable to Large-scale Charging Stations Based on Deep Reinforcement Learning

CHEN Guo; WANG Xiuli; YUAN Shengqi; SHUAI Xuanyue; ZHOU Qian

doi:10.7500/AEPS20211215012

您当前的位置：

首页 >

文章列表页 >

Coordinated Charging Strategy Applicable to Large-scale Charging Stations Based on Deep Reinforcement Learning

Basic Research | 更新时间：2025-06-16

- Coordinated Charging Strategy Applicable to Large-scale Charging Stations Based on Deep Reinforcement Learning
- Automation of Electric Power Systems Vol. 47, Issue 2, Pages: 88-95(2023)
- 作者机构：
  
  1.西安交通大学电气工程学院,陕西省西安市 710049
  2.国网江苏省电力有限公司电力科学研究院,江苏省南京市 210036
- 作者简介：
- 基金信息：
  
  State Grid Corporation of China(5400-202099508A-0-0-00)
- DOI：10.7500/AEPS20211215012
  CLC：
- Received：15 December 2021，
  
  Revised：2022-04-22，
  
  Published Online：16 September 2022，
  
  Published：25 January 2023
- 稿件说明：
移动端阅览
陈果,王秀丽,原晟淇等.适用于大规模充电场站的深度强化学习有序充电策略[J].电力系统自动化,2023,47(2):88-95.

CHEN Guo,WANG Xiuli,YUAN Shengqi,et al.Coordinated Charging Strategy Applicable to Large-scale Charging Stations Based on Deep Reinforcement Learning[J].Automation of Electric Power Systems,2023,47(2):88-95.
陈果,王秀丽,原晟淇等.适用于大规模充电场站的深度强化学习有序充电策略[J].电力系统自动化,2023,47(2):88-95. DOI： 10.7500/AEPS20211215012.

CHEN Guo,WANG Xiuli,YUAN Shengqi,et al.Coordinated Charging Strategy Applicable to Large-scale Charging Stations Based on Deep Reinforcement Learning[J].Automation of Electric Power Systems,2023,47(2):88-95. DOI： 10.7500/AEPS20211215012.

摘要

针对大型充电场站内规模化电动汽车的有序充电问题，提出一种基于双深度Q网络（DDQN）深度强化学习方法的电动汽车充电安排策略，能有效计及电动汽车出行模式和充电需求的不确定性，实现充电场站充电成本最小化的目标。首先，对电动汽车泊车时间和充电需求特征进行提取；其次，为解决维数灾问题，提出一种“分箱”方法和充电次序优化策略以控制状态和动作空间的大小，从而建立了一种适用于大规模电动汽车有序充电的马尔可夫决策过程（MDP）模型；然后，应用DDQN的强化学习算法对电动汽车有序充电策略进行求解；最后，通过仿真算例验证了所提方法的有效性

不仅能有效减少充电场站的充电成本，而且能使模型训练难度不受电动汽车规模影响。

Abstract

Aiming at the coordinated charging problem of large-scale electric vehicles (EVs) within a large charging station

an EV charging scheduling strategy based on double deep Q network (DDQN) of deep reinforcement learning is proposed

which can effectively take into account the uncertainty of traveling pattern and charging demand of EVs and achieve the objective of minimizing the charging cost of the charging station. Firstly

characteristics of the parking time and the charging demand of EVs are extracted. Secondly

to solve the curse of dimensionality problem

a binning method and an optimal charging order strategy are proposed to control the size of the state and action space

and thus a Markov decision process (MDP) model which is applicable to the coordinated charging of large-scale EVs is established. Then

the DDQN of reinforcement learning algorithm is used to calculate the EV coordinated charging strategy. Finally

the validity of the proposed method is verified through a simulation example. It is verified that not only the charging cost of the charging station can be effectively reduced

but also the difficulty of model training becomes unaffected by the scale of EVs.

关键词

Keywords

references

陈吕鹏，潘振宁，余涛，等 . 基于动态非合作博弈的大规模电动汽车实时优化调度［J］. 电力系统自动化， 2019 ， 43 （ 24 ）： 32 - 40 .

CHEN Lüpeng ， PAN Zhenning ， YU Tao ， et al . Real-time optimal dispatch for large-scale electric vehicles based on dynamic non-cooperative game theory ［J］. Automation of Electric Power Systems ， 2019 ， 43 （ 24 ）： 32 - 40 .

SUN B ， HUANG Z ， TAN X Q ， et al . Optimal scheduling for electric vehicle charging with discrete charging levels in distribution grid ［J］. IEEE Transactions on Smart Grid ， 2018 ， 9 （ 2 ）： 624 - 634 .

张良，孙成龙，蔡国伟，等 . 基于PSO算法的电动汽车有序充放电两阶段优化策略［J］. 中国电机工程学报， 2022 ， 42 （ 5 ）： 1837 - 1852 .

ZHANG Liang ， SUN Chenglong ， CAI Guowei ， et al . Two-stage optimization strategy for coordinated charging and discharging of EVs based on PSO algorithm ［J］. Proceedings of the CSEE ， 2022 ， 42 （ 5 ）： 1837 - 1852 .

程杉，王贤宁，冯毅煁 . 电动汽车充电站有序充电调度的分散式优化［J］. 电力系统自动化， 2018 ， 42 （ 1 ）： 39 - 46 .

CHENG Shan ， WANG Xianning ， FENG Yichen . Decentralized optimization of ordered charging scheduling in electric vehicle charging station ［J］. Automation of Electric Power Systems ， 2018 ， 42 （ 1 ）： 39 - 46 .

SHI Y ， TUAN H D ， SAVKIN A V ， et al . Model predictive control for smart grids with multiple electric-vehicle charging stations ［J］. IEEE Transactions on Smart Grid ， 2019 ， 10 （ 2 ）： 2127 - 2136 .

ZHENG Y ， SONG Y ， HILL D J ， et al . Online distributed MPC-based optimal scheduling for EV charging stations in distribution systems ［J］. IEEE Transactions on Industrial Informatics ， 2019 ， 15 （ 2 ）： 638 - 649 .

龙雪梅，杨军，吴赋章，等 . 考虑路网-电网交互和用户心理的电动汽车充电负荷预测［J］. 电力系统自动化， 2020 ， 44 （ 14 ）： 86 - 93 .

LONG Xuemei ， YANG Jun ， WU Fuzhang ， et al . Prediction of electric vehicle charging load considering interaction between road network and power grid and user’s psychology ［J］. Automation of Electric Power Systems ， 2020 ， 44 （ 14 ）： 86 - 93 .

SUTTON R ， BARTO A . Reinforcement learning： an introduction ［J］. IEEE Transactions on Neural Networks ， 2005 ， 16 ： 285 - 286 .

孙毅，刘迪，李彬，等 . 深度强化学习在需求响应中的应用［J］. 电力系统自动化， 2019 ， 43 （ 5 ）： 183 - 191 .

SUN Yi ， LIU Di ， LI Bin ， et al . Application of deep reinforcement learning in demand response ［J］. Automation of Electric Power Systems ， 2019 ， 43 （ 5 ）： 183 - 191 .

WAN Z Q ， LI H P ， HE H B ， et al . Model-free real-time EV charging scheduling based on deep reinforcement learning ［J］. IEEE Transactions on Smart Grid ， 2019 ， 10 （ 5 ）： 5246 - 5257 .

YAN L F ， CHEN X ， ZHOU J Y ， et al . Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors ［J］. IEEE Transactions on Smart Grid ， 2021 ， 12 （ 6 ）： 5124 - 5134 .

WANG S Y ， BI S Z ， ZHANG Y A . Reinforcement learning for real-time pricing and scheduling control in EV charging stations ［J］. IEEE Transactions on Industrial Informatics ， 2021 ， 17 （ 2 ）： 849 - 859 .

杜明秋，李妍，王标，等 . 电动汽车充电控制的深度增强学习优化方法［J］. 中国电机工程学报， 2019 ， 39 （ 14 ）： 4042 - 4049 .

DU Mingqiu ， LI Yan ， WANG Biao ， et al . Deep reinforcement learning optimization method for charging control of electric vehicles ［J］. Proceedings of the CSEE ， 2019 ， 39 （ 14 ）： 4042 - 4049 .

刘敦楠，王玲湘，汪伟业，等 . 基于深度强化学习的大规模电动汽车充换电负荷优化调度［J］. 电力系统自动化， 2022 ， 46 （ 4 ）： 36 - 46 .

LIU Dunnan ， WANG Lingxiang ， WANG Weiye ， et al . Optimal scheduling of electric vehicle load for large-scale battery charging and swapping based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 4 ）： 36 - 46 .

李航，李国杰，汪可友 . 基于深度强化学习的电动汽车实时调度策略［J］. 电力系统自动化， 2020 ， 44 （ 22 ）： 161 - 167 .

LI Hang ， LI Guojie ， WANG Keyou . Real-time dispatch strategy for electric vehicles based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2020 ， 44 （ 22 ）： 161 - 167 .

闫其粹 . 解决强化学习中维数灾问题的方法研究［D］. 苏州：苏州大学， 2010 .

YAN Qicui . Researches on the curse of dimensionality in reinforcement learning ［D］. Suzhou ： Suzhou University ， 2010 .

SADEGHIANPOURHAMAMI N ， DELEU J ， DEVELDER C . Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 1 ）： 203 - 214 .

JIN J L ， XU Y J . Optimal policy characterization enhanced actor-critic approach for electric vehicle charging scheduling in a power distribution network ［J］. IEEE Transactions on Smart Grid ， 2021 ， 12 （ 2 ）： 1416 - 1428 .

VAN HASSELT H ， GUEZ A ， SILVER D . Deep reinforcement learning with double Q-learning ［J］. Proceedings of the AAAI Conference on Artificial Intelligence ， 2016 ， 30 （ 1 ）： 10295 .

高宇，李昀，曹蓉蓉，等 . 基于多代理Double DQN算法模拟发电侧竞价行为［J］. 电网技术， 2020 ， 44 （ 11 ）： 4175 - 4183 .

GAO Yu ， LI Yun ， CAO Rongrong ， et al . Simulation of generators’ bidding behavior based on multi-agent Double DQN ［J］. Power System Technology ， 2020 ， 44 （ 11 ）： 4175 - 4183 .

YAO L ， LIM W H ， TSAI T S . A real-time charging scheme for demand response in electric vehicle parking station ［J］. IEEE Transactions on Smart Grid ， 2017 ， 8 （ 1 ）： 52 - 62 .

California independent system operator ［DB/OL］.［ 2021-12-06 ］. http：//oasis.caiso.com/mrioasis/logon.do http://oasis.caiso.com/mrioasis/logon.do .

LAHARIYA M ， BENOIT D F ， DEVELDER C . Synthetic data generator for electric vehicle charging sessions： modeling and evaluation using real-world data ［J］. Energies ， 2020 ， 13 （ 16 ）： 4211 - 4229 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Voltage Regulation Strategy for Active Distribution Network Based on Multi-agent Deep Reinforcement Learning

Bi-layer Coordinated Control Strategy of Distribution Network Considering Participation of Electric Vehicles and Microgrid

Deep Reinforcement Learning Approach for Dual-timescale Voltage Management in Distribution System

Related Author

Guo CHEN

Xiuli WANG

Shengqi YUAN

Xuanyue SHUAI

Qian ZHOU

HUANG Yu

ZHANG Xiaoxiao

YUE Dong

Related Institution

Institute of Advanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications

NARI Technology Co., Ltd.

NARI Group Corporation (State Grid Electric Power Research Institute)

Hubei Engineering and Technology Research Center for AC/DC Intelligent Distribution Network

School of Electrical Engineering and Automation, Wuhan University

⁰