Charging Path Planning for Electric Vehicles Based on Reinforcement Learning Environment Design Strategy

SONG Yuhang; CHEN Yufan; WEI Yanling; GAO Shan

doi:10.7500/AEPS20230621004

您当前的位置：

首页 >

文章列表页 >

Charging Path Planning for Electric Vehicles Based on Reinforcement Learning Environment Design Strategy

Basic Research | 更新时间：2025-06-16

- Charging Path Planning for Electric Vehicles Based on Reinforcement Learning Environment Design Strategy
- Automation of Electric Power Systems Vol. 48, Issue 11, Pages: 184-196(2024)
- 作者机构：
  
  1.东南大学自动化学院,江苏省南京市 210096
  2.东南大学复杂工程系统测量与控制教育部重点实验室,江苏省南京市 210096
  3.东南大学电气工程学院,江苏省南京市 210096
- 作者简介：
- 基金信息：
  
  National Key R&D Program of China(2021YFB2501600)
- DOI：10.7500/AEPS20230621004
  CLC：
- Received：21 June 2023，
  
  Revised：2023-11-16，
  
  Published Online：28 April 2024，
  
  Published：10 June 2024
- 稿件说明：
移动端阅览
宋宇航,陈宇帆,魏延岭等.基于强化学习环境设计策略的电动汽车充电路径规划[J].电力系统自动化,2024,48(11):184-196.

SONG Yuhang,CHEN Yufan,WEI Yanling,et al.Charging Path Planning for Electric Vehicles Based on Reinforcement Learning Environment Design Strategy[J].Automation of Electric Power Systems,2024,48(11):184-196.
宋宇航,陈宇帆,魏延岭等.基于强化学习环境设计策略的电动汽车充电路径规划[J].电力系统自动化,2024,48(11):184-196. DOI： 10.7500/AEPS20230621004.

SONG Yuhang,CHEN Yufan,WEI Yanling,et al.Charging Path Planning for Electric Vehicles Based on Reinforcement Learning Environment Design Strategy[J].Automation of Electric Power Systems,2024,48(11):184-196. DOI： 10.7500/AEPS20230621004.

摘要

针对电动汽车充电路径规划问题，提出了一种适用于强化学习的环境建模方法。该方法基于城市道路网格与充电站地理位置分布等现实情况，将电动汽车的基本行驶路径分为三段进行表达。在三段式表达方法的基础上，提出了状态空间、动作空间、状态转移与奖励函数的设计方案，将充电路径规划建模为马尔可夫决策过程，并利用Q学习方法与深度Q网络（DQN）方法求解。实验结果表明，基于三段式表达法的强化学习环境设计方案具有可解性与可迁移性，考虑了电动汽车从道路驶向充电站过程中的降速转弯等现实场景，同时将充电动作简化为一种行驶方向选择，提升了基于Q学习与DQN的强化学习算法效率。

Abstract

An environmental modeling method suitable for reinforcement learning is proposed for the charging path planning problem of electric vehicles. Based on the actual situation of urban road network and geographical distribution of charging stations

this method divides the basic driving path of electric vehicles into three segments for representation. Based on the three-segment expression method

the design scheme of state space

action space

state transition

and reward function is proposed. The charging path planning is modeled as a Markov decision process

and solved by the Q learning method and the deep Q network (DQN) method. The experimental results show that the design scheme of the reinforcement learning environment based on the three-segment expression method is solvable and portable. It takes into account the realistic scenarios such as the deceleration and turning of electric vehicles in the process of driving from the road to the charging station

and simplifies the charging action into a driving direction choice

which improves the efficiency of the reinforcement learning algorithm based on Q learning and DQN.

关键词

Keywords

references

国家能源局 . 2022中国电动汽车用户充电行为白皮书［EB/OL］.［ 2023-03-20 ］. http：//www.nea.gov.cn/2023-03/20/c_1310703965.htm http://www.nea.gov.cn/2023-03/20/c_1310703965.htm .

Chinese National Energy Administration . White paper on charging behavior of Chinese electric vehicle users in 2022 ［EB/OL］. ［ 2023-03-20 ］. http：//www.nea.gov.cn/2023-03/20/c_1310703965.htm http://www.nea.gov.cn/2023-03/20/c_1310703965.htm .

XU Z ， LIU X ， CHEN Q L . Application of improved Astar algorithm in global path planning of unmanned vehicles ［C］// Chinese Automation Congress （CAC）， November 22-24 ， 2019 ， Hangzhou， China .

CHU Y L ， ZHANG T ， GE P S ， et al . Unmanned vehicle path planning based on improved ant colony algorithm ［C］// 6th CAA International Conference on Vehicular Control and Intelligence ， October 28-30 ， 2022 ， Nanjing， China .

REN J M ， WANG H ， WEI Y ， et al . A novel genetic algorithm-based emergent electric vehicle charging scheduling scheme ［C］// 45th Annual Conference of the IEEE Industrial Electronics Society ， October 14-17 ， 2019 ， Lisbon， Portugal .

崔桂香 . 基于交通因素分析的电动汽车运行规划优化研究［D］. 杭州：杭州电子科技大学， 2021 .

CUI Guixiang . Research on optimization of electric vehicle operation planning based on traffic factor analysis ［D］. Hangzhou ： Hangzhou Dianzi University ， 2021 .

李玉璟 . 多信息互联环境下的电动汽车充电路径规划方法研究［D］. 北京：北京交通大学， 2020 .

LI Yujing . Research on the charging path planning method for electric vehicles in a multi information interconnected environment ［D］. Beijing ： Beijing Jiaotong University ， 2020 .

邢强，陈中，冷钊莹，等 . 基于实时交通信息的电动汽车路径规划和充电导航策略［J］. 中国电机工程学报， 2020 ， 40 （ 2 ）： 534 - 550 .

XING Qiang ， CHEN Zhong ， LENG Zhaoying ， et al . Route planning and charging navigation strategy for electric vehicles based on real-time traffic information ［J］. Proceedings of the CSEE ， 2020 ， 40 （ 2 ）： 534 - 550 .

葛晓琳，何鈜博，符杨，等 . 融合分层规划和A-star算法的共享电动汽车换车与充电路径规划［J］. 中国电机工程学报， 2021 ， 41 （ 22 ）： 7668 - 7681 .

GE Xiaolin ， HE Hongbo ， FU Yang ， et al . Interchange and charging path planning of shared electric vehicles based on A-star algorithm combined with hierarchical programming ［J］. Proceedings of the CSEE ， 2021 ， 41 （ 22 ）： 7668 - 7681 .

詹华，江昌旭，苏庆列 . 基于分层强化学习的电动汽车充电引导方法［J］. 电力自动化设备， 2022 ， 42 （ 10 ）： 264 - 272 .

ZHAN Hua ， JIANG Changxu ， SU Qinglie . Electric vehicle charging navigation method based on hierarchical reinforcement learning ［J］. Electric Power Automation Equipment ， 2022 ， 42 （ 10 ）： 264 - 272 .

JIN J L ， XU Y J . Shortest-path-based deep reinforcement learning for EV charging routing under stochastic traffic condition and electricity prices ［J］. IEEE Internet of Things Journal ， 2022 ， 9 （ 22 ）： 22571 - 22581 .

HOU W C ， LUO Q S ， WU X D ， et al . Multiobjective optimization of large-scale EVs charging path planning and charging pricing strategy for charging station ［J］. Complexity ， 2021 ： 1 - 17 .

BOEING G . OSMnx： new methods for acquiring， constructing， analyzing， and visualizing complex street networks ［J］. Computers， Environment and Urban Systems ， 2017 ， 65 ： 126 - 139 .

全国电动汽车充电站分布［EB/OL］.［ 2023-03-10 ］. http：//admin.bjev520.com/jsp/beiqi/pcmap/do/pcMap.jsp？ http://admin.bjev520.com/jsp/beiqi/pcmap/do/pcMap.jsp? .

Distribution of electric vehicle charging stations nationwide ［EB/OL］. ［ 2023-03-10 ］. http：//admin.bjev520.com/jsp/beiqi/pcmap/do/pcMap.jsp？ http://admin.bjev520.com/jsp/beiqi/pcmap/do/pcMap.jsp? .

百度地图API开发指南［EB/OL］.［ 2023-05-04 ］. https：//lbsyun.baidu.com/index.php？title=open/dev-res https://lbsyun.baidu.com/index.php?title=open/dev-res .

Baidu map API development guide ［EB/OL］. ［ 2023-05-04 ］. https：//lbsyun.baidu.com/index.php？title=open/dev-res https://lbsyun.baidu.com/index.php?title=open/dev-res .

王典，周阳，宋毅，等 . 基于Q学习的生物启发式目标导向导航路径规划模型［J］. 电子测量与仪器学报， 2023 ， 37 （ 6 ）： 68 - 76 .

WANG Dian ， ZHOU Yang ， SONG Yi ， et al . Model of path planning in biological inspired goal-oriented navigation based on Q-learning ［J］. Journal of Electronic Measurement and Instrumentation ， 2023 ， 37 （ 6 ）： 68 - 76 .

王冰晨，连晓峰，颜湘，等 . 基于深度Q网络和人工势场的移动机器人路径规划研究［J］. 计算机测量与控制， 2022 ， 30 （ 11 ）： 226 - 232 .

WANG Bingchen ， LIAN Xiaofeng ， YAN Xiang ， et al . Research on path planning of mobile robot based on deep Q-network and artificial potential field ［J］. Computer Measurement & Control ， 2022 ， 30 （ 11 ）： 226 - 232 .

周椿奇，向月，童话，等 . 轨迹数据驱动的电动汽车充电需求及V2G可调控容量估计［J］. 电力系统自动化， 2022 ， 46 （ 12 ）： 46 - 55 .

ZHOU Chunqi ， XIANG Yue ， TONG Hua ， et al . Trajectory-data-driven estimation of electric vehicle charging demand and vehicle-to-grid regulable capacity ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 12 ）： 46 - 55 .

DAS H S ， RAHMAN M M ， LI S ， et al . Electric vehicles standards， charging infrastructure， and impact on grid integration： a technological review ［J］. Renewable and Sustainable Energy Reviews ， 2020 ， 120 ： 109618 .

张廷 . 纯电动乘用车行驶里程建模与分析［D］. 北京：北京交通大学， 2017 .

ZHANG Ting . Modeling and analysis of mileage of pure electric passenger car ［D］. Beijing ： Beijing Jiaotong University ， 2017 .

LI N ， LIU Y S ， ZHANG J Z ， et al . Research on energy consumption evaluation of electric vehicles for thermal comfort ［J］. Environmental Science and Pollution Research ， 2022 ， 4 （ 6 ）： 60 - 68 .

苏粟，李玉璟，夏明超，等 . 基于时空耦合特性的充电站运行状态预测［J］. 电力系统自动化， 2022 ， 46 （ 3 ）： 23 - 32 .

SU Su ， LI Yujing ， XIA Mingchao ， et al . Operation state prediction of charging station based on spatio-temporal coupling characteristics ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 3 ）： 23 - 32 .

MapTiler . epsg：3857 ［EB/OL］. ［ 2023-05-04 ］. https：//epsg.io/3857 https://epsg.io/3857 .

CHOI D I ， LIM D E . Analysis of the state-dependent queueing model and its application to battery swapping and charging stations ［J］. Sustainability ， 2020 ， 12 （ 6 ）： 2343 .

刘敦楠，王玲湘，汪伟业，等 . 基于深度强化学习的大规模电动汽车充换电负荷优化调度［J］. 电力系统自动化， 2022 ， 46 （ 4 ）： 36 - 46 .

LIU Dunnan ， WANG Lingxiang ， WANG Weiye ， et al . Optimal scheduling of electric vehicle load for large-scale battery charging and swapping based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 4 ）： 36 - 46 .

HUNTER J D . Matplotlib： a 2D graphics environment ［J］. Computing in Science & Engineering ， 2007 ， 9 （ 3 ）： 90 - 95 .

但光华 . 基于深度强化学习的多智能体路径规划研究［D］. 成都：电子科技大学， 2023 .

DAN Guanghua . Research on multi-agent path planning based on deep reinforcement learning ［D］. Chengdu ： University of Electronic Science and Technology of China ， 2023 .

沈国辉，赵荣生，董晓，等 . 基于多信息交互与深度强化学习的电动汽车充电导航策略［J］. 南方电网技术， 2022 ， 16 （ 1 ）： 108 - 116 .

SHEN Guohui ， ZHAO Rongsheng ， DONG Xiao ， et al . Electric vehicle charging navigation strategy based on multi-information interaction and deep reinforcement learning ［J］. Southern Power System Technology ， 2022 ， 16 （ 1 ）： 108 - 116 .

HAO J J ， HOU H ， ZHANG Y B ， et al . Integrated electric vehicle charging path planning considering traffic network and power grid ［C］// 4th Asia Energy and Electrical Engineering Symposium （AEEES）， March 25-28 ， 2022 ， Chengdu， China .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Optimal Charging Strategies for Electric Vehicles Supporting Time-varying Demand Response Incentive Price

Day-ahead Topology Optimization for Wind Farm Without Forecasting Based on Improved Deep Q Network

Multi-period Online Preventive Control Strategy for Power Grid with High Proportion of Renewable Energy Based on Reinforcement Learning

Related Author

ZHENG Yingying

MIAO Xinyi

WANG Hui

ZHAO Yongning

ZHENG Yingying

MIAO Xinyi

WANG Hui

ZHAO Yongning

Related Institution

College of Information and Electrical Engineering, China Agricultural University

College of Electrical and Information Engineering, Hunan University

Guangzhou Power Supply Bureau of Guangdong Power Grid Co., Ltd.

School of Electrical Engineering and Automation, Wuhan University

Key Laboratory of Modern Power System Simulation and Control & Renewable Energy Technology, Ministry of Education (Northeast Electric Power University)

⁰