Multi-objective Optimal Scheduling Strategy for Energy Storage System Based on Improved Multi-agent Deep Deterministic Policy Gradient Algorithm

WANG Yufei; WANG Yishun; XUE Hua; TU Yiyun; LIN Shunfu

doi:10.7500/AEPS20231217002

您当前的位置：

首页 >

文章列表页 >

Multi-objective Optimal Scheduling Strategy for Energy Storage System Based on Improved Multi-agent Deep Deterministic Policy Gradient Algorithm

Basic Research | 更新时间：2025-06-16

- Multi-objective Optimal Scheduling Strategy for Energy Storage System Based on Improved Multi-agent Deep Deterministic Policy Gradient Algorithm
- Automation of Electric Power Systems Vol. 48, Issue 23, Pages: 98-111(2024)
- 作者机构：
  
  上海电力大学电气工程学院，上海市 200090
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(51977127);Shanghai Science and Technology Innovation Action Plan(22010501400)
- DOI：10.7500/AEPS20231217002
  CLC：
- Received：17 December 2023，
  
  Revised：2024-02-25，
  
  Published Online：03 June 2024，
  
  Published：10 December 2024
- 稿件说明：
移动端阅览
王育飞,王义顺,薛花等.基于改进MADDPG算法的储能系统多目标优化调度策略[J].电力系统自动化,2024,48(23):98-111.

WANG Yufei,WANG Yishun,XUE Hua,et al.Multi-objective Optimal Scheduling Strategy for Energy Storage System Based on Improved Multi-agent Deep Deterministic Policy Gradient Algorithm[J].Automation of Electric Power Systems,2024,48(23):98-111.
王育飞,王义顺,薛花等.基于改进MADDPG算法的储能系统多目标优化调度策略[J].电力系统自动化,2024,48(23):98-111. DOI： 10.7500/AEPS20231217002.

WANG Yufei,WANG Yishun,XUE Hua,et al.Multi-objective Optimal Scheduling Strategy for Energy Storage System Based on Improved Multi-agent Deep Deterministic Policy Gradient Algorithm[J].Automation of Electric Power Systems,2024,48(23):98-111. DOI： 10.7500/AEPS20231217002.

摘要

针对储能系统在新能源消纳中利用率低的问题，考虑利用其闲置功率和容量为配电网提供有功和无功辅助服务，提出一种基于改进多智能体深度确定性策略梯度（MADDPG）算法的储能系统多目标优化调度策略。首先，设计储能参与多应用场景协同运行模式，在满足新能源消纳要求的基础上，利用储能参与辅助调频和调压服务，建立计及全系统供电成本、节点电压偏移和调频功率偏差的储能多目标优化调度模型；然后，将储能调度问题转化为马尔可夫博弈过程，并通过构造双Critic网络结构和延迟Actor网络更新操作改进MADDPG算法，降低强化学习中动作价值高估问题对训练的影响；最后，在改进IEEE 33节点测试系统上进行仿真分析，结果验证了所提策略的有效性及优越性。

Abstract

Aiming at the problem of low utilization rate of the energy storage system in the renewable energy accommodation

considering using its idle power and capacity to provide active and reactive power auxiliary services for the distribution network

a multi-objective optimal scheduling strategy for energy storage system based on deep reinforcement learning using improved multi-agent deep deterministic policy gradient (MADDPG) algorithm is proposed. First

the collaborative operation mode of energy storage participating in multiple application scenarios is designed. Based on meeting the requirements of renewable energy accommodation

energy storage is used to participate in auxiliary frequency regulation and voltage regulation services

and a multi-objective optimal scheduling model of energy storage considering total system power supply cost

node voltage offset and frequency regulation power deviation is established. Then

the energy storage scheduling problem is transformed into a Markov game process

and the MADDPG algorithm is improved by constructing a dual-critic network structure and a delayed actor network update operation to reduce the impact of the action value overestimation problem on training in reinforcement learning. Finally

the simulation analysis is carried out on a modified IEEE 33-bus test system

and the results verify the effectiveness and superiority of the proposed strategy.

关键词

Keywords

references

国家发展改革委，国家能源局 . 关于建立健全可再生能源电力消纳保障机制的通知［EB/OL］.［ 2023-09-01 ］. https：//www.ndrc.gov.cn/xxgk/zcfb/tz/201905/t20190515_962446.html https://www.ndrc.gov.cn/xxgk/zcfb/tz/201905/t20190515_962446.html .

National Development and Reform Commission ， National Energy Administration . Notice on establishing and improving the renewable energy electricity consumption guarantee mechanism ［EB/OL］. ［ 2023-09-01 ］. https：//www.ndrc.gov.cn/xxgk/zcfb/tz/201905/t20190515_962446.html https://www.ndrc.gov.cn/xxgk/zcfb/tz/201905/t20190515_962446.html .

刘红 . 基于改进粒子群算法的储能调峰容量优化配置研究［J］. 广东电力， 2023 ， 36 （ 1 ）： 68 - 76 .

LIU Hong . Research on optimal configuration of energy storage peak shaving capacity based on improved particle swarm optimization algorithm ［J］. Guangdong Electric Power ， 2023 ， 36 （ 1 ）： 68 - 76 .

李慧玲，王维军，廖亚特，等 . 基于电动汽车充电管理的村级光伏发电系统公用储能配置优化研究［J］. 广东电力， 2023 ， 36 （ 12 ）： 30 - 38 .

LI Huiling ， WANG Weijun ， LIAO Yate ， et al . Optimization study of public energy storage configuration for village photovoltaic power generation system based on electric vehicle charging management ［J］. Guangdong Electric Power ， 2023 ， 36 （ 12 ）： 30 - 38 .

LIU W ， XU Y ， FENG X ， et al . Optimal fuzzy logic control of energy storage systems for V / f support in distribution networks considering battery degradation ［J］. International Journal of Electrical Power & Energy Systems ， 2022 ， 139 ： 107867 .

宋天昊，李柯江，韩肖清，等 . 储能系统参与多应用场景的协同运行策略［J］. 电力系统自动化， 2021 ， 45 （ 19 ）： 43 - 51 .

SONG Tianhao ， LI Kejiang ， HAN Xiaoqing ， et al . Coordinated operation strategy of energy storage system participating in multiple application scenarios ［J］. Automation of Electric Power Systems ， 2021 ， 45 （ 19 ）： 43 - 51 .

CHEN H H ， WANG D ， ZHANG R F ， et al . Optimal participation of ADN in energy and reserve markets considering TSO-DSO interface and DERs uncertainties ［J］. Applied Energy ， 2022 ， 308 ： 118319 .

王枭，何怡刚，马恒瑞，等 . 面向电网辅助服务的虚拟储能电厂分布式优化控制方法［J］. 电力系统自动化， 2022 ， 46 （ 10 ）： 181 - 188 .

WANG Xiao ， HE Yigang ， MA Hengrui ， et al . Distributed optimization control method of virtual energy storage plants for power grid ancillary services ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 10 ）： 181 - 188 .

周校聿，刘娆，鲍福增，等 . 百兆瓦级储能参与电网双重辅助服务调度的联合优化模型［J］. 电力系统自动化， 2021 ， 45 （ 19 ）： 60 - 69 .

ZHOU Xiaoyu ， LIU Rao ， BAO Fuzeng ， et al . Joint optimization model for hundred-megawatt-level energy storage participating in dual ancillary services dispatch of power grid ［J］. Automation of Electric Power Systems ， 2021 ， 45 （ 19 ）： 60 - 69 .

郑能，丁晓群，管志成，等 . 基于场景法的配电网有功-无功协调优化［J］. 电网技术， 2019 ， 43 （ 5 ）： 1640 - 1651 .

ZHENG Neng ， DING Xiaoqun ， GUAN Zhicheng ， et al . Coordinated optimization of active power and reactive power in distribution network based on scenario method ［J］. Power System Technology ， 2019 ， 43 （ 5 ）： 1640 - 1651 .

肖遥，别朝红，黄格超，等 . 基于二阶锥最优潮流的机会约束分布鲁棒优化研究［J］. 电网技术， 2021 ， 45 （ 4 ）： 1505 - 1518 .

XIAO Yao ， BIE Zhaohong ， HUANG Gechao ， et al . Chance-constrained distributional robust optimization based on second-order cone optimal power flow ［J］. Power System Technology ， 2021 ， 45 （ 4 ）： 1505 - 1518 .

梁栋，邱馨洁，刘琪，等 . 电动汽车与直流配电网协同的分布式自适应滚动预测调度［J/OL］. 电力系统自动化［ 2024-01-01 ］. https：//link.cnki.net/urlid/32.1180.TP.20231007.1722.006 https://link.cnki.net/urlid/32.1180.TP.20231007.1722.006 .

LIANG Dong ， QIU Xinjie ， LIU Qi ， et al . Distributed adaptive rolling predictive scheduling via coordination of electric vehicles and DC distribution networks ［J/OL］. Automation of Electric Power Systems ［ 2024-01-01 ］. https：//link.cnki.net/urlid/32.1180.TP.20231007.1722.006 https://link.cnki.net/urlid/32.1180.TP.20231007.1722.006 .

MORADI-SEPAHVAND M ， AMRAEE T . Secure expansion of energy storage and transmission lines considering bundling option under renewable penetration ［J］. Applied Energy ， 2023 ， 347 ： 121414 .

CANESE L ， CARDARILLI G C ， DI NUNZIO L ， et al . Multi-agent reinforcement learning： a review of challenges and applications ［J］. Applied Sciences ， 2021 ， 11 （ 11 ）： 4948 .

LOWE R ， WU Y ， TAMAR A ， et al . Multi-agent actor-critic for mixed cooperative-competitive environments ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems ， December 4-9 ， 2017 ， Long Beach， USA ： 6382 - 6393 .

SANCHEZ GOROSTIZA F ， GONZALEZ-LONGATT F M . Deep reinforcement learning-based controller for SOC management of multi-electrical energy storage system ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 6 ）： 5039 - 5050 .

XIANG Y ， LU Y ， LIU J Y . Deep reinforcement learning based topology-aware voltage regulation of distribution networks with distributed energy storage ［J］. Applied Energy ， 2023 ， 332 ： 120510 .

FUJIMOTO S ， VAN HOOF H ， MEGER D . Addressing function approximation error in actor-critic methods ［EB/OL］. ［ 2024-01-25 ］. http：//arxiv.org/abs/1802.09477 http://arxiv.org/abs/1802.09477 .

ZHANG Z Z ， PAN Z Y ， KOCHENDERFER M J . Weighted double Q-learning ［C］// Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ， August 19-26 ， 2017 ， Melbourne， Australia ： 3455 - 3461 .

ANSCHEL O ， BARAM N ， SHIMKIN N . Averaged-DQN： variance reduction and stabilization for deep reinforcement learning ［C］// Proceedings of the 34th International Conference on Machine Learning ， August 6-11 ， 2017 ， Sydney， Australia ： 176 - 185 .

VAN HASSELT H ， GUEZ A ， SILVER D . Deep reinforcement learning with double Q-learning ［C］// Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence ， February 12-17 ， 2016 ， Phoenix， USA ： 2094 - 2100 .

LIN C H ， WU W C ， ZHANG B M ， et al . Decentralized reactive power optimization method for transmission and distribution networks accommodating large-scale DG integration ［J］. IEEE Transactions on Sustainable Energy ， 2017 ， 8 （ 1 ）： 363 - 373 .

贾兆昊，张峰，丁磊 . 考虑功率四象限输出的配电网储能优化配置策略［J］. 电力系统自动化， 2020 ， 44 （ 2 ）： 105 - 113 .

JIA Zhaohao ， ZHANG Feng ， DING Lei . Optimal allocation strategy of energy storage in distribution network considering power four-quadrant output ［J］. Automation of Electric Power Systems ， 2020 ， 44 （ 2 ）： 105 - 113 .

ABOMAZID A M ， EL-TAWEEL N A ， FARAG H E Z . Optimal energy management of hydrogen energy facility using integrated battery energy storage and solar photovoltaic systems ［J］. IEEE Transactions on Sustainable Energy ， 2022 ， 13 （ 3 ）： 1457 - 1468 .

YI Z K ， XU Y L ， ZHOU J G ， et al . Bi-level programming for optimal operation of an active distribution network with multiple virtual power plants ［J］. IEEE Transactions on Sustainable Energy ， 2020 ， 11 （ 4 ）： 2855 - 2869 .

LI H ， WU Q W ， YANG L ， et al . Distributionally robust negative-emission optimal energy scheduling for off-grid integrated electricity-heat microgrid ［J］. IEEE Transactions on Sustainable Energy ， 2024 ， 15 （ 2 ）： 803 - 818 .

丁涛，贾文皓，黄雨涵，等 . 基于分布鲁棒机会约束的移动氢能系统制-储-运氢协同优化［J］. 电力系统自动化， 2023 ， 47 （ 23 ）： 1 - 11 .

DING Tao ， JIA Wenhao ， HUANG Yuhan ， et al . Collaborative optimization for hydrogen generation， storage， and transportation in mobile hydrogen energy systems based on distributionally robust chance constraint ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 23 ）： 1 - 11 .

WEN Y L ， HU Z C ， CHEN X L ， et al . Centralized distributionally robust chance-constrained dispatch of integrated transmission-distribution systems ［J］. IEEE Transactions on Power Systems ， 2024 ， 39 （ 2 ）： 2947 - 2959 .

邱威，贺静波，谭真，等 . 一种用于含直流馈入的受端电网故障筛选与排序方法［J］. 电网技术， 2020 ， 44 （ 9 ）： 3516 - 3524 .

QIU Wei ， HE Jingbo ， TAN Zhen ， et al . Contingency screening and ranking for receiving-end power grid with infeed HVDC ［J］. Power System Technology ， 2020 ， 44 （ 9 ）： 3516 - 3524 .

HAN X F ， HE H W ， WU J D ， et al . Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle ［J］. Applied Energy ， 2019 ， 254 ： 113708 .

韩平平，潘薇，张楠，等 . 基于负荷预测和非支配排序遗传算法的人工相序优化方法［J］. 电力系统自动化， 2020 ， 44 （ 20 ）： 71 - 78 .

HAN Pingping ， PAN Wei ， ZHANG Nan ， et al . Optimization method for artificial phase sequence based on load forecasting and non-dominated sorting genetic algorithm ［J］. Automation of Electric Power Systems ， 2020 ， 44 （ 20 ）： 71 - 78 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Active and Reactive Power Coordinated Optimal Dispatch of Networked Microgrids Based on Distributed Deep Reinforcement Learning

Multi-building Coordinated Dispatch in Smart Park for Carbon Emission Peak and Carbon Neutrality Considering Electricity and Hydrogen Complementary

Prediction and Decision Integrated Scheduling of Energy Storage System in Wind Farm Based on Deep Reinforcement Learning

Related Author

JU Yuntao

CHEN Xi

LI Jiawei

WANG Jie

FAN Hong

YU Weinan

LIU Lu

DOU Zhenlan

Related Institution

School of Electrical and Control Engineering, North China University of Technology

College of Information and Electrical Engineering, China Agricultural University

School of Electric Power Engineering, Shanghai University of Electric Power

Key Laboratory of Control of Power Transmission and Conversion, Ministry of Education(Shanghai Jiao Tong University)

State Grid Shanghai Comprehensive Energy Service Co., Ltd.

⁰