Energy Management Strategy for Multi-microgrid Based on Improved Federated Dueling Deep Q Network

LI Haitao; LIU Yiran; YANG Yanhong; XIAO Hao; XIE Dongxue; PEI Wei

doi:10.7500/AEPS20230521001

您当前的位置：

首页 >

文章列表页 >

Energy Management Strategy for Multi-microgrid Based on Improved Federated Dueling Deep Q Network

Basic Research | 更新时间：2025-06-16

- Energy Management Strategy for Multi-microgrid Based on Improved Federated Dueling Deep Q Network
- Automation of Electric Power Systems Vol. 48, Issue 8, Pages: 174-184(2024)
- 作者机构：
  
  1.北京工业大学信息学部，北京市 100124
  2.中国科学院电工研究所，北京市 100190
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(52277131);Youth Innovation Promotion Association, Chinese Academy of Sciences(2021136)
- DOI：10.7500/AEPS20230521001
  CLC：
- Received：21 May 2023，
  
  Revised：2023-12-12，
  
  Published Online：23 February 2024，
  
  Published：25 April 2024
- 稿件说明：
移动端阅览
黎海涛,刘伊然,杨艳红等.基于改进联邦竞争深度Q网络的多微网能量管理策略[J].电力系统自动化,2024,48(8):174-184.

LI Haitao,LIU Yiran,YANG Yanhong,et al.Energy Management Strategy for Multi-microgrid Based on Improved Federated Dueling Deep Q Network[J].Automation of Electric Power Systems,2024,48(8):174-184.
黎海涛,刘伊然,杨艳红等.基于改进联邦竞争深度Q网络的多微网能量管理策略[J].电力系统自动化,2024,48(8):174-184. DOI： 10.7500/AEPS20230521001.

LI Haitao,LIU Yiran,YANG Yanhong,et al.Energy Management Strategy for Multi-microgrid Based on Improved Federated Dueling Deep Q Network[J].Automation of Electric Power Systems,2024,48(8):174-184. DOI： 10.7500/AEPS20230521001.

摘要

目前，基于联邦深度强化学习的微网（MG）能量管理研究未考虑多类型能量转换与MG间电量交易的问题，同时，频繁交互模型参数导致通信时延较大。基于此，以一种包含风、光、电、气等多类型能源的MG为研究对象，构建了支持MG间电量交易和MG内能量转换的能量管理模型，提出基于正余弦算法的联邦竞争深度Q网络学习算法，并基于该算法设计了计及能量交易与转换的多MG能量管理与优化策略。仿真结果表明，所提能量管理策略在保护数据隐私的前提下，能够得到更高奖励且最大化MG经济收益，同时降低了通信时延。

Abstract

Current research on microgrid (MG) energy management based on federated deep reinforcement learning has not considered the problem of multiple types of energy conversion and power trading among MGs

while frequent interaction of model parameters leads to large communication latency. Based on this

an MC including multiple types of energies such as wind

solar

electricity

and gas is studied. An energy management model that allows for inter-MG electricity trading and intra-MG energy conversion is constructed. A federated dueling deep Q-network (Dueling DQN) learning algorithm based on the sine cosine algorithm (SCA) is proposed

and a multi-microgrid energy management and optimization strategy considering energy conversion and trading is designed based on the proposed algorithm. The simulation results show that the proposed energy management strategy can achieve higher rewards to maximize microgrid economic benefits while protecting data privacy and reducing communication latency.

关键词

Keywords

references

IZADIAN A ， GIRRENS N ， KHAYYER P . Renewable energy policies： a brief review of the latest U.S. and E.U. policies ［J］. IEEE Industrial Electronics Magazine ， 2013 ， 7 （ 3 ）： 21 - 34 .

WANG Z Y ， CHEN B K ， WANG J H ， et al . Coordinated energy management of networked microgrids in distribution systems ［J］. IEEE Transactions on Smart Grid ， 2015 ， 6 （ 1 ）： 45 - 53 .

ALI AREFIFAR S ， ORDONEZ M ， MOHAMED Y . Energy management in multi-microgrid systems—development and assessment ［C］// 2017 IEEE Power & Energy Society General Meeting ， July 16-20 ， 2017 ， Chicago， USA ： 910 - 922 .

NG E J ， EL-SHATSHAT R A . Multi-microgrid control systems （MMCS）［C］// IEEE PES General Meeting ， July 25-29 ， 2010 ， Minneapolis， USA ： 1 - 6 .

JIANG W ， YANG K X ， YANG J J ， et al . A multiagent-based hierarchical energy management strategy for maximization of renewable energy consumption in interconnected multi-microgrids ［J］. IEEE Access ， 2019 ， 7 ： 169931 - 169945 .

SAMADI E ， BADRI A ， EBRAHIMPOUR R . Decentralized multi-agent based energy management of microgrid using reinforcement learning ［J］. International Journal of Electrical Power & Energy Systems ， 2020 ， 122 ： 106211 .

XU Y ， YU L ， BI G ， et al . Deep reinforcement learning and blockchain for peer-to-peer energy trading among microgrids ［C］// 2020 International Conferences on Internet of Things （iThings） and IEEE Green Computing and Communications （GreenCom） and IEEE Cyber， Physical and Social Computing （CPSCom） and IEEE Smart Data （SmartData） and IEEE Congress on Cybermatics （Cybermatics）， November 2-6 ， 2020 ， Rhodes， Greece ： 360 - 365 .

NIE H H ， CHEN Y ， XIA Y ， et al . Optimizing the post-disaster control of islanded microgrid： a multi-agent deep reinforcement learning approach ［J］. IEEE Access ， 2020 ， 8 ： 153455 - 153469 .

LIU W ， WEN Z ， SHEN Y P ， et al . Reinforcement learning-based distributed secondary optimal control for multi-microgrids ［C］// 2017 IEEE Conference on Energy Internet and Energy System Integration （EI2）， November 26 - 28 ， Beijing， China ： 1 - 4 .

FANG X H ， ZHAO Q ， WANG J K ， et al . Multi-agent deep reinforcement learning for distributed energy management and strategy optimization of microgrid market ［J］. Sustainable Cities and Society ， 2021 ， 74 ： 103163 .

REZAZADEH F ， BARTZOUDIS N . A federated DRL approach for smart micro-grid energy control with distributed energy resources ［C］// 2022 IEEE 27th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks （CAMAD）， November 2-3 ， 2022 ， Paris， France ： 108 - 114 .

MUKHERJEE S ， HOSSAIN R R ， LIU Y ， et al . Enhancing cyber resilience of networked microgrids using vertical federated reinforcement learning ［C］// 2023 IEEE Power & Energy Society General Meeting （PESGM）， July 16-20 ， 2023 ， Orlando， USA ： 1 - 5 .

BOUACHIR O ， ALOQAILY M ， OZKASAP O ， et al . Federated grids： federated learning and blockchain-assisted P2P energy sharing ［J］. IEEE Transactions on Green Communications and Networking ， 2022 ， 6 （ 1 ）： 424 - 436 .

李彦君，裴玮，肖浩，等 . 基于深度学习的微网需求响应特性封装与配电网优化运行［J］. 电力系统自动化， 2021 ， 45 （ 10 ）： 157 - 165 .

LI Yanjun ， PEI Wei ， XIAO Hao ， et al . Deep learning based characteristic packaging of demand response for microgrids and optimal operation of distribution network ［J］. Automation of Electric Power Systems ， 2021 ， 45 （ 10 ）： 157 - 165 .

张帅，裴玮，马腾飞，等 . 考虑马尔可夫决策的产消者P2P电能交易非合作博弈模型［J］. 电力系统自动化， 2023 ， 47 （ 13 ）： 18 - 27 .

ZHANG Shuai ， PEI Wei ， MA Tengfei ， et al . Non-cooperative game model of prosumer P2P electricity trading considering Markov decision ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 13 ）： 18 - 27 .

周毅斌，肖浩，裴玮，等 . 基于纵向联邦学习的微电网群协同优化运行与策略进化［J］. 电力系统自动化， 2023 ， 47 （ 11 ）： 121 - 132 .

ZHOU Yibin ， XIAO Hao ， PEI Wei ， et al . Collaborative optimization operation and strategy evolution of microgrid cluster based on vertical federated learning ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 11 ）： 121 - 132 .

BARBOUR E ， PARRA D ， AWWAD Z ， et al . Community energy storage： a smart choice for the smart grid？［J］. Applied Energy ， 2018 ， 212 ： 489 - 497 .

KEINER D ， THOMA C ， BOGDANOV D ， et al . Seasonal hydrogen storage for residential on- and off-grid solar photovoltaics prosumer applications： revolutionary solution or niche market for the energy transition until 2050 ［J］. Applied Energy ， 2023 ， 340 ： 121009 .

NAKABI T A ， TOIVANEN P . Deep reinforcement learning for energy management in a microgrid with flexible demand ［J］. Sustainable Energy， Grids and Networks ， 2021 ， 25 ： 100413 .

TASDIGHI M ， GHASEMI H ， RAHIMI-KIAN A . Residential microgrid scheduling based on smart meters data and temperature dependent thermal load modeling ［J］// IEEE Transactions on Smart Grid ， 2014 ， 5 （ 1 ）： 349 - 357 .

NAKABI T A ， TOIVANEN P . Optimal price-based control of heterogeneous thermostatically controlled loads under uncertainty using LSTM networks and genetic algorithms ［J］. F1000Research ， 2019 ， 8 ： 1619 .

ZHANG C ， XU Y ， DONG Z Y ， et al . Robust coordination of distributed generation and price-based demand response in microgrids ［J］. IEEE Transactions on Smart Grid ， 2018 ， 9 （ 5 ）： 4236 - 4247 .

JIAN L N ， XUE H H ， XU G Q ， et al . Regulated charging of plug-in hybrid electric vehicles for minimizing load variance in household smart microgrid ［J］. IEEE Transactions on Industrial Electronics ， 2013 ， 60 （ 8 ）： 3218 - 3226 .

CHEN T Y ， BU S R ， LIU X ， et al . Peer-to-peer energy trading and energy conversion in interconnected multi-energy microgrids using multi-agent deep reinforcement learning ［J］. IEEE Transactions on Smart Grid ， 2022 ， 13 （ 1 ）： 715 - 727 .

DEMSETZ H . The cost of transacting ［J］. The Quarterly Journal of Economics ， 1968 ， 82 （ 1 ）： 33 .

TUSHAR W ， SAHA T K ， YUEN C ， et al . Peer-to-peer trading in electricity networks： an overview ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 4 ）： 3185 - 3200 .

LU X Z ， XIAO X Y ， XIAO L ， et al . Reinforcement learning-based microgrid energy trading with a reduced power plant schedule ［J］. IEEE Internet of Things Journal ， 2019 ， 6 （ 6 ）： 10728 - 10737 .

黎海涛，申保晨，杨艳红，等 . 基于改进竞争深度Q网络算法的微电网能量管理与优化策略［J］. 电力系统自动化， 2022 ， 46 （ 7 ）： 42 - 49 .

LI Haitao ， SHEN Baochen ， YANG Yanhong ， et al . Energy management and optimization strategy for microgrid based on improved dueling deep Q network algorithm ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 7 ）： 42 - 49 .

WANG Z Y ， SCHAUL T ， HESSEL M ， et al . Dueling network architectures for deep reinforcement learning ［C］// 33rd International Conference on International Conference on Machine Learning ， June 19-24 ， 2016 ， New York， USA ： 1995 - 2003 .

罗长银，王君宇，陈学斌，等 . 改进的联邦加权平均算法［J］. 计算机应用， 2022 ， 42 （ 4 ）： 1131 - 1136 .

LUO Changyin ， WANG Junyu ， CHEN Xuebin ， et al . Improved federated weighted average algorithm ［J］. Journal of Computer Applications ， 2022 ， 42 （ 4 ）： 1131 - 1136 .

MIRJALILI S . SCA： a sine cosine algorithm for solving optimization problems ［J］. Knowledge-Based Systems ， 2016 ， 96 ： 120 - 133 .

LI Y Z ， HE S Y ， LI Y ， et al . Federated multiagent deep reinforcement learning approach via physics-informed reward for multimicrogrid energy management ［J/OL］. IEEE Transactions on Neural Networks and Learning Systems ［ 2023-05-01 ］. https：//ieeexplore.ieee.org/document/10005155 https://ieeexplore.ieee.org/document/10005155 .

Fingrid . Fingrid open datasets ［DB/OL］. ［ 2023-05-01 ］. https：//data.fingrid.fi/en/dataset https://data.fingrid.fi/en/dataset .

Pecan Street Incorporated . Dataport ［DB/OL］. ［ 2023-05-01 ］ https：//www.pecanstreet.org/dataport/ https://www.pecanstreet.org/dataport/ .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Improved Energy Management Strategy for Wind-Photovoltaic-Hydrogen-Storage Coupling System Based on Dynamic Optimization Cycles

Energy Management Method for Port Areas Considering Hydrogen-Electricity Load Characteristics

Machine Learning Based Energy Management and Operation Control for Active Distribution Networks

Energy Management Strategy for Heating, Ventilation, and Air Conditioning Systems in Buildings Based on Multi-layer Rectified Linear Unit Network

Related Author

WANG Chen

TANG Yi

ZHENG Chenyi

SHI Changli

HE Yichen

HE Junqiang

ZHOU Jing

ZHU Meisheng

Related Institution

School of Electrical Engineering, Southeast University

College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University

University of Chinese Academy of Sciences

School of Electrical Engineering, Yanshan University

School of Electronic Information Engineering, Taiyuan University of Science and Technology

⁰