Collaborative Optimization Strategy of Distributed Generators Based on Federated Reinforcement Learning for Privacy Preservation

PU Tianjiao; DU Shuai; LI Ye; WANG Xinying

doi:10.7500/AEPS20220330008

您当前的位置：

首页 >

文章列表页 >

Collaborative Optimization Strategy of Distributed Generators Based on Federated Reinforcement Learning for Privacy Preservation

Cyber Security and Privacy Preservation in Perspective of Cyber-Physical-Society Systems | 更新时间：2025-06-16

- Collaborative Optimization Strategy of Distributed Generators Based on Federated Reinforcement Learning for Privacy Preservation
- Automation of Electric Power Systems Vol. 47, Issue 8, Pages: 62-70(2023)
- 作者机构：
  
  中国电力科学研究院有限公司，北京市 100192
- 作者简介：
- 基金信息：
  
  National Key R&D Program of China(2020YFB0905900);National Natural Science Foundation of China(U2066213)
- DOI：10.7500/AEPS20220330008
  CLC：
- Received：30 March 2022，
  
  Revised：2022-07-01，
  
  Published Online：22 November 2022，
  
  Published：25 April 2023
- 稿件说明：
移动端阅览
蒲天骄,杜帅,李烨等.面向隐私保护基于联邦强化学习的分布式电源协同优化策略[J].电力系统自动化,2023,47(8):62-70.

PU Tianjiao,DU Shuai,LI Ye,et al.Collaborative Optimization Strategy of Distributed Generators Based on Federated Reinforcement Learning for Privacy Preservation[J].Automation of Electric Power Systems,2023,47(8):62-70.
蒲天骄,杜帅,李烨等.面向隐私保护基于联邦强化学习的分布式电源协同优化策略[J].电力系统自动化,2023,47(8):62-70. DOI： 10.7500/AEPS20220330008.

PU Tianjiao,DU Shuai,LI Ye,et al.Collaborative Optimization Strategy of Distributed Generators Based on Federated Reinforcement Learning for Privacy Preservation[J].Automation of Electric Power Systems,2023,47(8):62-70. DOI： 10.7500/AEPS20220330008.

摘要

针对分布式电源优化调度面临的隐私保护和实时决策问题，提出了基于联邦强化学习的多智能体分布式协同优化策略。首先，构建了基于联邦强化学习的配电网分布式协同优化框架，利用联邦学习避免在多智能体深度强化学习过程中泄露隐私数据。在此框架下，提出了多智能体约束策略优化方法，利用离线训练缩短在线决策时间，支持智能体实时分布式决策。同时，该方法为智能体构建了考虑潮流方程等约束条件的可行域，允许智能体在训练过程中自由探索，提高了收敛速度，并确保实时调度策略满足电力系统安全运行约束。最后，通过算例进行仿真验证，结果表明离线训练时各智能体仅利用局部信息即可实现全局优化，并保证了实时决策和调度策略的安全性。

Abstract

Aiming at the privacy preservation and real-time decision-making problems of the optimal dispatch for distributed generators

a multi-agent distributed collaborative optimization strategy based on federated reinforcement learning is proposed. First

a distributed collaborative optimization framework for the distribution network based on federated reinforcement learning is constructed

which uses federated learning to avoid leaking private data in the process of multi-agent deep reinforcement learning. Under this framework

a multi-agent constrained policy optimization method is proposed

which uses off-line training to shorten the online decision-making time for supporting the real-time distributed decision-making of agents. At the same time

the proposed method constructs a feasible region for agents considering the constraints such as power flow equations

allowing them to explore freely during the training process

which improves the convergence speed and ensures that the real-time dispatch strategy can meet the power system security operation constraints. Finally

the simulation results show that each agent can achieve the global optimization only by using local information during off-line training

and the proposed method ensures the security of real-time decision-making and dispatch strategy.

关键词

Keywords

references

王洪坤，王守相，潘志新，等 . 含高渗透分布式电源配电网灵活性提升优化调度方法［J］. 电力系统自动化， 2018 ， 42 （ 15 ）： 86 - 93 .

WANG Hongkun ， WANG Shouxiang ， PAN Zhixin ， et al . Optimized dispatching method for flexibility improvement of distribution network with high-penetration distributed generation ［J］. Automation of Electric Power Systems ， 2018 ， 42 （ 15 ）： 86 - 93 .

蒲天骄，陈盛，赵琦，等 . 能源互联网数字孪生系统框架设计及应用展望［J］. 中国电机工程学报， 2021 ， 41 （ 6 ）： 2012 - 2029 .

PU Tianjiao ， CHEN Sheng ， ZHAO Qi ， et al . Framework design and application prospect for digital twins system of energy Internet ［J］. Proceedings of the CSEE ， 2021 ， 41 （ 6 ）： 2012 - 2029 .

CHENG Z Y ， CHOW M Y . Collaborative distributed AC optimal power flow： a dual decomposition based algorithm ［J］. Journal of Modern Power Systems and Clean Energy ， 2021 ， 9 （ 6 ）： 1414 - 1423 .

PARISIO A ， WIEZOREK C ， KYNTÄJÄ T ， et al . Cooperative MPC-based energy management for networked microgrids ［J］. IEEE Transactions on Smart Grid ， 2017 ， 8 （ 6 ）： 3066 - 3074 .

ZHAO T Q ， DING Z T . Distributed agent consensus-based optimal resource management for microgrids ［J］. IEEE Transactions on Sustainable Energy ， 2018 ， 9 （ 1 ）： 443 - 452 .

蒲天骄，陈乃仕，王晓辉，等 . 主动配电网多源协同优化调度架构分析及应用设计［J］. 电力系统自动化， 2016 ， 40 （ 1 ）： 17 - 23 .

PU Tianjiao ， CHEN Naishi ， WANG Xiaohui ， et al . Application and architecture of multi-source coordinated optimal dispatch for active distribution network ［J］. Automation of Electric Power Systems ， 2016 ， 40 （ 1 ）： 17 - 23 .

彭刘阳，孙元章，徐箭，等 . 基于深度强化学习的自适应不确定性经济调度［J］. 电力系统自动化， 2020 ， 44 （ 9 ）： 33 - 42 .

PENG Liuyang ， SUN Yuanzhang ， XU Jian ， et al . Self-adaptive uncertainty economic dispatch based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2020 ， 44 （ 9 ）： 33 - 42 .

LEI L ， TAN Y ， DAHLENBURG G ， et al . Dynamic energy dispatch based on deep reinforcement learning in IoT-driven smart isolated microgrids ［J］. IEEE Internet of Things Journal ， 2021 ， 8 （ 10 ）： 7938 - 7953 .

HUA H C ， QIN Y C ， HAO C T ， et al . Optimal energy management strategies for energy Internet via deep reinforcement learning approach ［J］. Applied Energy ， 2019 ， 239 ： 598 - 609 .

于一潇，杨佳峻，杨明，等 . 基于深度强化学习的风电场储能系统预测决策一体化调度［J］. 电力系统自动化， 2021 ， 45 （ 1 ）： 132 - 140 .

YU Yixiao ， YANG Jiajun ， YANG Ming ， et al . Prediction and decision integrated scheduling of energy storage system in wind farm based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2021 ， 45 （ 1 ）： 132 - 140 .

KAIROUZ P ， MCMAHAN H ， AVENT B ， et al . Advances and open problems in federated learning ［J］. Foundations and Trends® in Machine Learning ， 2021 ， 14 （ 1 ）： 1 - 210 .

QIANG Yang ， YANG Liu ， YONG Cheng ， et al . Federated learning ［M］. Kentfield， USA ： Morgan & Claypool Publishers ， 2019 ．

ALTMAN E . Constrained Markov decision processes ［M］. Boca Raton， USA ： Chapman and Hall/CRC ， 1999 .

PHONG L T ， AONO Y ， HAYASHI T ， et al . Privacy-preserving deep learning via additively homomorphic encryption ［J］. IEEE Transactions on Information Forensics and Security ， 2018 ， 13 （ 5 ）： 1333 - 1345 .

SATTLER F ， WIEDEMANN S ， MÜLLER K R ， et al . Robust and communication-efficient federated learning from non-i.i.d. data ［J］. IEEE Transactions on Neural Networks and Learning Systems ， 2020 ， 31 （ 9 ）： 3400 - 3413 .

吕朋蓬，赵晋泉，苏大威，等 . 基于一致性理论的独立微电网分布式动态经济调度［J］. 电力系统自动化， 2019 ， 43 （ 5 ）： 22 - 29 .

LYU Pengpeng ， ZHAO Jinquan ， SU Dawei ， et al . Consensus-based distributed dynamic economic dispatching for islanded microgrids ［J］. Automation of Electric Power Systems ， 2019 ， 43 （ 5 ）： 22 - 29 .

BOYD S . Distributed optimization and statistical learning via the alternating direction method of multipliers ［J］. Foundations and Trends® in Machine Learning ， 2010 ， 3 （ 1 ）： 1 - 122 .

YUAN D M ， XU S Y ， ZHAO H Y . Distributed primal-dual subgradient method for multiagent optimization via consensus algorithms ［J］. IEEE Transactions on Systems， Man， and Cybernetics， Part B （Cybernetics）， 2011 ， 41 （ 6 ）： 1715 - 1724 .

LOWE R ， WU Y ， TAMAR A ， et al . Multi-agent actor-critic for mixed cooperative-competitive environments ［C］// Neural Information Processing Systems （NIPS）， December 4-9 ， 2017 ， Long Beach， USA .

ACHIAM J ， HELD D ， TAMAR A . Constrained policy optimization ［C］// 34th International Conference on Machine Learning ， August 6-11 ， 2017 ， Sydney， Australia .

SUTTON R S ， BARTO A G . Reinforcement learning： an introduction ［J］. IEEE Transactions on Neural Networks ， 1998 ， 9 （ 5 ）： 1054 .

LI H P ， WAN Z Q ， HE H B . Constrained EV charging scheduling based on safe deep reinforcement learning ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 3 ）： 2427 - 2439 .

BESSON O ， ABRAMOVICH Y I . On the Fisher information matrix for multivariate elliptically contoured distributions ［J］. IEEE Signal Processing Letters ， 2013 ， 20 （ 11 ）： 1130 - 1133 .

SCHULMAN J ， WOLSKI F ， DHARIWAL P ， et al . Proximal policy optimization algorithms ［EB/OL］. ［ 2021-10-05 ］. https：//arxiv.org/pdf/1707.06347.pdf https://arxiv.org/pdf/1707.06347.pdf .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Multi-agent Game Collaborative Optimization Strategy for Active Distribution Networks Based on Federated Reinforcement Learning

Voltage Regulation Strategy for Active Distribution Network Based on Multi-agent Deep Reinforcement Learning

Synchronous Measurement Data-driven Classification Method for Disturbance Sources in Distribution Networks

Related Author

YANG Wenwei

PENG Xiangang

QUAN Huan

CHU Zhuozhuo

WANG Xinghua

ZHAO Zhuoli

HUANG Yu

ZHANG Xiaoxiao

Related Institution

College of Automation, Guangdong University of Technology

School of Electric Power Engineering, South China University of Technology

Institute of Advanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications

NARI Technology Co., Ltd.

State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (North China Electric Power University)

⁰