Voltage Control for Active Distribution Network Based on Bayesian Deep Reinforcement Learning

ZHANG Xiao; WU Zhi; ZHENG Shu; GU Wei; HU Bo; DONG Jichao

doi:10.7500/AEPS20240416006

您当前的位置：

首页 >

文章列表页 >

Voltage Control for Active Distribution Network Based on Bayesian Deep Reinforcement Learning

Active Control and Cooperative Dispatch for Distribution Networks with High-penetration Distributed Energy Resources | 更新时间：2025-06-16

- Voltage Control for Active Distribution Network Based on Bayesian Deep Reinforcement Learning
- Automation of Electric Power Systems Vol. 48, Issue 20, Pages: 81-90(2024)
- 作者机构：
  
  1.东南大学电气工程学院,江苏省南京市 210096
  2.南瑞集团有限公司(国网电力科学研究院有限公司),江苏省南京市 211106
  3.国网辽宁省电力有限公司大连供电公司,辽宁省大连市 116001
- 作者简介：
- 基金信息：
  
  State Grid Corporation of China(5400-202328548A-3-2-ZN)
- DOI：10.7500/AEPS20240416006
  CLC：
- Received：16 April 2024，
  
  Revised：2024-05-16，
  
  Published Online：24 July 2024，
  
  Published：25 October 2024
- 稿件说明：
移动端阅览
张晓,吴志,郑舒等.基于贝叶斯深度强化学习的主动配电网电压控制[J].电力系统自动化,2024,48(20):81-90.

ZHANG Xiao,WU Zhi,ZHENG Shu,et al.Voltage Control for Active Distribution Network Based on Bayesian Deep Reinforcement Learning[J].Automation of Electric Power Systems,2024,48(20):81-90.
张晓,吴志,郑舒等.基于贝叶斯深度强化学习的主动配电网电压控制[J].电力系统自动化,2024,48(20):81-90. DOI： 10.7500/AEPS20240416006.

ZHANG Xiao,WU Zhi,ZHENG Shu,et al.Voltage Control for Active Distribution Network Based on Bayesian Deep Reinforcement Learning[J].Automation of Electric Power Systems,2024,48(20):81-90. DOI： 10.7500/AEPS20240416006.

摘要

多分布式源荷接入导致配电网电压波动性增强。同时，上级主网电压的不确定性波动也会对配电网电压特性造成影响。为有效应对主配电网电压波动，文中提出一种基于数据驱动与模型求解结合的主动配电网多时间尺度电压控制框架。在慢时间尺度下，考虑主网电压波动，构建了上级主网非无穷大系统多馈线环境，对该环境下的电压控制问题进行了对抗性马尔可夫过程建模。在训练过程中，以投影梯度下降算法使主网电压发生扰动。利用贝叶斯深度Q网络算法感知上级主网电压波动，实现有载调压变压器挡位快速控制。在快时间尺度下，基于传统二阶锥优化方法控制光伏逆变器无功输出。算例结果表明，该方法可准确感知上级主网电压波动，在极短时间实现配电网无模型电压控制，保证各节点电压维持在安全范围内。

Abstract

The access of multiple distributed sources and loads leads to enhanced voltage volatility in the distribution network. Meanwhile

the uncertainty fluctuation in the voltage of the upper main grid also affect the voltage characteristics of the distribution network. In order to effectively deal with the voltage fluctuations of the main grid and the distribution network

this paper proposes a multi-timescale voltage control framework for active distribution networks based on the combination of data-driven and model solving. In the slow time scale

considering the voltage fluctuation of the main grid

a multiple-feeder environment with a non-infinity system of the upper main grid is constructed

and the voltage control problem in this environment is modeled as an adversarial Markov process. During the training process

the voltage of the main grid is perturbed with a projected gradient descent algorithm. The Bayesian deep Q network algorithm is utilized to sense the voltage fluctuation of the upper main grid and realize the fast control of taps of the on-load tap changer. In the fast time scale

the reactive power output of the photovoltaic inverter is controlled based on the traditional second-order cone optimization method. The case results show that the method can accurately sense the voltage fluctuation of the upper main grid

realize model-free voltage control of the distribution network in a very short time

and ensure that the voltage of each node is maintained within the safety range.

关键词

Keywords

references

魏泓屹，卓振宇，张宁，等 . 中国电力系统碳达峰·碳中和转型路径优化与影响因素分析［J］. 电力系统自动化， 2022 ， 46 （ 19 ）： 1 - 12 .

WEI Hongyi ， ZHUO Zhenyu ， ZHANG Ning ， et al . Transition path optimization and influencing factor analysis of carbon emission peak and carbon neutrality for power system of China ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 19 ）： 1 - 12 .

李鹏，习伟，李鹏，等 . 基于边缘计算的配电网数字化转型关键问题分析与展望［J］. 电力系统自动化， 2024 ， 48 （ 6 ）： 29 - 41 .

LI Peng ， XI Wei ， LI Peng ， et al . Analysis of key issues and prospect for digital transformation of distribution networks based on edge computing ［J］. Automation of Electric Power Systems ， 2024 ， 48 （ 6 ）： 29 - 41 .

杨浩，苏文栋，谷毅，等 . 面向耦合系统的交替方向滚动时域电压分层协同优化控制［J］. 电工技术学报， 2023 ， 38 （ 21 ）： 5846 - 5861 .

YANG Hao ， SU Wendong ， GU Yi ， et al . Voltage hierarchical cooperative control of coupled system using model predictive control and alternating direction method of multipliers ［J］. Transactions of China Electrotechnical Society ， 2023 ， 38 （ 21 ）： 5846 - 5861 .

WU Z ， LI Y Q ， GU W ， et al . Multi-timescale voltage control for distribution system based on multi-agent deep reinforcement learning ［J］. International Journal of Electrical Power & Energy Systems ， 2023 ， 147 ： 108830 .

胡维昊，曹迪，黄琦，等 . 深度强化学习在配电网优化运行中的应用［J］. 电力系统自动化， 2023 ， 47 （ 14 ）： 174 - 191 .

HU Weihao ， CAO Di ， HUANG Qi ， et al . Application of deep reinforcement learning in optimal operation of distribution network ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 14 ）： 174 - 191 .

HOSSAIN R R ， HUANG Q H ， HUANG R K . Graph convolutional network-based topology embedded deep reinforcement learning for voltage stability control ［J］. IEEE Transactions on Power Systems ， 2021 ， 36 （ 5 ）： 4848 - 4851 .

徐博涵，向月，潘力，等 . 基于深度强化学习的含高比例可再生能源配电网就地分散式电压管控方法［J］. 电力系统保护与控制， 2022 ， 50 （ 22 ）： 100 - 109 .

XU Bohan ， XIANG Yue ， PAN Li ， et al . Local decentralized voltage management of a distribution network with a high proportion of renewable energy based on deep reinforcement learning ［J］. Power System Protection and Control ， 2022 ， 50 （ 22 ）： 100 - 109 .

PETRUSEV A ， PUTRATAMA M A ， RIGO-MARIANI R ， et al . Reinforcement learning for robust voltage control in distribution grids under uncertainties ［J］. Sustainable Energy， Grids and Networks ， 2023 ， 33 ： 100959 .

LIU H T ， WU W C . Two-stage deep reinforcement learning for inverter-based volt-var control in active distribution networks ［J］. IEEE Transactions on Smart Grid ， 2021 ， 12 （ 3 ）： 2037 - 2047 .

LIU H Y ， ZHANG C ， CHAI Q M ， et al . Robust regional coordination of inverter-based volt/var control via multi-agent deep reinforcement learning ［J］. IEEE Transactions on Smart Grid ， 2021 ， 12 （ 6 ）： 5420 - 5433 .

习伟，李鹏，李鹏，等 . 基于深度强化学习的分布式电源就地自适应电压控制方法［J］. 电力系统自动化， 2022 ， 46 （ 22 ）： 25 - 31 .

XI Wei ， LI Peng ， LI Peng ， et al . Adaptive local voltage control method for distributed generator based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 22 ）： 25 - 31 .

ZHANG T Q ， SUN M Y ， QIU D W ， et al . A Bayesian deep reinforcement learning-based resilient control for multi-energy micro-gird ［J］. IEEE Transactions on Power Systems ， 2023 ， 38 （ 6 ）： 5057 - 5072 .

BRINKEL N B G ， GERRITSMA M K ， ALSKAIF T A ， et al . Impact of rapid PV fluctuations on power quality in the low-voltage grid and mitigation strategies using electric vehicles ［J］. International Journal of Electrical Power & Energy Systems ， 2020 ， 118 ： 105741 .

ZHAO W L ， SHI X C ， GUO Q L ， et al . Voltage security analysis with high PVs penetration considering the interaction of transmission and distribution grids： case studies ［C］// 2015 IEEE PES Asia-Pacific Power and Energy Engineering Conference （APPEEC）， November 15-18 ， 2015 ， Brisbane， Australia .

CHEN Z H ， YU R J ， HU C G ， et al . The voltage fluctuation characteristic analysis of regional power network with high-penetration PV system ［C］// 2017 4th International Conference on Systems and Informatics （ICSAI）， November 11-13 ， 2017 ， H angzhou， China .

路镇铭，徐英，仪忠凯，等 . 基于双向安德森加速分解协调算法的输配协同无功优化［J/OL］. 电力系统自动化［ 2024-01-29 ］. https：//kns.cnki.net/kcms/detail/32.1180.TP.20240126.2057.003.html https://kns.cnki.net/kcms/detail/32.1180.TP.20240126.2057.003.html .

LU Zhenming ， XU Ying ， YI Zhongkai ， et al . Transmission and distribution coordinated reactive power optimization based on bidirectional Anderson acceleration decomposition coordination algorithm ［J/OL］. Automation of Electric Power Systems ［ 2024-01-29 ］. https：//kns.cnki.net/kcms/detail/32.1180.TP.20240126.2057.003.html https://kns.cnki.net/kcms/detail/32.1180.TP.20240126.2057.003.html .

刘晨曦，辛焕海 . 考虑实际工况的新能源多馈入系统广义运行短路比［J/OL］. 电力系统自动化［ 2024-04-18 ］. https：//kns.cnki.net/kcms/detail/32.1180.tp.20240415.2009.010.html https://kns.cnki.net/kcms/detail/32.1180.tp.20240415.2009.010.html .

LIU Chenxi ， XIN Huanhai . Generalized operational short-circuit ratio of multi-infeed systems with renewable energy considering actual operation conditions ［J/OL］. Automation of Electric Power Systems ［ 2024-04-18 ］. https：//kns.cnki.net/kcms/detail/32.1180.tp.20240415.2009.010.html https://kns.cnki.net/kcms/detail/32.1180.tp.20240415.2009.010.html .

华文，董炜，阙凌燕，等 . 计及直流互联与电-气-热耦合的输配协同优化调度［J］. 浙江电力， 2022 ， 41 （ 11 ）： 31 - 38 .

HUA Wen ， DONG Wei ， QUE Lingyan ， et al . Collaborative optimal scheduling for coupled transmission and distribution systems considering HVDC interconnection and electricity-gas-heat coupling ［J］. Zhejiang Electric Power ， 2022 ， 41 （ 11 ）： 31 - 38 .

李胜男，高源，何廷一，等 . 新能源临界短路比对电网运行参数灵敏度分析方法［J］. 智慧电力， 2023 ， 51 （ 5 ）： 96 - 102 .

LI Shengnan ， GAO Yuan ， HE Tingyi ， et al . Sensitivity analysis method for critical short circuit ratio of new energy to power grid operating parameters ［J］. Smart Power ， 2023 ， 51 （ 5 ）： 96 - 102 .

王奕鑫，唐宇，叶琳，等 . 多馈入直流系统电压稳定中静态负荷影响的定量分析和评估方法［J/OL］. 电工技术学报［ 2024-05-15 ］. https：//doi.org/10.19595/j.cnki.1000-6753.tces.231870 https://doi.org/10.19595/j.cnki.1000-6753.tces.231870 .

WANG Yixin ， TANG Yu ， YE Lin ， et al . Quantitative analysis and evaluation method for the impact of static loads on voltage stability in multi-infeed direct current systems ［J/OL］. Transactions of China Electrotechnical Society ［ 2024-05-15 ］. https：//doi.org/10.19595/j.cnki.1000-6753.tces.231870 https://doi.org/10.19595/j.cnki.1000-6753.tces.231870 .

吴文传，张伯明，巨云涛 . 主动配电网网络分析与运行调控［M］. 北京：科学出版社， 2016 .

WU Wenchuan ， ZHANG Boming ， JU Yuntao . Active distribution networks analysis， operation and control ［M］. Beijing ： Science Press ， 2016 .

WANG R H ， BI X W ， BU S Q . Real-time coordination of dynamic network reconfiguration and volt-var control in active distribution network： a graph-aware deep reinforcement learning approach ［J］. IEEE Transactions on Smart Grid ， 2024 ， 15 （ 3 ）： 3288 - 3302 .

XU H C ， DOMINGUEZ-GARCIA A D ， SAUER P W . Optimal tap setting of voltage regulation transformers using batch reinforcement learning ［J］. IEEE Transactions on Power Systems ， 2020 ， 35 （ 3 ）： 1990 - 2001 .

WANG C . Attacking and defending deep reinforcement learning policies ［EB/OL］. ［ 2024-01-29 ］. https：//arxiv.org/abs/2205.07626 https://arxiv.org/abs/2205.07626 .

冯昌森，张瑜，文福拴，等 . 基于深度期望Q网络算法的微电网能量管理策略［J］. 电力系统自动化， 2022 ， 46 （ 3 ）： 14 - 22 .

FENG Changsen ， ZHANG Yu ， WEN Fushuan ， et al . Energy management strategy for microgrid based on deep expected Q network algorithm ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 3 ）： 14 - 22 .

GHAVAMZADEH M ， MANNOR S ， PINEAU J ， et al . Bayesian reinforcement learning： a survey ［EB/OL］. ［ 2024-04-16 ］. http：//arxiv.org/abs/1609.0443644 http://arxiv.org/abs/1609.0443644 .

SILVA B D . A Bayesian perspective on Q-learning ［EB/OL］. ［ 2024-04-16 ］. https：//brandinho.github.io/bayesian-per-spective-q-learning https://brandinho.github.io/bayesian-per-spective-q-learning .

李宏浩，张沛，刘曌 . 基于深度强化学习的暂态稳定紧急控制决策方法［J］. 电力系统自动化， 2023 ， 47 （ 5 ）： 144 - 152 .

LI Honghao ， ZHANG Pei ， LIU Zhao . Decision-making method for transient stability emergency control based on deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 5 ）： 144 - 152 .

陈珩 . 电力系统稳态分析［M］. 北京：中国电力出版社， 2007 .

CHEN Heng . Steady state analysis of power systems ［M］. Beijing ： China Electric Power Press ， 2007 .

BDQN training and testing set ［EB/OL］. ［ 2024-04-16 ］. https：//github.com/chilixiao/BDQN-training-and-testing-set https://github.com/chilixiao/BDQN-training-and-testing-set .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Voltage Regulation Strategy for Active Distribution Network Based on Multi-agent Deep Reinforcement Learning

Coordinated Optimal Scheduling of Transmission and Distribution Networks Based on Multiple Agents and Improved Target Cascading Method

Bi-layer Coordinated Control Strategy of Distribution Network Considering Participation of Electric Vehicles and Microgrid

Related Author

HUANG Yu

ZHANG Xiaoxiao

YUE Dong

HU Songlin

WANG Juan

LI Zhukun

WANG Juan

LI Zhukun

Related Institution

Institute of Advanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications

NARI Technology Co., Ltd.

NARI Group Corporation (State Grid Electric Power Research Institute)

College of Electrical Engineering and Automation, Shandong University of Science and Technology

Key Laboratory of Power System Intelligent Dispatch and Control of Ministry of Education (Shandong University)

⁰