Reinforcement Learning Based Dynamic Aggregation and Multi-timescale Bidding Strategy for Virtual Power Plants

GAO Yuanhao; ZHANG Jihang; ZHAO Yincheng; GUO Mingxing; WANG Su; JIANG Chuanwen; WANG Lingling

doi:10.7500/AEPS20240430012

您当前的位置：

首页 >

文章列表页 >

Reinforcement Learning Based Dynamic Aggregation and Multi-timescale Bidding Strategy for Virtual Power Plants

Basic Research | 更新时间：2026-02-28

- Reinforcement Learning Based Dynamic Aggregation and Multi-timescale Bidding Strategy for Virtual Power Plants
- Automation of Electric Power Systems Vol. 49, Issue 23, Pages: 108-118(2025)
- 作者机构：
  
  1.电力传输与功率变换控制教育部重点实验室（上海交通大学），上海市 200240
  2.国网天津市电力公司经济技术研究院，天津市 300171
  3.国网上海市电力公司经济技术研究院，上海市 200233
- 作者简介：
- 基金信息：
  
  State Grid Corporation of China(5400-202317213A-1-1-ZN)
- DOI：10.7500/AEPS20240430012
  CLC：
- Received：30 April 2024，
  
  Revised：2024-11-23，
  
  Online First：05 February 2025，
  
  Published：10 December 2025
- 稿件说明：
移动端阅览
高元昊,张继行,赵胤呈等.基于强化学习的虚拟电厂动态聚合及多时间尺度竞价策略[J].电力系统自动化,2025,49(23):108-118.

GAO Yuanhao,ZHANG Jihang,ZHAO Yincheng,et al.Reinforcement Learning Based Dynamic Aggregation and Multi-timescale Bidding Strategy for Virtual Power Plants[J].Automation of Electric Power Systems,2025,49(23):108-118.
高元昊,张继行,赵胤呈等.基于强化学习的虚拟电厂动态聚合及多时间尺度竞价策略[J].电力系统自动化,2025,49(23):108-118. DOI： 10.7500/AEPS20240430012.

GAO Yuanhao,ZHANG Jihang,ZHAO Yincheng,et al.Reinforcement Learning Based Dynamic Aggregation and Multi-timescale Bidding Strategy for Virtual Power Plants[J].Automation of Electric Power Systems,2025,49(23):108-118. DOI： 10.7500/AEPS20240430012.

摘要

随着新型电力系统建设进程不断加快，虚拟电厂成为聚合灵活性资源参与市场交易的重要手段。为提高虚拟电厂的市场竞争力，文中提出一种虚拟电厂动态聚合及多时间尺度竞价策略。首先，考虑灵活性资源的可调能力，构建了虚拟电厂“云-边-端”分层分区调控架构；其次，在此基础上，考虑虚拟电厂内部通信成本及市场契合度得到分层分区聚合策略，并进行外特性辨识；然后，构建了虚拟电厂多时间尺度竞价模型，并采用约束感知深度强化学习方法进行模型求解，进而得到竞价策略；最后，通过算例分析验证了文中所提动态聚合方法和竞价模型能够降低虚拟电厂成本和运行风险，提升整体收益。

Abstract

With the acceleration of the construction of new power systems

virtual power plants have become an important means to aggregate flexible resources for participating in market transactions. In order to improve the market competitiveness of virtual power plants

a dynamic aggregation and multi-timescale bidding strategy for virtual power plants is proposed. Firstly

considering the adjustable capability of flexibility resources

a “cloud-edge-end” hierarchical and partitioned regulation architecture of virtual power plants is constructed. Secondly

on this basis

considering the internal communication cost of virtual power plants and the degree of market fit

a hierarchical and partitioned aggregation strategy is obtained and the external characteristic identification is carried out. Then

a multi-timescale bidding model of virtual power plants is constructed and solved by using a constraint-aware deep reinforcement learning method to obtain the bidding strategy. Finally

the proposed dynamic aggregation method and bidding model are verified through case study to reduce the cost and operation risk of virtual power plants and improve the overall revenue.

关键词

Keywords

references

ZHAO Y C ， ZHANG G Z ， HU W H ， et al . Meta-learning based voltage control for renewable energy integrated active distribution network against topology change ［J］. IEEE Transactions on Power Systems ， 2023 ， 38 （ 6 ）： 5937 - 5940 .

ZHAO Y C ， ZHANG G Z ， HU W H ， et al . Meta-learning based voltage control strategy for emergency faults of active distribution networks ［J］. Applied Energy ， 2023 ， 349 ： 121399 .

NIKONOWICZ Ł ， MILEWSKI J . Virtual power plants-general review： structure， application and optimization ［J］. Journal of Power of Technologies ， 2012 ， 92 ： 135 - 149 .

ZHAO H T ， WANG B ， PAN Z G ， et al . Aggregating additional flexibility from quick-start devices for multi-energy virtual power plants ［J］. IEEE Transactions on Sustainable Energy ， 2021 ， 12 （ 1 ）： 646 - 658 .

蒋正威，张锋明，胡凤桐，等 . 虚拟电厂分布式资源的聚合响应能力评估方法［J］. 电力工程技术， 2022 ， 41 （ 6 ）： 39 - 49 .

JIANG Zhengwei ， ZHANG Fengming ， HU Fengtong ， et al . Evaluation method of aggregated responsiveness of distributed resources in virtual power plant ［J］. Electric Power Engineering Technology ， 2022 ， 41 （ 6 ）： 39 - 49 .

白雪岩，樊艳芳，王天生，等 . 计及可再生能源可靠性的虚拟电厂动态聚合方法［J］. 电力自动化设备， 2022 ， 42 （ 7 ）： 102 - 110 .

BAI Xueyan ， FAN Yanfang ， WANG Tiansheng ， et al . Dynamic aggregation method of virtual power plants considering reliability of renewable energy ［J］. Electric Power Automation Equipment ， 2022 ， 42 （ 7 ）： 102 - 110 .

云秋晨 . 虚拟电厂多类型资源优化组合及运行策略研究［D］. 北京：北京交通大学， 2022 .

YUN Qiuchen . Research on optimal combination and operation strategy of multi-type resources of virtual power plant ［D］. Beijing ： Beijing Jiaotong University ， 2022 .

CHEN X ， LI N . Leveraging two-stage adaptive robust optimization for power flexibility aggregation ［J］. IEEE Transactions on Smart Grid ， 2021 ， 12 （ 5 ）： 3954 - 3965 .

张卫国，徐青山，陈堃，等 . 考虑电价敏感用户行为的电动汽车虚拟电厂外特性［J］. 高电压技术， 2023 ， 49 （ 4 ）： 1372 - 1379 .

ZHANG Weiguo ， XU Qingshan ， CHEN Kun ， et al . External characteristics of electric vehicle virtual power plant considering electricity price-sensitive user behavior ［J］. High Voltage Engineering ， 2023 ， 49 （ 4 ）： 1372 - 1379 .

潘明杰，解大，王西田 . 计及分布式资源调节特性差异的虚拟电厂响应策略［J］. 电力系统自动化， 2022 ， 46 （ 18 ）： 108 - 117 .

PAN Mingjie ， XIE Da ， WANG Xitian . Response strategy for virtual power plant considering difference in regulation characteristics of distributed energy resources ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 18 ）： 108 - 117 .

DING Z T ， LI Y P ， ZHANG K F ， et al . Two-stage dynamic aggregation involving flexible resource composition and coordination based on submodular optimization ［J］. Applied Energy ， 2024 ， 360 ： 122829 .

LIU X ， LI Y ， WANG L ， et al . Dynamic aggregation strategy for a virtual power plant to improve flexible regulation ability ［J］. Energy ， 2024 ， 297 ： 131261 .

严强 . 日前-实时市场联合下的虚拟电厂优化调度研究［D］. 南京：东南大学， 2022 .

YAN Qiang . Research on optimal scheduling of virtual power plants in the combination of day-ahead and real-time markets ［D］. Nanjing ： Southeast University ， 2022 .

ZHANG J ， QIN D ， YE Y C ， et al . Multi-time scale economic scheduling method based on day-ahead robust optimization and intraday MPC rolling optimization for microgrid ［J］. IEEE Access ， 2021 ， 9 ： 140315 - 140324 .

矫英鹤 . 分布式电采暖负荷群的多时间尺度电力市场交易机制［D］. 吉林：东北电力大学， 2023 .

JIAO Yinghe . Multi-time scale electricity market trading mechanism of distributed electric heating load group ［D］. Jilin ： Northeast Dianli University ， 2023 .

FOROUGHI M ， PASBAN A ， MOEINI-AGHTAIE M ， et al . A bi-level model for optimal bidding of a multi-carrier technical virtual power plant in energy markets ［J］. International Journal of Electrical Power & Energy Systems ， 2021 ， 125 ： 106397 .

孙辉，范轩轩，胡姝博，等 . 虚拟电厂参与日前电力市场的内外协调竞标策略［J］. 电网技术， 2022 ， 46 （ 4 ）： 1248 - 1262 .

SUN Hui ， FAN Xuanxuan ， HU Shubo ， et al . Internal and external coordination bidding strategy of virtual power plant participating in day-ahead power market ［J］. Power System Technology ， 2022 ， 46 （ 4 ）： 1248 - 1262 .

刘方，徐耀杰，杨秀，等 . 考虑电能交互共享的虚拟电厂集群多时间尺度协调运行策略［J］. 电网技术， 2022 ， 46 （ 2 ）： 642 - 656 .

LIU Fang ， XU Yaojie ， YANG Xiu ， et al . Multi-time scale coordinated operation strategy of virtual power plant clusters considering power interactive sharing ［J］. Power System Technology ， 2022 ， 46 （ 2 ）： 642 - 656 .

GAO X ， CHAN K W ， XIA S W ， et al . A multiagent competitive bidding strategy in a pool-based electricity market with price-maker participants of WPPs and EV aggregators ［J］. IEEE Transactions on Industrial Informatics ， 2021 ， 17 （ 11 ）： 7256 - 7268 .

ZHANG G Z ， HU W H ， CAO D ， et al . Deep reinforcement learning-based approach for proportional resonance power system stabilizer to prevent ultra-low-frequency oscillations ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 6 ）： 5260 - 5272 .

CAO D ， HU W H ， ZHAO J B ， et al . Reinforcement learning and its applications in modern power and energy systems： a review ［J］. Journal of Modern Power Systems and Clean Energy ， 2020 ， 8 （ 6 ）： 1029 - 1042 .

YE Y J ， QIU D W ， SUN M Y ， et al . Deep reinforcement learning for strategic bidding in electricity markets ［J］. IEEE Transactions on Smart Grid ， 2020 ， 11 （ 2 ）： 1343 - 1355 .

XU H C ， SUN H B ， NIKOVSKI D ， et al . Deep reinforcement learning for joint bidding and pricing of load serving entity ［J］. IEEE Transactions on Smart Grid ， 2019 ， 10 （ 6 ）： 6366 - 6375 .

QIU D W ， WANG J H ， WANG J K ， et al . Multi-agent reinforcement learning for automated peer-to-peer energy trading in double-side auction market ［C］// Proceedings of the 30th International Joint Conference on Artificial Intelligence ， August 19-27 ， 2021 ， Montreal， Canada ： 2913 - 2920 .

LIU D N ， GAO Y ， WANG W Y ， et al . Research on bidding strategy of thermal power companies in electricity market based on multi-agent deep deterministic policy gradient ［J］. IEEE Access ， 2021 ， 9 ： 81750 - 81764 .

曲翀，王秀丽，谢绍宇，等 . 不同风速模型和可靠性指标对风电可信容量评估的影响［J］. 电网技术， 2013 ， 37 （ 10 ）： 2896 - 2903 .

QU Chong ， WANG Xiuli ， XIE Shaoyu ， et al . Impacts of different wind speed models and reliability indices on capacity credit evaluation of wind power ［J］. Power System Technology ， 2013 ， 37 （ 10 ）： 2896 - 2903 .

王晶晶，姚良忠，刘科研，等 . 面向区域自治的配电网动态区域划分方法［J/OL］. 电网技术： 1 - 12 ［ 2023-12-14 ］. https：//doi.org/10.13335/j.1000-3673.pst.2023.1565 https://doi.org/10.13335/j.1000-3673.pst.2023.1565 .

WANG Jingjing ， YAO Liangzhong ， LIU Keyan ， et al . Dynamic network partitioning method of distribution networks considering regional autonomy ［J/OL］. Power System Technology ： 1 - 12 ［ 2023-12-14 ］. https：//doi.org/10.13335/j.1000-3673.pst.2023.1565 https://doi.org/10.13335/j.1000-3673.pst.2023.1565 .

张继行，张一，王旭，等 . 基于多代理强化学习的多新型市场主体虚拟电厂博弈竞价及效益分配策略［J/OL］. 电网技术： 1 - 12 ［ 2024-03-27 ］. https：//doi.org/10.13335/j.1000-3673.pst.2023.1097 https://doi.org/10.13335/j.1000-3673.pst.2023.1097 .

ZHANG Jihang ， ZHANG Yi ， WANG Xu ， et al . Game bidding and benefit allocation strategy for virtual power plants with multiple new market entities based on multi-agent reinforcement learning ［J/OL］. Power System Technology ： 1 - 12 ［ 2024-03-27 ］. https：//doi.org/10.13335/j.1000-3673.pst.2023.1097 https://doi.org/10.13335/j.1000-3673.pst.2023.1097 .

HOU S R ， VERGARA P P ， SALAZAR DUQUE E M ， et al . Optimal energy system scheduling using a constraint-aware reinforcement learning algorithm ［J］. International Journal of Electrical Power & Energy Systems ， 2023 ， 152 ： 109230 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Scientific Problems and Research Framework of Virtual Power Plant with Enormous Flexible Distributed Energy Resources in New Power System

Frequency Regulation Package Design for Virtual Power Plants Based on Graph Neural Network Prediction and Stackelberg Game

Incentive Mechanism and Key Technologies of Ancillary Service for Virtual Power Plant Based on Customized Packages

Related Author

Chongqing KANG

Qixin CHEN

Jian SU

Qian AI

Yu JI

Weiwei PAN

Hongchao GAO

Ming WU

Related Institution

State Key Laboratory of Power System Operation and Control (Tsinghua University)

China Electric Power Research Institute

Key Laboratory of Control of Power Transmission and Conversion Control, Ministry of Education (Shanghai Jiao Tong University)

State Grid Shanghai Energy Internet Research Institute Co., Ltd.

State Grid Zhejiang Electric Power Co., Ltd.

⁰