Deep Reinforcement Learning-based Dynamic Capacity Planning for Consecutive Highway Charging Stations
-
摘要: 新能源汽车保有量激增与高速公路充电服务能力不足之间的矛盾日益突出，同时传统多目标充电站规划方法依赖静态假设、难以适应动态交通环境。为解决上述问题，提出基于深度强化学习的高速公路连续充电站动态定容模型。将多个连续的充电站容量规划问题建模为马尔可夫决策过程，其状态空间融合充电站实时运行指标及车辆动态特征，有效刻画了“交通流-充电需求-用户行为”的非线性耦合关系，并设计了兼顾用户满意度与运营成本的多目标奖励函数。在算法层面，设计D3QN-PER-2s算法，该算法结合Dueling深度Q网络（deep Q-network，DQN）分离状态价值与动作优势函数，并利用Double DQN解决Q值过估计问题，引入优先经验回放机制评估经验重要性并进行优先级采样；并进一步采用两步时序差分更新策略，融入未来步的奖励与价值信息，增强更新过程的稳定性。模型依托SUMO仿真环境开展实时交互式学习，无需依赖历史数据，为生成鲁棒性强的方案，引入经验累积分布函数分位数分析策略，在训练后对大量仿真决策进行统计分析，选取90%分位数对应值作为最终方案，有效平衡方案经济性与需求波动的鲁棒性。结果表明，所提出的基于深度强化学习的定容方案将充电排队车辆占比控制在10%以内，且平均排队时间小于3 min。相较于基于遗传算法求解的排队论多目标模型，建设成本降低24.3%，在充电高峰时段充电桩利用率提升17.4%。上述结果证明该方法在解决多目标充电桩容量规划问题上的优越性，提升了高速充电服务效率与用户充电体验。Abstract: The contradiction between the surging number of new energy vehicles and the inadequate charging services on highways becomes increasingly prominent. Compounding this issue, traditional multi-objective planning methods for charging stations rely on static assumptions and fail to adapt to dynamic traffic environments. To resolve these limitations, a dynamic capacity planning model based on deep reinforcement learning is proposed for consecutive highways charging stations. The capacity planning problem for consecutive charging stations is modeled as a Markov decision process. The state space integrates real-time operational metrics of charging stations and dynamic vehicular features to characterize the nonlinear coupling among traffic flow, charging demand, and user behavior. Furthermore, a multi-objective reward function balances user satisfaction and operational costs is designed. Algorithmically, the D3QN-PER-2s algorithm is developed. It combines the Dueling deep Q-network (DQN) architecture to separate state value from action advantage functions and employs Double DQN to mitigate Q-value overestimation problem. Additionally, a prioritized experience replay mechanism is introduced to evaluate experience importance and conduct priority sampling. To enhance learning stability, the algorithm further adopts a two-step temporal difference update strategy that incorporates future rewards and value estimates. The model interacts with the SUMO simulation environment for real-time learning, eliminating the dependence on historical data. To generate highly robust planning solutions, a quantile analysis strategy based on the empirical cumulative distribution function is introduced. Statistical analysis is conducted on extensive simulated decisions after training. The 90th percentile value is selected as the final plan, effectively balancing cost-effectiveness and robustness against demand fluctuations. Experimental results demonstrate that the proposed deep reinforcement learning-based scheme controls the proportion of queuing vehicles to within 10% reduces the average queuing time to less than 3 minutes. Compared to a multi-objective queuing model solved by genetic algorithm, this approach reduces construction costs by 24.3% and improves peak-hour charger utilization by 17.4%. These results prove the superiority of the proposed method in solving multi-objective capacity planning problem of chargers, enhancing the efficiency of highway charging services and user charging experience.
-
图 2 传统DQN神经网络与对决网络结构对比
Figure 2. Comparison between traditional DQN neural network and dueling network architectures
图 4 基于D3QN-PER-2s模型迭代奖励收敛曲线
Figure 4. Reward convergence curve of the D3QN-PER-2s model iteration
图 5 基于年平均日数据的1号服务区定容方案ECDF曲线
Figure 5. ECDF curve of capacity allocation scheme for Service Area 1 based on annual average daily data
表 1 基于D3QN-PER-2s的高速公路充电站定容模型参数
Table 1. Parameter settings of the expressway charging station capacity allocation model based on D3QN-PER-2s
参数 取值 服务区i布设充电桩的土建施工成本Di/(万元/个) 1.3 服务区i充电桩购置成本Mi/(万元/个) 5.5 服务区i配电扩容成本Pi/(万元/个) 2.2 折现率r0/% 5 充电桩运营年限z/年 8 充电桩功率P/kW 250 充电电价(元/kW·h) 1.485 5 充电桩数量超过约束的奖励调节系数kex 100 经验回放池大小b 10 000 训练批次大小N 50 每轮训练次数 50 折扣因子γ 0.82 学习率α 0.01 单层网络神经元个数 256 目标网络更新间隔d 10 
下载: 导出CSV
表 2 服务区充电需求数据表
Table 2. Service area charging demand data
服务区 基于年平均日的充电需求量/辆 基于春节的充电需求量/辆 1号 147 335 2号 221 503 3号 103 235
下载: 导出CSV
表 3 SUMO仿真关键参数设置
Table 3. Key parameter settings for SUMO simulation
关键参数 模型 车辆到达模型 泊松过程 跟驰模型 智能驾驶人模型(IDM) 车道变换模型 LC2013模型 排队规则 先到先服务
下载: 导出CSV
表 4 2种服务区充电站定容模型求解结果
Table 4. Solving results of two service area charging station capacity allocation models
模型 数据类型 求解结果(充电桩数量) Cmt/万元 URpeak/% 充电排队车辆占比/% 平均充电排队时间/s 1号服务区 2号服务区 3号服务区 基于排队论的多目标优化模型 年平均日 8 11 6 234 82.6 0 0 春节 15 20 11 430.56 83.9 0 0 基于深度强化学习模型 年平均日 6 8 4 168.48 100 7.2 146 春节 12 17 8 346.32 100 9.1 173
下载: 导出CSV
-
[1] YU Z, CHOU S Y. Research on ratio of new energy vehicles to charging piles in China[J]. Computer Systems Science and Engineering, 2022, 42(3): 963-984. doi: 10.32604/csse.2022.023129 [2] 姜涵, 张健, 张海燕, 等. 基于强化学习的交叉口智能网联车多目标通行控制方法[J]. 交通信息与安全, 2024, 42(1): 84-93. doi: 10.3963/j.jssn.1674-4861.2024.01.010JIANG H, ZHANG J, ZHANG H Y, et al. A multi-objective traffic control method for connected and automated vehicle at signalized intersection based on reinforcement learning[J]. Journal of Transport Information and Safety, 2024, 42(1): 84-93. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2024.01.010 [3] 辛琪, 荚胜琪, 徐猛, 等. 混行下CAV作业区分段式深度强化学习合流模型[J]. 交通信息与安全, 2025, 43(2): 95-108. doi: 10.3963/j.jssn.1674-4861.2025.02.011XIN Q, JIA S Q, XU M, et al. A merging model based on piecewise deep reinforcement learning for connected and autonomous vehicle in work zone under mixed autonomy[J]. Journal of Transport Information and Safety, 2025, 43(2): 95-108. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2025.02.011 [4] SONG L, LIN Y, ZHAO X, et al. Exploring the feasibility and sensitivity of deep reinforcement learning controlled traffic signals in bidirectional two-lane road work zones[J]. Expert Systems with Applications, 2025, 287: 128180. doi: 10.1016/j.eswa.2025.128180 [5] SHUVO S, AHMED M, SYMUM H, et al. Deep reinforcement learning based cost-benefit analysis for hospital capacity planning[C]. International Joint Conference on Neural Networks, Online: IEEE, 2021. [6] 王震坡, 张瑾, 刘鹏, 等. 电动汽车充电站规划研究综述[J]. 中国公路学报, 2022, 35(12): 230-252.WANG Z P, ZHANG J, LIU P, et al. Overview of planning of electric vehicle charging stations [J]. China Journal of Highway and Transport, 2022, 35(12): 230-252. (in Chinese) [7] ZHAO Z, XU M, LEE C K M. Capacity planning for an electric vehicle charging station considering fuzzy quality of service and multiple charging options[J]. IEEE Transactions on Vehicular Technology, 2021, 70(12): 12529-12541. doi: 10.1109/TVT.2021.3121440 [8] 杨亚璪, 宾涛. 电动汽车集中型充电站选址定容模型[J]. 交通运输系统工程与信息, 2024(3): 204-212.YANG Y Z, BIN T. Electric vehicle centralized charging station siting and capacity modeling [J]. Journal of Transportation Systems Engineering and Information Technology, 2024 (3): 204-212. (in Chinese) [9] 胡晓伟, 宋帅, 邱振洋, 等. 寒区电动公交充电站选址及定容规划研究[J]. 交通运输系统工程与信息, 2024(2): 281-292.HU X W, SONG S, QIU Z Y, et al. Location and capacity planning of electric bus charging station in cold regions [J]. Journal of Transportation Systems Engineering and Information Technology, 2024(2): 281-292. (in Chinese) [10] WANG C, HE F, LIN X, et al. Designing locations and capacities for charging stations to support intercity travel of electric vehicles: an expanded network approach[J]. Transportation Research Part C: Emerging Technologies, 2019, 102: 210-232. doi: 10.1016/j.trc.2019.03.013 [11] CHEN R, QIAN X, MIAO L, et al. Optimal charging facility location and capacity for electric vehicles considering route choice and charging time equilibrium[J]. Computers and Operations Research, 2020, 113: 104776. doi: 10.1016/j.cor.2019.104776 [12] KINAY Ö B, GZARA F, ALUMUR S A. Full cover charging station location problem with routing[J]. Transportation Research Part B: Methodological, 2021, 144: 1-22. [13] 张文会, 乔梓凡, 陈德启. 出租车轨迹数据驱动的充电站选址定容方法[J]. 交通运输系统工程与信息, 2025, 25(5): 291-301.ZHANG W H, QIAO Z F, CHEN D Q. Charging station location and capacity determination algorithm based on taxi trajectory data[J]. Journal of Transportation Systems Engineering and Information Technology, 2025, 25(5): 291-301. (in Chinese) [14] 孙健, 宋茂星, 邱果, 等. 基于电动汽车大数据的多等级充电站选址与服务能力研究[J]. 中国公路学报, 2024, 37 (4): 48-60.SUN J, SONG M X, QIU G, et al. Location and service capability of multilevel charging stations based on electric vehicle big data[J]. China Journal of Highway and Transport, 2024, 37(4): 48-60. (in Chinese) [15] XU M, MENG Q. Optimal deployment of charging stations considering path deviation and nonlinear elastic demand[J]. Transportation Research Part B: Methodological, 2020, 135: 120-142. doi: 10.1016/j.trb.2020.03.001 [16] KAVIANIPOUR M, FAKHRMOOSAVI F, SINGH H, et al. Electric vehicle fast charging infrastructure planning in urban networks considering daily travel and charging behavior[J]. Transportation Research Part D: Transport and Environment, 2021, 93: 102769. doi: 10.1016/j.trd.2021.102769 [17] 侯琦. 高速路网的充换电设施定容优化研究[D]. 天津: 天津工业大学, 2023.HOU Q. Optimization study of charging and switching facilities' fixed capacity for high-speed road networks[D]. Tianjin: Tianjing Polytechnic University, 2023. (in Chinese) [18] HU D, ZHANG J, LIU Z W. Charging stations expansion planning under government policy driven based on Bayesian regularization backpropagation learning[J]. Neurocomputing, 2020, 416: 47-58. doi: 10.1016/j.neucom.2019.03.092 [19] 侯慧, 唐俊一, 王逸凡, 等. 城区电动汽车充电站布局规划研究[J]. 电力系统保护与控制, 2022, 50(14): 181-187.HOU H, TANG J Y, WANG Y F, et al. Layout planning of electric vehicle charging stations in urban areas [J]. Power System Protection and Control, 2022, 50(14): 181-187. (in Chinese) [20] FAKHRMOOSAVI F, KAVIANIPOUR M, SHOIAEI M H, et al. Electric vehicle charger placement optimization in Michigan considering monthly traffic demand and battery performance variations[J]. Transportation Research Record: Journal of the Transportation Research Board, 2021, 2675(5): 13-29. doi: 10.1177/0361198120981958 [21] 朱永胜, 杨振涛, 丁同奎, 等. 考虑用户动态充电需求的电动汽车充电站规划[J]. 郑州大学学报(工学版), 2023, 44 (2): 82-90.ZHU Y S, YANG Z T, DING T K, et al. Electric vehicle charging station planning considering users' dynamic charging demand [J]. Journal of Zhengzhou University (Engineering Science), 2023, 44(2): 82-90. (in Chinese) [22] 詹钊弘, 傅成红. 异质交通流单点信号控制D3QN算法改进研究[J]. 交通工程, 2025, 25(5): 45-53.ZHAN Z H, FU C H. Research on the improvement of D3QN algorithm for Isolated intersection signal control of heterogeneous traffic flow[J]. Road Traffic & Safety, 2025, 25(5): 45-53. (in Chinese) [23] HOU Y, LIU L, WEI Q, et al. A novel DDPG method with prioritized experience replay[C]. 2017 IEEE International Conference on Systems, Man, and Cybernetics. Banff, Canada: IEEE, 2017. [24] XI L, ZHOU L, XU Y, et al. A multi-step unified reinforcement learning method for automatic generation control in multi-area interconnected power grid[J]. IEEE Transactions on Sustainable Energy, 2020, 12(2): 1406-1415. [25] 李帅兵, 朱宇辰, 谭九鼎, 等. 计及负荷时空特性的高速公路链式微网光-储-充容量优化配置方法[J]. 电网技术, 2025, 49 (7): 2768-2778.LI S B, ZHU Y C, TAN J D, et al. Optimal method for photo-voltaic-storage-charging capacity configuration of microgrid-in-chain on expressways considering load spatio-temporal characteristics [J]. Power System Technology, 2025, 49 (7): 2768-2788. (in Chinese) [26] 曾颖娇. 高速公路电动汽车充电站选址定容研究[D]. 南昌: 南昌航空大学, 2023.ZENG Y J. Study on the siting and capacity of electric vehicle charging stations on highways[D]. Nanchang: Nanchang Hangkong University, 2023. (in Chinese) -
点击查看大图
图(5) / 表(4)
计量
- 文章访问数: 44
- HTML全文浏览量: 16
- PDF下载量: 2
- 被引次数: 0