Deep Reinforcement Learning-based Dynamic Capacity Planning for Consecutive Highway Charging Stations

The contradiction between the surging number of new energy vehicles and the inadequate charging services on highways becomes increasingly prominent. Compounding this issue, traditional multi-objective planning methods for charging stations rely on static assumptions and fail to adapt to dynamic traffic environments. To resolve these limitations, a dynamic capacity planning model based on deep reinforcement learning is proposed for consecutive highways charging stations. The capacity planning problem for consecutive charging stations is modeled as a Markov decision process. The state space integrates real-time operational metrics of charging stations and dynamic vehicular features to characterize the nonlinear coupling among traffic flow, charging demand, and user behavior. Furthermore, a multi-objective reward function balances user satisfaction and operational costs is designed. Algorithmically, the D3QN-PER-2s algorithm is developed. It combines the Dueling deep Q-network (DQN) architecture to separate state value from action advantage functions and employs Double DQN to mitigate Q-value overestimation problem. Additionally, a prioritized experience replay mechanism is introduced to evaluate experience importance and conduct priority sampling. To enhance learning stability, the algorithm further adopts a two-step temporal difference update strategy that incorporates future rewards and value estimates. The model interacts with the SUMO simulation environment for real-time learning, eliminating the dependence on historical data. To generate highly robust planning solutions, a quantile analysis strategy based on the empirical cumulative distribution function is introduced. Statistical analysis is conducted on extensive simulated decisions after training. The 90th percentile value is selected as the final plan, effectively balancing cost-effectiveness and robustness against demand fluctuations. Experimental results demonstrate that the proposed deep reinforcement learning-based scheme controls the proportion of queuing vehicles to within 10% reduces the average queuing time to less than 3 minutes. Compared to a multi-objective queuing model solved by genetic algorithm, this approach reduces construction costs by 24.3% and improves peak-hour charger utilization by 17.4%. These results prove the superiority of the proposed method in solving multi-objective capacity planning problem of chargers, enhancing the efficiency of highway charging services and user charging experience.