A Method of Deep Reinforcement Learning-based Ramp Metering for Mainline-ramp Coordination

The merging area of expressway ramps is prone to traffic congestion and frequent accidents. To improve the performance of traditional ramp metering algorithms in terms of response speed and control accuracy, a ramp metering method based on reinforcement learning is studied. The ramp metering problem is formulated as a Markov decision process. The action space is designed using discrete signal phases to improve training efficiency. A state space and a multi-dimensional reward function are constructed to represent the operating states of the mainline and ramps. At the state perception level, a real time traffic detection mechanism is incorporated. To avoid high frequency phase switching, a minimum phase duration constraint is imposed on action outputs. Meanwhile, prioritized experience replay is used during the training process to enhance the model performance. Furthermore, the deep network structure is optimized to improve convergence speed and generalization in complex traffic environments. Residual connections and layer normalization are introduced to construct a lightweight and efficient multi-layer perception network. A microscopic simulation platform is used to conduct systematic experiments to verify the control effect of the proposed method. The results show that compared with the no-control scenario, the system throughput increased by 52.67% under the proposed mainline-ramp coordinated control. Meanwhile, the average travel time decreases by 58.21% under the proposed method. Moreover, traffic efficiency on the mainline and ramps improves significantly under the proposed method. The proposed method is deployed in the entrance traffic limiting project of the section from Hangzhou West to Yuqian Interchange on the Hangzhou-Huizhou Expressway. The road network structure and traffic flow characteristics of this section are accurately reproduced. The results indicate that network vehicle numbers and mainline average speed increase, while speed fluctuation is more moderate. These improvements demonstrate that the proposed method has high potential for engineering deployment.