A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN

YIN Xing; ZHANG Yu; ZHENG Qianqian; TANG Kexin

doi:10.3963/j.jssn.1674-4861.2022.06.009

Volume 40 Issue 6

Dec. 2022

Turn off MathJax

Article Contents

Article Navigation > Journal of Transport Information and Safety > 2022 > 40(6): 81-91

YIN Xing, ZHANG Yu, ZHENG Qianqian, TANG Kexin. A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN[J]. Journal of Transport Information and Safety, 2022, 40(6): 81-91. doi: 10.3963/j.jssn.1674-4861.2022.06.009

Citation:

YIN Xing, ZHANG Yu, ZHENG Qianqian, TANG Kexin. A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN[J]. Journal of Transport Information and Safety, 2022, 40(6): 81-91. doi: 10.3963/j.jssn.1674-4861.2022.06.009

Citation:

PDF( 2284 KB)

A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN

doi: 10.3963/j.jssn.1674-4861.2022.06.009

YIN Xing^1
,,
ZHANG Yu^{1, 2},
ZHENG Qianqian¹,
TANG Kexin^{1
,
,}

1.
School of Transportation and Logistics Engineering, Wuhan University of Technology, Wuhan 430063, China
2.
Inland Port and Shipping Industry Research Co. Ltd. of Guangdong Province, Shaoguan 512000, Guangdong, China

Received Date: 2022-06-13
Available Online: 2023-03-27

Abstract

Abstract

The interactive operations of quay cranes, artificial intelligent robots of transportation(ARTs), and yard cranes during automatic container terminal unloading are studied. A three-stages integrated scheduling model of automated container terminal based on hybrid flow shop scheduling problem is proposed, with the criterion of minimizing the makespan. In addition, the scheduling environment requires high real-time response. A deep reinforcement learning algorithm, namely double deep Q-network(DDQN), is used to solve the problem of dynamic characteristics of the automatic terminal scheduling environment. The input of the model is the real-time status data of the equipment at each stage. The neural network is used to fit the value-action function. The model is trained by experience playback mechanism. The single heuristic rule with the compound heuristic rule is taken as the equipment candidate behavior. By strengthening the learning action selection and action evaluation mechanism, the optimal container equipment combination strategy is obtained. According to the actual survey data of Tianjin Port Automation Terminal, different scales cases are designed for experimental comparison and analysis. The results show that: the total operation time of the proposed method is reduced by 7.84% on average compared with the current advanced particle swarm optimization algorithm, and the gap with the theoretical lower bound value is 6.0%, 5.6%, and 4.6%, respectively. In addition, the equipment loading in the three stages is relatively balanced. And the average utilization rate of equipment is 89%, which can meet the actual application requirements. In small-scale examples, the average error of the total completion time obtained by DDQN is 1.99% compared with Gurobi. With the increase of the size of the example, the solving time is increased by 59% at most, which verifies the feasibility and efficiency of the proposed method for improving the operation efficiency of the automated container terminal.
- intelligent transportation,
- automated container terminal,
- three-stage integrated scheduling,
- deep reinforcement learning,
- hybrid flow shop

FullText(HTML)

References(25)

References

[1]	高雪峰. 基于深度强化学习的自动化集装箱码头双场桥动态调度研究[D]. 大连: 大连理工大学, 2020. GAO X F. Research on dynamic scheduling of two YC in automated container terminal based on deep reinforcement learning[D]. Dalian: Dalian University of Technology, 2020. (in Chinese)
[2]	丁一, 袁浩, 方怀瑾, 等. 考虑冲突规避的自动化集装箱码头AGV优化调度方法[J]. 交通信息与安全, 2022, 40(3): 96-107. doi: 10.3963/j.jssn.1674-4861.2022.03.010 DING Y, YUAN H, FANG H J, et al. An optimal scheduling method of AGVs at automated container terminal considering conflict avoidance[J]. Journal of Transport Information and Safety, 2022, 40(3): 96-107. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2022.03.010
[3]	夏孟珏, 史学鑫, 李美贞. 集装箱码头岸桥突发故障情况下装卸船作业重调度研究[J]. 上海海事大学学报, 2022, 43(1): 30-37. https://www.cnki.com.cn/Article/CJFDTOTAL-SHHY202201005.htm XIA M J, SHI X X, LI M Z, et al. Study on handling operation rescheduling under sudden malfunction of container terminal quay cranes[J]. Journal of Shanghai Maritime University, 2022, 43(1): 30-37. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-SHHY202201005.htm
[4]	梁承姬, 杨全业. 双循环操作策略下集装箱码头岸桥与集卡多船作业联合调度[J]. 重庆交通大学学报(自然科学版), 2018, 37(3): 106-114. https://www.cnki.com.cn/Article/CJFDTOTAL-CQJT201803019.htm LIANG C J, YANG Q Y. Container terminal QC & IT integral scheduling model for multi-vessel operation under double-cycle strategy[J]. Journal of Chongqing Jiaotong University(Natural Science), 2018, 37(3): 106-114. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-CQJT201803019.htm
[5]	陈超, 邱建梅, 台伟力, 等. 出口箱随机入港下的集装箱码头泊位调度[J]. 交通信息与安全, 2014, 181(1): 91-96. doi: 10.3963/j.issn.1674-4861.2014.01.019 CHEN C, QIU J M, TAI W L, et al. Berth allocation planning in container terminals for the outbound containers arrival in random order[J]. Journal of Transport Information and Safety, 2014, 181(1): 91-96. (in Chinese) doi: 10.3963/j.issn.1674-4861.2014.01.019
[6]	常祎妹, 朱晓宁. 不确定因素下的集装箱码头车船间装卸作业集成调度[J]. 交通运输工程学报, 2017, 17(6): 115-124. https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201706017.htm CHANG Y M, ZHU X N. Integrated scheduling of handling operation between train and vessel in container terminal under uncertain factor[J]. Journal of Traffic and Transportation Engineering, 2017, 17(6): 115-124. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201706017.htm
[7]	KIM K H, PARK Y M. A crane scheduling method for port container terminals[J]. European Journal of Operational Research, 2004, 156(3): 752-768.
[8]	秦天保, 葛浩, 沙梅. 约束规划求解集装箱装卸系统集成调度问题[J]. 系统工程理论与实践, 2015, 35(8): 2127-2136. https://www.cnki.com.cn/Article/CJFDTOTAL-XTLL201508023.htm QIN T B, GE H, SHA M. Constraint programming for the integrated scheduling problem of container handling system in container terminals[J]. System Engineering-Theory & Practice, 2015, 35(8): 2127-2136. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-XTLL201508023.htm
[9]	ZHEN L, YU S C, WANG S A, et al. Scheduling quay cranes and yard trucks for unloading operations in container ports[J]. Annals of Operations Research, 2019, 273(1): 455-478.
[10]	钟祾充, 李文锋, 贺利军, 等. 集装箱码头混合零空闲柔性流水作业调度优化[J/OL]. 计算机集成制造系统: 1-22[2022-10-28]. http://kns.cnki.net/kcms/detail/11.5946.TP.20211228.1358.016.html. ZHONG L C, LI W F, HE L J, et al. Optimization of mixed no-idle flexible flow scheduling in container terminal[J]. Computer Integrated Manufacturing Systems: 1-22[2022-10-28]. http://kns.cnki.net/kcms/detail/11.5946.TP.20211228.1358.016.html. (in Chinese)
[11]	陈超, 张哲, 曾庆成. 集装箱码头混合交叉作业集成调度模型[J]. 交通运输工程学报, 2012, 12(3): 92-100. https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201203017.htm CHEN C, ZHANG Z, ZENG Q C. Integrated scheduling model of mixed cross-operation for container terminal[J]. Journal of Traffic and Transportation Engineering, 2012, 12(3): 92-100. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JYGC201203017.htm
[12]	杨彩云, 张煜, 徐亚军, 等. 自动化集装箱码头ART动态调速群智决策研究[J]. 武汉理工大学学报, 2022, 44(1): 28-35. https://www.cnki.com.cn/Article/CJFDTOTAL-WHGY202201005.htm YANG C Y, ZHANG Y, XU Y J, et al. Simulation analysis of ART dynamic speed regulation of automated container terminal based on MAS[J]. Journal of Wuhan University of Technology, 2022, 44(1): 28-35. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-WHGY202201005.htm
[13]	SUTTON R S, BARTO A G. Reinforcement learning: an introduction[J]. IEEE Transactions on Neural Networks, 1998, 9(5): 1054-1054.
[14]	张华胜. 基于深度强化学习的自动化码头跨运车集成调度研究[D]. 上海: 上海海事大学, 2021. ZHANG H S. Research on integrated scheduling of shuttle carriers in automated terminals based on deep reinforcement learning[D]. Shanghai: Shanghai Maritime University, 2021. (in Chinese)
[15]	尚晶, 徐长生. 基于强化学习的集装箱码头卡车调度策略研究[J]. 武汉理工大学学报, 2011, 33(3): 72-76. https://www.cnki.com.cn/Article/CJFDTOTAL-WHGY201103015.htm SHANG J, XU C S. Vehicle scheduling in container terminal based on reinforcement learning[J]. Journal of Wuhan University of Technology, 2011, 33(3): 72-76. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-WHGY201103015.htm
[16]	PARK S J, YOO Y H, PYO C W. Applying DQN solutions in fog-based vehicular networks: Scheduling, caching, and collision control[J]. Vehicular Communications, 2022, 33(C): 100397.
[17]	SALVADORl M S. A solution to a special class of flow shop scheduling problems[J]. Symposium on the Theory of Scheduling and Its Applications, 1972(86): 83-91.
[18]	SUTTON R S, BARTO A G. Reinforcement Learning: An Introduction[M]. second edition. 北京: 电子工业出版, 2019.
[19]	陆志强, 任逸飞, 许则鑫. 基于深度学习的资源投入问题算法[J]. 计算机集成制造系统, 2021, 27(6): 1558-1568. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJJ202106003.htm LU Z Q, REN Y F, XU Z X. Research on the deep learning algorithm for resource investment problem[J]. Computer Integrated Manufacturing Systems, 2021, 27(6): 1558-1568. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSJJ202106003.htm
[20]	HAN B A, YANG J J. Research on adaptive job shop scheduling problems based on dueling double DQN[J]. IEEE Access, 2020(8): 186474-186495.
[21]	WANG Y H, ZHOU X, SHI Y X, et al. Transmission network expansion planning considering wind power and load uncertainties based on multi-agent DDQN[J]. Energies, 2021, 14(19): 6073.
[22]	LIU C L, CHANG C C, TSENG C J. Actor-Critic deep reinforcement learning for solving job shop scheduling problem[J]. IEEE Access, 2020(8): 71752-71762.
[23]	邢曦文. 基于混合流水作业组织的集装箱码头装卸集成调度优化[D]. 大连: 大连海事大学, 2013. XING X W. Optimization of container loading/unloading integrated scheduling in a container terminal based on hybrid flowshop[D]. Dalian: Dalian Maritime University, 2013. (in Chinese)
[24]	肖鹏飞, 张超勇, 孟磊磊, 等. 基于深度强化学习的非置换流水车间调度问题[J]. 计算机集成制造系统, 2021, 27(1): 192-205. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJJ202101018.htm XIAO P F, ZHANG C Y, MENG L L, et al. Non-permutation flow shop scheduling problem based on deep reinforcement learning[J]. Computer Integrated Manufacturing Systems, 2021, 27(1): 192-205. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSJJ202101018.htm
[25]	YUTTHAPONG T, WERASAK K. Comparing nonlinear inertia weights and constriction factors in particle swarm optimization[J]. International Journal of Knowledge-Based and Intelligent Engineering Systems, 2011, 15(2): 65-70.

Relative Articles

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(8) / Tables(10)

Get Citation

PDF

XML

Article Metrics

Article views (813) PDF downloads(39)

A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN

doi: 10.3963/j.jssn.1674-4861.2022.06.009

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

A Study of Integrated Scheduling of Automated Container Terminal Based on DDQN

doi: 10.3963/j.jssn.1674-4861.2022.06.009

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Export File

Citation

Format

Content