A Novel Ship Driver Behavior Recognition Approach Based on Improved TSM

In maritime transportation, irregular operations by crew onboard represent a significant factor causing maritime accidents. The design of a real-time detection method for monitoring ship driver behavior holds substantial importance. Compared to automobilism driving and security surveillance, the ship's bridge environment is more complex, posing challenges such as the inability to simultaneously monitor multiple crew members, inefficiency and lower accuracy rates. To solve this problem, a two-step multi-person behavior recognition approach combining multi-target tracking and behavior recognition is proposed. Firstly, a multi-target tracker uses the YoloV7 and ByteTracker to generate continuous feature maps of crew. Based on the temporal shift module (TSM) algorithm for single-target behavior recognition, this approach utilizes techniques such as oversampling and cross-frame stitching to process continuous feature maps. Meanwhile, it leverages EfficientNet-B3 alongside the co-ordinate attention (CA) module to produce highly accurate recognition outcomes. The research establishes a ship's bridge behavior dataset "SC-Action", with data from different ship's bridge surveillance videos, including 2,000 behavior samples of both regular and irregular behaviors. Transfer learning and ablation experiments conducted on this dataset demonstrate that the proposed method achieves real-time behavior recognition of three crew at 24 frames per second, with both recognition speed and accuracy superior to mainstream algorithms. In tests targeting single-person behavior recognition, the method's accuracy improved by 1.3% compared to the baseline TSM model after applying the image enhancement module. Incorporating attention mechanism, the accuracy further increased by 1.78%, reaching 82.1%, with only a 0.1% increase in computational load. During multi-target testing, the method also surpasses leading approaches such as SlowFast in practical inference speed and performance, affirming its efficacy.