Objective This study addresses the following challenges in vehicle detection under road traffic environments small target sizes occlusions unclear targets and the consequent false positives and missed detections. Methods An index named ShapeIoU was proposed to assess the matching degree between the target detection box and the real shape of the target. By calculating the similarity of the target contour the matching degree between the detection box and the real shape was measured to enhance the precision and accuracy of object detection. A convolutional kernel module adjusted by the dynamic snake convolution DSConv module was introduced. This module could adaptively adjust the shape of the convolutional kernel according to the changes in the shape and size of the target so as to capture the features of the target more effectively. To improve the multi-scale detection ability enhance the detection accuracy and the expression ability of the model a P2 detection head was added during the improvement. Results The Precision Recall mAP@ 0. 5 and mAP@ 0. 5 0. 95 of the improved YOLOv8 model algorithm increased by 2. 6% 3. 2% 3. 2% and 3. 0% respectively reaching 87. 9% 84. 2% 89. 4% and 65. 4%. Conclusion The detection of small occluded and unclear targets in complex traffic environments has been significantly improved. It is verified that this improved method can effectively solve problems such as low accuracy in vehicle target detection.