Objective Aiming at the problems of scattered attention in existing Transformer models for polyp image segmentation and the low segmentation accuracy caused by information loss during the fusion of multi-level features extracted by the encoder a new segmentation model named PVT-CAMNet is proposed. Methods In this model the Pyramid Vision Transformer PVT was used as the encoder. Then a Multi-scale Feature Attention Extraction MFAE module and an Inter-layer Attention Aggregation IA module were designed. Among them the PVT ensured the generalization ability of the model through its self-attention mechanism. The MFAE used filters of different sizes to extract features at multiple scales to alleviate the problem of scattered attention. The IA interactively fused features at different levels to effectively solve the problem of information loss caused by the fusion of multi-level features. Finally a Global Context GC module was introduced to enable the model to better understand the pixel dependency relationship between feature maps. Results Evaluations were carried out on the Kvasir CVC-ClinicDB CVC-ColonDB and ETIS datasets.Comparing the performance of the proposed PVT-CAMNet model with that of the optimal baseline model the mDice valuesof PVT-CAMNet increased by 1. 76% 1. 51% 3. 15% and 1. 73% respectively and the mIoU values of PVT-CAMNet increased by 0. 81% 1. 74% 2. 65% and 3. 84% respectively on these four datasets. Conclusion PVT-CAMNet is superior to other advanced methods in both learning performance and generalization capability demonstrating significant application value in polyp image segmentation.