In the fields of cadastral surveying and disaster management automatic extraction of building contours and positions is crucial. To address the problem of inaccurate segmentation accuracy of buildings in high- resolution remote sensing images due to environmental factors an improved lightweight SOLOv2 instance segmentation model called NDFEDet-SOLOv2 was proposed. Methods This model adopted a lightweight EfficientDet network with a bidirectional feature pyramid network BiFPN feature fusion method. The EfficientNet in the backbone network was upgraded to EfficientNetv2 and the three-layer MBConv module SE attention in EfficientNetv2 was replaced with a lightweight normalized attention mechanism NAM containing DropBlock regularization forming the NAD-MBConv module. The feature fusion part of BiFPN incorporated a bi-level routing attention visual transformer BiFormer into each feature layer at its tail end to form a bi-directional horizontally routing attention feature pyramid network structure Bi- FPN-Former which focused on the contour information of small buildings and achieved higher-level feature fusion. Results Compared with traditional lightweight SOLOv2 instance segmentation algorithms the NDFEDet-SOLOv2 model has improved average accuracies by 3. 9% 3. 7% and 2. 5% for mAP mAP50 and mAP75 respectively and improved detection frame rate FPS by 4. 7 frames / s. Conclusion The lightweight NDFEDet-SOLOV2 instance segmentation algorithm model eliminates image distortion of building edges and corners and can accurately extract the basic contours of buildings in remote sensing images even in complex and uneven geographical environments. This provides a theoretical reference for the update of urban layouts and the detection of building changes.