Based on the perspective of new structural economics, this paper divided the infrastructure into soft infrastructure and hard infrastructure and examined the mechanism of the impact of infrastructure construction on economic growth and narrowing the cross-country income gap from labor division viewpoint: At the initial stage of modern economic growth, certain soft infrastructure is needed to boost technical innovation and to ensure labor division, the evolution of the labor division needs to improve hard infrastructure to enlarge market scale physically, furthermore, with the economic growth, both soft infrastructure and hard infrastructure need to be continuously improved to cooperatively boost optimized development of social labor division system and economic growth so as to realize sustainably endogeneously economic growth. In developing countries, because of technical innovation externality, the action of soft infrastructure is more significant than that of hard infrastructure not only in the initial stage of economic growth but also in the whole process of catching up and surpassing. The difference of infrastructure results in the heterogeneity of economic growth and further produces income gap.Because the difference of soft infrastructure is more difficult to be eliminated, the impact of soft infrastructure on income gap is more obvious. The empirical analysis results based on the data of 139 countries and regions during 1996-2014 support above theoretical analysis. Soft infrastructure has more significant boosting effect on economic growth, soft and hard infrastructure plays a significant role in economic growth in the OECD countries, however,in non-OECD countries, only the soft infrastructure plays a significant role.The larger the gap of infrastructure is, the larger per capita income gap is, and the impact of soft infrastructure on income gap in non-OECD countries is larger. The economic development in developing countries should attach more importance to soft infrastructure, however, the improvement of soft infrastructure should not take western value system as standard but should be based on China’s reality to take the promotion of formation and evolution of effective labor division as standard.