This article, based on the data from a survey of two destitute areas of Wuling and Luoxiao Mountains’ 407 migrant workers’ startups, explores the interaction effect and mechanism between the institutional environment, the organizational legitimacy and the growth of migrant workers’ startups. The study finds that all the three dimensions of institutional environment (institutional environment, cognitive environment and standard environment) in destitute areas and three dimensions of organizational legitimacy (practice legitimacy, moral legitimacy and cognitive legitimacy) have a significantly positive effect on the growth of migrant workers’ startups, that institutional environment has positive impact on organizational legitimacy of migrant workers’ startups, that the organizational legitimacy plays a intermediary role in the relationship between the institutional environment and the growth of migrant workers’ startups. The growth of migrant workers’ startups in the destitute areas can not be separated from the regulation and perfection of the institutional environment at each level, migrant workers’ startups can boost their growth by organizational legitimacy which is closely related to institutional environment, thus, the institutional environment should be further improved to actively raise the organizational legitimacy of migrant workers’ entrepreneurship to promote the healthy growth of migrant workers’ startups.