As a typical policy for the adjustment of administrative districts at the county level, the development dividend released by “removing counties and establishing districts” has become increasingly prominent. However, in the research on the policy effect of “removing counties and establishing districts”, the existing literature rarely discusses its influence on the innovation of private enterprises, especially the lack of empirical evidence. This paper holds that the market segmentation caused by administrative barriers is one of the main factors restricting the innovation and development of private enterprises, and the adjustment of administrative districts must promote the market integration between regions in order to effectively empower the innovation and development of private enterprises. The administrative barriers and market segmentation between the annexed counties and the municipal districts are weakened by “removing counties and establishing districts”, which can enhance the technological innovation capability of private enterprises in the annexed counties through promoting the improvement of human capital and increasing external financing. The innovation willingness of private enterprises in the removed counties can be enhanced through the competitive intensification effect of industrial agglomeration and the innovation cost reduction effect, which in turn promotes the innovation development of private enterprises. Based on the quasi-natural experiment of “removing counties and establishing districts”, this paper uses the data of private listed companies in Shanghai and Shenzhen A-shares from 2011 to 2020 to carry out multi-stage double differential test. The results show that “removing counties and establishing districts” has significantly promoted the increase of R&D intensity and patent authorization number of private enterprises in the removed counties, among which the level of human capital, external financing and regional industrial agglomeration have played a part in the intermediary effect. That is to say, there are three influencing paths, namely, “removing counties and establishing districts—enhancing human capital of enterprises—increasing input and output of technological innovation”, “removing counties and establishing districts—easing the financing constraints of enterprises—increasing input and output of technological innovation”, “removing counties and establishing districts—promoting regional industrial agglomeration—increasing input and output of technological innovation”. The innovation promotion effect of “removing counties and establishing districts” is mainly reflected in enterprises with high dependence on external financing and enterprises in the eastern region. “Removing counties and establishing districts” promotes the innovation of private enterprises in the secondary and tertiary industries, but has a negative impact on the innovation of private enterprises in the primary industry. While promoting industrial agglomeration and innovation of private enterprises in the merged counties, “removing counties and establishing districts ” has not produced significant “siphon effect” on the surrounding counties that have not been merged. Compared with the existing literature, this paper discusses the influence and mechanism of “removing counties and establishing districts” on the innovation behavior of private enterprises, which not only expands and deepens the research on the micro-economic effect of administrative division adjustment, but also provides empirical evidence and policy enlightenment for further effectively promoting the innovation and development of private enterprises. This paper shows that the administrative district adjustment represented by “removing counties and establishing districts” effectively promotes the innovation development of private enterprises by promoting regional market integration. Therefore, in the process of actively promoting the adjustment of administrative districts, we should focus on building a unified big market. On one hand, we should create a more open, fair, and shared market environment; on the other hand, we should reduce excessive intervention in the market and realize the deep coupling between “promising government” and “effective market”.