The change of public fund manager affects the investment value and risk return expectations of fund products. The change in domestic public fund becomes more frequent, but existing researches are mostly limited to the traditional principal-agent framework, focusing on the discussion of the linear relationship between the fund performance and the change of the fund manager. However, the impact of the evaluation mechanism and market environment characteristics on the above relationship is rarely discussed. Taking the three types of public funds from 2006 to 2019 as the research sample such as common stock, partial stock mixed fund, and flexible allocation fund, based on the data set of 1726 funds and 39227 valid samples, this article has made a multi-dimensional examination on the driving mechanism and constraints of domestic public fund manager changes. The study finds (1) The response of fund manager changes to fund performance is mainly presented in the form of relative return ranking, while the performance of the absolute net value growth rate is not sensitive; (2) The changes in market environment will restrict the actual impact of fund performance on fund manager changes, and non-steady markets are more likely to trigger fund manager changes; (3) Different market environments have heterogeneous effects on the change of fund managers. The bull market will form a positive push, while the bear market will produce a negative compression effect. Considering the dual driving mechanism of fund managers change and its market heterogeneity characteristics comprehensively, it is necessary to improve the internal evaluation system of fund companies to further standardize the principal-agent relationship and form a more comprehensive and complete external fund evaluation system.