The New Economics of Labor Migration believes that under different market environments, factor mobility can produce different impacts on agricultural production, therefore, the effect of labor transfer on agricultural production has differences. By taking rice production as an example, by using sampling survey household data of Zhejiang, Jiangsu, Guangdong, Sichuan, Chongqing and Guizhou, this paper empirically tests the adjusting effect of the influence of market-oriented level labor transfer on the farmers’ technical efficiency, and the analysis shows that the decrease of agricultural labor brought by labor transfer causes the loss of farmers’ rice production technical efficiency, that the remittance from transferred labor can not help the farmers to improve the technical efficiency of rice production, therefore, the increase of market-oriented level can not change the relation between migrant workers’ remittance and farmers’ technical efficiency but significantly inhibit the passive impact produced by labor run-off. Thus, the difference of market environment can lead to the difference in labor transfer effect. Local governments of China should make efforts to improve market environment, raise market-oriented level, and boost agricultural production efficiency.