Under the background of big difference between urban and rural areas and high cost of labor opportunity, agricultural labor transfer to non-agriculture continuously exists, however, under the traditional agricultural production mode, the labor transfer causes labor shortage and labor cost rise and finally triggers factor substitution in agricultural production. By using inter-provincial panel data from main provinces and municipalities planting crops during 2004—2015, this paper studies the effects of factor substitution caused by agricultural labor shifting and its influence, and analyzes the internal relation between agricultural labor transfer and fertilizer overuse. The results show that currently in three main grains production in China, the use of fertilizers is over the optimal quantity in economic meaning, there is the problem in “economic excessiveness”. Because the labor cost growth is far higher than the growth of fertilizer price, the labor transfer is the important reason for the farmers to overuse fertilizers, as a result, there is factor substitution relation between agricultural labor transfer and fertilizer use. Thus, only by transforming agricultural production and management mode and by optimizing agricultural production and management main body, can the excessive use of fertilizers caused by agricultural labor transfer and low allocation of agricultural production factors be basically solved.