Based on 1981—2011 sample data of China, the influence of financial development scale and efficiency difference and other main factors on the income gap between the urban and rural residents is examined by quantile regression method, and the results show that at each quantile point, the impact of fixed asset investment difference on the income gap between the urban and rural residents is strongest, that the influence of labor productivity is the second strongest and the impact of financial development scale and efficiency difference is the weakest. Thus, financial development scale and efficiency disparity are not the main factor for current income gap enlargement between the urban and rural residents of China. However, the disparity of fixed asset investment and labor productivity between the urban and rural inhabitants is the main cause. When the income gap between the urban and rural areas is at the high quantile, the influence of both financial scale and efficiency disparity is not significant. When the income gap is at the intermediate quantile, financial efficiency disparity has a significant negative influence on the income gap between the urban and rural areas. When the income gap is at the low quantile, financial scale disparity has obvious positive impact on the income gap between the urban and rural areas. Therefore, narrowing the income gap between the urban and rural areas of China should take differentiated policies according to the different quantile levels of the income gap itself between the urban and rural residents.