Agricultural product e-commerce has become a new kinetic energy to promote the industrialization and digitalization of agriculture and the sustainable increase of farmers’ income. Exploring the factors influencing the adoption of e-commerce by farmers and their logical relationship is of great practical significance for improving the adoption of e-commerce by farmers and promoting the organic connection between poverty alleviation and rural revitalization. This paper used data from 631 questionnaire surveys of farmers in the western mountainous regions in China. Firstly, the Logit model was used to determine the influencing factors of rural households’ e-commerce behavior, and then the ISM model was used to analyze the relationship and hierarchical structure of each influencing factor. The results of the study show that eight factors, including farmers’ health status, education level, household income level, income structure, work status of family members, topographic features, distance from townships, and broadband access, have a significant impact on farmers’ e-commerce behavior. Among them, broadband access to households is a direct factor at the surface level, household income level is the factor at the second level, health status, income structure, and work status of family members are the factors at the third level, and the education level of farmers, topographic features and distance from townships are the deep root causes factor. Based on this, policy suggestions including supporting the development of e-commerce in remote areas, strengthening education and training and talent introduction, increasing the construction of new infrastructures and integrated applications, and emphasizing the construction of e-commerce service stations are put forward.