Objective Raw multi-modal fault data collected in practical applications is usually nonlinear data containing a large amount of noise and redundant information. How to extract effective nonlinear fault features from different fault modalities is still a challenging problem. Methods A discriminant manifold sensitivity cross-modal fault diagnosis method was proposed. In the method the correlation coefficient between different modalities was first constructed in the cross-modal fault space using correlation analysis theory and the equivalent optimization model of the correlation coefficient was obtained by theoretical derivation. Then the discriminant manifold sensitivity scatter was constructed by using local neighborhood graphs and a discriminant manifold sensitivity cross-modal fault diagnosis model was constructed by maximizing the correlation between different modalities and minimizing the discriminant manifold sensitivity scatter. The analytical solutions of the optimization model were derived theoretically so that nonlinear fault features with well discriminant power can be obtained from fault data of different modalities. Results Targeted experiments were designed on the Germany Paderborn bearing dataset and the multi-modal bearing fault dataset. The experimental results showed that good diagnosis accuracy can be achieved with a small number of training fault samples. Conclusion The proposed method is an effective cross-modal fault diagnosis method.