According to the multivariate characteristics of college students′ selflearning ability in mathematics evaluation，a method of quantitative evaluation of students' selflearning ability in mathematics is proposed. Firstly，we set up an index system for self-study ability of college students by analyzing the influencing factors of college students′ selflearning ability. Secondly，by introducing the optimal transfer matrix，the consistency of judgment matrix in AHP can be naturally satisfied，and the AHP weights at all levels are obtained. Then，we build an evaluation model of self-study ability of college students based on whitening weight function. Finally，we establish the whitening weight function，obtain the grey evaluation weight matrix of indexes at all levels and make a comprehensive evaluation of college students′ selflearning ability in mathematics through examples. The results show that the model is reasonable,feasible and effective because this model can decrease the influence of subjective factors of evaluators.