It is difficult to recognize handwriting number, because writing styles are different, context is free and recognition requires high accuracy. According to the characteristics and requirements of handwriting number recognition, this paper uses the Deep Learning algorithm for classification, based on the sample of training to recognize the handwriting number.At the same time, the experiment compared the classification effect with that of SVM algorithm and BP neural network. The experimental results show that the Deep Learning of handwriting number in recognition has a higher accuracy.