Traditional gesture recognition methods need to select features manually， and the selected features are difficult to adapt to the variability of gestures. So，a gesture recognition method based on skin color feature and convolutional neural network is designed. Firstly， the elliptical model is used to segment the gesture samples in complex background. The obtained gesture regions are binarized and normalized. Then， the processed gesture samples are iteratively trained by convolution neural network. The key highdimensional features of the gestures are extracted， and then the gesture recognition model is obtained. The gesture model trained by this method can autonomously extract feature and classify gesture. Experiments show that the gesture recognition method has high recognition rate on the test set. In the real scene test， the method also achieves good gesture recognition effect and has good realtime performance and robustness.