In target tracking system, the key to obtaining good characterization is to determine target tracking performance, therefore, this paper proposes a target tracking algorithm about the correlated filters and convolutional neural networks. This algorithm firstly pre-selects reference blocks which can distinguish the target appearance in each video scene to construct the training samples and then build the two-way convolutional neural network which is not completely symmetric and which shares the weights. This convolutional neural network makes the target output characteristics outside the reference area as similar as possible to the target output characteristics in the reference area so as to get good characterization of the target in the reference area. The correlated filter module is added into a way to realize the combination of convolutional network and the correlated filter. Experimental results verified the feasibility of the algorithm.