Abstract: To effectively make use of deep learning technology automatic feature extraction ability,and solve the problem when the training sample size reduced or the iteration times reduced the recognition performance fell sharply,this paper proposed a deep learning algorithm based on Fisher criterion.In the feed forward spread,this method used convolution neural network to extract automatically image features such as structural information,and used convolution network of sharing weights and pooling,subsampling methods to reduce the weight number,and the method reduced the model complexity.When the back propagation adjusted the weights,it adopted the constraints based on Fisher criterion.At the same time,it kept the samples in small distance with-class and large distance between-class,so that the weights could be more close the optimal value for classification.It improved the recognition rate effectively when the sample size was insufficient or when it had few training iterations.A large number of experiments show that when the label samples are insufficient and the training iteration fewer,the hybrid deep learning algorithm based on Fisher criterion still achieves good recognition effect.