Convolutional Neural Networks (CNN) and Local Binary Patterns (LBP) can only extract single features of facial expression images during facial expression feature extraction, so it is difficult to extract the precise features related to facial changes. In order to solve this problem, this paper proposes a facial expression recognition method using feature fusion based on deep learning. The method combines the LBP feature and the features extracted by the CNN convolutional layer into the improved VGG-16 network connection layer by weighting. Finally, the fusion features are sent to the Softmax classifier to obtain the probability of various features, and complete the basic six expression classifications. The experimental results show that the average recognition accuracy of the proposed method on the CK+ and JAFFE datasets is 97.5% and 97.62%, respectively. The recognition results obtained by the fusion features are significantly superior to that of single feature recognition. Compared with other methods, this method can effectively improve the accuracy of expression recognition and is more robust to illumination changes.