Aiming at the problem that the current clothing classification algorithm has general accuracy in solving the multi-category clothing classification, this paper proposes an optimized clothing classification algorithm based on residual convolutional neural network, and uses the following three optimization methods in the network: 1) The orders of batch normalized layer (BN), activation function (Relu) and convolution layer in the network are adjusted; 2) A parallel pooling structure of "pool layer + convoluted layer" is adopted; 3) The full connection layer is replaced by the global mean pooling layer. Experiments on the multi-category large-scale clothing data set (DeepFashion) provided by the multimedia laboratory, the Chinese university of Hong Kong and the standard data set CIFAR-10 show that the proposed network model is superior to VGGNet and AlexNet in image processing speed and classification accuracy, and obtains the best classification accuracy on DeepFashion data set so far. The network can also be better applied to target detection and image segmentation.