Skin is the largest organ in the human body, and skin color is more convenient and stable than other biological properties of the human body. Therefore, it is very meaningful to design an effective skin color grading system. In this paper, the skin color grading system is divided into two parts: skin segmentation and skin color grading. For the skin segmentation,a multi-scale feature fusion network is built under the framework of the generative adversarial network. Compared with the traditional semantic segmentation networks, the proposed segmentation model makes full use of the information of each layer's feature map. In the face color grading experiment, the SVM classifier and the BP neural network are trained with 1 000 images in the normalized rgb, HSV, and Lab color spaces. 128 skin images are used as test sets, and the correct rate is between 73% and 76%. Then,the color features are combined with the LBP texture features of the skin region to do the learning. The correct rate of the SVM classifier is 85%, and the correct rate of the BP neural network is 91%.