YOLOv3 is a one-stage target detection algorithm that does not generate Region Proposal Network (RPN) to extract target information. Compared with the two-stage target detection algorithm, it has faster detection speed. However, the existing algorithms has the problems of low accuracy and missed detection in small target detection. Therefore, based on YOLOv3 algorithm, a detection method of training set optimization and layer processing is proposed. Firstly, K-means algorithm is used to cluster the standard dataset VOC2007+2012 and the self-built behavior dataset, so as to get the anchor size that fits the training size of the dataset. Then, The training is carried out by adjusting the training parameters and selecting a reasonable labeling method. Finally, the input image is processed by layer and the target is detected. The experimental results show that the average accuracy (mAP) of VOC2007 verification set is improved by 1.4% after clustering analysis, the problem of small feeling field on the higher convolution layer is effectively solved during the detection process of the original algorithm, and the accuracy of YOLOv3 algorithm is improved when detecting small target objects and reducing the missed detection rate.