Aiming at the problem that the tracking frame of the tracking-learning-detection (TLD) algorithm can cause tracking drift when the target is not rigidly deformed and rotated and the background is cluttered, an improved TLD real-time visual tracking algorithm incorporating the CN algorithm (TLD-CN) is proposed. Firstly, TLD-CN calculates the image saliency of the area within the tracking frame to obtain the threshold of the feature points sampled by the BRISK algorithm.  The appropriate feature points are obtained to establish the rotation and scale normalized descriptors. Secondly, the color features and texture features are fused to track the descriptors in the frame before and after the frame. The optimal similarity matching is performed to obtain a matched feature point set. After the discriminative encoding of the discriminative dictionary for the feature points in the set, the similarity measures are performed with the central pixel points of the CN tracking box and the TLD tracking box to obtain the weighting factors after output box adjustment. The experimental results show that the TLD-CN algorithm measures the weight value of the output box adjustment after fusing the two algorithms through the feature points, and has high accuracy and success in complex tracking scenarios such as target deformation, rotation, background clutter, and fast motion rate. The adaptive updating of the weight coefficients avoids model over-fitting and achieves real-time tracking effect.