Aiming at the problem that the target window cannot adapt to target scale change in the traditional spatio-temporal context tracking (STC) algorithm, which leads to inaccurate targeting, we propose a target tracking algorithm based on joint optimization of improved STC and SURF features (STC-SURF). Firstly, the feature points of two adjacent frames are extracted and matched by the speeded up robust feature (SURF) algorithm, and the random sample consensus (RANSAC) matching algorithm is used to eliminate the mismatch and increase the matching precision. Furthermore, the target window is adjusted according to the change of the matching feature points in the two frames of the image, and then outputted. Finally, the update method of the model of the STC algorithm is optimized to increase the accuracy of the tracking result. Experimental results show that the STC-SURF algorithm can adapt to the target scale change, and the target tracking success rate is better than the target-learning detection (TLD) algorithm and the traditional STC algorithm.