Because the traffic safety hazards are becoming more serious in the current life, it has certain practical significance to detect abnormal vehicles in scenes such as pedestrian streets and campuses where vehicles are prohibited from traveling. Aiming at the problem of ghost and cavity in the background model built by mixed Gaussian, an abnormal vehicle detection based on mixed Gaussian modeling of SSIM structural similarity is proposed. The similarity between two image pixels is calculated by SSIM. The secondary background modeling is carried out after the Gaussian modeling, and the exponential function is introduced to optimize the weight update process in the Gaussian modeling process, which improves the update speed. The graphical handle function is used to optimize the connected domain method to detect the abnormal vehicle in the foreground area, it is possible to detect the abnormal vehicle and the labeling frame is closer to the shape of the vehicle. Experimental results  of 580 images segmented by video show that the detection rate can reach 90.3%.