At present, the fatigue early warning algorithm mostly adopts real-time monitoring and alarming, which has great security risks in the high-speed driving environment. In view of the temporal correlation of human fatigue state, this paper proposes an early warning algorithm based on spatiotemporal feature extraction of facial motion. Firstly, a convolutional neural network with spatial transformation structure is constructed to identify the face region and detect and mark the facial feature points. Secondly, a spatiotemporal feature extraction network is established, and the real-time acquired facial image feature sequence is used to predict and output the future image sequence. Finally, in the outputted image sequence, the comprehensive states of eyes and mouth are used to determine whether a fatigue warning is issued or not.Experimental results show that, under the condition that the image is acquired at 15 frames per second and the 30 frames in the future 2 seconds are predicted, the proposed algorithm can achieve the accuracy of more than 90% when issuing a fatigue warning 26 frames (about 1.5 seconds) in advance, and the accuracy of 97% when issuing a fatigue warning 15 frames (1 second) in advance. Under the average speed of 100 km/h in China's expressways, it is equivalent to an early warning of 40 meters in advance, which can further reduce the occurrence of traffic accidents.