Abstract: Aiming at the problems of low accuracy and weak robustness of dynamic gesture recognition in complex environment, a dynamic gesture recognition algorithm based on multimodal fusion, named TF-MG, is proposed. TF-MG combines the depth information and hand skeleton information, extracts the corresponding feature information using two different networks, and then fuses the extract- ed features into the classification network to realize dynamic gesture recognition. According to the depth information, the motion history image method is used to compress the motion trajectory into a single frame image, and the feature is extracted by MobileNetV2. According to the hand skeleton information, DeepGRU composed of gated recurrent units is used to extract features from the hand skeleton information. The experimental results show that, on DHG-14/28 dataset, the recognition accuracy of 14 kinds of hand gestures reaches 93.29%, and that of 28 kinds of hand gestures reaches 92.25%. Compared with other algorithms, it achieves higher recognition accuracy.

Key words: multimodality, dynamic gesture recognition, gated recurrent unit, convolutional neural network