关键词: 语言模型, 知识蒸馏, 正则化, 低资源神经机器翻译

Abstract: The lack of large parallel corpora is one of the key issues in low-resource neural machine translation. This paper proposes a neural machine translation method based on language model distillation, which regularizes neural machine translation training by using a monolingual language model. This method introduces prior knowledge contained in the language model to improve translation results. Specifically, we draw on the idea of knowledge distillation, and use the target-side language model (teacher model) trained on rich monolingual data to construct the regularization factor of the low- resource neural machine translation model (student model), allowing the translation model to learn highly generalized prior knowledge from the language model. Unlike traditional monolingual language models that participate in the decoding process, the language model in this method is only used during training and does not participate in the inference stage, so it can effectively improve decoding speed. Experimental results on two low-resource translation datasets of Uyghur-Chinese and Tibetan-Chinese from the 17th national machine translation conference (CCMT2021) show that compared with the current state-of-the-art language model fusion baseline system, BLEU can be improved by 1.42 points  (Tibetan-Chinese) to 2.11 points (Chinese-Uyghu).

Key words: language model, knowledge distillation, regularization, low-resource neural machine translation