关键词: 神经机器翻译, 预训练语言模型, 词嵌入, 汉语-越南语 ,  ,

Abstract: Neural machine translation is the current mainstream machine translation method. However, in low-resource machine translation tasks such as Chinese-Vietnamese, the effect of neural machine translation is not ideal due to the small scale of bilingual parallel corpus. Considering that the pre-trained language model contains rich language information, incorporating the pre-trained language model into a neural machine translation system may have a positive effect on low-resource machine translation. Therefore, this paper proposes a low-resource neural machine translation method that combines the dual representation of BERT pre-training language model and word embedding. The pre-training language model and word embedding are used to learn the representation of the source language sequence. The connection between the two representations are established through the attention mechanism. The splic- ing operation is performed  to obtain the dual representation vector. Through the linear transformation and self-attention mechanism, the word embedding representation and the pre-trained language model representation are fully adaptively fused together to obtain a sufficient representation of the input text, thereby improving the performance of the neural machine translation model. The translation experiment on the Chinese-Vietnamese language pair shows that, compared with the benchmark system, the method obtains an increase of 1.99 BLEU in the 127k-scale Chinese-Vietnamese training data, and an increase of 4.34 BLEU in the 70k-scale Chinese-Vietnamese training data, which proves that the fusion of BERT pre-training language model and dual representation of word embedding can effectively improve the performance of Chinese-Vietnamese machine translation.

Key words: neural machine translation, pre-trained language model, word embedding, Chinese- Vietnamese