Abstract: Depending on the large-scale parallel corpus, neural machine translation has achieved great success in some language pairs. Subsequently, unsupervised neural machine translation (UNMT) has partly solved the problem that high quality corpus is difficult to obtain. Recent studies show that cross-lingual language model pretraining can significantly improve the translation performance of UNMT. This method models deep context information in cross-lingual language scenarios by using a large-scale monolingual corpus, and obtains significant results. This paper further explores UNMT based on cross-lingual language pretraining, proposes several improved methods of training model, and compares the performance between UNMT and baseline system on different language pairs. Aiming at the issue of unbalanced initialization of unsupervised NMT parameters when using pre-trained models, this paper proposes a secondary pre-training stage to continue pre-training, and propose to initialize the Cross attention sub-layer with the self-attention sub-layer in unsupervised NMT model. Meanwhile, as back- translation plays a critical role in unsupervised NMT, we propose to use Teacher-Student framework to guide back-translation.Experimental results show that, compared with the baseline system, these methods improve BLEU by 0.8~2.08 percentages at most.

Key words: neural network, neural machine translation, unsupervised, pre-training