Abstract:

Based on the idea of bilingual topic model, we analyze similarity of bilingual documents and propose a cross-lingual document similarity calculation method based on bilingual LDA. Firstly we use the bilingual parallel documents to train the bilingual LDA model and then use the trained model to predict the topic distribution of the new corpus. The new corpus's bilingual documents are mapped to the vector space of the same topic.  We use the cosine similarity method and topic distribution combined to calculate the similarity of the bilingual documents of the new corpus. We improve the topic frequency inverse document frequency method from the aspect of the dispersion of in-category and the between-category topic distribution, and utilize the improved method to calculate feature topic weights. Experimental results show that the improved weight calculation method can enhance the recall rate, enable the LDA similarity calculation algorithm not limited to certain categories, and it is reliable.

Key words: bilingual LDA, cross-lingual document similarity calculation, cosine similarity, topic frequency-inverse document frequency