关键词: 多模态知识图谱, 多模态实体对齐, 嵌入模型, 自适应融合, 匹配问题

Abstract: To address the issues of information loss during feature fusion and incorrect entity alignment caused by solely focusing on joint entity vectors in multimodal entity alignment, this paper proposes an adaptive fusion  for multimodal entity alignment method(ADMMEA). This method employs FastText, ResNet-152, and GAT models to extract multimodal entity features, obtaining feature representations for entity names, images, and structural data. It utilizes the Bray-Curtis dissimilarity matrix and Levenshtein distance to calculate the similarity between source and target entities, generating distance matrices for each modality. Through an adaptive fusion strategy, the text-image distance matrices are fused and then concatenated with the structural information matrix to obtain the final fused matrix. Leveraging a ranking approach, the fused matrix is sorted in descending order based on similarity scores to achieve multimodal entity alignment. Experimental evaluations are conducted on the ZH-EN, JA-EN, and FR-EN subsets of the DBP15K dataset, and the results are compared with 13 other methods, including JAPE, RDGCN, MOGNN, and MIMEA, etc. The findings demonstrate that ADMMEA achieves Hits@1 scores of 0.985, 0.995, and 0.994 on the ZH-EN,JA-EN and FR-EN datasets, respectively, validating the effectiveness of the ADMMEA.

Key words: multimodal knowledge graphs, multimodal entity alignment, embedding model, adaptive fusion, matching problem