关键词: CLIP, 多模态分类, 图文特征融合, 注意力机制, 多尺度特征提取, 细粒度特征, LSTM

Abstract: To address the diagnostic inaccuracies in traditional Chinese medicine (TCM) nephropathy caused by single-modal data and similar clinical symptoms, this study proposes MLC-CLIP, a novel TCM nephropathy diagnosis model based on CLIP and multimodal data fusion. The model systematically enhances the CLIP framework by: introducing ResNet50 to improve local feature and hierarchical information extraction, designing a multi-scale feature extraction module to strengthen CLIP's image encoding capability, employing LSTM to enhance semantic understanding of TCM clinical texts, and optimizing text encoding structures. Additionally, a weighted feature fusion module and Cross-modal gating attention feature fusion module are developed, combined with a stepwise fusion strategy to optimize multimodal feature integration. Experimental results on the nephropathy dataset from Heilongjiang Academy of Traditional Chinese Medicine demonstrate that the model achieves significant performance improvements in multimodal classification tasks, with accuracy, precision, recall, and F1-score reaching 94.94%, 95.07%, 94.89%, and 94.98%, respectively, representing increases of 10.68%, 9.31%, 9.58%, and 9.45% over the original CLIP model. The results confirm that MLC-CLIP effectively integrates multimodal information from tongue images and clinical texts, demonstrating enhanced discriminative capability for complex cases with similar symptoms, thus providing reliable decision support for TCM nephropathy diagnosis.

Key words: CLIP, multimodal classification, image-text feature fusion, attention mechanism, multi-scale feature extraction, fine-grained features, LSTM