关键词: automatic generation of medical image reports, federated learning, multimodal large model fine-tuning, medical data sharing, patient privacy protection

Abstract: Accurate and efficient automatic generation of medical image reports has become the key to improve diagnosis and treatment efficiency and alleviate the shortage of medical resources. At present, the research on automatic generation of medical image reports is mainly in English, which cannot meet the clinical needs of China, and there are conflicts between medical data sharing and patient privacy protection and high training cost of new models. To this end, we proposed a federated multimodal Chinese medical image report automatic generation model, which uses federated learning as the framework, the multimodal large model VisualGLM-6B as the base, and low-rank adaptive large model fine-tuning technology, which allows healthcare organisations to fine-tune to optimise the model performance without exposing the local data. Meanwhile, we designed a federated adaptive algorithm for dynamically calculating model aggregation coefficients for different institutions to improve the generalisation performance of the model. The experimental results show that the BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR and ROUGE metrics of this paper's model on the internal test set of MIMIC-CXR are improved by 18.06%, 17.85%, 11.37%, 1.61%, 13.44%, and 5.06%, respectively, compared to the baseline XrayGLM, and that the performance of the model on the external test dataset OPEN-I also shows good generalisation performance.