关键词: 预训练模型, 降噪微调, 图注意力机制, 藏文长文本分类

Abstract: In Tibetan long text classification tasks, the issue of long-distance dependencies is particularly prominent. Meanwhile, multilingual pre-trained models exhibit certain biases when handling Tibetan text classification tasks. To address these issues, this paper proposes a Tibetan long text classification method based on the pre-trained language model CINO-Large, which integrates denoising fine-tuning and a graph attention network. Firstly, the In-trust loss function is introduced into CINO-Large to enhance the model’s generalization ability in downstream tasks through task-adaptive loss. Secondly, sliding windows and linear classification are introduced into graph structure modeling to selectively increase document-document edges, thereby improving the feature distinguishability among nodes. Finally, the graph attention mechanism is utilized to capture the importance of different nodes in the graph, completing the Tibetan long text classification task. On the TNCC news long text dataset, the classification accuracy of the proposed method reaches 71.66%. Compared to the pre-trained language model CINO-Large, the accuracy, precision, and F1 score of the proposed model are improved by 1.77%, 2.67% and 2.03%, respectively. For some subclasses that are difficult to classify, the F1 score of the proposed method can be significantly improved by approximately 20%.

Key words: pre-training model, denoising fine-tuning, graph attention mechanism, Tibetan long text classification