Abstract: Federated learning has garnered significant attention in academia as it enables users to participate in model training without uploading their data. However, federated learning also faces various security challenges from malicious participants, such as Byzantine attacks and label flipping attacks. Existing defense methods exhibit diminished effectiveness under unevenly distributed data. To address these issues, this paper proposes a secure aggregation algorithm  in federated learning based on the one-class support vector machine (OC-SVM). This algorithm extracts appropriate feature parameters using OC-SVM and determines a threshold to separate normal data from anomalous data. Owing to its ability to construct an optimal hyperplane, the algorithm can effectively distinguish between normal and anomalous data. Moreover, it can select a more suitable threshold under different data conditions, demonstrating strong generalization capability and robustness. Through a series of experiments comparing the proposed algorithm with four different defense algorithms, the results show that, in environments with varying proportions of malicious clients and regardless of whether the data distribution is uniform or not, the proposed algorithm can effectively defend against attacks.

Key words: federated learning, Byzantine attack, label flipping attack, one-class support vector machine