关键词: 高性能计算系统软件, 算力网络, 并行传输, RSYNC, 集群安全

Abstract: In the construction of computational networks, how to conduct efficient and reliable data migration between clusters in different regional computing centers is a key research topic. In view of this, this paper designs and implements a high-performance transmission software based on RSYNC, namely SCOW-SYNC. The main research results are as follows: Firstly, SCOW-SYNC adopts the queue and thread pool architecture to optimize the traditional RSYNC. By parallelly establishing multiple TCP connections and parallel transmission, the bandwidth utilization rate is improved. In addition, SCOW-SYNC also supports functions such as automatic large file splitting, dynamic compression, background operation, real-time progress query, and SSH connection pool management. Through testing, SCOW-SYNC can achieve a speedup ratio of 125% to 130% compared with RSYNC. Secondly, in order to improve the security of transmission, this paper proposes a reliable cross-cluster transmission system architecture for computing centers. Data transmission only occurs between "transmission nodes" and is encrypted by "transmission keys", which are dynamically checked, generated, and distributed by the "management node". Finally, this paper integrates SCOW-SYNC into the high-performance computing portal and management platform SCOW, and implements the cross-cluster transmission module of SCOW, so that users can perform high-performance data migration between different clusters through the browser, and deploys it to the cross-cluster environment of Peking University through containerization technology, which improves the production efficiency.

Key words: high performance computing system software, computational network, parallel transmission, RSYNC, cluster security