基于HPC环境的深度学习并行优化

计算机工程与科学

基于HPC环境的深度学习并行优化

陈孟强，颜子杰，叶彦，吴维刚

（中山大学数据科学与计算机学院，广东广州 510006）

收稿日期:2018-07-20 修回日期:2018-09-27 出版日期:2018-11-26 发布日期:2018-11-25
基金资助:
国家重点研发计划（2016YFB0200404）；国家自然科学基金（U1711263）

Parallel optimization for deep learning

based on HPC environment

CHEN Mengqiang，YAN Zijie，YE Yan，WU Weigang

（School of Data and Computer Science,Sun Yatsen University,Guangzhou 510006,China）

Received:2018-07-20 Revised:2018-09-27 Online:2018-11-26 Published:2018-11-25

摘要/Abstract

摘要：

深度学习已被广泛应用于各领域，尤其是大数据分析，然而深度学习所需的计算越来越复杂，规模也越来越大。为了加速大规模深度学习的训练，学术界已经提出了各种分布式并行训练协议。设计了一种新的异步训练协议——加权异步并行协议（WASP），以更有效的方式更新神经网络参数。WASP的核心是对“陈旧梯度”的处理，即基于参数版本号来衡量梯度陈旧性并减少陈旧梯度对参数的影响。此外，通过周期性强制同步模型参数，WASP结合了同步和异步并行协议的优点，可以快速收敛并提高模型训练速度。我们在天河二号超级计算机上使用两个经典卷积神经网络LeNet5和ResNet101进行实验，结果表明，WASP可以比现有异步并行训练协议取得更高的加速比、更稳定的收敛。

关键词: 深度学习, 分布式并行, 天河二号, 参数服务器, 陈旧度

Abstract:

Deep learning technology has been widely applied for various purposes, especially big data analysis. However, computation demand for deep learning is getting more complex and larger. In order to accelerate the training of largescale deep networks, various distributed parallel training protocols have been proposed. We design a novel asynchronous training protocol, called weighted asynchronous parallel protocol (WASP), to update neural network parameters in a more effective way. The core of WASP is how to deal with “gradient staleness”, a parameter version number based metric to weight gradients and reduce the influence of the stale gradient on parameters. Moreover, by periodic forced synchronization of model parameters, the WASP combines the advantages of synchronous and asynchronous training models and can speed up training with a rapid convergence rate. We conduct experiments on the Tianhe2 supercomputing system using two classical convolutional neural networks, LeNet5 and ResNet101, and the results show that the WASP can achieve a much higher speedup and a more stable convergence than existing asynchronous parallel training protocols.

Key words: deep learning, distributed parallelization, Tianhe2, parameter server, staleness

陈孟强，颜子杰，叶彦，吴维刚. 基于HPC环境的深度学习并行优化[J]. 计算机工程与科学.

CHEN Mengqiang，YAN Zijie，YE Yan，WU Weigang.

Parallel optimization for deep learning

based on HPC environment

[J]. Computer Engineering & Science.

参考文献

［1］Lecun Y,Bengio Y,Hinton G.Deep learning［J］.Nature,2015,521(7553):436444.

［2］Li X,Zhang G,Huang H H,et al.Performance analysis of GPUbased convolutional neural networks［C］∥Proc of IEEE International Conference on Parallel Processing,2016:6776.

［3］Li M.Scaling distributed machine learning with the parameter server［C］∥Proc of ACM International Conference on Big Data Science and Computing,2014:3.

［4］Smola A,Narayanamurthy S.An architecture for parallel topic models［J］.VLDB Endowment,2010,3(12):703710.

［5］Ahmed A,Aly M,Gonzalez J,et al.Scalable inference in latent variable models［C］∥Proc of ACM International Conference on Web Search and Data Mining,2012:123132.

［6］Li M,Zhou L,Yang Z,et al.Parameter server for distributed machine learning［C］∥
Proc of ACM NIPS Workshop on Big Learning,2013:110.

［7］Dean J,Corrado G S,Monga R,et al.Large scale distributed deep networks［C］∥Proc of International Conference on Neural Information Processing Systems,
2012:12231231.

［8］Ho Q,Cipar J,Cui H,et al.More effective distributed ML via a stale synchronous parallel parameter server［C］∥Proc of
International Conference on Neural Information Processing Systems, 2013:12231231.
［9］Zhang H,Hu Z,Wei J,et al.Poseidon:A system architecture for efficient GPUbased deep learning on multiple machines［J］.Computer Science,2015,arXiv:1512.06216.

［10］Abadi M,Barham P,Chen J,et al.TensorFlow:A system for largescale machine learning［C］∥Proc of the 12th ACM USENIX Conference on Operating Systems Design and Implementation,2016:265283.

［11］Wang M,Xiao T,Li J,et al.Minerva:A scalable and highly efficient training platform for deep learning［C］∥Proc of NIPS Workshop on Distributed Machine Learning & Matrix Computations,2014:19.

［12］Valiant L G.A bridging model for parallel computation［J］.COMMUNICATIONS of the ACM,1990,33(8):103111.

［13］Mccoll W F.Bulk synchronous parallel computing［M］.Oxford:Oxford University Press,1995.

［14］Cui H,Cipar J,Ho Q,et al.Exploiting bounded staleness to speed up big data analytics［C］∥Proc of Usenix Technical Conference,2014:3748.

［15］Zhang W,Gupta S,Lian X,et al.Stalenessaware asyncSGD for distributed deep learning［C］∥
Proc of International Joint Conference on Artificial Intelligence,2016:23502356.

［16］Lecun Y,Bottou L,Bengio Y,et al.Gradientbased learning applied to document recognition［J］.Proceedings of the IEEE,
1998,86(11),22782324.

［17］He K,Zhang X,Ren S,et al.Deep residual learning for image recognition［C］∥Proc of IEEE Conference on Computer Vision and Pattern Recognition
,2016:770778.

[1]	马思远, 焦佳辉, 任晟岐, 宋伟. 基于注意力机制的城市多元空气质量数据缺失值填充[J]. 计算机工程与科学, 2023, 45(08): 1354-1364.
[2]	易啸, 马胜, 肖侬. 深度学习加速器在不同剪枝策略下的运行优化[J]. 计算机工程与科学, 2023, 45(07): 1141-1148.
[3]	康宇晗, 时洋, 陈照云, 文梅. 面向迈创+MatrixZone异构系统的深度学习编程框架[J]. 计算机工程与科学, 2023, 45(07): 1149-1158.
[4]	刘浩翰, 孙铖, 贺怀清, 惠康华. 基于改进YOLOv3的金属表面缺陷检测[J]. 计算机工程与科学, 2023, 45(07): 1226-1235.
[5]	田秀霞, 刘正, 刘秋旭, 李浩然. 一种改进Faster R-CNN的图像篡改检测模型[J]. 计算机工程与科学, 2023, 45(06): 1030-1039.
[6]	濮子俊, 张寿明. 基于特征融合与Transformer模型的声音事件定位与检测算法研究[J]. 计算机工程与科学, 2023, 45(06): 1097-1105.
[7]	邓姗姗, 黄慧, 马燕. 基于改进Faster R-CNN的小目标检测算法[J]. 计算机工程与科学, 2023, 45(05): 869-877.
[8]	霍爱清, 张书涵, 杨玉艳, 胥静蓉, 王泽文. 密集交通场景中改进YOLOv3目标检测优化算法[J]. 计算机工程与科学, 2023, 45(05): 878-884.
[9]	方雪杉, 杨云飞, 冯松. 基于多尺度多模态学习的光球亮点曲线轨迹段检测方法研究[J]. 计算机工程与科学, 2023, 45(05): 885-894.
[10]	黄星威, 陈曦, 张塑凡. 改进特征金字塔的小目标深度学习模型[J]. 计算机工程与科学, 2023, 45(04): 734-742.
[11]	董佩杰, 牛新, 魏自勉, 陈学晖. 单次神经网络结构搜索研究综述[J]. 计算机工程与科学, 2023, 45(02): 191-203.
[12]	强梓林, 刘建国, 刘云峰, 卫栋, 强彦. 基于时域-频域哈希编码的电网图像检索方法[J]. 计算机工程与科学, 2022, 44(10): 1877-1884.
[13]	何平, 李刚, 李慧斌, . 基于深度学习的视频异常检测方法综述[J]. 计算机工程与科学, 2022, 44(09): 1620-1629.
[14]	刘从军, 徐佳陈, 肖志勇, 柴志雷. 基于深度学习的心脏核磁共振图像自动分割算法[J]. 计算机工程与科学, 2022, 44(09): 1646-1654.
[15]	何涛, 施慧莉, 李大亮. 基于深度学习的SAR目标识别DSP设计[J]. 计算机工程与科学, 2022, 44(08): 1357-1363.