The explosive growth of information volume in the Internet era and the popularization of deep learning have made traditional generalpurpose computing unable to meet largescale, highconcurrency computing requirements. Heterogeneous computing can release greater computing power for deep learning, satisfy higher performance requirements, and be applied to a wider range of computing scenarios. We design and simulate a complete heterogeneous SoC architecture for deep learning. Firstly, we analyze the computational features of commonly used deep learning algorithms such as GoogleNet, VGG and SSD, and summarize them into a limited number of deep learning common operator classes which are displayed in charts and structure diagrams. At the same time, the pseudo instruction stream at the minimum operator level is generated. Then, based on extracted algorithm features, a hardwareaccelerated AI IP core for deep learning is designed, and a heterogeneous computing SoC architecture is constructed. Finally, experimental verification on the simulation modeling platform shows that the performance to power ratio of the SoC system is greater than 1.5 TOPS/W. The 10channel 1080p 30fps video can be processed frame by frame by the GoogleNet algorithm, and the end-to-end processing time of each frame is no more than 30ms.