Abstract: With the increase of transmission rate, transmission length and structure complexity of high-speed channel, channel modeling technology becomes more complex and difficult. This paper proposes a novel method by combining the popular machine learning method with high-speed channel. A large number of analog data are collected, and deep neural network (DNN) and recurrent neural network (RNN) methods are used to model the channel. Once the model is trained successfully, the eye diagram of the output signal can be predicted by the simulation model, and the signal integrity can be evaluated and analyzed quickly and accurately. In addition, in the high-speed channel, the serious interference and attenuation of the signal limits the transmission distance and transmission rate, which brings difficulties to the test and information collection. In order to recover the ideal signal, the high-speed serial link usually contains complex equalization blocks.The least mean square (LMS) algorithm is adopted to effectively eliminate the interference, reduce the bit error rate and improve the transmission rate.

Key words: high speed channel, deep neural network(DNN), recurrent neural network(RNN), eye diagram, machine learning, equalization, least mean square(LMS)