In order to solve the problems of strong correlation and high redundancy in the process of color image encryption, a color image encryption algorithm based on cloud model Fibonacci chaotic system and matrix convolution operation is proposed. Firstly, the algorithm permutates the pixel coordinates of the image stitched by the R, G, and B components of the color image. Secondly, the chaotic sequence values are used as the input values of the convolution kernel. Alternative matrix convolution operation is performed on the chaotic sequence values and the pixel values to achieve the pixel transformation. Thirdly, two opposite XOR operations with the cloud model Fibonacci chaotic sequence and the previous adjacent pixel values are done to generate the encrypted image. Finally, simulation experiments show that the histogram of the encrypted image is smoother, the pixel distribution is uniform, the correlation between adjacent image pixels is low, and the average horizontal, vertical and diagonal correlation coefficients of the RGB components of the encrypted image are -0.0010, 0.0016 and 0.0031, respectively. The encrypted image can resist the attack experiments such as differential attack, plaintext attack, noise attack and shear attack. The proposed new encryption algorithm has the characteristics of high encryption security, high anti-interference and strong robustness.