We focus on detecting the efficiency of extreme learning machine (ELM) on action recognition. To overcome the problems of computational complexity and time consumption of online learning and action classification, we propose a new action recognition algorithm (ELM-Cholesky). Firstly, a method based on Cholesky decomposition to seek the calculation of ELM is introduced into the algorithm. Secondly, according to the characteristics of kernel function matrix updates during  online learning, we utilize the partitioned Cholesky decomposition algorithm for online solution to ELM, which realizes online updating of the triangular factor matrix. Finally, we can obtain a new online learning algorithm, called ELM-Cholesky. The new algorithm can make full use of historical training data, reduce the complexity of calculation, and improve action identification accuracy. Moreover, extensive experiments on benchmark database verify the effectiveness of this online learning algorithm.