Given that the randomly selected input weight matrix and hidden layer deviation of the traditional extreme learning machine (ELM) lead to inaccurate diagnosis of breast cancer, we propose an ELM optimization method based on an improved fish swam optimization algorithm(AFSA-ELM). In the diagnosis of breast cancer, the advantages of the ELM including fast learning speed, good generalization performance and low adjustment parameters are fully utilized. In addition, the input layer weights and the hidden layer deviation of the ELM are optimized by the improved fish swarm algorithm, and the nonlinear mapping relationship between the breast tumor and the 10 eigenvectors extracted from the breast tumor sample data is constructed. Compared with the ELM、LVQ and BP neural network, the simulation results show that the proposed method has higher recognition accuracy, lower false negative rate and faster learning speed in breast tumor diagnosis.