Aiming at the problem of road detection in aerial images, a road extraction algorithm based on prediction and residual refinement networks is proposed. Firstly, the prediction network makes initial predictions. In order to improve the refinement ability of the segmentation network and learn higher- level road features, the dilated convolution and multi-kernel pooling modules are introduced in the prediction networks. Secondly, the residual refinement network will further refine the output of the prediction network and improve the ambiguity of the prediction network results. In addition, considering the small proportion of road pixels in aerial images, the network also combines binary cross entropy, structural similarity, and intersection over union loss functions to reduce road information loss. The experimental results on the Massachusetts road dataset show that the precision, recall, F value and accuracy reaches 99.3%, 95.7%, 97.3% and 95.1%, respectively. The intersection over union and structural similarity also reaches 94.8% and 84.3%, respectively. Compared with other algorithms, this proposed algorithm has certain application value.