Abstract:

Parallel I/O of sampled data constrains the operational efficiency of some parallel applications. We design and implement a parallel aggregation I/O method of sampled data. The method first uses the sampled data cache deployed on the client to reduce the number of I/O, and then collects the data to the output process by aggregating the traffic and stores it in the file. To guarantee the storage consistency of sampled data during the longrunning process of parallel programs, we monitor the running state of the application in the JASMIN framework and refresh or restore the data when parallel programs load or restart. During the output process, we use HDF5's chunk I/O to improve I/O efficiency. Test results show that the new method not only has good scalability, but also improves the parallel IO efficiency of sampled data by more than 7.5 times in parallel applications with complex functions such as load balancing or restart.

Key words: scientific computing, sampled data, parallel I/O, performance optimization, aggregated buffer