Abstract:

As the rapid growing of massive data, the storage method of cluster file system is developing from replication to Erasure Code. The storage system based on Erasure Code can provide higher reliability with less storage overhead. However, in the procedure of storing original data and the coded data, storage based on Erasure Code faces more disk I/O conflicts and unbalanced load, which jeopardizes the throughput of the system. Specially, there is a tradeoff between data consistency and data caching in the storage system based on Erasure Code when writing back the parity. And there are limitations for the use of non-datacaching machanism and alldatacaching machanism in the cluster file system. For these two issues, the paper proposed an I/O optimization method, which includes the data layout machanism balancing the load and the parity consistency machanism. In the cluster file system ECFS we developed base on Erasure Code, the throughput of the system can be improved by 95.53% after exploiting the I/O optimization.

Key words: cluster file system;massive storage;erasurecode;data placement;parity caching;consistency