• 中国计算机学会会刊
  • 中国科技核心期刊
  • 中文核心期刊

Computer Engineering & Science

Previous Articles     Next Articles

An HDFS storage optimization strategy based on Cauchy code  

XIE Guojun,SHEN Jiquan,YANG Huanhuan   

  1. (School of Computer Science and Technology,Henan Polytechnic University,Jiaozuo 454000,China)
  • Received:2018-04-26 Revised:2018-07-12 Online:2019-03-25 Published:2019-03-25

Abstract:

With the advent of the big data era, data storage is facing severe challenges. The traditional Hadoop distributed file system (HDFS) has problems such as high storage redundancy and insufficient load balancing. Aiming at these problems, based on Cauchy code, we propose a Cauchy dynamic decentralized storage (CDDS) strategy. For the data blocks in the system, this strategy can generate different storage schemes based on their heat levels while ensuring data availability. For the cold data and hot data in the system, we adopt the Cauchy based erasure code technology to perform singlecopy storage and multicopy storage respectively, which guarantees the reliability of the data and the I/O capability of the system. Test results show that the CDDS strategy reduces data storage space to 75% of the original, and enhances the system’s reliability and load balancing capability.
 

Key words: data storage, Cauchy code, dynamic replica, load balancing