基于云环境下一种小文件传输策略研究

J4 ›› 2016, Vol. 38 ›› Issue (01): 20-27.

基于云环境下一种小文件传输策略研究

周兰凤1，赵鹏飞1，彭俊杰2

(1.上海应用技术学院计算机科学与信息工程学院,上海 201418;
2.上海大学计算机信息工程与科学学院,上海 200444)

收稿日期:2015-08-19 修回日期:2015-10-29 出版日期:2016-01-25 发布日期:2016-01-25

A transfer strategy for small files
in cloud environment

ZHOU Langfeng1，ZHAO Pengfei1，PENG Junjie2

(1.Institute of Computer Science and Information Engineering,Shanghai Institution of Technology,Shanghai 201418;2.School of Computer Engineering and Science,Shanghai University,Shanghai 200444,China)

Received:2015-08-19 Revised:2015-10-29 Online:2016-01-25 Published:2016-01-25

摘要/Abstract

摘要：

基于互联网的文件传输策略将直接影响到文件传输效率和网络资源的使用效率，这一情况在各种资源集中的云计算环境表现得更为明显，特别是当传输大量小文件时尤为突出。针对这种现状，提出了一种基于打包策略的文件传输策略，给出了文件包优化处理机制，基于该策略，文件传输之前对小文件进行打包处理，然后进行传输，在接收端收到文件时首先进行解包再进行存储，从而减小传输过程中大量的IO操作，提升文件传输效率。大量的实验表明了本文所提策略的正确性，能够极大地提升云环境下小文件的传输效率，提高云资源的利用率。

关键词: 云环境, 小文件传输, 优化处理, 特定策略

Abstract:

File transfer strategies on the Internet directly affect the efficiency of resource utilization and file transfer, and this is more obvious in cloud computing environments where there are all kinds of resources, especially when transferring a large number of small files. Aiming at this problem, we present a transfer strategy for small files based on packing strategy, and an optimization mechanism for file packets as well. We pack small files before they are transferred and stored. Packaged files are unzipped and then stored at the serverend. The transfer process of IO operations is therefore greatly reduced and the efficiency of file transfer is also enhanced. Experimental results verify the correctness of the proposed strategy, which greatly improves the efficiency of small file transfer in cloud environments and resource utilization .

Key words: cloud environment;small file transfer;optimization;specific strategy

周兰凤1，赵鹏飞1，彭俊杰2. 基于云环境下一种小文件传输策略研究[J]. J4, 2016, 38(01): 20-27.

ZHOU Langfeng1，ZHAO Pengfei1，PENG Junjie2. A transfer strategy for small files
in cloud environment [J]. J4, 2016, 38(01): 20-27.

参考文献［10］

［1］	Chen Jie.Cloud storage technology and application ［J］.ZTE Technology,2012,18(6):4851.(in Chinese)
［2］	Mackey G, Sehrish S, Wang Jun.Improving metadata management for small files in HDFS［C］∥Proc of the 2009 IEEE International Conference on cluster computing and Workshops,2010:1.
［3］	Zheng Lining, Wang Yunlan.Research on adaptive file transfer and strategy ［J］.Computer Engineering and Applications,2012,48(36):9599.(in Chinese)
［4］	Ma Tinghuai,Lu Yinghua,Cao Hao.A large number of meteorological data small file adaptive optimization transmission ［J］.Applications Meteorological Journal,2014,25(5):629637.(in Chinese)
［5］	Cao Hao.Large number of small file transferred optimization ［D］.Nanjing:Nanjing University of Information Science and Technology,2013.(in Chinese)
［6］	Bo Dong,Jie Qiu.A novel approach to improving the efficiency of databases and accessing small files on Hadoop:A case study by PowerPoint files［C］∥Proc of International Conference on Services Computing,2010:6572.
［7］	Hong Xusheng,Lin Shiping.MapFile based on the HDFS small file storage efficiency problem ［J］.Computer System Applications,2012,21(11):179182.(in Chinese)
［8］	Yu Si,Gui Xiaolin,Huang Ruwei,et al.A program for improving the storage efficiency of small file storage in cloud storage ［J］.Journal of Xi’an Jiaotong University,2011,45(6):5963.(in Chinese)
［9］	Li Huiying.Research and optimization of small file storage method based on HDFS ［D］.Xi’an:Xi’an Electronic and Science University,2014.(in Chinese)
［10］	Zhao Yuelong,Xie Xiaoling,Cai Yongcai.A new research on the storage access strategy for small file with performance optimization［J］. Journal of Computer Research and Development,2012,49(7):15791586.(in Chinese)
	附中文参考文献：
［1］	陈杰.大数据场景下的云存储技术与应用［J］.中兴通讯技术,2012,18(6):4851.
［3］	郑利宁，王云岚.自适应文件传输与策略研究［J］.计算机工程与应用,2012,48(36):9599.
［4］	陆颖华，马廷淮，曹浩.大量气象数据小文件自适应优化传输［J］.应用气象学报，2014,25(5):629637.
［5］	曹浩.气象数据网格中大量小文件传输优化［D］.南京：南京信息工程大学，2013.
［7］	洪旭升,林世平.基于MapFile的HDFS小文件存储效率问题［J］.计算机系统应用,2012,21(11):179182.
［8］	余思，桂小林，黄汝维，等.一种提高云存储中小文件存储效率的方案［J］.西安交通大学学报,2011,45(6):5963.
［9］	李慧莹.基于HDFS的小文件存储方法的研究与优化［D］.西安：西安电子科技大学,2014.
［10］	赵跃龙,谢晓玲,蔡咏才.一种性能优化的小文件存储访问策略的研究［J］.计算机研究与发展,2012,49(7):15791586.

[1]	周兰凤1，赵鹏飞1，彭俊杰2. 基于云环境下一种小文件传输策略研究[J]. J4, 20160101, 38(01): 20-27.
[2]	仇开1,2，姜瑛1,2. 加权LOF结合上下文判断的云环境中服务运行数据异常检测方法[J]. 计算机工程与科学, 2020, 42(06): 951-958.
[3]	郭宏乐，陈旺虎，马生俊，李新田，乔保民. 一种云环境下科学工作流执行计划的优化方法[J]. 计算机工程与科学, 2019, 41(03): 433-439.