云计算环境下基于数据关联度的海洋监测大数据布局策略

摘要/Abstract

摘要：

海洋监测数据是具有强数据关联的大数据，如何高效地进行数据布局，是制约其有效管理和应用的关键问题之一。在云计算环境下，针对海洋监测大数据的特点，提出了一种基于数据关联度的海洋监测大数据布局策略。在保证数据中心存储均衡的情况下，综合考虑了监测任务、监测点和监测数据之间的关联，建立了海洋监测点间的关联度、监测数据间的关联度和监测数据全局关联度，从三个角度对海洋监测大数据进行布局，使得同一数据中心内的数据具有较高的关联度。通过实验分析，该方法降低了用户访问海洋监测大数据的响应时间，为海洋监测大数据提供了一种有效的布局策略。

关键词: 海洋监测大数据, 云计算环境, 数据布局, 关联度

Abstract:

Marine monitoring data is big and has strong data dependency. How to design a fast placement strategy and control the marine data efficiently becomes more and more important for the management and the application of big marine monitoring data. In cloud computing environments, according to the characteristics of big marine monitoring data we propose a placement strategy based on data dependency. Under the condition of guaranteeing the load balancing of data centers, the dependencies among monitoring tasks, monitoring points and monitoring data are considered comprehensively. The dependency of monitoring points, the dependency of monitoring data and the overall dependency of monitoring data are established.According to the three kinds of dependencies, the marine data placement is done so as to make the big marine monitoring data in the same data center with higher dependency.Experimental results show that our strategy reduces the response time of ocean monitoring data access and provides an effective placement strategy for ocean monitoring data.

Key words: big marine monitoring data;cloud computing environment;data placement;dependency

黄冬梅1,随宏运1,贺琪1,赵丹枫1,杜艳玲1,苏诚2. 云计算环境下基于数据关联度的海洋监测大数据布局策略[J]. J4, 2015, 37(11): 1989-1996.

HUANG Dongmei1,SUI Hongyun1,HE Qi1,ZHAO Danfeng1,DU Yanling1,SU Cheng2. A marine monitoring big data placement strategy in cloud
computing environment based on data dependency [J]. J4, 2015, 37(11): 1989-1996.

参考文献

［1］Petes L,Diamond J,Fisher B,et al. Ocean management challenges,adaptation approaches,and opportunities in a changing climate［M］∥Oceans and Marine Resources in a Changing Climate. Washington:Island Press/Center for Resource Economics,2013:140155.
［2］Park K,Lee D H,Woo Y,et al. Reliability and performance enhancement technique for SSD array storage system using RAID mechanism［C］∥Proc of the 9th International Symposium on Communications and Information Technology,2009:140145.
［3］Caron S,Giroire F,Mazauric D,et al. P2P storage systems:Study of different placement policies［J］. PeertoPeer Networking and Applications,2014,7(4):427443.
［4］Herlihy D R,Matula S P,Andreasen C. Swath mapping data management within the national iceanic and atmospheric administration［J］. The International Hydrographic Review,2015,65(2):1.
［5］Kameda H,Li J,Kim C,et al. Optimal load balancing in distributed computer systems［M］. Incorporated:Springer Publishing Company,2011.
［6］Song Jie,Li Tiantian,Yan Zhenxing,et al.Loadbalanced data layout approach in dataintensive computing［J］. Journal of Beijing University of Posts and Telecommunications,2013,36(4):7680. (in Chinese)
［7］Zhang Tiantian, Cui Lizhen. A data placement strategy based on relaxation and reconstruction for scientific workflow applications［J］. Journal of Computer Research and Development,2013,50(suppl):7176. (in Chinese)
［8］Chen Tao,Xiao Nong,Liu Fang,et al. Clusteringbased and consistent Hashingaware data placement algorithm［J］. Journal of Software,2010,21(12):31753185. (in Chinese)
［9］Liu Jingyu,Zheng Jun,Li Yuanzhang,et al. Hybrid SRAID:An energyefficient data layout for sequential data storage［J］. Journal of Computer Research and Development,2013,50(1):3748. (in Chinese)
［10］Yao W,Lu L. A selection algorithm of service providers for optimized data placement in multicloud storage environment［M］∥Intelligent Computation in Big Data Era. Berlin:Springer Berlin Heidelberg,2015:8192.
［11］Calder B,Wang J,Ogus A,et al. Windows azure storage:A highly available cloud storage service with strong consistency［C］∥Proc of the 23rd ACM Symposium on Operating Systems Principles,2011:143157.
［12］Maia G,Guidoni D L,Viana A C,et al. A distributed data storage protocol for heterogeneous wireless sensor networks with mobile sinks［J］. Ad Hoc Networks,2013,11(5):15881602.
［13］Yan Lin,Xing Jing,Huo Zhigang,et al. A survey on storage architectures and core algorithms for big data management on new storages［J］. Computer Engineering & Science, 2013,35(5):2027. (in Chinese)
［14］Wang Yijie,Sun Weidong,Zhong Song,et al. Key technologies of distributed storage for cloud computing［J］. Journal of Software,2012,23(4):962986. (in Chinese)
［15］Zheng Pai,Cui Lizhen,Wang Haiyang,et al. A data placement strategy for dataintensive applications in cloud［J］. Chinese Journal of Computers,2010,33(8):14721480. (in Chinese)
［16］Ma Fei. Data placement strategy research for scientific workflow in hybrid cloud computing［D］. Hefei:Anhui University,2014. (in Chinese)
［17］Zhang Peng,Wang Guiling,Xu Xuehui. A data placement approach for workflow in cloud［J］. Journal of Computer Research and Development,2013,50(3):636647. (in Chinese)
［18］Xie Pengfei,Sui Weina,Tao Guanfeng,et al. Cloud computing in the marine environment monitoring［J］. Marine Environmental Science,2013,32(4):576580. (in Chinese)
［19］Majeti D,Barik R,Zhao J,et al. Compilerdriven data layout transformation for heterogeneous platforms［C］∥Proc of EuroPar 2013:Parallel Processing Workshops,2014:188197.
［20］Wildani A,Miller E L,Adams I F,et al. PERSES:Data layout for low impact failures［C］∥Proc of 2014 IEEE 22nd International Symposium on Modelling,Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS),2014:7180.
［21］Zhao Wei, Zhuo Wei, Li Zhanbo,et al. A novel data exchange architecture based on cloud computing［J］. Computer Engineering & Science,2013,35(8):1519. (in Chinese)
［22］Wei L,Zhu H,Cao Z,et al. Security and privacy for storage and computation in cloud computing［J］. Information Sciences,2014,258(10):371386.
［23］Heath M A,Coker K T,Viraraghavan P. Data storage device overlapping host data transfer for a write command with intercommand delay:U.S. Patent 8,631,188［P］. 20140114.
［24］Liu C,Chen J,Yang L T,et al. Authorized public auditing of dynamic big data storage on cloud with efficient verifiable finegrained updates［J］. IEEE Transactions on Parallel and Distributed Systems,2014,25(9):22342244.
［25］Jenkins J,Zou X,Tang H,et al. RADAR:Runtime asymmetric dataaccess driven scientific data replication［C］∥Proc of the 28th International Supercomputing Conference, ISG’14,2014:296313.
［26］Yang K,Jia X. An efficient and secure dynamic auditing protocol for data storage in cloud computing［J］. IEEE Transactions on Parallel and Distributed Systems,2013,24(9):17171726.
［27］Barsoum A F,Hasan A. Enabling dynamic data and indirect mutual trust for cloud computing storage systems［J］. IEEE Transactions on Parallel and Distributed Systems,2013,24(12):23752385.
［28］Anjos J C S,Carrera I,Kolberg W,et al. MRA++:Scheduling and data placement on MapReduce for heterogeneous environments［J］. Future Generation Computer Systems,2015,42:2235.
［29］McCormick Jr W T,Schweitzer P J,White T W. Problem decomposition and data reorganization by a clustering technique［J］. Operations Research,1972,20(5):9931009.
［30］Song H,Yin Y,Sun X H,et al. A segmentlevel adaptive data layout scheme for improved load balance in parallel file systems［C］∥Proc of the 2011 11th IEEE/ACM International Symposium on Cluster,Cloud and Grid Computing,2011:414423.
［31］Sun X,Shi L,Luo Y,et al. Histogrambased normalization technique on human brain magnetic resonance images from different acquisitions［J］. Biomedical Engineering Online,2015,14(1):73.
附中文参考文献：
［6］宋杰,李甜甜,闫振兴,等. 数据密集型计算中负载均衡的数据布局方法［J］. 北京邮电大学学报,2013,36(4):7680.
［7］张甜甜,崔立真. 基于释放和重构的科学工作流数据布局策略［J］.计算机研究与发展,2013,50(suppl):7176.
［8］陈涛,肖侬,刘芳,等. 基于聚类和一致 Hash 的数据布局算法［J］. 软件学报,2010,21(12):31753185.
［9］刘靖宇,郑军,李元章,等. 混合 SRAID:一种适于连续数据存储的节能数据布局［J］. 计算机研究与发展,2013,50(1):3748.
［13］严林,邢晶,霍志刚,等. 面向海量数据存储的ErasureCode分布式文件系统I/O优化方法［J］. 计算机工程与科学,2013,35(5):2027.
［14］王意洁,孙伟东,周松,等. 云计算环境下的分布存储关键技术［J］. 软件学报,2012,23(4):962986.
［15］郑湃,崔立真,王海洋,等. 云计算环境下面向数据密集型应用的数据布局策略与方法［J］. 计算机学报,2010,33(8):14721480.
［16］马飞. 混合云环境下科学工作流数据布局研究［D］. 合肥：安徽大学,2014.
［17］张鹏,王桂玲,徐学辉. 云计算环境下适于工作流的数据布局方法［J］. 计算机研究与发展,2013,50(3):636647.
［18］解鹏飞,隋伟娜,陶冠峰,等. 云计算与海洋环境监测［J］. 海洋环境科学,2013,32(4):576580.
［21］赵伟,卓伟,李占波,等基于云计算的一种新的数据交换架构［J］. 计算机工程与科学,2013,35(8):1519.

[1]	刘磊,宋家友. 故障样本的多指标集成加权分配方法[J]. 计算机工程与科学, 2019, 41(04): 744-749.
[2]	戴小玲,唐明董,吕赛霞. 基于灰色关联分析的Web服务选择[J]. J4, 2016, 38(02): 297-304.
[3]	魏永山1，张峰1，陈欣1，韩燕波2，孙忠林1. 一种云计算环境下的XML查询数据服务的优化方法[J]. J4, 2013, 35(6): 30-36.
[4]	陈平增齐路刘晓光王刚. 基于面向对象存储系统的对象及数据副本布局方法设计[J]. J4, 2008, 30(11): 140-143.