• 中国计算机学会会刊
  • 中国科技核心期刊
  • 中文核心期刊

Computer Engineering & Science

    Next Articles

A hybrid big data platform based on
private cloud VMs and bare metals

WANG Yong-kun1,LUO Xuan1,JIN Yao-hui1,2   

  1. (1.Network and Information Center,Shanghai Jiao Tong University,Shanghai 200240;
    2.State Key Laboratory of Advanced Optical Communication System and Network,
    Shanghai Jiao Tong University,Shanghai 200240,China)
  • Received:2017-09-03 Revised:2017-10-30 Online:2018-02-25 Published:2018-02-25

Abstract:

The wide application of big data analysis technology cannot be separated from the support of big data platforms. Building big data platforms is an important demand of many enterprises and institutions. Building a big data platform requires sophisticated, system-wide technologies, and system performance and scalability should be considered especially. With the increasing volume of data, user needs continue to increase, and hence the scale of the planned data platform may not be able to meet the changing needs. Therefore, we design a hybrid big data platform that uses both bare metals and private cloud Virtual Machines (VM) . This takes into account performance and scalability. Because bare metals generally outperform private cloud VMs, the big data platforms built on bare metals generally perform better than the big data platforms built on private cloud VMs. It is very convenient and quick to start the cloud servers in the private cloud, so the computing and storage nodes of the big data platform can be flexibly expanded to the private cloud so as to ensure that the big data platform can still have sufficient processing capacity during the peak period. We implemented this hybrid design in a production environment. Tests in the production environment also demonstrate the effectiveness of this design.
 

Key words: big data, private cloud, big data analysis, big data processing, data platform, Hadoop, Openstack