• 中国计算机学会会刊
  • 中国科技核心期刊
  • 中文核心期刊

Computer Engineering & Science

Previous Articles     Next Articles

A parallel method for ETL process
based on agent and activity priority

CHEN Gang,DU Xin-lin,ZENG Si-feng,AN Bao-ran   

  1. (Institute of Computer Application,China Academy of Engineering Physics,Mianyang 621900,China)
  • Received:2015-11-10 Revised:2016-07-01 Online:2017-09-25 Published:2017-09-25

Abstract:

ETL is the essential step to obtain high-quality data for data warehouse, and plays an important role in the construction and implementation of data warehouse. Aiming at the deficiency of traditional serial ETL process, we propose a parallel method for ETL based on agent and activity priority. This method first calculates the priority of each ETL activity and then utilizes the agent theory and multi-thread computing techniques to achieve parallel execution of independent ETL activities with the same priority. Experimental results show that this method achieves high speedup when the data volume is large and improves the efficiency of ETL process.
 

Key words: agent, activity priority, ETL, parallel