• 中国计算机学会会刊
  • 中国科技核心期刊
  • 中文核心期刊

计算机工程与科学

• 高性能计算 • 上一篇    下一篇

空间科学卫星数据快速处理方法

孙小涓1,2,3,石涛2,3,李冰2,3 ,杨晓艳2,3 ,雷斌1,2,3,胡玉新1,2,3   

  1. (1.中国科学院大学,北京 100049;2.中国科学院电子学研究所,北京 100190;
    3.中国科学院空间信息处理与应用技术重点实验室,北京 100190)
  • 收稿日期:2018-01-11 修回日期:2018-03-20 出版日期:2018-08-25 发布日期:2018-08-25

A rapid data processing method
for space science satellites

SUN Xiaojuan1,2,3,SHI  Tao2,3,LI Bing2,3,YANG Xiaoyan2,3,LEI Bin1,2,3,HU Yuxin1,2,3   

  1. (1.University of Chinese Academy of Sciences,Beijing 100049;
    2.Institute of Electronics,Chinese Academy of Sciences,Beijing 100190;
    3.Key Laboratory of Technology in Geospatial Information Processing and Application System,
    Chinese Academy of Sciences,Beijing 100190,China)
  • Received:2018-01-11 Revised:2018-03-20 Online:2018-08-25 Published:2018-08-25

摘要:

针对卫星获取的大规模数据进行快速数据处理一直是空间信息处理系统建设中的关键。面对空间科学卫星全天候观测、探测载荷类型多、处理算法多样带来的数据处理难题,现有基于CCSDS标准格式的数据分析方法,难以满足目前在轨的多颗空间科学卫星数据处理系统在正确性和时效性方面的要求。针对空间科学卫星探测数据处理特点,提出了一种空间科学数据快速处理方法,设计两层联合索引结构,将空间科学大数据处理问题转化为索引表和源包数据单元的处理问题,提高了数据处理效率;采用科学工作流技术设计了数据驱动和业务驱动协同的处理框架,支持多样化的空间科学卫星数据处理流程,各类载荷数据处理任务并行调度。实验结果表明,这种方法处理速度可扩展,内存使用较少,已应用于空间科学卫星地面系统中,取得了良好的效果。
 
 

关键词: 数据处理, 空间科学卫星, 索引结构, 科学工作流

Abstract:

Rapid data processing for largescale data acquired by satellites has always been the key to build spatial information processing systems. Faced with space science satellite problems such as allweather observation, multiple types of detection loads, and various processing algorithms, the existing data analysis methods based on CCSDS
 standard format are difficult to satisfy the current requirements of data processing systems for the onorbit space science satellites in correctness and timeliness. According to the observing data characteristics of the space science satellites, the paper proposes a fast data processing method for space science, designs the twolayer joint index structure that transforms the processing problem from the space science large data into the index tables and the source packet data units, and improves the efficiency of data processing. The paper designs a processing framework using scientific workflow technique, which supports the cooperation between the datadriven processing and the businessdriven processing of space science satellite data, and also supports various data processing workflows and the parallel scheduling of numerous tasks for different satellite payload types. The experimental results show that this method has good scalability and less memory usage, which has been well applied to the spatial information processing system of space science satellites.
 

Key words: data processing, space science satellite, index structure, scientific workflow