Abstract:

Meteorological data is a typical non-structure data, which reaches dozens of TBs per day. Parsing, storage and access mode based on RDBMS and file systems become the bottleneck of weather forecast data processing system. To fulfill fast and in time queries of realtime data of the users of national weather forecast platform MICAPS’, we depict a stable 7*24 distributed data parsing system, supporting a realtime parsing system containing dozens of TBs per day. According to the multidimension model and the user behaviors of meteorological data, using nonrelational keyvalue DDBMS, we design and implement a high performance massive data storage system. Experiments prove that the proposed real time data parsing system and the massive data storage system based on non-relational key-value DDBMS can meet storage, query and applications requirements of massive meteorological data. This system is also the core system of China weather forecast data flow, possessing excellent functions and performance.

Key words: multi-dimension data;meteorological data;distributed;parse;storage;MICAPS