关键词: 多维索引, 八叉树, 数据稀疏化, 渐进式探索, 大数据可视化

Abstract: Visualizing large-scale scientific data requires high data transmission bandwidth and a large amount of memory. Efficient processing of visualization data poses a significant challenge. To improve the efficiency of scientific visualization, the most common and direct method is to reduce the amount of data that needs to be processed. A novel visualization scheme for large-scale volumes of data is proposed by designing a new data structure called EMRI-Tree as well as a flexible rendering workflow. The characteristics of our scheme can be summarized as follows. Firstly, the proposed EMRI-Tree supports memory-efficient data queries and ROI-data (ROI, region of interest) fetching on large 3D models, thus reducing the memory footprint significantly. Secondly, data blocks at different resolution levels in the EMRI-Tree are stored in a key-value (KV) storage system with variable-length indices, which improves the scalability of storage and the concurrency of reading. Lastly, a prefetching scheme is proposed, which supports progressive rendering based on ray marching to render a more accurate model as  interact. By combining the above optimizations, the proposed scheme facilitates the visualization of large volumes of high-resolution data with limited memory overhead. Evaluating the approach by using 80 GB of synthetic data in 10 simulated read tests. The experimental results demonstrate that the  scheme has the characteristics of 2 000+ QPS (queries per second) and linear growth in memory consumption, making it a robust and memory-efficient solution.  

Key words: multidimensional Indexing, octree, data-thinning, progressive exploration, large data visualization