多源图像融合方法的研究综述

摘要/Abstract

摘要： 由于成像机理不同，多源图像有本质区别，使得其在融合过程中存在差异。在参阅了大量中外文献的基础上，对融合方法进行分类，并重点论述了各类融合方法的融合过程和典型算法，详细阐述了其关键技术。同时，深入评述了当前的评价指标和分类。最后，结合关键技术的影响因素和技术的发展状况，从数据特征、时间效率、信息提取、评估角度和方法的普适性5个方面对融合图像领域的未来发展趋势进行了展望。

关键词: 图像融合, 多源图像, 融合方法, 评价指标

Abstract: Due to the different imaging mechanisms, the multi-source images are essentially diffe- rent, which makes the fusion process different. Based on a large number of Chinese and foreign literature, this paper classifies fusion methods, and focuses on the fusion processes and typical algorithms of various fusion methods, and elaborates its key techniques in detail. At the same time, an in-depth review of the current evaluation indicators and classifications is carried out. Finally, combining the influencing factors of key techniques and the technology development status, the future development trend of the fusion image field is prospected from the five aspects such as data characteristics, time efficiency, information extraction, evaluation angles and universal applicability of the method.

Key words: image fusion, multi-source image, fusion method, evaluation indicator

张丽霞, 曾广平, 宣兆成. 多源图像融合方法的研究综述[J]. 计算机工程与科学, 2022, 44(02): 321-334.

ZHANG Li-xia, ZENG Guang-ping, XUAN Zhao-cheng. A survey of fusion methods for multi-source image[J]. Computer Engineering & Science, 2022, 44(02): 321-334.

参考文献［78］

［1］	Piella G. A general framework for multiresolution image fusion:From pixels to regions［J］. Information Fusion, 2003,4(4):259-280.
［2］	Li S T,Yang B,Hu J. Performance comparison of different multi-resolution transforms for image fusion［J］. Information Fusion, 2011,12(2):74-84.
［3］	www.imagefusion.org.
［4］	Zhang L X,Zeng G P,Wei J J.Adaptive region-segmentation multi-focus image fusion based on differential evolution［J］. International Journal of Pattern Recognition & Artificial Intelligence,2019,33(3):No 3.
［5］	Zhang L X,Zeng G P,Wei J J,et al. Multi-modality image fusion in adaptive-parameters SPCNN based on inherent characteristics of image［J］. IEEE Sensors Journal,2020,20(20):11820-11827.
［6］	Li S,Kang X,Fang L,et al. Pixel-level image fusion:A survey of the state of the art［J］. Information Fusion, 2017,33(C):100-112.
［7］	Kong J, Zheng K,Zhang J,et al. Multi-focus image fusion using spatial frequency and genetic algorithm［J］. International Journal of Computer Science and Network Security,2008,8(2):220-224.
［8］	Aslantas V,Kurban R. Fusion of multi-focus images using differential evolution algorithm［J］. Expert Systems with Applications,2010,37(12):8861-8870.
［9］	Eltoukhy H A,Kavusi S. A computationally efficient algorithm for multi-focus image reconstruction［J］. Proceedings of SPIE—The International Society for Optical Engineering,2003,5017:332-341.
［10］	Wan T,Zhu C,Qin Z. Multifocus image fusion based on robust principal component analysis［J］. Pattern Recognition Letters,2013,34(9):1001-1008.
［11］	Mitianoudis N,Stathaki T. Pixel-based and region-based image fusion schemes using ICA bases［J］. Information Fusion,2007,8(2):131-142.
［12］	Jiang Y,Wang M. Image fusion with morphological component analysis［J］. Information Fusion,2014,18:107-118.
［13］	He K, Sun J,Tang X. Guided image filtering［J］. IEEE Transactions on Software Engineering,2013,35(6):1397-1409.
［14］	Li S T,Kang X,Hu J. Image fusion with guided filtering［J］. IEEE Transactions on Image Processing,2013,22(7):2864-2875.
［15］	Zhu J,Jin W Q,Li L,et al. Multiscale infrared and visible image fusion using gradient domain guided image filtering［J］. Infrared Physics & Technology,2018,89:8-19.
［16］	Zhang Y X,Wei W,Yuan Y T. Multi-focus image fusion with alternating guided filtering［J］. Signal,Image and Video Processing,2019,13:727-735.
［17］	Balasubramaniam P,Ananthi V P. Image fusion using intuitionistic fuzzy sets［J］. Information Fusion,2014,20:21-30.
［18］	Sun J,Zhu H,Xu Z,et al. Poisson image fusion based on Markov random field fusion model［J］. Information Fusion,2013,14(3):241-254.
［19］	Burt P J,Adelson E H. The Laplacian pyramid as a compact image code［J］. Readings in Computer Vision,1987,31(4):671-679.
［20］	Toet A,Ruyven J J V,Valeton J M. Merging thermal and visual images by a contrast pyramid［J］. Optical Engineering,1989,28(7):789-792.
［21］	Petrovic V S,Xydeas C S. Gradient-based multiresolution image fusion［J］. IEEE Transactions on Image Processing,2004,13(2):228-237.
［22］	Matsopoulos G K,Marshall S,Brunt J N H. Fusion of MR and CT images of the human brain using multiresolution morphology［M］
	∥Mathematical Morphology and Its Applications to Image Processing.Dordrecht:Springer,1994:137-142.
［23］	Chipman L J,Orr T M,Graham L N. Wavelets and image fusion［C］∥Proc of IEEE Computer Society International Conference on Image Processing,1995:32-48.
［24］	Lewis J J,Callaghan R J O,Nikolov S G,et al. Pixel- and region-based image fusion with complex wavelets［J］. Information Fusion,2007,8:119-130.
［25］	Pajares G, de la Cruz J M. A wavelet-based image fusion tutorial［J］. Pattern Recognition,2004,37(9):1855-1872.
［26］	Wang H H. A new multiwavelet-based approach to image fusion［J］. Journal of Mathematical Imaging and Vision,2004,21:177-192.
［27］	Ioannidou S,Karathanassi V. Investigation of the dual-tree complex and shift-invariant discrete wavelet transforms on quick bird image fusion［J］. IEEE Geoscience & Remote Sensing Letters,2007,4(1):166-170.
［28］	Upla K P,Joshi M V,Gajjar P P. An edge preserving multiresolution fusion:Use of contourlet transform and MRF prior［J］. IEEE Transactions on Geoscience and Remote Sensing,2015,53(6):3210-3220.
［29］	Candes E J,Donoho D L. Curvelets and curvilinear integrals［J］. Journal of Approximation Theory,2001,113(1):59-90.
［30］	Easley G,Labate D,Lim W Q. Sparse directional image re- presentations using the discrete shearlet transform［J］. Applied and Computational Harmonic Analysis,2008,25(1):25-46.
［31］	Bhatnagar G,Wu Q M J,Liu Z. Directive contrast based multimodal medical image fusion in NSCT domain［J］. IEEE Transactions on Multimedia,2013,15(5):1014-1024.
［32］	Ganasala P,Kumar V. Multimodality medical image fusion based on new features in NSST domain［J］. Biomedical Engineering Letters,2015,4(4):414-424.
［33］	Liu X,Mei W,Du H. Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion［J］. Neurocomputing,2017,235:131-139.
［34］	Liu Y,Liu S P,Wang Z F. A general framework for image fusion based on multi-scale transform and sparse representation［J］. Information Fusion,2015,24(C):147-164.
［35］	Yin Wen. Li Yuan-xiang,Zhou Ze-ming,et al. Remote sensing image fusion based on sparse representation［J］. Acta Optica Sinica,2013,33(4):428001-428003.(in Chinese)
［36］	Liu Y,Wang Z. Multi-focus image fusion based on sparse representation with adaptive sparse domain selection［C］∥Proc of the 7th International Conference on Image & Graphics,2013:591-596.
［37］	Li S,Yin H,Fang L. Group-sparse representation with dictionary learning for medical image denoising and fusion［J］. IEEE Transactions on Bio-Medical Engineering,2012,59(12):3450-3459.
［38］	Zhang Xiao,Xue Yue-ju,Tu Shu-qin,et al. Remote sensing image fusion based on structural group sparse representation［J］. Journal of Image and Graphics,2016,21(8):1106-1118.(in Chinese)
［39］	Zhang Q,Levine M D. Robust multi-focus image fusion using multi-task sparse representation and spatial context［J］.IEEE Transactions on Image Processing,2016,25(5):2045-2058.
［40］	Zhang Y,Prasad S. Multisource geospatial data fusion via local joint sparse representation［J］. IEEE Transactions on Geoscience and Remote Sensing,2016,54(6):3265-3276.
［41］	Huang W,Jing Z. Multi-focus image fusion using pulse coupled neural network［J］. Pattern Recognition Letters,2007,28(9):1123-1132.
［42］	Wang Z,Ma Y. Medical image fusion using m-PCNN［J］. Information Fusion,2008,9(2):176-185.
［43］	Miao Qi-guang,Wang Bao-shu. A novel image fusion algorithm based on local contrast and adaptive PCNN［J］. Chinese Journal of Computers,2008,31(5):875-880.(in Chinese)
［44］	Ganasala P,Kumar V. Feature-motivated simplified adaptive PCNN-based medical image fusion algorithm in NSST domain［J］. Journal of Digital Imaging,2016,29(1):73-85.
［45］	Qu X B,Yan J W, Xiao H Z,et al. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain［J］. Acta Automatica Sinica,2008,34(12):1508-1514.
［46］	Liu Y,Chen X,Peng H,et al. Multi-focus image fusion with a deep convolutional neural network［J］. Information Fusion,2017,36:191-207.
［47］	Liu Y,Chen X,Cheng J,et al. A medical image fusion method based on convolutional neural networks［C］∥Proc of 2017 20th International Conference on Information Fusion,2017:1-7.
［48］	Tang H,Xiao B,Li W,et al. Pixel convolutional neural network for multi-focus image fusion［J］. Information Sciences,2018,433:125-141.
［49］	Du C B,Gao S S. Multi-focus image fusion with the all convolutional neural network［J］. Optoelectronics Letters,2018,14(1):71-75.
［50］	Xia K J,Yin H S,Wang J Q. A novel improved deep convolutional neural network model for medical image fusion［J］. Cluster Computing,2019,22:1515-1527.
［51］	Mostafa A N,Ali A,Mehdi E. Ensemble of CNN for multi-
	focus image fusion［J］. Information Fusion,2019,51:201-214.
［52］	Niu Xiao-hui, Jia Ke-bin. Image fusion algorithm based on PCA & self-adaptive region variance［J］. Application Research of Computers,2010,27(8):3179-3181. (in Chinese)
［53］	Ma Xian-xi,Peng Li,Xu Hong. PCA-based Laplacian pyramid in image fusion［J］. Computer Engineering and Applications,2012,48(8):211-213.(in Chinese)
［54］	Tian J,Chen L. Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure［J］. Signal Processing,2012,92(9):2137-2146.
［55］	Wang Wei,Zhang Jia-e. Remote sensing image fusion method based on guided filter and sparse representation［J］. Journal of Chinese Computer Systems, 2017,38(3):601-604. (in Chinese)
［56］	Wang Wei,Zhang Jia-e. A remote sensing image fusion algorithm based on guided filtering and shearlet sparse base［J］. Computer Engineering & Science, 2018,40(8):1453-1458.(in Chinese)
［57］	Ouyang Ning,Zheng Xue-ying,Yuan Hua. Multi-focus image fusion based on NSCT and sparse representation［J］. Computer Engineering and Design,
20	17,38(1):177-182.(in Chinese)
［58］	Yin Ming,Duan Pu-hong,Chu Biao,et al. Fusion of infrared and visible images combined with NSDTCT and sparse representation［J］. Optics and Precision Engineering,
20	16,24(7):1763-1771. (in Chinese)
［59］	Ding S,Zhao X,Hui X,et al. NSCT-PCNN image fusion based on image gradient motivation［J］. IET Computer Vision,2018,12(4):377-383.
［60］	Cheng B,Jin L,Li G. A novel fusion framework of visible light and infrared images based on singular value decomposition and adaptive DUAL-PCNN in NSST domain［J］. Infrared Physics & Technology,2018,91:153-163.
［61］	Xia J,Chen Y,Chen A,et al. Medical image fusion based on sparse representation and PCNN in NSCT domain［J］. Computational & Mathematical Methods in Medicine,2018,5:1-12.
［62］	Dai Wen-zhan,Pan Shu-wei,Li Jun-feng. Medical image fusion algorithm based on human visual features and adaptive PCNN［J］. Journal of Optoelectronics·Laser,2017,28(7):808-816.(in Chinese)
［63］	Krizhevsky A,Sutskever I,Hinton G E. ImageNet classification with deep convolutional neural networks［J］.
	Communications of the ACM,2017,60(6):84-90.
［64］	Lawrence S,Giles C L,Tsoi A C,et al. Face recognition:A convolutional neural-network approach［J］. IEEE Transactions on Neural Networks,1997,8(1):98-113.
［65］	Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(4):640-651.
［66］	Dong C,Chen C L,He K M,et al. Image super-resolution using deep convolutional networks［J］. IEEE
	Transactions on Pattern Analysis and Machine Intelligence,2016,38(2):295-307.
［67］	Zhan K,Li Q Q,Teng J C,et al. Multifocus image fusion using phase congruency［J］. Journal of Electronic Imaging,2015,24(3):033014.
［68］	Zhan K,Teng J,Li Q,et al. A novel explicit multi-focus image fusion method［J］. Journal of Information Hiding and Multimedia Signal Processing,2015,3(6):600-612.
［69］	Huang W, Jing Z. Evaluation of focus measures in multi- focus image fusion［J］. Pattern Recognition Letters,2007,28(4):493-500.
［70］	Pu Tian,Fang Qing-zhe,Ni Guo-qiang. Contrast-based multiresolution image fusion［J］. Acta Electronica Sinica, 2000,28(12):116-118.(in Chinese)
［71］	Li S,Kwok J T,Wang Y. Combination of images with diverse focuses using the spatial frequency［J］. Information Fusion,2001,2(3):169-176.
［72］	Xydeas C S,Petrovic V. Objective image fusion performance measure［J］. Electronics Letters,2000,36(4):308-309.
［73］	Wang Chao, Ye Zhong-fu. Similarity-based objective mea- sure for performance of image fusion［J］. Journal of Software,2006,17(7):1580-1587.(in Chinese)
［74］	Luo Xiao-qing,Wu Xiao-jun. An evaluation method of image fusion based on region similarity［J］. Acta Electronica Sinica,2010,38(5):1152-1155.(in Chinese)
［75］	Luo Lan,Du Qin-sheng. Image fusion quality assessment algorithm based on multi-scale local variance［J］. Computer Engineering,2017,43(2):264-267.(in Chinese)
［76］	Jagalingam P,Hegde A V. A review of quality metrics for fused image［J］. Aquatic Procedia,2015,4:133-142.
［77］	Liu Z,Blasch E,Xue Z,et al. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision:A comparative study［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(1):94-109.
［78］	Zhang Xiao-li,Li Xiong-fei,Li Jun. Validation and correlation analysis of metrics for evaluating performance of image fusion［J］. Acta Automatica Sinica,2014,40(2):306-315.(in Chinese)
［79］	Liu Y,Liu S,Wang Z. Multi-focus image fusion with dense SIFT［J］. Information Fusion,2015,23:139-155.
	附中文参考文献：
［35］	尹雯,李元祥,周则明，等. 基于稀疏表示的遥感图像融合方法［J］. 光学学报,2013,33(4):428001-428003.
［38］	张晓，薛月菊，涂淑琴，等.基于结构组稀疏表示的遥感图像融合［J］.中国图象图形学报，2016,21(8):1106-1118.
［43］	苗启广,王宝树. 基于局部对比度的自适应PCNN图像融合［J］. 计算机学报,2008,31(5):875-880.
［52］	牛晓晖,贾克斌. 基于PCA和自适应区域方差的图像融合方法［J］. 计算机应用研究,2010,27(8):3179-3181.
［53］	马先喜,彭力,徐红. 基于PCA的拉普拉斯金字塔变换融合算法研究［J］. 计算机工程与应用,2012,48(8):211-213.
［55］	王威,张佳娥. 引导滤波和稀疏表示相结合的遥感图像融合算法［J］. 小型微型计算机系统,2017,38(3):601-604.
［56］	王威,张佳娥. 基于引导滤波和shearlet稀疏的遥感图像融合算法［J］. 计算机工程与科学, 2018,40(8):1453-1458.
［57］	欧阳宁,郑雪英,袁华. 基于NSCT和稀疏表示的多聚焦图像融合［J］. 计算机工程与设计,2017,38(1):177-182.
［58］	殷明,段普宏,褚标，等. 基于非下采样双树复轮廓波变换和稀疏表示的红外和可见光图像融合［J］. 光学精密工程,2016,24(7):1763-1771.
［62］	戴文战,潘树伟,李俊峰. 基于人眼视觉特性与自适应PCNN的医学图像融合算法［J］. 光电子·激光,2017,28(7):808-816.
［70］	蒲恬,方庆喆,倪国强. 基于对比度的多分辨图像融合［J］. 电子学报,2000,28(12):116-118.
［73］	王超,叶中付. 基于相似性的图像融合质量的客观评估方法［J］. 软件学报,2006,17(7):1580-1587.
［74］	罗晓清,吴小俊. 一种基于区域相似性的图像融合评价方法［J］. 电子学报,2010,38(5):1152-1155.
［75］	罗兰,杜钦生. 基于多尺度局部方差的图像融合质量评价算法［J］. 计算机工程,2017,43(2):264-267.
［78］	张小利,李雄飞,李军. 融合图像质量评价指标的相关性分析及性能评估［J］. 自动化学报,2014,40(2):306-315.

[1]	李彤彤, 王诗蕊, 张耀方, 王佰玲, 王子博, 刘红日, . 面向工控系统漏洞的多维属性评估[J]. 计算机工程与科学, 2023, 45(02): 261-268.
[2]	李一, 李阳, 苗壮, 王家宝, 张睿. 一种扩展VIFB的红外与可见光图像融合基准[J]. 计算机工程与科学, 2022, 44(06): 1072-1082.
[3]	张贵仓, 王静, 苏金凤. 改进稀疏表示与积化能量和的多聚焦图像融合[J]. 计算机工程与科学, 2022, 44(01): 124-131.
[4]	张丽霞, 曾广平, 宣兆成. NSST域下SPCNN与SR结合的多源图像融合[J]. 计算机工程与科学, 2020, 42(11): 2042-2049.
[5]	张贵仓, 苏金凤, 拓明秀. DTCWT域的红外与可见光图像融合算法[J]. 计算机工程与科学, 2020, 42(07): 1226-1233.
[6]	王杨1,2，向秀梅1,2，卢嘉1,2，郁振鑫1,2. 基于双目融合的无参考立体图像质量评价[J]. 计算机工程与科学, 2020, 42(03): 510-516.
[7]	苏金凤，张贵仓，汪凯. 图像差与加权核范数最小化的压缩图像融合[J]. 计算机工程与科学, 2019, 41(10): 1785-1794.
[8]	王健1,2,张修飞1,任萍1,院文乐1. 基于增补小波变换和PCNN的NSCT域图像融合算法[J]. 计算机工程与科学, 2018, 40(10): 1822-1828.
[9]	王威1,2，张佳娥1,2. 基于引导滤波和shearlet稀疏的遥感图像融合算法[J]. 计算机工程与科学, 2018, 40(08): 1453-1458.
[10]	王威1,2，张佳娥1,2. 基于引导滤波和shearlet稀疏的遥感图像融合算法[J]. 计算机工程与科学, 2018, 40(07): 1250-1255.
[11]	陈清江，张彦博，柴昱洲，魏冰蔗 . 结合区域特性的有限离散剪切波图像融合[J]. 计算机工程与科学, 2017, 39(02): 351-358.
[12]	曹重华1，2，夏家莉1，2，彭文忠2，毛阿敏1，陈位斌1，吴丹1. 克服阻塞的用户交互处理策略[J]. J4, 2016, 38(06): 1135-1140.
[13]	江铁成. 一种改进PCA与IHS融合的高光谱图像异常检测算法[J]. J4, 2016, 38(04): 733-738.
[14]	都琳，孙华燕，张廷华，王帅. 基于相机响应曲线的高动态范围图像融合[J]. J4, 2015, 37(07): 1331-1337.
[15]	王丽1，苗凤娟1，陶佰睿2. 结合Curvelet变换和LSWT的多聚焦图像融合算法[J]. J4, 2015, 37(06): 1203-1207.