基于LDA和卷积神经网络的半监督图像标注方法

计算机工程与科学 ›› 2022, Vol. 44 ›› Issue (01): 110-117.

基于LDA和卷积神经网络的半监督图像标注方法

王保成1，刘利军1，黄青松1,2

（1.昆明理工大学信息工程与自动化学院，云南昆明 650500； 2.云南省计算机技术应用重点实验室，云南昆明 650500）

收稿日期:2020-06-29 修回日期:2020-09-20 接受日期:2022-01-25 出版日期:2022-01-25 发布日期:2022-01-13
基金资助:
国家自然科学基金(81860318,81560296)

A semi supervised image annotation method based on LDA and convolutional neural network

WANG Bao-cheng1，LIU Li-jun1，HUANG Qing-song1,2

(1.Faculty of Information Engineering and Automation，Kunming University of Science and Technology，Kunming 650500;

2.Yunnan Key Laboratory of Computer Technology Application，

Kunming University of Science and Technology，Kunming 650500，China)

Received:2020-06-29 Revised:2020-09-20 Accepted:2022-01-25 Online:2022-01-25 Published:2022-01-13

摘要/Abstract

摘要： 随着智能设备的不断出现，图像数量急速增加，但是很多图像因为没有被标注所以未被充分利用。为了能够使该问题得到较好解决，提出了基于LDA和卷积神经网络的半监督图像标注方法。首先把图像训练集中的所有文字信息放入LDA中，生成图像的文字标注词;然后使用卷积神经网络获得图像的高层视觉特征，同时用加入注意力机制和修改损失函数的方法来对卷积神经网络进行优化;接着把LDA生成的标注词和已获得的图像的高层视觉特征进行结合并同时使用半监督学习来完成模型的训练；
最后把标注词间的相关性和使用最终模型预测的结果相结合来完成图像的最终标注。通过在IAPR TC-12 图像数据集上的相关实验对比可知，文中所提方法的标注更精确。

关键词: LDA, 卷积神经网络, 注意力机制, 半监督学习

Abstract: With the continuous emergence of intelligent devices, the number of pictures increases rapidly. However, many images are not fully utilized because they are not labeled. In order to solve this problem, a semi supervised image annotation method based on LDA and convolutional neural network is proposed. Firstly, all text information in the image training set is put into LDA to generate text tagging words. Secondly, the convolutional neural network is used to obtain the high-level visual features of the image, and the convolutional neural network is optimized by adding attention mechanism and modifying loss function. Thirdly, the label words generated by LDA are combined with the high-level visual features of the obtained image, and the semi supervised learning is used to complete the model training. Finally, the correlation between the tagging words and the prediction results using the final model are combined to complete the final tagging of the image. Comparative experiments on the IAPR TC-12 image data set show that the proposed labeling method is more accurate.

Key words: LDA, convolutional neural network, attention mechanism, semi supervised learning

王保成, 刘利军, 黄青松, . 基于LDA和卷积神经网络的半监督图像标注方法[J]. 计算机工程与科学, 2022, 44(01): 110-117.

WANG Bao-cheng, LIU Li-jun, HUANG Qing-song, . A semi supervised image annotation method based on LDA and convolutional neural network[J]. Computer Engineering & Science, 2022, 44(01): 110-117.

参考文献［28］

［1］	Duygulu P,Barnard K,de Freitas J F G,et a1.Object recognition as machine translation:Learning alexicon for a fixed image vocabulary ［C］∥Proc ofthe 7th European Conference on Computer Vision(ECCV 2002),2002：97-112.
［2］	Hofmann T.Unsupervised learning by probabilistic latent semantic analysis［J］.Machine Learning,2001,42(1-2):177-196.
［3］	Blei D M,Ng A Y,Jordan M I.Latent Dirichlet allocation［J］.The Journal of Machine Learning Research,2003,3:993-1022.
［4］	Zhao Peng,Wang Wen-bin,Zhu Wei-wei,et al.Automatic image annotation method integrating theme and visual semantics ［J］.Journal of Computer-Aided Design and Graphics,2013,25(11):1709-1714.(in Chinese)
［5］	Cao Jie, Lou Jü-xiang,Li Xiao-xu.Image annotation probabilistic topic model fusing class information［J］.Computer Engineering and Applications,2017,53(10):187-192.(in Chinese)
［6］	Grangier D, Bengio S. A discriminative kernel-based approach to rank images from text queries［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence,2008,30(8)：1371-1384.
［7］	Makadia A,Pavlovic V,Kumar S.Baselines for image annotation［J］.International Journal of Computer Vision,2010,90(1)：88-105.
［8］	Guillaumin M,Mensink T,Verbeek J,et a1.TaGProp：Discriminative metric learning in nearest neighbor models for image auto-annotation［C］∥Proc of 2019 IEEE 12th International Conference on Computer Vision,2009:309-316.
［9］	Murthy V N,Maji S,Manmatha R.Automatic image annotation using deep learning representations［C］∥Proc of the 5th ACM International Conference on Multimedia Retrieval,2015:603-606.
［10］	Gao Yao-dong,Hou Ling-yan,Yang Da-li. Automatic image annotation methodusingmulti-label learning convolutional neural network ［J］.Journal of Computer Applications,2017,37(1):228-232.(in Chinese)
［11］	Yu Ning,Song Hai-yu,Sun Dong-yang,et al.Image annotation based on
	middle-layerconvolution feature of deep learning ［J］.Journal of Graphics,2019,40(5):872-877.(in Chinese)
［12］	Zhang Lei. Research and application of image annotation method based on theme model ［D］.Wuxi:Jiangnan University,2019.(in Chinese)
［13］	Zhang W F,Hu H,Hu H Y.Neural ranking for automatic image annotation［J］.Multimedia Tools and Applications,2018,77(17):22385-22406.
［14］	Lecun Y, Kavukcuoglu K,Farabet C.Convolutional networks and applications in vision［C］∥Proc of 2010 IEEE International Symposium on Circuits and Systems,
20	10:253-256.
［15］	Li J W,Luong M T,Jurafsky D.A hierarchical neural auto encoder for paragraphs and documents ［J］.
	arXiv:1506.01057,2015.
［16］	Sheng X, Tao F, Hong H, et al. A novel method of aerial image classification based on attention-based local descript- ors［J］. Procedia Earth & Planetary Science, 2009, 1(1):1133-1139.
［17］	Stollenga M, Masci J,Gomez F,et al.Deep networks with internal selective attention through feedback connections ［J］.Advances in Neural Information Processing Systems,2014,4(2):3545-3553.
［18］	Huang Y,Wang W,Wang L.Instance-aware image and sentence matching with selective multimodal LSTM［C］∥Proc of 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2017:7254-7262.
［19］	Zhang X,Lai H,Feng J.Attention-aware deep adversarial hashing for cross-modal retrieval［M］.Cham:Springer,2018.
［20］	Ma Yong-jie,Li Xue-yan,Song Xiao-feng.Traffic sign recognition based on improved deep convolution neural network ［J］.Progress in Laser and Optoelectronics,2018,55(12):250-257.(in Chinese)
［21］	Russakovsky O,Deng J,Su H,et al.ImageNet large scale visual recognition challenge ［J］.International Journal of Computer Vision,2015,115(3):211-252.
［22］	Zhuang Fu-zhen, Luo Ping, He Qing, et al.Survey on transfer learning research ［J］.Journal of Software,2015,26(1):26-39.(in Chinese)
［23］	Li Zhi-xin, Zheng Yong-zhe,Zhang Can-long,et al.Image semantic annotation combined with depth feature and multi marker classification ［J］.Journal of Computer-Aided Design and Graphics,2018,30(2):318-326.(in Chinese)
［24］	He K M,Zhang X Y,Ren S Q,et al.Deep residual learning for image recognition ［C］∥Proc of International Conference on Computer Vision and Pattern Recognition,2016:770-778.
［25］	Sun Jun-ding,Li Hai-hua,Jin Jiao-lin,et al.Automatic image annotation based on multi feature fusion and PLSA-GMM［J］.Measurement and Control Technology,2017,36(4):31-35.(in Chinese)
［26］	Kuric E, Bielikova M. ANNOR:Efficient image annotation based on combing local and global features ［J］.Computers & Graphics,2015,47:1-15.
［27］	Zhang X,Liu C.Image annotation based on feature fusion and semantic similarity ［J］.Neurocomputing,2015,149:1658-1671.
［28］	Wang Peng, Zhang Ao-fan, Wang Li-qin, et al. Image automatic annotation based on transfer learning and multi label smoothing strategy ［J］.Journal of Computer Applications,2018,38(11):
31	99-3203.(in Chinese)
	附中文参考文献：
［4］	赵鹏,王文彬,朱伟伟,等.融合主题和视觉语义的图像自动标注方法［J］.计算机辅助设计与图形学学报,2013,25(11):1709-1714.
［5］	曹洁, 罗菊香, 李晓旭. 融入类别信息的图像标注概率主题模型［J］. 计算机工程与应用, 2017, 53(10):187-192.
［10］	高耀东,侯凌燕,杨大利.基于多标签学习的卷积神经网络的图像标注方法［J］.计算机应用,2017,37(1):228-232.
［11］	于宁,宋海玉,孙东洋,等.基于深度学习中间层卷积特征的图像标注［J］.图学学报,2019,40(5):872-877.
［12］	张蕾.基于主题模型的图像标注方法研究与应用［D］.无锡：江南大学,2019.
［20］	马永杰,李雪燕,宋晓凤.基于改进深度卷积神经网络的交通标志识别［J］.激光与光电子学进展,2018,55(12):250-257.
［22］	庄福振, 罗平, 何清, 等. 迁移学习研究进展［J］.软件学报,2015, 26(1):26-39.
［23］	李志欣,郑永哲,张灿龙,等.结合深度特征与多标记分类的图像语义标注［J］.计算机辅助设计与图形学学报,2018,30(2):318-326.
［25］	孙君顶,李海华,靳姣林,等.基于多特征融合与 PLSA-GMM 的图像自动标注［J］.测控技术,2017,36(4):31-35.
［28］	汪鹏,张奥帆,王利琴,等.基于迁移学习与多标签平滑策略的图像自动标注［J］.计算机应用,2018,38(11):3199-3203.

[1]	马思远, 焦佳辉, 任晟岐, 宋伟. 基于注意力机制的城市多元空气质量数据缺失值填充[J]. 计算机工程与科学, 2023, 45(08): 1354-1364.
[2]	尹春勇, 冯梦雪. 基于注意力机制的半监督日志异常检测方法[J]. 计算机工程与科学, 2023, 45(08): 1405-1415.
[3]	余子丞, 凌捷. 基于Transformer和多特征融合的DGA域名检测方法[J]. 计算机工程与科学, 2023, 45(08): 1416-1423.
[4]	唐剑, 车文刚, 高盛祥. 融入注意力机制的多尺度卷积图像去雾方法[J]. 计算机工程与科学, 2023, 45(08): 1453-1462.
[5]	吴栋梁, 刘知贵, . 基于轻量化YOLOX的电子元器件缺陷检测方法研究[J]. 计算机工程与科学, 2023, 45(08): 1463-1471.
[6]	刘俊奇, 涂文轩, 祝恩. 图卷积神经网络综述[J]. 计算机工程与科学, 2023, 45(08): 1472-1481.
[7]	易啸, 马胜, 肖侬. 深度学习加速器在不同剪枝策略下的运行优化[J]. 计算机工程与科学, 2023, 45(07): 1141-1148.
[8]	刘浩翰, 孙铖, 贺怀清, 惠康华. 基于改进YOLOv3的金属表面缺陷检测[J]. 计算机工程与科学, 2023, 45(07): 1226-1235.
[9]	崔克彬, 崔叶微. 基于卷积和Transformer的断路器动触头跟踪方法研究[J]. 计算机工程与科学, 2023, 45(07): 1236-1244.
[10]	白杉, 冯秀芳. 基于注意力增强的中心差分自适应图卷积的骨架行为识别[J]. 计算机工程与科学, 2023, 45(07): 1263-1273.
[11]	王剑, 姜林, 王琳钦, 余正涛, 张松, 高盛祥, . 基于BiLSTM的低资源老挝语文本正则化任务[J]. 计算机工程与科学, 2023, 45(07): 1292-1299.
[12]	田秀霞, 刘正, 刘秋旭, 李浩然. 一种改进Faster R-CNN的图像篡改检测模型[J]. 计算机工程与科学, 2023, 45(06): 1030-1039.
[13]	杨慧剑, 孟亮. 基于改进的YOLOv5的航拍图像中小目标检测算法[J]. 计算机工程与科学, 2023, 45(06): 1063-1070.
[14]	刘晓航, 姜晶菲, 许金伟. 基于脉动阵列的层融合注意力模型加速器结构[J]. 计算机工程与科学, 2023, 45(05): 802-809.
[15]	排日旦·阿布都热依木, 吐尔地·托合提, 艾斯卡尔·艾木都拉, . 基于深度学习的实体关系抽取方法研究[J]. 计算机工程与科学, 2023, 45(05): 895-902.