计算机工程与科学 ›› 2021, Vol. 43 ›› Issue (12): 2190-2205.
姬子恒,王斌
收稿日期:
2020-09-03
修回日期:
2020-11-04
接受日期:
2021-12-25
出版日期:
2021-12-25
发布日期:
2021-12-31
基金资助:
JI Zi-heng,WANG Bin
Received:
2020-09-03
Revised:
2020-11-04
Accepted:
2021-12-25
Online:
2021-12-25
Published:
2021-12-31
摘要: 草图检索(SBIR)是基于内容的图像检索(CBIR)的扩展,是一种灵活便捷的目标图像检索方式,其研究的焦点是如何减少手绘草图域与自然图像域之间的域差。传统方法提取手工特征完成草图域与图像域之间的近似转换以减少域差,但该类方法无法有效拟合2个域内容,导致检索精度不高。深度学习方法依赖大量数据进行图像高维特征的提取,突破了传统方法的局限,已被证明可以有效解决跨域建模问题。研究聚焦于基于深度学习的草图检索方法,在深度特征提取模型、公开的数据测试集、粗粒度和细粒度检索、哈希技术和类别泛化等几个方面对草图检索的深度学习方法的相关研究工作进行了综述和评论。然后进行了实验比较研究,一方面,对现有3个公开的SBIR测试集Sketchy、TU-Berlin和QuickDraw进行适用性评估;另一方面,选取3个最新的SBIR深度学习模型GRLZS模型、SEM-PCYC模型和SAKE模型进行性能分析与比较。最后,对草图检索面临的挑战和未来研究方向进行了总结与展望。
姬子恒, 王斌. 基于深度学习的草图检索方法研究进展[J]. 计算机工程与科学, 2021, 43(12): 2190-2205.
JI Zi-heng, WANG Bin. Research progress on deep learning based sketch retrieval[J]. Computer Engineering & Science, 2021, 43(12): 2190-2205.
[1] | Smeulders A W M,Worring M,Santini S,et al.Content-based image retrieval at the end of the early years[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(12):1349-1380. |
[2] | Shen Jin-ke.Research on content-based image retrieval and practice[D].Wuhan:Wuhan Institute of Technology,2011.(in Chinese) |
[3] | Zhou Ming-quan,Gen Guo-hua,Wei Na.Image retrieval technique based on content[M].Beijing:Tsinghua University Press,2007.(in Chinese) |
[4] | Pang K,Song Y Z,Xiang T,et al.Cross-domain generative learning for fine-grained sketch-based image retrieval[C]∥Proc of British Machine Vision Conference,2017:1-12. |
[5] | Duan Xiong.Sketch based image retrieval using deep feature[D].Wuhan:Huazhong University of Science and Technology,2017.(in Chinese) |
[6] | Eitz M,Hildebrand K,Boubekeur T,et al.An evaluation of descriptors for large-scale image retrieval from sketched feature lines[J].Computers & Graphics,2010,34(5):482-498. |
[7] | Eitz M,Hildebrand K,Boubekeur T,et al.Sketch-based image retrieval:Benchmark and bag-of-features descriptors[J].IEEE Transactions on Visualization and Computer Graphics,2010,17(11):1624-1636. |
[8] | Cao Y,Wang C H,Zhang L Q,et al.Edgel index for large-scale sketch-based image search[C]∥ |
Proc of the 2011 IEEE Conference on Computer Vision and Pattern Recognition,2011:761-768. | |
[9] | Cao Y, Wang H, Wang C H, et al. MindFinder:Interactive sketch-based image search on millions of images[C]∥Proc of the 18th ACM International Conference on Multimedia,2010:1605-1608. |
[10] | Wang C,Li Z,Zhang L.MindFinder:Image search by interactive sketching and tagging[C]∥Proc of the 19th International Conference on World Wide Web,2010:1309-1312. |
[11] | Hu R,Wang T,Collomosse J.A bag-of-regions approach to sketch-based image retrieval[C]∥Proc of 2011 18th IEEE International Conference on Image Processing,2011:3661-3664. |
[12] | Lin Y L,Huang C Y,Wang H J,et al.3D sub-query expansion for improving sketch-based multi-view image retrieval[C]∥Proc of the 2013 IEEE International Conference on Computer Vision,2013:3495-3502. |
[13] | James S,Fonseca M J,Collomosse J.ReEnact:Sketch based choreographic design from archival dance footage[C]∥Proc of International Conference on Multimedia Retrieval,2014:313-320. |
[14] | Saavedra J M. Sketch based image retrieval using a soft computation of the histogram of edge local orientations (S-HELO)[C]∥Proc of 2014 IEEE International Conference on Image Processing (ICIP),2014:2998-3002. |
[15] | Saavedra J M,Barrios J M,Orand S.Sketch based image retrieval using learned keyshapes (LKS)[C]∥Proc of British Machine Vision Conference,2015:7. |
[16] | Hu R,Collomosse J.A performance evaluation of gradient field hog descriptor for sketch based image retrieval[J].Computer Vision and Image Understanding,2013,117(7):790-806. |
[17] | |
Lowe D G. Object recognition from local scale-invariant features[C]∥Proc of the 7th IEEE International Conference on Computer Vision,1999:1150-1157. | |
[18] | Yu Deng,Liu Yu-jie,Xing Min-min,et al. Sketch-based image retrieval using cross-domain modeling and deep fusion network[J].Journal of Software,2019,30(11):3567-3577.(in Chinese) |
[19] | Bui T,Ribeiro L,Ponti M,et al.Generalisation and sharing in triplet convnets for sketch based visual search[J].arXiv:1611.05301,2016. |
[20] | Hadsell R, Chopra S, LeCun Y. Dimensionality reduction by learning an invariant mapping[C]∥ |
Proc of 2016 IEEE Conference on Computer Vision and Pattern Recognition,2006:1735-1742. | |
[21] | Song Y, Lei J,Peng B,et al.Edge-guided cross-domain learning with shape regression for sketch-based image retrieval[J].IEEE Access,2019,7:32393-32399. |
[22] | Zhang H,Zhang C,Wu M.Sketch-based cross-domain image retrieval via heterogeneous network[C]∥Proc of 2017 IEEE Visual Communications and Image Processing (VCIP),2017:1-4. |
[23] | |
Schroff F, Kalenichenko D, Philbin J. FaceNet: A unified embedding for face recognition and clustering[C]∥Proc of 2015 IEEE Conference on Computer Vision and Pattern Recognition,2015: 815-823. | |
[24] | Eitz M,Hays J,Alexa M. How do humans sketch objects?[J]. ACM Transactions on Graphics,2012,31(4):44:1-44:10. |
[25] | Liu L, Shen F, Shen Y, et al.Deep sketch hashing:Fast free-hand sketch-based image retrieval[C]∥Proc of |
20 | 17 IEEE Conference on Computer Vision and Pattern Recognition,2017:2862-2871. |
[26] | Sangkloy P, Burnell N, Ham C,et al. The sketchy database:Learning to retrieve badly drawn bunnies[J]. ACM Transactions on Graphics, 2016, 35(4):1-12. |
[27] | Yu Q,Liu F,Song Y Z,et al.Sketch me that shoe[C]∥Proc of 2016 IEEE Conference on Computer Vision and Pattern Recognition,2016:799-807. |
[28] | Dey S, Riba P, Dutta A, et al. Doodle to search: Practical zero-shot sketch-based image retrieval[C]∥Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019:2179-2188. |
[29] | Qi Y,Song Y Z,Zhang H,et al.Sketch-based image retrieval via siamese convolutional neural network[C]∥Proc of 2016 IEEE International Conference on Image Processing (ICIP),2016:2460-2464. |
[30] | Yu Q,Yang Y,Song Y Z,et al.Sketch-a-net that beats humans[J].arXiv:1501.07873,2015. |
[31] | Lei J,Zheng K,Zhang H,et al.Sketch based image retrieval via image-aided cross domain learning[C]∥Proc of 2017 IEEE International Conference on Image Processing (ICIP),2017:3685-3689. |
[32] | Wang F, Kang L,Li Y.Sketch-based 3D shape retrieval using convolutional neural networks[C]∥Proc of 2015 IEEE Conference on Computer Vision and Pattern Recognition,2015:1875-1883. |
[33] | Yu D,Liu Y J,Pang Y P,et al.A multi-layer deep fusion convolutional neural network for sketch based image retrieval[J].Neurocomputing,2018,296:23-32. |
[34] | Song J,Yu Q,Song Y Z,et al.Deep spatial-semantic attention for fine-grained sketch-based image retrieval[C]∥Proc of the 2017 IEEE International Conference on Computer Vision,2017:5551-5560. |
[35] | Xu P, Yin Q Y,Huang Y Y,et al.Cross-modal subspace learning for fine-grained sketch-based image retrieval[J].Neurocomputing,2018,278:75-86. |
[36] | Bui T,Ribeiro L,Ponti M,et al.Sketching out the details:Sketch-based image retrieval using convolutional neural networks with multi-stage regression[J].Computers & Graphics,2018,71:77-87. |
[37] | Qi Q, Huo Q, Wang J, et al. Personalized sketch-based image retrieval by convolutional neural network and deep transfer learning[J]. IEEE Access, 2019, 7:16537-16549. |
[38] | Bozas K, Izquierdo E.Large scale sketch based image retrieval using patch hashing[C]∥Proc of International Symposium on Visual Computing,2012:210-219. |
[39] | Gong Y C, Lazebnik S, Gordo A,et al. Iterative quantization:A procrustean approach to learning binary codes for large-scale image retrieval[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, |
20 | 12,35(12):2916-2929. |
[40] | Svante W, Esbensen K, Geladi P.Principal component analysis[J].Chemometrics and Intelligent Laboratory Systems, |
19 | 87,2(1-3):37-52. |
[41] | Wang J D, Zhang T, Song J K,et al. A survey on learning to hash[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, |
20 | 17,40(4):769-790. |
[42] | Furuya T, Ohbuchi R.Hashing cross-modal manifold for scalable sketch-based 3D model retrieval[C]∥Proc of 2014 2nd International Conference on 3D Vision,2014:543-550. |
[43] | Matsui Y, Aizawa K, Jing Y. Sketch2Manga:Sketch-based manga retrieval[C]∥Proc of 2014 IEEE International Conference on Image Processing (ICIP),2014:3097-3101. |
[44] | Siddiquie B,White B,Sharma A,et al.Multi-modal image retrieval for complex queries using small codes[C]∥Proc of International Conference on Multimedia Retrieval,2014:321-328. |
[45] | Sun X,Wang C,Xu C,et al.Indexing billions of images for sketch-based retrieval[C]∥Proc of the 21st ACM International Conference on Multimedia,2013:233-242. |
[46] | Tseng K Y,Lin Y L,Chen Y H,et al.Sketch-based image retrieval on mobile devices using compact hash bits[C]∥Proc of the 20th ACM International Conference on Multimedia,2012:913-916. |
[47] | Brian K, Grauman K. Kernelized locality-sensitive hashing[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, |
20 | 11,34(6):1092-1104. |
[48] | Xu P,Huang Y,Yuan T,et al. Sketchmate:Deep hashing for million-scale human sketch retrieval[C]∥Proc of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018:8090-8098. |
[49] | Ruslan S, Hinton G. Semantic hashing[J].International Journal of Approximate Reasoning, |
20 | 09,50(7):969-978. |
[50] | Zhang J Y, Shen F M, Liu L, et al.Generative domain- migration hashing for sketch-to-image retrieval[C]∥Proc of European Conference on Computer Vision (ECCV),2018:304-321. |
[51] | Mirza M,Osindero S.Conditional generative adversarial nets[J].arXiv:1411.1784,2014. |
[52] | Bui T,Ribeiro L,Ponti M,et al.Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network[J].Computer Vision and Image Understanding,2017,164:27-37. |
[53] | Yelamarthi S K,Reddy S K,Mishra A,et al.A zero-shot framework for sketch based image retrieval[C]∥Proc of European Conference on Computer Vision(ECCV),2018:316-333. |
[54] | Zhang Z M,Saligrama V.Zero-shot learning via joint latent similarity embedding |
[C] | ∥Proc of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016:6034-6042. |
[55] | Frome A,Corrado G S,Shlens J,et al.Devise:A deep visual-semantic embedding model[C]∥ |
Proc of the 26th International Conference on Neural Information Processing Systems, 2013:2121-2129. | |
[56] | Kodirov E,Xiang T,Gong S.Semantic autoencoder for zero-shot learning[C]∥Proc of 2017 IEEE Conference on Computer Vision and Pattern Recognition,2017:3174-3183. |
[57] | Zhang Z M,Saligrama V.Zero-shot learning via semantic similarity embedding[C]∥Proc of the 2015 IEEE International Conference on Computer Vision,2015:4166-4174. |
[58] | Socher R,Ganjoo M,Manning C D,et al.Zero-shot learning through cross-modal transfer[C]∥ |
Proc of the 26th International Conference on Neural Information Processing Systems,2013:935-943. | |
[59] | Norouzi M,Mikolov T,Bengio S,et al.Zero-shot learning by convex combination of semantic embeddings[J].arXiv:1312.5650,2013. |
[60] | Changpinyo S,Chao W L,Gong B,et al.Synthesized classifiers for zero-shot learning[C]∥Proc of 2016 IEEE Confe- rence on Computer Vision and Pattern Recognition,2016:5327-5336. |
[61] | Akata Z,Reed S,Walter D,et al.Evaluation of output embeddings for fine-grained image classification[C]∥Proc of 2015 IEEE Conference on Computer Vision and Pattern Recognition,2015:2927-2936. |
[62] | Xian Y, Akata Z, Sharma G,et al.Latent embeddings for zero-shot classification[C]∥Proc of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016:69-77. |
[63] | Shen Y,Liu L, Shen F, et al. Zero-shot sketch-image hash- ing[C]∥Proc of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018:3598-3607. |
[64] | Dutta A,Akata Z.Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval[C]∥Proc of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2019:5089-5098. |
[65] | Liu Q,Xie L,Wang H,et al.Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval[J].arXiv:1904.03208,2019. |
附中文参考文献: | |
[2] | 沈金科.基于内容的图像检索方法研究与实现[D].武汉:武汉工程大学,2011. |
[3] | 周明全,耿国华,韦娜.基于内容的图像检索技术[M].北京:清华大学出版社,2007. |
[5] | 段雄.基于深度特征的手绘草图检索图像方法研究[D].武汉:华中科技大学,2017. |
[18] | 于邓,刘玉杰,邢敏敏,等.包含跨域建模和深度融合网络的手绘草图检索[J] .软件学报,2019,30(11):3567-3577. |
[1] | 马思远, 焦佳辉, 任晟岐, 宋伟. 基于注意力机制的城市多元空气质量数据缺失值填充[J]. 计算机工程与科学, 2023, 45(08): 1354-1364. |
[2] | 易啸, 马胜, 肖侬. 深度学习加速器在不同剪枝策略下的运行优化[J]. 计算机工程与科学, 2023, 45(07): 1141-1148. |
[3] | 康宇晗, 时洋, 陈照云, 文梅. 面向迈创+MatrixZone异构系统的深度学习编程框架[J]. 计算机工程与科学, 2023, 45(07): 1149-1158. |
[4] | 刘浩翰, 孙铖, 贺怀清, 惠康华. 基于改进YOLOv3的金属表面缺陷检测[J]. 计算机工程与科学, 2023, 45(07): 1226-1235. |
[5] | 田秀霞, 刘正, 刘秋旭, 李浩然. 一种改进Faster R-CNN的图像篡改检测模型[J]. 计算机工程与科学, 2023, 45(06): 1030-1039. |
[6] | 濮子俊, 张寿明. 基于特征融合与Transformer模型的声音事件定位与检测算法研究[J]. 计算机工程与科学, 2023, 45(06): 1097-1105. |
[7] | 邓姗姗, 黄慧, 马燕. 基于改进Faster R-CNN的小目标检测算法[J]. 计算机工程与科学, 2023, 45(05): 869-877. |
[8] | 霍爱清, 张书涵, 杨玉艳, 胥静蓉, 王泽文. 密集交通场景中改进YOLOv3目标检测优化算法[J]. 计算机工程与科学, 2023, 45(05): 878-884. |
[9] | 方雪杉, 杨云飞, 冯松. 基于多尺度多模态学习的光球亮点曲线轨迹段检测方法研究[J]. 计算机工程与科学, 2023, 45(05): 885-894. |
[10] | 黄星威, 陈曦, 张塑凡. 改进特征金字塔的小目标深度学习模型[J]. 计算机工程与科学, 2023, 45(04): 734-742. |
[11] | 董佩杰, 牛新, 魏自勉, 陈学晖. 单次神经网络结构搜索研究综述[J]. 计算机工程与科学, 2023, 45(02): 191-203. |
[12] | 强梓林, 刘建国, 刘云峰, 卫栋, 强彦. 基于时域-频域哈希编码的电网图像检索方法[J]. 计算机工程与科学, 2022, 44(10): 1877-1884. |
[13] | 何平, 李刚, 李慧斌, . 基于深度学习的视频异常检测方法综述[J]. 计算机工程与科学, 2022, 44(09): 1620-1629. |
[14] | 刘从军, 徐佳陈, 肖志勇, 柴志雷. 基于深度学习的心脏核磁共振图像自动分割算法[J]. 计算机工程与科学, 2022, 44(09): 1646-1654. |
[15] | 何涛, 施慧莉, 李大亮. 基于深度学习的SAR目标识别DSP设计[J]. 计算机工程与科学, 2022, 44(08): 1357-1363. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||
湘公网安备 43010502000083号
湘ICP备10006030号
版权所有 © 《计算机工程与科学》 编辑部
地址:中国湖南省长沙市开福区德雅路109号(410073) 电话:0731-87002567 Email: jsjgcykx@vip.163.com
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持:support@magtech.com.cn