基于深度学习的视频异常检测方法综述

摘要/Abstract

摘要： 近年来，随着视频监控技术的广泛应用，对海量视频进行智能分析并及时发现其中的异常状态或事件的视频异常检测任务受到了广泛关注。对基于深度学习的视频异常检测方法进行了综述。首先，对视频异常检测问题进行概述，包括基本概念、基本类型、建模流程、学习范式及评价方式。其次，提出将现有基于深度学习的视频异常检测方法分为基于重构的方法、基于预测的方法、基于分类的方法及基于回归的方法4类并详细阐述了各类方法的建模思想、代表性工作及其优缺点。然后，在此基础上介绍了常用的单场景视频异常检测公开数据集和评估指标，并对比分析了代表性异常检测方法的性能。最后，总结全文并从数据集、方法及评估指标3方面对视频异常检测研究的未来发展方向进行了展望。

关键词: 视频监控, 异常检测, 深度学习, 单场景, 学习范式

Abstract: Recent years, with the widespread use of video surveillance technology, video anomaly detection, which can intelligently analyze massive videos and quickly discover the abnormalities, has received wide attention. This paper aims to give a comprehensive survey on deep learning based video anomaly detection methods. Firstly, a brief introduction of video anomaly detection is given, including the basic concepts, basic tasks, modeling process, learning paradigms as well as the evaluation perspectives. Secondly, the video anomaly detection methods are classified into four categories: reconstruction-based, prediction-based, classification-based, and regression-based. Their basic modeling ideas, typical algorithms, advantages, and disadvantages are discussed in detail. On this basis, the commonly used single-scene video anomaly detection public datasets and evaluation indicators are introduced, and the performance of representative anomaly detection algorithms is compared and analyzed. Finally, summary is conducted, and the future development directions related to datasets, algorithm and evaluation criteria of video anomaly detection are proposed.

Key words: surveillance video, anomaly detection, deep learning, single scene, learning paradigm

何平, 李刚, 李慧斌, . 基于深度学习的视频异常检测方法综述[J]. 计算机工程与科学, 2022, 44(09): 1620-1629.

HE Ping, LI Gang, LI Hui-bin, . A survey on deep learning based video anomaly detection[J]. Computer Engineering & Science, 2022, 44(09): 1620-1629.

参考文献［10］

［1］	Lu Bo-wen.Research on anomaly detection in surveillance videos based algorithm on deep neural network［D］.Nanjing：Nanjing University of Posts and Telecommunications,2020.(in Chinese)
［2］	Wang Si-qi,Hu Jing-tao,Yu Guang,et al.A survey of video abnormal event detection［J］.Computer Engineering & Science,2020,42(8):1393-1405.(in Chinese)
［3］	Saligrama V,Konrad J,Jodoin P M.Video anomaly identification［J］.IEEE Signal Processing Magazine,2010,27(5):18-33.
［4］	Zhou Pei-pei,Ding Qing-hai,Luo Hai-bo,et al.Anomaly detection and location in crowded surveillance videos［J］.Acta Optica Sinica,2018,38(8):97-105.(in Chinese)
［5］	Li Juan,Zhang Bing-yi,Feng Zhi-yong,et al.Anomaly detection based on hidden Markov model in videos［J］.Computer Engineering & Science,2017,39(7):1300-1308.(in Chinese)
［6］	Popoola O P,Wang K.Video-based abnormal human behavior recognition—A review［J］.IEEE Transactions on Systems,Man,and Cybernetics,2012,42(6):865-878.
［7］	Kiran B R,Thomas D M,Parakkal R.An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos［J］.Journal of Imaging,2018,4(2):36.
［8］	Peng Jia-li,Zhao Ying-liang,Wang Li-ming. Research on video abnormal behavior detection based on deep learning［J］.Laser and Optoelectronics Progress,2020,58(6):0600004.(in Chinese)
［9］	Hu Zheng-ping,Zhang Le,Li Shu-fang,et al.Review of abnormal behavior detection and location for intelligent video surveillance systems［J］.Journal of Yanshan University,2019,43(1):1-12.(in Chinese)
［10］	Wang Zhi-guo,Zhang Yu-jin.Anomaly detection in surveillance videos:A survey［J］.Journal of Tsinghua University (Science and Technology),2020,60(6):518-529.(in Chinese)
［11］	Xu D,Yan Y,Ricci E,et al.Detecting anomalous events in videos by learning deep representations of appearance and motion［J］.Computer Vision and Image Understanding,2017,156:117-127.
［12］	Hasan M,Choi J,Neumann J,et al.Learning temporal regularity in video sequences［C］∥Proc of IEEE Conference on Computer Vision and Pattern Recognition,2016:733-742.
［13］	Gong D,Liu L Q,Le V,et al.Memorizing normality to detect anomaly:Memory-augmented deep autoencoder for unsupervised anomaly detection［C］∥Proc of IEEE/CVF International Conference on Computer Vision,2019:1705-1714.
［14］	Park H,Noh J,Ham B.Learning memory-guided normality for anomaly detection［C］∥Proc of IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:14372-14381.
［15］	Zhao B, Li F F,Xing E P.Online detection of unusual events in videos via dynamic sparse coding［C］∥Proc of IEEE Conference on Computer Vision and Pattern Recognition,2011:3313-3320.
［16］	Cong Y,Yuan J S,Liu J.Sparse reconstruction cost for abnormal event detection［C］∥Proc of IEEE Conference on Computer Vision and Pattern Recognition,2011:3449-3456.
［17］	Luo W X,Liu W,Gao S H.A revisit of sparse coding based anomaly detection in stacked RNN framework［C］∥Proc of IEEE International Conference on Computer Vision,2017:341-349.
［18］	Luo W X,Liu W,Lian D Z,et al.Video anomaly detection with sparse coding inspired deep neural networks［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,43(3):1070-1084.
［19］	Wu P,Liu J,Li M M,et al.Fast sparse coding networks for anomaly detection in videos［J］.Pattern Recognition,2020,107:107515.
［20］	Liu W,Luo W X,Lian D Z,et al.Future frame prediction for anomaly detection—A new baseline［C］∥Proc of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2018:6536-6545.
［21］	Medel J R,Savakis A.Anomaly detection in video using predictive convolutional long short-term memory networks［J］.arXiv:1612.00390,2016.
［22］	Lu Y W, Kumar K M, Nabavi S S,et al.Future frame prediction using convolutional VRNN for anomaly detection［C］∥Proc of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS),2019:1-8.
［23］	Mathieu M, Couprie C,LeCun Y.Deep multi-scale video prediction beyond mean square error［J］.arXiv:1511.05440,2015.
［24］	Ye M C,Peng X J,Gan W H,et al.AnoPCN:Video anomaly detection via deep predictive coding network［C］∥Proc of the 27th ACM International Conference on Multimedia,2019:1805-1813.
［25］	Sabokrou M, Khalooei M,Fathy M,et al.Adversarially learned one-class classifier for novelty detection［C］∥Proc of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2018:3379-3388.
［26］	Wu P,Liu J,Shen F.A deep one-class neural network for anomalous event detection in complex scenes［J］.IEEE Transactions on Neural Networks and Learning Systems,2019,31(7):2609-2622.
［27］	Narasimhan M G,Kamath S.Dynamic video anomaly detection and localization using sparse denoising autoencoders［J］.Multimedia Tools and Applications,2018,77(11):13173-13195.
［28］	Ionescu R T,Khan F S,Georgescu M I,et al.Object-centric auto-encoders and dummy anomalies for abnormal event detection in video［C］∥Proc of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2019:7842-7851.
［29］	Xu K,Sun T F,Jiang X H.Video anomaly detection and localization based on an adaptive intra-frame classification network［J］.IEEE Transactions on Multimedia,2019,22(2):394-406.
［30］	Sultani W,Chen C,Shah M.Real-world anomaly detection in surveillance videos［C］∥Proc of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2018:6479-6488.
［31］	Kamoona A M,Gosta A K,Bab-Hadiashar A,et al.Multiple instance-based video anomaly detection using deep temporal encoding-decoding［J］.arXiv:2007.01548,2020.
［32］	Zhu Y,Newsam S.Motion-aware feature for improved video anomaly detection［J］.arXiv:1907.10211,2019.
［33］	Pang G S,Yan C,Shen C H,et al.Self-trained deep ordinal regression for end-to-end video anomaly detection［C］∥Proc of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:12173-12182.
［34］	Bird N,Atev S,Caramelli N,et al.Real time,online detection of abandoned objects in public areas［C］∥Proc of 2006 IEEE International Conference on Robotics and Automation,2006:3775-3780.
［35］	Adam A,Rivlin E,Shimshoni I,et al.Robust real-time unusual event detection using multiple fixed-location monitors［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence,2008,30(3):555-560.
［36］	Mahadevan V,Li W X,Bhalodia V,et al.Anomaly detection in crowded scenes［C］∥Proc of 2010 IEEE Conference on Computer Vision and Pattern Recognition,2010:1975-1981.
［37］	Lu C W,Shi J P,Jia J Y.Abnormal event detection at 150 FPS in matlab［C］∥Proc of IEEE International Conference on Computer Vision,2013:2720-2727.
［38］	Ramachandra B,Jones M.Street Scene:A new dataset and evaluation protocol for video anomaly detection［C］∥Proc of IEEE/CVF Winter Conference on Applications of Computer Vision,2020:2569-2578.
［39］	Rodrigues R,Bhargava N,Velmurugan R,et al.Multi-timescale trajectory prediction for abnormal human activity detection［C］∥Proc of IEEE/CVF Winter Conference on Applications of Computer Vision,2020:2626-2634.
［40］	Xu D,Ricci E,Yan Y,et al.Learning deep representations of appearance and motion for anomalous event detection［J］.arXiv:1510.01553,2015.
［41］	Yan S Y,Smith J S,Lu W J,et al.Abnormal event detection from videos using a two-stream recurrent variational autoencoder［J］.IEEE Transactions on Cognitive and Developmental Systems,2018,12(1):30-42.
［42］	Zhao Y R,Deng B,Shen C,et al.Spatio-temporal autoencoder for video anomaly detection［C］∥Proc of the 25th ACM International Conference on Multimedia,2017:1933-1941.
［43］	Chong Y S,Tay Y H.Abnormal event detection in videos using spatiotemporal autoencoder［C］∥Proc of the International Symposium on Neural Networks,2017:189-196.
［44］	Tran H T M,Hogg D.Anomaly detection using a convolutional winner-take-all autoencoder［C］∥Proc of the British Machine Vision Conference,2017:1-12.
［45］	Ravanbakhsh M,Nabi M,Sangineto E,et al.Abnormal event detection in videos using generative adversarial nets［C］∥Proc of 2017 IEEE International Conference on Image Processing (ICIP),2017:1577-1581.
［46］	Sun J Y,Wang X Z,Xiong N X,et al.Learning sparse representation with variational auto-encoder for anomaly detection［J］.IEEE Access,2018,6:33353-33361.
［47］	Zhou J T,Du J,Zhu H,et al.AnomalyNet:An anomaly detection network for video surveillance［J］.IEEE Transactions on Information Forensics and Security,2019,14(10):2537-2550.
［48］	Song H,Sun C,Wu X X,et al.Learning normal patterns via adversarial attention-based autoencoder for abnormal event detection in videos［J］.IEEE Transactions on Multimedia,2019,22(8):2138-2148.
［49］	Li Y Y,Cai Y H,Liu J Q,et al.Spatio-temporal unity networking for video anomaly detection［J］.IEEE Access,2019,7:172425-172432.
［50］	Zhou J T,Zhang L,Fang Z,et al.Attention-driven loss for anomaly detection in video surveillance［J］.IEEE Transactions on Circuits and Systems for Video Technology,2019,30(12):4639-4647.
［51］	Sabokrou M,Fayyaz M,Fathy M,et al.Deep-cascade:Cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes［J］.IEEE Transactions on Image Processing,2017,26(4):1992-2004.
［52］	Sabokrou M,Fayyaz M,Fathy M,et al.Deep-anomaly:Fully convolutional neural network for fast anomaly detection in crowded scenes［J］.Computer Vision and Image Understanding,2018,172:88-97.
［53］	Sabzalian B,Marvi H,Ahmadyfard A.Deep and sparse features for anomaly detection and localization in video［C］∥Proc of the 4th International Conference on Pattern Recognition and Image Analysis (IPRIA),2019:173-178.
［54］	Fan Y X,Wen G J,Li D R,et al.Video anomaly detection and localization via Gaussian mixture fully convolutional variational autoencoder［J］.Computer Vision and Image Understanding,2020,195:102920.
［55］	Doshi K,Yilmaz Y.Continual learning for anomaly detection in surveillance videos［C］∥Proc of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:254-255.
［56］	Vu H,Phung D,Nguyen T D,et al.Energy-based models for video anomaly detection［J］.arXiv:1708.05211,2017.
［57］	Zhang Y Y,Zhou D S,Chen S Q,et al.Single-image crowd counting via multi-column convolutional neural network［C］∥Proc of IEEE Conference on Computer Vision and Pattern Recognition,2016:589-597.
［58］	Wu P,Liu J,Shi Y J,et al.Not only look,but also listen:Learning multimodal violence detection under weak supervision［C］∥Proc of the European Conference on Computer Vision,2020:322-339.
	附中文参考文献：
［1］	卢博文.基于深度学习的监控视频中的异常行为的检测算法研究［D］.南京:南京邮电大学,2020.
［2］	王思齐,胡婧韬,余广,等.智能视频异常事件检测方法综述［J］.计算机工程与科学,2020,42(8):1393-1405.
［4］	周培培,丁庆海,罗海波,等.视频监控中的人群异常行为检测与定位［J］.光学学报,2018,38(8):97-105.
［5］	李娟,张冰怡,冯志勇,等.基于隐马尔可夫模型的视频异常场景检测［J］.计算机工程与科学,2017,39(7):1300-1308.
［8］	彭嘉丽,赵英亮,王黎明.基于深度学习的视频异常行为检测综述［J］.激光与光电子学进展,2020,58(6):0600004.
［9］	胡正平,张乐,李淑芳,等.视频监控系统异常目标检测与定位综述［J］.燕山大学学报,2019,43(1):1-12.
［10］	王志国,章毓晋.监控视频异常检测:综述［J］.清华大学学报(科学与技术),2020,60(6):518-529.

[1]	马思远, 焦佳辉, 任晟岐, 宋伟. 基于注意力机制的城市多元空气质量数据缺失值填充[J]. 计算机工程与科学, 2023, 45(08): 1354-1364.
[2]	尹春勇, 冯梦雪. 基于注意力机制的半监督日志异常检测方法[J]. 计算机工程与科学, 2023, 45(08): 1405-1415.
[3]	易啸, 马胜, 肖侬. 深度学习加速器在不同剪枝策略下的运行优化[J]. 计算机工程与科学, 2023, 45(07): 1141-1148.
[4]	康宇晗, 时洋, 陈照云, 文梅. 面向迈创+MatrixZone异构系统的深度学习编程框架[J]. 计算机工程与科学, 2023, 45(07): 1149-1158.
[5]	刘浩翰, 孙铖, 贺怀清, 惠康华. 基于改进YOLOv3的金属表面缺陷检测[J]. 计算机工程与科学, 2023, 45(07): 1226-1235.
[6]	田秀霞, 刘正, 刘秋旭, 李浩然. 一种改进Faster R-CNN的图像篡改检测模型[J]. 计算机工程与科学, 2023, 45(06): 1030-1039.
[7]	濮子俊, 张寿明. 基于特征融合与Transformer模型的声音事件定位与检测算法研究[J]. 计算机工程与科学, 2023, 45(06): 1097-1105.
[8]	邓姗姗, 黄慧, 马燕. 基于改进Faster R-CNN的小目标检测算法[J]. 计算机工程与科学, 2023, 45(05): 869-877.
[9]	霍爱清, 张书涵, 杨玉艳, 胥静蓉, 王泽文. 密集交通场景中改进YOLOv3目标检测优化算法[J]. 计算机工程与科学, 2023, 45(05): 878-884.
[10]	方雪杉, 杨云飞, 冯松. 基于多尺度多模态学习的光球亮点曲线轨迹段检测方法研究[J]. 计算机工程与科学, 2023, 45(05): 885-894.
[11]	黄星威, 陈曦, 张塑凡. 改进特征金字塔的小目标深度学习模型[J]. 计算机工程与科学, 2023, 45(04): 734-742.
[12]	董佩杰, 牛新, 魏自勉, 陈学晖. 单次神经网络结构搜索研究综述[J]. 计算机工程与科学, 2023, 45(02): 191-203.
[13]	强梓林, 刘建国, 刘云峰, 卫栋, 强彦. 基于时域-频域哈希编码的电网图像检索方法[J]. 计算机工程与科学, 2022, 44(10): 1877-1884.
[14]	杨斌, 王敬宇, 刘世超, 邵明山, 肖伟, 陈起, 何晓斌, 刘卫国, 薛巍, . Beacon+：面向E级超级计算机的轻量级端到端I/O性能监控与分析诊断系统[J]. 计算机工程与科学, 2022, 44(09): 1521-1531.
[15]	刘从军, 徐佳陈, 肖志勇, 柴志雷. 基于深度学习的心脏核磁共振图像自动分割算法[J]. 计算机工程与科学, 2022, 44(09): 1646-1654.