Research and Comparison of Intelligent Detection Methods of Pavement Distress Based on Deep Data Augmentation

HOU Yue; ZHANG Huiting; GAO Zhiwei; WANG Dawei; LIU Pengfei; OESER Markus; WANG Linbing; CHEN Ning

doi:10.11936/bjutxb2021110004

Journal of Beijing University of Technology > 2022 > 48(6): 622-634. > DOI: 10.11936/bjutxb2021110004

HOU Yue, ZHANG Huiting, GAO Zhiwei, WANG Dawei, LIU Pengfei, OESER Markus, WANG Linbing, CHEN Ning. Research and Comparison of Intelligent Detection Methods of Pavement Distress Based on Deep Data Augmentation[J]. Journal of Beijing University of Technology, 2022, 48(6): 622-634. DOI: 10.11936/bjutxb2021110004

Citation:

PDF (18055 KB)

Research and Comparison of Intelligent Detection Methods of Pavement Distress Based on Deep Data Augmentation

1.
Faculty of Architecture, Civil and Transportation Engineering, Beijing University of Technology, Beijing 100124, China
2.
James Watt School, Engineering University of Glasgow, Glasgow G128QQ, UK
3.
Institute of Highway Engineering, RWTH Aachen University, Aachen D52074, Germany
4.
Road Research Institute of the Federal Ministry of Transport, Nordrhein-westfaten 51427, Germany
5.
Department of Civil and Environmental Engineering, Virginia Polytechnic Institute and State University, Blacksburg VA24061, USA
6.
Toyota Transportation Research Institute, Toyota 471-0024, Japan

More Information

Received Date: November 07, 2021
Revised Date: December 02, 2021
Available Online: August 03, 2022
Published Date: June 09, 2022

Graphical Abstract

Abstract

Abstract

To solve the time-consuming problem of manual pavement distress detection and the possible low detection accuracy problem due to unbalanced sample dataset, a method of deep data augmentation was employed to enhance the dataset of high-definition road images taken by smartphones. The results after the data augmentation were evaluated and tested by using two different target detection algorithms. The main research contents of the paper included. First, considering the limitations of experimental conditions and acquisition environment, a deep data augmentation method was employed by combining WGAN-GP and Poisson transfer algorithm, which supplemented and balanced the training sample data by generating road pothole images under different lighting conditions. Then, multiple target detection algorithm variants of Yolo(Yolov5s, Yolov5m, Yolov5l, and Yolov5x) and Faster R-CNN algorithm were introduced, and the accuracy and efficiency of various target detection algorithms after applying the data augmentation were compared through experiments. Experimental results on the Japanese open road detection dataset show that the average improvement of P, R and F1 of five detection algorithms is 2.8%, 4.0% and 3.6%, respectively, after using the deep data augmentation method. Among the five detection algorithms, Yolov5l achieved the highest F1 value, reaching 60.9%. If conditions are suitable, such as in the test set with moderate light conditions, the F1 value of Yolov5l algorithm can reach 68.7%.
- road engineering,
- pavement distress,
- deep learning,
- convolutional neural network,
- data augmentation,
- target detection

FullText(HTML)

References (35)

References

[1]	刘红遍. 基于神经网络的高速公路沥青路面预防性养护预测模型研究[D]. 长安: 长安大学, 2015. LIU H B. Research on preventive maintenance prediction model of expressway asphalt pavement based on neural network[D]. Chang'an: Chang'an University, 2015. (in Chinese)
[2]	白日华. 沥青路面病害检测与养护决策研究[D]. 吉林: 吉林大学, 2013. BAI R H. Research onasphalt pavement disease detection and maintenance decision[D]. Jilin: Jilin University, 2013. (in Chinese)
[3]	薛佳瑶, 陈海勇, 周刚. 基于卷积循环神经网络的城市区域车流量预测模型[J]. 信息工程大学学报, 2019, 20(2): 236-241. doi: 10.3969/j.issn.1671-0673.2019.02.019 XUE J Y, CHEN H Y, ZHOU G. Traffic flow prediction model based on convolutional recurrent neural network[J]. Journal of Information Engineering University, 2019, 20(2): 236-241. (in Chinese) doi: 10.3969/j.issn.1671-0673.2019.02.019
[4]	GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C. : IEEE Computer Society, 2014: 580-587.
[5]	HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2015, 37(9): 1904-1916. https://arxiv.org/abs/1406.4729
[6]	GIRSHICK R. Fast R-CNN[C]//2015 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 1440-1448.
[7]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031
[8]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 779-788.
[9]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multi box detector[C]//2016 European Conference on Computer Vision. Berlin: Springer, 2016: 21-37.
[10]	MAEDA H, SEKIMOTO Y, SETO T, et al. Road damage detection and classification using deep neural networks with smartphone images[J]. Computer-Aided Civil and Infrastructure Engineering, 2018, 33(12): 1127-1141. doi: 10.1111/mice.12387
[11]	WANG W X, WANG M F, LI H X, et al. Pavement crack image acquisition methods and crack extraction algorithms: a review[J]. Journal of Traffic and Transportation Engineering, 2019, 6(6): 535-556. https://www.sciencedirect.com/science/article/pii/S2095756419303010
[12]	肖创柏, 柏鳗晏, 禹晶. 基于Faster R-CNN的缝隙检测与提取算法[J]. 北京工业大学学报, 2021, 47(2): 135-146. https://www.cnki.com.cn/Article/CJFDTOTAL-BJGD202102006.htm XIAO C B, BAI M Y, YU J. Crack detection and extraction based on Faster R-CNN[J]. Journal of Beijing University of Technology, 2021, 47(2): 135-146. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-BJGD202102006.htm
[13]	STANIEK M. Road pavement condition diagnostics using smartphone-based data crowdsourcing in smart cities[J]. Journal of Traffic and Transportation Engineering(English Edition), 2021, 8(4): 554-567. doi: 10.1016/j.jtte.2020.09.004
[14]	张宁. 基于Faster R-CNN的公路路面病害检测算法的研究[D]. 南昌: 华东交通大学, 2019. ZHANG N. Research on highway pavement disease detection algorithm based on Faster R-CNN[D]. Nanchang: East China Jiaotong University, 2019. (in Chinese)
[15]	ZHANG J, CHEN Z Q. Pixel-level crack delineation in images with convolutional feature fusion[J]. Structural Control and Health Monitoring, 2019, 26(1): 2286-2291. doi: 10.1002/stc.2286
[16]	GULAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of wasserstein gans[C]//Proceedings of the 31^stInternational Conference on Neural Information Processing Systems. New York: Curran Associate Inc, 2017: 5769-5779.
[17]	MAEDA H, SEKIMOTO Y, SETO T. Lightweight road manager: smartphone-based automatic determination of road damage status by deep neural network[C]//5th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. New York: Association for Computing Machinery, 2016: 37-45.
[18]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. Massachusetts: MIT Press, 2014: 2672-2680.
[19]	RATLIFF L J, BURDEN S A, SASTRY S. Characterization and computation of local nash equilibria in continuous games[C]//2013 51st Annual Allerton Conference on Communication, Control, and Computing. Piscataway: IEEE, 2013: 917-924.
[20]	黄攀, 杨小冈, 卢瑞涛, 等. 基于GAN的红外飞机数据增强方法[J]. 电光与控制, 2021, 28(11): 84-88. doi: 10.3969/j.issn.1671-637X.2021.11.018 HUANG P, YANG X G, LU R T, et al. Infrared aircraft data enhancement method based on GAN[J]. Electronics Optics & Control, 2021, 28(11): 84-88. (in Chinese) doi: 10.3969/j.issn.1671-637X.2021.11.018
[21]	ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. Massachusetts: MIT Press, 2017: 214-223.
[22]	MAEDA H, KASHIYAMA T, SEKIMOTO Y, et al. Generative adversarial networks for road damage detection[J]. Computer-Aided Civil and Infrastructure Engineering, 2020, 36(1): 47-60. https://www.researchgate.net/publication/341836638_Generative_adversarial_network_for_road_damage_detection
[23]	REINHARD E, ADHIKHMIN M, GOOCH B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41. https://ieeexplore.ieee.org/abstract/document/946629
[24]	中国软件开发网. 无缝融合两张图片——cv2. seamlessClone()泊松融合[CP/OL]. (2021-05-13)[2021-10-08]. Https://blog.csdn.net/AugustMe/article/details/116747204.
[25]	中国软件开发网. Python Opencv实现Reinhard颜色迁移算法[CP/OL]. (2020-12-11)[2021-10-11]. Https://blog.csdn.net/weixin_30466953/article/details/98125671.
[26]	尉迟姝毅. 基于反向映射的图像间颜色迁移算法仿真[J]. 计算机仿真, 2021, 38(1): 212-216. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJZ202101044.htm YUCHI S Y. Simulation of color transfer algorithm between images based on reverse mapping[J]. Computer Simulation, 2021, 38(1): 212-216. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JSJZ202101044.htm
[27]	NEUBEC K, GOOL L V. Efficient non-maximum suppression[C]//18th International Conference on Pattern Recognition. Piscataway: IEEE, 2006: 850-855.
[28]	BEKER D. Faster-RCNN-TensorFlow-Python3[CP/OL]. (2019-07-31)[2021-04-12]. Https://github.com/dBeker/Faster-RCNN-TensorFlow-Python3.
[29]	王思雨. 一种基于Yolo的交通目标实时检测方法[J]. 计算机与数字工程. 2020, 48(9): 2162-2166. doi: 10.3969/j.issn.1672-9722.2020.09.017 WANG S Y. A real-time traffic target detection method based on Yolo[J]. Computer and Digital Engineering, 2020, 48(9): 2162-2166. (in Chinese) doi: 10.3969/j.issn.1672-9722.2020.09.017
[30]	ULTRALYTICS. Yolov5[CP/OL]. (2021-01-13)[2021-04-15]. Https://github.com/ultralytics/yolov5.
[31]	LIU Z, WU W X, GU X Y. Application of combining Yolo models and 3D GPR images in road detection and maintenance[J]. Remote Sensing, 2021, 13(6): 1-18. https://www.researchgate.net/publication/350035261_Application_of_Combining_YOLO_Models_and_3D_GPR_Images_in_Road_Detection_and_Maintenance
[32]	REZATOFIGI H, TSOI N, GWAK J Y, et al. Generalized intersection over union: a metric and a loss for bounding box regression[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 658-666,
[33]	ZHENG Z H, WANG P, LIU W, et al. Distance-IOU loss: faster and better learning for bounding box regression[C]//34th AAAI Conference on Artificial Intelligence. California: AAAI, 2020: 12993-13000.
[34]	何文轩, 胡健, 柳小波. 矿石块度视觉识别判断方法[J]. 中国矿业, 2021, 30(6): 100-105. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGKA202106017.htm HE W X, HU J, LIU X B. Visual identification and judgment method of ore lumpiness[J]. China Mining, 2021, 30(6): 100-105. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-ZGKA202106017.htm
[35]	LI S, GU X Y, XU X R, et al. Detection of concealed cracks from ground penetrating radar images based on deep learning algorithm[J]. Construction and Building Materials, 2021, 273(1): 121949. https://www.sciencedirect.com/science/article/pii/S0950061820339532

[1]	REN Kun, LI Pan, HAN Honggui. Object Detection in Challenging Environments via Dual-scale CBAM Feature Fusion of mmWave Radar and Vision[J]. Journal of Beijing University of Technology, 2025, 51(3): 284-294. DOI: 10.11936/bjutxb2023070003
[2]	RUAN Xiaogang, ZHOU Chen, HUANG Jing. Semantic Visual SLAM Based on Target Detection Algorithm in Indoor Dynamic Scenes[J]. Journal of Beijing University of Technology, 2023, 49(8): 842-850. DOI: 10.11936/bjutxb2021090025
[3]	FANG Juan, FANG Zhenhu. Vision SLAM Optimization in Dynamic Scene Based on Object Detection Network[J]. Journal of Beijing University of Technology, 2022, 48(5): 466-475. DOI: 10.11936/bjutxb2021020005
[4]	YUAN Jiaojiao, HU Yongli, SUN Yanfeng, YIN Baocai. Survey of Small Object Detection Methods Based on Deep Learning[J]. Journal of Beijing University of Technology, 2021, 47(3): 293-302. DOI: 10.11936/bjutxb2020090019
[5]	JIA Yanli, PEI Liang, SUN Xu, GAO Lianru. Application of Sparse Representation Model in Target Detection of Hyperspectral Remote Sensing Image[J]. Journal of Beijing University of Technology, 2017, 43(5): 691-700. DOI: 10.11936/bjutxb2016070017
[6]	LIU Yanheng, GAO Siwei, WANG Jian, DENG Weiwen. Design and Implementation of Target Detection System for Miniature Intelligent Vehicles[J]. Journal of Beijing University of Technology, 2016, 42(10): 1509-1518. DOI: 10.11936/bjutxb2016030027
[7]	GAO Jing, CAI Xingfu, LIU Zhiqiang, CHANG Yan. Method of Target Detection Based on Region Growing[J]. Journal of Beijing University of Technology, 2016, 42(6): 856-861. DOI: 10.11936/bjutxb2015050002
[8]	GAO Jing, SUN Ji-yin, WU Kun, LIU Jing. FLIR Target Detection Algorithm Based on Shape Template Matching[J]. Journal of Beijing University of Technology, 2012, 38(9): 1359-1365. DOI: 10.3969/j.issn.0254-0037.2012.09.015
[9]	XU Dong-bin, GE Tao, XIAO Chuang-bai, HUANG Lei. Combining Motion and Statistical Features for Static Object Detection[J]. Journal of Beijing University of Technology, 2012, 38(7): 1079-1086. DOI: 10.3969/j.issn.0254-0037.2012.07.022
[10]	DU Jian-jun, LU Jian-rong, QIAO Ai-ke, LIU You-jun. Automatic Segmentation of Tomographic Images Based on Object Detecting and Region Growing Algorithms[J]. Journal of Beijing University of Technology, 2010, 36(4): 566-571. DOI: 10.3969/j.issn.0254-0037.2010.04.024

Cited By

Cited by

Periodical cited type(11)

1.	刘鹏宇，袁静，高倩，陈善继. 基于改进YOLOv5的路面病害检测方法. 北京工业大学学报. 2025(05): 552-559 . 本站查看
2.	庞荣，杨燕，冷雄进，张朋，刘言. 基于双分支点流语义先验的路面病害分割模型. 智能系统学报. 2024(01): 153-164 .
3.	王国忠，陈明星，姚辉，曹丹丹. 图像灰度处理对路面裂缝病害检测影响分析. 市政技术. 2024(04): 270-277 .
4.	崔闯，罗纯坤，邱师津，张清华. 基于数据深度增强的钢桥螺栓脱落智能检测方法研究. 桥梁建设. 2024(02): 39-47 .
5.	翁广良，刘冬生，吴送英，万强华. 基于改进的YOLOv7混凝土路面裂缝检测算法研究. 中国公路. 2024(04): 96-99 .
6.	姚楚羡，蔡皓楠，张远波，唐可懿，詹璐，周宝定. 基于轻量化车载设备的道路病害检测方法. 测绘通报. 2024(05): 147-150 .
7.	者甜甜，赵新旭，顾宙瑜，张博熠，刘庆华. 一种基于YOLOX优化的轻量级路面病害检测方法. 江苏科技大学学报(自然科学版). 2024(03): 55-62 .
8.	李兴久，冯增文，荆虹波. 改进的卷积神经网络在地铁保护区无人机安全巡查中的应用. 北京测绘. 2024(10): 1412-1417 .
9.	赫英策，李禹萱，孙尚宇，宋伟东. 基于改进YOLOv8的前视影像的路面病害检测方法. 时空信息学报. 2024(05): 605-617 .
10.	倪昌双，李林，罗文婷，秦勇，杨振，傅幼华. 改进YOLOv7的沥青路面病害检测. 计算机工程与应用. 2023(13): 305-316 .
11.	马正宇. 基于无人机影像的公路桥梁裂缝智能检测方法. 工程机械与维修. 2023(04): 268-270 .

Other cited types(7)

Get Citation

PDF

XML

Article views (385) PDF downloads (73) Cited by(18)

Turn off MathJax

Article Contents

Abstract

References

Research and Comparison of Intelligent Detection Methods of Pavement Distress Based on Deep Data Augmentation