基于深度双线性转换注意力机制网络的林业有害生物识别方法

doi:10.11707/j.1001-7488.LYKX20210787

摘要/Abstract

摘要：

目的: 针对林业有害生物种类多，不少物种之间相似度高，视觉差异小，不易区分，导致林业防控人员无法快速准确识别有害生物种类的问题，本文提出基于深度双线性转换注意力机制网络（DBTANet）的林业有害生物细粒度图像识别方法。方法: 以自然状态下拍摄的60种林业害虫和14种林业有害植物图像作为研究对象，利用水平镜像、亮度调节、高斯模糊和高斯噪声等方法对图像数据集进行增强，按6∶2∶2比例划分为训练集、验证集和测试集；采用双线性插值法将每幅图像缩放至统一尺寸；改进ResNet网络中残差模块，加入深度双线性转换模块和注意力机制模块，建立DBTANet-101网络进行特征提取与分类；利用平均准确率、平均召回率和平均F₁值3个指标评价不同模型对林业有害生物的识别结果。结果: VGGNet-19、ResNet-50、ResNet-101、改进残差模块的ResNet-50和ResNet-101共5个模型对74种林业有害生物平均准确率分别为78.6%、74.9%、76.3%、79.7%和81.1%；在改进残差模块的ResNet-101基础上，增加深度双线性转换模块和注意力机制模块后，74种林业有害生物的平均准确率和召回率分别提高了10.2%和12.1%，22种相似的有害生物细粒度图像平均准确率提高了15.7%。结论: 基于深度双线性转换注意力机制网络（DBTANet）的林业有害生物细粒度图像识别方法，对74种林业有害生物和22种相似有害生物细粒度图像的平均准确率分别为91.3%和85.1%；双线性转换模块和注意力机制可以有效提高林业有害生物识别模型的准确率。

关键词: 林业有害生物, 细粒度图像识别, 深度双线性转换, 注意力机制, ResNet

Abstract:

Objective: There are many forest pest species, and some of them are morphologically similar with small visual differences, and are difficult to be distinguished, which makes it difficult for forestry prevention and control technicians to quickly and accurately identify them. In regard to those problems, this paper proposes a fine-grained image identification method of forest pests based on deep bilinear transformation attention mechanism network (DBTANet). Method: The images of 60 forest pests and 14 harmful forest plants were targeted. The original images were enhanced by methods including horizontal mirror, brightness adjustment, Gaussian blur and Gaussian noise. All images were divided into training dataset, verification dataset and test dataset by a 6∶2∶2 ratio. The bilinear interpolation method was used to scale each image to a uniform size. The residual module in the ResNet network was improved. A deep bilinear conversion module and an attention mechanism module were added. DBTANet-101 model was developed to extract features and classify forest pest species. Three evaluation indexes of average accuracy, average recall rate and average F₁ score were used to evaluate the identification results of forestry pests by thd different models. Result: A total of 74 forest pest species were identified by five models of VGGNet-19, ResNet-50, ResNet-101, improved ResNet-50 and improved ResNet-101, respectively, and the average accuracy rates of the five models were 78.6%, 74.9%, 76.3%, 79.7% and 81.1%, respectively. On the basis of improved ResNet-101 of the residual module, the deep bilinear conversion module and the attention mechanism module were added, after then the average recognition rate and recall rate of 74 forestry pests increased by 10.2% and 12.1%, respectively. The average recognition rate of fine-grained images of 22 similar pests increased by 15.7%. Conclusion: The fine-grained image recognition method of forest pests based on deep bilinear transformation attention mechanism network (DBTANet) achieves the average identification rate of 74 forest pests by 91.3% and the average identification rate of 22 similar forest pest species by 85.1%. The bilinear transformation module and the attention mechanism can effectively improve the accuracy of the forest pest identification model.

Key words: forest pests, fine-grained image recognition, deep bilinear transformation, attention mechanism, ResNet

中图分类号:

苏佳杰,张哲宇,徐嘉俊,李彬,吕军,姚青. 基于深度双线性转换注意力机制网络的林业有害生物识别方法[J]. 林业科学, 2023, 59(2): 121-128.

Jiajie Su,Zheyu Zhang,Jiajun Xu,Bin Li,Jun Lü,Qing Yao. Forest Pest Identification Method Based on a Deep Bilinear Transformation Attention Mechanism Network[J]. Scientia Silvae Sinicae, 2023, 59(2): 121-128.

图/表 9

图1

74种林业有害生物图像 1. 斑凤蝶Chilasa clytia; 2. 碎斑青凤蝶Graphium chironides; 3. 黑脉蛱蝶Hestina assimilis; 4. 小红蛱蝶Vanessa cardui; 5. 美眼蛱蝶Junonia almana; 6. 大红蛱蝶Vanessa indica; 7. 榆凤蛾Epicopeia mencia; 8. 麝凤蝶Byasa alcinous; 9. 中华马蜂Polistes chinensis; 10. 黑带食蚜蝇Episyrphus balteatus; 11. 黑尾大叶蝉Bothrogonia ferruginea; 12. 黑尾叶蝉Nephotettix cincticeps; 13. 星天牛Anoplophora chinensis; 14. 光肩星天牛Anoplophora glabripennis; 15. 孔夫子锯锹Prosopocoilas confucius; 16. 扁锹甲Serrognathus titanus; 17. 绿螽蟖Holochlora nawae; 18. 短瓣优草螽Euconocephalus brachyxiphus; 19. 棉蝗Chondracris rosea; 20. 中华草螽Conocephalus chinensis; 21. 日本条螽蟖Ducetia japonica; 22. 东亚飞蝗Locusta migratoria; 23. 老豹蛱蝶Argyronom laodue; 24. 中环蛱蝶Neptis hylas; 25. 白弄蝶Abraximorpha davidii; 26. 蚜灰蝶Taraka hamada; 27. 曲纹蜘蛱蝶Arashnia doris; 28. 丝带凤蝶Sericinus montela; 29. 玉带凤蝶Papilio pdytes; 30. 金裳凤蝶Troides zeacus; 31. 重阳木锦斑蛾Histia rhodope; 32. 青凤蝶Graphium sarpedon; 33. 琉璃蛱蝶Kaniska canacae; 34. 黑弄蝶Daimio tothys; 35. 翠蓝眼蛱蝶Junonia orithya; 36. 白带螯蛱蝶Chayaxex bemardus; 37. 稻眉眼蝶Mycalesis gotama; 38. 直纹稻弄蝶Parnara gnnata; 39. 乌桕大蚕蛾Attacus atlas; 40. 甘薯腊龟甲Laccoptera quadrimaculata; 41. 绿灰蝶Artipe eryx; 42. 暗脉菜粉蝶Pieris napi; 43. 绢粉蝶Aporia crataegi; 44. 琉璃灰蝶Celastrina argiolus; 45. 美国白蛾Hyphantria cunea; 46. 咖啡透翅天蛾Cephonodes hylas; 47. 青背长喙天蛾Macroglos bomylans; 48. 苎麻双脊天牛Paraglenea fortunei; 49. 黄星天牛Psacothea hilaris; 50. 桑天牛Apriona germari; 51. 东方蝼蛄Gryllotalpa orientalis; 52. 紫茎甲Sagra femorata; 53. 中华蟋蟀Gryllus chinensis; 54. 黑额光叶甲Smaragdina nigrifrons; 55. 十星瓢萤叶甲Oides decempunc; 56. 豆芫菁Epicauta gorhami; 57. 长肩棘缘蝽Cletus trigonus; 58. 红脊长蝽Tropidothorax sinensis; 59. 透明疏广蜡蝉Euricaniaclara kato; 60. 黄杨绢野螟Diaphania perspectalis; 61. 菟丝子Cuscuta chinensis; 62. 羊蹄Rumex japonicus; 63. 葎草Humulus scandens; 64. 平车前Plantago depressa; 65. 豚草Ambrosia artemisiifolia; 66. 薜荔Ficus pumila; 67. 白茅Imperata cylindrica; 68. 桑寄生Taxillus sutchuenensis; 69. 络石Trachelospermum jasminoides; 70. 喜旱莲子草Alternanthera philoxeroides; 71. 野葛Pueraria lobata; 72. 五爪金龙Ipomoea cairica; 73. 凤眼莲Eichhornia crassipes; 74. 牛茄子Solanum capsicoides."

图1

图2

图3

图4

图5

表1

表2

图6

图7

参考文献 0

	陈　娟, 陈良勇, 王生生, 等基于改进残差网络的园林害虫图像识别. 农业机械学报, 2019, 50 (5): 187- 195. doi: 10.6041/j.issn.1000-1298.2019.05.022
	Chen J, Chen L Y, Wang S S, et al Pest image recognition of garden based on improved Residual Network. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50 (5): 187- 195. doi: 10.6041/j.issn.1000-1298.2019.05.022
	陈志威. 2017. 广州市林业有害生物普查. 广州 : 华南农业大学.
	Chen Z W. 2017. Investigation of forestry pest in Guangzhou city. Guangzhou: South China Agricultural University. [in Chinese]
	程　曦, 吴云志, 张友华, 等基于深度卷积神经网络的储粮害虫图像识别. 中国农学通报, 2018, 34 (1): 154- 158. doi: 10.11924/j.issn.1000-6850.casb16110146
	Cheng X, Wu Y Z, Zhang Y H, et al Image recognition of stored grain pests: based on Deep Convolutional Neural Network. Chinese Agricultural Science Bulletin, 2018, 34 (1): 154- 158. doi: 10.11924/j.issn.1000-6850.casb16110146
	高巨虎, 王桂林, 李慧杰, 等自动虫情测报灯在林业害虫测报中的应用. 中国森林病虫, 2011, 30 (3): 36- 39. doi: 10.3969/j.issn.1671-0886.2011.03.011
	Gao J H, Wang G L, Li H J, et al Application of automatic forecast light in the forecast forest pests. Forest Pest and Disease, 2011, 30 (3): 36- 39. doi: 10.3969/j.issn.1671-0886.2011.03.011
	孔建磊, 金学波, 陶　治, 等基于多流高斯概率融合网络的病虫害细粒度识别. 农业工程学报, 2020, 36 (13): 148- 157. doi: 10.11975/j.issn.1002-6819.2020.13.018
	Kong J L, Jin X B, Tao Z, et al Fine-grained recognition of diseases and pests based on multi-stream Gaussian probability fusion network. Transactions of the Chinese Society of Agricultural Engineering, 2020, 36 (13): 148- 157. doi: 10.11975/j.issn.1002-6819.2020.13.018
	李珏闻, 许胜利, 王立宇, 等应用云计算和物联网技术促进林草有害生物监测预报智慧发展. 林业和草原机械, 2020, 1 (3): 1- 6.
	Li J W, Xu S L, Wang L Y, et al Application of cloud computing and internet of things technology promoting intelligent development of monitoring and forecasting forest and grass pests. Forestry And Grassland Machinery, 2020, 1 (3): 1- 6.
	李孝华. 2020. 基于卷积神经网络的蝴蝶识别系统设计与实现. 重庆 : 西南大学.
	Li X H. 2020. The classification of butterfly based on Convolutional Neural Network. Chongqing : Southwest University. [in Chinese]
	林　强福建省林业有害生物灾害航空监测作业设计与效果评估系统. 福建农林大学学报 (自然科学版), 2015, 44 (2): 165- 168.
	Lin Q Route design and impact assessment system for forest pest aviation monitoring in Fujian Province. Journal of Fujian Agriculture and Forestry University (Natural Science Edition), 2015, 44 (2): 165- 168.
	刘文定, 田洪宝, 谢将剑, 等基于全卷积神经网络的林区航拍图像虫害区域识别方法. 农业机械学报, 2019, 50 (3): 179- 183. doi: 10.6041/j.issn.1000-1298.2019.03.019
	Liu W D, Tian H B, Xie J J, et al Identification methods for Forest Pest areas of UAV aerial photography based on Fully Convolutional Networks. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50 (3): 179- 183. doi: 10.6041/j.issn.1000-1298.2019.03.019
	吕玫霞. 2020. 无人机林地图像传输APP及虫害检测系统. 北京 : 北京林业大学.
	Lü M X. 2020. Research and development of UAV Forest Images transmission APP and pest detection system. Beijing : Beijing Forestry University. [in Chinese]
	聂丽郦. 2019. 基于卷积神经网络的榆紫叶甲虫识别研究. 哈尔滨 : 东北林业大学.
	Nie L Y. 2019. Identification and research of the Ambrostoma quadriimpressum motschlsky based on the Convolutional Neural Network. Harbin : Northeast Forestry University. [in Chinese]
	秦　放. 2018. 基于深度学习的昆虫图像识别研究. 成都: 西南交通大学.
	Qing F. 2018. Research on insect image recognition based on Deep Learning. Chengdu : Southwest Jiaotong University. [in Chinese]
	邵亚奎, 王　蕾, 朱长明, 等 GEE云平台支持下的西天山森林遥感监测与时空变化分析. 测绘通报, 2020, 2020 (8): 13- 17.
	Shao Y K, Wang L, Zhu C M, et al Forest survey and spatio-temporal analysis in West Tianshan mountains supported by Google Earth Engine. Bulletin of Surveying and Mapping, 2020, 2020 (8): 13- 17.
	史冰莹, 李佳琦, 张　磊, 等基于CNN的农作物病虫害图像识别模型. 计算机系统应用, 2020, 29 (6): 89- 96. doi: 10.15888/j.cnki.csa.007471
	Shi B Y, Li J Q, Zhang L, et al Recognition model of crop pests and diseases images based on CNN. Computer Systems & Applications, 2020, 29 (6): 89- 96. doi: 10.15888/j.cnki.csa.007471
	孙忠秋, 吴发云, 高显连, 等基于机载大光斑激光雷达的森林冠层高度估测. 林业资源管理, 2020, (3): 111- 117.
	Sun Z Q, Wu F Y, Gao X L, et al Estimation of forest canopy height based on large-footprint airborne LiDAR data. Forest Resources Management, 2020, (3): 111- 117.
	王　佳, 张隆裕, 吕春东, 等基于地面激光雷达点云数据的树种识别方法. 农业机械学报, 2018, 49 (11): 180- 188. doi: 10.6041/j.issn.1000-1298.2018.11.021
	Wang J, Zhang R Y, Lv C D, et al Tree species identification methods based on point cloud data using ground-based LiDAR. Transactions of the Chinese Society for Agricultural Machinery, 2018, 49 (11): 180- 188. doi: 10.6041/j.issn.1000-1298.2018.11.021
	王胜梅. 2018. 油松毛虫性诱技术优化及其幼虫空间格局研究. 北京 : 北京林业大学.
	Wang S M. 2018. Study on technology optimization of sex pheromone attractant and larvae spatial distribution of Dendrolimus tabulaeformis. Beijing : Beijing Forestry University. [in Chinese]
	谢　楠. 2017. 明山万亩生态林场林业有害生物调查及主要种类发生规律. 泰安: 山东农业大学.
	Xie N. 2017. Investigation of Forest Pests and occurrence rule of main kinds in MingShan ten-thousand mu ecological forestry farm. Tai'an: Shandong Agricultural University. [in Chinese]
	应新新. 2018. 基于支持向量机的果实蝇图像的自动识别系统研究与实现. 南昌: 江西农业大学.
	Ying X X. 2018. Research and realization of image automatic identification system for Bactrocera based on SVM. Nanchang: Jiangxi Agricultural University. [in Chinese]
	岳春宇, 郑永超, 邢艳秋, 等星载激光遥感林业应用发展研究. 红外与激光工程, 2020, 49 (11): 105- 114.
	Yue C Y, Zheng Y C, Xing Y Q, et al Technical and application development study of space-borne LiDAR in forestry remote sensing. Infrared and Laser Engineering, 2020, 49 (11): 105- 114.
	张超凡, 王儒敬, 谢成军. 2017. 基于多特征字典学习的害虫图像自动分类方法. 计算机应用与软件, 34 (3) : 142−147, 180.
	Zhang C F, Wang R J, Xie C J. 2017. Automatic classification method for Pest Image based on multi-feature dictionary learning. Computer Applications and Software, 34 (3) : 142−147, 180. [in Chinese]
	张广群, 李英杰, 汪杭军基于词袋模型的林业业务图像分类. 浙江农林大学学报, 2017, 34 (5): 791- 797. doi: 10.11833/j.issn.2095-0756.2017.05.004
	Zhang G Q, Li Y J, Wang H J Classification of forestry images based on the BoW Model. Journal of Zhejiang A & F University, 2017, 34 (5): 791- 797. doi: 10.11833/j.issn.2095-0756.2017.05.004
	赵慧颖. 2019. 基于贝叶斯卷积神经网络的害虫图像识别方法. 长春: 吉林大学.
	Zhao Y H. 2019. Pest image identification method based on Bayesian Convolutional Neural Network. Changchun: Jilin University. [in Chinese]
	赵仁辉, 刘　丽, 王志宝, 等基于开源GIS的林业有害生物监测与预警系统. 计算机工程与设计, 2015, 36 (3): 802- 808. doi: 10.16208/j.issn1000-7024.2015.03.048
	Zhao R H, Liu L, Wang Z B, et al Monitoring and warning system for forest pest based on open source GIS. Computer Engineering and Design, 2015, 36 (3): 802- 808. doi: 10.16208/j.issn1000-7024.2015.03.048
	Bahdanau D, Cho K, Bengio Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv : 1409.0473.
	He K, Zhang X, Ren S, et al. 2016. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 770−778.
	Ioffe S, Szegedy C Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning. PMLR, 2015, 37, 448- 456.
	Jaderberg M, Simonyan K, Zisserman A Spatial transformer networks. Advances in neural information processing systems, 2015, 28, 2017- 2025.
	Krizhevsky A, Sutskever I, Hinton G E Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60 (6): 84- 90. doi: 10.1145/3065386
	Qiao X, Li Y Z, Su G Y, et al MmNet: Identifying Mikania micrantha kunth in the wild via a deep Convolutional Neural Network . Journal of Integrative Agriculture, 2020, 19 (5): 1292- 1300. doi: 10.1016/S2095-3119(19)62829-7
	Simonyan K, Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv : 1409. 1556.
	Srivastava N, Hinton G, Krizhevsky A, et al Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014, 15 (1): 1929- 1958.
	Szegedy C, Ioffe S, Vanhoucke V, et al Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31 (1): 4278- 4284.
	Szegedy C, Liu W, Jia Y, et al. 2015. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, 1−9.
	Szegedy C, Vanhoucke V, Ioffe S, et al. 2016. Rethinking the inception architecture for computer vision. Proceedings of the IEEE conference on computer vision and pattern recognition, 2818−2826.
	Wang F, Jiang M, Qian C, et al. 2017. Residual attention network for image classification. Proceedings of the IEEE conference on computer vision and pattern recognition, 3156−3164.
	Xiao T, Xu Y, Yang K, et al. 2015. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. Proceedings of the IEEE conference on computer vision and pattern recognition, 842−850.
	Xin D J, Chen Y W, Li J J Fine-grained butterfly classification in ecological images using squeeze-and-excitation and spatial attention modules. Applied Sciences, 2020, 10 (5): 1681. doi: 10.3390/app10051681
	Ye H J, Han H, Zhu L N, et al Vegetable Pest image recognition method based on improved VGG convolution neural network. Journal of Physics:Conference Series, 2019, 1237 (3): 032018. doi: 10.1088/1742-6596/1237/3/032018
	Zhao B, Wu X, Feng J, et al Diversified visual attention networks for fine-grained object classification. IEEE Transactions on Multimedia, 2017, 19 (6): 1245- 1256. doi: 10.1109/TMM.2017.2648498
	Zheng H, Fu J, Zha Z J, et al Learning deep bilinear transformation for fine-grained image representation. Advances in Neural Information Processing Systems, 2019, 32, 4277- 4286.

模型 Models	平均准确率 Average Precision(%)	平均召回率 Average Recall	平均F₁值 Average F₁-measure
ResNet-50	74.9	72.7	71.5
ResNet-101	76.3	74.9	73.9
改进残差模块的ResNet-50	79.7	77.3	76.8
改进残差模块的ResNet-101	81.1	79.0	78.7

结果Results	VGGNet-19	改进残差模块的ResNet-101	改进残差模块的DBTNet-101	改进残差模块的DBTANet-101
74种平均准确率Average precision-74	78.6	81.1	88.8	91.3
22种平均准确率Average precision-22	67.5	69.4	81.1	85.1
74种平均召回率Average recall-74	74.1	79.0	89.5	91.1
22种平均召回率Average recall-22	56.4	59.3	77.6	82.5
74种平均F₁值Average F₁-74	74.5	78.7	88.5	90.8
22种平均F₁值Average F₁-22	59.6	62.8	78.0	83.0

[1]	孟贵,刘叶菲,张旭峰,赵程乐,吴水荣. 1998—2018年我国林业有害生物灾情的时序分析[J]. 林业科学, 2022, 58(7): 134-143.
[2]	张慧春,周宏平,郑加强,廖娟,AndrewJ.Hewitt. 农药助剂对空中和地面防控林业有害生物的雾滴粒径影响[J]. 林业科学, 2020, 56(5): 118-129.
[3]	刘萍, 梁倩玲, 陈梦, 刘忠军, 盛兆湖, 苏延乐. 林业有害生物灾害损失评估指标体系构建[J]. 林业科学, 2016, 52(6): 101-107.
[4]	常原飞;武红敢;董振辉;崔福东;刘丽;荐军;张迎;李卫国. 国家级林业有害生物灾害监测与预警系统[J]. 林业科学, 2011, 47(6): 93-100.