AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Plant Phenomics Article
Article Link
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article | Open Access

Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach

Yangmingrui Gao1,†, Yinglun Li1,†, Ruibo Jiang1, Xiaohai Zhan1, Hao Lu2, Wei Guo3, Wanneng Yang4, Yanfeng Ding1, Shouyang Liu1( )
Plant Phenomics Research Centre, Academy for Advanced Interdisciplinary Studies, Jiangsu Collaborative Innovation Center for Modern Crop Production, Nanjing Agricultural University, Nanjing, China
Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China
Graduate School of Agricultural and Life Sciences, The University of Tokyo, 1-1-1 Midori-cho, Nishitokyo City, Tokyo, Japan
National Key Laboratory of Crop Genetic Improvement, National Center of Plant Gene Research, and Hubei Key Laboratory of Agricultural Bioinformatics, Huazhong Agricultural University, Wuhan 430070, China

†These author contributed equally to this work.

Show Author Information

Abstract

The green fraction (GF), which is the fraction of green vegetation in a given viewing direction, is closely related to the light interception ability of the crop canopy. Monitoring the dynamics of GF is therefore of great interest for breeders to identify genotypes with high radiation use efficiency. The accuracy of GF estimation depends heavily on the quality of the segmentation dataset and the accuracy of the image segmentation method. To enhance segmentation accuracy while reducing annotation costs, we developed a self-supervised strategy for deep learning semantic segmentation of rice and wheat field images with very contrasting field backgrounds. First, the Digital Plant Phenotyping Platform was used to generate large, perfectly labeled simulated field images for wheat and rice crops, considering diverse canopy structures and a wide range of environmental conditions (sim dataset). We then used the domain adaptation model cycle-consistent generative adversarial network (CycleGAN) to bridge the reality gap between the simulated and real images (real dataset), producing simulation-to-reality images (sim2real dataset). Finally, 3 different semantic segmentation models (U-Net, DeepLabV3+, and SegFormer) were trained using 3 datasets (real, sim, and sim2real datasets). The performance of the 9 training strategies was assessed using real images captured from various sites. The results showed that SegFormer trained using the sim2real dataset achieved the best segmentation performance for both rice and wheat crops (rice: Accuracy = 0.940, F1-score = 0.937; wheat: Accuracy = 0.952, F1-score = 0.935). Likewise, favorable GF estimation results were obtained using the above strategy (rice: R2 = 0.967, RMSE = 0.048; wheat: R2 = 0.984, RMSE = 0.028). Compared with SegFormer trained using a real dataset, the optimal strategy demonstrated greater superiority for wheat images than for rice images. This discrepancy can be partially attributed to the differences in the backgrounds of the rice and wheat fields. The uncertainty analysis indicated that our strategy could be disrupted by the inhomogeneity of pixel brightness and the presence of senescent elements in the images. In summary, our self-supervised strategy addresses the issues of high cost and uncertain annotation accuracy during dataset creation, ultimately enhancing GF estimation accuracy for rice and wheat field images. The best weights we trained in wheat and rice are available: https://github.com/PheniX-Lab/sim2real-seg.

References

1

Li W, Fang H, Wei S, Weiss M, Baret F. Critical analysis of methods to estimate the fraction of absorbed or intercepted photosynthetically active radiation from ground measurements: Application to rice crops. Agric For Meteorol. 2021;297:108273.
Crossref Google Scholar
2
Baret F, Bruno A, Folmer J, Hanocq J, Sarrouy C. Gap fraction measurement from hemispherical infrared photography and its use to evaluate PAR interception efficiency. In: Varlet-Grancher C, Bonhomme R, Sinoquet H, editors. Crop structure and light microclimate. Characterisation and applications. Paris (France): INRA; 1993. p. 359–372.
3

Baret F, Andrieu B, Steven M. Gap frequency and canopy architecture of sugar beet and wheat crops. Agric For Meteorol. 1993;65(3-4):261–279.
Crossref Google Scholar
4

Liu S, Baret F, Abichou M, Boudon F, Thomas S, Zhao K, Fournier C, Andrieu B, Irfan K, Hemmerlé M, et al. Estimating wheat green area index from ground-based LiDAR measurement using a 3D canopy structure model. Agric For Meteorol. 2017;247:12–20.
Crossref Google Scholar
5

Luis Araus J, Cairns JE. Field high-throughput phenotyping: The new crop breeding frontier. Trends Plant Sci. 2014;19(1):52–61.
Crossref Google Scholar
6

Weiss M, Baret F, Smith GJ, Jonckheere I, Coppin P. Review of methods for in situ leaf area index (LAI) determination part Ⅱ. Estimation of LAI, errors and sampling. Agric For Meteorol. 2004;121(1-2):37–53.
Crossref Google Scholar
7

Castillo-Martínez MÁ, Gallegos-Funes FJ, Carvajal-Gámez BE, Urriolagoitia-Sosa G, Rosales-Silva AJ. Color index based thresholding method for background and foreground segmentation of plant images. Comput Electron Agric. 2020;178:105783.
Crossref Google Scholar
8

Meyer GE, Neto JC. Verification of color vegetation indices for automated crop imaging applications. Comput Electron Agric. 2008;63(2):282–293.
Crossref Google Scholar
9

Hamuda E, Glavin M, Jones E. A survey of image processing techniques for plant extraction and segmentation in the field. Comput Electron Agric. 2016;125:184–199.
Crossref Google Scholar
10

Ruiz-Ruiz G, Gómez-Gil J, Navas-Gracia LM. Testing different color spaces based on hue for the environmentally adaptive segmentation algorithm (EASA). Comput Electron Agric. 2009;68(1):88–96.
Crossref Google Scholar
11

Zheng L, Shi D, Zhang J. Segmentation of green vegetation of crop canopy images based on mean shift and fisher linear discriminant. Pattern Recogn Lett. 2010;31(9):920–925.
Crossref Google Scholar
12

Guo W, Rage UK, Ninomiya S. Illumination invariant segmentation of vegetation for time series wheat images based on decision tree model. Comput Electron Agric. 2013;96:58–66.
Crossref Google Scholar
13

Giménez-Gallego J, González-Teruel JD, Jiménez-Buendía M, Toledo-Moreo AB, Soto-Valles F, Torres-Sánchez R. Segmentation of multiple tree leaves pictures with natural backgrounds using deep learning for image-based agriculture applications. Appl Sci. 2019;10(1):202.
Crossref Google Scholar
14

Wang H, Lyu S, Ren Y. Paddy rice imagery dataset for panicle segmentation. Agronomy. 2021;11(8):1542.
Crossref Google Scholar
15

Itakura K, Hosoi F. Automatic leaf segmentation for estimating leaf area and leaf inclination angle in 3D plant images. Sensors. 2018;18(10):3576.
Crossref Google Scholar
16

LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–444.
Crossref Google Scholar
17
Serouart M, Madec S, David E, Velumani K, Lopez Lozano R, Weiss M, Baret F. SegVeg: Segmenting RGB images into green and senescent vegetation by combining deep and shallow methods. 2022;2022:9803570.
18
Lanham M. Generating a new reality: From autoencoders and adversarial networks to deepfakes. Canada: Apress; 2021. Chapter 7, Attention is all we need!, p. 195–222.
19

Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E. Deep learning for computer vision: A brief review. Comput Intell Neurosci. 2018;2018:7068349.
Google Scholar
20
Garcia-Garcia A, Orts-Escolano S, Oprea SO, Villena-Martinez V, Garcia-Rodriguez J. A review on deep learning techniques applied to semantic segmentation. arXiv. 2017. https://doi.org/10.48550/arXiv.1704.06857
21

Zhang Q, Liu Y, Gong C, Chen Y, Yu H. Applications of deep learning for dense scenes analysis in agriculture: A review. Sensors. 2020;20(5):1520.
Crossref Google Scholar
22
Richter S, Vineet V, Roth S, Koltun V. Playing for data: Ground truth from computer games. Paper presented at: the European Conference on Computer Vision (ECCV); 2016 October 11–14; Amsterdam, The Netherlands.
23
Ros G, Sellart L, Materzynska J, Vazquez D, Lopez AM. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. Paper presented at: IEEE: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 June 27–30; Las Vegas, NV.
24
Gaidon A, Wang Q, Cabon Y, Vig E. Virtual worlds as proxy for multi-object tracking analysis. Paper presented at: the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 27–30 Jun 2016; Las Vegas, NV.
25

Liu S, Martre P, Buis S, Andrieu MAB, Baret F. Estimation of plant and canopy architectural traits using the digital plant phenotyping platform. Plant Physiol. 2019;181(3):881–890.
Crossref Google Scholar
26

Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley W, Ozair S, Courville A, Bengio Y. Generative adversarial networks. Commun ACM. 2020;63(11):139–144.
Crossref Google Scholar
27

Zhang J, Tai L, Yun P, Xiong Y, Liu M, Boedecker J, Burgard W. VR-goggles for robots: Real-to-Sim domain adaptation for visual control. IEEE Robot Autom Lett. 2019;4(2):1148–1155.
Crossref Google Scholar
28
Stein GJ, Roy N. GeneSIS-Rt: Generating synthetic images for training secondary real-world tasks. Paper presented at: 2018 IEEE International Conference on Robotics and Automation (ICRA); 2018 May 21–25; Brisbane, Australia.
29

Li Y, Zhan X, Liu S, Lu H, Jiang R, Guo W, Chapman S, Ge Y, Solan B, Ding Y, et al. Self-supervised plant phenotyping by combining domain adaptation with 3D plant model simulations: Application to wheat leaf counting at seedling stage. Plant Phenomics. 2023;5:Article 0041.
Google Scholar
30
Zhu J-Y, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. Paper presented at: 2017 IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22–29; Venice, Italy.
31

Zenkl R et al. Outdoor plant segmentation with deep learning for high-throughput field phenotyping on a diverse wheat dataset. Front Plant Sci. 2021;12:774068.
Google Scholar
32
Tangseng P, Wu Z, Yamaguchi K. Looking at outfit to parse clothing. arXiv. 2017. https://doi.org/10.48550/arXiv.1703.01386.
33
Fournier C, Andrieu B, Ljutovac S, Saint-Jean S. ADEL-wheat: a 3D architectural model of wheat development. In: Hu B-G, Jaeger M, editors. 2003 International Symposium on plant growth modeling, simulation, visualization and their applications. Beijing (P.R. China): Tsinghua University Press - Springer Verlag; 2003. p. 54–63.
34

Pradal C, Dufour-Kowalski S, Boudon F, Fournier C, Godin C. OpenAlea: A visual programming and component-based software platform for plant modelling. Funct Plant Biol. 2008;35(10):751–760.
Crossref Google Scholar
35

Li J, Lu B-L. An adaptive image Euclidean distance. Pattern Recogn. 2009;42(3):349–357.
Crossref Google Scholar
36
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Paper presented at: Medical Image Computing and Computer-Assisted Intervention; 2015 May 18; PA, USA.
37
Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. Paper presented at: ECCV 2018: 15th European Conference; 2018 Sep 8–14; Munich, Germany.
38
Xie E, Wang W, Yu Z, Anandkumar A, Alvarez J, Luo P. SegFormer: Simple and efficient design for semantic segmentation with transformers. Paper presented at: Advances in Neural Information Processing Systems 34 (NeurIPS 2021); 2021 Jun 12.
39
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. Paper presented at: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2015 Jun 7–12; Boston, MA.
40

Zhou T, Dong Y, Huo B, Liu S, Ma Z. U-net and its applications in medical image segmentation: A review. J Image Graph. 2021;26:2058–2077.
Google Scholar
41

Yin X-X, Sun L, Fu Y, Lu R, Zhang Y. U-net-based medical image segmentation. J Healthc Eng. 2022;2022:Article 4189781.
Google Scholar
42

Wei Y, Liu X, Lei J, Yue R, Feng J. Multiscale feature U-net for remote sensing image segmentation. J Appl Remote Sens. 2022;16(1):Article 016507.
Google Scholar
43

Chen LC, Papandreou, Kokkinos I, Murphy K, Yuille AL. DeepLab: Semantic image segmentation with deep convolutional nets, Atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell. 2018;40(4):834–848.
Crossref Google Scholar
44
Kamann C, Rother C. Benchmarking the robustness of semantic segmentation models. Paper presented at the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020 Jun 13–19; Seattle, WA.
45

Liu L, Lu H, Li Y, Cao Z. High-throughput Rice density estimation from transplantation to Tillering stages using deep networks. Plant Phenomics. 2020;2020:Article 1375957.
Google Scholar
46
Lòpez F, González JM, Baldrich R, Vanrell M. Fast surface grading using color statistics in the CIE Lab Space. Paper presented at: IbPRIA'05: Proceedings of the Second Iberian conference on Pattern Recognition and Image Analysis; 2005 June 7; Berlin, Heidelberg.
47

Bai XD, Cao ZG, Wang Y, Yu ZH, Zhang XF, Li CN. Crop segmentation from images by morphology modeling in the CIE L*a*b* color space. Comput Electron Agric. 2013;99:21–34.
Crossref Google Scholar
48

Hoyez H, Schockaert C, Rambach J, Mirbach B, Stricker D. Unsupervised image-to-image translation: A review. Sensors (Basel). 2022;22(21):Article 8540.
Crossref Google Scholar
49

Zou K, Chen X, Wang Y, Zhang C, Zhang F. A modified U-net with a specific data argumentation method for semantic segmentation of weed images in the field. Comput Electron Agric. 2021;187(C):929.
Google Scholar
Plant Phenomics
Volume 5,
2023
Article number: 0064
DOI: 10.34133/plantphenomics.0064
Cite this article:
Gao Y, Li Y, Jiang R, et al. Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach. Plant Phenomics, 2023, 5: 0064. https://doi.org/10.34133/plantphenomics.0064

144

Views

14

Crossref

13

Web of Science

15

Scopus

0

CSCD

Altmetrics
Received: 07 April 2023
Accepted: 12 June 2023
Published: 18 July 2023
© 2023 Yangmingrui Gao et al. Exclusive licensee Nanjing Agricultural University. No claim to original U.S. Government Works.

Distributed under a Creative Commons Attribution License 4.0 (CC BY 4.0).
Return