Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit...

ver descrição completa

Detalhes bibliográficos
Autores: Gené Mola, Jordi, Sanz Cortiella, Ricardo, Rosell Polo, Joan Ramon, Morros Rubió, Josep Ramon|||0000-0002-1395-487X, Ruiz Hidalgo, Javier|||0000-0001-6774-685X, Vilaplana Besler, Verónica|||0000-0001-6924-9961, Gregorio, Eduard
Tipo de documento: artigo
Data de publicação:2020
País:España
Recursos:Universitat Politècnica de Catalunya (UPC)
Repositório:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglês
OAI Identifier:oai:upcommons.upc.edu:2117/175526
Acesso em linha:https://hdl.handle.net/2117/175526
https://dx.doi.org/10.1016/j.compag.2019.105165
Access Level:Acceso aberto
Palavra-chave:Remote sensing
Fruit -- Breeding
Structure-from-motion
Fruit detection
Fruit location
Mask R-CNN
Terrestrial remote sensing
Teledetecció -- Aplicacions agrícoles
Fructicultura
Àrees temàtiques de la UPC::Enginyeria agroalimentària
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Radiocomunicació i exploració electromagnètica::Teledetecció
id ES_aec6f752746d4e2f275a9b212f3fb045
oai_identifier_str oai:upcommons.upc.edu:2117/175526
network_acronym_str ES
network_name_str España
repository_id_str
spelling Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetryGené Mola, JordiSanz Cortiella, RicardoRosell Polo, Joan RamonMorros Rubió, Josep Ramon|||0000-0002-1395-487XRuiz Hidalgo, Javier|||0000-0001-6774-685XVilaplana Besler, Verónica|||0000-0001-6924-9961Gregorio, EduardRemote sensingFruit -- BreedingStructure-from-motionFruit detectionFruit locationMask R-CNNTerrestrial remote sensingTeledetecció -- Aplicacions agrícolesFructiculturaÀrees temàtiques de la UPC::Enginyeria agroalimentàriaÀrees temàtiques de la UPC::Enginyeria de la telecomunicació::Radiocomunicació i exploració electromagnètica::TeledeteccióThe development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html.Peer Reviewed20202020-01-1320202020-01-23journal articlehttp://purl.org/coar/resource_type/c_6501AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/175526https://dx.doi.org/10.1016/j.compag.2019.105165reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/1755262026-05-27T15:37:01Z
dc.title.none.fl_str_mv Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
title Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
spellingShingle Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
Gené Mola, Jordi
Remote sensing
Fruit -- Breeding
Structure-from-motion
Fruit detection
Fruit location
Mask R-CNN
Terrestrial remote sensing
Teledetecció -- Aplicacions agrícoles
Fructicultura
Àrees temàtiques de la UPC::Enginyeria agroalimentària
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Radiocomunicació i exploració electromagnètica::Teledetecció
title_short Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
title_full Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
title_fullStr Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
title_full_unstemmed Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
title_sort Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry
dc.creator.none.fl_str_mv Gené Mola, Jordi
Sanz Cortiella, Ricardo
Rosell Polo, Joan Ramon
Morros Rubió, Josep Ramon|||0000-0002-1395-487X
Ruiz Hidalgo, Javier|||0000-0001-6774-685X
Vilaplana Besler, Verónica|||0000-0001-6924-9961
Gregorio, Eduard
author Gené Mola, Jordi
author_facet Gené Mola, Jordi
Sanz Cortiella, Ricardo
Rosell Polo, Joan Ramon
Morros Rubió, Josep Ramon|||0000-0002-1395-487X
Ruiz Hidalgo, Javier|||0000-0001-6774-685X
Vilaplana Besler, Verónica|||0000-0001-6924-9961
Gregorio, Eduard
author_role author
author2 Sanz Cortiella, Ricardo
Rosell Polo, Joan Ramon
Morros Rubió, Josep Ramon|||0000-0002-1395-487X
Ruiz Hidalgo, Javier|||0000-0001-6774-685X
Vilaplana Besler, Verónica|||0000-0001-6924-9961
Gregorio, Eduard
author2_role author
author
author
author
author
author
dc.subject.none.fl_str_mv Remote sensing
Fruit -- Breeding
Structure-from-motion
Fruit detection
Fruit location
Mask R-CNN
Terrestrial remote sensing
Teledetecció -- Aplicacions agrícoles
Fructicultura
Àrees temàtiques de la UPC::Enginyeria agroalimentària
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Radiocomunicació i exploració electromagnètica::Teledetecció
topic Remote sensing
Fruit -- Breeding
Structure-from-motion
Fruit detection
Fruit location
Mask R-CNN
Terrestrial remote sensing
Teledetecció -- Aplicacions agrícoles
Fructicultura
Àrees temàtiques de la UPC::Enginyeria agroalimentària
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Radiocomunicació i exploració electromagnètica::Teledetecció
description The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html.
publishDate 2020
dc.date.none.fl_str_mv 2020
2020-01-13
2020
2020-01-23
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/175526
https://dx.doi.org/10.1016/j.compag.2019.105165
url https://hdl.handle.net/2117/175526
https://dx.doi.org/10.1016/j.compag.2019.105165
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869416609914814464
score 15,301629

Registros relacionados