RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video

TouchDesign (M1792)

Detalles Bibliográficos
Autores: WANG, JIAYI, MUELLER, FRANZISKA, BERNARD, FLORIAN, SORLI, SUZANNE, SOTNYCHENKO, OLEKSANDR, QIAN, NENG, OTADUY, MIGUEL A., CASAS, DAN, THEOBALT, CHRISTIAN
Tipo de recurso: artículo
Fecha de publicación:2020
País:España
Institución:Universidad Rey Juan Carlos
Repositorio:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
OAI Identifier:oai:burjcdigital.urjc.es:10115/17670
Acceso en línea:http://hdl.handle.net/10115/17670
Access Level:acceso abierto
Palabra clave:hand tracking
hand pose estimation
hand reconstruction
two hands
monocular RGB
RGB video
computing methodologies
Computer vision
Neural networks
id ES_84adcf8f3d86f8e4fedac6c2d1c8c58b
oai_identifier_str oai:burjcdigital.urjc.es:10115/17670
network_acronym_str ES
network_name_str España
repository_id_str
spelling RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB VideoWANG, JIAYIMUELLER, FRANZISKABERNARD, FLORIANSORLI, SUZANNESOTNYCHENKO, OLEKSANDRQIAN, NENGOTADUY, MIGUEL A.CASAS, DANTHEOBALT, CHRISTIANhand trackinghand pose estimationhand reconstructiontwo handsmonocular RGBRGB videocomputing methodologiesComputer visionNeural networksTouchDesign (M1792)© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM Transactions on Graphics, https://doi.org/10.1145/3414685.3417852.Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings (e.g., considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods.Association for Computing Machinery (ACM)202120212020info:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10115/17670reponame:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlosinstname:Universidad Rey Juan CarlosInglésTouchDesign (M1792)Attribution-NonCommercial-NoDerivatives 4.0 Internacionalhttp://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:burjcdigital.urjc.es:10115/176702026-06-24T12:48:17Z
dc.title.none.fl_str_mv RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
title RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
spellingShingle RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
WANG, JIAYI
hand tracking
hand pose estimation
hand reconstruction
two hands
monocular RGB
RGB video
computing methodologies
Computer vision
Neural networks
title_short RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
title_full RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
title_fullStr RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
title_full_unstemmed RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
title_sort RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
dc.creator.none.fl_str_mv WANG, JIAYI
MUELLER, FRANZISKA
BERNARD, FLORIAN
SORLI, SUZANNE
SOTNYCHENKO, OLEKSANDR
QIAN, NENG
OTADUY, MIGUEL A.
CASAS, DAN
THEOBALT, CHRISTIAN
author WANG, JIAYI
author_facet WANG, JIAYI
MUELLER, FRANZISKA
BERNARD, FLORIAN
SORLI, SUZANNE
SOTNYCHENKO, OLEKSANDR
QIAN, NENG
OTADUY, MIGUEL A.
CASAS, DAN
THEOBALT, CHRISTIAN
author_role author
author2 MUELLER, FRANZISKA
BERNARD, FLORIAN
SORLI, SUZANNE
SOTNYCHENKO, OLEKSANDR
QIAN, NENG
OTADUY, MIGUEL A.
CASAS, DAN
THEOBALT, CHRISTIAN
author2_role author
author
author
author
author
author
author
author
dc.subject.none.fl_str_mv hand tracking
hand pose estimation
hand reconstruction
two hands
monocular RGB
RGB video
computing methodologies
Computer vision
Neural networks
topic hand tracking
hand pose estimation
hand reconstruction
two hands
monocular RGB
RGB video
computing methodologies
Computer vision
Neural networks
description TouchDesign (M1792)
publishDate 2020
dc.date.none.fl_str_mv 2020
2021
2021
dc.type.none.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv http://hdl.handle.net/10115/17670
url http://hdl.handle.net/10115/17670
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv TouchDesign (M1792)
dc.rights.none.fl_str_mv Attribution-NonCommercial-NoDerivatives 4.0 Internacional
http://creativecommons.org/licenses/by-nc-nd/4.0/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv Attribution-NonCommercial-NoDerivatives 4.0 Internacional
http://creativecommons.org/licenses/by-nc-nd/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Association for Computing Machinery (ACM)
publisher.none.fl_str_mv Association for Computing Machinery (ACM)
dc.source.none.fl_str_mv reponame:BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
instname:Universidad Rey Juan Carlos
instname_str Universidad Rey Juan Carlos
reponame_str BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
collection BURJC-Digital. Repositorio Institucional de la Universidad Rey Juan Carlos
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869412242463653888
score 15.812455

Similares en LA Referencia