Accurate hand contact detection from RGB images via image-to-image translation

Fecha

2025-05

Título de la revista

ISSN de la revista

Título del volumen

Editor

Elsevier

Citas

0 citas en WOS
0 citas en

Resumen

Hand tracking is a growing research field that can potentially provide a natural interface to interact with virtual environments. However, despite the impressive recent advances, the 3D tracking of two interacting hands from RGB video remains an open problem. While current methods are able to infer the 3D pose of two hands in interaction reasonably, residual errors in depth, shape, and pose estimation prevent the accurate detection of hand-to-hand contact. To mitigate these errors, in this paper, we propose an image-based data-driven method to estimate the contact in hand-to-hand interactions. Our method is built on top of 3D hand trackers that predict the articulated pose of two hands, enriching them with camera-space probability maps of contact points. To train our method, we first feed motion capture data of interacting hands into a physics-based hand simulator, and compute dense 3D contact points. We then render such contact maps from various viewpoints and create a dataset of pairs of pixel-to-surface hand images and their corresponding contact labels. Finally, we train an image-to-image network that learns to translate pixel-to-surface correspondences to contact maps. At inference time, we estimate pixel-to-surface correspondences using state-of-the-art hand tracking and then use our network to predict accurate hand-to-hand contact. We qualitatively and quantitatively validate our method in real-world data and demonstrate that our contact predictions are more accurate than state-of-the-art hand-tracking methods.

Descripción

This work has been partially funded by: the Spanish State Agency of Research under grant agreement TED2021-132003B-I00; by the Universidad Rey Juan Carlos through the Distinguished Researcher position INVESDIST-04 under the call from 17/12/2020; and by Spanish Ministry of Science and Innovation under grant agreement CNS2022-135996.

Citación

Suzanne Sorli, Marc Comino-Trinidad, Dan Casas, Accurate hand contact detection from RGB images via image-to-image translation, Computers & Graphics, Volume 128, 2025, 104200, ISSN 0097-8493, https://doi.org/10.1016/j.cag.2025.104200
license logo
Excepto si se señala otra cosa, la licencia del ítem se describe como Attribution 4.0 International