Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

Abstract

Wepresentanovelmethodforreal-timeposeandshapereconstructionof twostronglyinteractinghands.Ourapproachisthefirsttwo-handtracking solutionthatcombinesanextensivelistoffavorableproperties,namelyitis marker-less,usesasingleconsumer-leveldepthcamera,runsinrealtime, handlesinter-andintra-handcollisions,andautomaticallyadjuststothe user’shandshape.Inordertoachievethis,weembedarecentparametric handposeandshapemodelandadensecorrespondencepredictorbasedon adeepneuralnetworkintoasuitableenergyminimizationframework.For trainingthecorrespondencepredictionnetwork,wesynthesizeatwo-hand dataset based on physical simulations that includes both hand pose and shapeannotationswhileatthesametimeavoidinginter-handpenetrations. Toachievereal-timerates,wephrasethemodelfittingintermsofanonlinear least-squaresproblemsothattheenergycanbeoptimizedbasedonahighly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art resultsinscenesthatexceedthecomplexityleveldemonstratedbyprevious