STGAE: Spatial Temporal Graph Auto-Encoder for Hand Motion Denoising

Kanglei Zhou, Jiaying Chen, Hubert P. H. Shum, Frederick W. B. Li and Xiaohui Liang
Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2021

Core A* Conference Core A* Conference

STGAE: Spatial Temporal Graph Auto-Encoder for Hand Motion Denoising
‡ According to Core Ranking 2023"

Abstract

Hand object interaction in mixed reality (MR) relies on the accurate tracking and estimation of human hands, which provide users with a sense of immersion. However, raw captured hand motion data always contains errors such as joints occlusion, dislocation, high-frequency noise, and involuntary jitter. Denoising and obtaining the hand motion data consistent with the user’s intention are of the utmost importance to enhance the interactive experience in MR. To this end, we propose an end-to-end method for hand motion denoising using the spatial-temporal graph auto-encoder (STGAE). The spatial and temporal patterns are recognized simultaneously by constructing the consecutive hand joint sequence as a spatial-temporal graph. Considering the complexity of the articulated hand structure, a simple yet effective partition strategy is proposed to model the physic-connected and symmetry-connected relationships. Graph convolution is applied to extract structural constraints of the hand, and a self-attention mechanism is to adjust the graph topology dynamically. Combining graph convolution and temporal convolution, a fundamental graph encoder or decoder block is proposed. We finally establish the hourglass residual auto-encoder to learn a manifold projection operation and a corresponding inverse projection through stacking these blocks. In this work, the proposed framework has been successfully used in hand motion data denoising with preserving structural constraints between joints. Extensive quantitative and qualitative experiments show that the proposed method has achieved better performance than the state-of-the-art approaches.

Downloads

YouTube

Citations

BibTeX

@inproceedings{zhou21stgae,
 author={Zhou, Kanglei and Chen, Jiaying and Shum, Hubert P. H. and Li, Frederick W. B. and Liang, Xiaohui},
 booktitle={Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality},
 series={ISMAR '21},
 title={STGAE: Spatial Temporal Graph Auto-Encoder for Hand Motion Denoising},
 year={2021},
 month={10},
 pages={41--49},
 numpages={9},
 doi={10.1109/ISMAR52148.2021.00018},
 issn={1554-7868},
 publisher={IEEE},
 location={Bari, Italy},
}

RIS

TY  - CONF
AU  - Zhou, Kanglei
AU  - Chen, Jiaying
AU  - Shum, Hubert P. H.
AU  - Li, Frederick W. B.
AU  - Liang, Xiaohui
T2  - Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality
TI  - STGAE: Spatial Temporal Graph Auto-Encoder for Hand Motion Denoising
PY  - 2021
Y1  - 10 2021
SP  - 41
EP  - 49
DO  - 10.1109/ISMAR52148.2021.00018
SN  - 1554-7868
PB  - IEEE
ER  - 

Plain Text

Kanglei Zhou, Jiaying Chen, Hubert P. H. Shum, Frederick W. B. Li and Xiaohui Liang, "STGAE: Spatial Temporal Graph Auto-Encoder for Hand Motion Denoising," in ISMAR '21: Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality, pp. 41-49, Bari, Italy, IEEE, Oct 2021.

Supporting Grants

Similar Research

Zhiying Leng, Jiaying Chen, Hubert P. H. Shum, Frederick W. B. Li and Xiaohui Liang, "Stable Hand Pose Estimation under Tremor via Graph Neural Network", Proceedings of the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2021
Kanglei Zhou, Hubert P. H. Shum, Frederick W. B. Li and Xiaohui Liang, "Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising", IEEE Transactions on Visualization and Computer Graphics (TVCG), 2024
Kanglei Zhou, Chen Chen, Yue Ma, Zhiying Leng, Hubert P. H. Shum, Frederick W. B. Li and Xiaohui Liang, "A Mixed Reality Training System for Hand-Object Interaction in Simulated Microgravity Environments", Proceedings of the 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2023
Qi Feng, Hubert P. H. Shum and Shigeo Morishima, "Resolving Hand-Object Occlusion for Mixed Reality with Joint Deep Learning and Model Optimization", Computer Animation and Virtual Worlds (CAVW) - Proceedings of the 2020 International Conference on Computer Animation and Social Agents (CASA), 2020

 

 

Last updated on 14 April 2024
RSS Feed