Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling

He Wang, Edmond S. L. Ho, Hubert P. H. Shum and Zhanxing Zhu
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021

REF 2021 Submitted Output Impact Factor: 4.7 Top 25% Journal in Computer Science, Software Engineering Citation: 81#

Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling
# According to Google Scholar 2024

Abstract

Data-driven modeling of human motions is ubiquitous in computer graphics and computer vision applications, such as synthesizing realistic motions or recognizing actions. Recent research has shown that such problems can be approached by learning a natural motion manifold using deep learning on a large amount data, to address the shortcomings of traditional data-driven approaches. However, previous deep learning methods can be sub-optimal for two reasons. First, the skeletal information has not been fully utilized for feature extraction. Unlike images, it is difficult to define spatial proximity in skeletal motions in the way that deep networks can be applied for feature extraction. Second, motion is time-series data with strong multi-modal temporal correlations between frames. On the one hand, a frame could be followed by several candidate frames leading to different motions; on the other hand, long-range dependencies exist where a number of frames in the beginning correlate to a number of frames later. Ineffective temporal modeling would either under-estimate the multi-modality and variance, resulting in featureless mean motion or over-estimate them resulting in jittery motions, which is a major source of visual artifacts. In this paper, we propose a new deep network to tackle these challenges by creating a natural motion manifold that is versatile for many applications. The network has a new spatial component for feature extraction. It is also equipped with a new batch prediction model that predicts a large number of frames at once, such that long-term temporally-based objective functions can be employed to correctly learn the motion multi-modality and variances. With our system, long-duration motions can be predicted/synthesized using an open-loop setup where the motion retains the dynamics accurately. It can also be used for denoising corrupted motions and synthesizing new motions with given control signals. We demonstrate that our system can create superior results comparing to existing work in multiple applications.

Downloads

YouTube



Citations

BibTeX

@article{wang21spatiotemporal,
 author={Wang, He and Ho, Edmond S. L. and Shum, Hubert P. H. and Zhu, Zhanxing},
 journal={IEEE Transactions on Visualization and Computer Graphics},
 series={TVCG '24},
 title={Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling},
 year={2021},
 volume={27},
 number={1},
 pages={216--227},
 numpages={12},
 doi={10.1109/TVCG.2019.2936810},
 publisher={IEEE},
}

RIS

TY  - JOUR
AU  - Wang, He
AU  - Ho, Edmond S. L.
AU  - Shum, Hubert P. H.
AU  - Zhu, Zhanxing
T2  - IEEE Transactions on Visualization and Computer Graphics
TI  - Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling
PY  - 2021
VL  - 27
IS  - 1
SP  - 216
EP  - 227
DO  - 10.1109/TVCG.2019.2936810
PB  - IEEE
ER  - 

Plain Text

He Wang, Edmond S. L. Ho, Hubert P. H. Shum and Zhanxing Zhu, "Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling," IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 1, pp. 216-227, IEEE, 2021.

Supporting Grants

Similar Research

Edmond S. L. Ho, Hubert P. H. Shum, He Wang and Li Yi, "Synthesizing Motion with Relative Emotion Strength", Proceedings of the 2017 ACM SIGGRAPH Asia Workshop on Data-Driven Animation Techniques (D2AT), 2017
Qianhui Men, Edmond S. L. Ho, Hubert P. H. Shum and Howard Leung, "A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction", IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021
Edmund J. C. Findlay, Haozheng Zhang, Ziyi Chang and Hubert P. H. Shum, "Denoising Diffusion Probabilistic Models for Styled Walking Synthesis", Proceedings of the 2022 ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG) Posters, 2022
Hubert P. H. Shum, Ludovic Hoyet, Edmond S. L. Ho, Taku Komura and Franck Multon, "Natural Preparation Behavior Synthesis", Computer Animation and Virtual Worlds (CAVW), 2013
Hubert P. H. Shum, Ludovic Hoyet, Edmond S. L. Ho, Taku Komura and Franck Multon, "Preparation Behaviour Synthesis with Reinforcement Learning", Proceedings of the 2013 International Conference on Computer Animation and Social Agents (CASA), 2013
Ziyi Chang, Edmund J. C. Findlay, Haozheng Zhang and Hubert P. H. Shum, "Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Models", Proceedings of the 2023 International Conference on Computer Graphics Theory and Applications (GRAPP), 2023
Hubert P. H. Shum, Taku Komura and Pranjul Yadav, "Angular Momentum Guided Motion Concatenation", Computer Animation and Virtual Worlds (CAVW) - Proceedings of the 2009 International Conference on Computer Animation and Social Agents (CASA), 2009
Baiyi Li, Edmond S. L. Ho, Hubert P. H. Shum and He Wang, "Two-Person Interaction Augmentation with Skeleton Priors", Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024
Qianhui Men, Hubert P. H. Shum, Edmond S. L. Ho and Howard Leung, "GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction", Computers and Graphics (C&G), 2022

 

 

Last updated on 15 July 2024
RSS Feed