GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction

Qianhui Men, Hubert P. H. Shum, Edmond S. L. Ho and Howard Leung
Computers and Graphics (C&G), 2022

 Impact Factor: 2.5 Citation: 16#

GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction
# According to Google Scholar 2023"

Abstract

Creating realistic characters that can react to the users' or another character's movement can benefit computer graphics, games and virtual reality hugely. However, synthesizing such reactive motions in human-human interactions is a challenging task due to the many different ways two humans can interact. While there are a number of successful researches in adapting the generative adversarial network (GAN) in synthesizing single human actions, there are very few on modelling human-human interactions. In this paper, we propose a semi-supervised GAN system that synthesizes the reactive motion of a character given the active motion from another character. Our key insights are two-fold. First, to effectively encode the complicated spatial-temporal information of a human motion, we empower the generator with a part-based long short-term memory (LSTM) module, such that the temporal movement of different limbs can be effectively modelled. We further include an attention module such that the temporal significance of the interaction can be learned, which enhances the temporal alignment of the active-reactive motion pair. Second, as the reactive motion of different types of interactions can be significantly different, we introduce a discriminator that not only tells if the generated movement is realistic or not, but also tells the class label of the interaction. This allows the use of such labels in supervising the training of the generator. We experiment with the SBU and the HHOI datasets. The high quality of the synthetic motion demonstrates the effective design of our generator, and the discriminability of the synthesis also demonstrates the strength of our discriminator.

Downloads

YouTube

Citations

BibTeX

@article{men22gan,
 author={Men, Qianhui and Shum, Hubert P. H. and Ho, Edmond S. L. and Leung, Howard},
 journal={Computers and Graphics},
 title={GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction},
 year={2022},
 volume={102},
 pages={634--645},
 numpages={12},
 doi={10.1016/j.cag.2021.09.014},
 issn={0097-8493},
 publisher={Elsevier},
}

RIS

TY  - JOUR
AU  - Men, Qianhui
AU  - Shum, Hubert P. H.
AU  - Ho, Edmond S. L.
AU  - Leung, Howard
T2  - Computers and Graphics
TI  - GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction
PY  - 2022
VL  - 102
SP  - 634
EP  - 645
DO  - 10.1016/j.cag.2021.09.014
SN  - 0097-8493
PB  - Elsevier
ER  - 

Plain Text

Qianhui Men, Hubert P. H. Shum, Edmond S. L. Ho and Howard Leung, "GAN-Based Reactive Motion Synthesis with Class-Aware Discriminators for Human-Human Interaction," Computers and Graphics, vol. 102, pp. 634-645, Elsevier, 2022.

Supporting Grants

Similar Research

Baiyi Li, Edmond S. L. Ho, Hubert P. H. Shum and He Wang, "Two-Person Interaction Augmentation with Skeleton Priors", Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024
Qianhui Men, Edmond S. L. Ho, Hubert P. H. Shum and Howard Leung, "A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction", IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021
He Wang, Edmond S. L. Ho, Hubert P. H. Shum and Zhanxing Zhu, "Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling", IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021
Taku Komura, Hubert P. H. Shum and Edmond S. L. Ho, "Simulating Interactions of Characters", Proceedings of the 2008 ACM International Conference on Motion in Games (MIG), 2008
Hubert P. H. Shum and Taku Komura, "Generating Realistic Fighting Scenes by Game Tree", Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) Posters, 2006
Hubert P. H. Shum and Taku Komura, "Technical Note: Generating Realistic Fighting Scenes by Game Tree", arXiv Preprint, 2006
Hubert P. H. Shum, "Simulating Interactions Among Multiple Characters", PhD Thesis from the University of Edinburgh, 2010
Edmond S. L. Ho, Hubert P. H. Shum, Yiu-ming Cheung and P. C. Yuen, "Topology Aware Data-Driven Inverse Kinematics", Computer Graphics Forum (CGF) - Proceedings of the 2013 Pacific Conference on Computer Graphics and Applications (PG), 2013

 

 

Last updated on 10 June 2024
RSS Feed