Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-Centric Videos

Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei, Hubert P. H. Shum and Edmond S. L. Ho
Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026

H5-Index: 92#Core A Conference

Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-Centric Videos
‡ According to ICORE Ranking 2026
# According to Google Scholar 2026

Abstract

Pedestrian trajectory prediction from an egocentric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian. By modelling correlation and intent from the historical and future trajectories of the pedestrian, it will usually result in a multimodal (i.e. multiple modes) distribution. Existing stochastic predictors often sample multiple futures from a single unimodal distribution, which can yield sub-optimal 'mixed-mode' trajectories that lie between distinct motion patterns and become implausible in real scenes. In this paper, we propose MMPM, a mode-aware framework that separately models future trajectory distributions into semantically meaningful modes based on the pedestrian’s crossing behavior. MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) that jointly captures pedestrian–vehicle and pedestrian–environment interactions by introducing gaze, head and hand gesture, and a CVAE-based Mode-aware Trajectory Predictor (MTP) module to model the future trajectory distributions on two modes, crossing and non-crossing the road, separately. A query-based decoder further enforces mode consistency during decoding. Experiments on PIE and JAAD datasets show that our method surpasses state-of-the-art baselines. Our proposed MTP is model-agnostic, which can be integrated into existing frameworks such as BiTrap-NP and SGNet-ED to further improve future trajectory prediction performance. We additionally introduce a data-driven validation protocol that matches predictions to spatiotemporally consistent ground-truth trajectories, demonstrating improved frame-wise displacement errors over previous work.


Downloads


YouTube


Cite This Research

Plain Text

Yuxuan Xie, Nicolas Pugeault, Chongfeng Wei, Hubert P. H. Shum and Edmond S. L. Ho, "Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-Centric Videos," in Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems, Pittsburgh, USA, IEEE/RSJ, 2026.

BibTeX

@inproceedings{xie26where,
 author={Xie, Yuxuan and Pugeault, Nicolas and Wei, Chongfeng and Shum, Hubert P. H. and Ho, Edmond S. L.},
 booktitle={Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems},
 title={Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-Centric Videos},
 year={2026},
 publisher={IEEE/RSJ},
 location={Pittsburgh, USA},
}

RIS

TY  - CONF
AU  - Xie, Yuxuan
AU  - Pugeault, Nicolas
AU  - Wei, Chongfeng
AU  - Shum, Hubert P. H.
AU  - Ho, Edmond S. L.
T2  - Proceedings of the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems
TI  - Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-Centric Videos
PY  - 2026
PB  - IEEE/RSJ
ER  - 


Supporting Grants


Similar Research

Ruochen Li, Stamos Katsigiannis, Tae-Kyun Kim and Hubert P. H. Shum, "BP-SGCN: Behavioral Pseudo-Label Informed Sparse Graph Convolution Network for Pedestrian and Heterogeneous Trajectory Prediction", IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2025
Ben Rainbow, Qianhui Men and Hubert P. H. Shum, "Semantics-STGCNN: A Semantics-Guided Spatial-Temporal Graph Convolutional Network for Multi-Class Trajectory Prediction", Proceedings of the 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021
Ruochen Li, Tanqiu Qiao, Stamos Katsigiannis, Zhanxing Zhu and Hubert P. H. Shum, "Unified Spatial-Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction", IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2025
Ruochen Li, Zhanxing Zhu, Tanqiu Qiao and Hubert P. H. Shum, "ViTE: Virtual Graph Trajectory Expert Router for Pedestrian Trajectory Prediction", Proceedings of the 2026 AAAI Conference on Artificial Intelligence (AAAI), 2026
Ruochen Li, Stamos Katsigiannis and Hubert P. H. Shum, "Multiclass-SGCN: Sparse Graph-Based Trajectory Prediction with Agent Class Embedding", Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), 2022
Qianhui Men and Hubert P. H. Shum, "PyTorch-Based Implementation of Label-Aware Graph Representation for Multi-Class Trajectory Prediction", Software Impacts (SIMPAC), 2021

HomeGoogle ScholarLinkedInYouTubeGitHubORCIDResearchGateEmail
 
Print