Towards Graph Representation Learning Based Surgical Workflow Anticipation

Francis Xiatian Zhang, Noura Al Moubayed and Hubert P. H. Shum
Proceedings of the 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 2022

 Oral Presentation

Towards Graph Representation Learning Based Surgical Workflow Anticipation

Abstract

Surgical workflow anticipation can give predictions on what steps to conduct or what instruments to use next, which is an essential part of the computer-assisted intervention system for surgery, e.g. workflow reasoning in robotic surgery. However, current approaches are limited to their insufficient expressive power for relationships between instruments. Hence, we propose a graph representation learning framework to comprehensively represent instrument motions in the surgical workflow anticipation problem. In our proposed graph representation, we maps the bounding box information of instruments to the graph nodes in the consecutive frames and build inter-frame/inter-instrument graph edges to represent the trajectory and interaction of the instruments over time. This design enhances the ability of our network on modeling both the spatial and temporal patterns of surgical instruments and their interactions. In addition, we design a multi-horizon learning strategy to balance the understanding of various horizons indifferent anticipation tasks, which significantly improves the model performance in anticipation with various horizons. Experiments on the Cholec80 dataset demonstrate the performance of our proposed method can exceed the state-of-the-art method based on richer backbones, especially in instrument anticipation (1.27 v.s. 1.48 for inMAE; 1.48 v.s. 2.68 for eMAE). To the best of our knowledge, we are the first to introduce a spatial-temporal graph representation into surgical workflow anticipation.

Downloads

YouTube

Citations

BibTeX

@inproceedings{zhang22towards,
 author={Zhang, Francis Xiatian and Moubayed, Noura Al and Shum, Hubert P. H.},
 booktitle={Proceedings of the 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics},
 series={BHI '22},
 title={Towards Graph Representation Learning Based Surgical Workflow Anticipation},
 year={2022},
 month={9},
 pages={1--4},
 numpages={4},
 doi={10.1109/BHI56158.2022.9926801},
 publisher={IEEE},
 location={Ioannina, Greece},
}

RIS

TY  - CONF
AU  - Zhang, Francis Xiatian
AU  - Moubayed, Noura Al
AU  - Shum, Hubert P. H.
T2  - Proceedings of the 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics
TI  - Towards Graph Representation Learning Based Surgical Workflow Anticipation
PY  - 2022
Y1  - 9 2022
SP  - 1
EP  - 4
DO  - 10.1109/BHI56158.2022.9926801
PB  - IEEE
ER  - 

Plain Text

Francis Xiatian Zhang, Noura Al Moubayed and Hubert P. H. Shum, "Towards Graph Representation Learning Based Surgical Workflow Anticipation," in BHI '22: Proceedings of the 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics, pp. 1-4, Ioannina, Greece, IEEE, Sep 2022.

Supporting Grants

Similar Research

Francis Xiatian Zhang, Shuang Chen, Xianghua Xie and Hubert P. H. Shum, "Depth-Aware Endoscopic Video Inpainting", Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2024
Tanqiu Qiao, Ruochen Li, Frederick W. B. Li and Hubert P. H. Shum, "From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos", Proceedings of the 2024 International Conference on Pattern Recognition (ICPR), 2024
Tanqiu Qiao, Qianhui Men, Frederick W. B. Li, Yoshiki Kubotani, Shigeo Morishima and Hubert P. H. Shum, "Geometric Features Informed Multi-Person Human-Object Interaction Recognition in Videos", Proceedings of the 2022 European Conference on Computer Vision (ECCV), 2022
Qianhui Men, Howard Leung, Edmond S. L. Ho and Hubert P. H. Shum, "A Two-Stream Recurrent Network for Skeleton-Based Human Interaction Recognition", Proceedings of the 2020 International Conference on Pattern Recognition (ICPR), 2020

 

 

Last updated on 15 July 2024
RSS Feed