Depth-Aware Endoscopic Video Inpainting

Francis Xiatian Zhang, Shuang Chen, Xianghua Xie and Hubert P. H. Shum
Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2024

 H5-Index: 96# Core A Conference

Depth-Aware Endoscopic Video Inpainting
‡ According to Core Ranking 2023
# According to Google Scholar 2024

Abstract

Video inpainting fills in corrupted video content with plausible replacements. While recent advances in endoscopic video inpainting have shown potential for enhancing the quality of endoscopic videos, they mainly repair 2D visual information without effectively preserving crucial 3D spatial details for clinical reference. Depth-aware inpainting methods attempt to preserve these details by incorporating depth information. Still, in endoscopic contexts, they face challenges including reliance on pre-acquired depth maps, less effective fusion designs, and ignorance of the fidelity of 3D spatial details. To address them, we introduce a novel Depth-aware Endoscopic Video Inpainting (DAEVI) framework. It features a Spatial-Temporal Guided Depth Estimation module for direct depth estimation from visual features, a Bi-Modal Paired Channel Fusion module for effective channel-by-channel fusion of visual and depth information, and a Depth Enhanced Discriminator to assess the fidelity of the RGB-D sequence comprised of the inpainted frames and estimated depth images. Experimental evaluations on established benchmarks demonstrate our framework’s superiority, achieving a 2% improvement in PSNR and a 6% reduction in MSE compared to state-of-the-art methods. Qualitative analyses further validate its enhanced ability to inpaint fine details, highlighting the benefits of integrating depth information into endoscopic inpainting.


Downloads


YouTube


Cite This Research

Plain Text

Francis Xiatian Zhang, Shuang Chen, Xianghua Xie and Hubert P. H. Shum, "Depth-Aware Endoscopic Video Inpainting," in MICCAI '24: Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 143-153, Marrakesh, Morocco, Springer, 2024.

BibTeX

@inproceedings{zhang24endoscopic,
 author={Zhang, Francis Xiatian and Chen, Shuang and Xie, Xianghua and Shum, Hubert P. H.},
 booktitle={Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention},
 series={MICCAI '24},
 title={Depth-Aware Endoscopic Video Inpainting},
 year={2024},
 pages={143--153},
 doi={10.1007/978-3-031-72089-5_14},
 isbn={978-3-031-72089-5},
 publisher={Springer},
 location={Marrakesh, Morocco},
}

RIS

TY  - CONF
AU  - Zhang, Francis Xiatian
AU  - Chen, Shuang
AU  - Xie, Xianghua
AU  - Shum, Hubert P. H.
T2  - Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention
TI  - Depth-Aware Endoscopic Video Inpainting
PY  - 2024
SP  - 143
EP  - 153
DO  - 10.1007/978-3-031-72089-5_14
SN  - 978-3-031-72089-5
PB  - Springer
ER  - 


Supporting Grants

The Engineering and Physical Sciences Research Council
Northern Health Futures Hub (NortHFutures)
EPSRC Digital Health Hub Pilot Scheme (Ref: EP/X031012/1): £4.17 million, Co-Investigator (PI: Prof. Abigail Durrant)
Received from The Engineering and Physical Sciences Research Council, UK, 2023-2026
Project Page

Similar Research

Francis Xiatian Zhang, Jingjing Deng, Robert Lieck and Hubert P. H. Shum, "Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation", IEEE Transactions on Medical Robotics and Bionics (TMRB), 2025
Abril Corona-Figueroa, Hubert P. H. Shum and Chris G. Willcocks, "Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling", Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024
Francis Xiatian Zhang, Noura Al Moubayed and Hubert P. H. Shum, "Towards Graph Representation Learning Based Surgical Workflow Anticipation", Proceedings of the 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), 2022

 

RSS Feed