Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling

Abril Corona-Figueroa, Hubert P. H. Shum and Chris G. Willcocks
Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024

 Best Paper Award H5-Index: 115#

Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling
# According to Google Scholar 2024

Abstract

This paper investigates a 2D to 3D image translation method with a straightforward technique, enabling correlated 2D X-ray to 3D CT-like reconstruction. We observe that existing approaches, which integrate information across multiple 2D views in the latent space, lose valuable signal information during latent encoding. Instead, we simply repeat and concatenate the 2D views into higher-channel 3D volumes and approach the 3D reconstruction challenge as a straightforward 3D to 3D generative modeling problem, sidestepping several complex modeling issues. This method enables the reconstructed 3D volume to retain valuable information from the 2D inputs, which are passed between channel states in a Swin UNETR backbone. Our approach applies neural optimal transport, which is fast and stable to train, effectively integrating signal information across multiple views without the requirement for precise alignment; it produces non-collapsed reconstructions that are highly faithful to the 2D views, even after limited training. We demonstrate correlated results, both qualitatively and quantitatively, having trained our model on a single dataset and evaluated its generalization ability across six datasets, including out-of-distribution samples.


Downloads


YouTube


Cite This Research

Plain Text

Abril Corona-Figueroa, Hubert P. H. Shum and Chris G. Willcocks, "Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling," in CVPRW '24: Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, USA, IEEE/CVF, 2024.

BibTeX

@inproceedings{coronafigueroaa24repeat,
 author={Corona-Figueroa, Abril and Shum, Hubert P. H. and Willcocks, Chris G.},
 booktitle={Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
 series={CVPRW '24},
 title={Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling},
 year={2024},
 publisher={IEEE/CVF},
 location={Seattle, USA},
}

RIS

TY  - CONF
AU  - Corona-Figueroa, Abril
AU  - Shum, Hubert P. H.
AU  - Willcocks, Chris G.
T2  - Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops
TI  - Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling
PY  - 2024
PB  - IEEE/CVF
ER  - 


Supporting Grants


Similar Research

Abril Corona-Figueroa, Jonathan Frawley, Sam Bond-Taylor, Sarath Bethapudi, Hubert P. H. Shum and Chris G. Willcocks, "MedNeRF: Medical Neural Radiance Fields for Reconstructing 3D-Aware CT-projections from a Single X-ray", Proceedings of the 2022 International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2022
Abril Corona-Figueroa, Sam Bond-Taylor, Neelanjan Bhowmik, Yona Falinie A. Gaus, Toby P. Breckon, Hubert P. H. Shum and Chris G. Willcocks, "Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers", Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023
Francis Xiatian Zhang, Shuang Chen, Xianghua Xie and Hubert P. H. Shum, "Depth-Aware Endoscopic Video Inpainting", Proceedings of the 2024 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2024

 

Last updated on 6 October 2024
RSS Feed