AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Computational Visual Media Article
PDF (13 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Review Article | Open Access

Addressing missing modality challenges in MRI images: A comprehensive review

Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen 52074, Germany
School of Advanced Technologies in Medicine, Iran University of Medical Sciences, Tehran 14496-14535, Iran
NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal, Montreal H3T 1J4, Canada
Functional Neuroimaging Unit, CRIUGM, University of Montreal, Montreal H3T 1J4, Canada
MILA, Quebec AI Institute, Montreal H2S 3H1, Canada
Faculty of Informatics and Data Science, University of Regensburg, Regensburg 93053, Germany
Fraunhofer Institute for Digital Medicine MEVIS, Bremen 28359, Germany
Show Author Information

Graphical Abstract

Abstract

Magnetic resonance imaging (MRI) is one of the most prevalent imaging modalities used for diagnosis, treatment planning, and outcome control in various medical conditions. MRI sequences provide physicians with the ability to view and monitor tissues at multiple contrasts within a single scan and serve as input for automated systems to perform downstream tasks. However, in clinical practice, there is usually no concise set of identically acquired sequences for a whole group of patients. As a consequence, medical professionals and automated systems both face difficulties due to the lack of complementary information from such missing sequences. This problem is well known in computer vision, particularly in medical image processing tasks such as tumor segmentation, tissue classification, and image generation. With the aim of helping researchers, this literature review examines a significant number of recent approaches that attempt to mitigate these problems. Basic techniques such as early synthesis methods, as well as later approaches that deploy deep learning, such as common latent space models, knowledge distillation networks, mutual information maximization, and generative adversarial networks (GANs) are examined in detail. We investigate the novelty, strengths, and weaknesses of the aforementioned strategies. Moreover, using a case study on the segmentation task, our survey offers quantitative benchmarks to further analyze the effectiveness of these methods for addressing the missing modalities challenge. Furthermore, a discussion offers possible future research directions.

Keywords

missing modalitysurveydeep learningmagnetic resonance imaging (MRI)

References

[1]
Ouyang, J.; Adeli, E.; Pohl, K. M.; Zhao, Q.; Zaharchuk, G. Representation disentanglement for multi-modal brain MRI analysis. In: Information Processing in Medical Imaging. Lecture Notes in Computer Science, Vol. 12729. Feragen, A.; Sommer, S.; Schnabel, J.; Nielsen, M. Eds. Springer Cham, 321–333, 2021.
Crossref
[2]

Dinsdale, N. K.; Bluemke, E.; Smith, S. M.; Arya, Z.; Vidaurre, D.; Jenkinson, M.; Namburete, A. Learning patterns of the ageing brain in MRI using deep convolutional networks. NeuroImage Vol. 224, 117401, 2021.
Crossref Google Scholar
[3]

Feng, C. M.; Wang, K.; Lu, S.; Xu, Y.; Li, X. Brain MRI super-resolution using coupled-projection residual network. Neurocomputing Vol. 456, 190–199, 2021.
Crossref Google Scholar
[4]
Conze, P. H.; Kavur, A. E.; Gall, E. C.; Gezer, N. S.; Le Meur, Y.; Selver, M. A.; Rousseau, F. Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks. arXiv preprint arXiv: 2001.09521, 2020.
Crossref
[5]

Biondetti, P.; Vangel, M. G.; Lahoud, R. M.; Furtado, F. S.; Rosen, B. R.; Groshar, D.; Canamaque, L G.; Umutlu, L.; Zhang, E. W.; Mahmood, U.; et al. PET/MRI assessment of lung nodules in primary abdominal malignancies: sensitivity and outcome analysis. European Journal of Nuclear Medicine and Molecular Imaging Vol. 48, No. 6, 1976–1986, 2021.
Crossref Google Scholar
[6]
Azad, R.; Asadi-Aghbolaghi, M.; Fathy, M.; Escalera, S. Bi-directional ConvLSTM U-Net with densley connected convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
Crossref
[7]

Reyngoudt, H.; Marty, B.; Boisserie, J. M.; Le Louër, J.; Koumako, C.; Baudin, P. Y.; Wong, B.; Stojkovic, T.; Béhin, A.; Gidaro, T.; et al. Global versus individual muscle segmentation to assess quantitative MRI-based fat fraction changes in neuromuscular diseases. European Radiology Vol. 31, No. 6, 4264–4276, 2021.
Crossref Google Scholar
[8]

Lee, Y. H. Efficiency improvement in a busy radiology practice: determination of musculoskeletal magnetic resonance imaging protocol using deep-learning convolutional neural networks. Journal of Digital Imaging Vol. 31, No. 5, 604–610, 2018.
Crossref Google Scholar
[9]
Azad, R.; Rouhier, L.; Cohen-Adad, J. Stacked hourglass network with a multi-level attention mechanism: Where to look for intervertebral disc labeling. In: Machine Learning in Medical Imaging. Lecture Notes in Computer Science, Vol. 12966. Lian, C.; Cao, X.; Rekik, I.; Xu, X.; Yan, P. Eds. Springer Cham, 406–415, 2021.
Crossref
[10]
Bozorgpour, A.; Azad, R.; Showkatian, E.; Sulaiman, A. Multi-scale regional attention Deeplab3+: Multiple myeloma plasma cells segmentation in microscopic images. arXiv preprint arXiv: 2105.06238, 2021.
[11]
Feyjie, A. R.; Azad, R.; Pedersoli, M.; Kauffman, C.; Ben Ayed, I.; Dolz, J. Semi-supervised few-shot learning for medical image segmentation. arXiv preprint arXiv: 2003.08462, 2020.
[12]

Brady, A. P. Error and discrepancy in radiology: Inevitable or avoidable? Insights into Imaging Vol. 8, No. 1, 171–182, 2017.
Crossref Google Scholar
[13]

Yao, W.; Liu, C.; Wang, N.; Zhou, H.; Chen, H.; Qiao, W. An MRI-guided targeting dual-responsive drug delivery system for liver cancer therapy. Journal of Colloid and Interface Science Vol. 603, 783–798, 2021.
Crossref Google Scholar
[14]

Delli Pizzi, A.; Chiarelli, A. M.; Chiacchiaretta, P.; D'Annibale, M.; Croce, P.; Rosa, C.; Mastrodicasa, D.; Trebeschi, S.; Lambregts, D. M. J.; Caposiena, D.; et al. MRI-based clinical-radiomics model predicts tumor response before treatment in locally advanced rectal cancer. Scientific Reports Vol. 11, No. 1, Article No. 5379, 2021.
Crossref Google Scholar
[15]

Yao, W.; Liu, C.; Wang, N.; Zhou, H.; Chen, H.; Qiao, W. Anisamide-modified dual-responsive drug delivery system with MRI capacity for cancer targeting therapy. Journal of Molecular Liquids Vol. 340, Article No. 116889, 2021.
Crossref Google Scholar
[16]

Bleker, J.; Yakar, D.; van Noort, B.; Rouw, D.; de Jong, I. D.; Dierckx, R.; Kwee, T.; Huisman, H. Single-center versus multi-center biparametric MRI radiomics approach for clinically significant peripheral zone prostate cancer. Insights into Imaging Vol. 12, Article No. 150, 2021.
Crossref Google Scholar
[17]

Park, Y. M.; Lim, J. Y.; Koh, Y. W.; Kim, S. H.; Choi, E. C. Prediction of treatment outcome using MRI radiomics and machine learning in oropharyngeal cancer patients after surgical treatment. Oral Oncology Vol. 122, Article No. 105559, 2021.
Crossref Google Scholar
[18]
Azad, R.; Khosravi, N.; Merhof, D. SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. arXiv preprint arXiv: 2204.02961, 2022.
[19]

Graves, M. J.; Mitchell, D. G. Body MRI artifacts in clinical practice: A physicist's and radiologist's perspective. Journal of Magnetic Resonance Imaging Vol. 38, No. 2, 269–287, 2013.
Crossref Google Scholar
[20]

Bekiesińska-Figatowska, M. Artifacts in magnetic resonance imaging. Polish Journal of Radiology Vol. 80, 93–106, 2015.
Crossref Google Scholar
[21]
Hamghalam, M.; Frangi, A. F.; Lei, B.; Simpson, A. L. Modality completion via gaussian process prior variational autoencoders for multi-modal glioma segmentation. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 12907. De Bruijne, Marleen.; Cattin, P. C.; Cotin, S.; Padoy, N.; Speidel, S.; Zheng, Y.; Essert, C. Eds. Springer Cham, 442–452, 2021.
Crossref
[22]

Dalmaz, O.; Yurt, M.; Çukur, T. ResViT: Residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging Vol. 41, No. 10, 2598–2614, 2022.
Crossref Google Scholar
[23]
Zhang, Y.; Yang, J.; Tian, J.; Shi, Z.; Zhong, C.; Zhang, Y.; He, Z. Modality-aware mutual learning for multi-modal medical image segmentation. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 12901. De Bruijne, Marleen.; Cattin, P. C.; Cotin, S.; Padoy, N.; Speidel, S.; Zheng, Y.; Essert, C. Eds. Springer Cham, 589–599, 2021.
Crossref
[24]

Zhou, T.; Canu, S.; Vera, P.; Ruan, S. Feature-enhanced generation and multi-modality fusion based deep neural network for brain tumor segmentation with missing MR modalities. Neurocomputing Vol. 466, 102–112, 2021.
Crossref Google Scholar
[25]
Zhu, Y.; Wang, S.; Lin, R.; Hu, Y.; Chen, Q. Brain tumor segmentation for missing modalities by supplementing missing features. In: Proceedings of the IEEE 6th International Conference on Cloud Computing and Big Data Analytics, 652–656, 2021.
Crossref
[26]
Yu, Z.; Zhai, Y.; Han, X.; Peng, T.; Zhang, X.-Y. MouseGAN: GAN-based multiple MRI modalities synthesis and segmentation for mouse brain structures. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 12901. De Bruijne, Marleen.; Cattin, P. C.; Cotin, S.; Padoy, N.; Speidel, S.; Zheng, Y.; Essert, C. Eds. Springer Cham, 442–450, 2021.
Crossref
[27]
Tulder, G. V.; de Bruijne, M. Why does synthesized data improve multi-sequence classification? In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 9349. Navab, N.; Hornegger, J.; Wells, W. M.; Frangi, A. Eds. Springer Cham, 531–538, 2021.
[28]

Jog, A.; Carass, A.; Roy, S.; Pham, D. L.; Prince, J. L. Random forest regression for magnetic resonance image synthesis. Medical Image Analysis Vol. 35, 475–488, 2017.
Crossref Google Scholar
[29]
Havaei, M.; Guizard, N.; Chapados, N.; Bengio, Y. Hemis: Hetero-modal image segmentation. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 9901. Ourselin, S.; Joskowicz, L.; Sabunce, M. R.; Unal, G.; Wells, W. Eds. Springer Cham, 469–477, 2016.
Crossref
[30]

Choi, Y.; Al-masni, M. A.; Jung, K. J.; Yoo, R. E.; Lee, S. Y.; Kim, D. H. A single stage knowledge distillation network for brain tumor segmentation on limited MR image modalities. Computer Methods and Programs in Biomedicine Vol. 240, Article No. 107644, 2023.
Crossref Google Scholar
[31]
Vadacchino, S.; Mehta, R.; Sepahvand, N.; Nichyporuk, B.; Clark, J. J.; Arbel, T. HAD-net: A hierarchical adversarial knowledge distillation network for improved enhanced tumour segmentation without post-contrast images. arXiv preprint arXiv: 2103.16617, 2021.
[32]
Wang, Q.; Zhan, L.; Thompson, P.; Zhou, J. Multimodal learning with incomplete modalities by knowledge distillation. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1828–1838, 2020.
Crossref
[33]
Karimijafarbigloo, S.; Azad, R.; Kazerouni, A.; Ebadollahi, S.; Merhof, D. MMCFormer: Missing modality compensation transformer for brain tumor segmentation. In: Proceedings of the Machine Learning Research, 1144–1162, 2023.
[34]

Zhou, T.; Canu, S.; Vera, P.; Ruan, S. Latent correlation representation learning for brain tumor segmentation with missing MRI modalities. IEEE Transactions on Image Processing Vol. 30, 4263–4274, 2021.
Crossref Google Scholar
[35]
Sylvain, T.; Dutil, F.; Berthier, T.; Di Jorio, L.; Luck, M.; Hjelm, D.; Bengio, Y. Cross-modal information maximization for medical imaging: CMIM. arXiv preprint arXiv: 2010.10593, 2020.
[36]
Pan, Y.; Chen, Y.; Shen, D.; Xia, Y. Collaborative image synthesis and disease diagnosis for classification of neurodegenerative disorders with incomplete multi-modal neuroimages. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 12905. Navab, N.; Hornegger, J.; Wells, W. M.; Frangi, A. Eds. Springer Cham, 480–489, 2021.
Crossref
[37]

Sharma, A.; Hamarneh, G. Missing MRI pulse sequence synthesis using multi-modal generative adversarial network. IEEE Transactions on Medical Imaging Vol. 39, No. 4, 1170–1183, 2020.
Crossref Google Scholar
[38]
Yu, B.; Zhou, L.; Wang, L.; Fripp, J.; Bourgeat, P. 3D cGAN based cross-modality MR image synthesis for brain tumor segmentation. In: Proceedings of the IEEE 15th International Symposium on Biomedical Imaging, 626–630, 2018.
Crossref
[39]
Isola, P.; Zhu, J. Y.; Zhou, T.; Efros, A. A. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5967–5976, 2017.
Crossref
[40]

Zhan, B.; Li, D.; Wang, Y.; Ma, Z.; Wu, X.; Zhou, J.; Zhou, L. LR-cGAN: Latent representation based conditional generative adversarial network for multi-modality MRI synthesis. Biomedical Signal Processing and Control Vol. 66, Article No. 102457, 2021.
Crossref Google Scholar
[41]

Hofmann, M.; Steinke, F.; Scheel, V.; Charpiat, G.; Farquhar, J.; Aschoff, P.; Brady, M.; Schölkopf, B.; Pichler, B. J. MRI-based attenuation correction for PET/MRI: A novel approach combining pattern recognition and atlas registration. Journal of Nuclear Medicine Vol. 49, No. 11, 1875–1883, 2008.
Crossref Google Scholar
[42]
Varsavsky, T.; Eaton-Rosen, Z.; Sudre, C. H.; Nachev, P.; Cardoso, J. M. PIMMS: Permutation invariant multi-modal segmentation. In: Deep Learning in Medical Imaging Analysis and Multimodal Learning for Clinical Decision Support. Lecture Notes in Computer Science, Vol. 11045. Stoyanov, D.; Taylor, Z.; Carneiro, G.; Syeda-Mahmood, T.; Martel, A.; Maier-Hein, L.; Tavares, J. M. R. S.; Bradley, A.; Papa, J. P.; Belagiannis, V.; et al. Eds. Springer Cham, 201–209, 2018.
Crossref
[43]
Mehta, R.; Arbel, T. RS-Net: Regression-segmentation 3D CNN for synthesis of full resolution missing brain MRI in the presence of tumours. arXiv preprint arXiv: 1807.10972, 2018.
Crossref
[44]
Shen, Y.; Gao, M. Brain tumor segmentation on MRI with missing modalities. In: Information Processing in Medical Imaging. Lecture Notes in Computer Science, Vol. 11492. Chung, A. C. S.; Gee, J. C.; Yushkevich, P. A.; Bao, S. Eds. Springer Cham, 417–428, 2019.
Crossref
[45]
Dorent, R.; Joutard, S.; Modat, M.; Ourselin, S.; Vercauteren, T. Hetero-modal variational encoder-decoder for joint modality completion and segmentation. In: Machine Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 11765. Navab, N.; Hornegger, J.; Wells, W. M.; Frangi, A. Eds. Springer Cham, 74–82, 2019.
Crossref
[46]
Wang, Y.; Zhang, Y.; Liu, Y.; Lin, Z.; Tian, J.; Zhong, C.; Shi, Z.; Fan, J.; He, Z. ACN: Adversarial co-training network for brain tumor segmentation with missing modalities. arXiv preprint arXiv: 2106.14591, 2021.
Crossref
[47]
Ding, Y.; Yu, X.; Yang, Y. RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 3955–3964, 2021.
Crossref
[48]
Lau, K.; Adler, J.; Sjölund, J. A unified representation network for segmentation with missing modalities. arXiv preprint arXiv: 1908.06683, 2019.
[49]
Konwer, A.; Hu, X.; Bae, J.; Xu, X.; Chen, C.; Prasanna, P. Enhancing modality-agnostic representations via meta-learning for brain tumor segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 21415–21425, 2023.
Crossref
[50]
Liu, H.; Wei, D.; Lu, D.; Sun, J.; Wang, L.; Zheng, Y. M3AE: Multimodal representation learning for brain tumor segmentation with missing modalities. arXiv preprint arXiv: 2303.05302, 2023.
Crossref
[51]
Zhou, T.; Canu, S.; Vera, P.; Ruan, S. Conditional generator and multi-sourcecorrelation guided brain tumor segmentation with missing MR modalities. arXiv preprint arXiv: 2105.13013, 2021.
[52]

Zhou, T. Feature fusion and latent feature learning guided brain tumor segmentation and missing modality recovery network. Pattern Recognition Vol. 141, Article No. 109665, 2023.
Crossref Google Scholar
[53]

Cao, B.; Zhang, H.; Wang, N.; Gao, X.; Shen, D. Auto-GAN: Self-supervised collaborative learning for medical image synthesis. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 7, 10486–10493, 2020.
Crossref Google Scholar
[54]
Li, H.; Paetzold, J. C.; Sekuboyina, A.; Kofler, F.; Zhang, J.; Kirschke, J. S.; Wiestler, B.; Menze, B. DiamondGAN: Unified multi-modal generative adversarial networks for MRI sequences synthesis. arXiv preprint arXiv: 1904.12894, 2019.
Crossref
[55]

Qian, S.; Wang, C. COM: Contrastive Masked-attention model for incomplete multimodal learning. Neural Networks Vol. 162, 443–455, 2023.
Crossref Google Scholar
[56]
Islam, S. K.; Nasim, M. D.; Hossain, I.; Ullah, M. A.; Gupta, K. D.; Bhuiyan, M. M. H. Introduction of medical imaging modalities. arXiv preprint arXiv: 2306.01022, 2023.
Crossref
[57]

Webb, G. A. Introduction to Biomedical Imaging. John Wiley & Sons, 2017.
[58]
Weishaupt, D.; Köchli, V. D.; Marincek, B. How does MRI Work? An Introduction to the Physics and Function of Magnetic Resonance Imaging, 2nd edn. Springer, 2006.
[59]
Baba, Y.; Jones, J. A. T1 weighted image. 2009. Available at https://radiopaedia.org/articles/5852
Crossref
[60]
Haouimi, A.; Jones, J. T2 weighted image. 2009. Available at https://radiopaedia.org/articles/6345
[61]
Baba, Y.; Niknejad, M. Fluid attenuated inversion recovery. 2013. Available at https://radiopaedia.org/articles/21760
Crossref
[62]

Tanner, M.; Gambarota, G.; Kober, T.; Krueger, G.; Erritzoe, D.; Marques, J. P.; Newbould, R. Fluid and white matter suppression with the MP2RAGE sequence. Journal of Magnetic Resonance Imaging Vol. 35, No. 5, 1063–1070, 2012.
Crossref Google Scholar
[63]

Yu, H.; Buch, K.; Li, B.; O'Brien, M.; Soto, J.; Jara, H.; Anderson, S. W. Utility of texture analysis for quantifying hepatic fibrosis on proton density MRI. Journal of Magnetic Resonance Imaging Vol. 42, No. 5, 1259–1265, 2015.
Crossref Google Scholar
[64]

Carass, A.; Roy, S.; Jog, A.; Cuzzocreo, J. L.; Magrath, E.; Gherman, A.; Button, J.; Nguyen, J.; Bazin, P. L.; Calabresi, P. A.; et al. Longitudinal multiple sclerosis lesion segmentation data resource. Data in Brief Vol. 12, 346–350, 2017.
Crossref Google Scholar
[65]

Menze, B. H.; Jakab, A.; Bauer, S.; Kalpathy-Cramer, J.; Farahani, K.; Kirby, J.; Burren, Y.; Porz, N.; Slotboom, J.; Wiest, R.; et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging Vol. 34, No. 10, 1993–2024, 2015.
Crossref Google Scholar
[66]
Ashton, E. A.; Kim, S. H. Evaluation of reproducibility for perfusion assessment of tumors in MRI. In: Proceedings of the 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 824–827, 2004.
Crossref
[67]

Brown, M. A.; Semelka, R. C. MRI: Basic Principles and Applications. John Wiley & Sons, 2015.
[68]

Rauf, N.; Alam, D. Y.; Jamaluddin, M.; Samad, B. A. Improve image quality of transversal relaxation time PROPELLER and FLAIR on magnetic resonance imaging. Journal of Physics: Conference Series Vol. 979, Article No. 012079, 2018.
Crossref Google Scholar
[69]
Mudgal, P. Case courtesy of Dr. Prashant Mudgal. 2012. Available at https://radiopaedia.org/cases/26952/studies/27131
[70]
Ballinger, J. R. Polycystic ovaries. 2013. Available at https://radiopaedia.org/cases/polycystic-ovaries
Crossref
[71]
Fischer, A.; Igel, C. An introduction to restricted boltzmann machines. In: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. Lecture Notes in Computer Science, Vol. 7741. Alvarez, L.; Mejail, M.; Gomez, L.; Jacobo, J. Eds. Springer Cham, 14–36, 2012.
Crossref
[72]

Wang, Y.; Hu, H.; Yu, S.; Yang, Y.; Guo, Y.; Song, X.; Chen, F.; Liu, Q. A unified hybrid transformer for joint MRI sequences super-resolution and missing data imputation. Physics in Medicine & Biology Vol. 68, No. 13, Article No. 135006, 2023.
Crossref Google Scholar
[73]
Çiçek, Ö.; Abdulkadir, A.; Lienkamp, S. S.; Brox, T.; Ronneberger, O. 3D U-Net: Learning dense volumetric segmentation from sparse annotation. arXiv preprint arXiv: 1606.06650, 2016.
[74]
Wu, M.; Goodman, N. Multimodal generative models for scalable weakly-supervised learning. arXiv preprint arXiv: 1802.05335, 2018.
[75]

Liu, J.; Pasumarthi, S.; Duffy, B.; Gong, E.; Datta, K.; Zaharchuk, G. One model to synthesize them all: Multi-contrast multi-scale transformer for missing data imputation. IEEE Transactions on Medical Imaging Vol. 42, No. 9, 2577–2591, 2023.
Crossref Google Scholar
[76]
Bucilua, C.; Caruana, R.; Niculescu-Mizil, A. Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 535–541, 2006.
Crossref
[77]
Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02531, 2015.
[78]
Isensee, F.; Kickingereder, P.; Martin Bendszus, W.; Maier-Hein, K. H. No new-net. arXiv preprint arXiv: 1809.10483, 2018.
[79]
Belghazi, M. I.; Baratin, A.; Rajeswar, S.; Ozair, S.; Bengio, Y.; Courville, A.; Hjelm, R. D. MINE: Mutual information neural estimation. arXiv preprint arXiv: 1801.04062, 2018.
[80]
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. arXiv preprint arXiv: 1406.2661, 2014.
[81]
Pan, K.; Cheng, P.; Huang, Z.; Lin, L.; Tang, X. Transformer-based T2-weighted MRI synthesis from T1-weighted images. In: Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society, 5062–5065, 2022.
Crossref
[82]

Chen, X.; Lian, C.; Wang, L.; Deng, H.; Kuang, T.; Fung, S.; Gateno, J.; Yap, P. T.; Xia, J. J.; Shen, D. Anatomy-regularized representation learning for cross-modality medical image segmentation. IEEE Transactions on Medical Imaging Vol. 40, No. 1, 274–285, 2021.
Crossref Google Scholar
[83]
Hu, M.; Maillard, M.; Zhang, Y.; Ciceri, T.; La Barbera, G.; Bloch, I.; Gori, P. Knowledge distillation from multi-modal to mono-modal segmentation networks. arXiv preprint arXiv: 2106.09564, 2020.
Crossref
[84]

Mirzadeh, S. I.; Farajtabar, M.; Li A.; Levine, N.; Matsukawa, A.; Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. Proceedings of the AAAI Conference on Artificial Intelligence Vol. 34, No. 4, 5191–5198, 2020.
Crossref Google Scholar
[85]
Cho, J. H.; Hariharan, B. On the efficacy of knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4793–4801, 2019.
Crossref
[86]
Cho, J. H.; Hariharan, B. On the efficacy of knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 4793–4801, 2019.
Crossref
[87]
Chang, Q.; Yan, Z.; Baskaran, L.; Qu, H.; Zhang, Y.; Zhang, T.; Zhang, S.; Metaxas, D. N. Multi-modal AsynDGAN: Learn from distributed medical image data without sharing private information. arXiv preprint arXiv: 2012.08604, 2020.
[88]
Huang, P.; Li, D.; Jiao, Z.; Wei, D.; Li, G.; Wang, Q.; Zhang, H.; Shen, D. CoCa-GAN: Common-feature-learning-based context-aware generative adversarial network for glioma grading. In: Medical Imaging Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, Vol. 11766. Shen, D.; Liu, T.; Peters, T. M.; Staib, L. H.; Essert, C.; Zhou, S.; Yap, P.-T.; Khan, A. Eds. Springer Cham, 155–163, 2019.
Crossref
[89]
Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved techniques for training GANs. arXiv preprint arXiv: 1606.03498, 2016.
[90]
Karras, T.; Aittala, M.; Hellsten, J.; Laine, S.; Lehtinen, J.; Aila, T. Training generative adversarial networks with limited data. arXiv preprint arXiv: 2006.06676, 2020.
[91]
Myronenko, A. 3D MRI brain tumor segmentation using autoencoder regularization. arXiv preprint arXiv: 1810.11654, 2024.
[92]
Styner, M.; Lee, J.; Chin, B.; Chin, M.; Commowick, O.; Tran, H.; Markovic-Plese, S.; Jewells, V.; Warfield, S. 3D segmentation in the clinic: A grand challenge Ⅱ: MS lesion segmentation. The MIDAS Journal 2008, http://hdl.handle.net/10380/1509
Crossref
[93]

Tillin, T.; Forouhi, N. G.; McKeigue, P. M.; Chaturvedi, N. Southall and Brent REvisited: Cohort profile of SABRE, a UK population-based comparison of cardiovascular disease and diabetes in people of European, Indian Asian and African Caribbean origins. International Journal of Epidemiology Vol. 41, No. 1, 33–42, 2012.
Crossref Google Scholar
[94]
Kuijf, H. J. WMH segmentation challenge. 2017. Available at https://wmh.isi.uu.nl/
[95]

Maier, O.; Menze, B. H.; von der Gablentz, J.; Häni, L.; Heinrich, M. P.; Liebrand, M.; Winzeck, S.; Basit, A.; Bentley, P.; Chen, L.; et al. A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Medical Image Analysis Vol. 35, 250–269, 2017.
Crossref Google Scholar
[96]

Kavur, A. E.; Gezer, N. S.; Barıç, M.; Aslan, S.; Conze, P. H.; Groza, V.; Pham, D. D.; Chatterjee, S.; Ernst, P.; Özkan, S.; et al. CHAOS Challenge-combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis Vol. 69, Article No. 101950, 2021.
Crossref Google Scholar
[97]

Commowick, O.; Kain, M.; Casey, R.; Ameli, R.; Ferré, J. C.; Kerbrat, A.; Tourdias, T.; Cervenansky, F.; Camarasu-Pop, S.; Glatard, T.; et al. Multiple sclerosis lesions segmentation from multiple experts: The MICCAI 2016 challenge dataset. NeuroImage Vol. 244, Article No. 118589, 2021.
Crossref Google Scholar
[98]
Brudfors, M.; Ashburner, J.; Nachev, P.; Balbastre, Y. Empirical bayesian mixture models for medical image translation. arXiv preprint arXiv: 1908.05926, 2019.
Crossref
[99]

MacDonald, J. A.; Zhu Z.; Konkel, B.; Mazurowski, M. A.; Wiggins, W. F.; Bashir, M. R. Duke liver dataset: A publicly available liver MRI dataset with liver segmentation masks and series labels. Radiology: Artificial Intelligence Vol. 5, No. 5, Article No. e220275, 2023.
Crossref Google Scholar
[100]
Giacomello, E.; Loiacono, D.; Mainardi, L. Brain MRI tumor segmentation with adversarial networks. arXiv preprint arXiv: 1910.02717, 2019.
[101]
Chen, C.; Dou, Q.; Jin, Y.; Chen, H.; Qin, J.; Heng, P.-A. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion. arXiv preprint arXiv: 2002.09708, 2019.
Crossref
[102]

Islam, M.; Wijethilake, N.; Ren, H. Glioblastoma multiforme prognosis: MRI missing modality generation, segmentation and radiogenomic survival prediction. Computerized Medical Imaging and Graphics Vol. 91, Article No. 101906, 2021.
Crossref Google Scholar
[103]

Zhou, T.; Fu, H.; Chen, G.; Shen, J.; Shao, L. Hi-net: Hybrid-fusion network for multi-modal MR image synthesis. IEEE Transactions on Medical Imaging Vol. 39, No. 9, 2772–2781, 2020.
Crossref Google Scholar
[104]

Jack Jr, C. R.; Bernstein, M. A.; Fox, N. C.; Thompson, P.; Alexander, G.; Harvey, D.; Borowski, B.; Britson, P. R.; Whitwell, J. L.; Ward, C.; et al. The Alzheimer's disease neuroimaging initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging Vol. 27, No. 4, 685–691, 2008.
Crossref Google Scholar
[105]
Orbes-Arteaga, M.; Cardoso, M. J.; SØrensen, L.; Modat, M.; Ourselin, S.; Nielsen, M.; Pai, A. Simultaneous synthesis of FLAIR and segmentation of white matter hypointensities from T1 MRIs. arXiv preprint arXiv: 1808.06519, 2018.
[106]

Yuan, W.; Wei, J.; Wang, J.; Ma, Q.; Tasdizen, T. Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images. Medical Image Analysis Vol. 64, Article No. 101731, 2020.
Crossref Google Scholar
[107]
Simpson, A. L.; Antonelli, M.; Bakas, S.; Bilello, M.; Farahani, K.; van Ginneken, B.; Kopp-Schneider, A.; Landman, B. A.; Litjens, G.; Menze, B.; et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv: 1902.09063, 2019.
[108]

Zhu, Z.; Mittendorf, A.; Shropshire, E.; Allen, B.; Miller, C.; Bashir, M. R.; Mazurowski, M. A. 3D pyramid pooling network for abdominal MRI series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 44, No. 4, 1688–1698, 2022.
Crossref Google Scholar
[109]
Milletari, F.; Navab, N.; Ahmadi, S. A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: Proceedings of the 4th International Conference on 3D Vision, 565–571, 2016.
Crossref
[110]
Isensee, F.; Petersen, J.; Klein, A.; Zimmerer, D.; Jaeger, P. F.; Kohl, S.; Wasserthal, J.; Koehler, G.; Norajitra, T.; Wirkert, S.; et al. nnU-Net: Self-adapting framework for U-Net-Based medical image segmentation. arXiv preprint arXiv: 1809.10486, 2019.
Crossref
[111]
Jacob, B.; Kligys, S.; Chen, B.; Zhu, M.; Tang, M.; Howard, A.; Adam, H.; Kalenichenko, D. Quantization and training of neural networks for efficient integerarithmetic-only inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–2713, 2018.
Crossref
[112]
Huang, J.; Rathod, V.; Sun, C.; Zhu, M.; Korattikara, A.; Fathi, A.; Fischer, I.; Wojna, Z.; Song, Y.; Guadarrama, S.; et al. Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3296–3297, 2017.
Crossref
[113]
Tan, M.; Le, Q. V. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1905.11946, 2019.
[114]

Matsuo, K.; Tanaka, Y.; Sarmenta, L. F. G.; Nakai, T.; Bagarinao, E. Enabling on-demand real-time functional MRI analysis using grid technology. Methods of Information in Medicine Vol. 44, No. 5, 665–673, 2005.
Crossref Google Scholar
[115]
Azad, R.; Aghdam, E. K.; Rauland, A.; Jia, Y.; Avval, A. H.; Bozorgpour, A.; Karimijafarbigloo, S.; Cohen, J. P.; Adeli, E.; Merhof, D. Medical image segmentation review: The success of U-Net. arXiv preprint arXiv: 2211.14830, 2022.
[116]
Wang, H.; Wang, Z.; Du, M.; Yang, F.; Zhang, Z.; Ding, S.; Mardziel, P.; Hu, X. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 24–25, 2020.
Crossref
[117]
Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, 618–626, 2017.
Crossref
[118]
Gros, C.; Lemay, A.; Vincent, O.; Rouhier, L.; Bucquet, A.; Cohen, J. P.; Cohen-Adad, J. Ivadomed: A medical imaging deep learning toolbox. arXiv preprint arXiv: 2010.09984, 2020.
Crossref
[119]
Cardoso, M. J.; Li, W.; Brown, R.; Ma, N.; Kerfoot, E.; Wang, Y.; Murrey, B.; Myronenko, A.; Zhao, C.; Yang, D.; et al. MONAI: An open-source framework for deep learning in healthcare. arXiv preprint arXiv: 2211.02701, 2022.
[120]
Jaeger, P. F.; Kohl, S. A. A.; Bickelhaupt, S.; Isensee, F.; Kuder, T. A.; Schlemmer, H. P.; Maier-Hein, K. H. Retina U-Net: Embarrassingly simple exploitation of segmentation supervision for medical object detection. arXiv preprint arXiv: 1811.08661, 2018.
[121]
Pawlowski, N.; Ktena, S. I.; Lee, M. C. H.; Kainz, B.; Rueckert, D.; Glocker, B.; Rajchl, M. DLTK: State of the art reference implementations for deep learning on medical images. arXiv preprint arXiv: 1711.06853, 2017.
[122]
Kazerouni, A.; Aghdam, E. K.; Heidari, M.; Azad, R.; Fayyaz, M.; Hacihaliloglu, I.; Merhof, D. Diffusion models in medical imaging: A comprehensive survey. arXiv preprint arXiv: 2211.07804, 2022.
Crossref
[123]
Jiang, L.; Mao, Y.; Wang, X.; Chen, X.; Li, C. CoLa-Diff: Conditional latent diffusion model for multi-modal MRI synthesis. arXiv preprint arXiv: 2303.14081, 2023.
Crossref
[124]
Meng, X.; Gu, Y.; Pan, Y.; Wang, N.; Xue, P.; Lu, M.; He, X.; Zhan, Y.; Shen, D. A novel unified conditional score-based generative framework for multi-modal medical image completion. arXiv preprint arXiv: 2207.03430, 2022.
[125]
Molaei, A.; Aminimehr, A.; Tavakoli, A.; Kazerouni, A.; Azad, B.; Azad, R.; Merhof, D. Implicit neural representation in medical imaging: A comparative survey. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2373–2383, 2023.
Crossref
[126]
Iddrisu, K.; Malec, S.; Crimi, A. 3D reconstructions of brain from MRI scans using neural radiance fields. In: Artificial Intelligence and Soft Computing. Lecture Notes in Computer Science, Vol. 14126. Rutkowski, L.; Scherer, R.; Korytkowski, M.; Pedrycz, W.; Tadeusiewicz, R.; Zurada, J. M. Eds. Springer Cham, 207–218, 2023.
Crossref
Computational Visual Media
Volume 11 Issue 2,
April 2025
Pages 241-268
DOI: 10.26599/CVM.2025.9450399
Cite this article:
Azad R, Dehghanmanshadi M, Khosravi N, et al. Addressing missing modality challenges in MRI images: A comprehensive review. Computational Visual Media, 2025, 11(2): 241-268. https://doi.org/10.26599/CVM.2025.9450399

43

Views

6

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 27 May 2023
Accepted: 27 December 2023
Published: 08 May 2025
© The Author(s) 2025.

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

To submit a manuscript, please go to https://jcvm.org.
Return