Journal Home > Volume 28 , Issue 4

As a deep learning network with an encoder-decoder architecture, UNet and its series of improved versions have been widely used in medical image segmentation with great applications. However, when used to segment targets in 3D medical images such as magnetic resonance imaging (MRI), computed tomography (CT), these models do not model the relevance of images in vertical space, resulting in poor accurate analysis of consecutive slices of the same patient. On the other hand, the large amount of detail lost during the encoding process makes these models incapable of segmenting small-scale tumor targets. Aiming at the scene of small-scale target segmentation in 3D medical images, a fully new neural network model SUNet++ is proposed on the basis of UNet and UNet++. SUNet++ improves the existing models mainly in three aspects: 1) the modeling strategy of slice superposition is used to thoroughly excavate the three dimensional information of the data; 2) by adding an attention mechanism during the decoding process, small scale targets in the picture are retained and amplified; 3) in the up-sampling process, the transposed convolution operation is used to further enhance the effect of the model. In order to verify the effect of the model, we collected and produced a dataset of hyperintensity MRI liver-stage images containing over 400 cases of liver nodules. Experimental results on both public and proprietary datasets demonstrate the superiority of SUNet++ in small-scale target segmentation of three-dimensional medical images.


menu
Abstract
Full text
Outline
About this article

SUNet++: A Deep Network with Channel Attention for Small-Scale Object Segmentation on 3D Medical Images

Show Author's information Lan Zhang1Kejia Zhang1( )Haiwei Pan1
College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, China

Abstract

As a deep learning network with an encoder-decoder architecture, UNet and its series of improved versions have been widely used in medical image segmentation with great applications. However, when used to segment targets in 3D medical images such as magnetic resonance imaging (MRI), computed tomography (CT), these models do not model the relevance of images in vertical space, resulting in poor accurate analysis of consecutive slices of the same patient. On the other hand, the large amount of detail lost during the encoding process makes these models incapable of segmenting small-scale tumor targets. Aiming at the scene of small-scale target segmentation in 3D medical images, a fully new neural network model SUNet++ is proposed on the basis of UNet and UNet++. SUNet++ improves the existing models mainly in three aspects: 1) the modeling strategy of slice superposition is used to thoroughly excavate the three dimensional information of the data; 2) by adding an attention mechanism during the decoding process, small scale targets in the picture are retained and amplified; 3) in the up-sampling process, the transposed convolution operation is used to further enhance the effect of the model. In order to verify the effect of the model, we collected and produced a dataset of hyperintensity MRI liver-stage images containing over 400 cases of liver nodules. Experimental results on both public and proprietary datasets demonstrate the superiority of SUNet++ in small-scale target segmentation of three-dimensional medical images.

Keywords: segmentation, attention mechanism, 3D medical images, small-scale target

References(37)

[1]
J. Long, E. Shelhamer, and T. Darrell, Fully convolutional networks for semantic segmentation. in Proc. 2015 IEEE Conf.on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 3431–3440, 2015.
[2]
O. Ronneberger, P. Fischer, and T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in Proc. 18th Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 2015, pp. 234–241.
[3]
L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, Semantic image segmentation with deep convolutional nets and fully connected CRFs, presented at the 3rd Int. Conf. on Learning Representations, San Diego, CA, USA, 2015.
[4]
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, Pyramid scene parsing network. in Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 6230–6239.
[5]
R. Girshick, J. Donahue, T. Darrell, and J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in Proc. 2014 IEEE Conf. on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014, pp. 580–587, 2014.
[6]
S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, 2017.
[7]
Z. Zhou, M. R. Siddiquee, N. Tajbakhsh, and J. Liang, UNet++: A nested U-net architecture for medical image segmentation, in Proc. 4th Int. Workshop on Deep Learning in Medical Image Analysis, Granada, Spain, 2018, pp. 3–11.
[8]
F. Milletari, N. Navab, and S. A. Ahmadi, V-Net: Fully convolutional neural networks for volumetric medical image segmentation, in Proc. 4th Int. Conf. on 3D Vision (3DV), Stanford, CA, USA, 2016, pp. 565–571.
[9]
Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, 3D U-Net: Learning dense volumetric segmentation from sparse annotation, in Proc. 19th Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, 2016, pp. 424–432.
[10]
Z. Cai, X. Zheng, and J. Yu, A differential-private framework for urban traffic flows estimation via taxi companies, IEEE Trans. Industr. Inform., vol. 15, no. 12, pp. 6492–6499, 2019.
[11]
Y. Liang, Z. Cai, J. Yu, Q. Han, and Y. Li, Deep learning based inference of private information using embedded sensors in smart devices, IEEE Network, vol. 32, no. 4, pp. 8–14, 2018.
[12]
A. Alansary, K. Kamnitsas, A. Davidson, R. Khlebnikov, M. Rajchl, C. Malamateniou, M. Rutherford, J. V. Hajnal, B. Glocker, D. Rueckert, et al., Fast fully automatic segmentation of the human placenta from motion corrupted MRI, in Proc. 19th Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, 2016, pp. 589–597.
[13]
M. Shakeri, S. Tsogkas, E. Ferrante, S. Lippe, S. Kadoury, N. Paragios, and I. Kokkinos, Sub-cortical brain structure segmentation using F-CNN’S, in Proc. 13th Int. Sympos. on Biomedical Imaging (ISBI), Prague, Czech Republic, 2016, pp. 269–272.
[14]
P. Moeskops, J. M. Wolterink, B. H. M. van der Velden, K. G. A. Gilhuijs, T. Leiner, M. A. Viergever, and I. Išgum, Deep learning for multi-task medical image segmentation in multiple modalities, in Proc. 19th Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, 2016, pp. 478–486.
[15]
X. Zhou, T. Ito, R. Takayama, S. Wang, T. Hara, and H. Fujita, Three-dimensional CT image segmentation by combining 2D fully convolutional network with 3D majority voting, in Proc. 1st Int. Workshop on Deep Learning in Medical Image Analysis, Athens, Greece, 2016, pp. 111–120.
[16]
R. Korez, B. Likar, F. Pernuš, and T. Vrtovec, Model-based segmentation of vertebral bodies from MR images with 3D CNNs, in Proc. 19th Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, 2016, pp. 433–441.
[17]
M. Drozdzal, E. Vorontsov, G. Chartrand, S. Kadoury, and C. Pal, The importance of skip connections in biomedical image segmentation, in Proc. 1st Int. Workshop on Deep Learning in Medical Image Analysis, Athens, Greece, 2016, pp. 179–187.
[18]
K. Kamnitsas, C. Ledig, V. F. J. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation, Med. Image Anal., vol. 36, pp. 61–78, 2017.
[19]
M. Ghafoorian, N. Karssemeijer, T. Heskes, I. W. M. van Uder, F. E. de Leeuw, E. Marchiori, B. van Ginneken, and B. Platel, Non-uniform patch sampling with deep convolutional neural networks for white matter hyperintensity segmentation. in Proc. 13th Int. Sympos. on Biomedical Imaging, Prague, Czech Republic, 2016, pp. 1414–1417.
[20]
T. Brosch, L. Y. W. Tang, Y. Yoo, D. K. B. Li, A. Traboulsee, and R. Tam, Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation, IEEE Trans. Med. Imaging, vol. 35, no. 5, pp. 1229–1239, 2016.
[21]
H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y. W. Chen, and J. Wu, UNet 3+: A full-scale connected UNet for medical image segmentation, in Proc. 2020 IEEE Int. Conf. on Acoustics, Speech and Signal Processing, Barcelona, Spain, 2020, pp. 1055–1059.
[22]
A. E. Kavur, N. S. Gezer, M. Barış S. Aslan, P. H. Conze, V. Groza, D. D. Pham, S. Chatterjee, P. Ernst, S. Özkan, et al., CHAOS challenge-combined (CT-MR) healthy abdominal organ segmentation, Med. Image Anal., vol. 69, p. 101950, 2021.
[23]
A. Andreopoulos and J. K. Tsotsos, Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI, Med. Image Anal., vol. 12, no. 3, pp. 335–357, 2008.
[24]
O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P. A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, et al., Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Trans. Med. Imaging, vol. 37, no. 11, pp. 2514–2525, 2018.
[25]
G. Litjens, R. Toth, W. van de Ven, C. Hoeks, S. Kerkstra, B. van Ginneken, G. Vincent, G. Guillard, N. Birbeck, J. Zhang, et al., Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge, Med. Image Anal., vol. 18, no. 2, pp. 359–373, 2014.
[26]
P. Christ, LiTS-liver tumor segmentation challenge (LiTS17), https://academictorrents.com/details/27772adef6f563a1ecc0ae19a528b956e6c803ce, 2017.
[27]
Z. Cai, Z. He, X. Guan, and Y. Li, Collective data-sanitization for preventing sensitive information inference attacks in social networks, IEEE Trans. Dependable Secure Comput., vol. 15, no. 4, pp. 577–590, 2018.
[28]
Z. Cai and X. Zheng, A private and efficient mechanism for data uploading in smart cyber-physical systems, IEEE Trans. Netw. Sci. Eng., vol. 7, no. 2, pp. 766–775, 2020.
[29]
X. Zheng and Z. Cai, Privacy-preserved data sharing towards multiple parties in industrial IoTs. IEEE J. Sel. Areas Commun., vol. 38, no. 5, pp. 968–979, 2020.
[30]
Z. Cai, Z. Xiong, H. Xu, P. Wang, W. Li, and Y. Pan, Generative adversarial networks: A survey toward private and secure applications, ACM Comput. Surv., vol. 54, no. 6, p. 132, 2022.
[31]
K. Simanyan and A. Zisserman, Very deep convolutional network for large-scale image recognition, arXiv preprint arXiv: 1409.1556, 2015.
[32]
J. Wang, Z. Cai, and J. Yu, Achieving personalized k-anonymity-based content privacy for autonomous vehicles in CPS, IEEE Trans. Industr. Inform., vol. 16, no. 6, pp. 4242–4251, 2020.
[33]
X. Zheng, Z. Cai, J. Li, and H. Gao, Location-privacy-aware review publication mechanism for local business service systems, in Proc. 2017 IEEE Conf. on Computer Communications, Atlanta, GA, USA, 2017, pp. 1–9.
[34]
X. Zheng, Z. Cai, J. Yu, C. Wang, and Y. Li, Follow but no track: Privacy preserved profile publishing in cyber-physical social systems, IEEE Internet Things J., vol. 4, no. 6, pp. 1868–1878, 2017.
[35]
X. Zheng, Z. Cai, and Y. Li, Data linkage in smart internet of things systems: A consideration from a privacy perspective, IEEE Commun. Mag., vol. 56, no. 9, pp. 55–61, 2018.
[36]
B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., The multimodal brain tumor image segmentation benchmark (BRATS), IEEE Trans. Med. Imaging, vol. 34, no. 10, pp. 1993–2024, 2015.
[37]
F. Isersee, J. Petersen, A. Klein, D. Zimmerer, P. F. Jaeger, S, Kohl, J. Wasserthal, G. Koehler, T. Norajrtra, S. Wirkert, et al., nnU-Net: Self-adapting framework for U-Net-based modical image segmantation, arXiv preprint arXiv: 1809.10486, 2018.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 17 May 2022
Accepted: 21 June 2022
Published: 06 January 2023
Issue date: August 2023

Copyright

© The author(s) 2023.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 62072135), Natural Science Foundation of Ningxia Hui Autonomous Region (No. 2022AAC03346), and Fundamental Research Funds for the Central Universities (No. 3072020CF0602).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return