Journal Home > Volume 8 , Issue 1
International Journal of Crowd Science 2024, 8(1): 56-63 https://doi.org/10.26599/IJCS.2023.9100031
Open Access | Issue | Published: 27 February 2024

Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text

Show Author's Information Zhichao Wang1,2,  Xiangwei Zheng1,2( ),  Jinsong Zhang1,2,  Mingzhe Zhang1,2( )
School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China
State Key Laboratory of High-end Server & Storage Technology, Jinan 250013, China
Keywords:
text classification, gastroscopy diagnostic text, Bidirectional Encoder Representations from Transformers (BERT)
Cite this article:
Wang Z, Zheng X, Zhang J, et al. Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text. International Journal of Crowd Science, 2024, 8(1): 56-63. https://doi.org/10.26599/IJCS.2023.9100031
Download citation
EndNote(RIS)
BibTeX

127

Views

16

Downloads

Citations

0

Crossref

N/A

WoS

0

Scopus

N/A

CSCD

Abstract Full text About this article

During a hospital visit, a significant volume of Gastroscopy Diagnostic Text (GDT) data are produced, representing the unstructured gastric medical records of patients undergoing gastroscopy. As such, GDTs play a crucial role in evaluating the patient’s health, shaping treatment plans, and scheduling follow-up visits. However, given the free-text nature of GDTs, which lack a formal structure, physicians often find it challenging to extract meaningful insights from them. Furthermore, while deep learning has made significant strides in the medical domain, to our knowledge, there are not any readily available text-based pre-trained models tailored for GDT classification and analysis. To address this gap, we introduce a Bidirectional Encoder Representations from Transformers (BERT) based three-branch classification network tailored for GDTs. We leverage the robust representation capabilities of the BERT pre-trained model to deeply encode the texts. A unique three-branch decoder structure is employed to pinpoint lesion sites and determine cancer stages. Experimental outcomes validate the efficacy of our approach in GDT classification, with a precision of 0.993 and a recall of 0.784 in the early cancer category. In pinpointing cancer lesion sites, the weighted F1 score achieved was 0.849.


menu
Abstract
Full text
Outline
About this article

Three-Branch BERT-Based Text Classification Network for Gastroscopy Diagnosis Text

Show Author's information Zhichao Wang1,2, Xiangwei Zheng1,2( ), Jinsong Zhang1,2, Mingzhe Zhang1,2( )
School of Information Science and Engineering, Shandong Normal University, Jinan 250358, China
State Key Laboratory of High-end Server & Storage Technology, Jinan 250013, China

Abstract

During a hospital visit, a significant volume of Gastroscopy Diagnostic Text (GDT) data are produced, representing the unstructured gastric medical records of patients undergoing gastroscopy. As such, GDTs play a crucial role in evaluating the patient’s health, shaping treatment plans, and scheduling follow-up visits. However, given the free-text nature of GDTs, which lack a formal structure, physicians often find it challenging to extract meaningful insights from them. Furthermore, while deep learning has made significant strides in the medical domain, to our knowledge, there are not any readily available text-based pre-trained models tailored for GDT classification and analysis. To address this gap, we introduce a Bidirectional Encoder Representations from Transformers (BERT) based three-branch classification network tailored for GDTs. We leverage the robust representation capabilities of the BERT pre-trained model to deeply encode the texts. A unique three-branch decoder structure is employed to pinpoint lesion sites and determine cancer stages. Experimental outcomes validate the efficacy of our approach in GDT classification, with a precision of 0.993 and a recall of 0.784 in the early cancer category. In pinpointing cancer lesion sites, the weighted F1 score achieved was 0.849.

Keywords: text classification, gastroscopy diagnostic text, Bidirectional Encoder Representations from Transformers (BERT)

References(24)

[1]

Y. Cui, G. Cheng, G. Tian, S. He, and Y. Yan, Secular trends in the mortality of gastrointestinal cancers across China, Japan, the US, and India: An age-period-cohort, joinpoint analyses, and Holt forecasts, Front. Public Health, vol. 10, p. 925011, 2022.
DOI Google Scholar
[2]

C. Herrera-Pariente, S. Montori, J. Llach, A. Bofill, E. Albeniz, and L. Moreira, Biomarkers for gastric cancer screening and early diagnosis, Biomedicines, vol. 9, no. 10, p. 1448, 2021.
DOI Google Scholar
[3]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, in Proc. 31st Int. Conf. Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 6000–6010.
[4]
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810.04805, 2019.
[5]
W. Zaremba, I. Sutskever, and O. Vinyals, Recurrent neural network regularization, arXiv preprint arXiv: 1409.2329, 2014.
[6]
A. Graves, Long short-term memory, in Supervised Sequence Labelling with Recurrent Neural Networks, A. Graves, ed. Berlin, Germany: Springer, 2012, pp. 37–45.
DOI
[7]

A. Onan and M. A. Toçoğlu, A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification, IEEE Access, vol. 9, pp. 7701–7722, 2021.
DOI Google Scholar
[8]
W. Zhou, J. Du, and X. Ren, Improving BERT fine-tuning with embedding normalization, arXiv preprint arXiv: 1911.03918, 2019.
[9]

L. Jiang, C. Li, S. Wang, and L. Zhang, Deep feature weighting for naive Bayes and its application to text classification, Eng. Appl. Artif. Intell., vol. 52, pp. 26–39, 2016.
DOI Google Scholar
[10]
T. Joachims, Making large scale SVM learning practical, https://api.semanticscholar.org/CorpusID:61116019, 1998.
[11]

J. R. Quinlan, Induction of decision trees, Mach. Learn., vol. 1, no. 1, pp. 81–106, 1986.
DOI Google Scholar
[12]

L. Breiman, Random forests, Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
DOI Google Scholar
[13]
Y. Yang and X. Liu, A re-examination of text categorization methods, in Proc. 22nd Annual Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Berkeley, CA, USA, 1999, pp. 42–49.
DOI
[14]

S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, Deep learning: Based text classification, ACM Comput. Surv., vol. 54, no. 3, pp. 1–40, 2022.
DOI Google Scholar
[15]

M. Hughes, I. Li, S. Kotoulas, and T. Suzumura, Medical text classification using convolutional neural networks, Stud. Health Technol. Inform., vol. 235, pp. 246–250, 2017.
Google Scholar
[16]

A. Onan, Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification, J. King Saud Univ. Comput. Inf. Sci., vol. 34, no. 5, pp. 2098–2117, 2022.
DOI Google Scholar
[17]

A. Onan, S. Korukoğlu, and H. Bulut, Ensemble of keyword extraction methods and classifiers in text classification, Expert Syst. Appl., vol. 57, pp. 232–247, 2016.
DOI Google Scholar
[18]
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, arXiv preprint arXiv: 1907.11692, 2019.
[19]

A. Onan, Hierarchical graph-based text classification framework with contextual node embedding and BERT-based dynamic fusion, J. King Saud Univ. Comput. Inf. Sci., vol. 35, no. 7, p. 101610, 2023.
DOI Google Scholar
[20]
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, Improving language understanding by generative pre-training, https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf, 2018.
[21]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, in Proc. 34th Int. Conf. Neural Information Processing Systems, Vancouver, Canada, 2020, pp. 1877–1901.
[22]

S. Ding, S. Hu, X. Li, Y. Zhang, and D. D. Wu, Leveraging multimodal semantic fusion for gastric cancer screening via hierarchical attention mechanism, IEEE Trans. Syst. Man Cybern. Syst., vol. 52, no. 7, pp. 4286–4299, 2022.
DOI Google Scholar
[23]
Y. Liu, Fine-tune BERT for extractive summarization, arXiv preprint arXiv: 1903.10318, 2019.
[24]
N. Ketkar, Introduction to keras, in Deep Learning with Python, N. Ketkar, ed. Berkeley, CA, USA: Apress, 2017, pp. 97–111.
DOI
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 08 September 2023
Revised: 02 November 2023
Accepted: 16 November 2023
Published: 27 February 2024
Issue date: March 2024

Copyright

© The author(s) 2024.

Acknowledgements

Acknowledgment

This work was supported by the Key R&D Program of Shandong Province, China (Major Scientific and Technological Innovation Project) (No. 2021CXGC010506), Key R&D Program of Shandong Province, China (No. 2021SFGC0104), Natural Science Foundation of Shandong Province China (Nos. ZR2020LZH008, ZR2021MF118, and ZR2022LZH003) and National Natural Science Foundation of China (No. 62101311).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return