Journal Home > Volume 28 , Issue 1
Tsinghua Science and Technology 2023, 28(1): 59-68 https://doi.org/10.26599/TST.2021.9010079
Open Access | Issue | Published: 21 July 2022

Supervised Contrastive Learning with Term Weighting for Improving Chinese Text Classification

Show Author's Information Jiabao Guo1,  Bo Zhao1( ),  Hui Liu1,  Yifan Liu1,  Qian Zhong1
School of Cyber Science and Engineering, Wuhan University, Wuhan 430000, China
Keywords:
Chinese text classification, Supervised Contrastive Learning (SCL), Term Weighting (TW), Temporal Convolution Network (TCN)
Cite this article:
Guo J, Zhao B, Liu H, et al. Supervised Contrastive Learning with Term Weighting for Improving Chinese Text Classification. Tsinghua Science and Technology, 2023, 28(1): 59-68. https://doi.org/10.26599/TST.2021.9010079
Download citation
EndNote(RIS)
BibTeX

587

Views

54

Downloads

Citations

4

Crossref

4

WoS

4

Scopus

0

CSCD

Abstract Full text About this article

With the rapid growth of information retrieval technology, Chinese text classification, which is the basis of information content security, has become a widely discussed topic. In view of the huge difference compared with English, Chinese text task is more complex in semantic information representations. However, most existing Chinese text classification approaches typically regard feature representation and feature selection as the key points, but fail to take into account the learning strategy that adapts to the task. Besides, these approaches compress the Chinese word into a representation vector, without considering the distribution of the term among the categories of interest. In order to improve the effect of Chinese text classification, a unified method, called Supervised Contrastive Learning with Term Weighting (SCL-TW), is proposed in this paper. Supervised contrastive learning makes full use of a large amount of unlabeled data to improve model stability. In SCL-TW, we calculate the score of term weighting to optimize the process of data augmentation of Chinese text. Subsequently, the transformed features are fed into a temporal convolution network to conduct feature representation. Experimental verifications are conducted on two Chinese benchmark datasets. The results demonstrate that SCL-TW outperforms other advanced Chinese text classification approaches by an amazing margin.


menu
Abstract
Full text
Outline
About this article

Supervised Contrastive Learning with Term Weighting for Improving Chinese Text Classification

Show Author's information Jiabao Guo1, Bo Zhao1( ), Hui Liu1, Yifan Liu1, Qian Zhong1
School of Cyber Science and Engineering, Wuhan University, Wuhan 430000, China

Abstract

With the rapid growth of information retrieval technology, Chinese text classification, which is the basis of information content security, has become a widely discussed topic. In view of the huge difference compared with English, Chinese text task is more complex in semantic information representations. However, most existing Chinese text classification approaches typically regard feature representation and feature selection as the key points, but fail to take into account the learning strategy that adapts to the task. Besides, these approaches compress the Chinese word into a representation vector, without considering the distribution of the term among the categories of interest. In order to improve the effect of Chinese text classification, a unified method, called Supervised Contrastive Learning with Term Weighting (SCL-TW), is proposed in this paper. Supervised contrastive learning makes full use of a large amount of unlabeled data to improve model stability. In SCL-TW, we calculate the score of term weighting to optimize the process of data augmentation of Chinese text. Subsequently, the transformed features are fed into a temporal convolution network to conduct feature representation. Experimental verifications are conducted on two Chinese benchmark datasets. The results demonstrate that SCL-TW outperforms other advanced Chinese text classification approaches by an amazing margin.

Keywords: Chinese text classification, Supervised Contrastive Learning (SCL), Term Weighting (TW), Temporal Convolution Network (TCN)

References(39)

[1]
T. A. Almeida, T. P. Silva, I. Santos, and J. M. Gómez Hidalgo, Text normalization and semantic indexing to enhance instant messaging and SMS spam filtering, Knowl. Based Syst., vol. 108, pp. 25–32, 2016.
DOI Google Scholar
[2]
A. Watanabe, R. Sasano, H. Takamura, and M. Okumura, Generating personalized snippets for web page recommender systems, in Proc. 2014 IEEE/WIC/ACM Int. Joint Conf. on Web Intelligence (WI) and Intelligent Agent Technologies, Warsaw, Poland, 2014, pp. 218–225.
DOI Google Scholar
[3]
X. H. Chen, Y. Zhang, J. Xu, C. X. Xing, and H. Chen, Deep learning based topic identification and categorization: Mining diabetes-related topics on Chinese health websites, in Proc. 21st Int. Conf. on Database Systems for Advanced Applications, Dallas, TX, USA, 2016, pp. 481–500.
DOI Google Scholar
[4]
W. Zhong, N. Yu, and C. Y. Ai, Applying big data based deep learning system to intrusion detection, Big Data Mining and Analytics, vol. 3, no. 3, pp. 181–195, 2020.
DOI Google Scholar
[5]
J. W. Tang, R. X. Li, K. P. Wang, X. W. Gu, and Z. Y. Xu, A novel hybrid method to analyze security vulnerabilities in Android applications, Tsinghua Science and Technology, vol. 25, no. 5, pp. 589–603, 2020.
DOI Google Scholar
[6]
B. Zhao, P. Y. Zhao, and P. R. Fan, ePUF: A lightweight double identity verification in IoT, Tsinghua Science and Technology, vol. 25, no. 5, pp. 625–635, 2020.
DOI Google Scholar
[7]
L. H. Lee, D. Isa, W. O. Choo, and W. Y. Chue, High relevance keyword extraction facility for Bayesian text classification on different domains of varying characteristic, Expert Syst. Appl., vol. 39, no. 1, pp. 1147–1155, 2012.
DOI Google Scholar
[8]
W. X. Hu, Z. Q. Gu, Y. S. Xie, L. Wang, and K. K. Tang, Chinese text classification based on neural networks and word2vec, in Proc. 2019 IEEE Fourth Int. Conf. on Data Science in Cyberspace, Hangzhou, China, 2019, pp. 284–291.
DOI Google Scholar
[9]
V. N. Phu, V. T. N. Tran, V. T. N. Chau, N. D. Dat, and K. L. D. Duy, A decision tree using ID3 algorithm for English semantic analysis, Int. J. Speech Technol., vol. 20, no. 3, pp. 593–613, 2017.
DOI Google Scholar
[10]
H. Q. Tao, S. W. Tong, H. K. Zhao, T. Xu, B. B. Jin, and Q. Liu, A radical-aware attention-based model for Chinese text classification, in Proc. Thirty-Third AAAI Conf. on Artificial Intelligence, Honolulu, HI, USA, 2019, pp. 5125–5132.
DOI Google Scholar
[11]
X. X. Chen, L. Xu, Z. Y. Liu, M. S. Sun, and H. B. Luan, Joint learning of character and word embeddings, in Proc. 24th Int. Conf. on Artificial Intelligence, Buenos, Argentina, 2015, pp. 1236–1242.
Google Scholar
[12]
J. L. Li, G. Q. Cheng, Q. P. Yin, J. C. Huang, W. C. Chen, and J. M. Du, A practical method for the expert academic personas classification based on text classifier, Evol. Intell., .
DOI Google Scholar
[13]
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, Distributed representations of words and phrases and their compositionality, in Proc. 26th Int. Conf. on Neural Information Processing Systems, Lake Tahoe, UT, USA, 2013, pp. 3111–3119.
Google Scholar
[14]
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2019, pp. 4171–4186.
Google Scholar
[15]
L. Zhang, and C. C. Chen, Sentiment classification with convolutional neural networks: An experimental study on a large-scale Chinese conversation corpus, in Proc. 2016 12th Int. Conf. on Computational Intelligence and Security, Wuxi, China, 2016, pp. 165–169.
DOI Google Scholar
[16]
Y. Wang, S. Feng, D. L. Wang, Y. F. Zhang, and G. Yu, Context-aware Chinese microblog sentiment classification with bidirectional LSTM, in Proc. 18th Asia-Pacific Web Conf., Suzhou, China, 2016, pp. 594–606.
DOI Google Scholar
[17]
J. C. Du, L. Gui, R. F. Xu, and Y. L. He, A convolutional attention model for text classification, in Proc. 6th Natural Language Processing and Chinese Computing, Dalian, China, 2017, pp. 183–195.
DOI Google Scholar
[18]
Y. J. Zhou, B. Xu, J. M. Xu, L. Yang, C. L. Li, and B. Xu, Compositional recurrent neural networks for Chinese short text classification, in Proc. 2016 IEEE/WIC/ACM Int. Conf. on Web Intelligence, Omaha, NE, USA, 2017, pp. 137–144.
DOI Google Scholar
[19]
K. S. Jones, A statistical interpretation of term specificity and its application in retrieval, J. Doc., vol. 60, no. 5, pp. 493–502, 2004.
DOI Google Scholar
[20]
F. Debole and F. Sebastiani, Supervised term weighting for automated text categorization, in Proc. 2003 ACM Symp. on Applied Computing, Melbourne, Australia, 2003, pp. 784–788.
DOI Google Scholar
[21]
H. B. Wu, X. D. Gu, and Y. W. Gu, Balancing between over-weighting and under-weighting in supervised term weighting, Inf. Process. Manage., vol. 53, no. 3, pp. 547–557, 2017.
DOI Google Scholar
[22]
H. J. Escalante, M. A. García-Limón, A. Morales-Reyes, M. Graff, M. Montes-y-Gómez, E. F. Morales, and J. Martínez-Carranza, Term-weighting learning via genetic programming for text classification, Knowl. Based Syst., vol. 83, pp. 176–189, 2015.
DOI Google Scholar
[23]
G. Domeniconi, G. Moro, R. Pasolini, and C. Sartori, A study on term weighting for text categorization: A novel supervised variant of tf.idf, in Proc. 4th Int. Conf. on Data Management Technologies and Applications, Colmar, France, 2015, pp. 26–37.
DOI Google Scholar
[24]
C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, Temporal convolutional networks for action segmentation and detection, in Proc. 2017 IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 1003–1012.
DOI Google Scholar
[25]
A. Ghasemi, C. Parekh, and P. Guinand, Spectrum sensing for modulated radio signals using deep temporal convolutional networks, in Proc. 2019 IEEE Wireless Communications and Networking Conf. Workshop, Marrakech, Morocco, 2019, pp. 1–5.
DOI Google Scholar
[26]
W. Jiang, Y. Wang, and Y. Tang, A sequence-to-sequence transformer premised temporal convolutional network for Chinese word segmentation, in Proc. 10th Int. Symp. on Parallel Architectures, Algorithms and Programming, Guangzhou, China, 2019, pp. 541–552.
DOI Google Scholar
[27]
J. X. You, Y. C. Wang, A. Pal, P. Eksombatchai, C. Rosenburg, and J. Leskovec, Hierarchical temporal convolutional networks for dynamic recommender systems, in Proc. World Wide Web Conf., San Francisco, CA, USA, 2019, pp. 2236–2246.
DOI Google Scholar
[28]
L. Kuang, C. B. Hua, J. G. Wu, Y. Y. Yin, and H. H. Gao, Traffic volume prediction based on multi-sources GPS trajectory data by temporal convolutional network, Mobile Netw. Appl., vol. 25, no. 4, pp. 1405–1417, 2020.
DOI Google Scholar
[29]
S. J. Bai, J. Z. Kolter, and V. Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint arXiv: 1803.01271, 2018.
Google Scholar
[30]
X. H. Zhang, Y. X. Zou, and W. Shi, Dilated convolution neural network with leakyrelu for environmental sound classification, in Proc. 2017 22nd Int. Conf. on Digital Signal Processing, London, UK, 2017, pp. 1–5.
DOI Google Scholar
[31]
Z. P. Guo, Y. Zhao, Y. B. Zheng, X. C. Si, Z. Y. Liu, and M. S. Sun, Thuctc: An efficient Chinese text classifier, (in Chinese), http://github.com/diuzi/THUCTC, 2016.
[32]
S. I. Wang and C. D. Manning, Baselines and bigrams: Simple, good sentiment and topic classification, in Proc. 50th Ann. Meeting of the Association for Computational Linguistics, Jeju Island, Republic of Korea, 2012, pp. 90–94.
Google Scholar
[33]
Y. Kim, Convolutional neural networks for sentence classification, in Proc. 2014 Conf. on Empirical Methods in Natural Language Processing, Doha, Qatar, 2014, pp. 1746–1751.
DOI Google Scholar
[34]
Z. C. Yang, D. Y. Yang, C. Dyer, X. D. He, A. J. Smola, and E. H. Hovy, Hierarchical attention networks for document classification, in Proc. 2016 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA, 2017, pp. 1480–1489.
DOI Google Scholar
[35]
S. Gao, A. Ramanathan, and G. D. Tourassi, Hierarchical convolutional attention networks for text classification, in Proc. Third Workshop on Representation Learning for NLP, Melbourne, Australia, 2018, pp. 11–23.
DOI Google Scholar
[36]
Y. Zhang, J. E. Meng, R. Venkatesan, N. Wang, and M. Pratama, Sentiment classification using comprehensive attention recurrent models, in Proc. 2016 Int. Joint Conf. on Neural Networks, Vancouver, Canada, 2016, pp. 1562–1569.
DOI Google Scholar
[37]
Y. J. Zhou, J. M. Xu, J. Cao, B. Xu, C. L. Li, and B. Xu, Hybrid attention networks for Chinese short text classification, Comput. Sist., vol. 21, no. 4, pp. 759–769, 2017.
DOI Google Scholar
[38]
X. Qiao, C. Peng, Z. Liu, and Y. F. Hu, Word-character attention model for Chinese text classification, Int. J. Mach. Learn. Cybern., vol. 10, no. 12, pp. 3521–3537, 2019.
DOI Google Scholar
[39]
K. Q. Zhang, S. P. Wang, B. B. Li, F. Mei, and J. Y. Zhang, Hierarchical convolutional attention networks using joint Chinese word embedding for text classification, in Proc. 16th Pacific Rim Int. Conf. on Artificial Intelligence, Cuvu, Fiji, 2019, pp. 234–246.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 10 October 2021
Accepted: 25 October 2021
Published: 21 July 2022
Issue date: February 2023

Copyright

© The author(s) 2023.

Acknowledgements

The work was supported by the National Natural Science Foundation of China (No. U1936122) and Primary Research & Developement Plan of Hubei Province (Nos. 2020BAB101 and 2020BAA003).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return