Journal Home > Volume 27 , Issue 5

The performances of semisupervised clustering for unlabeled data are often superior to those of unsupervised learning, which indicates that semantic information attached to clusters can significantly improve feature representation capability. In a graph convolutional network (GCN), each node contains information about itself and its neighbors that is beneficial to common and unique features among samples. Combining these findings, we propose a deep clustering method based on GCN and semantic feature guidance (GFDC) in which a deep convolutional network is used as a feature generator, and a GCN with a softmax layer performs clustering assignment. First, the diversity and amount of input information are enhanced to generate highly useful representations for downstream tasks. Subsequently, the topological graph is constructed to express the spatial relationship of features. For a pair of datasets, feature correspondence constraints are used to regularize clustering loss, and clustering outputs are iteratively optimized. Three external evaluation indicators, i.e., clustering accuracy, normalized mutual information, and the adjusted Rand index, and an internal indicator, i.e., the Davidson-Bouldin index (DBI), are employed to evaluate clustering performances. Experimental results on eight public datasets show that the GFDC algorithm is significantly better than the majority of competitive clustering methods, i.e., its clustering accuracy is 20% higher than the best clustering method on the United States Postal Service dataset. The GFDC algorithm also has the highest accuracy on the smaller Amazon and Caltech datasets. Moreover, DBI indicates the dispersion of cluster distribution and compactness within the cluster.


menu
Abstract
Full text
Outline
About this article

Graph Convolutional Network Combined with Semantic Feature Guidance for Deep Clustering

Show Author's information Junfen ChenJie HanXiangjie Meng( )Yan Li( )Haifeng Li
Key Laboratory of Machine Learning and Computational Intelligence of Hebei Province, the College of Mathematics and Information Science, Hebei University, Baoding 071002, China
School of Applied Mathematics, Beijing Normal University Zhuhai, Zhuhai 519087, China
Department of Computer Teaching, Hebei University, Baoding 071002, China

Abstract

The performances of semisupervised clustering for unlabeled data are often superior to those of unsupervised learning, which indicates that semantic information attached to clusters can significantly improve feature representation capability. In a graph convolutional network (GCN), each node contains information about itself and its neighbors that is beneficial to common and unique features among samples. Combining these findings, we propose a deep clustering method based on GCN and semantic feature guidance (GFDC) in which a deep convolutional network is used as a feature generator, and a GCN with a softmax layer performs clustering assignment. First, the diversity and amount of input information are enhanced to generate highly useful representations for downstream tasks. Subsequently, the topological graph is constructed to express the spatial relationship of features. For a pair of datasets, feature correspondence constraints are used to regularize clustering loss, and clustering outputs are iteratively optimized. Three external evaluation indicators, i.e., clustering accuracy, normalized mutual information, and the adjusted Rand index, and an internal indicator, i.e., the Davidson-Bouldin index (DBI), are employed to evaluate clustering performances. Experimental results on eight public datasets show that the GFDC algorithm is significantly better than the majority of competitive clustering methods, i.e., its clustering accuracy is 20% higher than the best clustering method on the United States Postal Service dataset. The GFDC algorithm also has the highest accuracy on the smaller Amazon and Caltech datasets. Moreover, DBI indicates the dispersion of cluster distribution and compactness within the cluster.

Keywords: feature correspondence, evaluation indicator, graph convolutional network, self-supervised clustering, semantic feature guidance, confusion matrix

References(43)

[1]
H. J. Zhang and I. Davidson, Deep descriptive clustering, arXiv preprint arXiv: 2105.11549, 2021.
[2]
B. T. Li, D. C. Pi, Y. X. Lin, and L. Cui, DNC: A deep neural network-based clustering-oriented network embedding algorithm, J. Netw. Comput. Appl., vol. 173, p. 102854, 2021.
[3]
J. J. Gao, F. Z. Li, B. J. Wang, and H. L. Liang, Unsupervised nonlinear adaptive manifold learning for global and local information, Tsinghua Science and Technology, vol. 26. no. 2, pp. 163–171, 2021.
[4]
P. H. Huang, Y. Huang, W. Wang, and L. Wang, Deep embedding network for clustering, presented at the 22nd Int. Conf. Pattern Recognition, Stockholm, Sweden, 2014, pp. 1532–1537.
DOI
[5]
C. Niu and G. Wang, SPICE: Semantic pseudo-labeling for image clustering, arXiv preprint arXiv: 2103.09382, 2021.
[6]
J. J. Zhao, D. H. Lu, K. Ma, Z. Zhang, and Y. F. Zheng, Deep image clustering with category-style representation, presented at the 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 54–70.
DOI
[7]
B. L. Zhang and J. B. Qian, Autoencoder-based unsupervised clustering and hashing, Appl. Intell., vol. 51, no. 1, pp. 493–505, 2021.
[8]
X. Ji, A. Vedaldi, and J. Henriques, Invariant information clustering for unsupervised image classification and segmentation, presented at the 2019 IEEE/CVF Int. Conf. Computer Vision, Seoul, Republic of Korea, 2019, pp. 9864–9873.
DOI
[9]
K. Sohn, D. Berthelot, C. L. Li, Z. Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, FixMatch: Simplifying semi-supervised learning with consistency and confidence, arXiv preprint arXiv: 2001.07685, 2020.
[10]
Z. H. Wu, S. R Pan, F. W. Chen, G. D. Long, C. Q. Zhang, and P. S. Yu, A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 1, pp. 4–24, 2021.
[11]
T. N. Kipf and M. Welling, Semi-supervised classification with graph convolutional networks, in Proc. 5th Int. Conf. Learning Representations, Toulon, France, 2017.
[12]
J. MacQueen, Some methods for classification and analysis of multivariate observations, in Proc. 5th Berkeley Symp. Mathematical Statistics and Probability, Berkeley, CA, USA, 1967, pp. 281–297.
[13]
J. Y. Xie, R. Girshick, and A. Farhadi, Unsupervised deep embedding for clustering analysis, in Proc. 33rd Int. Conf. Machine Learning, New York, NY, USA, 2016, pp. 478–487.
[14]
M. Caron, P. Bojanowski, A. Joulin, and M. Douze, Deep clustering for unsupervised learning of visual features, presented at the 15th European Conf. Computer Vision, Munich, Germany, 2018, pp. 139–156.
DOI
[15]
Y. F. Li, P. Hu, Z. T. Liu, D. Z. Peng, J. T. Zhou, and P. Xi, Contrastive clustering, presented at the 35th AAAI Conf. Artificial Intelligence, New York, USA, 2021, pp. 8547–8555.
[16]
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. A. Manzagol, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, J. Mach. Learn. Res., vol. 11, no. 12, pp. 3371–3408, 2010.
[17]
L. Zhang, J. C. Liu, F. X Shang, G. Li, J. M. Zhao, and Y. Q. Zhang, Robust segmentation method for noisy images based on an unsupervised denosing filter, Tsinghua Science and Technology, vol. 26, no. 5, pp. 736–748, 2021.
[18]
K. G. Dizaji, A. Herandi, C. Deng, W. D. Cai, and H. Huang, Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization, presented at the 2017 IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 5747–5756.
DOI
[19]
F. F. Li, H. Qiao, and B. Zhang, Discriminatively boosted image clustering with fully convolutional auto-encoders, Pattern Recogn., vol. 83, pp. 161–173, 2018.
[20]
N. Dilokthanakul, P. A. M. Mediano, M. Garnelo, M. C. H. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan, Deep unsupervised clustering with Gaussian mixture variational autoencoders, presented at 2017 Int. Conf. Learning Representations, Toulon, France, 2017.
[21]
P. Zhou, Y. Q. Hou, and J. S. Feng, Deep adversarial subspace clustering, presented at the 2018 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 1596–1604.
DOI
[22]
X. F. Guo, L. Gao, X. W. Liu, and J. P. Yin, Improved deep embedded clustering with local structure preservation, in Proc. 26th Int. Joint Conf. Artificial Intelligence, Melbourne, Australia, 2017, pp. 1753–1759.
DOI
[23]
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, A simple framework for contrastive learning of visual representations, in Proc. 37th Int. Conf. Machine Learning, 2020, pp. 1597–1607.
[24]
J. L. Wu, K. Y. Long, F. Wang, C. Qian, C. Li, Z. C. Lin, and H. B. Zha, Deep comprehensive correlation mining for image clustering, presented at the 2019 IEEE/CVF Int. Conf. Computer Vision, Seoul, Republic of Korea, 2019, pp. 8149–8158.
DOI
[25]
J. B. Huang, S. G. Gong, and X. T. Zhu, Deep semantic clustering by partition confidence maximisation, presented at 2020 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 8846–8855.
DOI
[26]
C. Niu, J. Zhang, G. Wang, and J. M. Liang, GATcluster: Self-supervised Gaussian-attention network for image clustering, presented at the 16th European Conf. Computer Vision, Glasgow, UK, 2020, pp. 735–751.
DOI
[27]
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, The graph neural network model. IEEE Trans. Neural Netw., vol. 20, no. 1, pp. 61–80, 2009.
[28]
J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun, Spectral networks and locally connected networks on graph, in Proc. 2nd Int. Conf. Learning Representations, Banff, Canada, 2014.
[29]
Q. M. Li, Z. C. Han, and X. M. Wu, Deeper insights into graph convolutional networks for semi-supervised learning, in Proc. 32nd AAAI Conf. Artificial Intelligence, New Orleans, LA, USA, 2018, pp. 3538–3545.
[30]
Z. D Wang, L. Zheng, Y. L. Li, and S. J. Wang, Linkage based face clustering via graph convolution network, presented at the 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 1117–1125.
DOI
[31]
Y. Bengio, A. Courville, and P. Vincent, Representation learning: A review and new perspectives, IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798–1828, 2013.
[32]
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[33]
J. J. Hull, A database for handwritten text recognition research, IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, no. 5, pp. 550–554, 1994.
[34]
A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, Handbook of Systemic Autoimmune Diseases, vol. 1, no. 4, pp. 1–58, 2009.
[35]
A. Coates, A. Ng, and H. Lee, An analysis of single-layer networks in unsupervised feature learning, in Proc. 14th Int. Conf. Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 2011, pp. 215–223.
[36]
T. Li and C. Ding, The relationships among various nonnegative matrix factorization methods for clustering, presented at the 6th Int. Conf. Data Mining, Hong Kong, China, 2006, pp. 362–371.
DOI
[37]
A. Strehl and J. Ghosh, Cluster ensembles—A knowledge reuse framework for combining multiple partitions, J. Mach. Learn. Res., vol. 3, pp. 583–617, 2003.
[38]
L. Hubert and P. Arabie, Comparing partitions. J. Classif., vol. 2, no. 1, pp. 193–218, 1985.
[39]
A. Fahad, N. Alshatri, Z. Tari, A. Alamri, I. Khalil, A. Y. Zomaya, S. Foufou, and A. Bouras, A survey of clustering algorithms for big data: Taxonomy and empirical analysis, IEEE Trans. Emerg. Top. Comput., vol. 2, no. 3, pp. 267–279, 2014.
[40]
D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, in Proc. 3rd Int. Conf. Learning Representations, San Diego, CA, USA, 2015, pp. 1–15.
DOI
[41]
J. L. Chang, L. F. Wang, G. F. Meng, S. M. Xiang, and C. H. Pan, Deep adaptive image clustering, presented at 2017 IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 5880–5888.
DOI
[42]
H. S. Zhong, J. L. Wu, C. Chen, J. Q. Huang, M. H. Deng, L. Q. Nie, Z. C. Lin, and X. S. Hua, Graph contrastive clustering. arXiv preprint arXiv: 2104.01429, 2021.
[43]
L. van der Maaten and G. Hinton, Visualizing data using t-SNE, J. Mach. Learn. Res., vol. 9, no. 86, pp. 2579–2605, 2008.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 26 July 2021
Revised: 23 August 2021
Accepted: 25 August 2021
Published: 17 March 2022
Issue date: October 2022

Copyright

© The author(s) 2022.

Acknowledgements

This work was supported by the Hebei Province Introduction of Studying Abroad Talent Funded Project (No. C20200302), the Opening Fund of Hebei Key Laboratory of Machine Learning and Computational Intelligence (Nos. 2019-2021-A and ZZ201909-202109-1), the National Natural Science Foundation of China (No. 61976141), and the Social Science Foundation of Hebei Province (No. HB20TQ005).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return