AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Tsinghua Science and Technology Article
PDF (8.3 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Causality-Based Contrastive Incremental Learning Framework for Domain Generalization

School of Computer Science and Technology, Beijing Institute with of Technology, Beijing 100081, China
Beijing Institute of Control Engineering, Beijing 100190, China
Show Author Information

Abstract

Learning domain-invariant feature representations is critical to alleviate the distribution differences between training and testing domains. The existing mainstream domain generalization approaches primarily pursue to align the across-domain distributions to extract the transferable feature representations. However, these representations may be insufficient and unstable. Moreover, these networks may also undergo catastrophic forgetting because the previous learned knowledge is replaced by the new learned knowledge. To cope with these issues, we propose a novel causality-based contrastive incremental learning model for domain generalization, which mainly includes three components: (1) intra-domain causal factorization, (2) inter-domain Mahalanobis similarity metric, and (3) contrastive knowledge distillation. The model extracts intra and inter domain-invariant knowledge to improve model generalization. Specifically, we first introduce a causal factorization to extract intra-domain invariant knowledge. Then, we design a Mahalanobis similarity metric to extract common inter-domain invariant knowledge. Finally, we propose a contrastive knowledge distillation with exponential moving average to distill model parameters in a smooth way to preserve the previous learned knowledge and mitigate model forgetting. Extensive experiments on several domain generalization benchmarks prove that our model achieves the state-of-the-art results, which sufficiently show the effectiveness of our model.

Keywords

domain generalizationcausal factorizationmahalanobis similarityincremental learningdomain-invariant feature

References

[1]

M. E. Mercha and H. Benbrahim, Machine learning and deep learning for sentiment analysis across languages: A survey, Neurocomputing, vol. 531, pp. 195–216, 2023.
Crossref Google Scholar
[2]

P. J. G. Lisboa, S. Saralajew, A. Vellido, R. Fernández-Domenech, and T. Villmann, The coming of age of interpretable and explainable machine learning models, Neurocomputing, vol. 535, pp. 25–39, 2023.
Crossref Google Scholar
[3]

Y. Gu, B. Li, and Q. Meng, Hybrid interpretable predictive machine learning model for air pollution prediction, Neurocomputing, vol. 468, pp. 123–136, 2022.
Crossref Google Scholar
[4]

C. Zhang, Q. Zhao, and Y. Wang, Transferable attention networks for adversarial domain adaptation, Inf. Sci., vol. 539, pp. 422–433, 2020.
Crossref Google Scholar
[5]

Z. Wang, Y. Zhang, X. Xu, Z. Fu, H. Yang, and W. Du, Federated probability memory recall for federated continual learning, Inf. Sci., vol. 629, pp. 551–565, 2023.
Crossref Google Scholar
[6]
L. Xiang, X. Jin, G. Ding, J. Han, and L. Li, Incremental few-shot learning for pedestrian attribute recognition, in Proc. Twenty-Eighth Int. Joint Conf. Artificial Intelligence, Macao, China, 2019, pp. 3912–3918.
Crossref
[7]

M. K. Singh, S. Dhople, F. Dörfler, and G. B. Giannakis, Time-domain generalization of Kron reduction, IEEE Control Syst. Lett., vol. 7, pp. 259–264, 2023.
Crossref Google Scholar
[8]

C. X. Tian, H. Li, X. Xie, Y. Liu, and S. Wang, Neuron coverage-guided domain generalization, IEEE Trans. Patt. Anal. Mach. Intellig., vol. 45, no. 1, pp. 1302–1311, 2023.
Crossref Google Scholar
[9]
M. Wang, J. Yuan, Q. Qian, Z. Wang, and H. Li, Semantic data augmentation based distance metric learning for domain generalization, in Proc. 30 th ACM Int. Conf. Multimedia, Lisboa, Portugal, 2022, pp. 3214–3223.
Crossref
[10]
F. C. Borlino, A. D’Innocente, and T. Tommasi, Domain generalization vs. data augmentation: An unbiased perspective, in Proc. European Conf. Computer Vision, Glasgow, UK, 2020, pp. 726–730.
Crossref
[11]
Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, Open domain generalization with domain-augmented meta-learning, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 9619–9628.
Crossref
[12]

K. Chen, D. Zhuang, and J. Chang, Discriminative adversarial domain generalization with meta-learning based cross-domain validation, Neurocomputing, vol. 467, pp. 418–426, 2022.
Crossref Google Scholar
[13]

Y. F. Zhang, Z. Zhang, D. Li, Z. Jia, L. Wang, and T. Tan, Learning domain invariant representations for generalizable person re-identification, IEEE Trans. Image Process., vol. 32, pp. 509–523, 2023.
Crossref Google Scholar
[14]
Q. Xu, L. Yao, Z. Jiang, G. Jiang, W. Chu, W. Han, W. Zhang, C. Wang, and Y. Tai, DIRL: Domain-invariant representation learning for generalizable semantic segmentation, in Proc. Thirty-Sixth AAAI Conf. Artificial Intelligence, Palo Alto, CA, USA, 2022, pp. 2884–2892.
Crossref
[15]
F. Lv, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu, Causality inspired representation learning for domain generalization, in Proc. 2022 IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022, pp. 8036–8046.
Crossref
[16]

S. Li, Q. Zhao, C. Zhang, and Y. Zou, Deep discriminative causal domain generalization, Inf. Sci., vol. 645, p. 119335, 2023.
Crossref Google Scholar
[17]
H. Zhao, R. T. Des Combes, K. Zhang, and G. J. Gordon, On learning invariant representations for domain adaptation, in Proc. 36 th Int. Conf. Machine Learning, Long Beach, CA, USA, 2019, pp. 7523–7532.
[18]

C. Lin, Z. Zhu, S. Wang, Z. Shi, and Y. Zhao, CoI2A: Collaborative interdomain and intra-domain alignments for multisource domain adaptation, IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–8, 2023.
Crossref Google Scholar
[19]

Z. Li and D. Hoiem, Learning without forgetting, IEEE Trans. Patt. Anal. Mach. Intellig., vol. 40, no. 12, pp. 2935–2947, 2018.
Crossref Google Scholar
[20]
C. Simon, P. Koniusz, and M. Harandi, On learning the geodesic path for incremental learning, in Proc. 2021 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 1591–1600.
Crossref
[21]
Z. Huo, D. Hwang, K. C. Sim, S. Garg, A. Misra, N. Siddhartha, T. Strohman, and F. Beaufays, Incremental layer-wise self-supervised learning for efficient unsupervised speech domain adaptation on device, in Proc. Interspeech 2022, 23 rd Ann. Conf. Int. Speech Communication Association, Incheon, Korea, 2022, pp. 4845–4849.
Crossref
[22]
K. Thandiackal, L. Piccinelli, P. Pati, and O. Goksel, Multi-scale feature alignment for continual learning of unlabeled domains, arXiv preprint arXiv: 2302.01287, 2023.
[23]
C. Simon, M. Faraki, Y. H. Tsai, X. Yu, S. Schulter, Y. Suh, M. Harandi, and M. Chandraker, On generalizing beyond domains in cross-domain continual learning, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022, pp. 9255–9264.
Crossref
[24]

M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. Van De Weijer, Class-incremental learning: Survey and performance evaluation on image classification, IEEE Trans. Patt. Anal. Mach. Intellig., vol. 45, no. 5, pp. 5513–5533, 2023.
Crossref Google Scholar
[25]

Z. Qiu, L. Xu, Z. Wang, Q. Wu, F. Meng, and H. Li, ISM-Net: Mining incremental semantics for class incremental learning, Neurocomputing, vol. 523, pp. 130–143, 2023.
Crossref Google Scholar
[26]

W. Sun, Q. Li, J. Zhang, W. Wang, and Y. Geng, Class incremental learning based on identically distributed parallel one-class classifiers, Neurocomputing, vol. 556, p. 126579, 2023.
Crossref Google Scholar
[27]

M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, A continual learning survey: Defying forgetting in classification tasks, IEEE Trans. Patt. Anal. Mach. Intellig., vol. 44, no. 7, pp. 3366–3385, 2022.
Google Scholar
[28]

Y. Shi, D. Shi, Z. Qiao, Z. Wang, Y. Zhang, S. Yang, and C. Qiu, Multi-granularity knowledge distillation and prototype consistency regularization for class-incremental learning, Neural Netw., vol. 164, pp. 617–630, 2023.
Crossref Google Scholar
[29]
Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C. Y. Lee, X. Ren, G. Su, V. Perot, J. G. Dy, and T. Pfister, DualPrompt: Complementary prompting for rehearsal-free continual learning, in Proc. 17 th European Conf. Computer Vision, Tel Aviv, Israel, 2022, pp. 631–648.
Crossref
[30]
M. De Lange and T. Tuytelaars, Continual prototype evolution: Learning online from non-stationary data streams, in Proc. 2021 IEEE/CVF Int. Conf. Computer Vision, Montreal, Canada, 2021, pp. 8230–8239.
Crossref
[31]

S. Lee, K. Chang, and J. G. Baek, Incremental learning using generative-rehearsal strategy for fault detection and classification, Exp. Syst. Appl., vol. 184, p. 115477, 2021.
Crossref Google Scholar
[32]
S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, Learning a unified classifier incrementally via rebalancing, in Proc. 2019 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 831–839.
Crossref
[33]
E. Belouadah and A. Popescu, IL2M: Class incremental learning with dual memory, in Proc. 2019 IEEE/CVF Int. Conf. Computer Vision, Seoul, Republic of Korea, 2019, pp. 583–592.
Crossref
[34]
J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin, Generalizing to unseen domains: A survey on domain generalization, in Proc. Thirtieth Int. Joint Conf. Artificial Intelligence, Montreal, Canada, 2021, pp. 4627–4635.
Crossref
[35]
Q. Xu, R. Zhang, Y. Zhang, Y. Wang, and Q. Tian, A Fourier-based framework for domain generalization, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 14378–31487.
Crossref
[36]
K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, Domain generalization with MixStyle, presented at the 9th Int. Conf. Learning Representations, Vienna, Austria, 2021.
[37]
R. Wang, M. Yi, Z. Chen, and S. Zhu, Out-of-distribution generalization with causal invariant transformations, in Proc. 2022 IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022, pp. 375–385.
Crossref
[38]

H. Zhang, L. Wang, K. A. Lee, M. Liu, J. Dang, and H. Meng, Meta-generalization for domain-invariant speaker verification, IEEE/ACM Trans. Audio Speech Language Process., vol. 31, pp. 1024–1036, 2023.
Crossref Google Scholar
[39]

B. Zhang, S. Wang, and F. Gao, Contrastive metric learning for lithium super-ionic conductor screening, SN Comput. Sci., vol. 3, no. 6, p. 465, 2022.
Crossref Google Scholar
[40]
X. Yao, Y. Bai, X. Zhang, Y. Zhang, Q. Sun, R. Chen, R. Li, and B. Yu, PCL: Proxy-based contrastive learning for domain generalization, in Proc. 2022 IEEE/CVF Conf. Computer Vision and Pattern Recognition, New Orleans, LA, USA, 2022, pp. 7087–7097.
Crossref
[41]

Y. Chen, P. Song, H. Liu, L. Dai, X. Zhang, R. Ding, and S. Li, Achieving domain generalization for underwater object detection by domain mixup and contrastive learning, Neurocomputing, vol. 528, pp. 20–34, 2023.
Crossref Google Scholar
[42]

Q. Zhao, X. Wang, B. Wang, L. Wang, W. Liu, and S. Li, A dual-attention deep discriminative domain generalization model for hyperspectral image classification, Remote Sens., vol. 15, no. 23, p. 5492, 2023.
Crossref Google Scholar
[43]

Y. Liu, H. Ge, L. Sun, and Y. Hou, Complementary attention-driven contrastive learning with hard-sample exploring for unsupervised domain adaptive person Re-ID, IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 1, pp. 326–341, 2023.
Crossref Google Scholar
[44]
Q. Miao, J. Yuan, and K. Kuang, Domain generalization via contrastive causal learning, arXiv preprint arXiv: 2210.02655, 2022.
[45]
Z. Chen, W. Wang, Z. Zhao, and A. Men, Causality-based dual-contrastive learning framework for domain generalization, arXiv preprint arXiv: 2301.09120, 2023.
[46]
H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan, Deep hashing network for unsupervised domain adaptation, in Proc. 2017 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 5385–5394.
Crossref
[47]

Y. He, Z. Shen, and P. Cui, Towards non-I.I.D. image classification: A dataset and baselines, Patt. Recognit., vol. 110, p. 107383, 2021.
Crossref Google Scholar
[48]
D. Li, Y. X. Yang, Y. Z. Song, and T. M. Hospedales, Deeper, broader and artier domain generalization, in Proc. 2017 IEEE Int. Conf. Computer Vision, Venice, Italy, 2017, pp. 5543–5551.
Crossref
[49]
M. M. Zhang, H. Marklund, A. Gupta, S. Levine, and C. Finn, Adaptive risk minimization: A meta-learning approach for tackling group shift, presented at the Int. Conf. Learning Representations, Vienna, Austria, 2021.
[50]
V. N. Vapnik, Statistical Learning Theory. New York, NY, USA: Wiley, 1998.
Tsinghua Science and Technology
Volume 30 Issue 4,
August 2025
Pages 1636-1647
DOI: 10.26599/TST.2024.9010072
Cite this article:
Wang X, Zhao Q, Wang L, et al. Causality-Based Contrastive Incremental Learning Framework for Domain Generalization. Tsinghua Science and Technology, 2025, 30(4): 1636-1647. https://doi.org/10.26599/TST.2024.9010072

54

Views

9

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 26 December 2023
Revised: 18 March 2024
Accepted: 13 April 2024
Published: 03 March 2025
© The Author(s) 2025.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return