Knowledge Distillation Method Based on Incremental Class Activation Knowledge

Due to the fact that features are not category-deterministic and the equipment resources are usually limit to support the category structure learning of samples, existing knowledge distillation methods possibly ignore the category knowledge distillation of samples. Therefore, this paper proposes a distillation method based on incremental class activation knowledge (ICAKD) . First, this paper uses the class activation gradient map to extract class-discriminative sample features and proposes a class-activation constraint loss. Then, an incremental memory bank is built to store class-deterministic features, and multiple training batch samples are saved and updated iteratively. Finally, our proposed method calculates the quasi-quality center of the samples in the memory bank to construct the category structure relationship, and further performs the category knowledge distillation according to the class-activation constraint and the category structure relationship. Experimental results on the Cifar10, Cifar100, Tiny-ImageNet, and ImageNet datasets show that the proposed method achieves a 0.4%～1.21% improvement in term of accuracy when compared with the Category Structure Knowledge Distillation(CSKD) methods, demonstrating the promising effectiveness of the characteristics and increment of category judgment for category knowledge distillation.