A multi-exit network is an important technique for achieving adaptive inference by dynamically allocating computational resources based on different input samples. The existing works mainly treat the final classifier as the teacher, enhancing the classification accuracy by transferring knowledge to the intermediate classifiers. However, this traditional self-distillation training strategy only utilizes the knowledge contained in the final classifier, neglecting potentially distinctive knowledge in the other classifiers. To address this limitation, we propose a novel multi-level collaborative self-distillation learning strategy (MLCSD) that extracts knowledge from all the classifiers. MLCSD dynamically determines the weight coefficients for each classifier’s contribution through a learning process, thus constructing more comprehensive and effective teachers tailored to each classifier. These new teachers transfer the knowledge back to each classifier through a distillation technique, thereby further improving the network’s inference efficiency. We conduct experiments on three datasets, CIFAR10, CIFAR100, and Tiny-ImageNet. Compared with the baseline network that employs traditional self-distillation, our MLCSD-Net based on ResNet18 enhances the average classification accuracy by 1.18%. The experimental results demonstrate that MLCSD-Net improves the inference efficiency of adaptive inference applications, such as anytime prediction and budgeted batch classification. Code is available at https://github.com/deepzlk/MLCSD-Net.

多出口网络是通过根据不同输入样本动态分配计算资源来实现自适应推理的重要技术。现有的工作主要将最终分类器视为老师，通过将知识传递给中间分类器来提高分类精度。然而，这种传统的自蒸馏训练策略仅利用最终分类器中包含的知识，忽略了其他分类器中潜在的独特知识。为了解决这个限制，我们提出了一种新颖的多级协作自蒸馏学习策略（MLCSD），该策略从所有分类器中提取知识。 MLCSD通过学习过程动态确定每个分类器贡献的权重系数，从而为每个分类器构建更全面、更有效的教师。这些新教师通过蒸馏技术将知识传回每个分类器，从而进一步提高网络的推理效率。我们在 CIFAR10、CIFAR100 和 Tiny-ImageNet 三个数据集上进行实验。与采用传统自蒸馏的基线网络相比，我们基于ResNet18的MLCSD-Net将平均分类精度提高了1.18%。实验结果表明，MLCSD-Net 提高了自适应推理应用的推理效率，例如随时预测和预算批量分类。代码可在 https://github.com/deepzlk/MLCSD-Net 获取。