Edge applications are increasingly empowered by deep neural networks (DNN) and face the challenges of adapting or retraining models for the changes in input data domains and learning tasks. The existing techniques to enable DNN retraining on edge devices are to configure the memory-related hyperparameters, termed m-hyperparameters, via batch size reduction, parameter freezing, and gradient checkpoint. While those methods show promising results for static DNNs, little is known about how to online and opportunistically optimize all their m-hyperparameters, especially for retraining tasks of edge applications. In this paper, we propose, MPOptimizer, which jointly optimizes an ensemble of m-hyperparameters according to the input distribution and available edge resources at runtime. The key feature of MPOptimizer is to easily emulate the execution of retraining tasks under different m-hyperparameters and thus effectively estimate their influence on task performance. We implement MPOptimizer on prevalent DNNs and demonstrate its effectiveness against state-of-the-art techniques, i.e. successfully find the best configuration that improves model accuracy by an average of 13% (up to 25.3%) while reducing memory and training time by 4.1x and 5.3x under the same model accuracies.

中文翻译：

---

在边缘重新训练 DNN 时内存相关超参数的自适应集成优化


边缘应用程序越来越多地受到深度神经网络 （DNN） 的支持，并面临着根据输入数据域和学习任务的变化调整或重新训练模型的挑战。在边缘设备上启用 DNN 重新训练的现有技术是通过批量缩减、参数冻结和梯度检查点来配置与内存相关的超参数，称为 m 超参数。虽然这些方法对静态 DNN 显示出有希望的结果，但人们对如何在线和机会性地优化其所有 m 超参数知之甚少，尤其是对于边缘应用程序的重新训练任务。在本文中，我们提出了 MPOptimizer，它根据运行时的输入分布和可用的边缘资源联合优化 m 超参数的集合。MPOptimizer 的主要特点是轻松模拟不同 m 超参数下再训练任务的执行，从而有效地估计它们对任务性能的影响。我们在流行的 DNN 上实施了 MPOptimizer，并证明了它对最先进技术的有效性，即成功找到最佳配置，在相同的模型精度下，将模型准确性平均提高 13%（高达 25.3%），同时将内存和训练时间减少 4.1 倍和 5.3 倍。