Efficient learning and model compression algorithm for deep neural network (DNN) is a key workhorse behind the rise of deep learning (DL). In this work, we propose a message passing-based Bayesian deep learning algorithm called EM-TDAMP to avoid the drawbacks of traditional stochastic gradient descent (SGD)-based learning algorithms and regularization-based model compression methods. Specifically, we formulate the problem of DNN learning and compression as a sparse Bayesian inference problem, in which group sparse prior is employed to achieve structured model compression. Then, we propose an expectation maximization (EM) framework to estimate posterior distributions for parameters (E-step) and update hyperparameters (M-step), where the E-step is realized by a newly proposed turbo deep approximate message passing (TDAMP) algorithm. We further extend the EM-TDAMP and propose a novel Bayesian federated learning framework, in which the clients perform TDAMP to efficiently calculate the local posterior distributions based on the local data, and the central server first aggregates the local posterior distributions to update the global posterior distributions and then update hyperparameters based on EM to accelerate convergence. We detail the application of EM-TDAMP to Boston housing price prediction and handwriting recognition, and present extensive numerical results to demonstrate the advantages of EM-TDAMP.

中文翻译：

---

通过期望最大化和涡轮深度近似消息传递的贝叶斯深度学习


深度神经网络 (DNN) 的高效学习和模型压缩算法是深度学习 (DL) 兴起的关键主力。在这项工作中，我们提出了一种名为 EM-TDAMP 的基于消息传递的贝叶斯深度学习算法，以避免传统的基于随机梯度下降（SGD）的学习算法和基于正则化的模型压缩方法的缺点。具体来说，我们将 DNN 学习和压缩问题表述为稀疏贝叶斯推理问题，其中采用组稀疏先验来实现结构化模型压缩。然后，我们提出了一个期望最大化（EM）框架来估计参数的后验分布（E-step）并更新超参数（M-step），其中E-step是通过新提出的涡轮深度近似消息传递（TDAMP）实现的算法。我们进一步扩展了 EM-TDAMP 并提出了一种新颖的贝叶斯联邦学习框架，其中客户端执行 TDAMP 以基于本地数据有效计算局部后验分布，而中央服务器首先聚合局部后验分布以更新全局后验分布分布，然后基于 EM 更新超参数以加速收敛。我们详细介绍了 EM-TDAMP 在波士顿房价预测和手写识别中的应用，并提供了大量的数值结果来证明 EM-TDAMP 的优势。