稀疏和密集深度神经网络的训练：参数更少，性能相同,Physical Review E

当前位置： X-MOL 学术 › Phys. Rev. E › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

稀疏和密集深度神经网络的训练：参数更少，性能相同
Physical Review E ( IF 2.2 ) Pub Date : 2021-11-29 , DOI: 10.1103/physreve.104.054312
Lorenzo Chicchi ₁ , Lorenzo Giambagli ₁ , Lorenzo Buffoni ₁ , Timoteo Carletti ₂ , Marco Ciavarella ₁ , Duccio Fanelli ₁

Affiliation

通过对直接空间中合适的传输算子的特征值和特征向量起作用，可以在倒数空间中训练深度神经网络。在冻结特征向量的同时调整特征值会产生参数空间的实质性压缩。后者根据定义与计算神经元的数量成比例。然而，由显示的准确度衡量的分类分数低于在相同架构的直接空间中进行学习并使用全套可训练参数（与相邻层的大小成二次关系）时获得的分数. 在本文中，我们提出了 Giambagli等人的光谱学习方法的变体。[纳特。社区。 12, 1330 (2021)]，它利用相邻层之间每个映射的两组特征值。特征值充当名副其实的旋钮，可以自由调整，以便 (1) 增强或静音输入节点的贡献，以及 (2) 使用我们将其解释为人工模拟的机制来调节接收节点的兴奋性稳态可塑性。可训练参数的数量仍然是网络大小的线性函数，但经过训练的设备的性能更接近于通过传统算法获得的性能，然而，后者需要相当大的计算成本。通过对特征向量矩阵的非平凡块采用合适的分解，最终可以填补传统训练和谱训练之间的剩余差距。

"点击查看英文标题和摘要"

Training of sparse and dense deep neural networks: Fewer parameters, same performance

Deep neural networks can be trained in reciprocal space by acting on the eigenvalues and eigenvectors of suitable transfer operators in direct space. Adjusting the eigenvalues while freezing the eigenvectors yields a substantial compression of the parameter space. This latter scales by definition with the number of computing neurons. The classification scores as measured by the displayed accuracy are, however, inferior to those attained when the learning is carried in direct space for an identical architecture and by employing the full set of trainable parameters (with a quadratic dependence on the size of neighbor layers). In this paper, we propose a variant of the spectral learning method as in Giambagli et al. [Nat. Commun. 12, 1330 (2021)], which leverages on two sets of eigenvalues for each mapping between adjacent layers. The eigenvalues act as veritable knobs which can be freely tuned so as to (1) enhance, or alternatively silence, the contribution of the input nodes and (2) modulate the excitability of the receiving nodes with a mechanism which we interpret as the artificial analog of the homeostatic plasticity. The number of trainable parameters is still a linear function of the network size, but the performance of the trained device gets much closer to those obtained via conventional algorithms, these latter requiring, however, a considerably heavier computational cost. The residual gap between conventional and spectral trainings can be eventually filled by employing a suitable decomposition for the nontrivial block of the eigenvectors matrix. Each spectral parameter reflects back on the whole set of internode weights, an attribute which we effectively exploit to yield sparse networks with stunning classification abilities as compared to their homologs trained with conventional means.

更新日期：2021-11-29

点击分享查看原文

点击收藏

阅读更多本刊新发论文本刊介绍/投稿指南