We present a unified representation of the most popular neural network activation functions. Adopting Mittag-Leffler functions of fractional calculus, we propose a flexible and compact functional form that is able to interpolate between various activation functions and mitigate common problems in training deep neural networks such as vanishing and exploding gradients. The presented gated representation extends the scope of fixed-shape activation functions to their adaptive counterparts whose shape can be learnt from the training data. The derivatives of the proposed functional form can also be expressed in terms of Mittag-Leffler functions making it suitable for backpropagation algorithms. By training an array of neural network architectures of different complexities on various benchmark datasets, we demonstrate that adopting a unified gated representation of activation functions offers a promising and affordable alternative to individual built-in implementations of activation functions in conventional machine learning frameworks.

我们提出了最流行的神经网络激活函数的统一表示。采用分数阶微积分的 Mittag-Leffler 函数，我们提出了一种灵活紧凑的函数形式，能够在各种激活函数之间进行插值，并缓解训练深度神经网络中的常见问题，例如梯度消失和爆炸。所提出的门控表示将固定形状激活函数的范围扩展到其形状可以从训练数据中学习的自适应对应函数。所提出的函数形式的导数也可以用 Mittag-Leffler 函数来表示，使其适用于反向传播算法。通过在各种基准数据集上训练一系列不同复杂度的神经网络架构，我们证明了采用激活函数的统一门控表示为传统机器学习框架中激活函数的单个内置实现提供了一种有前途且经济实惠的替代方案。