Activation Functions

The activation function of the artificial neuron transforms the input values into outputs. Common activation functions include

• step (Heaviside) $h(x) = \left\{ \begin{array}{ll} 1 & \text{if } x \geq 0 \\ 0 & \text{if } x < 0 \end{array}\right.$
• sigmoid $\sigma(x) = \frac{1}{1 + e^{-x}}$
• softsign $s(x) = \frac{1}{1 + \vert x\vert}$
• hyperbolic tangent $\tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$
• ramp $r(x) = \max(0, x)$

Originally, only sigmoid activation functions (sigmoid and hyperbolic tangent) were commonly used; however, recently, the class of ramp functions has also been employed, which, in contrast to sigmoid functions, are unbounded. Artificial Neurons with the ramp activation function are defined as Rectified Linear Units (ReLUs) (KSH12a).