Activation Functions

The activation function (activation function) of an artificial neuron transforms the input values into outputs. Common activation functions include

• step function (Heaviside) $h(x) = \left\{ \begin{array}{ll} 1 & \text{if } x \geq 0 \\ 0 & \text{if } x < 0 \end{array}\right.$
• sigmoid $\sigma(x) = \frac{1}{1 + e^{-x}}$
• softsign $s(x) = \frac{1}{1 + \vert x\vert}$
• hyperbolic tangent $\tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$
• ramp $r(x) = \max(0, x)$

Originally, almost exclusively sigmoid activation functions (the sigmoid and hyperbolic tangent) were used, but more recently the class of ramp functions has also been adopted. Unlike sigmoid functions, ramp functions are unbounded. Artificial neurons with a ramp activation function are called Rectified Linear Units (ReLUs) (KSH12a).