Activation Functions

The activation function (activation function) of an artificial neuron transforms input values into outputs. Common activation functions include

• step (Heaviside) $h(x) = \left\{ \begin{array}{ll} 1 & \text{if } x \geq 0 \\ 0 & \text{if } x < 0 \end{array}\right.$
• sigmoid $\sigma(x) = \frac{1}{1 + e^{-x}}$
• softsign $s(x) = \frac{1}{1 + \vert x\vert}$
• hyperbolic tangent $\tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$
• ramp $r(x) = \max(0, x)$

Originally, almost exclusively sigmoid activation functions (the sigmoid and hyperbolic tangent) were used, but more recently the class of ramp functions has also been adopted; unlike sigmoid functions, ramp functions are unbounded. Artificial neurons with a ramp activation function are called Rectified Linear Units (ReLUs) (KSH12a).