Auto-Encoders

Auto-Encoders are a specific type of unsupervised neural networks that aim to approximate the RBMs trained with the Contrastive Divergence algorithm. An Auto-Encoder (and in this context, RBMs serve as a perfect representation) allows for encoding the input $\mathbf {x}$ into a representation $\mathbf{c}(\mathbf{x})$ such that the input can still be reconstructed in some manner, minimizing the negative log-likelihood

$$-\log P \left(\mathbf{x} \vert \mathbf{c}(\mathbf{x}) \right)$$ (4.86)

. It is noteworthy that in the case where the distribution is Gaussian, one would recover the classical form of least squares regression (see section 2.8).

When the inputs $\mathbf{x}_i$ are binary (or the distribution is of a binomial type), the cost function becomes

$$-\log (P \left(\mathbf{x} \vert \mathbf{c}(\mathbf{x}) \rig... ...\log \left( 1 - \mathbf{f}_i( \mathbf{c}(\mathbf{x}) ) \right)$$ (4.87)

where $\mathbf{f}(\cdot)$ is the decoder associated with the encoder $\mathbf{c}(\cdot)$.

The function $\mathbf{c}(\mathbf{x})$ is a lossy compression function. This function is effectively a good compression method only for the data observed during the unsupervised training phase, but it will not perform well for all data in general that were not involved in the learning phase.