Auto-Encoders

Auto-Encoders are particular types of unsupervised neural networks that aim to encode the input $\mathbf {x}$ into a compact representation $\mathbf{c}(\mathbf{x})$, allowing the input itself to be reconstructed. In this sense, Auto-Encoders can be viewed as an alternative to RBMs trained using the Contrastive Divergence algorithm, with which they share the goal of learning meaningful latent representations.

The network consists of two parts:

• an encoder $\mathbf{c}(\cdot)$ that transforms the input into a latent representation;
• a decoder $\mathbf{f}(\cdot)$ that attempts to reconstruct the input from the latent representation.

The objective is to minimize the negative log-likelihood of the reconstruction:

$$-\log P \left(\mathbf{x} \vert \mathbf{c}(\mathbf{x}) \right)$$ (5.114)

When the data distribution is Gaussian, this expression reduces to ordinary least-squares regression (see Section 2.7).

If, instead, the inputs $\mathbf{x}_i$ are binary (or follow a binomial distribution), the cost function becomes:

$$-\log P \left(\mathbf{x} \vert \mathbf{c}(\mathbf{x}) \righ... ...\log \left( 1 - \mathbf{f}_i( \mathbf{c}(\mathbf{x}) ) \right)$$ (5.115)

where $\mathbf{f}(\cdot)$ is the decoder associated with the encoder $\mathbf{c}(\cdot)$.

The function $\mathbf{c}(\mathbf{x})$ represents lossy compression (lossy compression). It is effective for representing data observed during unsupervised training, but may be suboptimal for data outside the training domain.