Recurrent Neural Networks and Transformers (for Sequences and Images)

Figure 5.11: Conceptual diagram of the self-attention mechanism in a Transformer.

Convolutional Neural Networks (CNNs) represented a breakthrough in static-image processing, thanks to their ability to capture hierarchies of local features through convolutional filters and pooling. However, conventional CNNs are not suited to sequential data such as text, temporal signals, or video, in which the order and temporal relationship between elements are crucial. To address these problems, Recurrent Neural Networks (RNNs) were introduced. Their neurons have recurrent connections that enable them to retain a memory of the past state. RNNs have been applied to tasks such as speech recognition, machine translation, time-series analysis, and automatic image description (image captioning). However, conventional RNNs have difficulty learning long-term dependencies because of the vanishing or exploding gradient problem. To mitigate these limitations, more sophisticated architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have been developed. These architectures can control which information to retain or forget through gating mechanisms.

Despite this, the first generation of deep sequential neural networks was based on an encoder-decoder paradigm, in which the information in the input sequence was compressed into a lower-dimensional tensor that had to preserve as much useful information as possible for the task. However, this approach has an inherent limitation: relevant input elements may be overlooked or attenuated during compression.

A fundamental change came with the “attention mechanism” (attention mechanism). The idea behind attention is simple but powerful: instead of compressing all information into a single vector, the model can dynamically “focus” on the most relevant parts of the input sequence. Attention weights are computed based on the relevance of each element to the others, overcoming the limitations of representations compressed into recurrent states.

Let $X \in \mathbb{R}^{n \times d}$ be the input matrix of the sequence, where $n$ is the number of tokens (the individual units into which the algorithm divides the input sequence, varying from sequence to sequence) and $d$ is the embedding dimension (a numerical vector representing each token, fixed for the model and capable of encoding semantic and syntactic information).

The matrices called query ($Q$), key ($K$), and value ($V$) are obtained through linear projections:

$$% Etichetta per riferimento futuro Q = X W^Q, \quad K = X W^K, \quad V = X W^V$$ (5.117)

where:

• $W^Q, W^K \in \mathbb{R}^{d \times d_k}$ and $W^V \in \mathbb{R}^{d \times d_v}$ are weight matrices learned during training;
• $Q, K \in \mathbb{R}^{n \times d_k}$ and $V \in \mathbb{R}^{n \times d_v}$.

The self-attention mechanism can then be expressed in scalar form (for a single token) as:

$$% Etichetta per riferimento futuro \alpha_{i,j} = \text{sof... ...rt{d_k}}\right), \quad z_i = \sum_{j=1}^{n} \alpha_{i,j} V_j$$ (5.118)

where:

• $Q_i, K_j \in \mathbb{R}^{d_k}$ are, respectively, the query of token $i$ and the key of token $j$;
• $V_j \in \mathbb{R}^{d_v}$ is the value of token $j$;
• $\alpha_{i,j}$ is the scalar indicating how much token $i$ attends to token $j$;
• $z_i \in \mathbb{R}^{d_v}$ is the new representation of token $i$ resulting from the weighted combination of the values.
• The softmax function, which normalizes the weights $\alpha_{i,j}$ to values between 0 and 1, is defined as:
  

  $$\text{softmax}(\mathbf{x})_i = \frac{e^{x_i}}{\sum_{j=1}^{n} e^{x_j}}, \quad i = 1, \dots, n$$ (5.119)

  
  
  where $\mathbf {x}$ is the input vector (in our case $\mathbf{x} = \frac{Q_i K^\top}{\sqrt{d_k}}$ for row $i$).

Intuitively, attention can be viewed as a dynamic generalization of weighting methods such as Bag of Words or TF-IDF. However, whereas TF-IDF assigns static weights to terms, attention assigns contextual, task-dependent weights, allowing the model to focus selectively on the most relevant parts. Semantically, the resulting matrix $Z$ has the same length as the input ($n$ input tokens and $n$ output tokens), but each token is enriched with information from the global context of the sequence.

The attention mechanism led directly to the development of Transformers (VSP$^+$17), now the de facto standard for sequence modeling in natural language, computer vision, and multimodal learning. In Transformers, the central operator is self-attention, which makes it possible to model the relationships among all sequence elements directly and in parallel. Compared with RNNs, Transformers offer significant advantages in terms of parallelization, numerical stability, and the ability to learn long-range dependencies.

In computer vision, the application of Transformers led to the development of Vision Transformers (ViT) (DBK$^+$20), in which an image is divided into small regions (patches) treated as a sequence, similarly to words in a text. These models have demonstrated performance competitive with or superior to that of CNNs on various classification, recognition, and segmentation tasks, especially when large amounts of data are available.

In practical applications, a single self-attention module is not sufficient to extract enough information from the input tokens. The multi-head self-attention mechanism extends the idea of self-attention by allowing the model to examine the sequence from several perspectives simultaneously. In practice:

• $h$ different heads are used. For each head $p = 1, \dots, h$, separate projection matrices are defined:
  

  $$Q^{(p)} = X W^{Q,(p)}, \quad K^{(p)} = X W^{K,(p)}, \quad V^{(p)} = X W^{V,(p)},$$ (5.120)

  
  
  where each matrix $W^{Q,(p)}, W^{K,(p)}, W^{V,(p)} \in \mathbb{R}^{d \times d_k}$.
• Each head computes self-attention (scaling, softmax, and combination) in its own subspace. The compact matrix form is used, where $\text{Attention}$ is the scaled dot-product attention operator:
  

  $$Z^{(p)} = \text{Attention}\left(Q^{(p)}, K^{(p)}, V^{(p)}\r... ...left(\frac{Q^{(p)} (K^{(p)})^\top}{\sqrt{d_k}}\right) V^{(p)}$$ (5.121)

  
  

• The outputs of the different heads are concatenated:
  

  $$Z_{\text{concat}} = \big[Z^{(1)} \, ; \, Z^{(2)} \, ; \, \dots \, ; \, Z^{(h)}\big] \in \mathbb{R}^{n \times (h \cdot d_v)}$$ (5.122)

  
  
  and then projected again into a space of dimension $d$ (the model dimension) by means of an output matrix $W^O \in \mathbb{R}^{(h \cdot d_v) \times d}$:
  

  $$Z = Z_{\text{concat}} \, W^O$$ (5.123)

  
  

• Typically, $d_v = d_k$ and $d = h \cdot d_k$ are chosen, so that the concatenation followed by the projection produces a dimension consistent with the input to the next layer.
• This makes it possible to model different “types” of attention (e.g., syntactic and semantic relationships, local vs. global dependencies) within the same layer.

Table 5.1: Examples of geometry (layers, embedding, heads) for well-known Transformer models.

Model	Layers	$\mathbf{d}$	$\mathbf{h}$	$\mathbf{d_k} = d / h$
Transformer (base) (VSP$^+$17)	6 encoder + 6 decoder	512	8	64
BERT-Base	12 encoder	768	12	64
BERT-Large	24 encoder	1024	16	64
GPT-3 (175B)	many decoders	12288	96	128

Today, RNNs and Transformers are complementary tools: the former remain useful in scenarios involving relatively short sequences or limited resources, whereas the latter form the basis of the most advanced architectures in modern deep learning. The evolution from recurrent mechanisms to attention-based mechanisms marked a paradigm shift: from the idea of compressed memory to a dynamic, contextual representation in which the model autonomously decides “what to look at” for each sequence element.