The Transformer is fundamentally an **encoder-decoder** model. The encoder's job is to process the entire input sentence (e.g., in English) and compress it into a rich numerical representation that captures its meaning and context. The decoder then takes this representation and generates the output sentence (e.g., in French), one word at a time. Both the encoder and decoder are actually stacks of multiple identical layers (the original paper used 6), allowing the model to learn increasingly abstract features.

Explore the diagram below. The key information flow is from the input, through the encoder stack, then to the decoder stack, which also considers its own previous outputs to generate the next word. This structure is highly effective for sequence-to-sequence tasks like machine translation.

Transformers don't work with raw text. The first step is to break the input sentence into smaller pieces called **tokens**. These can be words or sub-word units. Each token is then mapped to a numerical vector called an **embedding**. This vector, which is learned during training, captures the token's semantic meaning. For example, the vectors for "king" and "queen" would be closer to each other in the vector space than to "apple".

Because the attention mechanism processes all words at once, it has no inherent sense of word order. "The dog chased the cat" and "The cat chased the dog" would look identical to it. To solve this, we inject **positional encodings**. These are vectors created using sine and cosine functions of different frequencies. This vector is added to each token's embedding, giving the model a unique signal for each position in the sequence.

This is the core of the Transformer. For each word, **self-attention** allows it to look at all other words in the input and decide which ones are most relevant. It does this by creating three vectors for each word: a **Query (Q)**, a **Key (K)**, and a **Value (V)**. The Query is like "what I'm looking for." The Key is like "what I contain." The model matches the Query of one word with the Keys of all other words. The strength of this match determines how much of each word's Value gets passed along.

**Multi-head** attention simply means doing this process multiple times in parallel with different Q, K, and V projections. This allows the model to learn different types of relationships simultaneously (e.g., one "head" might focus on grammatical relationships, another on semantic ones).

After the attention mechanism gathers contextual information from across the sequence, the output for each word's position is passed through a **Feed-Forward Network (FFN)**. This component processes each position's vector independently. It consists of two linear layers with a ReLU activation in between. A key feature is that it first expands the dimensionality of the vector (e.g., from 512 to 2048) and then contracts it back down. This provides additional processing depth and non-linearity for each token representation.

Each major sub-layer in the Transformer (like Multi-Head Attention and the FFN) is wrapped in an "Add & Norm" component. This consists of two crucial operations: a **Residual Connection** and **Layer Normalization**.

• Residual Connection (Add): This is a "skip connection" that takes the input to the sub-layer and adds it to the output of the sub-layer. This helps prevent the "vanishing gradient" problem in deep networks, making it easier to train them. It essentially ensures that the model doesn't "forget" the original information as it passes through transformations.
• Layer Normalization (Norm): This operation stabilizes the training process by normalizing the values within each layer. It ensures the numbers don't get too large or too small, which can destabilize learning.

The decoder's job is to generate the output sequence one token at a time. It has a structure similar to the encoder but with two key differences in its attention mechanisms.

After passing through the final decoder block, the resulting vector needs to be turned into a predicted word. This is done in two final steps:

• Linear Layer: A final linear layer acts as a classifier. It takes the vector from the decoder stack and expands its dimension to the size of the entire vocabulary (e.g., from 512 to 50,000+). The output is a raw score, or **logit**, for every possible word in the vocabulary.
• Softmax Layer: The softmax function is applied to these logits. It converts the raw scores into a probability distribution, where all values are between 0 and 1 and sum to 1. The word with the highest probability is then chosen as the next word in the output sequence.

This process repeats, with the newly predicted word being fed back into the decoder as input for the next step, until the model predicts a special "end-of-sentence" token.

The original Transformer had both an encoder and a decoder, which is ideal for sequence-to-sequence tasks. However, researchers found that the individual components are powerful on their own, leading to two major families of models: