How does Transformer self-attention mechanism ($Q, K, V$) compute contextual token representations?

Self‑attention computes each token’s representation by projecting the input sequence into query, key, and value matrices, scaling the dot‑product of queries and keys, applying a softmax, and weighting the values.

Step‑by‑step

1. Linear projections:
Q = XW_Q, K = XW_K, V = XW_V where X∈ℝ^{L×d_model} and W_*∈ℝ^{d_model×d_k}. In PyTorch: Q = X @ W_Q.
2. Scaled dot‑product:
scores = Q @ Kᵀ / √d_k.
3. Masking (optional): add ‑inf to illegal positions before softmax.
4. Attention weights:
A = softmax(scores, dim=-1).
5. Weighted sum:
Z = A @ V.
6. Multi‑head concat & projection: split Q,K,V into h heads, repeat steps 1‑5, concatenate results Z_h, then output = Z_h @ W_O.

Single‑head vs Multi‑head (text table)

| Aspect | Single‑head | Multi‑head (h=8) |
|-----------------|-------------|------------------|
| Dim per head | d_k = d_model | d_k = d_model/h |
| Expressivity | Limited | Captures diverse subspaces |
| Compute (GFLOPs)| lower | ≈h× higher but parallelizable |

PyTorch snippet (v2.4)

import torch
def self_attention(X, W_q, W_k, W_v, W_o, mask=None, heads=8):
    B, L, D = X.shape
    d_h = D // heads
    Q = X @ W_q   # (B, L, D)
    K = X @ W_k
    V = X @ W_v
    Q = Q.view(B, L, heads, d_h).transpose(1,2)   # (B, h, L, d_h)
    K = K.view(B, L, heads, d_h).transpose(1,2)
    V = V.view(B, L, heads, d_h).transpose(1,2)
    scores = torch.matmul(Q, K.transpose(-2,-1)) / (d_h**0.5)
    if mask is not None:
        scores = scores.masked_fill(mask==0, float('-inf'))
    A = torch.softmax(scores, dim=-1)
    Z = torch.matmul(A, V).transpose(1,2).contiguous().view(B, L, D)
    return Z @ W_o

The resulting output replaces the original token embeddings and can be stacked in transformer layers, enabling contextualized representations for downstream tasks such as language modeling, translation, or vision‑language fusion.