Attention Is All You Need

A sample reading note for trying out the vault and publishing workflow. Replace it with your own notes as you read.

One-sentence Summary

The Transformer uses attention to build a sequence-to-sequence model and demonstrates its effectiveness on machine translation tasks.

Key Ideas

• Self-attention lets each position aggregate information from other positions.
• Multi-head attention computes attention in several representation spaces.
• Positional encodings provide information about sequence order.

Scaled dot-product attention is defined as:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\mathsf{T}}}{\sqrt{d_k}}\right)V$$

Here $Q$, $K$, and $V$ are queries, keys, and values; $d_k$ is the key dimension.

My Understanding and Questions

• Self-attention enables direct information exchange between tokens, but attention weights alone do not establish a causal explanation.
• Why divide by $\sqrt{d_k}$? How do compute and memory costs change with sequence length?

Relevance to My Work

A foundation for understanding language model architectures and inference costs. Related papers can reuse the same keyword tags.

Reference

Vaswani et al., Attention Is All You Need, 2017