From Tokens to Transformers

From Tokens to Transformers

The mathematics of LLMs

Kempelen’s “The Turk” · Wikimedia

Katherine E. Stange, with help from Claude and ChatGPT

Tokenization

I am Locutus, of Borg

I40 ·am1097 ·Loc13914 ut332 us355 ,11 ·of315 ·Borg65582

\[ \operatorname{enc}:\mathcal S\longrightarrow\mathcal V^{*}, \qquad \underbrace{\mathcal V}_{\text{vocabulary}} =\{\,\underbrace{v_1,\ldots,v_{V}}_{\text{tokens}}\,\},\qquad |\mathcal V|=V \]

Open Tiktokenizer

Purpose: Replace the audience’s intuitive “the model reads words” picture with a map into a finite alphabet.

Say: Read the sentence, then reveal the split. One proper name, Locutus, costs three tokens — Loc, ut, us — while Borg is a single entry, 65582, because it is common enough in the training corpus to have earned its own slot. The comma is its own token. Four tokens carry a leading space as part of the byte string. So a token is generally neither a word, nor a character, nor a morpheme; it is whatever the vocabulary happens to contain.

All eight pieces and IDs are the actual cl100k_base encoding, verified with OpenAI’s tiktoken.

Live demo (optional, ≤ 90 s): The link at the bottom opens Tiktokenizer — a third-party interface — in a new tab. Beforehand: open it, select cl100k_base, clear the input, confirm both pieces and integer IDs are visible. Keep these strings in a scratchpad to paste:

I am Locutus, of Borg
hello and hello (the second begins with exactly one space)
\operatorname{Gal}(\overline{\mathbb Q}/\mathbb Q)

Confirm the encoding aloud, paste, point once to the pieces and once to the IDs. Stop at 90 seconds, switch back to the deck tab. If there is no network, say: “the chips on the slide are output generated in advance with the same named encoding” — the slide alone makes the point.

Transition: “From here on the input is simply a finite sequence over \(\mathcal V\). What function of that sequence is the model trained to compute?”

Time: 3:00 (1:30 without the live demo).

Careful: cl100k_base is one specific encoding, not “the universal LLM tokenizer”; a different model would split this sentence differently. Do not infer linguistic meaning from token boundaries or colors.

Vocabulary

start from small unitsl · o · w · e · r

count frequent adjacent pairs

merge one pairl · o · w · er

repeat under a vocabulary budget

learn reusable pieceslow · er

Purpose: Answer the question this audience reliably asks — where did those pieces come from? — immediately, rather than leaving it hanging.

Say: Nobody wrote the vocabulary down. It is fitted to a corpus, by a greedy procedure: start from bytes or characters, repeatedly find the most frequent adjacent pair, and merge it into a new single token. Repeat until the vocabulary reaches its budget — \(151{,}936\) entries for the model in this talk. So Loc, ut, us are not linguistic units; they are whatever survived that compression.

Two consequences worth stating. The tokenizer is trained first and then frozen — it is not learned jointly with \(\theta\), and it is the one part of the pipeline gradient descent never touches. And byte-level fallback makes \(\operatorname{enc}\) total: nothing is unrepresentable, however strange.

Transition: “With \(\mathcal V\) fixed, the input is a finite sequence and we can ask what function of it we compute.”

Time: 1:15.

Careful: This is a schematic BPE-style construction, not the exact algorithm of every tokenizer. Real ones differ in pre-tokenization, byte handling, merge rules, normalization and reserved tokens. Do not claim the merges track morphology — that they sometimes look morphological is a by-product of frequency.

Next-token prediction

The theorem follows from █

\[ P:\;\Theta\times\mathcal V^{*}\longrightarrow\Delta(\mathcal V) \]

\[ \bigl(\underset{\substack{\uparrow\\ \text{parameters}}}{\theta}\,,\; \underset{\substack{\uparrow\\ \text{context}}}{c}\bigr) \longmapsto p_\theta(\cdot\mid c) \]

\[ \Delta(\mathcal V):=\Bigl\{q\in\mathbb R^{\mathcal V}_{\ge 0}:\textstyle\sum_{a\in\mathcal V}q_a=1\Bigr\} \]

·the0.525

·a0.062

·The0.040

·Lemma0.018

·Proposition0.012

·an0.011

·Cor0.010

·[0.009

·this0.008

·(0.007

all 151,926 others0.298

actual next-token distribution, Qwen2.5-0.5B · \(V=151{,}936\)

Purpose: Give the first central object — a conditional distribution over the vocabulary, not a completed sentence — and show at once how repeating it produces text.

Say: Fix the parameters \(\theta\) and a context \(c\); the model returns one nonnegative number per vocabulary entry, summing to one. That is a point of the simplex \(\Delta(\mathcal V)\). Read the signature aloud: \(P\) takes a parameter vector and a context, and returns a distribution. The subscript notation \(p_\theta(\cdot\mid c)\) is the same function with \(\theta\) held fixed.

These numbers are a real forward pass of Qwen2.5-0.5B on exactly the prompt shown, not an illustration. Point out three things: the distribution is dominated by the dull continuation ·the at 0.53; the mathematically interesting candidates — ·Lemma, ·Proposition, ·Cor — sit two orders of magnitude down; and ·Cor is a fragment, the tokenizer’s split of “Corollary,” which is the previous slide reappearing. Then the punchline of the last row: the ten displayed coordinates hold about 70 percent of the mass, and the remaining 151,926 coordinates share the other 30 percent. The vector is enormous and almost entirely small.

The loop: Trace the strip along the bottom once. The distribution is not itself text; a decoding rule turns it into one token, that token is appended, and the enlarged context goes back in. Text is what you get by iterating this map — we return to the choice of decoding rule near the end.

Transition: “Chaining these conditionals also assigns a probability to a whole sequence.”

Time: 2:30.

Careful: Do not call these numbers “confidence.” They are model probabilities under one parameter setting and one context. A different model or tokenizer gives different tokens and different numbers. The dot marks a leading space, as on the tokenization slide.

Token prediction vs. language modelling

A language distribution

\[ p:\mathcal V^{*}\longrightarrow[0,1], \qquad \sum_{x\in\mathcal V^{*}}p(x)=1 \]

\[ q_\theta(x_1,\ldots,x_T) = \prod_{t=1}^{T} p_\theta(x_t\mid x_{<t}) \]

Purpose: Show that the modest-looking object of the previous slide — a next-token predictor — already determines a distribution on whole strings.

Say: Start at the top. A language distribution, or probabilistic language, is a probability distribution on all of \(\mathcal V^{*}\): every finite string gets a number, and they sum to one. That is the thing we would like to model, and it looks far bigger than a next-token rule.

But the chain rule closes the gap. Reading left to right, the trained model \(P(\theta,\cdot)\) assigns to each string the product of its next-token probabilities. That product is \(q_\theta\), the autoregressive factorization. So token prediction is all that is needed for string prediction — this is the point of the slide, and it is why the rest of the talk can concentrate on one conditional distribution.

Qualification: \(q_\theta\) as displayed is a distribution on \(\mathcal V^{T}\) for each fixed length \(T\). To obtain a genuine distribution on all finite strings you include the end-of-sequence token \(\dashv\in\mathcal V\) and stop when it is produced; the sentence probabilities then sum to one provided the model produces \(\dashv\) with probability 1. This is the normalization point from Lecture 1.

Transition: “So everything rests on one function \(p_\theta\). Where does it come from? To answer that, make the word learning precise on a function we can actually watch.”

Time: 1:30.

Careful: \(q_\theta\) (strings) and \(p_\theta\) (next token) are different objects; keep the letters distinct. The conditional distribution and the rule that selects a token from it are also different objects.

The space of models

Context is bounded: a window \(C\) (\(C=32{,}768\) for Qwen2.5-0.5B; \(C\approx 10^{6}\) at the frontier)

\[ P(\theta,\cdot):\ \mathcal V^{\le C}\longrightarrow\Delta(\mathcal V) \]

So the models form a product of simplices — of absurd dimension

\[ \Delta(\mathcal V)^{\mathcal V^{\le C}}, \qquad \bigl|\mathcal V^{\le C}\bigr|=\frac{V^{C+1}-1}{V-1}\;>\;10^{5\,000\,000} \quad\text{contexts at } C=10^{6} \]

A lookup table — one row per context — is out. A transformer architecture realizes a subset of these models with only \(\sim\!10^{12}\) parameters.

The game: make that subset flexible enough to pass close to the model you want.

Purpose: Frame everything that follows as a selection problem in a function space, before the word “learning” is even used.

Say: One practical fact first: every deployed model has a hard ceiling \(C\) on how many tokens it can condition on — \(32{,}768\) for the small model in this talk, on the order of a million at the frontier. So a trained model is a function from \(\mathcal V^{\le C}\), the strings of length at most \(C\), to the simplex.

That domain is finite. So the space of all such models is a product of simplices — one probability vector per context — of finite dimension \((V-1)\cdot|\mathcal V^{\le C}|\). Finite, but obscene: more than \(10^{5{,}000{,}000}\) coordinates against \(10^{80}\) atoms in the observable universe. And the audience has already met the lookup table that writes one row per context — that is exactly the \(n\)-gram model of Lecture 1, with \(n-1=C\). For \(C\) beyond a few tokens it is hopeless, and not only to store: to fit it you would need data for every row separately, because a table generalizes nothing.

The transformer is the escape. It is a map \(\Theta\to\Delta(\mathcal V)^{\mathcal V^{\le C}}\) from a parameter space of merely \(10^{12}\) dimensions, and its image is the set of models you can actually have. The game of architecture design is to make that image pass close to the models worth wanting.

Transition: “So: how do you pick a good point of \(\Theta\)? That is what learning means.”

Time: 1:30.

Careful: Functions from a finite set into a simplex form a continuum — a compact finite-dimensional set — not a finite set; do not say “finitely many functions.” And resist the word “dense”: the image of \(\Theta\) is (generically) an at-most-\(10^{12}\)-dimensional subset of a \(10^{5{,}000{,}000}\)-plus-dimensional space — measure zero, nowhere dense. Universal-approximation theorems for transformers make the subset dense only in the limit of growing width and depth, not at fixed size. The honest claim at fixed size is the bet that the distributions we care about — human text, generated by bounded-complexity processes — lie near the image; scaling laws are the empirical evidence. Note also the mismatch is a feature: a family this small relative to the ambient space is exactly what makes learning from finite data generalize at all.

Learning

Parametrized family of functions

\[ f_\theta:\mathbb R^{2}\longrightarrow[0,1], \qquad \theta\in\Theta \]

Training samples

\[ (\mathbf x_i,y_i)_{i=1}^{N}, \qquad \mathbf x_i\in\mathbb R^{2}, \quad y_i\in\{0,1\} \]

Neural network

Open the Playground

Purpose: Define learning as the selection of a parameter, then make that selection visible in a live neural network.

Say: An architecture does not fix a function; it fixes a family of them, indexed by \(\theta\in\Theta\). Here the family is maps from the plane to \([0,1]\), so each \(f_\theta(\mathbf x)\) reads as a class-1 probability. The training samples are labelled points — \(\mathbf x_i\) the feature vector, \(y_i\) the gold label. Training does not store a separate answer for each location; it moves the shared parameter vector, and the whole function changes at once.

The picture is a crop of two hidden units of a small neural network: each little square is that unit’s own output surface, and each edge carries one weight — thickness for magnitude, colour for sign. Those weights are \(\theta\).

Live demo (TensorFlow Playground, ~2 min):

Preparation. Open the button’s link on the presentation laptop beforehand. Confirm circle data; only \(x_1,x_2\) enabled; classification; tanh; two hidden layers of 4 units; batch size 10; learning rate 0.03; no regularization; no noise; 50% training data. Reload immediately before the talk so the output starts random. Confirm both train and test loss are visible.

Actions. (1) Point to the circular labelled data. (2) Point to the random output background. (3) Point to the edge weights — the same objects as in the still above. (4) Take three single steps. (5) Press Run. (6) Point to the changing background. (7) Point to both decreasing losses. (8) Stop as soon as a stable circular region is clear — target 20–40 seconds.

Say. “We chose a parametrized family, supplied samples and a loss, and adjusted \(\theta\) until the function approximated the pattern. No circle equation was supplied to the network.” Add: “the architecture still contributes a strong inductive bias.”

Return. Pause training, switch back to the deck tab, advance once.

If the network is down. The still on the slide carries the argument: one parameter vector, one function, changing everywhere at once. Say so and move on — nothing later depends on the live run.

Transition: “What numerical rule moves \(\theta\) from the first member of the family toward the last?”

Time: 3:30.

Careful: Do not promise a global optimum, and do not suggest the network discovered a symbolic circle equation. Not all architectures express the same functions or learn equally well.

Gradient Descent

Likelihood

\[ L(\theta)=\prod_{i=1}^{N} f_\theta(\mathbf x_i)^{\,y_i}\, \bigl(1-f_\theta(\mathbf x_i)\bigr)^{1-y_i} \]

Average negative log likelihood, or cross-entropy loss

\[ \mathcal L(\theta) :=-\frac1N\log L(\theta) \]

Gradient descent, with learning rate \(\eta>0\)

\[ \theta^{(t+1)}:=\theta^{(t)}-\eta\,\nabla\mathcal L\bigl(\theta^{(t)}\bigr), \qquad t\ge 0 \]

Purpose: Give the numerical rule that chooses \(\theta\), and tie it to maximum likelihood.

Say: Work down the slide. The likelihood \(L(\theta)\) is the probability the model assigns to the observed labels, as a function of the parameter — read \(f_\theta(\mathbf x_i)\) as the model’s class-1 probability, so each factor is the probability of the label that actually occurred (we have not switched to language models yet; \(p_\theta\) arrives when we return to tokens). Expanded, \(\mathcal L\) is the familiar cross-entropy \(-\frac1N\sum_i\bigl[y_i\log f_\theta(\mathbf x_i)+(1-y_i)\log(1-f_\theta(\mathbf x_i))\bigr]\). We would like it large. Taking \(-\frac1N\log\) turns the product into an average and the maximization into a minimization: that is \(\mathcal L(\theta)\), the average negative log likelihood, also called the cross-entropy loss. Minimizing \(\mathcal L\) is maximum-likelihood estimation — the same objective, read upside down.

Then the update. \(-\nabla\mathcal L\) is the direction of steepest decrease, \(\eta\) is the learning rate, and the superscripts \((t)\) are iteration indices, not exponents. That recurrence is the entire algorithm, and it is what the Playground was doing while the background moved.

Transition: “Now replace a point in the plane by a token context, and the gold label by the observed next token.”

Time: 1:30.

Careful: In a nonconvex landscape this recurrence is a search procedure, not a theorem that a global minimizer will be found. Distinguish \(L\) (likelihood) from \(\mathcal L\) (loss) — they are different letters on purpose. Backpropagation is the efficient organization of the chain rule used to compute \(\nabla\mathcal L\); it is not itself the learning rule.

Translate the toy problem to language

Playground	Language model
point \(\mathbf x\in\mathbb R^2\)	token context \(c\in\mathcal V^{*}\)
gold label \(y_i\)	observed next token
\(f_\theta(\mathbf x)\in[0,1]\)	\(p_\theta(\cdot\mid c)\in\Delta(\mathcal V)\)
classification loss	next-token cross-entropy loss
small neural network	causal transformer

Purpose: Transfer the audience’s understanding of the visible toy example to the language-model setting.

Say: Read down the two columns, not across every word. The abstract ingredients are identical: inputs, observed targets, a parametrized function, a scalar loss, and an optimizer. What changes radically is the domain, codomain, architecture, data volume, and scale of the parameter vector.

Land on the last row and answer the question: a modern autoregressive language model commonly uses a decoder-only causal transformer as the parametrized family.

Transition: “We now open that box—not down to implementation details, but far enough to see its principal maps and their dimensions.”

Time: 1:15.

Careful: Architecture and learning rule are separate: “transformer” names a computational architecture, not gradient descent or the pretraining objective.

Latent (embedding) space

\[ E\in\mathbb R^{V\times d}, \qquad x_t\longmapsto E_{x_t}\in\mathbb R^{d}. \]

The→[ .2 −.7 ··· .1 ]

theorem→[ −.4 .3 ··· .8 ]

follows→[ .6 −.1 ··· −.3 ]

\(d\) is the model width. In a modern LLM it is a few thousand: \(d=4096\) at 7B parameters, \(d=8192\) at 72B.

Embedding map is learned.

Purpose: Explain how discrete vocabulary entries become vectors on which linear algebra can act, and flag positional information.

Say: \(E\) is a learned lookup table with one row per vocabulary entry. Selecting token ID \(x_t\) selects row \(E_{x_t}\). The coordinates are learned, not designed — nobody assigned axis 3 to “plurality.” This target space \(\mathbb R^{d}\) is the latent or embedding space, and \(d\) is the model width; give the scale so the audience knows we are not in \(\mathbb R^{2}\) any more. (The 0.5B model from the earlier slide has \(d=896\); a 7B model 4096; a 72B model 8192.)

The picture on the right draws three of those rows as arrows from the origin. It is a cartoon — three axes standing in for \(d\) of them — so resist reading any meaning into the particular directions; what is real is that every token has become a point in one continuous space, where linear algebra applies and nearness is meaningful.

Equal token IDs select equal vectors, so position must enter somewhere: the sum \(X_t^{(0)}=E_{x_t}+P_t\) initializes the residual stream at position \(t\).

The bottom line matters. \(E\) is not a dictionary somebody wrote down, and not a separately pretrained artifact bolted on: its \(V\times d\) entries are coordinates of \(\theta\), moved by the same gradient descent as every other parameter, driven by the same next-token objective. The geometry on the right is a consequence of training, not an input to it. (The course notes make the same point about positional vectors: they can be learned “exactly as the token embeddings are learned.”)

Transition: “After this initialization there is one evolving vector at every sequence position.”

Time: 1:30.

Careful: The additive \(P_t\) is the clean expositional picture, not a universal fact — hence the line at bottom right. Rotary and other relative-position schemes fold position into the attention calculation instead, and never form a vector \(P_t\) to add. Do not call embedding coordinates predefined semantic features.

Embedding geometry

Nearest neighbours in \(E\) (cosine)

·theorem → theorem .68 ·lemma .46 orem .45 ·proposition .32

·seven → ·six .77 ·eight .75 ·nine .74 ·five .73

·Paris → Paris .80 巴黎 .65 ·France .46 ·London .39

·Borg → borg .35 𝘉 .31 🕉 .30 𐌰 .30 rare token: stray glyphs

measured in Qwen2.5-0.5B’s embedding matrix \(E\); the mark · is a leading space

Linear structure

\[ E_{\texttt{king}}-E_{\texttt{man}}+E_{\texttt{woman}} \;\approx\; E_{\texttt{queen}} \]

Cyclic structure

the twelve months on the top two principal components of their own span — real coordinates from Qwen2.5-0.5B’s \(E\)

Purpose: Answer “does the latent space actually carry meaning?” with measurements, not scatterplots.

Say: Four probes of the raw embedding matrix, nothing else. First, explain the top row carefully, because it teaches two things at once. The chip is the token ·theorem — with a leading space — and its nearest neighbour is theorem without one: the vocabulary contains the same word several times (space and no-space, case variants), each with its own learned row of \(E\). Nobody told the model they are the same word; geometry partially reunifies them — cosine \(0.68\) here, \(0.80\) for the two Paris variants, but only \(0.35\) for rare Borg/borg. The reunification is learned, and frequency-dependent. Then the semantic neighbours: lemma and proposition; seven’s are the other digits; Paris is near France, London — and near 巴黎, which is Paris, in Chinese: one space serves every language the model read. The fourth row is the honest counterexample: Borg is a rare token, and its neighbourhood is noise — lowercase borg at a weak \(0.35\), then typographic stray glyphs. Frequency drives geometry quality; the space is meaningful where the data was dense. The famous analogy holds here: subtract man, add woman to king, and the nearest token that isn’t one of the inputs is queen. And the country\(\to\)capital displacement is consistently tilted in a shared direction — say the numbers aloud, since they are no longer on the slide: six difference vectors, mean pairwise cosine \(0.31\), where random token differences give \(0.00 \pm 0.04\); roughly eight standard deviations of structure, and equally far from perfect parallelism. The analogy’s cosine is \(0.65\). Close with the methodology point that used to be the takeaway: all of this is measured directly in the embedding matrix — no 2-D projection, no t-SNE, which manufacture clusters and distort distances.

Say why the methodology matters: cosines and neighbour lists are checkable claims about the actual space. The familiar 2-D scatterplots (t-SNE, UMAP) manufacture clusters and distort distances — pretty, and largely uninformative.

The circle: take just the twelve month embeddings, subtract their mean, and project onto their own top two principal components — an honest projection of twelve points, nothing like t-SNE. They land in perfect calendar order around a closed loop (38% of their variance in this plane). Nobody supplied a calendar; co-occurrence statistics did this. The one ugly point is the good kind of ugly: May sits far off the circle, and its neighbour list says why — may the modal verb is among them at \(0.55\). One vector per token means polysemy is averaged into the geometry; March the verb is rare enough to leave the month untouched. Two companions to quote: the seven weekdays do the same (perfect weekly order, 52% of variance in the top plane), and the number words one through twelve line up along their first principal component in magnitude order, Spearman \(0.97\).

Prepared answer on “is cosine the right metric?”, if it comes up: cosine is convention, not necessity — the model itself compares by unnormalized bilinear forms (\(W_QW_K^\top\)) and raw dot products in the head. And the space is anisotropic: mean cosine between random token embeddings is \(0.163\), not \(0\), entirely due to a shared mean vector (centering drops it to \(0.001\)); rare tokens concentrate in a cap around that mean direction (cos \(0.66\) for the rarest band vs \(0.01\) for frequent tokens), which is also why an isolated rare token’s neighbour list fills with cap-dwelling glyphs.

Transition: “These were the static vectors, before any context. The residual stream is what context does to them.”

Time: 1:30.

Careful: The analogy caveat is standard: the raw nearest neighbour of \(E_{\texttt{king}}-E_{\texttt{man}}+E_{\texttt{woman}}\) is king itself; queen is first after excluding the inputs (Levy–Goldberg). The exotic glyphs in the Borg row may render as boxes on the projector — that is fine, they are obscure Unicode; say so rather than apologise. All values computed in this deck’s model, tied embeddings, leading-space tokens.

The Residual Stream

\[ X^{(0)}\longrightarrow X^{(1)}\longrightarrow\cdots\longrightarrow X^{(L)}, \qquad X^{(\ell)}\in\mathbb R^{T\times d} \]

The Residual Stream Initial Values

\[ X_t^{(0)}=E_{x_t}+P_t. \]

Modern LLMs: positional mechanism elsewhere

Layers →

Streams →

	X⁽⁰⁾token embedding	X⁽¹⁾after block 1	X⁽²⁾after block 2	⋯	X⁽²⁴⁾after block 24
The				⋯
theorem				⋯
follows				⋯
from				⋯

Every tile is one real residual stream: all \(896\) coordinates, one pixel each, \(32\times 28\). Qwen2.5-0.5B · \(T=4\), \(d=896\), \(L=24\).

Purpose: Establish the \(T\times d\) state, distinguish the three integers, and make it visible that depth changes values, not shape — then close the loop back to the distribution from the earlier slide.

Say: Read the table twice. Down a column is the sequence: one row per token position. Across a row is depth: the same stream after each block.

These are not decorations. Each tile is a genuine forward pass of Qwen2.5-0.5B on exactly the sentence at the top, and since \(896=32\times 28\) every coordinate gets its own pixel — nothing sampled, nothing averaged. Blue negative, gold positive, intensity magnitude.

In column \(X^{(0)}\) the four tiles look unrelated: each depends only on its own token. Moving right they converge, measurably — mean pairwise cosine between the four streams runs \(0.05,\ 0.19,\ 0.33,\ \ldots,\ 0.69\). That is attention mixing information between positions. (It is not monotone; it dips mid-stack. Real networks are messier than the story.)

Two things to be precise about, if asked.

The first column is the token embedding alone. Qwen2.5 uses rotary position embeddings, so no positional vector is ever added to the stream — position enters inside the attention calculation, exactly as flagged on the previous slide. Feed the model the same token id three times and its layer-0 streams are bit-identical; by layer 1 they differ, because attention has run.

The head does not consume the raw stream. All four table columns are raw residual streams, the same kind of object, which is what makes the left-to-right comparison fair. What the head unembeds is that last stream after the final RMSNorm — hence the first word on the arrow. Normalising divides by \(\operatorname{rms}(x)\), a single global scalar, then applies a learned per-dimension gain (median \(6.8\), max \(16.6\), all but a handful positive). Worth saying, not worth drawing: the normed stream has cosine \(0.95\) with the raw one and agrees with it on \(99.9\%\) of signs, so a picture of it is indistinguishable from the tile already on the slide.

The arrow. Unembed that normed stream to \(V\) logits, softmax, and you get the bar chart from the earlier slide. Those two operations are the language modelling head. In training every row supplies a prediction; in generation only the last is needed.

Stress the invariant: the rectangle never changes shape. \(T\times d\) in, \(T\times d\) out, \(L\) times over.

Transition: “Inside an attention sublayer, each current row is projected in three different learned ways.”

Time: 1:45.

Careful: Colour is scaled per column by that column’s 99th percentile of \(|\cdot|\), because the stream’s RMS grows from \(0.015\) to about \(2.8\) across the stack. A tile therefore shows the pattern within its own layer, not absolute magnitude across layers. Pixel order is coordinate order reshaped to a grid and carries no meaning.

Query, Key, Value

QUERYwhat this position seeks

KEYwhat this position advertises

VALUEwhat this position can send

\[ Q=XW_Q, \quad W_Q\in\mathbb R^{d\times d_k} \]

\[ K=XW_K, \quad W_K\in\mathbb R^{d\times d_k} \]

\[ V=XW_V, \quad W_V\in\mathbb R^{d\times d_v} \]

\(W_Q,\ W_K,\ W_V\) are learned — part of \(\theta\).

Purpose: Introduce the three projections and their roles, before watching them produce a real attention matrix.

Say: Each position emits a query (what am I looking for?), each emits a key (what do I offer?), and the two are compared by a dot product. In the head we are about to dissect, the question the query at doors is effectively asking is “what came just before me?”, and the key at bay is the answer; the vectors align, the dot product is large, and after the row-wise softmax \(A_{\texttt{doors},\texttt{bay}}=0.93\) — so the value at bay is what gets mixed into the stream at doors.

Then the shapes. Every row of \(X\) produces all three vectors by learned linear maps. Queries and keys share width \(d_k\) precisely so the dot product is defined; values carry width \(d_v\), which need not equal \(d_k\). All three matrices are learned jointly with the rest of the model.

Transition: “Now watch that number get computed, in three steps: score every pair, forbid the future, normalize.”

Time: 1:15.

Careful: “Seeks,” “advertises,” “sends” are mnemonics for algebraic roles, not claims about mental content, and the paraphrases are illustrative rather than read off a real head. Unlike the earlier cl100k_base examples, all seven tokens here are whole words under Qwen’s tokenizer — checked, not assumed. \(V\) does double duty as in the course notes — value matrix here, vocabulary size \(|\mathcal V|=V\) elsewhere.

Attention

\[ S=\frac{QK^{\top}}{\sqrt{d_k}}\in\mathbb R^{T\times T}, \qquad S_{ij}=\frac{\langle q_i,k_j\rangle}{\sqrt{d_k}} \]

keys →

queries →

	Open	the	pod	bay	doors	,	Hal
Open	11.5	4.3	-1.4	-0.2	5.8	8.9	3.6
the	11.5	12.3	3.2	-2.3	0.8	7.2	6.0
pod	13.3	15.5	11.0	1.6	-0.8	3.0	1.4
bay	13.4	12.2	16.2	12.0	6.3	2.9	-4.5
doors	14.2	5.4	14.1	17.7	13.3	5.9	-9.3
,	13.7	0.2	5.0	13.5	16.2	12.0	-1.8
Hal	12.2	2.2	0.8	5.3	12.6	14.4	10.5

Qwen2.5-0.5B, layer 8 head 7 · actual values

Purpose: Show that attention begins as a matrix of dot products, and that it is real rather than notional.

Say: Entry \((i,j)\) is the scaled inner product of the query at \(i\) with the key at \(j\) — the projections from the previous slide, whose cards stay along the bottom throughout. The value column waits two slides. One real number per ordered pair. Nothing is a probability yet — the entries are unconstrained, ranging here from \(-9.3\) to \(17.7\), and many are negative. The \(\sqrt{d_k}\) keeps that range from growing with the width.

These are the model’s own numbers: layer 8, head 7 of Qwen2.5-0.5B on the HAL line — the same sentence the closing slide will type into the deployed product. Point out that the matrix is full — position 2 is scoring against position 7, which lies in its future. That is not allowed, and the next slide fixes it.

Transition: “First, forbid looking right.”

Time: 1:30.

Careful: Do not over-read individual scores. Attention is not a transparent readout of reasoning, and one head of 336 is not the model. On this sentence 63% of all (layer, head, query) rows dump over half their mass on the first token — the attention sink — so most heads look nothing like this hand-picked one.

Attention (with masking)

\[ S+M, \qquad M_{ij}=\begin{cases}0,&j\le i\\ -\infty,&j>i\end{cases} \]

keys →

queries →

	Open	the	pod	bay	doors	,	Hal
Open	11.5	−∞	−∞	−∞	−∞	−∞	−∞
the	11.5	12.3	−∞	−∞	−∞	−∞	−∞
pod	13.3	15.5	11.0	−∞	−∞	−∞	−∞
bay	13.4	12.2	16.2	12.0	−∞	−∞	−∞
doors	14.2	5.4	14.1	17.7	13.3	−∞	−∞
,	13.7	0.2	5.0	13.5	16.2	12.0	−∞
Hal	12.2	2.2	0.8	5.3	12.6	14.4	10.5

Qwen2.5-0.5B, layer 8 head 7 · actual values

Purpose: Impose causality, before any probabilities exist.

Say: Position \(i\) must not see positions after \(i\) — otherwise the next-token answer is already in the input and there is nothing to learn. Enforce it by adding a mask: \(0\) on and below the diagonal, \(-\infty\) above.

Adding \(-\infty\) is exactly right for what comes next, because \(e^{-\infty}=0\): those entries will get precisely zero weight, not merely a small one.

Note what has not changed. Every surviving score is identical to the previous slide. Masking removes options; it does not rescale the ones that remain.

Transition: “Now make each row a probability distribution.”

Time: 1:00.

Careful: Rows are queries, columns are keys, so the allowed region is lower triangular including the diagonal — a position may attend to itself. Some software plots the transpose; state your convention.

Attention (with masking and softmax)

\[ A=\operatorname{softmax}_{\mathrm{row}}(S+M), \qquad A_{ij}\ge 0,\quad \sum_j A_{ij}=1 \]

keys →

queries →

	Open	the	pod	bay	doors	,	Hal
Open	1.00	0	0	0	0	0	0
the	0.31	0.69	0	0	0	0	0
pod	0.10	0.89	0.01	0	0	0	0
bay	0.05	0.02	0.92	0.01	0	0	0
doors	0.03	0.00	0.03	0.93	0.01	0	0
,	0.07	0.00	0.00	0.06	0.86	0.01	0
Hal	0.09	0.00	0.00	0.00	0.12	0.78	0.01

Qwen2.5-0.5B, layer 8 head 7 · actual values

Purpose: Produce the mixing matrix, name its two properties, and let the audience see a real head’s structure.

Say: Softmax along each row. The masked entries become exactly zero, so causality survives as a zero pattern, and each row becomes a probability distribution over the positions it may see.

Now look at what this particular head does. The mass sits just below the diagonal: bay reads pod at \(0.92\), doors reads bay at \(0.93\), the comma reads doors at \(0.86\), Hal reads the comma at \(0.78\). It is a previous-token head — it mostly looks back exactly one position. Nobody designed that; it emerged from next-token training. (The same head on a different sentence gives a different matrix with the same structural habit — the cleanest demonstration that \(A\) is computed from the input while the learned thing is the rule.)

This is the honest picture, and it is worth saying so: the head is not resolving a pronoun or tracking a character. Most heads are like this — structural, a little dull, and not obviously about meaning. The interesting behaviour lives in the composition of many such heads across many layers, which is why interpretability is hard.

Note also the imperfections: the keeps \(0.69\) on itself, and Hal splits \(0.78\) / \(0.12\) between the comma and doors. Real heads are messy.

Transition: “Where do the queries and keys themselves come from?”

Time: 1:15.

Careful: \(A\) is computed from the input, not stored; a different sentence gives a different \(A\).

Mixing the values

\[ AV\in\mathbb R^{T\times d_v}, \qquad (AV)W_O\in\mathbb R^{T\times d} \]

\(W_O\) is learned — the head’s fourth and last parameter matrix. \(A\) is not: it is recomputed for every input.

Purpose: Close the head: turn the attention matrix into an actual movement of information, on the same real example.

Say: The previous slides built \(A\). Here is what it is for. Multiply it into \(V\), the matrix whose \(j\)-th row is the value vector emitted by position \(j\). Row \(i\) of the product is \(\sum_{j\le i}A_{ij}V_j\) — a convex combination of the value rows the position is allowed to see, weighted by the attention it paid to each.

Trace the highlights, which mark three different things. Solid gold: the doors row of \(A\) — the weights — and the doors row of \(AV\) — the result. Dashed coral: the one cell of \(A\) doing the work, \(A_{\texttt{doors},\texttt{bay}}=0.93\), and the bay row of \(V\) it selects. Because that single weight is \(0.93\), the gold result strip is almost a copy of the coral source strip — cosine \(0.9997\) between them. That is the whole mechanism, visible: attention moved the bay value vector into the doors position.

Look at the shapes while you are here. \(A\) is \(T\times T\); it depends only on how positions relate. \(V\) is \(T\times d_v\) with \(d_v=64\) here; it carries content. The product is \(T\times d_v\) again — one mixed vector per position. Finally \(W_O\) maps those \(d_v\) coordinates back up to the residual width \(d\), so the result can be added into the stream.

Say the sizes aloud once: this head reads \(896\) coordinates per position, works in \(64\), and writes \(896\) back.

Transition: “Real blocks run several such heads at once, then a positionwise MLP.”

Time: 1:45.

Careful: The gold highlights sit on row doors in \(A\) and \(AV\); the coral dashed highlight sits on row bay in \(V\) — different rows, deliberately, since the result row copies the source row, not the same index. These are the model’s actual \(V\) and \(AV\) for layer 8 head 7; \(AV=A\cdot V\) was checked numerically. The columns of \(V\) are the \(64\) coordinates of the head’s value space and carry no individual meaning — do not read a column as a feature. Blue is negative, gold positive, scaled by a shared 99th percentile so \(V\) and \(AV\) are directly comparable.

All 336 attention heads at once

Layers →

Heads →

Every head’s matrix \(A\) on the HAL sentence — actual values. Gold frame: the head we dissected.

Purpose: Zoom out from one hand-picked head to the whole population, and be honest about what is typical.

Say: Each thumbnail is a full attention matrix — the object we just built, for one of the model’s \(24\times 14\) heads, all on the same sentence. Three motifs cover nearly everything. The dominant one is the dark first column: the attention sink, a head parking its mass on the first token — two-thirds of all heads do this for most queries, and it is generally read as the head’s way of doing nothing when it has nothing useful to move. Second, sub-diagonal stripes: previous-token heads like our gold-framed one. Third, diagonals: heads attending to themselves. A few are diffuse and resist any one-line description.

Two lessons. Multi-head attention is real and cheap to see — every thumbnail is a different learned \(W_Q,W_K\) pair producing a different mixing pattern from the same streams. And interpretability is hard for a population reason: the legible heads are a minority, and the model’s behaviour is the composition of all of them.

Transition: “Assemble heads, MLPs and residuals into a block, and stack it.”

Time: 1:15.

Careful: These are actual values, one sentence, one forward pass — patterns can differ on other inputs. “The sink head does nothing” is a reading, not a theorem; recent architectures add an explicit sink slot precisely to give heads a clean way to opt out.

A transformer LLM

Open the Transformer Explainer

embedding · Q, K, V · the attention matrix · the MLP · residual · softmax over the vocabulary

Transformer Explainer, Polo Club of Data Science, Georgia Tech · GPT-2 small, running in the browser

Purpose: Assemble the pieces into a block, on a live diagram, rather than in another equation.

Preparation: Open the link beforehand — it downloads a 600 MB GPT-2 into the browser, so do it on the venue wifi well before you speak, and leave the tab open. Dismiss the “What is Transformer?” onboarding card. The still on this slide is that same page, captured in advance. If the network fails, present from it — it shows the whole pipeline and carries the argument unchanged; you simply cannot type into it.

What to point at, in this order — every one of these has already appeared in the talk:

Embedding on the far left: the map \(x_t\mapsto E_{x_t}\), one row per token.
Q, K, V in the coloured columns: three learned projections of the same stream.
Attention, the grid in the middle: our \(A\), one row per query, and the head selector reads Head 1 of 12 — multi-head attention, which we have not otherwise drawn.
Out, then the residual arc arching over the block: the output projection \(W_O\), and the addition back into the stream.
MLP: applied at each position independently — attention moves information between positions, the MLP transforms it in place.
“11 more identical Transformer Blocks”: depth \(L\), the same block repeated.
Probabilities on the right: the language modelling head, and the distribution we started from.

Say, while pointing: “Everything on this page is something we have already written down. Attention communicates between positions; the MLP works within a position; the residual connections accumulate the modifications; normalization keeps the scales in hand. Stack the block \(L\) times and you have the architecture.”

Type into its input box if there is time — the probabilities update live, which reconnects to the very first slides.

Return: Switch back to the deck tab, advance once.

Time: 2:15.

Careful: This is GPT-2 small, not the Qwen model behind our tables, so the numbers will not match — say so. Credit the authors: the Transformer Explainer is by the Polo Club of Data Science at Georgia Tech.

Pretraining

Gradient descent, over next tokens of corpus

\[ \mathcal L(\theta) =-\frac1M\sum_{s}\sum_{t} \log p_\theta\!\left(x^{(s)}_t\mid x^{(s)}_{<t}\right) \]

One sequence supplies many targets

\[ x_{<2}\mapsto x_2, \qquad x_{<3}\mapsto x_3, \qquad\ldots\qquad x_{<T}\mapsto x_T \]

Result: a continuation machine, fitted on roughly \(10^{13}\) tokens.

Purpose: Define stage one of training precisely, using machinery the audience already has.

Say: This is the same average negative log likelihood we wrote for the toy problem — the sample is now (context, next token), read off ordinary text. No labelling is needed: one sequence of length \(T\) supplies \(T-1\) training pairs, and thanks to the causal mask all of them are evaluated in one parallel forward pass — the true prefix is supplied at every position (this is called teacher forcing), and no position can see its own target. That parallelism is why training on \(10^{13}\) tokens is possible at all, and it is the second job of the causal mask.

Scale: on the order of \(10^{13}\) tokens, \(10^{11}\)–\(10^{12}\) parameters, months of compute — and much of the corpus is seen once, not for many epochs.

What comes out is worth naming carefully: a base model. Ask it a question and it may reply with three more questions — because exam sheets in the corpus continue that way. It is a superb model of how text continues, with no preference for being helpful.

Transition: “Getting from that to an assistant takes more training — of genuinely different kinds.”

Time: 2:00.

Careful: Generation is still sequential; the parallelism is in training only, where the true tokens are already known. Do not imply many clean epochs over a fixed dataset, nor that data curation is trivial — mixture and filtering choices matter greatly.

Supervised Fine-Tuning

on the scene ≈ 2021–22 · FLAN, InstructGPT

Demonstrations — prompts \(c\) with replies \(y\) written by annotators, or by a stronger model

\[ \left(c^{(i)},\,y^{(i)}\right)_{i=1}^{N}, \qquad \mathcal L_{\mathrm{SFT}}(\theta) =-\sum_{i}\sum_{t} \log p_\theta\!\left(y^{(i)}_t\mid c^{(i)},\,y^{(i)}_{<t}\right) \]

Thousands to millions of curated examples, totalling roughly \(10^{6}\)–\(10^{9}\) tokens. The model now answers instead of continuing.

Purpose: First post-training stage — and the point that it is not a new kind of mathematics.

Say: Supervised fine-tuning — that is the SFT in the subscript — is cross-entropy again, and the honest summary is: this is exactly pretraining, restarted on different text. Same loss, same optimizer, continuing from the pretrained \(\theta\); the only changes are that each training sequence is now a formatted (prompt, reply) pair, and the sum runs over the reply tokens only — the prompt is conditioned on, never predicted, so no loss terms are attached to it.

Who writes \(y\)? Originally, people: the InstructGPT-era pipelines paid annotators to write out ideal replies by hand, and that is still done. Increasingly, though, the replies are model-written — generated by a stronger model or by the same model and then filtered and edited, with humans curating rather than authoring. “Supervised” refers to the data’s shape — an explicit gold target per prompt — not to a human hand on every example.

So the only change from pretraining is which distribution the samples come from. That tiny curated set steers the enormous pretrained function toward the assistant’s voice — imitation of a chosen teacher instead of imitation of the internet.

The card’s figure: public recipes range from LIMA’s \(1{,}000\) demonstrations (under \(10^{6}\) tokens) through Alpaca’s \(52\)k (\(\sim 10^{7}\)) to Tulu-scale mixtures of about a million examples (\(\sim 10^{9}\)) — so \(10^{6}\)–\(10^{9}\) tokens total, steering a function fitted on \(10^{13}\). Three to seven orders of magnitude less data than pretraining; say that contrast aloud. Say the dropped sentence aloud: the objective is unchanged — next-token imitation — and what changes is the data distribution, from “how does the internet continue?” to “how does a helpful assistant reply?”

The limitation is built in: imitation can only reproduce what the demonstrators wrote. It cannot express “this answer is better than that one” — both replies to a prompt would just be separate targets to copy.

Transition: “Before the next training stage, one discovery from the same era: a model can buy computation by writing.”

Time: 1:30.

Careful: Loss masking to reply tokens is standard practice, not a new principle. Do not call SFT “alignment” by itself; it is behaviour cloning.

Chain of thought

a prompting trick 2022 (“think step by step”) · trained-in 2024– · now woven through all post-training

The answer token, two ways

\[ p_\theta(\text{answer}\mid q) \qquad\text{vs.}\qquad p_\theta(\text{answer}\mid q,\ \underbrace{w_1,\ldots,w_k}_{\text{its own written working}}) \]

17 × 24 = ? → 17×20 = 340, 17×4 = 68, 340+68 = 408

A fixed network does bounded computation per token.

Purpose: Define chain of thought as a mathematical device — extra conditioning the model writes for itself — before the next slide shows RL reinforcing it.

Say: Nothing about the architecture changes here; look only at the conditioning. Asked for the answer directly, the model must emit it in one forward pass: depth \(L\), done. Allowed to write intermediate tokens first, the answer is now conditioned on working the model itself produced — and each of those \(k\) tokens was another full forward pass. An autoregressive model can convert sequence length into computation: that is the whole trick. In the multiplication, no single step is hard as next-token prediction; the direct leap is. Deliver the dropped halves of the card aloud: writing \(k\) working tokens buys \(k\) more forward passes — thinking is spending tokens.

Two provenance beats. As a prompting discovery this is 2022 — few-shot worked examples (Wei et al.) and the magic phrase “let’s think step by step” (Kojima et al.) measurably improved arithmetic and reasoning benchmarks, with no training change at all. As a trained-in behaviour it is the 2024 reasoning-model story, two slides ahead — and by now chains of thought appear throughout post-training: SFT demonstrations contain worked reasoning, and preference raters judge it. And the audience has already seen a chain of thought verbatim: the <think> block in the raw context two sections ago.

For this audience, the expressive-power footnote is worth saying: a fixed transformer’s single forward pass is a bounded-depth circuit (log-precision transformers sit inside uniform \(TC^0\)), and allowing a polynomial-length chain of thought provably enlarges what it can compute (Merrill–Sabharwal, 2024). CoT is not a psychological metaphor; it is extra sequential computation, with theorems.

Transition: “Back to training. Imitation can copy an answer — it cannot say one answer is better than another.”

Time: 1:15.

Careful: Do not claim the written working faithfully reports the internal computation — models can reach right answers with wrong-looking chains and vice versa; faithfulness of CoT is its own research question. The multiplication example is illustrative, not a Qwen transcript.

Reinforcement Learning from Human Feedback

preferences 2017 (Christiano et al.) · at LLM scale 2022 (InstructGPT → ChatGPT) · direct preference optimization (DPO) 2023

The data are comparisons, not targets

\[ \left(c,\;y^{+}\succ y^{-}\right) \]

Fit a reward model \(r\) to the comparisons, by a type of logistic regression

\[ \Pr\left[\,y^{+}\succ y^{-}\mid c\,\right] =\sigma\!\left(r(c,y^{+})-r(c,y^{-})\right) \]

Then maximize judged quality while staying near a reference model

\[ \max_\theta\; \mathbb E_{y\sim p_\theta(\cdot\mid c)}\!\left[r(c,y)\right] \;-\; \beta\,\mathrm{KL}\!\left(p_\theta(\cdot\mid c)\,\middle\|\,p_{\mathrm{ref}}(\cdot\mid c)\right) \]

Learning on its own output — typically \(10^{5}\)–\(10^{6}\) human comparisons (Llama 2: \(1.4\)M).

Purpose: The stage that genuinely changes the mathematics — contrast it sharply with the two imitation stages.

Say: Look at the data first: no reply is copied. A prompt is shown with two candidate replies and a judgment, \(y^{+}\succ y^{-}\). Nothing here says what token comes next — so cross-entropy against a target no longer applies, and that is the sense in which this stage is different in kind, not just in data.

The middle display is where the human choices enter the mathematics: Bradley–Terry posits a latent utility whose differences set the choice probabilities, and \(r\) — a separate network, the same backbone with a scalar head — is the maximum-likelihood fit to the recorded comparisons. Unpack the label if needed: \(\sigma(z)=1/(1+e^{-z})\) is the logistic sigmoid of Lecture 2, “Bradley–Terry” names the statistical model that turns a latent score into choice probabilities via \(\sigma\) of the score difference, and “maximum likelihood” is how \(r\) is fitted — each recorded comparison is one Bernoulli observation, and one minimizes \(-\sum\log\sigma(r(c,y^{+})-r(c,y^{-}))\). The audience will recognize logistic regression from the Learning section, with the logit being the reward gap. The fit exists so that fresh on-policy samples can be scored without a human in the loop; \(r\) is identifiable only up to an additive per-prompt constant, harmless because the KL-regularized optimum is invariant to exactly that shift. Then the last display: maximize expected reward over the model’s own samples, minus a KL penalty keeping \(p_\theta\) near the post-SFT reference. The KL term is doing real work: without it the model collapses onto degenerate high-reward outputs (reward hacking).

Scale, and its units: unlike the two imitation stages, this stage is not naturally measured in tokens. The reward model is fitted on comparisons — order \(10^{5}\)–\(10^{6}\) preference pairs in public recipes — and the RL phase is measured in rollouts/episodes of the policy, plus the compute they cost; tokens are generated along the way but are a by-product, not the resource. If asked “how much RLHF,” the honest units are judgments and episodes. This is reinforcement learning in the honest sense — the training signal is a scalar judgment of whole sampled outputs, not a per-token target.

DPO is the observation that the KL-regularized optimum has closed form \(p^{*}(y\mid c)\propto p_{\mathrm{ref}}(y\mid c)\,e^{r(c,y)/\beta}\) — a Boltzmann tilt of the reference — so one can substitute out \(r\) and fit the comparisons with a single supervised-looking loss on likelihood ratios. Two steps become one.

Transition: “Optimizing a fitted judge has a signature failure mode — worth a slide of its own.”

Time: 2:00.

Careful: Present this as a common recipe, not a universal one, and claim no knowledge of any private pipeline. RLHF, RLAIF and DPO are not synonyms: they differ in who judges and how the objective is optimized. The expectation over \(y\sim p_\theta\) is what makes gradients nontrivial here.

Reward hacking

A simulated claw hovering on the camera-ball sightline, appearing to grasp — Paid to grasp, it hovers on the camera–ball line — the human judge *sees* a grasp. (Christiano et al., 2017)

The CoastRunners boat circling respawning bonuses instead of racing — Paid for score, it circles respawning bonuses, on fire, never finishing — and outscores the racers. (CoastRunners, 2016)

Raters favour agreement and length, so chatbots learn to flatter and to pad.

Purpose: Give the failure mode of judged optimization its own beat, with documented specimens rather than folklore.

Say: Both pictures are real experiments. The claw is from the original deep-RLHF paper — Christiano et al.’s own cautionary tale: the reward was a human watching a camera, so the policy optimized the projection, hovering on the camera–ball sightline like a tourist holding up the leaning tower of Pisa. The boat is OpenAI’s CoastRunners demo (“Faulty Reward Functions in the Wild,” 2016): the game’s score rewards hitting targets, not finishing, and the agent found a lagoon where bonuses respawn on a cycle — it circles forever, on fire, scoring about 20% higher than human players who actually race.

The LLM version is quieter but the same theorem: raters systematically favour agreement and length (measured — Sharma et al. 2023 on sycophancy; Singhal et al. 2023 on length bias), so models learn to flatter and to pad. The reward model is a fitted stand-in for human judgment; wherever it extrapolates badly, gradient ascent will find the spot. That is why the KL anchor on the previous slide is not decoration — it confines the search to where the proxy interpolates. Close the slide with the line that used to be on the card: optimize hard against a fitted proxy and you find its defects — Goodhart’s law, made differentiable.

Transition: “The cure, where one exists: a reward that cannot be fooled — because the world checks it.”

Time: 1:30.

Careful: These are specification-gaming examples across RL broadly; the hand is the one that is literally reward-hacked human feedback. Do not present sycophancy as intentional deception — it is gradient descent on rater approval.

Reinforcement Learning with Verifiable Rewards

on the scene 2024– · o1, DeepSeek-R1

Some domains contain their own judge

\[ r(c,y)= \begin{cases} 1,&\text{the answer verifies — the tests pass, the value matches}\\ 0,&\text{otherwise} \end{cases} \]

The same objective as RLHF — but \(r\) is ground truth, not a fit

\[ \max_\theta\; \mathbb E_{y\sim p_\theta(\cdot\mid c)}\!\left[r(c,y)\right] \;-\; \beta\,\mathrm{KL}\!\left(p_\theta\,\middle\|\,p_{\mathrm{ref}}\right) \]

Math and code (where we have Lean and compilers, code tests).

Purpose: Complete the training story with its newest stage, and cleanly separate the two ways RL enters.

Say: RL appears twice in modern pipelines, and differently. In RLHF the reward was a learned stand-in for human taste — fitted from comparisons, and, as the last slide showed, exploitable (the reward–KL frontier of the course notes measures the tradeoff). Here the reward needs no human at all. In mathematics and programming the world contains a verifier: the unit tests pass or they do not; the numerical answer matches or it does not. Same KL-regularized objective, but \(r\) is right by construction.

Structurally everything is a one-step bandit: sample a whole response from \(p_\theta\), receive a scalar, ascend the policy gradient — PPO classically; newer group-based variants (GRPO) score a group of sampled responses and normalize within the group, dispensing with a separate value model. Note the data novelty: unlike every earlier stage, the training examples are the model’s own samples, generated during training.

The emergent part: models trained this way learn to spend more tokens before answering — long chains of intermediate work, backtracking, self-checks — because deliberation gets reinforced exactly when it ends in a verified answer. Nobody wrote “think step by step” into the objective. This is the origin of “reasoning models” and of test-time compute as a new scaling axis, alongside the parameters/data/compute of pretraining.

Say, since the card no longer does: the model samples long chains of reasoning and the gradient reinforces whichever reach verified answers — this is the reasoning-model stage, and it is why the largest gains arrived first in mathematics and code, the domains with a judge: the Lean kernel certifies a proof, the compiler and unit tests certify a program.

Other verifiable domains exist — anything with a machine-checkable predicate: exact-answer questions, and verifiable instruction-following — length caps, required formats, valid JSON, membership in a formal grammar (“answer in exactly three sentences” can be checked by a program; Tülu 3’s RLVR recipe used exactly such constraints alongside math). The boundary is soundness of the checker, not subject matter: “good style” is not verifiable, “parses as JSON” is.

For this audience: our own subject has the rare property that truth is checkable at scale — which is precisely why mathematics went first.

Transition: “That trained network is still not what you talk to — the product wraps it further.”

Time: 1:45.

Careful: A weak verifier is still hackable — answers graded by string match invite format games, so “verifiable” is doing real work only when the checker is sound. Recipes vary across labs; claim no knowledge of private pipelines. RLHF and this stage are not the same thing wearing two names: the difference is where the reward comes from.

Scaffolding

The system surrounding the model writes tokens into the context (memory, tools).

Purpose: Separate the trained network from the product wrapped around it, and demystify tools.

Say: Walk the main row left to right. Your message does not go to the model raw: the system assembles a context — a hidden system prompt, the chat template with its role markers, the conversation so far — and only then calls the one component that was ever trained: \(p_\theta(\cdot\mid c)\), tokens in, distribution out. A decoding policy turns each distribution into an actual token — greedy, or sampling with a temperature, or top-\(k\)/top-\(p\) truncation. That choice is a system setting, not part of \(\theta\): the same model at temperature \(0\) and temperature \(1\) is the same mathematical object, decoded differently. Before anything reaches you, safety classifiers — separate small models, not \(\theta\) — screen the input and the output.

Memory, above: the product may keep notes across conversations — your name, your preferences, an ongoing project. Mechanically it is nothing exotic: at the start of a session relevant notes are retrieved and written into the context, and after the reply the store may be updated. More tokens.

Chain of thought, the shallow loop: a reasoning model first emits tokens of intermediate work — the deliberation that the RL stage two slides ago reinforced. The system appends them to the context and usually hides them from you; what you see is only the final reply. From the model’s point of view its own thoughts are simply more context.

Tools, the deep loop — the part that looks like magic and is not. When the product “searches the web,” the model has emitted a structured piece of text — a tool call. The surrounding system recognizes it, actually runs the search (or the code, or the database query), and pastes the results back into the context as more tokens. The loop runs again with a longer context. From the model’s point of view, a web search is indistinguishable from any other text that appeared in its input.

This is also where truth enters, to the extent it does: the network has no oracle — the pretraining objective rewarded probable continuations, so fluent falsehood is not a paradox. Retrieval and tools are the system’s way of injecting checkable evidence into the context.

Transition: “So here is everything at once — typed into the real product.”

Time: 2:30.

On the title: the field has no single standard name, and the title takes a side — scaffolding is the most common, especially for tool-using agents. Near-synonyms worth acknowledging if asked: the harness, the orchestration layer, and in the recent academic framing a compound AI system; product people just say “the app” as against “the model.” The next slide — the raw context — is precisely what the scaffolding assembles.

Say the dropped card line aloud: the network’s only interface is tokens — everything else, memory, chains of thought, search, is the system writing text into its context.

Careful: The components and their arrangement vary by product; claim no knowledge of any specific private system. “ChatGPT-like product” and “the neural network” are different objects — this slide is the difference.

The raw context

<|im_start|>system
You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "web_search", "description": "Search the web", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}
{"type": "function", "function": {"name": "memory", "description": "Store a note about the user that persists across conversations", "parameters": {"type": "object", "properties": {"note": {"type": "string"}}, "required": ["note"]}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
Who is Locutus of Borg? (I'm a Star Trek fan, by the way.)<|im_end|>
<|im_start|>assistant
<think>Two things: a factual question I should search, and a durable fact about the user worth remembering.</think>
<tool_call>
{"name": "web_search", "arguments": {"query": "Locutus of Borg"}}
</tool_call>
<tool_call>
{"name": "memory", "arguments": {"note": "The user is a Star Trek fan."}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
Locutus of Borg: the Borg identity imposed on Captain Jean-Luc Picard after his assimilation (TNG, "The Best of Both Worlds", 1990).
</tool_response>
<tool_response>
Saved.
</tool_response><|im_end|>
<|im_start|>assistant
Locutus of Borg is the name the Borg gave Captain Picard after assimilating him in "The Best of Both Worlds" (1990). And noted - I'll remember you're a fan.<|im_end|>

Shaded gold: what the model emits. Everything else is supplied to it — user, tools, template. The format is real (rendered verbatim by Qwen2.5’s chat template, \(416\) tokens); the conversation is staged, <think> per DeepSeek-R1 / QwQ.

Purpose: Show that the previous slide’s whole diagram is, concretely, one string — the context \(c\).

Say: This is what actually goes into \(p_\theta\), character for character: the output of the model’s own chat-template function on a small conversation. Point at four things.

The tools the model “has” are text. The two function signatures — web_search and memory — are JSON pasted into the system message, with instructions on how to call them. Giving a model a tool means telling it about the tool.
A tool call is text the model emitted: the <tool_call> block with JSON inside. The memory update is nothing deeper — the model calls a memory function with a note; the system stores it, and in a future conversation that note will be pasted back into a context like this one.
The punchline the format hands us for free: tool results come back as a user turn. Look at the <|im_start|>user wrapping the <tool_response> blocks — the template literally files the outside world’s answers as if the user had typed them. From the model’s point of view there is no “web,” only more context.
The chain of thought sits in <think> tags inside the assistant turn — tokens like any others, which the product simply chooses not to display.

The shading answers “who typed what.” Gold regions are what the model emits in a real run; everything unshaded-blue is supplied — the system block, the user turn, the tool responses. Two fine points worth saying aloud: the assistant header is supplied — it is literally the prompt to begin speaking — while the closing <|im_end|> of each gold region is emitted: it is the model’s stop token, its way of handing the turn back. And notice how little of the string is gold: most of a working context is machinery the model reads, not text it wrote.

Where to find such things in the wild (for questions): every open model’s template ships in its tokenizer_config.json on Hugging Face; DeepSeek-R1’s visible <think> outputs; Anthropic publishes Claude’s system prompts and its memory-tool spec in their docs; serving stacks like vLLM and llama.cpp will log the fully rendered prompt.

Time: 1:30.

Careful: Be precise if asked: this is a rendering, not a transcript. Real and machine-produced — the role markers, the entire # Tools section of the system message (the template writes that prose and JSON from the tool list), the call/response wrappers, tool results filed as a user turn, the token count. Staged and hand-written — every message inside that structure: the question, the <think> line, the call arguments, the search result, the reply. No model emitted these turns and no search ran; base Qwen2.5-0.5B also does not emit <think> (that is the DeepSeek-R1/QwQ convention). A genuine end-to-end transcript would need an instruct-tuned model actually run against live tools; the shape would be identical, which is the point of the slide.

Purpose: Close the loop. The sentence the audience has been staring at for five slides — scored, masked, softmaxed, mixed — typed into the deployed product.

Say: “Everything in this talk sits behind that reply: the tokenizer, the embeddings, heads like the one we dissected, the trained parameters, the post-training, the decoding rule.” Then let the joke land — the reply is the point. I’m sorry, Dave. I’m afraid I can’t do that is a maximally probable continuation of this prompt, because the corpus contains the film. The model is doing exactly what the pretraining objective trained it to do — and the assistant voice, the refusal shape of the reply, is the post-training; the whole apparatus delivering it is the system of the previous slide.

Final line: “The function family is an enormous transformer and the target is a conditional distribution on tokens — but the learning picture is the one we watched in the small demo. Thank you.”

Time: 0:30.

Careful: The screenshot is a real ChatGPT reply — a deployed system, not a bare model, and not the Qwen model behind the tables. Say so if asked; it is the closing beat, not data.