When someone says "an AI model", the thing they're pointing at is a file on a disk. Open it and there's no code in there, no rules, no sentences. There is a list of numbers. A very, very long one.

Each number in that list is called a parameter — you'll also hear them called weights, and the two words mean the same thing. When you see a model called "Llama 3.3 70B", the 70B is the count: seventy billion numbers.

A dense decoder-only transformer's weights are a set of 2-D tensors — per layer, four attention projections and three feed-forward projections, plus embeddings. The count is the headline: 70B parameters at 2 bytes each in BF16 is 140 GB of weights.

The operative property isn't the size, though — it's the reuse. In autoregressive decode you process one token at a time, so every weight matrix is multiplied against a single activation vector. The kernels are GEMV, not GEMM. There is no blocking, tiling or scheduling trick that creates data reuse that the problem doesn't contain: each weight is loaded, used for exactly one multiply-accumulate, and discarded.

Where did the numbers come from? An automatic process called training, which module 4 covers. For now the only thing that matters is that training happened once, somewhere else, and produced this list. The numbers are now frozen. Your computer never changes them.

Using the model is called inference, and it goes like this: your text becomes numbers, those numbers get combined with the model's numbers through a great deal of multiplication, and out comes one more word. Then the whole thing runs again for the next word. And again. One complete pass per word.

Weights are frozen after training, so unlike a training step there's no write traffic and no optimiser state — just a read-only stream. That sounds like the easy case. It isn't, because the stream is the entire parameter set and it recurs every token, with a serial dependency between tokens that forbids pipelining across them.

Put a number on the intensity. Each parameter participates in one multiply and one add per token, so the arithmetic is 2N FLOPs, and the traffic is N × bytes_per_param. At BF16 that's 2N / 2N = 1 FLOP per byte, exactly, independent of model size, layer count or hidden dimension.

So how much space does that take? Each number is stored in one or two bytes. Do the multiplication and a 70-billion-parameter model comes to somewhere between 40 and 140 gigabytes, depending on how tightly it's packed — which is module 4's subject.

Now the other half. Strip a computer down and two parts matter for our purposes.

The processor does the arithmetic. It is spectacularly fast — a good one performs trillions of calculations every second.

The memory is where the numbers sit while they're being worked on. It's a separate place, physically elsewhere on the board, joined to the processor by what amounts to a pipe.

And here's the part nobody mentions: the pipe is the problem. The processor can consume numbers far faster than memory can deliver them. Not slightly faster — roughly a hundred times faster.

The obvious architectural response to a read-only stream with no reuse is to stop streaming it: put the working set on-die and eat the fetch once. Run the area numbers and that door closes hard.

SRAM bitcell scaling has effectively stopped. TSMC's N5→N3 transition delivered roughly 0–5% bitcell shrink, and N3E none at all, against 1.6–1.7× logic density gains over the same step. N2 gets the high-density cell to about 0.0175 µm² — roughly 38 Mb/mm² — and TSMC's claimed ~22% SRAM density improvement over N3E comes mostly from periphery rather than the cell.

At 38 Mb/mm², holding a 40 GB (320 Gb) quantised model in on-die SRAM needs about 8,400 mm² of bitcell area alone — before periphery, before any logic, before yield. A full-field reticle is 858 mm². You are an order of magnitude past the largest die that can be printed, for the cells only.

So the weights live off-die in DRAM, permanently, and the entire discipline becomes an exercise in off-chip bandwidth and the energy of moving bits across a package boundary.

Two words for the two properties of that warehouse. You'll use both constantly from here on:

• Capacity — how much fits, in gigabytes. This one is binary: either your model fits or it doesn't.
• Bandwidth — how fast things come out, in gigabytes per second, written GB/s. This one is the speed limit.

Memory isn't one thing, either. It's a ladder, and each rung trades size against speed. This shape — tiny and instant at the top, vast and sluggish at the bottom — is true of every computer ever built, including the one you're reading this on.

The hierarchy is the usual one, but the ratios are what matter here. Note that the capacity you'd need sits three rungs below the bandwidth you'd want:

A model is 40–140 GB of numbers. A machine can pull numbers out of memory at some rate — call it 500 GB per second for a good laptop. And every word needs a complete pass through the whole pile.

So: 40 gigabytes to read, at 500 gigabytes per second, is 0.08 seconds. That's about 12 words per second, and that is your answer — not because of any calculation the chip did, but because that's how long the fetching took. The arithmetic occupied the processor for well under a thousandth of that time. It spent the rest waiting.

Decode throughput is therefore set by bandwidth ÷ bytes_per_token and essentially nothing else. A 40 GB quantised 70B on 500 GB/s of LPDDR gives ~12 tok/s; the same model on an 8 TB/s HBM3e part gives ~200. The FLOPS column of both spec sheets is, for this workload at this batch size, decorative.

Everything that follows is a way of attacking one of the three terms: reduce bytes per token (quantisation, MoE sparsity, speculative decode), raise bandwidth (HBM stacks, wider buses, unified memory), or raise the intensity so the FLOPS start to matter again (batching). Module 3 formalises this, and modules 6–7 are a tour of who chose which.

Models don't see letters, and they don't quite see words either. Before your text reaches the model, a piece of software called a tokenizer chops it into chunks from a fixed list. A token is one of those chunks: roughly four characters of English, or about three-quarters of a word.

Common words are usually one token each. Rare ones get broken into pieces. Numbers, code, emoji and non-English scripts fragment much harder — which is why the same paragraph can cost twice as much in Japanese as in English.

Why care? Because the token is the unit of everything: what you're billed, what counts against limits, and what the machine's time is spent on. A 40-page document isn't "40 pages" to the machine — it's about 20,000 tokens, and every one of them is work.

Tokenizers are byte-pair encoding: start from bytes, iteratively merge the most frequent adjacent pair, stop at a target vocabulary size. Two consequences worth holding.

First, vocabulary size sets the width of the embedding table and the output projection — vocab × d_model parameters each. Going from a 100k to a 200k vocabulary on a model with d_model = 8192 adds roughly 1.6 B parameters across the two, which is real capacity and real bandwidth.

Second, fertility — tokens emitted per unit of source text — is the multiplier on everything downstream. Frontier tokenizers sit at 100k–200k entries (cl100k_base holds 100,256; the o200k generation about 200,000), and the larger vocabulary was adopted substantially to cut fertility on CJK and code, where a naive tokenizer can burn several tokens per character.

Here's the surprising part. Answering you involves two phases that stress the machine in completely different ways.

Prefill is reading your question. Crucially, the machine can process every token of your input at once, because they're all already there — like skimming a whole page in one glance rather than word by word. That means it can load a chunk of the model once and use it against thousands of your tokens before moving on. Lots of work per delivery. This phase is limited by how fast the chip can calculate.

Decode is writing the answer, and it cannot work that way. Word five depends on word four, so they have to happen in order, one at a time. Each one drags the entire model out of memory to produce a single word. Almost no work per delivery. This phase is limited by the pipe.

Prefill processes the full prompt in parallel: weight matrices are loaded once and multiplied against a [seq_len × d] activation matrix. Arithmetic intensity scales with sequence length, so a 2,000-token prompt lands you far to the right of the ridge — genuinely compute-bound, and the one regime where peak FLOPS is the honest figure of merit.

Decode has a serial dependency across tokens, so the effective batch is 1 (or the server's concurrency) and the kernels degenerate to GEMV. Intensity collapses to batch FLOP/byte at BF16. Same weights, same code path, opposite side of the roofline.

Worth noting the second-order term: attention itself is O(seq²) in prefill, so ingest cost is superlinear in prompt length — at 100k-token prompts the attention term stops being a rounding error against the 2N·seq weight term.

Speed-for-you and speed-for-everyone are in tension, and the knob between them is how many people's requests get handled together. A server that waits 40 milliseconds to gather 64 users' requests into one bundle reads the model once for all 64 — sixty-four times the total output for the same fetching. Your laptop, serving only you, gets the worst possible efficiency and the best possible privacy. Both facts come from the same equation.

Latency and throughput trade off through batch size, and the mechanism is pure amortisation of weight traffic: one weight sweep serves B tokens, so intensity rises linearly with B until you hit the ridge. Continuous batching (admitting new sequences into an in-flight batch rather than waiting for the slowest to finish) is what makes this practical at serving scale, and paged KV cache is what stops the memory cost from exploding while you do it.

The context window is how much the model can hold in front of it at once: your prompt, plus its answer, plus everything said earlier in the conversation, because the whole history gets re-sent every turn. Current models advertise 128,000 to 1,000,000 tokens.

For hardware purposes, context isn't free storage — it's a second pile of numbers that grows while you talk. Every token processed leaves behind some notes so the model doesn't have to re-read the entire conversation from scratch each time. Those notes are called the KV cache, and for a 70B model they cost about 0.33 MB per token — roughly 42 GB at a full 128,000-token context, on top of the model itself.

Hold the shape of that: the model is a fixed cost, the conversation is a growing tax. Module 5 does the arithmetic properly.

Context length is a memory allocation, not a capability flag. Every processed token leaves K and V projections resident so attention doesn't recompute the prefix, and that cache is read in full on every subsequent decode step — so it's both a capacity cost and a bandwidth cost that grow linearly with sequence length.

For a Llama-3-class 70B with GQA (80 layers, 8 KV heads, 128 head dim, FP16 cache): 2 × 80 × 8 × 128 × 2 = 327,680 bytes/token ≈ 0.33 MB, so ~42 GB at 128k. Without GQA's 8× head sharing that same cache would be ~336 GB. Module 5 has the full derivation and the interactive.

The arithmetic inside a model is repetitive to the point of being boring: take a list of numbers, multiply each one by a corresponding parameter, add up the results, repeat a few hundred billion times. There's no branching, no searching, no clever algorithm. Just multiply-and-add, at colossal scale.

Which gives us the most useful estimate in the field, and it's almost embarrassingly simple: every parameter takes part in exactly one multiply and one add per word. Two operations per parameter.

A transformer layer is seven weight matrices — W_q, W_k, W_v, W_o plus gate/up/down in the MLP. The forward pass is a chain of projections against those, and at short context the weight matmuls dominate the FLOP count; attention's score computation is a second-order term until sequence length gets large.

Each parameter participates in one MAC per token, giving the standard estimate:

This ratio has a name: arithmetic intensity — how much calculating you get per byte fetched. Every task has one. Every machine has a break-even point where its calculating and its fetching are equally busy.

Below the break-even you're waiting on memory and the chip's advertised speed is meaningless. Above it, the advertised speed is real. Generating text one word at a time sits far below it, on every machine ever made.

There's one exception that matters enormously: if the machine handles several people's requests at the same time, one fetch of the model serves all of them. Two people doubles your work-per-delivery. Three hundred people is roughly where a big datacenter chip finally becomes busy.

Intensity for decode collapses to something clean. A weight block of P params, batch B: 2·P·B FLOPs ÷ (P · bytes_per_param) bytes. At BF16 that's exactly B — arithmetic intensity equals batch size. At INT8 it's 2B, at INT4 4B: lower precision buys intensity as well as capacity, which is worth remembering when people describe quantisation as purely a memory-saving trick.

Ridge point is peak FLOPS ÷ bandwidth. The right-hand column below is the batch you'd need before the FLOPS column means anything:

Everything so far has been about running a finished model. Making one in the first place is a different kind of job entirely.

Training shows the machine enormous amounts of text and, every time it guesses the next word wrong, nudges every one of its billions of numbers very slightly in a direction that would have been less wrong. Do that trillions of times and the numbers settle into something useful.

The arithmetic is about three times heavier than running the finished model. That part is just a bigger number. The part that reshapes the hardware is what training has to keep: for every parameter, you're also storing which direction you're currently nudging it, and a running memory of how you've been nudging it recently. Several extra numbers per parameter, all of which have to be in fast memory at once.

Backprop is roughly 3× the forward FLOPs (≈6 FLOPs/param/token against 2), which is unremarkable. The structural difference is resident state.

Standard mixed-precision with Adam: BF16 weights (2 B) + FP32 master copy (4 B) + gradients (2–4 B) + first and second moments (4 B each) ≈ 16 bytes per parameter, before activations. Activation memory then scales with batch × sequence × depth, which is what activation checkpointing trades against recompute FLOPs.

That 1.1 terabytes is why training hardware looks the way it does. The numbers get split across hundreds or thousands of machines, and at the end of every single step all of them have to compare notes with all the others. The wires between machines stop being plumbing and become part of the computer.

Sharding (ZeRO/FSDP for state, tensor parallel within a node, pipeline parallel across them) puts an all-reduce on the critical path of every step. At that point interconnect bandwidth is a first-order performance term, not I/O — which is precisely why NVLink domains and TPU ICI toruses exist, and why training capacity is sold by the rack rather than the card. Module 7 has the topologies.

Here's the bridge between the two worlds, and it's a genuinely elegant idea.

Every one of those billions of numbers has to be written down in some amount of space. Write it very precisely — lots of decimal places — and it takes four bytes. Round it off a bit and it takes two. Round it aggressively and it takes half a byte.

Think of prices. "£12.3847" is precise. "£12.38" loses almost nothing useful. "£12" loses a bit more. "About a tenner" loses quite a lot. The question is how much rounding a model tolerates before it gets noticeably worse — and the surprising answer is: a great deal.

Training needs precision, because it's making tiny adjustments and tiny adjustments get lost in rounding. But once the model is finished and the numbers are frozen, you can round them hard. That process is called quantisation, and it is the single reason running AI on your own computer is possible at all.

Training needs dynamic range more than mantissa precision, because gradients span many orders of magnitude — which is exactly why BF16 displaced FP16. Identical width, but BF16 keeps FP32's 8-bit exponent and spends only 7 bits on mantissa, so it rarely overflows and generally doesn't need loss scaling.

Inference has the opposite freedom: weights are frozen, their distributions are known and measurable, so you can compress hard provided you keep a scale factor per small block to preserve local range. That's the microscaling idea behind MXFP4 and friends, and behind llama.cpp's K-quants. The practical gotcha is outlier channels — a handful of dimensions with much larger magnitudes — which is what AWQ and GPTQ-style methods exist to handle.

Where the model's numbers physically sit turns out to matter enormously, and there are three arrangements in the wild.

A discrete graphics card has its own private memory, very fast, and usually not very much of it. A plain computer has ordinary system memory: plenty of it, but slow. And then there's unified memory, where everything shares one pool that is both large and reasonably fast.

The three arrangements differ in where the DRAM sits relative to the compute die, and that placement determines both the achievable bandwidth and the practical capacity ceiling:

• Discrete GDDR — high pin-rate DRAM on a wide bus around the die. Great bandwidth per dollar; capacity limited by board area and the bus width you can route (5090: 512-bit, 32 GB).
• Host DDR — enormous capacity, but a narrow dual-channel path (~90 GB/s) and a PCIe hop to any accelerator.
• Unified LPDDR — DRAM on package, wide and short. Apple's tiers run 153–1,200 GB/s at up to 512 GB, with zero-copy between CPU, GPU and NPU because there's only one pool.
• HBM — stacked DRAM dies with TSVs on an interposer beside the compute die. Bandwidth comes from thousands of slow pins rather than hundreds of fast ones, which is also why it's more energy-efficient per bit (~4 pJ/bit for HBM3E). Capacity per stack is the constraint, hence 192 GB parts.

Think about what an ordinary processor does for a single multiply-and-add: fetch two numbers from their hiding places, multiply them, add the result to a running total, put the total back. The multiplication is nearly free. All the cost is in the fetching and the putting back.

A systolic array deletes the trips. Lay out a grid of tiny multiplying units and load a chunk of the model into them, permanently. Then pump your numbers in at the edge and let each unit hand its result to its neighbour, like a bucket brigade. One number entering the edge gets used by every unit in its row, and nothing goes back to memory in between.

A weight-stationary systolic array amortises operand fetch spatially: an N×N array performs N² MACs per cycle while consuming ~2N operands per cycle, so operand reuse scales with N rather than depending on a cache hierarchy to find it. No register file pressure, no scheduling, minimal control overhead per MAC — which is where the energy advantage over a general-purpose SIMT pipeline comes from.

Run the panel below and watch the MACs-per-read figure climb as the wavefront fills the array. That ratio is the entire justification for the architecture.

Until recently, "bigger model" always meant "proportionally slower", because every number was read for every word. Mixture-of-experts models broke that link. The model is divided into many sections and only a few of them are consulted for any given word.

So the cost splits in two: you must store all of it, but you only read a little of it per word. Storage is charged on the total; speed is charged on the part that fires.

*OpenAI's gpt-oss-120b is the clean example. It holds 116.8 billion numbers — about 61 GB, so it needs a serious machine — but only 5.1 billion are used per word. It therefore generates far faster than a traditional model of its size ever could. This, more than any hardware change, is why running capable AI at home got dramatically better in 2025–26.

Sparse MoE routing means capacity cost scales with total parameters while bandwidth cost scales with active parameters. gpt-oss-120b: 116.8 B total, 5.1 B active per token, shipped natively in MXFP4 at ~61 GB, 128k context. That's a ~23× reduction in per-token weight traffic against a dense model of equal capacity.

The systems consequence is that the binding constraint flips from bandwidth to capacity for this class of model — which is exactly the regime where unified memory's capacity advantage beats GDDR's bandwidth advantage. It also changes the arithmetic intensity picture: fewer bytes per token at constant FLOPs per active parameter, so MoE is an intensity win as well as a speed win.