Sarah Drasner

<start of post>

💥 Just finished a new code drawing in my AI series, this one’s about how Transformers actually move information around in practice. Enjoy!

I spent a good amount of time on this one because it’s easy to memorize the words (attention, embeddings, heads) without ever getting a feel for what’s happening at each step. My goal was that someone new to the subject could come away with basic high-level understanding, but also that an experienced engineer could point at a box in the diagram and say ‘yep, that part matters’.

If you’ve only ever seen the “attention is all you need” headline version, the first surprise is that most of the work is just clean bookkeeping: taking tokens, turning them into vectors, and then doing the same few operations repeatedly, layer after layer, with slightly different learned weights.

One thing I wanted to make visually obvious is what changes across layers and what doesn’t. The tokens don’t magically become “meaning”, they become a set of numbers that are useful for predicting the next token, and the model keeps remixing those numbers through the same pattern of projections, dot products, and mixes.

The attention step is the star, but it’s also the easiest part to misunderstand if you never slow it down. You’re not “searching the internet” inside the model. You’re computing similarity between one vector and other vectors in the same context window, turning that into weights, and then using those weights to blend information forward. That’s it. Still incredibly powerful, but not mystical.

multi-head attention isn’t just “more attention”, it’s parallel subspaces that each get to specialize

masking is a constraint that shapes what can influence what, and it’s foundational for next-token prediction

the feed-forward block looks boring on paper, but it’s a huge chunk of the compute and capacity

Also, I tried to highlight where shapes matter (sequence length vs hidden size) because a lot of confusion comes from not knowing what dimension you’re multiplying by what. Once you can track the shapes, the whole thing becomes less intimidating, and you can start asking better questions about performance and memory.

If you’re coming from mobile or web performance work, this ends up feeling familiar: you’re moving and transforming big buffers of numbers, and the system is fast when you keep things batched, predictable, and hardware-friendly. That’s part of why GPU vs CPU intuition is useful here, even if you never write CUDA.

I’ve also been thinking a lot about the “practical” side lately: what you do when you want to apply this to a product. The model architecture is one piece, but the surrounding workflow (data, evaluation, guardrails, latency budgets) is where most real teams spend their time. It’s similar to WASM in that way: the tech is exciting, but the leverage shows up when it fits into a real pipeline and stays maintainable.

If you read this and think “ok, but when would I ever need to know this level of detail?”, you probably don’t for day-to-day API usage. But it’s extremely helpful when you’re debugging odd outputs, trying to understand why a model is slow, or evaluating tradeoffs between context length, model size, and quality.

I’d love to hear what’s been most confusing (or most surprising) as you’ve learned this stuff.

https://sarah.dev/projects

<end of post>