# How Large Language Models Work

> Tokens, context windows, attention mechanisms, and why LLMs are surprisingly good at code, and where they still fail.

- **Source:** https://ainativesoftware.engineering/roadmap/day-2/how-large-language-models-work
- **Site:** AI-Native Software Engineering — https://ainativesoftware.engineering/book

- **Day:** 2 · position 1 of 7
- **Reading time:** 3 minutes
- **Day overview:** [Day 2](https://ainativesoftware.engineering/roadmap/day-2.md)

Tokens, context windows, attention mechanisms, and why LLMs are surprisingly good at code, and where they still fail.

## Theory

Before you can use AI tools well, you need a working model of what's happening under the hood. You don't need a PhD, but you do need enough to reason about outputs, diagnose failures, and set realistic expectations.

### Tokens, not words

LLMs don't read text the way you do. They operate on **tokens**: chunks of characters, roughly 3–4 characters each. "engineering" is one token; "unbelievable" might be two. This matters because:

- Context windows are measured in tokens, not words
- Rare words and code symbols cost more tokens than common English
- Long inputs compress a lot of meaning into a limited budget

### The context window

Every LLM has a **context window**: the maximum number of tokens it can process in one call. Everything outside that window is invisible to the model. Modern frontier models have large windows (100K–1M tokens), but they still degrade: attention is not uniform. Content at the very beginning and very end of the window tends to receive more attention than content buried in the middle.

**Practical implication:** When you give an agent a large codebase, the model may effectively ignore files that land in the middle of a long context. Structuring your context carefully matters.

### Attention and next-token prediction

The core mechanism of a Transformer is **attention**: each token attends to every other token in the context and learns which ones are relevant. This is why LLMs can track variable names across a file, complete function signatures, and follow complex logic.

At inference time, the model does one thing: **predict the next token**, then the next, then the next. It doesn't plan ahead the way a human might. This explains why long outputs can drift or contradict themselves; there is no global planner revising the whole answer.

### Why LLMs are good at code

Code has properties that make it a strong fit for next-token prediction:

- **Structure is enforced**: Syntax, indentation, brackets: errors are unambiguous
- **Training signal is strong**: GitHub and open-source repos provide billions of correct examples
- **Verifiability**: Code either runs or it doesn't. Models trained with Reinforcement Learning from Human Feedback (RLHF) and RLVR learn to produce outputs that pass tests, not just look plausible
- **Repetitive patterns**: Boilerplate is highly predictable. Glue code, CRUD endpoints, test setups: models have seen thousands of near-identical examples

### Where LLMs still fail

- **Novel reasoning**: Anything that requires chaining together concepts the model hasn't seen combined before
- **Long-horizon consistency**: Models are evolving fast, but they still struggle to build long-horizon plans and to follow them consistently
- **Precise arithmetic and counting**: Next-token prediction doesn't naturally compose to reliable math
- **Up-to-date knowledge**: Training data has a cutoff; anything after that is unknown unless injected into context
- **Self-knowledge**: Models are often confidently wrong about what they can and can't do

**Theory resources**

- [Intro to Large Language Models](https://www.youtube.com/watch?v=zjkBMFhNj_g)
- [What Are LLMs? – Hugging Face Agents Course](https://huggingface.co/learn/agents-course/en/unit1/what-are-llms)

## Practice

Run this in a repository you already know, not a toy project.

Spend 20–30 minutes on [Artificial Analysis](https://artificialanalysis.ai/), an independent benchmarking site that tracks and compares frontier models across intelligence, speed, cost, and other dimensions.

**What to explore:**
1. Look at the **Intelligence Index** leaderboard. Which models are currently at the top? Are there any surprises?
2. Pick two or three models you've heard of and compare them on: intelligence score, output speed, and price per million tokens.
3. Check the **Intelligence vs. Cost** chart. Which models sit in the "most attractive" quadrant? What tradeoffs do you see between raw capability and cost?
4. Look at one benchmark in detail (e.g. coding, reasoning, or instruction following). Does the ranking match your intuition from using these models day-to-day?

**Reflect:**
- For the typical tasks you use AI for (code generation, review, explanation), which model looks like the best fit based on what you've just seen?
- Why might a team choose a cheaper, slightly less capable model over the top-ranked one?
- How often do you think these rankings shift? What does that tell you about locking in to one model?

**Practice resources**

- [Artificial Analysis – Independent AI Model Benchmarking](https://artificialanalysis.ai/)

- **Previous topic:** [Why This Matters Now: Career and Relevance](https://ainativesoftware.engineering/roadmap/day-1/why-this-matters-now-career-and-relevance.md)
- **Next topic:** [Prompt Engineering for Engineers](https://ainativesoftware.engineering/roadmap/day-2/prompt-engineering-for-engineers.md)

---

_AI-Native Software Engineering by Alfonso Graziano (O'Reilly Media, Early Release; print edition February 2027). Every page of ainativesoftware.engineering is also served as Markdown: append `.md` to any URL. Index: https://ainativesoftware.engineering/llms.txt — whole site in one file: https://ainativesoftware.engineering/llms-full.txt._