How Large Language Models Work

Before you can use AI tools well, you need a working model of what's happening under the hood. You don't need a PhD, but you do need enough to reason about outputs, diagnose failures, and set realistic expectations.

Tokens, not words

LLMs don't read text the way you do. They operate on tokens: chunks of characters, roughly 3–4 characters each. "engineering" is one token; "unbelievable" might be two. This matters because:

• Context windows are measured in tokens, not words
• Rare words and code symbols cost more tokens than common English
• Long inputs compress a lot of meaning into a limited budget

The context window

Every LLM has a context window: the maximum number of tokens it can process in one call. Everything outside that window is invisible to the model. Modern frontier models have large windows (100K–1M tokens), but they still degrade: attention is not uniform. Content at the very beginning and very end of the window tends to receive more attention than content buried in the middle.

Practical implication: When you give an agent a large codebase, the model may effectively ignore files that land in the middle of a long context. Structuring your context carefully matters.

Attention and next-token prediction

The core mechanism of a Transformer is attention: each token attends to every other token in the context and learns which ones are relevant. This is why LLMs can track variable names across a file, complete function signatures, and follow complex logic.

At inference time, the model does one thing: predict the next token, then the next, then the next. It doesn't plan ahead the way a human might. This explains why long outputs can drift or contradict themselves; there is no global planner revising the whole answer.

Why LLMs are good at code

Code has properties that make it a strong fit for next-token prediction:

• Structure is enforced: Syntax, indentation, brackets: errors are unambiguous
• Training signal is strong: GitHub and open-source repos provide billions of correct examples
• Verifiability: Code either runs or it doesn't. Models trained with Reinforcement Learning from Human Feedback (RLHF) and RLVR learn to produce outputs that pass tests, not just look plausible
• Repetitive patterns: Boilerplate is highly predictable. Glue code, CRUD endpoints, test setups: models have seen thousands of near-identical examples

Where LLMs still fail

• Novel reasoning: Anything that requires chaining together concepts the model hasn't seen combined before
• Long-horizon consistency: Models are evolving fast, but they still struggle to build long-horizon plans and to follow them consistently
• Precise arithmetic and counting: Next-token prediction doesn't naturally compose to reliable math
• Up-to-date knowledge: Training data has a cutoff; anything after that is unknown unless injected into context
• Self-knowledge: Models are often confidently wrong about what they can and can't do