Theory
One of the most common sources of agent failure is memory, or the lack of it. Understanding how agents manage state helps you design workflows that stay coherent across long tasks.
The context window as working memory
The simplest form of agent memory is the context window itself: everything in the current conversation, including past tool calls and results. This is fast and requires no external system, but it's bounded. Once the window fills up, something has to give.
Types of agent memory
Short-term (in-context) memory The active conversation or task thread. The agent "remembers" everything that fits in the window. When the session ends or the window overflows, this memory is lost.
Long-term (persistent) memory Information stored outside the model: files, databases, vector stores. The agent retrieves relevant pieces on demand rather than keeping everything in the window at once. This is how agents work with large codebases: they read only what's relevant to the current step.
Retrieved memory (RAG) Retrieval-Augmented Generation: the agent embeds a query, searches a vector index of past notes, docs, or code, and injects the top results into context. This lets agents work with knowledge bases far larger than any context window.
Agent-generated memory Some agent frameworks let agents write their own notes as they work: summaries, decisions, open questions, and re-read them later. This is a key technique for long-running tasks that span multiple sessions.
Context rot
Context rot is what happens when a long conversation accumulates noise: stale assumptions, superseded decisions, failed attempts. The agent starts reasoning from an increasingly unreliable history. Symptoms: repeated mistakes, contradicting earlier decisions, losing track of the goal.
How to prevent it:
- Keep tasks scoped. One agent session per task, not one session for the whole project.
- Start fresh sessions when a task is complete. Bring only what's necessary into the next.
- Use files (SPEC.md, PLAN.md, TASKS.md) as the persistent source of truth instead of relying on conversation history.
- Summarize and compress periodically if your tool supports it.
Memory best practices for engineers
- Prefer files over chat history for persistent state. Files survive session resets and are readable by humans and agents alike.
- Be selective. Not everything needs to be in context. More context is not always better: it increases cost, slows inference, and can dilute attention.
- Name things clearly. Agents retrieve information by semantic similarity. Descriptive file names, clear section headings, and explicit variable names help agents find what they need.
Practice
Run this in a repository you already know, not a toy project. The point is to feel where the practice helps and where it gets in the way on code that has history.
- Run an agent task that takes at least 5–10 steps (e.g., implement a small feature end-to-end).
- At the end of the session, inspect the conversation history. How much of it is still relevant to the final state of the code? Estimate the "noise ratio."
- Now design the same task differently: write a SPEC.md before you start, point the agent to it at the beginning of each step, and discard stale conversation history between steps.
- Compare the two approaches. Did the structured approach produce more consistent results? Did the agent need fewer corrections?