# Context engineering for AI agents

> The design and runtime management of everything an LLM sees before it answers — system prompts, tools, memory, retrieval — and the discipline of giving an agent just enough of it.

- **Source:** https://ainativesoftware.engineering/pillars/context-engineering
- **Site:** AI-Native Software Engineering — https://ainativesoftware.engineering/book

- **Published:** 2026-08-08
- **Updated:** 2026-08-08
- **Reading time:** 12 minutes
- **Series:** Pillar 1 of the pillars of AI-native engineering
- **Tags:** context engineering, prompt engineering, RAG, AI agents, Model Context Protocol

![A map of the components that make up an LLM's context window](https://ainativesoftware.engineering/images/pillars/context_diagram.png)

## Why context engineering matters

### The discovery: inference-time learning

**Prompts and context act as a form of learning that happens at inference time. We can dramatically change and improve a model's output simply by carefully crafting the context we provide — without any weight updates or retraining.**

Research published in July 2025 revealed that transformer architectures have an incredible ability: they can learn new patterns on-the-fly during inference. When you provide examples or instructions in the prompt, the model can adapt its behavior to match those patterns, even if it never saw them during training.

![Diagram from the paper Learning without training, showing context converted into an implicit weight update](https://ainativesoftware.engineering/images/pillars/learning_without_training.webp)

*Learning without training — arXiv 2507.16003*

**How it works:** the combination of self-attention layers and MLP (Multi-Layer Perceptron) layers in transformer blocks allows the model to implicitly modify how it processes information based on the context. Think of it as the model temporarily "rewiring" itself to better handle the specific task you're asking it to perform, all without changing its actual weights.

This discovery, detailed in [Learning without training: the implicit dynamics of in-context learning](https://arxiv.org/abs/2507.16003), shows that transformer blocks can transform context into low-rank weight updates of the MLP layer. This is why few-shot learning and prompt engineering work so effectively.

### The reality check: context window limitations

**It is tempting to think more context equals better results. Reality is more nuanced: two limitations prevent us from simply maxing out the window.**

**Context window size limits**

Every LLM has a maximum context window — a hard limit on how many tokens it can process in a single request. Even models that claim to support millions of tokens have practical limits.

**Performance degradation**

More context doesn't always mean better results. As you add more information, the model struggles to retain and effectively use all of it, leading to decreased accuracy and quality.

![Chart showing accuracy falling well before a model's advertised context limit](https://ainativesoftware.engineering/images/pillars/context_is_what_you_need.webp)

*Context is what you need — arXiv 2509.21361*

Research from [Context is what you need: the maximum effective context window for real world limits of LLMs](https://arxiv.org/abs/2509.21361) reveals a critical finding: the **Maximum Effective Context Window (MECW)** is drastically different from the advertised Maximum Context Window (MCW).

- Some top-tier models failed with as little as **100 tokens** in context.
- Most models showed severe accuracy degradation by **1,000 tokens**.
- All models fell short of their advertised maximum by as much as **99 percent**.
- The effective window size **varies based on problem type** — what works for one task may not work for another.

This means that even if a model claims to support one million tokens, the actual effective context window where it performs well might be only a fraction of that.

### The challenge: finding the balance

**We need to simultaneously maximise context to improve performance and minimise it to maintain quality and control costs. This is where context engineering comes in.**

Unfortunately, we cannot simply max out the context window, pass all our data, and hope for better results. Beyond the performance degradation, there are practical considerations:

- **Cost:** most LLM providers charge per token. Filling a large context window can become prohibitively expensive.
- **Latency:** processing more tokens takes more time, slowing down your application.
- **Quality:** as research shows, more context often leads to worse results, not better ones.

This creates a fundamental challenge: we need to find techniques that allow us to **maximise the amount of relevant context** we provide, to improve task performance, while **minimising the total context size**, to maintain quality and control costs.

> **The definition**
>
> **Context engineering** is the art and science of retrieving, selecting and structuring the right context so that LLMs can correctly perform tasks while keeping context size under control. It is about being intentional and strategic with every piece of information we include.

## What is this "context"?

### Context: from text to tokens

**Context for an LLM is just… numbers. Specifically, tokens. We pass text to an LLM — and images, if it is multimodal — and this is converted and processed as tokens.**

From a semantic point of view, we can divide the context into multiple components which we might add or not. The interesting thing about context is how we retrieve it.

One of the biggest challenges we have at the moment while working with agents is how we can retrieve and pass to the LLM the right context, always being cautious about the limitation of the window size and the accuracy loss as soon as we bring more and more context.

The key thing to understand is that, while interacting with an LLM, the only mandatory thing is the user query. Everything else is optional, and its main goal is to provide more information to the LLM so it can give a better answer.

### System prompt

### The system prompt is the foundation layer

**It defines the identity, behaviour and boundaries of the LLM or agent. Everything else — user input, memory, tools — builds on top of it.**

A good system prompt often includes several key sections:

**Role definition**

Who the model is. Example: "You are a technical assistant specialised in software engineering."

**Goals**

What the model should achieve. Example: "Your goal is to help users write clean, efficient TypeScript code."

**Tone and style**

How the model should communicate. Example: "Use clear and simple English. Be concise and professional."

**Behavioural rules**

What to do and what not to do. Example: "Always explain your reasoning briefly before giving the answer. Do not write unsafe code."

This means the system prompt directly influences the model's reasoning and style throughout the conversation. When we design context for an LLM, the system prompt is the first and most stable part. It helps with:

- **Consistency:** all outputs follow the same logic, tone and goals.
- **Safety:** prevents the model from performing unwanted actions.
- **Efficiency:** reduces the need to repeat instructions in every user prompt.
- **Alignment:** keeps the model focused on the task or role we expect.

In short, a well-written system prompt reduces confusion, improves quality, and helps the model stay in character. Usually the system prompt is **static**: it is written in a config file and loaded into the agent every time a conversation starts.

### Available tools

### Tools are external capabilities — and a way to fetch context

**Tools are the external capabilities the model, or the agent wrapper around the model, can call upon. They expand what the model can do beyond generating text.**

A tool is a function or interface that:

- has a clear name summarising its purpose
- has a description that explains what it does
- requires a set of parameters to work
- produces a defined output
- has a schema, often in JSON, that defines what a valid call looks like

For example, in one agent framework a tool might be a web search API, or a filesystem read function. Using well-defined schemas ensures that the LLM can reliably call tools and interpret their outputs. Proper tooling helps keep the **context size manageable**: instead of stuffing everything into the prompt, we can rely on tools and retrieve information when needed.

> **The key point**
>
> Tools are both part of the context — we have to inject the tool definition — and a way to retrieve more context dynamically, directly from the agent.

### User input and user-provided context

### User input is the trigger

**User input is the immediate request or command from the user. It is the piece of context that triggers the agent's action: it tells the agent what the user wants now.**

User input can take many forms:

**Natural-language question**

"Generate unit tests for this function."

**Command**

"Search the codebase for occurrences of `TODO`."

**Specification**

"Refactor the module `auth.ts` to follow the new architecture."

**Parameterised request**

"Use library X version 5.2 to implement feature Y."

The key point is that user input is the **latest turn** in the conversation or workflow, and it tells the agent what _now_ needs to be done.

When we design the context for an agent, user input matters because:

- It defines the **task boundary**: it tells the agent what to focus on.
- It shapes the **retrieval** of relevant context: the agent must pick the right tools, memory and documents based on what the user asked.
- It is a **dynamic input**: unlike the static environment or user profile, this changes turn by turn and must be processed correctly to maintain coherence and relevance.

In some cases you end up working on the same task type over and over. When that happens the user prompt is usually similar, and only a few things change, like parameters in a function.

The tooling evolved to have **prompt templates**, which work like helper functions: you recall the template, it gets injected into the context, then you add your customisations. Examples of this are [commands in Cursor](https://cursor.com/docs/agent/chat/commands) or [prompts in MCP](https://modelcontextprotocol.info/docs/concepts/prompts/).

Apart from defining what we want to achieve, we can also pass more context to explain _how_ we want to achieve it. [Rules](https://cursor.com/docs/context/rules) are a nice example. While writing the user input we can recall and explicitly add one or more rules to the context just by tagging them with `@ruleName`. A rule is usually a Markdown file containing style guides, restrictions and so on.

Depending on the task, injecting the right rules can make the difference. In some cases it is also possible to recall a rule directly from a prompt template. Standards are emerging to do something similar, such as [AGENTS.md](https://agents.md/).

Thanks to user-provided context, the human interacting with the agent can manually supply more information and steer the agent in the right direction.

After the user starts the interaction, the agent takes over and begins the process of reasoning, planning and acting based on the given context. At this point the **LLM has a full view of the context** it can access — system prompt, environment, available tools and user input — and it uses all of this to decide _what to do next_.

## From request to action: how the flow works

![The lifecycle of a request through an agent: context assembly, planning, tool execution, reasoning loop, final answer](https://ainativesoftware.engineering/images/pillars/context_lifecycle.png)

**1. User input arrives**

The user issues a request, for example "Refactor the authentication service and add logging for failed login attempts." The agent receives this as text, which is part of the current context. This is the latest turn in the conversation, and it tells the agent what now needs to be done.

**2. Context assembly**

The orchestration layer — an agent runtime like Cursor, Claude Code or GitHub Copilot — assembles all relevant context before the model starts reasoning. The system prompt defines the agent's identity and behaviour. The environment provides static and dynamic information about the system: repo, architecture, OS, date. Rules, skills and commands are loaded from static files if relevant. The available tools, declared as JSON schemas, are included so the model knows what actions it can perform. Conversation history and memory are added for continuity. This assembled context is then passed to the model as the input window.

**3. Planning phase (context discovery)**

Once the model has all the context, it creates an internal plan: understanding what additional information it needs ("What does the auth service currently look like?"), identifying which tools to use to retrieve it, and deciding the logical order of operations — inspect, edit, test, summarise. This process is sometimes called context discovery. The model uses reasoning techniques such as chain-of-thought to figure out what it needs to know, and how to gather it efficiently.

**4. Tool execution and external calls**

The model starts using the tools defined in its context. Each execution is mediated by a protocol or API layer; one emerging standard is the Model Context Protocol (MCP), which defines how LLMs and agents discover, call and exchange data with external tools in a structured and secure way. An agent can call a filesystem tool to read code, query a database or internal API, run commands like `grep`, `build` or `test`, or query external services over HTTP or RPC. Each call returns structured output, typically JSON, which is added back into the context for the next reasoning step.

**5. Iterative reasoning loop**

After every tool call the agent evaluates the results. Did the tool return what was expected? Is more data needed? Has the task been completed? This forms an iterative loop: reason, act, observe, adjust. It continues until the agent determines the task is complete, or that no further progress can be made. Some frameworks add a feedback mechanism — from the user, or automatic validation rules — to check the output before proceeding.

**6. Producing the final answer**

Once the agent has gathered all required information and executed all necessary actions, it produces a final output. Depending on the design that might include the final artifact (the refactored code, a generated file), a summary of the steps executed, logs or reports about tool calls and test results, or next-step suggestions and validation notes. This final message is what the user sees as the result of the interaction.

**Example: coding agent workflow**

**User**

Add logging to failed login attempts in the auth service.

**Agent**

- Loads the system prompt, environment via [AGENTS.md](https://agents.md/) (Node.js v18, Express, PostgreSQL), and tool definitions.
- Analyses user input and decides to read `auth.ts`.
- Calls the `read_file` tool through MCP.
- Parses the result and identifies where to insert logging.
- Generates code for the new logging statement.
- Writes changes using the `write_file` tool.
- Runs tests with the `run_tests` tool.
- Summarises the result and returns it to the user.

Each step includes a tool call, a reasoning phase and a feedback check.

### Dynamic context retrieval

**Static context — system prompts, rules, environment configuration — gives the agent a foundation. Most real-world tasks require fresh, situational information the model can only get by interacting with its environment.**

To achieve this, agents use tools and protocols that let them fetch, explore and query data while they run. These are the main sources an agent can leverage to gather context dynamically:

**Fetch (API requests)**

One of the most common ways to retrieve data. Agents use a fetch tool or an HTTP client to send requests to APIs, microservices or backend endpoints. Responses come back as structured JSON and become part of the agent's runtime context.

**Browser interaction**

Through tools like a Playwright MCP server, the agent can interact with real web pages — clicking buttons, filling forms, reading page content. Especially useful when APIs are not available and the only way in is a web interface.

**Filesystem**

The agent can inspect local or remote files to understand what exists in a project. It can read configuration, check code structure or analyse logs, retrieving domain-specific context directly from the source.

**Terminal**

Agents can execute terminal commands in a controlled environment to gather information about the system state: `ls` to list files, `git status` to see repository changes, `npm test` to verify code quality.

**RAG (retrieval-augmented generation)**

Used when the agent needs information from large knowledge bases or document stores. The system indexes documents into vector embeddings and retrieves the most relevant chunks for a query. RAG ranges from simple document lookup to complex multi-source retrieval pipelines.

**Web search**

When the information is not available locally, agents can search the web for public data, often through specialised APIs or search tools such as Tavily. This gives access to up-to-date information beyond the training data.

**Code sandbox**

Sometimes the agent needs to write and execute a small script to compute intermediate results, transform data or inspect artifacts. Code sandboxes provide a safe, isolated runtime for running snippets, testing logic or analysing outputs without affecting the main system.

**Other local or networked resources**

Agents can access any other authorised data source on the local system or the network: internal APIs, databases, third-party services behind authentication. Standards such as OAuth 2 are often used to handle secure access tokens.

The [Model Context Protocol](https://modelcontextprotocol.io/specification/draft/basic/authorization) already supports authorization and secure resource access, making it easier to standardise how agents communicate with multiple systems.

## How do you give agents the right context?

![An agent surrounded by the context sources it can draw on](https://ainativesoftware.engineering/images/pillars/context_agent.png)

### The art of context engineering

**Think of an agent as an exceptionally capable coworker who, however, always starts each day as if it were their first.**

Without proper documentation, clear instructions and accessible resources, even the best model will struggle to perform effectively. Maintaining consistent and comprehensive context files — rules, style guides, documentation — is essential. Every time something changes in your environment or workflows, take the time to update these references so the agent stays aligned with reality.

The most durable of those files is the one that states what you actually want. That is the whole argument of [spec-driven development](https://ainativesoftware.engineering/pillars/spec-driven-development): a specification is context that outlives the session it was written for, and it is the piece an agent is least able to invent for itself.

**Documentation and maintenance**

Keep your context files up to date and well structured. Update rules, style guides and documentation whenever your environment or workflows change.

- Maintain consistent and comprehensive context files
- Update references when environment or workflows change
- Keep documentation clear and accessible

**Tool selection**

More is not always better. Provide only the necessary tools, with clear descriptions and examples. In some cases, explicitly mention which tools to use in your request.

- Provide only necessary tools, not all available ones
- Include clear descriptions and examples for each tool
- Explicitly mention tools in user input when needed

**Context cleanliness**

As conversations grow longer the context window fills up and quality degrades. Start fresh chats when tasks are completed, or when responses lose precision.

- Start new chats after task completion
- Restart mid-way through complex tasks if quality degrades
- Keep context focused and relevant

**Token optimisation**

Use compact, machine-friendly formats for large structured information. Compression formats like Toon, or custom JSON minifiers, help optimise token usage.

- Use compact formats for JSON data, logs and configs
- Consider compression formats like Toon
- Optimise token usage for large payloads

> **The golden rule**
>
> Providing the right context is not only about quantity but about intentionality. It means being deliberate in what you include and what you leave out. **Too little** makes the agent blind. **Too much** makes it distracted. **Just right** enables effective reasoning.

The goal is to give just enough information for the model to reason effectively while staying within the context window. This balance — between **precision**, **relevance** and **clarity** — is what ultimately determines how well an agent can understand and execute a user's intent.

You will not get it right first time, and you are not supposed to. Context engineering sets the agent up; the two pillars after it deal with what happens when the setup was still wrong. [Human-in-the-loop collaboration](https://ainativesoftware.engineering/pillars/human-in-the-loop) is how a person corrects the context mid-run, before a small misreading compounds into a large one. [Verification and quality gates](https://ainativesoftware.engineering/pillars/verification-and-quality-gates) is the machinery that catches the output when nobody was watching.

## Frequently asked questions

### What is context engineering?

Context engineering is the practice of retrieving, selecting and structuring the information an LLM receives — system prompt, tools, memory, retrieved documents and the user's request — so the model can perform a task correctly while keeping the total context size under control.

### How is context engineering different from prompt engineering?

Prompt engineering is about wording a single request well. Context engineering is about the whole input window an agent assembles at runtime, most of which the user never types: the system prompt, tool schemas, project rules, conversation history and anything fetched by a tool mid-task.

### Why does adding more context make an LLM worse?

Research on the Maximum Effective Context Window shows models degrade well before their advertised limit — some fail with as little as a hundred tokens of extra context, and most degrade sharply by a thousand. More context also costs more money and adds latency, so quantity works against quality on all three axes.

### How does an agent get context at runtime?

Through tools. An agent fetches APIs, reads files, runs terminal commands, drives a browser, queries a vector store, searches the web or executes code in a sandbox, and each result is added back into the context window for the next reasoning step.

---

_AI-Native Software Engineering by Alfonso Graziano (O'Reilly Media, Early Release; print edition February 2027). Every page of ainativesoftware.engineering is also served as Markdown: append `.md` to any URL. Index: https://ainativesoftware.engineering/llms.txt — whole site in one file: https://ainativesoftware.engineering/llms-full.txt._