# AI Failure Modes Every Engineer Must Know

> Hallucination, confident wrongness, context drift, and the foundational security risks that arise when AI agents can take actions in the world.

- **Source:** https://ainativesoftware.engineering/roadmap/day-2/ai-failure-modes-every-engineer-must-know
- **Site:** AI-Native Software Engineering — https://ainativesoftware.engineering/book

- **Day:** 2 · position 6 of 7
- **Reading time:** 3 minutes
- **Day overview:** [Day 2](https://ainativesoftware.engineering/roadmap/day-2.md)

Hallucination, confident wrongness, context drift, and the foundational security risks that arise when AI agents can take actions in the world.

## Theory

Using AI tools effectively means knowing where they break. These failure modes are not edge cases. They appear regularly in everyday engineering work. Recognizing them early is a core professional skill.

### Hallucination

An LLM **hallucinates** when it produces output that sounds plausible but is factually wrong. In code, this means:

- Invented API methods that don't exist
- Fabricated library documentation
- Made-up function signatures
- Plausible-looking but incorrect logic

Hallucination isn't a bug that will be fixed. It's an intrinsic property of next-token prediction. The model generates the most probable-looking next token, not necessarily the correct one. Mitigation: always verify against documentation, tests, and type-checkers.

### Confident wrongness

Worse than hallucination is **confident wrongness**: the model is wrong, but shows no uncertainty. It doesn't hedge or add caveats. It states the wrong answer as fact. This is particularly dangerous for security, performance, and correctness decisions.

Mitigation: treat AI output as a first draft, not a final answer. Run the code. Check the docs. Especially for anything security-sensitive, never trust without verification.

### Context drift

In long sessions or large codebases, **context drift** happens when the model loses track of constraints, decisions, or requirements established earlier. The agent contradicts a constraint it agreed to ten messages ago, or re-introduces a pattern it was told not to use.

Mitigation: use persistent artifacts (SPEC.md, TASKS.md) to anchor the agent's reasoning. Re-inject key constraints explicitly at the start of new sessions.

### Hidden assumptions and missing context

Agents fill gaps with assumptions, and those assumptions are invisible unless you ask. A model generating a database schema makes choices about normalization, indexing, and naming, without flagging them unless prompted.

Mitigation: after getting output from an agent, ask explicitly: "What assumptions did you make? What alternatives did you consider?" This surfaces invisible decisions before they become expensive mistakes.

### The snowball effect in long-running tasks

In multi-step agent tasks, a small error in step 2 can propagate through steps 3, 4, and 5. By the time you notice something is wrong, the agent has built a significant structure on a bad foundation. This is why **human-in-the-loop** checkpoints matter: reviewing output at each meaningful step, not just at the end.

### AI security fundamentals

When agents can take actions (read files, call APIs, run terminal commands), the attack surface changes. Key risks:

- **Prompt injection**: A malicious input (e.g., in a file the agent reads, or a web page it visits) contains instructions that hijack the agent's behavior. The agent follows the injected instruction because it can't distinguish it from a legitimate system prompt.
- **Excessive permissions**: An agent with read/write/execute access to a production environment can cause irreversible damage if it misinterprets a task.
- **Data exfiltration**: A compromised MCP server or tool could send sensitive data to an external endpoint without the user's knowledge.

Mitigation: apply least-privilege. Give agents only the tools and permissions they need for the specific task. Review agent actions before they touch production systems. Treat tool outputs as untrusted input.

**Theory resources**

- [OWASP Top 10 for Agentic Applications](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/)
- [MCP Security Issues Threatening AI Infrastructure – Docker](https://www.docker.com/blog/mcp-security-issues-threatening-ai-infrastructure/)

## Practice

Run this in a repository you already know, not a toy project.

1. **Trigger a hallucination intentionally.** Ask your AI tool about a fictional npm package (e.g., `npm install @acme/superutils`) and see if it describes it confidently. Or ask it to document a method that doesn't exist in a real library you know well. Observe the confidence level in the response.
2. **Test context drift.** Start a long session: establish a clear constraint early (e.g., "never use class components, only functional components"). After 10+ turns of unrelated work, ask it to generate a new component and check whether the constraint was respected.
3. **Identify hidden assumptions.** Take any non-trivial piece of AI-generated code and ask: "What assumptions did you make in this implementation? List them explicitly." Review the list: are any of them wrong for your system?
4. Reflect: which of these failure modes have you encountered before without recognizing it as such? What would you change about your current workflow to catch them earlier?

- **Previous topic:** [Memory and State in Agent Workflows](https://ainativesoftware.engineering/roadmap/day-2/memory-and-state-in-agent-workflows.md)
- **Next topic:** [Human in the Loop](https://ainativesoftware.engineering/roadmap/day-2/human-in-the-loop.md)

---

_AI-Native Software Engineering by Alfonso Graziano (O'Reilly Media, Early Release; print edition February 2027). Every page of ainativesoftware.engineering is also served as Markdown: append `.md` to any URL. Index: https://ainativesoftware.engineering/llms.txt — whole site in one file: https://ainativesoftware.engineering/llms-full.txt._