AI Failure Modes Every Engineer Must Know

Using AI tools effectively means knowing where they break. These failure modes are not edge cases. They appear regularly in everyday engineering work. Recognizing them early is a core professional skill.

Hallucination

An LLM hallucinates when it produces output that sounds plausible but is factually wrong. In code, this means:

• Invented API methods that don't exist
• Fabricated library documentation
• Made-up function signatures
• Plausible-looking but incorrect logic

Hallucination isn't a bug that will be fixed. It's an intrinsic property of next-token prediction. The model generates the most probable-looking next token, not necessarily the correct one. Mitigation: always verify against documentation, tests, and type-checkers.

Confident wrongness

Worse than hallucination is confident wrongness: the model is wrong, but shows no uncertainty. It doesn't hedge or add caveats. It states the wrong answer as fact. This is particularly dangerous for security, performance, and correctness decisions.

Mitigation: treat AI output as a first draft, not a final answer. Run the code. Check the docs. Especially for anything security-sensitive, never trust without verification.

Context drift

In long sessions or large codebases, context drift happens when the model loses track of constraints, decisions, or requirements established earlier. The agent contradicts a constraint it agreed to ten messages ago, or re-introduces a pattern it was told not to use.

Mitigation: use persistent artifacts (SPEC.md, TASKS.md) to anchor the agent's reasoning. Re-inject key constraints explicitly at the start of new sessions.

Hidden assumptions and missing context

Agents fill gaps with assumptions, and those assumptions are invisible unless you ask. A model generating a database schema makes choices about normalization, indexing, and naming, without flagging them unless prompted.

Mitigation: after getting output from an agent, ask explicitly: "What assumptions did you make? What alternatives did you consider?" This surfaces invisible decisions before they become expensive mistakes.

The snowball effect in long-running tasks

In multi-step agent tasks, a small error in step 2 can propagate through steps 3, 4, and 5. By the time you notice something is wrong, the agent has built a significant structure on a bad foundation. This is why human-in-the-loop checkpoints matter: reviewing output at each meaningful step, not just at the end.

AI security fundamentals

When agents can take actions (read files, call APIs, run terminal commands), the attack surface changes. Key risks:

• Prompt injection: A malicious input (e.g., in a file the agent reads, or a web page it visits) contains instructions that hijack the agent's behavior. The agent follows the injected instruction because it can't distinguish it from a legitimate system prompt.
• Excessive permissions: An agent with read/write/execute access to a production environment can cause irreversible damage if it misinterprets a task.
• Data exfiltration: A compromised MCP server or tool could send sensitive data to an external endpoint without the user's knowledge.

Mitigation: apply least-privilege. Give agents only the tools and permissions they need for the specific task. Review agent actions before they touch production systems. Treat tool outputs as untrusted input.