Skip to content

engineer@aine:~/book/chapters$ cat 08_verification-and-quality-gates.md

CHAPTER 08Verification and Quality Gates

The biggest chapter in the book. Twenty-one deterministic gates, then LLM review for the judgment calls, then deployment strategies as a safety layer, then runtime and AI-powered ops. It ends with the trust ladder: how authority gets earned one reversible step at a time.

  • 66 sections
  • 21k words
  • ~93 min
  • 7 figures
key takeaways7 points
  • 01The bottleneck has moved from writing code to trusting it.
  • 02Verification is a stack of four automated layers, with a human directing the whole thing.
  • 03Deterministic gates are the floor, and they should run on everything first.
  • 04LLM review covers the judgment calls deterministic checks can't make.
  • 05After merge, the deployment itself becomes a safety layer.
  • 06Observability is the price of admission for AIOps, and the loop closes back to the start.
  • 07Two simple mechanisms decide when a human is pulled in: a path gate, then a risk scorer.
the verification stackascii

inside this chapter

  • The Bottleneck Moves
  • Why Manual Review Doesn't Scale
  • What This Chapter Promises
  • The Two Big Questions
  • Harness Validations and Self-Check Loops
  • Before Merge, Layer 1: Deterministic Guardrails
  • Linting and Formatting as a Baseline
  • Dead Code and Unused Dependencies
  • Type Checks and Compilation
  • The Test Suite
  • Mutation Testing as a Test Quality Gate
  • Property-Based Testing for Invariants
  • Static Application Security Testing (SAST)
  • Dependency and Container Scanning
  • Secrets Scanning
  • Infrastructure as Code Scanning
  • Supply-Chain Integrity
  • Architecture Fitness Functions
  • Performance and Bundle Budgets
  • Query and Data-Access Performance
  • Dependency Updates
  • Contract Gates: API and Schema Breakage
  • Accessibility and Internationalization
  • Pull-Request Size and Scope Discipline
  • Before Merge, Layer 2: LLM-Based Review
  • Why Deterministic Checks Are Not Enough
  • Adversarial Review Before the PR
  • How an LLM Reviewer Works
  • Two Ways to Organize the Review
  • What to Look For
  • Tuning the Reviewer to Your Codebase
  • After Merge, Layer 3: Safe Deployment Strategies
  • Feature Flags as the Default
  • Blue-Green Deployments
  • Canary Deployments
  • Shadow Traffic and Dark Launches
  • Progressive Delivery by User Segment
  • Automated Rollback on Error Budget Burn
  • Choosing Strategies
  • Data Safety
  • After Merge, Layer 4: Runtime Safety and AI-Powered Ops
  • The Three Pillars, Briefly
  • OpenTelemetry as the Common Substrate
  • SLOs and Error Budgets
  • Reducing Alert Fatigue
  • Where AI Changes the Game
  • AI Anomaly Detection on Metrics and Logs
  • Post-Deploy Observability Analysis
  • AI-Driven Root-Cause Analysis
  • The Bug-Resolution Pipeline
  • Closing the Loop Back to the Coding Agent
  • Synthetic Monitoring and Chaos Engineering
  • The Human in the Loop: Deciding What Needs You
  • Two Mechanisms to Decide What Needs a Human
  • The Deterministic Gate: Changes That Always Need a Human
  • The AI Risk Scorer
  • How Human Review Is Changing
  • Reviewing the Reviewer
  • Putting It Together
  • A Reference Pipeline, End to End
  • What to Adopt First
  • Metrics for the System Itself
  • Failure Modes
  • The Trust Ladder
  • Further Reading
  • Summary

figures

fig 8-1
The bottleneck has moved. Verification is now the constraint.
The bottleneck has moved. Verification is now the constraint.
fig 8-2
Four automated layers do the mechanical work; the human in the loop steers the pipeline from outside it.
Four automated layers do the mechanical work; the human in the loop steers the pipeline from outside it.
fig 8-3
Two ways to organize LLM review. Start simple; specialize where it pays off.
Two ways to organize LLM review. Start simple; specialize where it pays off.
fig 8-4
The detect-to-fix loop. A production bug reenters the same verification stack as any other change, and leaves behind a test that prevents its return.
The detect-to-fix loop. A production bug reenters the same verification stack as any other change, and leaves behind a test that prevents its return.
fig 8-5
Two mechanisms decide every PR: a deterministic path gate, then the AI scorer for everything else.
Two mechanisms decide every PR: a deterministic path gate, then the AI scorer for everything else.
fig 8-6
One change, end to end. Cheap checks run first and catch the most; expensive human attention is spent last and only where it is needed.
One change, end to end. Cheap checks run first and catch the most; expensive human attention is spent last and only where it is needed.
fig 8-7
The trust ladder. Authority is earned one reversible step at a time, never assumed.
The trust ladder. Authority is earned one reversible step at a time, never assumed.

vocabulary introduced here

man aine →

read this chapter

Chapter 8 is in the Early Release.

The code ANSE2026 gives you 30 days of free access to the O'Reilly platform, which covers this book and everything else on it.

Read it free ↗View on O'Reilly ↗