nousresearch/hermes-agent

AI-Native Readiness Report

Repo: nousresearch/hermes-agent, a fresh clone of main at commit 17d6a7d (2026-08-14) Date: 2026-08-14 Stack: Python 3.11 (primary — pyproject.toml, uv + setuptools, ~2,910 pytest files) plus TypeScript across four npm workspaces (apps/desktop Electron, ui-tui Ink, web Vite dashboard, website Docusaurus). Checks: pytest via scripts/run_tests.sh, vitest, ruff, ty, eslint, prettier. CI: GitHub Actions (27 workflows behind one ci.yml orchestrator). Packaging: Nix flake, Dockerfile, scripts/install.sh.

Result: 33 passed, 13 failed, 0 not applicable

How to read this. Every item is a numbered question about this repo, answered from evidence in this repo. Proof is what was actually found — a path, a file, a command and its output. An item with no proof is not an answer. The italic line under each question says how to check it; it stays in the report so the next run checks the same way.

FAIL vs N/A. FAIL means the practice would help here and it is missing — including when a whole artifact is missing, so several questions about it fail together with a one-line proof pointing at the item that found it missing. N/A means the question could never apply to a project of this kind, and the proof says why. "It does not exist" is a FAIL, never an N/A.

Priority is fixed by the template, not by the person filling it in. High means an agent working here will hit this on day one, or the risk is real. Low means it makes things better but nothing breaks without it.

1. Context

• 1. Does an agent instruction file exist (AGENTS.md, CLAUDE.md, .cursorrules, .github/copilot-instructions.md, or this ecosystem's equivalent), and what does it point at? Look for a file whose purpose is to be loaded into a coding agent's context at the start of every session. That file is the entry point. Then follow its pointers: instructions are commonly spread across several files, with the entry point routing to them — "commands are in docs/development.md", "conventions in .claude/rules/". The entry point plus every file it points at is the instruction set, and every other agent-instruction question is answered against that whole set, so list it here in the proof. Follow pointers as deep as they go, but a pointer only counts when it names a target and says what is in it: "test commands are in CONTRIBUTING.md" counts, a bare "see the docs" does not, and a file nothing points at is outside the set however good it is. A README is not an entry point, however good — it gets its credit at the README item and the specs-and-architecture item — but a README the entry point explicitly routes to is inside the set. Vendored skill libraries are not instructions either.

  • Status: PASS
  • Proof: Entry point is AGENTS.md at the root (1,543 lines, 81,342 bytes), opening "Instructions for AI coding assistants and developers working on the hermes-agent codebase." No CLAUDE.md, .cursorrules or .github/copilot-instructions.md exists (git ls-files | grep -iE 'AGENTS.md|CLAUDE.md|cursorrules|copilot-instructions' returns only AGENTS.md and apps/desktop/AGENTS.md). Instruction set = AGENTS.md + the files it names with content: apps/desktop/AGENTS.md (routed at AGENTS.md:533, "read apps/desktop/AGENTS.md"), gateway/platforms/ADDING_A_PLATFORM.md (AGENTS.md:287), website/docs/developer-guide/plugins/index.md (AGENTS.md:803), website/docs/developer-guide/model-provider-plugin.md, website/docs/user-guide/features/curator.md, website/docs/user-guide/features/kanban.md. All six exist. CONTRIBUTING.md and README.md are strong but nothing in the set routes to them, so they are outside it.
  • Recommendation: —
  • Priority: High

• 2. Do the agent instructions name the commands to build, test and check this project? Answer this against the instruction set mapped at the entry-point item and nothing outside it. If no entry point exists, FAIL with a one-line proof pointing at that item. The commands may sit in a file the entry point routes to rather than in the entry point itself — that is progressive disclosure working as intended, and it passes. What fails is a command an agent would have to guess its way to: if the commands live only in the README or the manifest and nothing in the set points at them, that is a FAIL, because the question measures what an agent can reach without being told where to look. Name the file each command was found in.

  • Status: FAIL
  • Proof: Test and build commands are in the set: AGENTS.md:1352-1367 gives scripts/run_tests.sh with four invocations and forbids bare pytest; AGENTS.md:504-517 gives the TUI's npm run dev/build/typecheck/lint/fmt/test; AGENTS.md:547 gives npx vitest run src/lib/desktop-slash-commands.test.ts. The Python check commands are absent: grep -ci ruff returns 0 for AGENTS.md, CONTRIBUTING.md, apps/desktop/AGENTS.md, gateway/platforms/ADDING_A_PLATFORM.md and website/docs/developer-guide/plugins/index.md, and ty appears nowhere either. ruff check . is a blocking required job (.github/workflows/lint.yml:128-152, "No --exit-zero, no || true"), so an agent can only learn the command that gates its merge by reading the CI YAML.
  • Recommendation: Add a three-line block to the AGENTS.md "Testing" section: ruff check . (blocking, rules in pyproject.toml), ty check (advisory), python scripts/check-windows-footguns.py --all (blocking), and npm run check at the root for the JS workspaces.
  • Priority: High

• 3. Does everything the agent instructions name still exist — the commands, the paths, the libraries, and the files they point at? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. Verify, do not trust: check every named command against the manifest or build file, spot-check the paths, run the cheap read-only ones. Then resolve every pointer in every file of the set — a link to a moved or deleted file is the most common rot in a multi-file instruction set, and it fails silently: the agent reads the entry point, follows nothing, and carries on without the rules.

  • Status: FAIL
  • Proof: Commands verified live: scripts/run_tests.sh tests/hermes_cli/test_profiles.py tests/test_hermes_state.py -q ran 276 tests, 0 failed, in 6.4s. Paths spot-checked and present: tests/conftest.py:538 _isolate_hermes_home, toolsets.py:31 _HERMES_CORE_TOOLS, hermes_cli/curses_ui.py, tools/registry.py, tools/delegate_tool.py, scripts/ci/classify_changes.py. Of 7 file pointers, 6 resolve; references/new-skill-pr-salvage.md (AGENTS.md:1029-1032, "load it before polishing contributor skill PRs") does not — it lives in a hermes-agent-dev skill that is not in this tree, and no references/ directory exists. One stale figure: AGENTS.md:308 says "~17k tests across ~900 files as of May 2026"; git ls-files tests | grep -c 'test_.*\.py$' is 2,910 today.
  • Recommendation: Either vendor references/new-skill-pr-salvage.md into the repo (e.g. docs/new-skill-pr-salvage.md) and repoint AGENTS.md:1031 at it, or say plainly in that paragraph that the checklist is maintainer-only and lives outside the repo; while there, refresh the test count at AGENTS.md:308.
  • Priority: High

• 4. Are the agent instructions specific to this repo, rather than advice that would read the same in any codebase? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The test: could this be pasted into another project unchanged? "Write clean code" and "add tests for new features" would fit anywhere and count for nothing. Judge the whole set, but weigh the files differently: an entry point that is mostly a routing table is fine, even good, when what it routes to is specific — while generic filler in the entry point costs more than generic filler three hops down, because it is loaded into every session whether it is needed or not. Say which files carried the specifics.

  • Status: PASS
  • Proof: Nothing here would survive a copy-paste into another project. AGENTS.md:19-27 names the two lenses ("Per-conversation prompt caching is sacred", "the core is a narrow waist"); AGENTS.md:182-211 is a six-rung Footprint Ladder naming real subcommands (hermes webhook, hermes cron, hermes tools); AGENTS.md:1291-1348 is nine named traps with file paths and PR numbers; AGENTS.md:598-620 is a pinning table with the incidents that produced it. apps/desktop/AGENTS.md is the same in miniature ("Decide state by authority", "Switching context is a re-home, not a reboot"). The entry point is not a routing table and carries almost no filler — the one generic line is AGENTS.md:5, "Never give up on the right solution."
  • Recommendation: —
  • Priority: High

• 5. Do the agent instructions explain where things live and how this project is laid out? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The map may live in a routed-to file. Judge coverage against the real tree, not against what the files mention: if the set maps one package well but is silent about sibling packages or directories an agent would land in, that is a FAIL with the omission named. In a multi-file set, check the routing too — a layout document nothing points at is a document the agent never opens.

  • Status: FAIL
  • Proof: AGENTS.md:263-314 maps the tree well for the code an agent usually edits: 8 root modules with one-line jobs, then agent/, hermes_cli/, tools/, gateway/, plugins/, skills/, optional-skills/, ui-tui/, tui_gateway/, acp_adapter/, cron/, scripts/, website/, tests/, plus config and log locations. Whole tracked directories are absent from the set: grep -c on AGENTS.md returns 0 for nix/, evals/, locales/, native/, contributors/, docker/ and optional-mcps/, and the string "nix" appears 0 times in the whole file even though flake.nix + nix/ (15 files, including nix/checks.nix) is a supported install and packaging path and scripts/run_tests.sh tells you to "enter the Nix devShell" when no venv is found. apps/ and web/ are covered in prose (AGENTS.md:531-533).
  • Recommendation: Add rows to the AGENTS.md:269-309 tree for nix/ (flake packaging + nix/checks.nix), evals/, optional-mcps/, locales/, native/, docker/ and contributors/, one line each saying what an agent would go there for.
  • Priority: High

• 6. Do the agent instructions state the rules that are not obvious from the code — the things never to do here? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. These are the tribal-knowledge traps: the flag that must be exactly this string, the import that breaks the build, the directory that is generated and must not be edited. Rules in a routed-to file count. Two extra checks in a multi-file set: that the entry point signposts the rules clearly enough for an agent to open them before it needs them, since a trap found afterwards has already been sprung; and that the files do not contradict each other, because nothing tells the agent which one wins.

  • Status: PASS
  • Proof: AGENTS.md:1291-1348 "Known Pitfalls" is exactly this: don't hardcode ~/.hermes (use get_hermes_home(), "the source of 5 bugs fixed in PR #3575"), menu pickers must use hermes_cli/curses_ui.py, no \033[K under prompt_toolkit's patch_stdout, _last_resolved_tool_names is a process-global, no cross-tool names in schema descriptions, the gateway's two message guards, stale squash merges silently reverting fixes, tests must not write to ~/.hermes/. AGENTS.md:1204-1233 "Important Policies" adds prompt-cache invariants. Both are top-level ## headings in the entry point itself, so they are visible before they are needed, and I found no contradiction between AGENTS.md and apps/desktop/AGENTS.md.
  • Recommendation: —
  • Priority: Low

• 7. Has the agent instruction file been updated recently enough to still be true, given how active the repo is? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Compare the last commit touching the file against the repo's tempo, then spot-check two or three of its claims against the code — a recently touched file can still lie.

  • Status: PASS
  • Proof: git log -1 -- AGENTS.md is 2026-08-14 00:26, eighteen minutes before HEAD (17d6a7d, 2026-08-14 00:42) — on a repo landing ~15 commits a day, the file is being edited with the code. Three claims spot-checked against the tree and all true: _isolate_hermes_home at tests/conftest.py:538, _HERMES_CORE_TOOLS at toolsets.py:31, and scripts/run_tests.sh probing .venv then venv (it printed both paths when it found neither). The one drifted number is the self-dated test count at AGENTS.md:308, covered at the pointers item.
  • Recommendation: —
  • Priority: Low

• 8. Is the agent instruction file small enough to load into every session without crowding out the actual task? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Measure it: wc -c, and bytes divided by four is a fair token estimate. Under ~5k tokens is comfortable; past ~10k it is eating the context window.

  • Status: FAIL
  • Proof: wc -c AGENTS.md = 81,342 bytes, roughly 20k tokens at four bytes per token — double the ~10k point where the file starts eating the context window, and four times the comfortable ~5k. apps/desktop/AGENTS.md adds 11,227 bytes (~2.8k tokens) on top for desktop work. The content is good, but all of it loads on every session, including the ~250 lines of TUI/desktop slash-command internals (AGENTS.md:467-547) that a Python-only change never needs.
  • Recommendation: Split AGENTS.md into a ~600-line entry point (What Hermes Is, rubric, dev environment, project structure, testing, known pitfalls) and move the per-subsystem deep dives — TUI/desktop (467-547), skins (685-773), plugins (774-924), skills (925-1035), toolsets/delegation/curator/cron/kanban (1036-1203) — into docs/ files, leaving a one-line pointer for each that says what is inside.
  • Priority: Low

• 9. Can a fresh session pick up a half-finished task — is there a file or convention where progress, decisions and what is left get written down? This is not about the agent instruction file. Any durable place in-flight state gets written counts: a TODO.md, a plans or notes directory, task files, linked issues, a specs folder whose entries carry progress and open questions, an agent memory file or directory, a scratchpad or working-notes convention. The test is whether a fresh session could read it and know what was decided and what is left — not what the place is called. Git history alone does not count: it records what happened, not what was decided or what remains.

  • Status: PASS
  • Proof: Decisions and open questions are written into the tree, not just into git. docs/ADR.md records each decision as Status / Context / Decision (newest entry 2026-07-13, plugin manager keyed by Hermes home). docs/design/profile-builder.md carries a live "Open product decisions (need Teknium)" section plus "What already exists (reuse, don't rebuild)" and "Verification plan (when built)" — a fresh session can read it and know what is settled and what is not. docs/rfcs/plugin-config-state-bridge.md:3 records status inline ("config + state slice implemented by #64227"). The issue/PR convention is written down too: .github/PULL_REQUEST_TEMPLATE.md requires "Related Issue / Fixes #", and CONTRIBUTING.md:21-38 tells contributors to gh search issues and gh search prs first and to comment on the issue to claim larger work.
  • Recommendation: —
  • Priority: Low

• 10. Is there a README that says what this project is and how to run it? This is where a good README earns its credit. It needs three things: what the project is, how to run it, and how to check a change. Judge what is on the page, not the file's existence.

  • Status: PASS
  • Proof: README.md (17,688 bytes) does all three. What it is: lines 5-34 plus "CLI vs Messaging Quick Reference" (143-162). How to run it: "Quick Install" (35-104) with a one-line installer for Linux/macOS/WSL2/Termux and native PowerShell, "Getting Started" (105-123), and a Troubleshooting block. How to check a change: the Contributing section (217-231) gives the full loop — installer, cd "${HERMES_HOME:-$HOME/.hermes}/hermes-agent", uv pip install -e ".[all,dev]", scripts/run_tests.sh — plus a manual-clone fallback that warns why the venv must live outside the source tree.
  • Recommendation: —
  • Priority: High

2. Specs

• 11. Is the thinking behind this system written down somewhere durable — a specs, RFC, proposals, design-doc or ADR directory, or architecture notes that record not just what but why? Look for the place and for the convention: specs/, docs/adr/, rfcs/, proposals/, .specify/, a docs or design folder, architecture notes anywhere in the tree, or this ecosystem's equivalent. Judge substance, not location: "we use X because Y" is a decision, a list of technologies is not, an essay about specs is not a spec, and a docs folder of usage guides with no reasoning is a FAIL whose proof says what was in there instead. Other items are answered against whatever this item finds, so name it precisely — and where forward-looking specs and after-the-fact architecture records live in different places, name both, since a decision log cannot answer a question about acceptance criteria.

  • Status: PASS
  • Proof: Two places, both with real reasoning. After-the-fact decisions: docs/ADR.md, dated entries in Status / Context / Decision form — the 2026-07-13 entry spends ~40 lines on why a process-global plugin-manager singleton broke profile switching (sys.modules caching hermes_plugins.<slug>.<submodule>) before stating the keyed-cache decision. Forward-looking specs: docs/rfcs/ (plugin-config-state-bridge.md, 2026-07-plugin-architecture-lessons-pi-opencode.md) and docs/design/profile-builder.md. Alongside them sit contract and incident docs that also record why: docs/relay-connector-contract.md, docs/chronos-managed-cron-contract.md, docs/security/network-egress-isolation.md, docs/rca-ssl-cacert-post-git-pull.md. AGENTS.md:29-211 carries the intent layer (the rubric and the Footprint Ladder).
  • Recommendation: —
  • Priority: High

• 12. Does the specs directory hold recent entries, or is it an archive nobody has touched? If no specs directory exists, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise compare the newest entry's date against the repo's recent activity.

  • Status: PASS
  • Proof: Measured against full history after git fetch --deepen=600 (the shallow clone's boundary date would have made everything look identical): docs/ADR.md last touched 2026-08-12 19:13, docs/rfcs/plugin-config-state-bridge.md 2026-08-12 16:26, docs/design/profile-builder.md and docs/observability/README.md 2026-08-08. HEAD is 2026-08-14. Two days behind a repo that lands ~15 commits a day is live, not archived.
  • Recommendation: —
  • Priority: Low

• 13. Is there a spec template, or an SDD framework, so every spec comes out the same shape? Scaffolding can exist even where no specs directory does — look for a TEMPLATE.md, a .specify/ directory, or framework config. If neither a directory nor any scaffolding exists, FAIL.

  • Status: FAIL
  • Proof: No spec scaffolding anywhere: no TEMPLATE.md, no .specify/, no framework config (git ls-files | grep -iE 'TEMPLATE|\.specify' returns only .github/ISSUE_TEMPLATE/* and .github/PULL_REQUEST_TEMPLATE.md, which shape issues and PRs, not specs). The three newest design documents come out in three different shapes: docs/ADR.md uses Status / Context / Decision; docs/rfcs/plugin-config-state-bridge.md uses Scope / Config API / Durable state API / State vs. config / Verification contract / Related; docs/design/profile-builder.md uses Why this / What already exists / Seams / Proposed backend / Proposed frontend / Open product decisions / Verification plan.
  • Recommendation: Add docs/rfcs/TEMPLATE.md with the headings the existing RFCs already converge on — Status, Scope, Non-goals, Design, Verification contract, Related — and one line in docs/ADR.md pointing new decisions at the Status / Context / Decision shape it already uses.
  • Priority: Low

• 14. Do the specs state acceptance criteria a machine could check? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise open the two newest specs and quote a criterion: "the endpoint returns 403 for expired tokens" is checkable; "the feature works well" is not.

  • Status: PASS
  • Proof: The two newest specs both state criteria a machine can check. docs/rfcs/plugin-config-state-bridge.md:101-115: "the complete serialized state is limited to 10 MiB per plugin. A rejected update leaves the previous file untouched", "state writes use temp-file + fsync + os.replace", "State keys are 1-128 characters and may contain letters, numbers, _, -, ., or :. Path separators and .. are rejected." Its "Verification contract" (127-139) then lists the cases the implementation must cover, including "concurrent settings writes without lost siblings" and "two-profile isolation after the ambient profile changes". docs/design/profile-builder.md:138 carries a "Verification plan (when built)".
  • Recommendation: —
  • Priority: High

• 15. Open the newest spec: do its criteria go past the happy path — what happens when a step fails, and how the change gets undone? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Look for error cases, edge inputs, and a rollback or undo story, not just the success flow.

  • Status: PASS
  • Proof: The newest spec, docs/rfcs/plugin-config-state-bridge.md, goes well past the happy path. Failure cases: "Fail closed: malformed/non-object JSON is reported and never overwritten"; quota rejection; "direct global, cross-plugin, POSIX traversal, and Windows traversal rejection" (127-139). Concurrency: a sibling lock file serialising read-modify-write across threads and processes, fcntl on POSIX and msvcrt on Windows. Undo story at the data level: "A rejected update leaves the previous file untouched" and atomic temp-file + os.replace, so a failed write cannot leave a half-written state file behind.
  • Recommendation: —
  • Priority: Low

• 16. Do the specs state non-goals, so an agent knows where to stop? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Non-goals written elsewhere (a README's "what this is not" list) are worth naming in the proof, but they do not turn this into a PASS — the question is whether specs carry them.

  • Status: PASS
  • Proof: docs/rfcs/plugin-config-state-bridge.md:9-17 scopes the slice to two capabilities and then says what is deliberately out: "Config schema registration, config defaults, and the cron facade from the original RFC remain separate follow-up work. No core model tool is added." docs/design/profile-builder.md:28 has "What already exists (reuse, don't rebuild)" and 126 has "Open product decisions (need Teknium)" — both fences an agent can stop at. The repo-wide non-goals are in AGENTS.md:96-136 ("What we don't want, rejected even when well-built").
  • Recommendation: —
  • Priority: Low

• 17. Can recent shipped work be traced back to a spec? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise take the last few substantial commits or PRs and look for a reference to a spec, an issue, or a design doc in the message or description.

  • Status: PASS
  • Proof: The trace runs both ways. docs/rfcs/plugin-config-state-bridge.md:3-7 names the shipped work and its origin: "Status: config + state slice implemented by #64227", "Original design: Topher Ross (@thebizfixer), RFC PR #58542", "Concrete consumer: kanban-advanced". Recent commits carry the PR number in the subject — 17d6a7d fix(desktop): avatar misses expire after 30s instead of caching forever (#85908), 5e8d25d fix(cli): --in accepts Git Bash / MSYS-style paths on Windows (#85865) — and .github/PULL_REQUEST_TEMPLATE.md requires a "Fixes #" link. AGENTS.md:1296 traces a rule back to PR #3575 and 598-620 traces the pinning policy to #2796, #2810 and #9801.
  • Recommendation: —
  • Priority: Low

3. Verification

• 18. Does this project have an automated test suite, in whatever form this ecosystem uses? Work out this ecosystem's convention before concluding anything is missing — check the manifest, the build file, the CI config, the README. A shell script that diffs output files is a test suite. If you find one, run it and record the result.

  • Status: PASS
  • Proof: Ran it. uv sync --locked --python 3.11 --extra dev on a bare clone, then HERMES_TEST_FILE_RETRIES=0 ./scripts/run_tests.sh tests/hermes_cli/test_profiles.py tests/test_hermes_state.py -q: "2 files, 276 tests passed, 0 failed, 4 skipped (100% complete) in 6.4s (20 workers)", with a note that the windows_only tests were skipped on darwin and run on the tests-os CI lane. The suite is 2,910 pytest files under tests/ (testpaths = ["tests"] in pyproject.toml:429) plus a vitest suite per npm workspace (tests-js/, ui-tui, apps/desktop, web). I did not run the full suite — CI shards it across slices.
  • Recommendation: —
  • Priority: High

• 19. Can the test command be discovered without guessing — is it written down where an agent reads? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise check the places an agent looks: the agent instruction file, the README, the manifest's scripts or targets.

  • Status: PASS
  • Proof: AGENTS.md:1352-1367, a top-level ## Testing heading in the entry point, opens "ALWAYS use scripts/run_tests.sh — do not call pytest directly" and gives four invocations (full suite, one directory, one file with -k, pass-through flags), plus the reason: hermetic parity with CI. CONTRIBUTING.md:201-213 and README.md:231 repeat it. The JS side is at AGENTS.md:504-517 and 547.
  • Recommendation: —
  • Priority: High

• 20. Do the tests assert real behaviour, rather than asserting that a mock was called? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise open the largest test files and read the assertions: calling real code on real inputs passes; expect(mock).toHaveBeenCalled() as the main dish fails.

  • Status: PASS
  • Proof: Opened the largest suites. tests/test_hermes_state.py (4,763 lines) drives a real SessionDB against a real SQLite file and asserts behaviour — assert [match["session_id"] for match in base_matches] == [...] on FTS5 search results, assert session["model"] == "test-model" after a round trip, assert list(tmp_path.glob("*malformed-backup*")) after corruption recovery — and subclasses sqlite3.Cursor to simulate a build without the fts5 module rather than mocking the call. Ratio across the suite: 65,089 assert lines, with assert_called/assert_has_calls appearing in 362 files out of 2,910. The rules are written down too: AGENTS.md:1441-1489 bans change-detector tests and 1490-1543 bans tests that read source text, both with worked before/after examples.
  • Recommendation: —
  • Priority: High

• 21. Is there a linter or static analysis configured for this language, and does it pass on a clean checkout? Configured is not enough — run it. A linter that exits non-zero on an untouched checkout is a FAIL with the error count in the proof, because an agent cannot tell its own damage from the baseline noise.

  • Status: PASS
  • Proof: Ran it on the clean checkout: uvx ruff@latest check . → "All checks passed!", exit 0, with one warning about an invalid # noqa directive at run_agent.py:108. It is enforced: .github/workflows/lint.yml:148-152 runs ruff check . as a blocking required job ("No --exit-zero, no || true"). Scope is deliberately narrow — pyproject.toml:454-462 selects exactly one rule, PLW1514, with a comment explaining that all others are off "while we wrangle typechecks" and that this one stays because bare open() in text mode corrupts non-ASCII on Windows. Two more static gates back it up: scripts/check-windows-footguns.py --all (blocking) and eslint with typescript-eslint, perfectionist, react-hooks and unused-imports per workspace. I did not run eslint locally — it needs npm ci across 11 workspaces.
  • Recommendation: —
  • Priority: High

• 22. Is there a formatter, so an agent's diffs do not churn on style? Look for the config file and the dependency in this ecosystem's form — .prettierrc, rustfmt, gofmt, black, an .editorconfig doing real work. If the language ships one formatting standard with the toolchain, that is a PASS and the proof says so.

  • Status: FAIL
  • Proof: The TypeScript half is covered: .prettierrc (printWidth 120, no semicolons, single quotes), .prettierignore, an fmt script in each workspace, npm run fix at the root, and merges that run it (bee9ef3 fmt(js): npm run fix on merge). The Python half — the primary language, ~2,910 test files plus the whole core — has no formatter: pyproject.toml has [tool.ruff] and [tool.ruff.lint] but no [tool.ruff.format], there is no black or blue in the dev extras, and there is no .editorconfig in the tree.
  • Recommendation: Add [tool.ruff.format] to pyproject.toml, run ruff format . once as a single mechanical commit, then add ruff format --check . to the existing blocking ruff job at .github/workflows/lint.yml:128.
  • Priority: Low

• 23. Is there a compile-time or type-level gate, if this language offers one? N/A only when the language genuinely has no such gate. If the language offers one and the repo does not use it — no strict mode, no typecheck script, no compiler step — that is a FAIL. Run the gate if it exists and record the result.

  • Status: FAIL
  • Proof: A type checker is configured and run, but it does not gate anything. pyproject.toml:444-449 configures Astral's ty (python-version = "3.13", two rule overrides) and .github/workflows/lint.yml:31-127 runs it on HEAD and on the base ref, then feeds both into scripts/lint_diff.py to write a summary — the step ends || true and only appends to $GITHUB_STEP_SUMMARY, while the sibling job comment states plainly that only ruff is "blocking". I ran uvx ty@latest check: Found 14754 diagnostics on an untouched checkout, which is why it cannot block. The TypeScript side does have a real gate (typecheck: tsc --noEmit inside each workspace's check script).
  • Recommendation: Make the existing base-vs-HEAD ty comparison in .github/workflows/lint.yml fail the job when HEAD introduces diagnostics the base did not have — the two JSON reports are already produced, so this is a change to scripts/lint_diff.py's exit code, not new infrastructure.
  • Priority: Low

• 24. Can an agent prove its own work before it pushes — one command, task-runner target or commit hook that runs every check this project has? One command, not a list to remember: a check or verify target, a precommit script, a Makefile target that chains them. Separate commands documented side by side are close but FAIL — the question is whether the agent can run the whole gauntlet without knowing its parts.

  • Status: FAIL
  • Proof: There is no one command. An agent must run at least four: scripts/run_tests.sh (pytest), ruff check ., python scripts/check-windows-footguns.py --all, and npm run check at the root (which is npm run --ws check, JS workspaces only). package.json:25 defines check but it never touches Python; there is no Makefile, justfile or Taskfile (ls | grep -iE 'makefile|justfile|taskfile' is empty); nix/checks.nix defines nix flake check but it verifies packaging (cross-eval, build-package, build-devshell, package-contents), not the test suite or the linters, and the string "nix" appears 0 times in AGENTS.md, CONTRIBUTING.md and README.md, so an agent would never find it.
  • Recommendation: Add scripts/check.sh that runs ruff check ., python scripts/check-windows-footguns.py --all, scripts/run_tests.sh and npm run check in sequence with a non-zero exit on the first failure, and name it in the AGENTS.md "Testing" section as the pre-push command.
  • Priority: High

• 25. Is there continuous integration, and does it run before a change lands rather than after? A deploy pipeline that runs on push to the default branch runs after the change has landed — that is a FAIL with the trigger quoted, not a PASS with a caveat. Look for a pre-merge trigger: pull_request events, merge trains, or this forge's equivalent.

  • Status: PASS
  • Proof: .github/workflows/ci.yml:17-20 triggers on pull_request first and push: branches: [main] second, so every change is checked before it lands. It is an orchestrator: a detect job classifies the diff (scripts/ci/classify_changes.py, failing open on push so post-merge validation is never weakened), then calls the sub-workflows by workflow_call, and "a final all-checks-pass gate job aggregates results so branch protection only needs to require a single check". Concurrency cancels in-progress runs on PRs only.
  • Recommendation: —
  • Priority: High

• 26. Does CI actually run the tests and the checks that exist in this repo? If no pipeline of any kind exists, FAIL with a one-line proof pointing at the CI item. Otherwise list what the pipeline runs and diff it against every check found in the Verification section — name each check that exists in the repo but is missing from the pipeline.

  • Status: PASS
  • Proof: Every check that exists in the repo has a CI lane. tests.yml shards pytest across slices with a duration cache (uv sync --locked --python 3.11 --extra all --extra dev ...); tests-os.yml runs the macOS and Windows lanes off scripts/ci/list_os_marked_tests.py; js-tests.yml enumerates the npm workspaces and runs each one's check script (typecheck + vitest + eslint) after npm ci; lint.yml runs blocking ruff, advisory ty, and the blocking Windows-footgun checker; osv-scanner.yml, supply-chain-audit.yml, uv-lockfile-check.yml, lockfile-diff.yml, docker-lint.yml, docker.yml, docs-site-checks.yml, installer-tests.yml, install-e2e.yml, e2e-desktop.yml, skills-index-freshness.yml, contributor-check.yml and history-check.yml cover the rest. The one gap is documented rather than hidden: AGENTS.md:1391-1401 warns that the change classifier means a Python test asserting about JS artifacts will not run on a JS-only PR, and tells you to put such tests in the vitest suite instead.
  • Recommendation: —
  • Priority: High

• 27. Does anything here get an LLM to read a change and go looking for problems — a review skill, a saved review prompt, or an AI reviewer on the pull requests? Look for a committed review skill or slash command, a CI step calling an AI reviewer, or bot config. Human review rules are a Safety item; this one is specifically about machine review.

  • Status: FAIL
  • Proof: Nothing gets an LLM to read a diff here. .github/workflows/ci-review-comment.yml sounds like it but is not: scripts/ci/live_comment.py "Polls the GitHub Actions API for job statuses in the CI run, assembles the review comment from whatever results are available, and upserts it as a PR comment" — a status aggregator, and grep -rilE 'openrouter|anthropic|openai|llm' scripts/ci/*.py matches only publish_e2e_evidence.py. supply-chain-audit.yml:68-115 is grep over the diff for .pth files and base64+exec, not a model. AGENTS.md:36-41 describes an automated "triage sweeper" that closes PRs on three allowed reasons, but git grep -ril sweeper -- .github scripts returns nothing, so whatever runs it lives outside this repo. There is no .claude/commands, no committed review skill, and no AI reviewer bot config.
  • Recommendation: Add a review skill or slash command under .claude/skills/ (or a prompts/ directory) that encodes the checks AGENTS.md:29-211 already spells out — verify the premise on current main, no change-detector tests, no source-reading tests, prompt-cache safety, Footprint Ladder rung — so a contributor's agent can self-review before opening the PR.
  • Priority: Low

4. Tooling

• 28. Can an agent reach the outside systems this project depends on, with that access committed to the repo rather than set up per laptop? MCP servers are one form: look for a committed .mcp.json or this ecosystem's equivalent. A CLI is another and often the better one — aws, gh, psql, kubectl, stripe, a vendor's own tool — and it counts when the repo names which tools the work needs and how to authenticate, so an agent is not guessing at a tool it cannot see. Judge the axis, not the mechanism: access that exists only in someone's shell history or laptop config is a FAIL, and so is a committed config that turns out to be gitignored, with the distinction in the proof. Project task scripts have their own item; this one is about reaching past the repo's edge. If this project genuinely talks to nothing outside itself, N/A with that as the reason.

  • Status: PASS
  • Proof: Access is committed, not per-laptop. .env.example (tracked) names every external service the project talks to and where to get each credential (Fireworks, OpenRouter, Novita and ~20 more), with the real .env gitignored. CONTRIBUTING.md:21-38 names the CLI for the dev loop and how to use it: gh search issues --repo NousResearch/hermes-agent "<terms>" and gh search prs --repo ... --state all. optional-mcps/*/manifest.yaml commits five MCP server manifests (comfy-cloud, figma, linear, n8n, unreal-engine) and hermes_cli/mcp_catalog.py + mcp_config.py + mcp_security.py generate the runtime config. Nothing important is reachable only from someone's shell history. Gap worth naming: there is no committed .mcp.json for an agent working on this repo, so a coding agent gets no MCP access of its own.
  • Recommendation: —
  • Priority: Low

• 29. Is there a skills, commands or reusable-prompt library in the repo? Look for .claude/skills, .agents/skills, .claude/commands, or a prompts directory. Committed and pinned beats committed; note whether anything ties the copies to a source.

  • Status: PASS
  • Proof: skills/ holds 485 tracked files across 15 categories (software-development, devops, github, research, …) as SKILL.md prompt bundles, optional-skills/ holds 550 more, and the library is maintained rather than dumped: scripts/build_skills_index.py builds an index, .github/workflows/skills-index.yml and skills-index-freshness.yml keep it current in CI, and AGENTS.md:925-1035 sets authoring standards ("Skill authoring standards (HARDLINE)", frontmatter contract). The honest caveat: these are the product's skills shipped to Hermes users, not a harness for agents working on this repo — the dev-facing hermes-agent-dev skill referenced at AGENTS.md:1030 is not in the tree.
  • Recommendation: —
  • Priority: Low

• 30. Do the skills, servers and tools that exist cover the work this team plainly repeats? First identify the repeated work from the README and the commit history. If skills, servers or documented tooling exist but miss it, FAIL naming the gap. If none exist at all and the repo plainly repeats work, FAIL pointing at the outside-systems item and the skills-library item. If the project is too small to repeat anything, N/A.

  • Status: PASS
  • Proof: The repeated work is visible in the commit history — platform adapters, model providers, plugins, skills, desktop/TUI features, releases — and each has a written recipe or a script. Adding a platform: gateway/platforms/ADDING_A_PLATFORM.md. Adding a tool: AGENTS.md:548-597. Adding a slash command: AGENTS.md:434-466. Adding a skin: AGENTS.md:730-773. Plugins and memory/model providers: AGENTS.md:774-924 plus website/docs/developer-guide/model-provider-plugin.md. Releases: scripts/release.py (CalVer + changelog). Contributor attribution, an actually repeated chore here: scripts/add_contributor.py, scripts/contributor_audit.py, scripts/audit_pr_attribution.py, .mailmap, contributor-check.yml. Plus 80 files under scripts/, 12 of them CI helpers.
  • Recommendation: —
  • Priority: Low

• 31. Are there project CLI scripts or task-runner targets for the common jobs? Check the manifest's scripts, the Makefile, the justfile, or this ecosystem's equivalent. The test: does routine work need a raw multi-flag command that someone has to remember?

  • Status: PASS
  • Proof: package.json:13-27 gives root targets — check (npm run --ws check), fix, and per-workspace install:* / audit:* / audit:fix:* — and each workspace defines the same verbs, e.g. ui-tui: check = build:ink && typecheck && test && lint, plus dev, build, fmt, lint:fix. Python side: scripts/run_tests.sh (hermetic wrapper with --file-retries, pass-through pytest flags), scripts/run_tests_parallel.py, scripts/release.py, scripts/install.sh, setup-hermes.sh, scripts/dev-sandbox.sh, and docker-compose.yml. pyproject.toml:364 declares the hermes console script. Routine work does not need a remembered multi-flag command.
  • Recommendation: —
  • Priority: High

• 32. Are those scripts named somewhere the agent will actually read them? If no scripts exist, FAIL with a one-line proof pointing at the task-scripts item. Otherwise check the agent instruction file, the README, and whether the manifest itself is self-explanatory.

  • Status: PASS
  • Proof: The ones an agent needs daily are in the instruction set: scripts/run_tests.sh with four invocations at AGENTS.md:1362-1367 and again at CONTRIBUTING.md:201-213 and README.md:231; the TUI's npm run dev/build/typecheck/lint/fmt/test at AGENTS.md:504-517; the desktop's npx vitest run src/lib/desktop-slash-commands.test.ts at AGENTS.md:547; scripts/release.py usage in its own docstring. The root check and fix targets are not named in prose but package.json:25-26 is self-explanatory. The linter commands are the exception, and they are failed at the build-test-check-commands item rather than here.
  • Recommendation: —
  • Priority: High

• 33. Can an agent get this project running — is there a reproducible environment or a documented setup path? Look for a pinned runtime (.nvmrc, .tool-versions, rust-toolchain), a lockfile, a container or nix file, and written setup steps. Try the first step if it is cheap and safe. This one bites hardest the moment work happens in a fresh git worktree or a new clone — the normal way to run agents in parallel. A new worktree has no installed dependencies, no .env, no build cache, so anything that works today only because of untracked state sitting on someone's machine simply does not run there. The test: would a bare checkout plus the written steps get this project up? Name any prerequisite nothing creates — an env file someone hand-made, a seeded database, a manual login — because each one is a wall a worktree hits on its first command.

  • Status: PASS
  • Proof: Proved on a bare clone in a temp folder, no prior state: uv sync --locked --python 3.11 --extra dev succeeded from uv.lock, and scripts/run_tests.sh then ran 276 tests green. Runtimes are pinned (.python-version = 3.11, .nvmrc = 26, engines.node >= 22.22.0), dependencies are locked (uv.lock + five package-lock.json), and there are three documented paths: CONTRIBUTING.md:105-213 (installer or manual clone, with the warning to create the venv outside the source tree), flake.nix + nix/devShell.nix, and Dockerfile + docker-compose.yml. Two walls a fresh worktree hits, neither of which anything creates for you: scripts/run_tests.sh needs a .venv/venv at the checkout root or HERMES_PYTHON (it exits with exactly that message and both paths it probed), and actually running the agent needs ~/.hermes/config.yaml plus at least one provider key in ~/.hermes/.env (CONTRIBUTING.md:174-184). Tests themselves need neither — the wrapper unsets credentials on purpose.
  • Recommendation: —
  • Priority: High

• 34. Can an agent see the results of a failed run — do the tools here produce output it can read and act on? Judge from the runs you already did in Verification: does a failure print a path, a line, a name — something actionable — or a wall of noise? If nothing could be run, FAIL saying why.

  • Status: PASS
  • Proof: Everything I ran printed something actionable. scripts/run_tests.sh with no venv: "error: no virtualenv with pytest found in <path>/.venv or <path>/venv, and HERMES_PYTHON is not a python with pytest (enter the Nix devShell or create a venv)" — the two paths it probed and three ways to fix it. The test runner prints a live per-file progress line, a summary ("276 tests passed, 0 failed, 4 skipped"), a host-skip note naming the CI lane that does run them, a ⚠ FLAKY section with both attempts when a retry saves a file, and a per-file duration table. ruff prints file:line:col with the rule; ty prints the source line with a caret.
  • Recommendation: —
  • Priority: Low

5. Safety

• 35. Are credentials kept out of the repo — nothing secret committed, ignore rules in place, an example env file for the shape? Three checks: grep tracked files for key-shaped strings, read the ignore rules for env and key patterns, and look for an example env file. Report each of the three separately.

  • Status: PASS
  • Proof: Three checks, all clean. (1) Grep of tracked files for key shapes — git grep -nIE '(sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{30,}|AKIA[0-9A-Z]{12,}|-----BEGIN .*PRIVATE KEY-----)' returns exactly one hit, a comment describing a redaction pattern at agent/redact.py:360. (2) Ignore rules — .gitignore covers .env, .op.env, .env.local, .env.development, .env.test, .env.*.local (lines 14-22) and *.pem under a "Private keys" heading (59-61). (3) Example file — .env.example is tracked, keys commented out, each with a description and the URL to obtain it; AGENTS.md:647-663 states the rule that .env is secrets only and everything else belongs in config.yaml.
  • Recommendation: —
  • Priority: High

• 36. Does anything scan for secrets automatically? Look for gitleaks, trufflehog, detect-secrets or this ecosystem's equivalent, wherever it is wired in — a CI step, a pre-commit hook, or forge-level push protection visible from the repo. CI is where this normally lives, and that is a PASS; a local hook on top is better, because it catches the key before it is pushed rather than after, but its absence is a line in the proof, not a FAIL. Say where the scan runs. "Nothing secret exists today" does not make this N/A — the scan is for the day that changes.

  • Status: FAIL
  • Proof: No secret scanner anywhere. grep -rilE 'gitleaks|trufflehog|detect-secrets|secret-scan' over .github/, scripts/ and .pre-commit-config.yaml returns nothing, and there is no pre-commit config in the tree. The nearest thing, .github/workflows/supply-chain-audit.yml:68-115, greps the PR diff for malware signatures — .pth files added under site-packages/ and base64.b64decode on the same line as exec(/eval( (the litellm attack pattern) — which is a different job. agent/redact.py redacts secrets at agent runtime, not in the repo. Forge-level push protection may be on, but nothing in the tree shows it.
  • Recommendation: Add a gitleaks step to the existing supply-chain-audit.yml scan job — it already computes the base...head diff and posts findings into the CI review comment, so it is one more step in a job that runs on every PR.
  • Priority: High

• 37. Are dependencies pinned, so a build is reproducible? Look for lockfiles in every package of the repo, exact versions for load-bearing dependencies, a pinned runtime, and an install command that respects the lock (npm ci, not npm install).

  • Status: PASS
  • Proof: Pinned at every level, with the policy written down at AGENTS.md:598-620 (a table: PyPI >=floor,<next_major, git URLs by 40-char SHA, GitHub Actions by SHA + version comment, CI-only pip ==exact) and the incidents that produced it (litellm compromise PR #2796/#2810, Mini Shai-Hulud May 2026). Artefacts: uv.lock with hashes, package-lock.json at the root and in website/, plugins/platforms/photon/sidecar/, scripts/whatsapp-bridge/, nix/; .python-version 3.11; .nvmrc 26; package.json overrides pinning 13 transitive packages; devDependencies pinned to exact versions. Install respects the lock everywhere: uv sync --locked in tests.yml, tests-os.yml, docker.yml, e2e-desktop.yml; npm ci in js-tests.yml, deploy-site.yml, docs-site-checks.yml, e2e-desktop.yml. Two workflows guard it — uv-lockfile-check.yml and lockfile-diff.yml — and supply-chain-audit.yml:181-258 fails the build on an unbounded PyPI dependency. Actions are SHA-pinned throughout (actions/checkout@de0fac2e… # v6.0.2).
  • Recommendation: —
  • Priority: High

• 38. Is anything watching those dependencies for known vulnerabilities? Look for dependabot or renovate config, an audit step in CI, or this ecosystem's equivalent. Check every lockfile in the repo is covered, not just the root one.

  • Status: PASS
  • Proof: .github/workflows/osv-scanner.yml:42-55 scans every lockfile in the repo by name — uv.lock, package-lock.json, website/package-lock.json, plugins/platforms/photon/sidecar/package-lock.json, scripts/whatsapp-bridge/package-lock.json — and uploads SARIF to code scanning. .github/dependabot.yml covers github-actions weekly and explains, in 24 lines of comment, why source ecosystems are deliberately excluded (exact pins are moved after review, not on a schedule) while CVE-triggered security updates stay enabled at the repo setting. package.json:19-24 adds audit:root, audit:web, audit:tui for manual npm audits.
  • Recommendation: —
  • Priority: Low

• 39. Are the review rules written down — who reads a change, and what they check? Look for CONTRIBUTING, a PR template, or a review checklist in the agent instruction set. On a solo repo the "who" is N/A-shaped but the "what gets checked before it lands" still matters — judge that half.

  • Status: PASS
  • Proof: Written down in three layers. CONTRIBUTING.md (48,835 bytes) covers priorities, search-first, skill-vs-tool, dev setup, code style, and how to add each kind of thing. .github/PULL_REQUEST_TEMPLATE.md asks for what/why, a linked issue, type of change, changed files, reproduction-and-proof steps, and a checklist. AGENTS.md:29-211 is the actual review rubric — what gets merged, what is rejected even when well built, the four wrong-premise patterns distilled from real closes, and when not to close a PR. Enforcement exists too: contributor-check.yml (attribution), review-labels.yml, history-check.yml, and SECURITY.md for vulnerability reports.
  • Recommendation: —
  • Priority: Low

• 40. Are the operations that need a human named somewhere an agent will read them? Look for a "never without asking" list in the agent instruction file or the README. Docs that hand out production commands with no fence around them count against, and the proof should quote one.

  • Status: FAIL
  • Proof: There is no "never without asking" list. Grepping AGENTS.md for approval, never-without, ask-a-maintainer and do-not-run language returns only unrelated hits (the gateway's approval message guards at 1310, a dependency-bounds rule at 614). apps/desktop/AGENTS.md has none either. SECURITY.md is about the product's trust model — OS isolation, credential scoping, plugin trust — not about which repo operations an agent must hand back to a human. Meanwhile the docs hand out publishing commands with no fence: scripts/release.py:12 documents "# Create the release" / python scripts/release.py --bump minor --publish, and .github/workflows/deploy-site.yml:35-40 documents "manually via gh workflow run deploy-site.yml when an out-of-band main commit needs to ship live".
  • Recommendation: Add a short "Never without a human" section to AGENTS.md listing the operations an agent must stop at: scripts/release.py --publish, gh workflow run deploy-site.yml, git push to main, force-pushes, closing or merging PRs, and any command that spends a provider API key (evals/, scripts/*_livetest.py, batch_runner.py, mini_swe_runner.py).
  • Priority: High

• 41. Does every action that spends money, destroys data or changes production have a human in the way? Start from the damage, not from the tooling: list what in reach of this repo could charge a card, drop or overwrite data, or alter what users are running. Then trace the shortest route an agent could take to each one — a push that auto-deploys, a script carrying live credentials, a migration that runs on merge, an infrastructure apply with no plan-and-approve step. PASS when every route meets a human first, whether that is a review, a manual trigger or a protected environment. FAIL when even one route runs start to finish unattended, and quote that route in the proof so the fix is obvious.

  • Status: FAIL
  • Proof: Three things in reach can change what users run: the container images, the docs site, and a GitHub release. Two of the routes meet a human — docker.yml publishes :main/:latest only on a push to main and behind a named environment: container-publish, and deploy-site.yml deploys on a push to main touching website/** — and both sit downstream of a PR merge gated by the all-checks-pass required check (ci.yml:33-36). One route does not: python scripts/release.py --bump minor --publish run from a checkout with a gh token creates a public GitHub release, and deploy-site.yml:3-5 and docker.yml both fire on release: types: [published], so a single unattended command ships a release, the site and the container tags. Its own docstring advertises it. Whether container-publish requires reviewers is a repo setting I cannot see from the tree.
  • Recommendation: Make scripts/release.py --publish refuse to run outside CI unless a --i-am-a-maintainer style flag is passed, and move the release cut into a workflow_dispatch workflow behind a protected environment with required reviewers, so the human gate is enforced rather than assumed.
  • Priority: High

• 42. If a prompt injection landed tonight, how far would it reach — are the credentials an agent can get to here scoped to the job, with nothing production-grade in reach? Inventory what an agent in this repo can reach: env files, cloud CLI profiles, tokens named in docs or config, deploy commands that work from a laptop. Scoped-or-absent passes; production-grade reach fails with the item named.

  • Status: FAIL
  • Proof: Repo-provided credentials are clean: nothing secret is committed, .env is gitignored, and CI is scoped hard — ci.yml:12-14 says "Do not add secrets: inherit or GitHub App credentials here. Trusted main-only automation uses protected environments in its own workflows", actions are SHA-pinned, and SECURITY.md §2.3 covers credential scoping. What an agent can still reach on a maintainer's machine is not scoped and not fenced: ~/.hermes/.env holds live paid provider keys, and the repo ships commands that spend them at scale — evals/, scripts/tool_search_livetest*.py (five of them), batch_runner.py, mini_swe_runner.py, scripts/benchmark_browser_eval.py — plus the laptop publish routes at the human-gate item. Nothing in the instruction set says which of these bill real money.
  • Recommendation: In the same AGENTS.md section, state that ~/.hermes/.env holds live paid keys, that scripts/run_tests.sh is hermetic and safe by design while evals/, scripts/*_livetest.py, batch_runner.py and mini_swe_runner.py bill real money, and that agents should use a separate low-limit key for any of those.
  • Priority: High

• 43. Can a change reach production a slice at a time — a feature flag that defaults to off, a canary, a staged rollout — rather than everyone at once? Look for a flag system and check the default, or canary and staged-rollout config in the deploy pipeline. Flags that need a rebuild to flip are worth naming in the proof — they gate exposure but they are not a kill switch.

  • Status: PASS
  • Proof: New capability lands off by default and flips at runtime, which is the pattern the instructions enforce rather than merely allow. AGENTS.md:621-663 requires behavioural settings to live in ~/.hermes/config.yaml — read at startup, no rebuild — and rejects new HERMES_* env vars for anything but secrets; AGENTS.md:182-211 puts service-gated tools behind a check_fn so they are absent until a prerequisite is configured; toolsets are opt-in per profile, optional-skills/ is "shipped but NOT active by default", plugins are discovered only from ~/.hermes/plugins/, and features like kanban.dispatch_in_gateway and curator.enabled are plain config booleans. What does not exist is a canary or percentage rollout for the released binaries themselves — a published release reaches every updater at once (git grep -inE 'canary|staged rollout|release channel' finds no release-gating machinery).
  • Recommendation: —
  • Priority: High

• 44. Once a change is live, can anyone see what it is doing — logs, metrics, traces, alerts that fire on their own, and can an agent read them too? Look for logging setup, an error tracker, analytics, alerting config — and then ask the second half: could an agent reach any of it (a CLI, an MCP server, an API named in the docs), or does observability stop at a dashboard behind a login?

  • Status: PASS
  • Proof: Both halves. Production: hermes_logging.py sets up agent.log (INFO+), errors.log (WARNING+) and gateway.log, profile-aware via get_hermes_home(); docs/observability/README.md defines a read-only observer-hook contract for trace/metrics/audit/replay exporters (Langfuse, OTel-style collectors, NeMo Relay); docs/observability/monitoring.md covers alerting including a killed-gateway drill; plugins/observability/ ships the plugin. Agent-readable: AGENTS.md:311-314 names hermes logs [--follow] [--level ...] [--session ...] and the on-disk paths, and hermes_cli/subcommands/logs.py implements it — so an agent reads logs with a CLI, not through a dashboard behind a login.
  • Recommendation: —
  • Priority: Low

• 45. Is there a way back — can a bad change be undone without a rebuild and a redeploy, including the ones that touched a database or a queue? Look for a documented rollback path, a revert-and-redeploy story, down-migrations, or a flag that can turn the change off at runtime. Deployment docs that only say how to go forward are worth quoting.

  • Status: PASS
  • Proof: website/docs/getting-started/updating.md:195-223 is a "Rollback instructions" section: list recent versions, roll back to a specific commit, or check out a previous release tag (git tag --sort=-version:refname), then restart the gateway. It is not the only path. The updater takes a pre-update snapshot by default — pairing data, cron jobs, config.yaml, .env, auth.json — controlled by updates.pre_update_backup (quick/full/off, with --backup/--no-backup on hermes update), restorable through the flow in website/docs/user-guide/checkpoints-and-rollback.md. There is an automatic one too: after pulling, Hermes compiles the nine critical startup files and runs git reset --hard <pre-pull-sha> if any fails to parse, "so your shell stays bootable". Nix installs roll back with nix profile rollback; container users pin :<release_tag> instead of :latest. Runtime state is covered by the plugin-state atomic-replace guarantees in the newest RFC.
  • Recommendation: —
  • Priority: Low

• 46. Does a past failure leave a check behind — is there a test or a rule in here that exists because something broke once? Look for tests, lint rules or comments that name the incident they guard against — "regression", "this broke when", a linked issue. On a very young repo with no failures yet, N/A with the age as the reason.

  • Status: PASS
  • Proof: Repeatedly, and the checks name their incident. pyproject.toml:454-462 keeps ruff's PLW1514 on alone because "We had three separate Windows sandbox regressions in one debug session before adding the explicit encoding." scripts/check-windows-footguns.py is a blocking CI checker built from Windows-unsafe primitives that bit before (os.kill(pid, 0), os.killpg, signal.SIGKILL without a getattr fallback). AGENTS.md:1293-1296 traces the get_hermes_home() rule to "5 bugs fixed in PR #3575"; 598-604 traces the pinning policy to the litellm compromise; 1329-1333 exists because dead code was wired in without E2E proof. supply-chain-audit.yml:88-91 links the exact upstream issue its .pth check guards against. docs/rca-ssl-cacert-post-git-pull.md is a full RCA, and sqlite_leak_fix.png sits in the root as the artefact of another. 357 test files mention a regression.
  • Recommendation: —
  • Priority: Low

Learn more: https://ainativesoftware.engineering/