home-assistant/core

AI-Native Readiness Report

Repo: home-assistant/core (shallow clone of dev, newest commit 7372ce4b, 2026-08-14) Date: 2026-08-14 Stack: Python 3.14 (.python-version 3.14.5, requires-python >=3.14.2), setuptools + pyproject.toml, pytest + syrupy for tests, Playwright/pnpm for a small e2e suite, ruff + pylint + mypy + hassfest for checks, prek (pre-commit runner) as the local gate, GitHub Actions for CI, uv for environment management, script/ as the task runner.

Result: 34 passed, 12 failed, 0 not applicable

How to read this. Every item is a numbered question about this repo, answered from evidence in this repo. Proof is what was actually found — a path, a file, a command and its output. An item with no proof is not an answer. The italic line under each question says how to check it; it stays in the report so the next run checks the same way.

FAIL vs N/A. FAIL means the practice would help here and it is missing — including when a whole artifact is missing, so several questions about it fail together with a one-line proof pointing at the item that found it missing. N/A means the question could never apply to a project of this kind, and the proof says why. "It does not exist" is a FAIL, never an N/A.

Priority is fixed by the template, not by the person filling it in. High means an agent working here will hit this on day one, or the risk is real. Low means it makes things better but nothing breaks without it.

1. Context

• 1. Does an agent instruction file exist (AGENTS.md, CLAUDE.md, .cursorrules, .github/copilot-instructions.md, or this ecosystem's equivalent), and what does it point at? Look for a file whose purpose is to be loaded into a coding agent's context at the start of every session. That file is the entry point. Then follow its pointers: instructions are commonly spread across several files, with the entry point routing to them — "commands are in docs/development.md", "conventions in .claude/rules/". The entry point plus every file it points at is the instruction set, and every other agent-instruction question is answered against that whole set, so list it here in the proof. Follow pointers as deep as they go, but a pointer only counts when it names a target and says what is in it: "test commands are in CONTRIBUTING.md" counts, a bare "see the docs" does not, and a file nothing points at is outside the set however good it is. A README is not an entry point, however good — it gets its credit at the README item and the specs-and-architecture item — but a README the entry point explicitly routes to is inside the set. Vendored skill libraries are not instructions either.

  • Status: PASS
  • Proof: AGENTS.md at the root, 5,556 bytes, titled "GitHub Copilot & Claude Code Instructions". ls -la shows CLAUDE.md -> AGENTS.md — a symlink, so both agents load the same file. It points at four targets and says what each holds: AI_POLICY.md ("This project follows the Open Home Foundation AI Policy"), .github/PULL_REQUEST_TEMPLATE.md ("use the repository's PR template"), .vscode/tasks.json ("contains useful commands used for development"), and script/setup ("run script/setup to set up the virtual environment with all development dependencies"). The instruction set is AGENTS.md plus those four. A second, generated entry point exists for Copilot: .github/copilot-instructions.md (11,920 bytes) and the path-scoped .github/instructions/integrations.instructions.md (applyTo: "homeassistant/components/**, tests/components/**"), both produced by script/gen_copilot_instructions.py from AGENTS.md plus the skills, with a pre-commit hook and the CI job gen-copilot-instructions keeping them in sync. .claude/skills/ and .claude/agents/ are a skills library, judged at the skills-library item, not part of the instruction set.
  • Recommendation: —
  • Priority: High

• 2. Do the agent instructions name the commands to build, test and check this project? Answer this against the instruction set mapped at the entry-point item and nothing outside it. If no entry point exists, FAIL with a one-line proof pointing at that item. The commands may sit in a file the entry point routes to rather than in the entry point itself — that is progressive disclosure working as intended, and it passes. What fails is a command an agent would have to guess its way to: if the commands live only in the README or the manifest and nothing in the set points at them, that is a FAIL, because the question measures what an agent can reach without being told where to look. Name the file each command was found in.

  • Status: PASS
  • Proof: All four command classes are inside the set. Setup: AGENTS.md line 17, "run script/setup to set up the virtual environment with all development dependencies (pylint, pre-commit hooks, etc.)", with the uv-too-old recovery step spelled out. Check: AGENTS.md line 19, "After finishing a code session, run uv run --no-sync prek run --all-files to check for linting and formatting issues." Test: AGENTS.md line 29, "Use uv run --no-sync pytest to run tests", plus line 30, "python3 -m script.translations develop --integration <integration_name>" before running tests after a strings.json change. More commands sit in the routed-to .vscode/tasks.json, which carries labelled tasks for Pytest, Ruff, Prek, Pylint, Code Coverage, Update syrupy snapshots, Generate Requirements, Compile English translations, Run scaffold and Create new integration. The routed-to .github/PULL_REQUEST_TEMPLATE.md adds python3 -m script.hassfest, python3 -m script.gen_requirements_all and ruff format homeassistant tests.
  • Recommendation: —
  • Priority: High

• 3. Does everything the agent instructions name still exist — the commands, the paths, the libraries, and the files they point at? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. Verify, do not trust: check every named command against the manifest or build file, spot-check the paths, run the cheap read-only ones. Then resolve every pointer in every file of the set — a link to a moved or deleted file is the most common rot in a multi-file instruction set, and it fails silently: the agent reads the entry point, follows nothing, and carries on without the rules.

  • Status: PASS
  • Proof: Every pointer resolves: AI_POLICY.md, .github/PULL_REQUEST_TEMPLATE.md, .vscode/tasks.json and script/setup all exist in git ls-files. Every named tool exists in the manifest: prek==0.2.28 and the pytest plugin set (pytest-asyncio, pytest-cov, pytest-timeout, pytest-socket, …) are pinned in requirements_test.txt; script/translations, script/hassfest, script/gen_requirements_all.py and script/setup all exist under script/. The Python-version claims hold: AGENTS.md says 3.14 is the minimum, .python-version is 3.14.5 and pyproject.toml line 24 is requires-python = ">=3.14.2". Ran the cheap read-only check: uvx ruff@0.16.1 check homeassistant tests script pylint → "All checks passed!", matching the ruff-check hook in .pre-commit-config.yaml. Not executed here: script/setup, pytest, mypy and pylint, which all need the full dependency install — CI runs them as the prek, mypy, pylint and pytest-full jobs in .github/workflows/ci.yaml.
  • Recommendation: —
  • Priority: High

• 4. Are the agent instructions specific to this repo, rather than advice that would read the same in any codebase? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The test: could this be pasted into another project unchanged? "Write clean code" and "add tests for new features" would fit anywhere and count for nothing. Judge the whole set, but weigh the files differently: an entry point that is mostly a routing table is fine, even good, when what it routes to is specific — while generic filler in the entry point costs more than generic filler three hops down, because it is loaded into every session whether it is needed or not. Say which files carried the specifics.

  • Status: PASS
  • Proof: AGENTS.md carries the specifics itself and has almost no filler. Examples that could not be pasted anywhere else: "Python 3.14 explicitly allows except TypeA, TypeB: without parentheses. Never flag this as an issue"; "Python 3.14 evaluates annotations lazily (PEP 649)"; "Tests load translations from the generated translations/en.json, not directly from strings.json"; "We use Syrupy for snapshot testing. Leverage .ambr snapshots"; "Register them with the async_register_admin_service service helper"; "Integrations with Platinum or Gold level in the Integration Quality Scale … The level is indicated in the manifest.json of the integration." The routed-to .github/PULL_REQUEST_TEMPLATE.md is equally concrete (hassfest, gen_requirements_all, requirements_all.txt). The nearest thing to generic advice is "Keep comments concise", and even that is followed by two HA-specific rules about restating comments and divider comments.
  • Recommendation: —
  • Priority: High

• 5. Do the agent instructions explain where things live and how this project is laid out? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The map may live in a routed-to file. Judge coverage against the real tree, not against what the files mention: if the set maps one package well but is silent about sibling packages or directories an agent would land in, that is a FAIL with the omission named. In a multi-file set, check the routing too — a layout document nothing points at is a document the agent never opens.

  • Status: FAIL
  • Proof: No file in the instruction set contains a layout map. AGENTS.md names individual files in passing (.vscode/tasks.json, manifest.json, strings.json) but never says where anything lives; AI_POLICY.md and .github/PULL_REQUEST_TEMPLATE.md are process documents; .vscode/tasks.json is a command list. The only real map in the repo is .claude/skills/ha-integration-knowledge/SKILL.md, which opens with "Integration code: ./homeassistant/components/<integration_domain>/" and "Integration tests: ./tests/components/<integration_domain>/" — and nothing in the instruction set routes to it. It also covers only integrations. Directories an agent will land in and that nothing maps: homeassistant/helpers/, homeassistant/util/, homeassistant/generated/ (generated, must not be hand-edited), script/hassfest/, pylint/plugins/ (custom pylint checkers), mypy_plugins/, machine/ and rootfs/.
  • Recommendation: Add a short "Repository layout" section to AGENTS.md naming homeassistant/components/<domain>/, homeassistant/helpers/, homeassistant/util/, homeassistant/generated/ (generated — do not edit), tests/components/<domain>/, script/hassfest/ and pylint/plugins/, and route to .claude/skills/ha-integration-knowledge/SKILL.md for integration work.
  • Priority: High

• 6. Do the agent instructions state the rules that are not obvious from the code — the things never to do here? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. These are the tribal-knowledge traps: the flag that must be exactly this string, the import that breaks the build, the directory that is generated and must not be edited. Rules in a routed-to file count. Two extra checks in a multi-file set: that the entry point signposts the rules clearly enough for an agent to open them before it needs them, since a trap found afterwards has already been sprung; and that the files do not contradict each other, because nothing tells the agent which one wins.

  • Status: PASS
  • Proof: AGENTS.md states the traps up front, in the first screen an agent reads: "Do NOT amend, squash, or rebase commits that have already been pushed to the PR branch after the PR is opened"; "NEVER REMOVE ANYTHING from the template"; "Do not flag syntax or features that require Python 3.14 as issues"; "After modifying strings.json … regenerate the English translation file before running tests"; "Avoid using conditions/branching in tests"; "Do not add section or divider comments"; and the AI-policy rule "Do not open issues or pull requests autonomously". No contradictions between AGENTS.md, AI_POLICY.md and the PR template — the generated Copilot files are byte-derived from AGENTS.md, so they cannot drift. One gap worth naming: the generated-file rule is signposted only in the file headers themselves (requirements.txt, requirements_all.txt, CODEOWNERS and .github/copilot-instructions.md each open with "Automatically generated … do not edit"), not in AGENTS.md, so an agent learns it on opening the file rather than before.
  • Recommendation: —
  • Priority: Low

• 7. Has the agent instruction file been updated recently enough to still be true, given how active the repo is? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Compare the last commit touching the file against the repo's tempo, then spot-check two or three of its claims against the code — a recently touched file can still lie.

  • Status: PASS
  • Proof: git log -1 --date=short -- AGENTS.md → 09eaf900 2026-08-12, two days before the newest commit in this clone (2026-08-14). The repo's tempo is very high — the last 15 commits all landed within days — so a two-day-old instruction file is current. Three claims spot-checked against the code and all held: prek is real and pinned (prek==0.2.28 in requirements_test.txt, and a prek job in .github/workflows/ci.yaml); the Python 3.14 minimum matches .python-version 3.14.5 and requires-python = ">=3.14.2"; python3 -m script.translations develop resolves to the existing script/translations package. The file is also protected from drift by the gen_copilot_instructions pre-commit hook, which fires on any change to AGENTS.md.
  • Recommendation: —
  • Priority: Low

• 8. Is the agent instruction file small enough to load into every session without crowding out the actual task? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Measure it: wc -c, and bytes divided by four is a fair token estimate. Under ~5k tokens is comfortable; past ~10k it is eating the context window.

  • Status: PASS
  • Proof: wc -c AGENTS.md → 5,556 bytes, roughly 1.4k tokens. Well inside the comfortable range. The generated Copilot copy is larger at 11,920 bytes (~3k tokens) because script/gen_copilot_instructions.py inlines the whole PR template into it, and the path-scoped .github/instructions/integrations.instructions.md is 3,559 bytes (~0.9k tokens) and only loads under homeassistant/components/**. Every file in the set stays under the 5k-token line.
  • Recommendation: —
  • Priority: Low

• 9. Can a fresh session pick up a half-finished task — is there a file or convention where progress, decisions and what is left get written down? This is not about the agent instruction file. Any durable place in-flight state gets written counts: a TODO.md, a plans or notes directory, task files, linked issues, a specs folder whose entries carry progress and open questions, an agent memory file or directory, a scratchpad or working-notes convention. The test is whether a fresh session could read it and know what was decided and what is left — not what the place is called. Git history alone does not count: it records what happened, not what was decided or what remains.

  • Status: PASS
  • Proof: The convention is GitHub-side and committed. .github/ISSUE_TEMPLATE/task.yml is a Task issue form ("For staff only — Create a task") whose fields are exactly in-flight state: a Description covering "What needs to be done / Why this task is needed / Expected outcome / Any constraints", and an Additional context field prompting for "Roadmap opportunity / Epic / Feature request / Technical design documents / Prototype/mockup / Dependencies" links. .github/PULL_REQUEST_TEMPLATE.md requires the link back ("This PR fixes or closes issue: fixes #"). Decisions taken mid-review are recoverable through a committed skill built for the job: .claude/skills/ha-pr-comment-audit/SKILL.md — "Audits the review comment threads on a Home Assistant GitHub pull request, flagging unaddressed comments and requests for clarification" — resolving the PR with gh pr view and the threads with gh api. Limit worth naming: all of it lives in GitHub, so a resumed session needs an authenticated gh; an offline clone carries no in-flight state at all.
  • Recommendation: —
  • Priority: Low

• 10. Is there a README that says what this project is and how to run it? This is where a good README earns its credit. It needs three things: what the project is, how to run it, and how to check a change. Judge what is on the page, not the file's existence.

  • Status: FAIL
  • Proof: README.rst is 34 lines and delivers one of the three. What it is: yes — "Open source home automation that puts local control and privacy first." How to run it: no — there is not a single command on the page; it links out to home-assistant.io/getting-started/ for end-user installation, which is not a developer run path. How to check a change: no — no mention of script/setup, pytest, prek, or of AGENTS.md and CONTRIBUTING.md where those live. grep for a shell command in the file returns nothing; the rest of the page is two screenshots, an architecture link to developers.home-assistant.io, and badges.
  • Recommendation: Add a short "Development" section to README.rst with the four lines that already exist elsewhere — script/setup, uv run --no-sync pytest, uv run --no-sync prek run --all-files — and a link to AGENTS.md and CONTRIBUTING.md.
  • Priority: High

2. Specs

• 11. Is the thinking behind this system written down somewhere durable — a specs, RFC, proposals, design-doc or ADR directory, or architecture notes that record not just what but why? Look for the place and for the convention: specs/, docs/adr/, rfcs/, proposals/, .specify/, a docs or design folder, architecture notes anywhere in the tree, or this ecosystem's equivalent. Judge substance, not location: "we use X because Y" is a decision, a list of technologies is not, an essay about specs is not a spec, and a docs folder of usage guides with no reasoning is a FAIL whose proof says what was in there instead. Other items are answered against whatever this item finds, so name it precisely — and where forward-looking specs and after-the-fact architecture records live in different places, name both, since a decision log cannot answer a question about acceptance criteria.

  • Status: FAIL
  • Proof: No specs, RFC, proposals, design-doc or ADR directory exists. git ls-files filtered for docs|adr|rfcs|specs|proposals|design|architecture path segments returns two zwave_js PNG assets and nothing else. There is no .specify/. git grep -l 'Architecture Decision\|ADR-0' over all markdown and rst returns nothing. The complete set of tracked markdown outside homeassistant/components/ and .github/ is AGENTS.md, AI_POLICY.md, CLA.md, CLAUDE.md, CODE_OF_CONDUCT.md, CONTRIBUTING.md, LICENSE.md, plus pylint/plugins/README.md and tests/components/knx/README.md — process and licence documents, no design reasoning. Three per-integration READMEs exist (prometheus, tplink, zwave_js) and are usage notes. The nearest thing to a decision record is the 337 homeassistant/components/*/quality_scale.yaml files, which do carry real per-rule reasoning — homeassistant/components/acaia/quality_scale.yaml records test-before-setup: status: exempt, comment: "Device is expected to be offline most of the time, but needs to connect quickly once available." — but that is a compliance checklist for one ruleset, not the system's design thinking. System architecture reasoning lives outside this repo, at developers.home-assistant.io/docs/architecture_index/, linked once from README.rst line 17, so an agent working in the clone cannot read it.
  • Recommendation: Add docs/adr/ to this repo and seed it with the decisions an agent hits weekly — why integrations must be thin wrappers over a PyPI library, why polling intervals are not user-configurable, why the recorder schema is forward-only — then route AGENTS.md at the directory so it is loaded rather than discovered.
  • Priority: High

• 12. Does the specs directory hold recent entries, or is it an archive nobody has touched? If no specs directory exists, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise compare the newest entry's date against the repo's recent activity.

  • Status: FAIL
  • Proof: No specs or architecture-decision directory exists — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs and architecture-decision directory.
  • Priority: Low

• 13. Is there a spec template, or an SDD framework, so every spec comes out the same shape? Scaffolding can exist even where no specs directory does — look for a TEMPLATE.md, a .specify/ directory, or framework config. If neither a directory nor any scaffolding exists, FAIL.

  • Status: FAIL
  • Proof: No spec scaffolding of any kind. find . -maxdepth 2 -iname '*TEMPLATE*' returns only .github/PULL_REQUEST_TEMPLATE.md and .github/ISSUE_TEMPLATE/, which shape change descriptions and issues, not specs. No .specify/ and no SDD framework config. What the repo does have, and is worth naming as the pattern to copy: homeassistant/components/*/quality_scale.yaml all come out the same shape because script/hassfest validates them against a fixed rule set, and script/scaffold generates uniform integration boilerplate — the same discipline applied to code and compliance, but never to design documents.
  • Recommendation: Add docs/adr/TEMPLATE.md with the sections an HA decision needs — context, decision, why, non-goals, how to verify — and add an adr check to script/hassfest so entries stay the same shape the way quality_scale.yaml already does.
  • Priority: Low

• 14. Do the specs state acceptance criteria a machine could check? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise open the two newest specs and quote a criterion: "the endpoint returns 403 for expired tokens" is checkable; "the feature works well" is not.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs and architecture-decision directory.
  • Priority: High

• 15. Open the newest spec: do its criteria go past the happy path — what happens when a step fails, and how the change gets undone? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Look for error cases, edge inputs, and a rollback or undo story, not just the success flow.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs and architecture-decision directory.
  • Priority: Low

• 16. Do the specs state non-goals, so an agent knows where to stop? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Non-goals written elsewhere (a README's "what this is not" list) are worth naming in the proof, but they do not turn this into a PASS — the question is whether specs carry them.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs and architecture-decision directory.
  • Priority: Low

• 17. Can recent shipped work be traced back to a spec? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise take the last few substantial commits or PRs and look for a reference to a spec, an issue, or a design doc in the message or description.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs and architecture-decision directory.
  • Priority: Low

3. Verification

• 18. Does this project have an automated test suite, in whatever form this ecosystem uses? Work out this ecosystem's convention before concluding anything is missing — check the manifest, the build file, the CI config, the README. A shell script that diffs output files is a test suite. If you find one, run it and record the result.

  • Status: PASS
  • Proof: git ls-files 'tests/**/*.py' | wc -l → 8,085 tracked test modules under tests/, mirroring homeassistant/components/ one directory per integration. Configured in pyproject.toml [tool.pytest.ini_options]: testpaths = ["tests"], asyncio_mode = "auto", an error::sqlalchemy.exc.SAWarning filter, and syrupy .ambr snapshots. A second suite exists for end-to-end: tests/e2e/onboarding.spec.ts with playwright.config.ts and pnpm-lock.yaml, run by .github/workflows/e2e-tests.yml. Not executed here: the suite needs the full install (script/setup → requirements_all.txt + requirements_test.txt), which is far past this audit's time budget on a fresh machine. CI runs it on every pull request across the pytest-full, pytest-partial, pytest-mariadb and pytest-postgres jobs in .github/workflows/ci.yaml.
  • Recommendation: —
  • Priority: High

• 19. Can the test command be discovered without guessing — is it written down where an agent reads? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise check the places an agent looks: the agent instruction file, the README, the manifest's scripts or targets.

  • Status: PASS
  • Proof: In the agent instruction file, AGENTS.md line 29: "Use uv run --no-sync pytest to run tests". In the routed-to .vscode/tasks.json, three labelled variants: "Pytest" (-m pytest --timeout=10 tests), "Pytest (changed tests only)" (--picked), and "Code Coverage" per integration. In the manifest, pyproject.toml sets testpaths = ["tests"] so a bare pytest works. Not in README.rst — see the README item.
  • Recommendation: —
  • Priority: High

• 20. Do the tests assert real behaviour, rather than asserting that a mock was called? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise open the largest test files and read the assertions: calling real code on real inputs passes; expect(mock).toHaveBeenCalled() as the main dish fails.

  • Status: PASS
  • Proof: Opened the two largest test modules. tests/components/sensor/test_recorder.py asserts against real recorder database rows and the real issue registry — assert db_state.statistic_id == expected_result[i]["statistic_id"], assert len(issue_registry.issues) == len(expected_issues), assert issue_registry.issues[(DOMAIN, issue_id)] == expected_issue. tests/components/mqtt/test_config_flow.py has 485 assert lines against 4 assert_called* — the config flow is driven for real and its resulting state checked. Syrupy .ambr snapshots assert whole entity and state payloads rather than call counts.
  • Recommendation: —
  • Priority: High

• 21. Is there a linter or static analysis configured for this language, and does it pass on a clean checkout? Configured is not enough — run it. A linter that exits non-zero on an untouched checkout is a FAIL with the error count in the proof, because an agent cannot tell its own damage from the baseline noise.

  • Status: PASS
  • Proof: Ran it on an untouched checkout: uvx ruff@0.16.1 check homeassistant tests script pylint → "All checks passed!" — zero findings, matching the pinned ruff-check hook at .pre-commit-config.yaml rev v0.16.1. Rules configured in pyproject.toml [tool.ruff.lint] with per-file ignores, banned APIs and import conventions. Three more analysers are configured and run by CI rather than here, because they need the installed venv: pylint with repo-specific checkers in pylint/plugins/ (pyproject.toml [tool.pylint], CI jobs pylint and pylint-tests), mypy (CI job mypy), and script/hassfest for manifest and metadata validation (CI job hassfest). zizmor --pedantic additionally lints the workflow files.
  • Recommendation: —
  • Priority: High

• 22. Is there a formatter, so an agent's diffs do not churn on style? Look for the config file and the dependency in this ecosystem's form — .prettierrc, rustfmt, gofmt, black, an .editorconfig doing real work. If the language ships one formatting standard with the toolchain, that is a PASS and the proof says so.

  • Status: PASS
  • Proof: Ran it: uvx ruff@0.16.1 format --check homeassistant tests script pylint → "18192 files already formatted", zero drift on a clean checkout. Wired as the ruff-format hook in .pre-commit-config.yaml and named in the PR checklist ("The code has been formatted using Ruff (ruff format homeassistant tests)"). Non-Python files are covered too: prettier v3.6.2 with prettier-plugin-sort-json (.prettierrc.js, .prettierignore), yamllint with .yamllint, and hadolint with .hadolint.yaml for the Dockerfiles.
  • Recommendation: —
  • Priority: Low

• 23. Is there a compile-time or type-level gate, if this language offers one? N/A only when the language genuinely has no such gate. If the language offers one and the repo does not use it — no strict mode, no typecheck script, no compiler step — that is a FAIL. Run the gate if it exists and record the result.

  • Status: PASS
  • Proof: mypy is the gate and it is used seriously. mypy.ini is generated by script/hassfest -p mypy_config from .strict-typing, a 664-line list of modules opted into strict checking ("If component is fully covered with type annotations, please add it here to enable strict mypy checks"), with strict typing on by default for core files. Enforced three ways: the mypy hook in .pre-commit-config.yaml via script/run-in-env.sh, the mypy CI job with a problem matcher at .github/workflows/matchers/mypy.json, and the hassfest-mypy-config hook that regenerates mypy.ini whenever .strict-typing changes. AGENTS.md reinforces it at the test level: "ensure all test function parameters have type annotations. Prefer concrete types … over Any." Not run here — mypy needs the installed venv; CI runs it on every pull request.
  • Recommendation: —
  • Priority: Low

• 24. Can an agent prove its own work before it pushes — one command, task-runner target or commit hook that runs every check this project has? One command, not a list to remember: a check or verify target, a precommit script, a Makefile target that chains them. Separate commands documented side by side are close but FAIL — the question is whether the agent can run the whole gauntlet without knowing its parts.

  • Status: PASS
  • Proof: One command, and AGENTS.md names it: "After finishing a code session, run uv run --no-sync prek run --all-files." That single invocation runs every static check in .pre-commit-config.yaml — ruff-check, ruff-format, codespell, zizmor, check-json, yamllint, prettier, mypy, pylint, gen_requirements_all, gen_copilot_instructions, hassfest, hassfest-metadata and hassfest-mypy-config — which is the same set CI runs as its prek job. script/setup installs the hooks (prek install) so it also fires on commit, and the no-commit-to-branch hook blocks dev, master and rc. Two narrower helpers exist for speed: script/lint (changed files only) and script/lint_and_test.py, whose own docstring is honest about scope — "This is NOT a full CI/linting replacement, only a quick check during development." The test suite is deliberately not in the prek run, and is named separately in AGENTS.md.
  • Recommendation: —
  • Priority: High

• 25. Is there continuous integration, and does it run before a change lands rather than after? A deploy pipeline that runs on push to the default branch runs after the change has landed — that is a FAIL with the trigger quoted, not a PASS with a caveat. Look for a pre-merge trigger: pull_request events, merge trains, or this forge's equivalent.

  • Status: PASS
  • Proof: .github/workflows/ci.yaml triggers on pull_request: ~ — every pull request, unfiltered — alongside push to dev, rc and master and a workflow_dispatch with granular inputs (lint-only, pylint-only, mypy-only, skip-coverage). The pre-merge trigger is present and primary. .github/workflows/e2e-tests.yml and the dependency-review job are likewise pull-request-scoped, and concurrency with cancel-in-progress: true keeps re-pushes cheap.
  • Recommendation: —
  • Priority: High

• 26. Does CI actually run the tests and the checks that exist in this repo? If no pipeline of any kind exists, FAIL with a one-line proof pointing at the CI item. Otherwise list what the pipeline runs and diff it against every check found in the Verification section — name each check that exists in the repo but is missing from the pipeline.

  • Status: PASS
  • Proof: .github/workflows/ci.yaml defines 19 jobs: info, prek, zizmor, lint-hadolint, base, hassfest, gen-requirements-all, gen-copilot-instructions, dependency-review, audit-licenses, pylint, pylint-tests, mypy, prepare-pytest-full, pytest-full, pytest-mariadb, pytest-postgres, pytest-partial, coverage-full, coverage-partial, upload-test-results. Diffed against every check found in this section: ruff-check and ruff-format run inside prek; codespell, yamllint, prettier and check-json also inside prek; mypy in mypy; pylint in pylint and pylint-tests; hassfest in hassfest; pytest across the four pytest jobs against MariaDB 10.3 through 11.4, MySQL 8.0 and PostgreSQL; Playwright e2e in the separate .github/workflows/e2e-tests.yml; CodeQL in codeql.yml; requirements determinism in check-requirements-deterministic.yml. Nothing found in the repo is missing from the pipeline.
  • Recommendation: —
  • Priority: High

• 27. Does anything here get an LLM to read a change and go looking for problems — a review skill, a saved review prompt, or an AI reviewer on the pull requests? Look for a committed review skill or slash command, a CI step calling an AI reviewer, or bot config. Human review rules are a Safety item; this one is specifically about machine review.

  • Status: PASS
  • Proof: Three separate mechanisms, all committed. An AI reviewer on the pull requests: .github/copilot-instructions.md opens with "# Copilot code review instructions" and carries repo-specific rules for it — "Do not comment on code style, formatting or linting issues", "A Pull Request with a dependency version bump should only contain changes required for the version bump" — plus the path-scoped .github/instructions/integrations.instructions.md; both are generated by script/gen_copilot_instructions.py and enforced by the gen-copilot-instructions CI job. Committed review skills: .claude/skills/ha-review/SKILL.md, .claude/skills/ha-pr-reviewer/SKILL.md ("the top skill to use for reviewing Pull Requests from GitHub", driving gh pr view and gh pr diff), .claude/skills/ha-pr-comment-audit/SKILL.md, and .claude/skills/ha-quality-scale-verify/SKILL.md. An AI CI step: .github/workflows/detect-duplicate-issues.yml requests models: read "For AI-based duplicate detection". AI_POLICY.md sets the boundary: "Some of our projects use AI tools to assist with code reviews … As with any automated tooling, these comments are not always correct."
  • Recommendation: —
  • Priority: Low

4. Tooling

• 28. Can an agent reach the outside systems this project depends on, with that access committed to the repo rather than set up per laptop? MCP servers are one form: look for a committed .mcp.json or this ecosystem's equivalent. A CLI is another and often the better one — aws, gh, psql, kubectl, stripe, a vendor's own tool — and it counts when the repo names which tools the work needs and how to authenticate, so an agent is not guessing at a tool it cannot see. Judge the axis, not the mechanism: access that exists only in someone's shell history or laptop config is a FAIL, and so is a committed config that turns out to be gitignored, with the distinction in the proof. Project task scripts have their own item; this one is about reaching past the repo's edge. If this project genuinely talks to nothing outside itself, N/A with that as the reason.

  • Status: PASS
  • Proof: The two systems past this repo's edge are GitHub and PyPI, and the CLI for each is named in committed files. GitHub via gh: .claude/agents/raise-pull-request.md:193 (gh pr create --repo home-assistant/core --base dev), .claude/skills/bump-dependency/SKILL.md:75, .claude/skills/ha-pr-reviewer/SKILL.md:9-11 (gh pr view, gh pr diff), .claude/skills/ha-pr-comment-audit/SKILL.md:10 (gh api for review threads). PyPI via a committed helper: .claude/skills/bump-dependency/scripts/resolve_dependency.py. No .mcp.json exists, and none is needed given the CLI route. .gitignore lines 144-145 exclude only .claude/settings.local.json and .claude/worktrees/, so nothing in the committed agent tooling is silently ignored. Gap worth naming: authentication is never stated — no file says gh auth login is a prerequisite, so a fresh worktree hits it unannounced on the first gh call.
  • Recommendation: —
  • Priority: Low

• 29. Is there a skills, commands or reusable-prompt library in the repo? Look for .claude/skills, .agents/skills, .claude/commands, or a prompts directory. Committed and pinned beats committed; note whether anything ties the copies to a source.

  • Status: PASS
  • Proof: .claude/skills/ holds six committed, first-party skills — bump-dependency (with a scripts/resolve_dependency.py helper), ha-integration-knowledge (plus platform-diagnostics.md and platform-repairs.md), ha-pr-comment-audit, ha-pr-reviewer, ha-quality-scale-verify, ha-review — all with proper frontmatter, totalling about 22 KB. .claude/agents/raise-pull-request.md adds a scoped subagent (tools: Read, Bash, Grep, Glob). The library is shared across vendors by symlink rather than copy: .agent/skills -> ../.claude/skills/ and .gemini/skills -> ../.claude/skills, both tracked by git, and script/gen_copilot_instructions.py folds ha-integration-knowledge into the Copilot instruction files. Pinning does not apply — nothing here is vendored from elsewhere, and the one derived copy is regenerated and CI-enforced.
  • Recommendation: —
  • Priority: Low

• 30. Do the skills, servers and tools that exist cover the work this team plainly repeats? First identify the repeated work from the README and the commit history. If skills, servers or documented tooling exist but miss it, FAIL naming the gap. If none exist at all and the repo plainly repeats work, FAIL pointing at the outside-systems item and the skills-library item. If the project is too small to repeat anything, N/A.

  • Status: PASS
  • Proof: The repeated work is visible in git log --oneline -15: dependency bumps dominate ("Bump midea-local to 8.0.1", "Bump blebox-uniapi to v2.5.7", "Bump pydaikin to 2.19.0", "Bump tesla-fleet-api to 1.9.0"), then per-integration fixes, then codeowner and quality-scale housekeeping. Each has a tool. Bumps: .claude/skills/bump-dependency/SKILL.md — "Bumps a Python package dependency across Home Assistant Core integrations, regenerates core requirement files, runs verification tests and prek lint, and prepares a pull request" — backed by script/gen_requirements_all.py and Renovate's homeassistant-manifest manager. New and edited integrations: script/scaffold, script/hassfest, .claude/skills/ha-integration-knowledge/SKILL.md. Quality-scale housekeeping: .claude/skills/ha-quality-scale-verify/SKILL.md and script/quality_scale_summary.py. Review: the three review skills. Codeowners: generated by script/hassfest/codeowners.py. No obvious repeated job is left without a tool.
  • Recommendation: —
  • Priority: Low

• 31. Are there project CLI scripts or task-runner targets for the common jobs? Check the manifest's scripts, the Makefile, the justfile, or this ecosystem's equivalent. The test: does routine work need a raw multi-flag command that someone has to remember?

  • Status: PASS
  • Proof: script/ is the task runner: setup, bootstrap, lint, lint_and_test.py, check_format, check_dirty, check_requirements, hassfest, scaffold, gen_requirements_all.py, gen_copilot_instructions.py, install_integration_requirements.py, translations, split_tests.py, quality_scale_summary.py, version_bump.py, licenses.py, run-in-env.sh. .vscode/tasks.json adds 14 labelled tasks that wrap the multi-flag invocations an agent would otherwise have to remember — the per-integration coverage run, for instance, is one label over -m pytest ./tests/components/${integrationName}/ --cov=homeassistant.components.${integrationName} --cov-report term-missing --durations-min=1 --durations=0 --numprocesses=auto.
  • Recommendation: —
  • Priority: High

• 32. Are those scripts named somewhere the agent will actually read them? If no scripts exist, FAIL with a one-line proof pointing at the task-scripts item. Otherwise check the agent instruction file, the README, and whether the manifest itself is self-explanatory.

  • Status: PASS
  • Proof: AGENTS.md names script/setup and python3 -m script.translations develop --integration <integration_name> directly, and routes to .vscode/tasks.json for the rest — ".vscode/tasks.json contains useful commands used for development" — where every task carries a human label. .github/PULL_REQUEST_TEMPLATE.md, which AGENTS.md also routes to, names python3 -m script.hassfest and python3 -m script.gen_requirements_all with the reason each is needed. script/ filenames are self-describing for the remainder. Not in README.rst — see the README item.
  • Recommendation: —
  • Priority: High

• 33. Can an agent get this project running — is there a reproducible environment or a documented setup path? Look for a pinned runtime (.nvmrc, .tool-versions, rust-toolchain), a lockfile, a container or nix file, and written setup steps. Try the first step if it is cheap and safe. This one bites hardest the moment work happens in a fresh git worktree or a new clone — the normal way to run agents in parallel. A new worktree has no installed dependencies, no .env, no build cache, so anything that works today only because of untracked state sitting on someone's machine simply does not run there. The test: would a bare checkout plus the written steps get this project up? Name any prerequisite nothing creates — an env file someone hand-made, a seeded database, a manual login — because each one is a wall a worktree hits on its first command.

  • Status: PASS
  • Proof: The worktree case is handled explicitly — AGENTS.md line 17: "When entering a new environment or worktree, run script/setup … This is required before committing", including the recovery step when uv is too old. Pinned runtime: .python-version 3.14.5 plus requires-python = ">=3.14.2". Pinned dependencies: requirements.txt, requirements_all.txt, requirements_test.txt and requirements_test_pre_commit.txt, all exact == and generated, with homeassistant/package_constraints.txt applied as a -c constraint. Container path: .devcontainer/devcontainer.json with "postCreateCommand": "git config --global --add safe.directory ${containerWorkspaceFolder} && script/setup" and a devcontainer-lock.json, plus Dockerfile.dev. Nothing hand-made is required — reading script/setup, it creates the venv, copies .vscode/settings.default.jsonc to settings.json if absent, creates config/, runs hass --script ensure_config, and appends a logger block; there is no .env in the design at all. Two walls worth naming: uv must be reachable (script/setup pip-installs it if missing, so this self-heals), and script/lint runs git merge-base upstream/dev HEAD, which fails in a fresh clone that has no upstream remote — the prek route named in AGENTS.md does not have this problem.
  • Recommendation: —
  • Priority: High

• 34. Can an agent see the results of a failed run — do the tools here produce output it can read and act on? Judge from the runs you already did in Verification: does a failure print a path, a line, a name — something actionable — or a wall of noise? If nothing could be run, FAIL saying why.

  • Status: PASS
  • Proof: From the runs done here, ruff reports path:line:col with a rule code and a fix hint, and ruff format --check names each unformatted file; the clean-checkout runs returned "All checks passed!" and "18192 files already formatted", so there is no baseline noise for an agent to filter out. CI goes further and normalises the rest: .github/workflows/matchers/ holds nine problem matchers — mypy.json, pylint.json, python.json, pytest-slow.json, yamllint.json, hadolint.json, codespell.json, check-json.json, check-executables-have-shebangs.json — each turning tool output into file-and-line annotations. pyproject.toml sets log_format to include %(filename)s:%(lineno)s for pytest, and pytest-github-actions-annotate-failures is pinned in requirements_test.txt.
  • Recommendation: —
  • Priority: Low

5. Safety

• 35. Are credentials kept out of the repo — nothing secret committed, ignore rules in place, an example env file for the shape? Three checks: grep tracked files for key-shaped strings, read the ignore rules for env and key patterns, and look for an example env file. Report each of the three separately.

  • Status: PASS
  • Proof: Check one, grep of tracked files for AKIA[0-9A-Z]{16}, -----BEGIN … PRIVATE KEY-----, ghp_, sk- and xox[baprs]- patterns: six hits, all source-code string literals that validate a user-supplied PEM — homeassistant/components/harbor/config_flow.py:49, homeassistant/components/mqtt/config_flow.py:5241-5244, homeassistant/components/weatherkit/config_flow.py:117. No committed key material. Check two, ignore rules: .gitignore covers .token (line 9), .env (line 116), and a # Secrets section with .lokalise_token (lines 128-129), alongside .venv/venv. Check three, example env file: none exists, and git ls-files | grep -iE '\.env|env\.example' is empty — but there is no env-var contract to document here, because runtime configuration comes from config/configuration.yaml, which script/setup generates with hass --script ensure_config. Two of three pass on the evidence and the third does not apply to this design.
  • Recommendation: —
  • Priority: High

• 36. Does anything scan for secrets automatically? Look for gitleaks, trufflehog, detect-secrets or this ecosystem's equivalent, wherever it is wired in — a CI step, a pre-commit hook, or forge-level push protection visible from the repo. CI is where this normally lives, and that is a PASS; a local hook on top is better, because it catches the key before it is pushed rather than after, but its absence is a line in the proof, not a FAIL. Say where the scan runs. "Nothing secret exists today" does not make this N/A — the scan is for the day that changes.

  • Status: FAIL
  • Proof: grep -rniE 'gitleaks|trufflehog|detect-secrets' .github .pre-commit-config.yaml returns nothing. No secret scanner runs in either place. .pre-commit-config.yaml has eleven hook sources and none of them scans for credentials. The nearest neighbours are not substitutes: codeql.yml is SAST and, in any case, fires only on schedule: cron "30 18 * * 4" — once a week, never on a pull request; zizmor --pedantic audits workflow permissions and action pinning, not committed keys. GitHub's own push protection may be enabled at the organisation level, but that is an account setting and is not visible from the clone, so it cannot be counted here.
  • Recommendation: Add the gitleaks hook to .pre-commit-config.yaml — the file already has a working hook chain and the CI prek job runs the whole set on every pull request, so those few lines buy local and pre-merge coverage in one change.
  • Priority: High

• 37. Are dependencies pinned, so a build is reproducible? Look for lockfiles in every package of the repo, exact versions for load-bearing dependencies, a pinned runtime, and an install command that respects the lock (npm ci, not npm install).

  • Status: PASS
  • Proof: Python: requirements.txt, requirements_all.txt, requirements_test.txt and requirements_test_pre_commit.txt are exact == throughout (aiohttp==3.14.3, attrs==26.1.0, prek==0.2.28), generated by script/gen_requirements_all.py and enforced by the gen-requirements-all CI job, with homeassistant/package_constraints.txt applied as a -c constraint on top. Runtime: .python-version 3.14.5. Node: tests/e2e/pnpm-lock.yaml. Container: .devcontainer/devcontainer-lock.json. Hooks: every .pre-commit-config.yaml repo carries an exact rev: tag. Actions: pinned to commit SHAs with a version comment (actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1), plus .github/aw/actions-lock.json, and zizmor in CI to catch any unpinned use. Determinism is itself tested by .github/workflows/check-requirements-deterministic.yml. One line worth naming: script/bootstrap installs with uv pip install … --upgrade, which is not lock-respecting in spirit — it resolves identically only because every requirement file is exact-pinned.
  • Recommendation: —
  • Priority: High

• 38. Is anything watching those dependencies for known vulnerabilities? Look for dependabot or renovate config, an audit step in CI, or this ecosystem's equivalent. Check every lockfile in the repo is covered, not just the root one.

  • Status: PASS
  • Proof: Both bots plus CI gates. .github/dependabot.yml watches github-actions daily with a seven-day cooldown. .github/renovate.json enables pep621, pip_requirements, pre-commit, dockerfile, npm, custom.regex and a homeassistant-manifest manager, with file patterns explicitly covering requirements*.txt, homeassistant/package_constraints.txt, Dockerfile and every homeassistant/components/*/manifest.json. In CI, .github/workflows/ci.yaml runs a dependency-review job on pull requests and an audit-licenses job, and codeql.yml runs weekly SAST. Lockfile coverage checked one by one: requirements*.txt covered by pip_requirements; tests/e2e/pnpm-lock.yaml covered by the npm manager, whose only exclusion is **/node_modules/**; .pre-commit-config.yaml covered by the pre-commit manager; .github/aw/actions-lock.json and the workflow SHAs covered by Dependabot. One gap: .devcontainer/devcontainer-lock.json matches no manager's file pattern, so devcontainer features are unwatched.
  • Recommendation: —
  • Priority: Low

• 39. Are the review rules written down — who reads a change, and what they check? Look for CONTRIBUTING, a PR template, or a review checklist in the agent instruction set. On a solo repo the "who" is N/A-shaped but the "what gets checked before it lands" still matters — judge that half.

  • Status: PASS
  • Proof: The "who" is generated and automatic: CODEOWNERS is 2,161 lines, produced by script/hassfest/codeowners.py, mapping every integration to its maintainers so reviewers are requested without anyone deciding. The "what" is a real checklist: .github/PULL_REQUEST_TEMPLATE.md requires that local tests pass ("Your PR cannot be merged unless tests pass"), no commented-out code, Ruff formatting, a filled manifest with python3 -m script.hassfest run, requirements_all.txt regenerated, a diff and changelog link for dependency bumps, documentation PRs linked, and "I understand the code I am submitting and can explain how it works." CONTRIBUTING.md sets the flow — fork, ensure tests work, PR against dev. The machine-side checklist is committed too, in .claude/skills/ha-review/SKILL.md ("Should be used when a review is requested to provide a consistent review behavior and output format") and .github/copilot-instructions.md.
  • Recommendation: —
  • Priority: Low

• 40. Are the operations that need a human named somewhere an agent will read them? Look for a "never without asking" list in the agent instruction file or the README. Docs that hand out production commands with no fence around them count against, and the proof should quote one.

  • Status: PASS
  • Proof: AGENTS.md closes with the fence, in the file loaded every session: "Autonomous contributions are not accepted: a human must review, understand, and be able to explain every change before it is submitted. Do not open issues or pull requests autonomously, and do not post comments on behalf of a user without their review." The routed-to AI_POLICY.md expands it — "We will close any pull requests or issues that we believe were created autonomously, and may mark automated comments as spam" — and CONTRIBUTING.md repeats it for humans. Two mechanical fences back the words: the no-commit-to-branch pre-commit hook with --branch=dev --branch=master --branch=rc, and .github/workflows/restrict-task-creation.yml. No document in the repo hands out an unfenced production command; the only publishing paths live in .github/workflows/builder.yml and wheels.yml, both gated on github.repository_owner == 'home-assistant' and neither runnable from a laptop.
  • Recommendation: —
  • Priority: High

• 41. Does every action that spends money, destroys data or changes production have a human in the way? Start from the damage, not from the tooling: list what in reach of this repo could charge a card, drop or overwrite data, or alter what users are running. Then trace the shortest route an agent could take to each one — a push that auto-deploys, a script carrying live credentials, a migration that runs on merge, an infrastructure apply with no plan-and-approve step. PASS when every route meets a human first, whether that is a review, a manual trigger or a protected environment. FAIL when even one route runs start to finish unattended, and quote that route in the proof so the fix is obvious.

  • Status: PASS
  • Proof: The damage in reach is: publishing container images to ghcr.io and Docker Hub (builder.yml), publishing wheels (wheels.yml), pushing strings to Lokalise (translations.yml), and altering the recorder database schema on user machines (homeassistant/components/recorder/migration.py, forward-only). Routes traced. Stable release: builder.yml fires on release: published — a human cuts the release. Nightly: builder.yml also fires on schedule: cron "0 2 * * *" and publishes the dev channel unattended, but it packages only what is already on dev, and nothing reaches dev except through a reviewed pull request; the same holds for wheels.yml on push: branches: [dev, rc]. Direct pushes are fenced by the no-commit-to-branch hook on dev/master/rc and by CODEOWNERS review requests. Every job is additionally gated on github.repository_owner == 'home-assistant', so a fork cannot publish. Stated plainly: branch protection is a repository setting and is not visible in a clone, so the in-repo evidence for the human gate is the commit hook, CODEOWNERS, and the PR-only flow in CONTRIBUTING.md — not the setting itself.
  • Recommendation: —
  • Priority: High

• 42. If a prompt injection landed tonight, how far would it reach — are the credentials an agent can get to here scoped to the job, with nothing production-grade in reach? Inventory what an agent in this repo can reach: env files, cloud CLI profiles, tokens named in docs or config, deploy commands that work from a laptop. Scoped-or-absent passes; production-grade reach fails with the item named.

  • Status: PASS
  • Proof: The blast radius is small by construction. No .env file exists or is expected — configuration comes from config/configuration.yaml generated by script/setup. No cloud CLI profile is named anywhere in the repo. Every powerful credential lives in GitHub Actions secrets and is unreadable from a checkout: DOCKERHUB_TOKEN, DOCKERHUB_USERNAME, WHEELS_KEY, LOKALISE_TOKEN, GIT_TOKEN, ISSUE_TRIAGE_APP_PEM, ISSUE_TRIAGE_APP_ID, CODECOV_TOKEN, COPILOT_GITHUB_TOKEN, GH_AW_GITHUB_TOKEN. Workflows declare permissions: {} at the top level and grant narrowly per job (contents: read, issues: write, security-events: write), and zizmor --pedantic runs in both pre-commit and CI to keep it that way. No deploy command works from a laptop. The one genuine reach is an authenticated gh: an injected instruction could open a pull request or post a comment as the user — which is exactly the act AI_POLICY.md forbids and AGENTS.md restates in-session.
  • Recommendation: —
  • Priority: High

• 43. Can a change reach production a slice at a time — a feature flag that defaults to off, a canary, a staged rollout — rather than everyone at once? Look for a flag system and check the default, or canary and staged-rollout config in the deploy pipeline. Flags that need a rebuild to flip are worth naming in the proof — they gate exposure but they are not a kill switch.

  • Status: PASS
  • Proof: The rollout is staged through release channels, visible in the pipeline: .github/workflows/builder.yml derives channel from the version (steps.version.outputs.channel) and gates dev-only publishing steps on needs.init.outputs.channel == 'dev', so a change lands first for nightly users, then for the beta/rc population, then for stable. A second slice exists at the product level: every integration is opt-in, so a new or changed integration reaches only users who have added it. homeassistant/helpers/issue_registry.py and the repairs component let a known problem be surfaced to exactly the affected installs without shipping code. The limit, named: there is no runtime feature-flag system and no kill switch — git ls-files finds no flag component, and moving between channels means installing a different version, so exposure is gated by which build a user runs, not by a value someone can flip.
  • Recommendation: —
  • Priority: High

• 44. Once a change is live, can anyone see what it is doing — logs, metrics, traces, alerts that fire on their own, and can an agent read them too? Look for logging setup, an error tracker, analytics, alerting config — and then ask the second half: could an agent reach any of it (a CLI, an MCP server, an API named in the docs), or does observability stop at a dashboard behind a login?

  • Status: FAIL
  • Proof: The first half is strong. In-product observability ships as first-class components: homeassistant/components/logger, system_log, system_health, diagnostics, repairs, analytics, and a sentry integration for error tracking; pyproject.toml configures structured pytest logging; CI uploads coverage to Codecov (codecov.yml, CODECOV_TOKEN). The second half fails. Nothing in this repo gives an agent a route to post-release signal: there is no CLI, no .mcp.json, and no API endpoint named anywhere in the instruction set, in README.rst, or in the skills for reading production behaviour back. The only channels are human-mediated — a user downloading a diagnostics file from the UI, or pasting logs into the form at .github/ISSUE_TEMPLATE/bug_report.yml. An agent finishing a change here has no way to ask what happened next.
  • Recommendation: Name the post-release signal routes in AGENTS.md — the public aggregate at analytics.home-assistant.io, and gh issue list --repo home-assistant/core --label <integration> for reports against an integration — so an agent can check what a shipped change is doing instead of stopping at the merge.
  • Priority: Low

• 45. Is there a way back — can a bad change be undone without a rebuild and a redeploy, including the ones that touched a database or a queue? Look for a documented rollback path, a revert-and-redeploy story, down-migrations, or a flag that can turn the change off at runtime. Deployment docs that only say how to go forward are worth quoting.

  • Status: FAIL
  • Proof: No rollback path is written down anywhere in the repo. grep -rniE 'rollback|roll back|downgrade' across the markdown, rst and yml outside homeassistant/components/ returns nothing but this report file. README.rst, CONTRIBUTING.md and AGENTS.md describe only how to go forward — "Create a Pull Request against the dev branch", "run script/setup" — and say nothing about undoing a shipped change. The database half is worse: homeassistant/components/recorder/migration.py is built around a single forward SCHEMA_VERSION with _apply_update steps and no downgrade path, so a user who installs a version that bumps the schema and then reverts is left with a database the older code did not write. Two real safety nets exist but are undocumented here: recovery_mode in homeassistant/bootstrap.py, which starts a minimal instance when setup fails, and out-of-repo backup and version-pinning routes on the user's side.
  • Recommendation: Add a short "Rolling back" section to CONTRIBUTING.md or AGENTS.md covering the three cases — revert the PR on dev, pin the previous version on an install, and restore a backup when the recorder schema has moved — and state plainly that recorder migrations are forward-only, since that is the trap.
  • Priority: Low

• 46. Does a past failure leave a check behind — is there a test or a rule in here that exists because something broke once? Look for tests, lint rules or comments that name the incident they guard against — "regression", "this broke when", a linked issue. On a very young repo with no failures yet, N/A with the age as the reason.

  • Status: PASS
  • Proof: The pattern is everywhere. git grep -c 'Regression test' across tests/ hits many integrations, including tests/components/airq/test_config_flow.py, tests/components/apple_tv/test_binary_sensor.py, tests/components/cert_expiry/test_sensor.py, tests/components/ecobee/test_climate.py (two) and tests/components/esphome/test_assist_satellite.py; tests/helpers/test_condition.py:5838 guards "failing cleanly rather than hanging on a regression". Beyond tests: homeassistant/block_async_io.py blocks blocking I/O inside the event loop, a rule that exists because that class of bug shipped; pylint/plugins/ holds custom checkers encoding HA-specific mistakes; tests/ignore_uncaught_exceptions.py is a standing allow-list built from real failures; and the filterwarnings block in pyproject.toml carries dated, per-library comments explaining each ignore, including "-- DeprecationWarning already fixed in our codebase" with the upstream issue link.
  • Recommendation: —
  • Priority: Low

Learn more: https://ainativesoftware.engineering/