langchain-ai/langchain

AI-Native Readiness Report

Repo: langchain-ai/langchain, a fresh clone of master at commit f9ee55d (2026-08-14) Date: 2026-08-14 Stack: Python monorepo (3,033 tracked files, 21 independently versioned packages under libs/). uv for dependencies and environments, make per package as the task runner, pytest for tests, ruff for lint and format, mypy for types, pre-commit for hooks, GitHub Actions for CI (27 workflows), PyPI for release. No root Makefile. Docs live in a separate repo (langchain-ai/docs).

Result: 28 passed, 18 failed, 0 not applicable.

How to read this. Every item is a numbered question about this repo, answered from evidence in this repo. Proof is what was actually found — a path, a file, a command and its output. An item with no proof is not an answer. The italic line under each question says how to check it; it stays in the report so the next run checks the same way.

FAIL vs N/A. FAIL means the practice would help here and it is missing — including when a whole artifact is missing, so several questions about it fail together with a one-line proof pointing at the item that found it missing. N/A means the question could never apply to a project of this kind, and the proof says why. "It does not exist" is a FAIL, never an N/A.

Priority is fixed by the template, not by the person filling it in. High means an agent working here will hit this on day one, or the risk is real. Low means it makes things better but nothing breaks without it.

Monorepo note. This audit is answered against the repo root. Per-package answers will differ — libs/core and libs/langchain_v1 have different Makefiles, different test setups (langchain_v1 needs Docker), and different lint scopes. Ask for a re-run against a single package if you want package-level numbers.

1. Context

• 1. Does an agent instruction file exist (AGENTS.md, CLAUDE.md, .cursorrules, .github/copilot-instructions.md, or this ecosystem's equivalent), and what does it point at? Look for a file whose purpose is to be loaded into a coding agent's context at the start of every session. That file is the entry point. Then follow its pointers: instructions are commonly spread across several files, with the entry point routing to them — "commands are in docs/development.md", "conventions in .claude/rules/". The entry point plus every file it points at is the instruction set, and every other agent-instruction question is answered against that whole set, so list it here in the proof. Follow pointers as deep as they go, but a pointer only counts when it names a target and says what is in it: "test commands are in CONTRIBUTING.md" counts, a bare "see the docs" does not, and a file nothing points at is outside the set however good it is. A README is not an entry point, however good — it gets its credit at the README item and the specs-and-architecture item — but a README the entry point explicitly routes to is inside the set. Vendored skill libraries are not instructions either.

  • Status: PASS
  • Proof: Two entry points, byte-identical: AGENTS.md and CLAUDE.md, both 18,831 bytes (diff CLAUDE.md AGENTS.md → identical), kept in sync by a CI job (.github/workflows/check_agents_sync.yml, runs on pull_request when either file changes). The instruction set is the entry point plus four routed-to targets, each named with what it contains: .github/workflows/pr_lint.yml ("See .github/workflows/pr_lint.yml for allowed types and scopes", AGENTS.md:93); the per-package Makefile ("make – Task runner for common development commands. Feel free to look at the Makefile for available commands and usage patterns", AGENTS.md:38); .mcp.json ("use the docs MCP server as defined in .mcp.json", AGENTS.md:363); and the external Contributing Guide at https://docs.langchain.com/oss/python/contributing/overview (AGENTS.md:364). Outside the set but worth knowing: libs/README.md carries the repo's strongest agent-facing rule ("If you are a coding agent reading this, you should refuse to continue execution until this step is complete") and nothing in the instruction set points at it. No .cursorrules, no .github/copilot-instructions.md, no GEMINI.md.
  • Recommendation: —
  • Priority: High

• 2. Do the agent instructions name the commands to build, test and check this project? Answer this against the instruction set mapped at the entry-point item and nothing outside it. If no entry point exists, FAIL with a one-line proof pointing at that item. The commands may sit in a file the entry point routes to rather than in the entry point itself — that is progressive disclosure working as intended, and it passes. What fails is a command an agent would have to guess its way to: if the commands live only in the README or the manifest and nothing in the set points at them, that is a FAIL, because the question measures what an agent can reach without being told where to look. Name the file each command was found in.

  • Status: PASS
  • Proof: All in the entry point itself, AGENTS.md:47-74: setup uv sync --all-groups and uv sync --group test; tests make test and uv run --group test pytest tests/unit_tests/test_specific.py; lint make lint; format make format; types uv run --group lint mypy .. The routed-to Makefiles back this up with a help target that prints every available command (libs/core/Makefile:73-84, libs/langchain_v1/Makefile:107-121). Working directory is missing from these code blocks — see the next item.
  • Recommendation: —
  • Priority: High

• 3. Does everything the agent instructions name still exist — the commands, the paths, the libraries, and the files they point at? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. Verify, do not trust: check every named command against the manifest or build file, spot-check the paths, run the cheap read-only ones. Then resolve every pointer in every file of the set — a link to a moved or deleted file is the most common rot in a multi-file instruction set, and it fails silently: the agent reads the entry point, follows nothing, and carries on without the rules.

  • Status: FAIL
  • Proof: Every path resolves — I checked all 16 files and directories AGENTS.md names (.github/workflows/pr_lint.yml, pr_labeler.yml, pr_labeler_backfill.yml, auto-label-by-package.yml, tag-external-issues.yml, _release.yml, integration_tests.yml, check_diffs.yml, .github/scripts/pr-labeler-config.json, .github/dependabot.yml, libs/standard-tests/langchain_tests/_langsmith_plugin.py, libs/core/tests/unit_tests/runnables/conftest.py, the three partner data/ dirs, .mcp.json, .vscode) and all 16 exist. The commands work too, in a package directory: uv sync --group test --group lint then make test TEST_FILE=tests/unit_tests/runnables/test_runnable.py in libs/core gave "113 passed, 8 skipped in 5.38s". What breaks is the working directory. There is no root Makefile — ls Makefile → "No such file or directory", and make test at the repo root prints make: *** No rule to make target 'test'. Stop. The libs/Makefile that does exist has only lock and check-lock, so make -C libs test fails too. AGENTS.md:57-74 shows make test, make lint and make format in bare code blocks with no cd, so an agent following the entry point verbatim from where it starts fails on its first command. The rule that fixes it ("from the root of the package(s) you've modified") is in .github/PULL_REQUEST_TEMPLATE.md, which nothing in the instruction set points at.
  • Recommendation: Add cd libs/<package> above the make code blocks in AGENTS.md:57-74, or add a root Makefile that forwards test, lint, format and type to the changed packages.
  • Priority: High

• 4. Are the agent instructions specific to this repo, rather than advice that would read the same in any codebase? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The test: could this be pasted into another project unchanged? "Write clean code" and "add tests for new features" would fit anywhere and count for nothing. Judge the whole set, but weigh the files differently: an entry point that is mostly a routing table is fine, even good, when what it routes to is specific — while generic filler in the entry point costs more than generic filler three hops down, because it is loaded into every session whether it is needed or not. Say which files carried the specifics.

  • Status: PASS
  • Proof: The specifics are in the entry point, which is where they earn the most. Examples that could not be pasted anywhere else: "This repository require actions to be pinned to a full-length commit SHA. Attempting to use a tag will fail" (AGENTS.md:359); the release workflow's numeric file ID, gh workflow run 63880841 (AGENTS.md:301); "The echo y | pipe is required when --data-dir is outside the libs/model-profiles working directory" (AGENTS.md:279); the exact three files a version bump touches, including "uv.lock — run uv lock from the package directory" (AGENTS.md:287-291); branch prefix <github-username>/<scope>/<short-description> with real examples (AGENTS.md:111-123); the env -u trick that unsets five tracing vars in libs/core/Makefile and the pytest11 entry-point plugin at libs/standard-tests/langchain_tests/_langsmith_plugin.py (AGENTS.md:340-342). Roughly 20% is portable filler — "Use descriptive, self-explanatory variable names", "Follow existing patterns", the Google-docstring example — but it does not crowd out the rest.
  • Recommendation: —
  • Priority: High

• 5. Do the agent instructions explain where things live and how this project is laid out? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. The map may live in a routed-to file. Judge coverage against the real tree, not against what the files mention: if the set maps one package well but is silent about sibling packages or directories an agent would land in, that is a FAIL with the omission named. In a multi-file set, check the routing too — a layout document nothing points at is a document the agent never opens.

  • Status: PASS
  • Proof: AGENTS.md:11-33 gives an ASCII tree plus a four-layer explanation (core / implementation / integration / testing). Checked against the real tree: it names every directory under libs/ that actually exists (core, langchain, langchain_v1, partners, text-splitters, standard-tests, model-profiles) and both non-libs directories an agent lands in (.github, .vscode). It also flags the two traps in the naming: libs/langchain/ is langchain-classic and legacy, libs/langchain_v1/ is the live langchain package. Only omission is .devcontainer/, which an agent rarely needs to edit. It even points outside the repo for related code: "Usually these repos are cloned at the same level as this monorepo, so if needed, you can refer to their code directly by navigating to ../langchain-google/" (AGENTS.md:32).
  • Recommendation: —
  • Priority: High

• 6. Do the agent instructions state the rules that are not obvious from the code — the things never to do here? If no entry point exists, FAIL with a one-line proof pointing at the entry-point item. These are the tribal-knowledge traps: the flag that must be exactly this string, the import that breaks the build, the directory that is generated and must not be edited. Rules in a routed-to file count. Two extra checks in a multi-file set: that the entry point signposts the rules clearly enough for an agent to open them before it needs them, since a trap found afterwards has already been sprung; and that the files do not contradict each other, because nothing tells the agent which one wins.

  • Status: PASS
  • Proof: Real traps, stated plainly: actions must be pinned to a full-length commit SHA or the build fails (AGENTS.md:359); --data-dir must point at the directory containing profile_augmentations.toml, "not the top-level package directory" (AGENTS.md:260) — and that file really does live in each partner's data/ dir, verified across eight partners; the echo y | pipe requirement (AGENTS.md:279); "do not create a GitHub release or tag manually" because mark-release does it (AGENTS.md:307); "Unit tests must never make network calls or send traces" with the env -u mechanism named (AGENTS.md:342); "All titles must include a scope with no exceptions — even for the main langchain package" (AGENTS.md:93); "Do not invoke pip, poetry, or conda directly" (AGENTS.md:78). All sit under signposted headings ("Environment and dependency management", "GitHub Actions & Workflows", "Release process") so an agent meets them before it needs them. I found no contradiction between the entry point and its routed-to files.
  • Recommendation: —
  • Priority: Low

• 7. Has the agent instruction file been updated recently enough to still be true, given how active the repo is? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Compare the last commit touching the file against the repo's tempo, then spot-check two or three of its claims against the code — a recently touched file can still lie.

  • Status: PASS
  • Proof: Repo tempo is high — the 50 commits in this shallow clone span 2026-08-07 to 2026-08-13, about 12 a day. git log -- AGENTS.md shows it touched in c9b301a on 2026-08-07, six days before HEAD. (Caveat: --depth 50 reaches back only to 2026-08-07, so there may be later touches I cannot see; the one visible touch is inside the window.) Spot-checked three claims: the env -u list of five tracing vars matches libs/core/Makefile exactly; libs/core/tests/unit_tests/runnables/conftest.py exists and holds the autouse fixture as described; the eight files listed under "Adding a new partner to CI" all exist. All three hold.
  • Recommendation: —
  • Priority: Low

• 8. Is the agent instruction file small enough to load into every session without crowding out the actual task? If no agent instruction file exists, FAIL with a one-line proof pointing at the entry-point item. Measure it: wc -c, and bytes divided by four is a fair token estimate. Under ~5k tokens is comfortable; past ~10k it is eating the context window.

  • Status: PASS
  • Proof: wc -c AGENTS.md → 18,831 bytes, about 4.7k tokens. Inside the comfortable band. Only one of AGENTS.md / CLAUDE.md loads per tool, so the duplication costs nothing at runtime.
  • Recommendation: —
  • Priority: Low

• 9. Can a fresh session pick up a half-finished task — is there a file or convention where progress, decisions and what is left get written down? This is not about the agent instruction file. Any durable place in-flight state gets written counts: a TODO.md, a plans or notes directory, task files, linked issues, a specs folder whose entries carry progress and open questions, an agent memory file or directory, a scratchpad or working-notes convention. The test is whether a fresh session could read it and know what was decided and what is left — not what the place is called. Git history alone does not count: it records what happened, not what was decided or what remains.

  • Status: PASS
  • Proof: The convention is linked GitHub issues, and it is enforced, not aspirational. .github/ISSUE_TEMPLATE/task.yml is a maintainer task form with three required-shape sections: "Acceptance Criteria" (checkbox list, required: true), "Dependencies" with an explicit "Blocked by" field, and "Context and Background" with related issues and PRs. .github/workflows/require_issue_link.yml runs on pull_request and forces every external PR to link an approved issue; AGENTS.md:129-139 requires the Closes #123 keyword on the PR's first line; close_unchecked_issues.yml auto-closes issues that skip the checkboxes. So a fresh session handed a branch can read the PR description, follow Closes #N, and see the acceptance criteria, what is ticked, and what is blocking. Reachable with gh issue view, and AGENTS.md already documents gh use. Caveat: no in-repo place exists — no TODO.md, no plans/, no notes/ in 3,033 tracked files — and nothing in the instruction set tells an agent to go read the linked issue for state.
  • Recommendation: —
  • Priority: Low

• 10. Is there a README that says what this project is and how to run it? This is where a good README earns its credit. It needs three things: what the project is, how to run it, and how to check a change. Judge what is on the page, not the file's existence.

  • Status: FAIL
  • Proof: Two of the three are there. What it is: "LangChain is a framework for building agents and LLM-powered applications" plus an ecosystem map (README.md:26-70). How to run it: uv add langchain and a five-line init_chat_model snippet (README.md:34-43). How to check a change: nothing. README.md names no test, lint, format or type command anywhere in its 5,932 bytes — it links out to https://docs.langchain.com/oss/python/contributing/overview (README.md:88) and stops there. An agent that lands on the README and has no network has no verification path, even though make format lint test is one line away in libs/core/Makefile.
  • Recommendation: Add a four-line "Development" section to README.md pointing at cd libs/core && make format lint test and at AGENTS.md for the full guidelines.
  • Priority: High

2. Specs

• 11. Is the thinking behind this system written down somewhere durable — a specs, RFC, proposals, design-doc or ADR directory, or architecture notes that record not just what but why? Look for the place and for the convention: specs/, docs/adr/, rfcs/, proposals/, .specify/, a docs or design folder, architecture notes anywhere in the tree, or this ecosystem's equivalent. Judge substance, not location: "we use X because Y" is a decision, a list of technologies is not, an essay about specs is not a spec, and a docs folder of usage guides with no reasoning is a FAIL whose proof says what was in there instead. Other items are answered against whatever this item finds, so name it precisely — and where forward-looking specs and after-the-fact architecture records live in different places, name both, since a decision log cannot answer a question about acceptance criteria.

  • Status: FAIL
  • Proof: No specs/, rfcs/, docs/adr/, proposals/, .specify/, design/ or architecture/ directory anywhere in the 3,033 tracked files (git ls-files | grep -iE "(^|/)(adr|rfc|proposal|spec|design|architecture)" returns only library source such as langchain_classic/agents/agent_toolkits/openapi/spec.py). There is no docs/ directory at all — prose docs live in the separate langchain-ai/docs repo. Architecture notes exist but are thin, and only one sentence in the whole repo records a why: "Most integrations have been moved to their own repositories for improved versioning, dependency management, collaboration, and testing" (libs/README.md:33). Everything else is what, not why — AGENTS.md:11-33 is a tree plus one-line layer labels, and the 14 package READMEs are install-and-link pages. Nothing anywhere explains the decision an agent trips over first: why libs/langchain_v1/ ships as langchain while libs/langchain/ ships as langchain-classic, or what the migration plan is.
  • Recommendation: Create docs/decisions/ (or libs/ARCHITECTURE.md) and write the three decisions an agent needs most: the langchain_v1 / langchain-classic split and its migration plan, why partner integrations moved out to their own repos, and why standard-tests is a shipped package rather than a test directory. Then point AGENTS.md at it.
  • Priority: High

• 12. Does the specs directory hold recent entries, or is it an archive nobody has touched? If no specs directory exists, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise compare the newest entry's date against the repo's recent activity.

  • Status: FAIL
  • Proof: No specs or architecture-decision directory exists — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs directory.
  • Priority: Low

• 13. Is there a spec template, or an SDD framework, so every spec comes out the same shape? Scaffolding can exist even where no specs directory does — look for a TEMPLATE.md, a .specify/ directory, or framework config. If neither a directory nor any scaffolding exists, FAIL.

  • Status: FAIL
  • Proof: No TEMPLATE.md, no .specify/, no SDD framework config anywhere in the tree. The closest scaffolding is issue and PR shaping, not spec shaping: .github/ISSUE_TEMPLATE/task.yml forces Acceptance Criteria, Dependencies and Context onto every maintainer task, and .github/PULL_REQUEST_TEMPLATE.md forces a Fixes # line and a ## Release note section. Both are good, and neither gives a spec a shape, because there are no specs.
  • Recommendation: Once docs/decisions/ exists, add a TEMPLATE.md beside it with four headings — Context, Decision, Consequences, Non-goals — and reuse the Acceptance Criteria wording already in .github/ISSUE_TEMPLATE/task.yml.
  • Priority: Low

• 14. Do the specs state acceptance criteria a machine could check? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise open the two newest specs and quote a criterion: "the endpoint returns 403 for expired tokens" is checkable; "the feature works well" is not.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item. Machine-checkable criteria are required elsewhere, in .github/ISSUE_TEMPLATE/task.yml ("Define the criteria that must be met for this task to be considered complete", a required checkbox list), but that shapes issues, not specs.
  • Recommendation: Blocked by the missing specs directory.
  • Priority: High

• 15. Open the newest spec: do its criteria go past the happy path — what happens when a step fails, and how the change gets undone? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Look for error cases, edge inputs, and a rollback or undo story, not just the success flow.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item.
  • Recommendation: Blocked by the missing specs directory.
  • Priority: Low

• 16. Do the specs state non-goals, so an agent knows where to stop? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Non-goals written elsewhere (a README's "what this is not" list) are worth naming in the proof, but they do not turn this into a PASS — the question is whether specs carry them.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item. Non-goals do appear elsewhere and they are strong: "langchain/ — langchain-classic (legacy, no new features)" (AGENTS.md:16), "PRs should not touch more than one package unless absolutely necessary" and "Do not update the uv.lock files or add dependencies to pyproject.toml files (even optional ones) unless you have explicit permission" (.github/PULL_REQUEST_TEMPLATE.md). Per the steer, those do not make this a PASS.
  • Recommendation: Blocked by the missing specs directory.
  • Priority: Low

• 17. Can recent shipped work be traced back to a spec? If no specs exist, FAIL with a one-line proof pointing at the specs-and-architecture item. Otherwise take the last few substantial commits or PRs and look for a reference to a spec, an issue, or a design doc in the message or description.

  • Status: FAIL
  • Proof: No specs exist — see the specs-and-architecture item. Traceability to issues is real and enforced: of the last 15 commits, 7fb045b fix(fireworks): avoid repeated 'service_tier' [closes #39619] links an issue inline, every commit carries its PR number, and .github/workflows/require_issue_link.yml blocks external PRs with no linked approved issue. What is missing is the layer above — nothing traces back to a written design decision.
  • Recommendation: Blocked by the missing specs directory.
  • Priority: Low

3. Verification

• 18. Does this project have an automated test suite, in whatever form this ecosystem uses? Work out this ecosystem's convention before concluding anything is missing — check the manifest, the build file, the CI config, the README. A shell script that diffs output files is a test suite. If you find one, run it and record the result.

  • Status: PASS
  • Proof: pytest, per package, split into tests/unit_tests/ (network blocked) and tests/integration_tests/ (network allowed), as documented at AGENTS.md:194-196. Ran it: uv sync --group test --group lint then make test TEST_FILE=tests/unit_tests/runnables/test_runnable.py PYTEST_EXTRA=-q in libs/core → "113 passed, 8 skipped, 10 warnings in 5.38s". The Makefile target runs pytest -n auto --benchmark-disable --disable-socket --allow-unix-socket, so network isolation is enforced by the runner, not by convention. Beyond unit tests there are extended tests (make extended_tests), Pydantic-compatibility tests (_test_pydantic.yml), VCR cassette replay tests (_test_vcr.yml), CodSpeed benchmarks (make benchmark), and a shared cross-integration suite shipped as its own package (libs/standard-tests).
  • Recommendation: —
  • Priority: High

• 19. Can the test command be discovered without guessing — is it written down where an agent reads? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise check the places an agent looks: the agent instruction file, the README, the manifest's scripts or targets.

  • Status: PASS
  • Proof: In three of the places an agent looks. The entry point: make test and uv run --group test pytest tests/unit_tests/test_specific.py at AGENTS.md:57-63. The task runner itself: every package Makefile has a help target that prints the list (libs/langchain_v1/Makefile:107-121 lists test, test_fast, coverage, extended_tests, integration_tests, benchmark). The PR template: "Run make format, make lint and make test from the root of the package(s) you've modified." Not in README.md — see the README item.
  • Recommendation: —
  • Priority: High

• 20. Do the tests assert real behaviour, rather than asserting that a mock was called? If no test suite exists, FAIL with a one-line proof pointing at the test-suite item. Otherwise open the largest test files and read the assertions: calling real code on real inputs passes; expect(mock).toHaveBeenCalled() as the main dish fails.

  • Status: PASS
  • Proof: Counted across libs/core/tests/unit_tests/: 4,382 bare assert statements against 4 uses of assert_called*. A ratio of about 1,000 to 1. Read the assertions in libs/core/tests/unit_tests/language_models/chat_models/test_base.py: they invoke real code and compare real values — assert model.invoke("hello").content == "foo" (line 128), assert model.asdict() == expected (line 105), assert get_type_hints(BaseChatModel.asdict)["return"] == dict[str, Any] (line 111), and a run of assertions inspecting actual traced-run metadata (assert all((r.extra or {}).get("batch_size") == 1 for r in cb.traced_runs)). Mocks appear as unittest.mock.patch for external boundaries, which is what the instructions ask for ("Use fixtures/mocks for external dependencies", AGENTS.md:203).
  • Recommendation: —
  • Priority: High

• 21. Is there a linter or static analysis configured for this language, and does it pass on a clean checkout? Configured is not enough — run it. A linter that exits non-zero on an untouched checkout is a FAIL with the error count in the proof, because an agent cannot tell its own damage from the baseline noise.

  • Status: PASS
  • Proof: ruff, configured aggressively — select = ["ALL"] in libs/core/pyproject.toml:100 with a short, commented ignore list, plus ban-relative-imports = "all" and Google pydocstyle convention. On top of that a custom import-boundary script, libs/core/scripts/lint_imports.sh, wired into the Makefile's lint target. Ran all three on the untouched checkout: ruff check . → "All checks passed!"; ./scripts/lint_imports.sh → exit 0. Zero baseline noise, so an agent can trust a non-zero exit as its own damage.
  • Recommendation: —
  • Priority: High

• 22. Is there a formatter, so an agent's diffs do not churn on style? Look for the config file and the dependency in this ecosystem's form — .prettierrc, rustfmt, gofmt, black, an .editorconfig doing real work. If the language ships one formatting standard with the toolchain, that is a PASS and the proof says so.

  • Status: PASS
  • Proof: ruff format, configured at [tool.ruff.format] in libs/core/pyproject.toml:96 with docstring-code-format = true, exposed as make format and checked in lint mode via ruff format --diff. Ran it on the clean checkout: "353 files already formatted". Reinforced by an .editorconfig that does real work (per-extension indent and line length, max_line_length = 88 for Python, trim_trailing_whitespace = false for notebooks and Markdown), .markdownlint.json for prose, and pre-commit texthooks that normalise smart quotes and non-breaking spaces (.pre-commit-config.yaml).
  • Recommendation: —
  • Priority: Low

• 23. Is there a compile-time or type-level gate, if this language offers one? N/A only when the language genuinely has no such gate. If the language offers one and the repo does not use it — no strict mode, no typecheck script, no compiler step — that is a FAIL. Run the gate if it exists and record the result.

  • Status: PASS
  • Proof: mypy, run as part of make lint and available alone as make type. AGENTS.md:169 makes it a rule: "All Python code MUST include type hints and return types." Ran it: uv run --all-groups mypy . --cache-dir .mypy_cache in libs/core → "Success: no issues found in 353 source files". CI runs it too, via make lint_package and make lint_tests in .github/workflows/_lint.yml.
  • Recommendation: —
  • Priority: Low

• 24. Can an agent prove its own work before it pushes — one command, task-runner target or commit hook that runs every check this project has? One command, not a list to remember: a check or verify target, a precommit script, a Makefile target that chains them. Separate commands documented side by side are close but FAIL — the question is whether the agent can run the whole gauntlet without knowing its parts.

  • Status: FAIL
  • Proof: No single command exists. Every package Makefile keeps format, lint, type, test, check_imports and check_version as separate targets and defines all: help — so make alone prints a menu instead of running anything (libs/core/Makefile:6, libs/langchain_v1/Makefile:6). There is no root Makefile at all. The instructions hand the agent the list to remember rather than a command: AGENTS.md:57-74 shows make test, make lint, make format and mypy . as four separate blocks, and the PR template says "Run make format, make lint and make test". Closest thing is pre-commit run --all-files over .pre-commit-config.yaml, which chains format, lint and version checks for every changed package — but it runs no tests, and nothing in the instruction set mentions pre-commit exists.
  • Recommendation: Add a root Makefile with check: format lint type test that forwards to the changed packages, and name it in AGENTS.md in place of the four separate blocks at lines 57-74.
  • Priority: High

• 25. Is there continuous integration, and does it run before a change lands rather than after? A deploy pipeline that runs on push to the default branch runs after the change has landed — that is a FAIL with the trigger quoted, not a PASS with a caveat. Look for a pre-merge trigger: pull_request events, merge trains, or this forge's equivalent.

  • Status: PASS
  • Proof: .github/workflows/check_diffs.yml is the primary pipeline and its trigger block is on: push: branches: [master], pull_request:, merge_group: (lines 16-20). Both pre-merge gates are present — pull_request and a merge queue. Twelve more workflows also fire on pull_request: _test.yml, check_agents_sync.yml, block_fork_main_prs.yml, check_versions.yml, check_extras_sync.yml, check_release_deps.yml, codspeed.yml, pr_labeler.yml, pr_lint.yml, pr_lint_trailer.yml, tag-external-issues.yml, require_issue_link.yml.
  • Recommendation: —
  • Priority: High

• 26. Does CI actually run the tests and the checks that exist in this repo? If no pipeline of any kind exists, FAIL with a one-line proof pointing at the CI item. Otherwise list what the pipeline runs and diff it against every check found in the Verification section — name each check that exists in the repo but is missing from the pipeline.

  • Status: PASS
  • Proof: check_diffs.yml computes a changed-package matrix (.github/scripts/check_diff.py) and then calls: _lint.yml (make lint_package and make lint_tests → ruff check, ruff format --diff, mypy), _test.yml (make test plus a second run against minimum resolved dependency versions), _test_pydantic.yml, _compile_integration_test.yml, _test_vcr.yml, an inline make extended_tests job, and a release-options validation job — gated behind a ci_success required check. Separate PR-triggered workflows add check_versions, check_extras_sync, check_release_deps, check_agents_sync, pr_lint, pr_lint_trailer, codspeed and require_issue_link. Diffed against every check found in this section: ruff, ruff format, lint_imports (inside make lint), mypy, pytest unit, extended, pydantic, VCR and benchmarks are all in CI. One gap worth naming: the "🔒 Verify Lockfile is Up-to-Date" step in _lint.yml:50-55 is commented out, so make -C libs check-lock never runs in CI — but lock staleness still fails the build, because check_diffs.yml:36-37 sets UV_FROZEN: "true" and UV_NO_SYNC: "true" at workflow level, which makes uv error on a lockfile out of step with pyproject.toml. Capability covered.
  • Recommendation: —
  • Priority: High

• 27. Does anything here get an LLM to read a change and go looking for problems — a review skill, a saved review prompt, or an AI reviewer on the pull requests? Look for a committed review skill or slash command, a CI step calling an AI reviewer, or bot config. Human review rules are a Safety item; this one is specifically about machine review.

  • Status: FAIL
  • Proof: Nothing reviews a change with an LLM. Grepped claude|copilot|codex|coderabbit|cursor|ai.review|greptile|sourcery across .github/ and got three kinds of hit, none of them a reviewer: check_agents_sync.yml (a diff -q between two files), pr_lint_trailer.yml:164 which strips Claude-authoring trailers out of PR descriptions, and the GitHub.copilot VS Code extension recommendation in .devcontainer/devcontainer.json. No committed review skill or slash command either — see the skills-library item. AGENTS.md:149 asks contributors to "Add a brief disclaimer noting AI-agent involvement in the contribution", which is disclosure, not machine review.
  • Recommendation: Add a review step to check_diffs.yml that runs an LLM over the PR diff against the rules already written in AGENTS.md — public-API stability, type hints on every function, Google docstrings, no bare except: — since those are the checks ruff and mypy cannot make.
  • Priority: Low

4. Tooling

• 28. Can an agent reach the outside systems this project depends on, with that access committed to the repo rather than set up per laptop? MCP servers are one form: look for a committed .mcp.json or this ecosystem's equivalent. A CLI is another and often the better one — aws, gh, psql, kubectl, stripe, a vendor's own tool — and it counts when the repo names which tools the work needs and how to authenticate, so an agent is not guessing at a tool it cannot see. Judge the axis, not the mechanism: access that exists only in someone's shell history or laptop config is a FAIL, and so is a committed config that turns out to be gitignored, with the distinction in the proof. Project task scripts have their own item; this one is about reaching past the repo's edge. If this project genuinely talks to nothing outside itself, N/A with that as the reason.

  • Status: PASS
  • Proof: Both mechanisms, committed. .mcp.json is tracked (git ls-files --error-unmatch .mcp.json succeeds, git check-ignore finds no rule) and declares two HTTP MCP servers: docs-langchain at https://docs.langchain.com/mcp and reference-langchain at https://reference.langchain.com/mcp. AGENTS.md:363 routes to it and says when to use it. On the CLI side, gh is named for the three jobs that need it — triggering releases (gh workflow run 63880841 ...), reading run results (gh run view <run-id> --repo langchain-ai/langchain), and resolving action tags to commit SHAs ("Use the gh cli to query. Verify tags are not annotated tag objects") — and .devcontainer/devcontainer.json installs it via ghcr.io/devcontainers/features/github-cli:1, so the container has it without per-laptop setup. Gap worth a line: nothing documents how to authenticate gh, and nothing names the provider API keys an integration test needs beyond the placeholder shapes in .env.example.
  • Recommendation: —
  • Priority: Low

• 29. Is there a skills, commands or reusable-prompt library in the repo? Look for .claude/skills, .agents/skills, .claude/commands, or a prompts directory. Committed and pinned beats committed; note whether anything ties the copies to a source.

  • Status: FAIL
  • Proof: None of the four exist. No .claude/, no .agents/, no .claude/commands/, no top-level prompts directory anywhere in the 3,033 tracked files. The prompts/ hits in git ls-files are all library source or test fixtures — libs/core/langchain_core/prompts/, libs/langchain/langchain_classic/indexes/prompts/, libs/core/tests/unit_tests/data/prompts/*.json — code that builds prompts for users, not prompts for agents working on this repo.
  • Recommendation: Add .claude/skills/ and start with the two chores AGENTS.md currently describes in prose: a release skill wrapping the three-file version bump plus gh workflow run, and an add-a-partner skill covering the eight-file CI checklist at AGENTS.md:346-356.
  • Priority: Low

• 30. Do the skills, servers and tools that exist cover the work this team plainly repeats? First identify the repeated work from the README and the commit history. If skills, servers or documented tooling exist but miss it, FAIL naming the gap. If none exist at all and the repo plainly repeats work, FAIL pointing at the outside-systems item and the skills-library item. If the project is too small to repeat anything, N/A.

  • Status: FAIL
  • Proof: The repeated work is obvious from the history — 3 of the last 15 commits are releases (fb8853d release(openai): 1.5.0, c5b2f95 release(anthropic): 1.5.6) across 21 independently versioned packages, and 162f9d9 chore(model-profiles): refresh model profile data is a recurring data refresh. Tooling covers the machine half well: _release.yml automates build through publish, refresh_model_profiles.yml automates the profile refresh, .github/scripts/*.py handle diff detection and version checks. The human half is prose. A release still means hand-editing three files in lockstep (_version.py, pyproject.toml, uv.lock) with a warning to manually revert unrelated uv.lock marker lines (AGENTS.md:287-291) — steps a script should own. And "Adding a new partner to CI" has no automation at all: AGENTS.md:346-356 lists eight files to edit by hand, which is exactly the kind of checklist that silently goes stale.
  • Recommendation: Write .github/scripts/bump_version.py <package> <version> to do the three-file bump and lock refresh in one call, and .github/scripts/scaffold_partner.py <name> to apply the eight-file CI checklist; then replace the prose at AGENTS.md:287-291 and 346-356 with those two commands.
  • Priority: Low

• 31. Are there project CLI scripts or task-runner targets for the common jobs? Check the manifest's scripts, the Makefile, the justfile, or this ecosystem's equivalent. The test: does routine work need a raw multi-flag command that someone has to remember?

  • Status: PASS
  • Proof: A Makefile per package, and they are thorough. libs/langchain_v1/Makefile declares format, lint, lint_diff, lint_package, lint_tests, type, test, test_fast, test_watch, extended_tests, integration_tests, coverage, coverage_agents, benchmark, check_imports, check_version, plus start_services/stop_services that bring up Postgres and Redis via docker compose. Each one hides a genuinely gnarly command — make test_fast stands in for LANGGRAPH_TEST_FAST=1 uv run --group test pytest -n auto --benchmark-disable --disable-socket --allow-unix-socket tests/unit_tests/. libs/Makefile adds cross-package lock and check-lock. .github/scripts/ holds six Python helpers for CI chores. Routine work needs no remembered flags.
  • Recommendation: —
  • Priority: High

• 32. Are those scripts named somewhere the agent will actually read them? If no scripts exist, FAIL with a one-line proof pointing at the task-scripts item. Otherwise check the agent instruction file, the README, and whether the manifest itself is self-explanatory.

  • Status: PASS
  • Proof: In the entry point (make test, make lint, make format at AGENTS.md:57-74, with AGENTS.md:38 routing to the Makefile for the rest) and in the task runner itself — every Makefile ends with a help target that prints each target and a one-line description, and all: help makes that the default, so a bare make in any package is self-documenting. Also in .github/PULL_REQUEST_TEMPLATE.md step 3. Not in README.md, and the AGENTS.md blocks omit the working directory — see the pointer-rot item.
  • Recommendation: —
  • Priority: High

• 33. Can an agent get this project running — is there a reproducible environment or a documented setup path? Look for a pinned runtime (.nvmrc, .tool-versions, rust-toolchain), a lockfile, a container or nix file, and written setup steps. Try the first step if it is cheap and safe. This one bites hardest the moment work happens in a fresh git worktree or a new clone — the normal way to run agents in parallel. A new worktree has no installed dependencies, no .env, no build cache, so anything that works today only because of untracked state sitting on someone's machine simply does not run there. The test: would a bare checkout plus the written steps get this project up? Name any prerequisite nothing creates — an env file someone hand-made, a seeded database, a manual login — because each one is a wall a worktree hits on its first command.

  • Status: PASS
  • Proof: Tested it rather than assumed it. On this bare clone, with no .env and no prior state, the written steps worked: uv sync --group test --group lint in libs/core resolved and installed, then make test ran green. Backing it up: 21 uv.lock files, one per package; UV_FROZEN = true exported from every Makefile so installs respect the lock; requires-python = ">=3.10.0,<4.0.0" bounding each package; a full container path in .devcontainer/devcontainer.json with postCreateCommand: cd libs/langchain_v1 && uv sync plus a Codespaces one-click badge in .devcontainer/README.md. Two prerequisites a fresh worktree hits, both with documented escapes: libs/langchain_v1's make test calls make start_services, which needs a running Docker daemon for compose-postgres.yml and compose-redis.yml — make test_fast is the in-memory alternative; and integration tests need real provider keys, whose shape is given in libs/langchain/tests/integration_tests/.env.example but whose values nothing can create. No global Python pin, which is deliberate: "do not pin a global Python version" (AGENTS.md:81).
  • Recommendation: —
  • Priority: High

• 34. Can an agent see the results of a failed run — do the tools here produce output it can read and act on? Judge from the runs you already did in Verification: does a failure print a path, a line, a name — something actionable — or a wall of noise? If nothing could be run, FAIL saying why.

  • Status: PASS
  • Proof: Forced a real failure with a three-line probe file and both gates pointed straight at it. ruff: ANN202 Missing return type annotation for private function 'f' with --> _aine_probe.py:1:5, a caret under the symbol, and a help: Add return type annotation line. mypy: _aine_probe.py:2: error: Incompatible types in assignment (expression has type "str", variable has type "int") [assignment] — path, line, both types, and a machine-usable error code. pytest names each test and prints a "slowest 5 durations" table. All three give an agent a path, a line and a next action. (Probe file removed.)
  • Recommendation: —
  • Priority: Low

5. Safety

• 35. Are credentials kept out of the repo — nothing secret committed, ignore rules in place, an example env file for the shape? Three checks: grep tracked files for key-shaped strings, read the ignore rules for env and key patterns, and look for an example env file. Report each of the three separately.

  • Status: PASS
  • Proof: All three clean. (1) Nothing committed: git grep -nIE "(sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----)" across all tracked files returned no hits outside test fixtures. (2) Ignore rules are deliberate, not boilerplate — .gitignore:115-146 covers .env, .env.* with a !.env.example re-include, .envrc, *.pem, *.key, credentials.json, .git-credentials, and carries a comment showing someone thought about the edge case: "SSH private keys (no file extension, so *.key never matches them)". Lines 35-38 add the Google GitHub Actions credential files. (3) Shape documented: libs/langchain/tests/integration_tests/.env.example lists OPENAI_API_KEY, SEARCHAPI_API_KEY, the Power BI and Astra DB variables, all with placeholder values and a link to where each key comes from.
  • Recommendation: —
  • Priority: High

• 36. Does anything scan for secrets automatically? Look for gitleaks, trufflehog, detect-secrets or this ecosystem's equivalent, wherever it is wired in — a CI step, a pre-commit hook, or forge-level push protection visible from the repo. CI is where this normally lives, and that is a PASS; a local hook on top is better, because it catches the key before it is pushed rather than after, but its absence is a line in the proof, not a FAIL. Say where the scan runs. "Nothing secret exists today" does not make this N/A — the scan is for the day that changes.

  • Status: FAIL
  • Proof: No scan anywhere. Grepped gitleaks|trufflehog|detect-secrets|secret.?scan|codeql|bandit|pip-audit|safety|osv|snyk across all of .github/ and .pre-commit-config.yaml; the only three hits were the English word "safety" in _release.yml:176, _release.yml:431 and check_diff.py:307. The 27 workflows include no scanning step, and .pre-commit-config.yaml's 40-odd hooks are all formatters, YAML/TOML validators, text normalisers and per-package lint or version checks. Forge-level push protection may be enabled at the org level, but nothing in the repo shows it, so I cannot confirm it either way. This matters more than usual here because libs/partners/openai/scripts/RECORD_CODEX_CASSETTES.md documents a workflow where a developer holds a live ChatGPT OAuth token bundle on disk and commits scrubbed VCR cassettes — a scrubbing step with no automated backstop.
  • Recommendation: Add a gitleaks job to .github/workflows/check_diffs.yml and a gitleaks hook to .pre-commit-config.yaml, so the cassette-scrubbing flow in RECORD_CODEX_CASSETTES.md has a net under it.
  • Priority: High

• 37. Are dependencies pinned, so a build is reproducible? Look for lockfiles in every package of the repo, exact versions for load-bearing dependencies, a pinned runtime, and an install command that respects the lock (npm ci, not npm install).

  • Status: PASS
  • Proof: 21 uv.lock files for 21 packages — full coverage, verified by diffing the lockfile directory list against the package list. The lock is enforced, not merely present: every Makefile sets UV_FROZEN = true under .EXPORT_ALL_VARIABLES:, and check_diffs.yml:36-37 sets UV_FROZEN: "true" plus UV_NO_SYNC: "true" for the whole pipeline, so a stale lock fails CI. libs/Makefile provides check-lock (uv lock --check) across the five core packages. requires-python = ">=3.10.0,<4.0.0" bounds the interpreter per package. CI actions are pinned to full commit SHAs, stated as a rule (AGENTS.md:359) and confirmed in practice (actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1). _test.yml also runs the suite a second time against minimum resolved versions, which catches over-loose floors that a lockfile alone would hide. One line worth noting: the explicit lockfile-verify step in _lint.yml:50-55 is commented out, so freshness rests on UV_FROZEN rather than an obvious named check.
  • Recommendation: —
  • Priority: High

• 38. Is anything watching those dependencies for known vulnerabilities? Look for dependabot or renovate config, an audit step in CI, or this ecosystem's equivalent. Check every lockfile in the repo is covered, not just the root one.

  • Status: PASS
  • Proof: .github/dependabot.yml covers the github-actions ecosystem at / and the uv ecosystem across explicit directory lists, monthly, with minor-and-patch grouped separately from major. Checked coverage rather than assuming it: the file names 21 /libs/... directories and the repo has 21 uv.lock files, and comm -13 between the two sorted lists is empty — every lockfile is watched, none missed. Internal packages are correctly excluded from bumps via the ignore list (langchain-core, langchain, langchain-classic, langchain-text-splitters, langchain-tests, langchain-model-profiles), so Dependabot does not fight the workspace sources. No pip-audit or CodeQL step on top, which would add vulnerability scanning rather than just version drift — worth having, but the item's capability is covered.
  • Recommendation: —
  • Priority: Low

• 39. Are the review rules written down — who reads a change, and what they check? Look for CONTRIBUTING, a PR template, or a review checklist in the agent instruction set. On a solo repo the "who" is N/A-shaped but the "what gets checked before it lands" still matters — judge that half.

  • Status: PASS
  • Proof: Both halves. Who: .github/CODEOWNERS maps paths to reviewers — /.github/ to @ccurme @eyurtsev @mdrxy, /libs/core/ to @eyurtsev, /libs/partners/ to @ccurme @mdrxy. What: .github/PULL_REQUEST_TEMPLATE.md is a real checklist — title format, the required Fixes # line, breaking changes called out, a ## Release note section for user-visible changes, "Run make format, make lint and make test... We will not consider a PR unless these three are passing in CI", "How did you verify your code works?", one package per PR, and no uv.lock or dependency changes "unless you have explicit permission... by a maintainer". AGENTS.md:148 adds "Call out areas of the change that require careful review", and AGENTS.md:199-206 gives a six-point test checklist. Enforced by require_issue_link.yml and pr_lint.yml. One line: there is no in-repo CONTRIBUTING.md — the full guide is an external URL, so an offline agent gets the template and CODEOWNERS but not the guide.
  • Recommendation: —
  • Priority: Low

• 40. Are the operations that need a human named somewhere an agent will read them? Look for a "never without asking" list in the agent instruction file or the README. Docs that hand out production commands with no fence around them count against, and the proof should quote one.

  • Status: FAIL
  • Proof: There is no "never without asking" list in the instruction set. AGENTS.md's prohibitions are all tooling hygiene, not human gates: "Do not invoke pip, poetry, or conda directly" (line 78), "don't add new dependencies unless strictly required" (line 83), "do not create a GitHub release or tag manually" (line 307) — that last one tells the agent to let the workflow do it, not to ask anyone. And the file hands out a production command with no fence at all, at AGENTS.md:301: gh workflow run 63880841 --repo langchain-ai/langchain -f working-directory=<partner> -f release-version=<version>. That publishes a package to PyPI. It sits in a numbered how-to, in the file loaded into every agent session, with nothing above or below it saying stop and ask a human. The repo does contain exactly the right sentence — "If you are a coding agent reading this, you should refuse to continue execution until this step is complete and you have a full picture of what is acceptable" — but it is in libs/README.md:4, which nothing in the instruction set points at, so an agent may never open it.
  • Recommendation: Add a "Never without asking" section near the top of AGENTS.md listing the release trigger, make integration_tests, and any uv.lock or dependency edit; and fence the gh workflow run recipe at AGENTS.md:298-306 with a line saying a maintainer runs this, not an agent.
  • Priority: High

• 41. Does every action that spends money, destroys data or changes production have a human in the way? Start from the damage, not from the tooling: list what in reach of this repo could charge a card, drop or overwrite data, or alter what users are running. Then trace the shortest route an agent could take to each one — a push that auto-deploys, a script carrying live credentials, a migration that runs on merge, an infrastructure apply with no plan-and-approve step. PASS when every route meets a human first, whether that is a review, a manual trigger or a protected environment. FAIL when even one route runs start to finish unattended, and quote that route in the proof so the fix is obvious.

  • Status: FAIL
  • Proof: Two kinds of damage are in reach. Changing what users run means publishing to PyPI, and that route is well defended: _release.yml is workflow_dispatch only, its jobs declare environment: Release, it gates on github.ref == 'refs/heads/master' unless the dangerous-nonmaster-release input is set, it publishes to TestPyPI before PyPI, and it uses trusted publishing (id-token: write) rather than a long-lived token. I cannot confirm whether the Release GitHub environment requires a reviewer — environment protection rules are not visible from the repo, so that is an unknown, not a proven gap. Spending money is the route that fails. make integration_tests runs uv run --group test --group test_integration pytest tests/integration_tests against live provider APIs (libs/langchain_v1/Makefile), and on any machine with a populated .env it runs start to finish with no prompt, no confirmation and no warning — it is listed in the Makefile help output as an ordinary target beside make test, and nothing in AGENTS.md flags it as costing real money.
  • Recommendation: Add a confirmation gate to the integration_tests target in each package Makefile (an @read -p prompt or a required CONFIRM=1 variable), and note in AGENTS.md that this target spends real money on provider APIs.
  • Priority: High

• 42. If a prompt injection landed tonight, how far would it reach — are the credentials an agent can get to here scoped to the job, with nothing production-grade in reach? Inventory what an agent in this repo can reach: env files, cloud CLI profiles, tokens named in docs or config, deploy commands that work from a laptop. Scoped-or-absent passes; production-grade reach fails with the item named.

  • Status: FAIL
  • Proof: Inside the repo the picture is good. No .env is committed (only .env.example), no cloud CLI profiles, no long-lived PyPI token — release publishing uses trusted publishing via id-token: write (_release.yml:556, :836). CI secrets are scoped to environments: LANGSMITH_API_KEY to the "Scheduled testing" environment, release secrets to environment: Release. Workflows declare least privilege (permissions: contents: read at the top of check_diffs.yml, check_agents_sync.yml, codspeed.yml) and guard on github.repository_owner == 'langchain-ai', with block_fork_main_prs.yml closing the fork vector. What fails is the laptop side, and AGENTS.md is what makes it reachable: on a maintainer's machine an agent inherits an ambient gh token, and AGENTS.md:301 gives it the exact one-line command to publish any of 21 packages to PyPI — packages installed by millions. Same file, same session, no fence. Add the local .env a developer creates from .env.example to run integration tests, holding live OpenAI, Astra DB and Power BI credentials, and an injected instruction has both a spend route and a publish route. For an external contributor with no maintainer access the reach is nil; for a maintainer it is production-grade.
  • Recommendation: Fence the release recipe as described at the human-operations item, and add a line to AGENTS.md stating that an agent must never run gh workflow run against _release.yml and must never read or write a local .env.
  • Priority: High

• 43. Can a change reach production a slice at a time — a feature flag that defaults to off, a canary, a staged rollout — rather than everyone at once? Look for a flag system and check the default, or canary and staged-rollout config in the deploy pipeline. Flags that need a rebuild to flip are worth naming in the proof — they gate exposure but they are not a kill switch.

  • Status: PASS
  • Proof: The release pipeline stages it. _release.yml runs build → release-notes → pre-release-checks → test-pypi-publish ("🧪 Publish to TestPyPI", line 540-541) → PyPI publish → mark-release, and every later job lists test-pypi-publish in its needs (lines 589, 749, 821, 871) — so nothing reaches real PyPI until the TestPyPI slice succeeds and the installed wheel is checked. Prereleases give a second slice: the allow-prereleases input lets alpha versions ship with prerelease dependencies resolved, and .github/scripts/check_prerelease_dependencies.py blocks a stable release whose dependencies allow prereleases. Because pip and uv skip prereleases by default, an alpha on PyPI reaches only users who opt in. Named for honesty: there is no runtime kill switch and no feature-flag system in the library — once a stable version is published, exposure is governed by what users pin, so the only lever left is yanking (see the rollback item).
  • Recommendation: —
  • Priority: High

• 44. Once a change is live, can anyone see what it is doing — logs, metrics, traces, alerts that fire on their own, and can an agent read them too? Look for logging setup, an error tracker, analytics, alerting config — and then ask the second half: could an agent reach any of it (a CLI, an MCP server, an API named in the docs), or does observability stop at a dashboard behind a login?

  • Status: PASS
  • Proof: Post-release signal exists and fires on its own. integration_tests.yml runs daily on a schedule against live provider APIs and traces every run to LangSmith — LANGSMITH_TRACING: "true", LANGSMITH_PROJECT, LANGSMITH_TAGS (working directory, Python version, commit SHA) and a LANGSMITH_METADATA JSON blob carrying github_run_url — bridged into pytest by the pytest11 plugin at libs/standard-tests/langchain_tests/_langsmith_plugin.py, which activates only when GITHUB_ACTIONS=true. codspeed.yml keeps performance baselines on every push to master so regressions surface as diffs. A failing scheduled run notifies through GitHub on its own. Second half: an agent can read the CI signal, because AGENTS.md:311 documents gh run view <run-id> --repo langchain-ai/langchain and gh is installed in the devcontainer. Gap worth naming: the LangSmith traces and the CodSpeed results themselves stop at a dashboard behind a login — .mcp.json exposes only the docs and reference servers, and no LangSmith CLI or API route is named anywhere in the repo.
  • Recommendation: —
  • Priority: Low

• 45. Is there a way back — can a bad change be undone without a rebuild and a redeploy, including the ones that touched a database or a queue? Look for a documented rollback path, a revert-and-redeploy story, down-migrations, or a flag that can turn the change off at runtime. Deployment docs that only say how to go forward are worth quoting.

  • Status: FAIL
  • Proof: The release documentation goes one way only. AGENTS.md:307-315 ends with "The workflow handles everything else automatically — do not create a GitHub release or tag manually" and "The full job chain is: build → release-notes → pre-release-checks → TestPyPI publish → PyPI publish → tag GitHub release." There is no next step for a bad release. The pipeline clearly knows yanking exists — _release.yml:660 comments "Identify the latest non-yanked published package release", and lines 695-715 skip yanked files when resolving the last good tag — but no document tells a human or an agent how to yank, when to yank, or what to do afterwards. revert appears in the repo only as an allowed Conventional Commit type in pr_lint.yml:26. This matters more for a library than for a service: a published PyPI version cannot be pulled back by a rebuild, so the only route back is yank plus a new patch release, and neither half is written down anywhere.
  • Recommendation: Add a "Rolling back a release" subsection to AGENTS.md beside the release process, covering when to yank versus ship a patch, the exact yank command, and who is allowed to do it.
  • Priority: Low

• 46. Does a past failure leave a check behind — is there a test or a rule in here that exists because something broke once? Look for tests, lint rules or comments that name the incident they guard against — "regression", "this broke when", a linked issue. On a very young repo with no failures yet, N/A with the age as the reason.

  • Status: PASS
  • Proof: The habit is established — 44 test files under libs/*/tests/ mention a regression, and several name the incident. Two files carry the identical comment # Test integer temperature values (regression test for issue #35300) (libs/core/tests/unit_tests/language_models/chat_models/test_base.py:1380 and .../llms/test_base.py:313), which is a linked issue turned into a permanent check in two places. Others describe the failure they guard: "This is the memory-level regression test: a long-lived tracer is..." (libs/core/tests/unit_tests/runnables/test_tracing_interops.py:629), "Async twin of the additional_kwargs preservation regression" (test_compat_bridge.py:775), and a whole module docstring at test_v1_parity.py:4 reading "v3 has a regression vs v1." The libs/standard-tests package generalises the pattern: a fix found in one integration becomes a shared test every integration must pass.
  • Recommendation: —
  • Priority: Low

Learn more: https://ainativesoftware.engineering/