Skip to content

feat(release): add an agent-driven release verifier - #884

Draft
nieblara wants to merge 8 commits into
cursor/dependency-pr-verifier-9b50from
cursor/release-verifier-7138
Draft

nieblara wants to merge 8 commits into
cursor/dependency-pr-verifier-9b50from
cursor/release-verifier-7138

Conversation

@nieblara

@nieblara nieblara commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Requirements

  • I have added test coverage for new or changed functionality (scripts/release/test/run.sh: 21 unit tests. scripts/dependency-pr/test/run.sh still passes.)
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions (Linux only, with Docker)

Related issues

This PR is stacked on #849, the Dependabot verifier. Merge #849 first. This PR reuses the check protocol, the hermetic environment, the secret scrubbing, and most baseline checks of #849.

Describe the solution you've provided

This PR adds a verifier for a full release. It compares the release-please PR with the last release tag. A run has a fixed part and a part that an agent decides.

The fixed part is scripts/release/verify.sh:

  • --phase prepare makes two worktrees (the last tag and the candidate), reads CI, and writes inventory.md. The inventory lists each commit with its PR, its changelog section, and its product areas from scripts/release/risk-map.json. It proposes units (groups of commits), and it flags a commit and its revert, and shipped changes that the changelog leaves out.
  • --phase baseline runs 22 checks. 15 come from feat(deps): agent-driven Dependabot PR verifier (phase 1) #849. The new ones check the version files and the bump, the changelog, CI on the release head, the CLI surface (commands, flags, and defaults, 4 levels deep), and the npm package contents.
  • --phase units runs the agent checks on the last release, on the candidate, and on the candidate without given commits (revert). A check counts as proof only if it tells the versions apart.
  • --phase render writes result.json and report.md. The verdict is safe-to-release, needs-human, block, or incomplete.

The agent part is the verify-release skill (.agents/skills/verify-release/). The agent groups the commits into units and writes claims from the PRs and the code. It writes a check for each claim. Parallel subagents can each own units (agent/units/<unit>.json). A separate judge agent then reviews every check and finding, and names gaps. The verdict stays incomplete without a judge review, without an adversarial check in each high-risk unit, or if a commit is in no unit.

Checks can use a cached binary build, a local fake LaunchDarkly API that records requests, and a read-only proxy to a live account. The proxy keeps the token in the parent process and refuses each write with 405.

Result on #826 (3.13.0)

The verdict is block. 46 commits in 11 units, 22 baseline checks, and 71 agent checks. 68 agent checks count as proof, and 3 found confirmed bugs. The judge reviewed all of them. All 194 live requests were reads.

Blocks:

  1. sync prompts: after a 409 on the manifest save, the output is null (JSON) or unknown error occurred. The message "run sync again" is lost.
  2. sync prompts link writes the variation file before it asks for confirmation, and keeps it after a "no". The next sync creates the variation that the user canceled.
  3. The changelog lists "Add local flag overrides using dev server" as a feature and as a fix, but fix: Add local flag overrides using dev server (#837) #850 reverts feat: Add local flag overrides using dev server #837.
  4. aws-devops-agent setup: after a failed token replacement, the rerun that setup advises restores only one of the agent spaces.
  5. The published Docker image: dev-server start with a published port no longer answers, because fix(dev-server): listen on 127.0.0.1 by default #845 listens on 127.0.0.1. 3.12.0 worked.

Questions for a person: 15 agent files in the npm package; the new listen address as a fix in a minor release; the move to API version 20240415 (#832) without a changelog entry; the dev server stops if a flag lacks a field that v24 requires; the Alpine 3.24 image removes 45 root certificates; and the release path of release-please-action v5, which no local check can run.

Describe alternatives you've considered

  • A fixed list of checks for each release. It cannot test new features such as sync prompts, so the agent writes checks for the claims of each release.
  • One agent that plans, checks, and judges. It grades its own work, so a separate judge reviews the evidence.
  • A GitHub workflow. GITHUB_TOKEN starts no workflows on the release-please PR, and the agent part needs a VM. reference/automation.md gives a Cursor Automation prompt instead.

Additional context

  • Two small changes touch feat(deps): agent-driven Dependabot PR verifier (phase 1) #849 files. run_one now stops processes that a check leaves running: they held ports and the worktree lock. The docker-image message now says that the dev server does not answer, not that it does not start.
  • Output goes to .verify-out/release-pr-<N>/, which git ignores. This PR does not post a comment on chore(main): release 3.13.0 #826.
  • Limits: the published artifacts are not checked after the release, the arm64 image did not run, and no real browser or client-side SDK ran.
Open in Web Open in Cursor 

cursoragent and others added 8 commits October 20, 2026 03:09
…eline checks

scripts/release/verify.sh compares the release-please PR with the last
release tag. It builds an inventory of the commits, runs the baseline checks
(most of them from scripts/dependency-pr), runs the agent checks of each
unit, and calculates one verdict.

Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…LI surface

Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
A check that starts a server and exits without stopping it left the server
running. The server kept its port and the lock file that the release runner
holds for the worktree, so the next check waited forever. timeout leads its
own process group, so the runner now stops that group after the check.

Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
A finding that explains a blocking check now replaces the block of that
check, so the report lists each bug one time, with the explanation of the
agent. A claim that a blocking check disproves is 'disproved', not
incomplete. The report no longer cuts the text of blocks and questions.

Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…that it does not start

Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants