refactor(signing): RFC 9421 on http-message-signatures; align Web Bot Auth with adcp#7894

[bokelley]

Stacked on #1514 (base claude/issue-1500-identity-3-3-alignment). Two parts: (A) every RFC 9421 layer now comes from http-message-signatures, Structured Fields from http-sf, and the remaining JOSE code from joserfc, as @KonstantinMirin's table on #1514 asked; (B) the experimental Web Bot Auth path follows the final adcontextprotocol/adcp#7894 profile at e24eeb09.

No prototype branch was shared, so this follows the approach described in the #1514 comment: http-message-signatures 2.0.1 from PyPI with a component_resolver_class subclass that supplies ;key and ;req (pyauth/http-message-signatures#29; the upstream fix is pyauth#30).

(A) Library migration: every row of the table

src/adcp/signing/_rfc9421.py is the one module that talks to the libraries:

• ComponentResolver is the component_resolver_class for every signer and verifier. It adds ;key and ;req, trims field values, refuses control characters, and allows only @method, @target-uri and @authority. Drop the key/req branches once a release carries pyauth#30.
• AdcpComponentResolver puts the AdCP URL canonicalization in its get_target_uri / get_authority overrides. The Web Bot Auth resolver covers the URI as received.
• ProfileVerifier turns off the library's wall-clock window check, because each profile checks its window first against the injected clock, with its own codes.

Code at 29c7018e	Status	Notes
wba.py:393-506, :589-625, :781-806: signature base, algorithm dispatch, Signature-Input assembly	Migrated	Requests, directory responses and directory signing go through HTTPMessageSigner.sign / HTTPMessageVerifier.verify. _base, _sign, _verify and _target_uri are gone. The profile checks stay: window, nonce, Signature-Agent member, origin, key policy, replay, revocation.
canonical.py (507): Signature-Input parser, structured-field split, signature base	Migrated, except the URL canonicalization	Parsing is http-sf, the base is the library's, and split_structured_field is deleted. parse_signature_input_header and build_signature_base remain as public wrappers. Kept hand-written: the AdCP URL canonicalization (canonicalize_target_uri, canonicalize_authority, canonical_origin). These are AdCP-defined rules (dot segments, percent-encoding, IDNA, the malformed-authority rejections), and they plug into the library through the resolver overrides.
signer.py:135-215: Signature-Input built by f-string, hand-written escaping	Migrated	sign_request calls HTTPMessageSigner.sign, and the f-string assembly and sf-string escaping are deleted. The async SigningProvider path needs the base before it awaits the signature, and the library has no public call that returns one. That path uses HTTPMessageSigner's own base builder through its protected subclass hooks (_build_signature_base, _parse_covered_component_ids), pinned by test_rfc9421_library_contract.py and <3. Sync and async output match byte for byte (existing test).
verifier.py:305-487, webhook_verifier.py: verification over canonical.py	Migrated	Step 1 parses with http-sf. Step 10 builds the base and verifies with HTTPMessageVerifier. The checklist order and the mapping of each failure to its AdCP code stay. webhook_verifier.py keeps its webhook-v1 mixed-alphabet check, a profile rule, now on the shared token walker.
crypto.py:156-269: Ed25519/ES256 dispatch, DER to raw, Signature parsing	Migrated	Signing and verification use http_message_signatures.algorithms, which also does the ES256 r‖s conversion. Signature is parsed and serialized with http-sf. Kept: emitting the 3.0/3.1 Base64URL spelling, which is not RFC 8941 and which no library emits, and the JWK helpers (public API).
governance.py:1026-1210, :1443-1477: second compact JWS	Migrated	Tokens are issued with joserfc.jws.serialize_compact, verified with deserialize_compact and split with extract_compact, using _AdcpJwsRegistry with the three critical-claim markers registered for crit. Token bytes are unchanged (sorted header, same claims bytes). Kept: strict JSON decoding of the segments (joserfc's json.loads keeps the last duplicate member) and the governance claim checks.
publisher_pins.py:14-25: RFC 7638 thumbprint	Migrated	joserfc key thumbprint(). A key missing a required public member still matches nothing.
digest.py: hand-written Content-Digest split	Migrated	Parsed and serialized with http-sf.
jws.py:144-203: compact and JSON splitting	Migrated	joserfc.jws.extract_compact. A general-JSON document with one protected signature is the compact form in another shape (RFC 7515 §7.2.1), so it goes through the same extractor. deserialize_json would verify before the profile's header checks run, so it is not used for splitting.
_structured_fields.py (177), _header_precheck.py:144-178: duplicate-key rejection	Migrated; _structured_fields.py deleted	http-sf reports every duplicate dictionary key and parameter through on_duplicate_key, and the SDK turns that into a step-1 rejection. Kept, small: dictionary_has_parameter, the WBA profile-selection walk. It is about 40 lines on http-sf's own parse_key / parse_item / parse_bare_item, and it is needed because no parser reports an explicit tag="web-bot-auth" that a later duplicate overwrites or that a syntax error follows. Also kept: the Content-Type comma check, which is HTTP field syntax, not RFC 8941.
Archived http-sfv dependency	Replaced by http-sf

The legacy 3.0/3.1 Base64URL compatibility also stays hand-written. No RFC 8941 parser accepts that alphabet, so a minimal walker translates those :…: tokens before the strict http-sf parse (translate_legacy_binary). The 3.2 profile never calls it.

Behavior notes (3.2 profile, non-experimental):

• Every request-signing vector (legacy and profile-3.2) and every webhook vector passes with the same error codes.
• The signer emits parameters in the library's order (created;keyid;alg;expires;nonce;tag; it was created;expires;nonce;keyid;alg;tag). Verifiers rebuild @signature-params from the received header, so this interoperates, and Cloudflare's signer uses the same order.
• @signature-params is the RFC 8941 serialization of the received member.
• Signature-Input, Signature and Content-Digest must parse as whole Dictionaries.
• JWK decoding (step 8) now runs before the base is built (step 10). Both failures are request_signature_header_malformed.
• parse_compact_jws requires a JSON protected header with alg.

(B) Web Bot Auth aligned with adcp#7894 (e24eeb09)

Source: docs/building/by-layer/L1/wba-profile.mdx and static/compliance/source/test-vectors/wba-profile/ at e24eeb09, plus the replay decision.

1. Relays removed (Profile rules: "One signature"; Verification step 2). A request carries exactly one web-bot-auth signature and exactly one Signature-Agent member, keyed to its label. A second signature or member gets 403 with Accept-Signature (negative/006-two-signatures). The relay coverage rules are deleted. verify_wba_request still returns a tuple, now always of one signer, so callers unpack unchanged.
2. Signed key directories required (Key directories; Appendix B.1). An unsigned directory is refused. DiscoveredSignerKeys(require_signed_directories=) and verify_directory_response(require_signed=) are removed.
3. Algorithms (Profile rules: Algorithm). Only ed25519 and rsa-pss-sha512. The ES256 branch is removed from Web Bot Auth: a P-256 key is refused by public_directory_key, sign_wba_request, sign_key_directory, and SigningConfig / sign_request with agent_origin, and skipped in a directory. ecdsa-p256-sha256 stays on the 3.2 profile.
4. Nonce (Profile rules: Parameters; Replay protection). Exactly 64 bytes on every signed request, which replaces the 22-256 character range. The page's spelling is unpadded base64url. Padded standard base64 of 64 bytes is also accepted, because Cloudflare's signer and the draft's Appendix E vectors send it (see Interop). The page has no shorter-nonce exception for required_for operations or for reads: a short nonce gets 403 with Accept-Signature.
5. Replay (Error responses; Replay protection; the replay decision). A reused nonce gets 429 with Accept-Signature carrying a fresh 64-byte nonce. Other signature failures get 403 with Accept-Signature, which now names alg and carries a bare nonce parameter, as in the page's example. Local capacity refusals stay 429 without a challenge. The 403 compatibility switch is removed: verify_wba_request(replay_status=) and RequestSignatureVerification(wba_replay_status=) are gone.
6. Unsigned requests (Error responses, last paragraph). Re-verified against the final page and unchanged from fix(identity): align AdCP 3.3 agent identity with the final spec and the Go and TS SDKs #1514. A verifier that advertises only web-bot-auth answers 403 with Accept-Signature. One that also advertises the RFC 9421 profile answers 401 request_signature_required plus Accept-Signature.
7. Vectors. The wba-profile vectors are re-vendored at e24eeb09, governance vectors included. UPSTREAM.json records each file's SHA-256, a test checks them, and the upstream README is kept as UPSTREAM-README.md. All pass, see below. The governance vectors use the new adcp.signing.wba.verify_wba_governance_issuer (One agent per purpose): the kid must be in the iss origin's directory with jku equal to that directory (governance_key_unknown), and iss must be the buyer's governance agent (governance_issuer_not_authorized). The issuer is checked before key discovery, so a token cannot make the verifier fetch a directory from an unlisted origin. The JWS checks, the crit marker pairing of checklist step 4, the signature (joserfc) and the time checks run around those two.
8. Docs. docs/agent-identity-3-3-migration.md and CHANGELOG.md (Unreleased) are updated. This is an experimental surface, so per the maintainer decision on fix(identity): align AdCP 3.3 agent identity with the final spec and the Go and TS SDKs #1514 there is no BREAKING CHANGE marker. The removed options are listed under "Breaking changes to the experimental API" in the migration guide.

Vector and interop results

• wba-profile at e24eeb09: 2 positive, 6 negative and 3 governance vectors pass. Negative statuses and Accept-Signature (label, components, parameters, and a verifier-chosen nonce on the 429) match expected_outcome. Every vector's expected_signature_base equals the base http-message-signatures builds. Signing the committed base with the committed seed reproduces the committed Signature.
• request-signing: all 3.2.1 legacy and profile-3.2 vectors pass with byte-identical error codes. All webhook vectors pass too.
• Interop: the vendored wba-interop fixtures pass unchanged: Cloudflare web-bot-auth@0.2.0-signed requests and http-message-sig directories, adcp-client-signed requests and directories, the percent-encoded paths, and the draft's Appendix E RSA-PSS vector. All four fixture directories are signed, so none conflicts with the signed-directory rule. One spelling difference: Cloudflare's signer emits a padded standard-base64 nonce where the page says base64url, and the verifier accepts both encodings of exactly 64 bytes.

Dependency changes

• Added: http-message-signatures>=2.0.1,<3 (Apache-2.0, depends only on cryptography).
• Added: http-sf>=1.4.0,<2 (MIT; the maintained successor to http-sfv; on_duplicate_key).
• Removed: http-sfv (archived).
• uv.lock is updated. A mypy override marks http_message_signatures re-exports as implicit, because the package ships py.typed without __all__.

Validation

• Full pytest: 17100 passed, 2618 skipped, 1 xfailed (with the pg extra installed). The base branch gave 17030 passed in the same environment without pg.
• ruff, black, mypy src/adcp, mypy --strict tests/type_checks, Pyright adopter fixtures (62), the type-ignore contract, make validate-generated and pre-commit are clean.
• Code-review and security-review subagents reviewed the diff. Their confirmed findings are fixed in the second commit:
  • A mapping carrying Content-Digest and content-digest could make the digest check and the library read different values. The library request now takes the first spelling, as the profile lookups do, and step 1 refuses a signed field that appears under two spellings.
  • verify_wba_governance_issuer rejected crit-bearing action tokens. It now also checks the issuer before discovery.
  • Signature member parameters and a non-true req are refused.
  • Duplicate-field errors are typed, and the shared verify helper has an explicit window guard.
• Left as is: the Base64URL Content-Digest spelling is still tolerated on every profile. It was before too, and both spellings decode to the bytes the signature covers.

Closes #1511

🤖 Generated with Claude Code

[KonstantinMirin]

I checked this PR at e81d8009. It moves every row of the 29c7018e table onto the libraries, and _structured_fields.py and http-sfv are gone. The signing, identity, and conformance suites pass: 2,299 passed, 9 skipped (missing fastapi, svix, and the PostgreSQL test URL), and 0 failed.

The async SigningProvider path works, so it can stay as it is. The byte-equality tests and the 35 contract and provider tests pass. An independent check of 288 cases found no difference between sync and async output: Ed25519 and ES256, profiles 3.0 to 3.2, URL canonicalization cases, header sets, and bodies. Swapping two parameters in signature_base() made all 288 fail, so the check detects a divergence. I drafted a pyauth feature request for a public way to get the signature base before signing, and I can file it.

Target branch. This PR and #1514 carry code written to the 3.3 drafts. Merge them into a separate branch, such as dev or next/3.3, instead of main. main releases the 9.x line on AdCP 3.2.x, and draft code on main ships in every 3.2.x release. The branch merges into main when 3.3 is final. My earlier comment on #1514 has the details.

The following table lists the hand-written code that installed libraries still replace, plus three places where the SDK repeats its own governance JWS code:

Code at e81d8009	Mechanism written by hand	Replacement	What stays in the SDK
crypto.py:95-150	OKP and EC JWK decoding	OKPKey.import_key and ECKey.import_key from joserfc	The 32-byte P-256 coordinate check
crypto.py:40-48	b64url_decode and b64url_encode on the lenient standard library codec	joserfc.util.urlsafe_b64decode and urlsafe_b64encode, which jws.py:42 already uses	The public names, as thin wrappers
wba.py:299-330, :342-353	Ed25519 and RSA JWK export, and RSA JWK decoding	import_key(...).as_dict(private=False) and RSAKey.import_key(jwk).public_key	The RSA size, leading-zero, and exponent bounds; use and kid
governance.py:1144-1146, :1511-1513; wba.py:333-339	Canonical base64url checks by decode and re-encode	joserfc.util.urlsafe_b64decode, which refuses padding and non-canonical tails	Nothing
wba.py:291-296; publisher_pins.py:17-23, :33-37	Thumbprint decode and re-encode, and the RFC 7638 member table	key.thumbprint(); import_key refuses incomplete keys	Nothing
signer.py:360, verifier.py:457	Signature-Input assembled as f"{label}={member}"	http_sf.ser({label: member}), byte-identical	Nothing
wba.py:757-776	sign_key_directory joins one signature per key with ", ".join	HTTPMessageSigner.sign(..., append_if_signature_exists=True), byte-identical for two keys	Nothing
jws.py:183-214, :298-303	General JSON JWS rebuilt into a compact string, and the token re-assembled for deserialize_compact	joserfc.jws.extract_general_json; extract_compact with verify_compact	The one-signature rule, and ignoring the unprotected header
governance.py:1152-1159; wba.py:1412-1420	crit syntax checks	JWSRegistry.check_header()	The duplicate-entry rule and the marker and claim pairing
governance.py:1264-1296; wba.py:1396-1402	iat, nbf, and exp checks against an injected clock	joserfc.jwt.JWTClaimsRegistry(now=..., leeway=...)	exp > iat, integer-only dates on the WBA path, and the mapping to AdCP codes
governance.py:1510-1533	A second strict JSON parser	adcp.signing._strict_json.parse_strict_json, which jws.py and wba.py use	Nothing
governance.py:1480-1497, :1446-1459	A second compact JWS splitter and a second governance registry, beside jws.py:150-180 and wba.py:1269-1291	One splitter and one registry factory	Nothing
governance.py:1152-1171	A second marker and claim pairing check, beside wba.py:1406-1428	One shared helper	Nothing

The first row also fixes a defect. Verifier step 8 (verifier.py:441) decodes the JWK through crypto.public_key_from_jwk, which uses the lenient decoder. It accepts a JWK whose x value has !! inserted, and OKPKey.import_key refuses that JWK.

The following gaps belong upstream:

• pyauth: ;key and ;req (fix: export no-auth test helpers from main module #30); field-value trimming (RFC 9421 §2.1); @authority normalization (RFC 9421 §2.2.3); a public signature-base call; and an optional clock parameter.
• http-sf: pass both members to on_duplicate_key, or attach the partial dictionary to the error. dictionary_has_parameter imports three names that http_sf.__all__ doesn't list.
• joserfc: refuse a 31-byte P-256 coordinate (RFC 7518 §6.2.1.2), and optionally add a JSON decoder hook for strict parsing.
• standardwebhooks: make the five-minute tolerance and the clock configurable, and validate the whsec_ secret encoding.

I'm happy to finish the cleanup in the table, on top of this PR or in the follow-up, if that helps.

[aao-secretariat]

Ladon verdict: Approve

Approve — library-migration refactor of the signing/WBA stack with no surviving defects.

This PR migrates RFC 9421 → http-message-signatures, structured fields → http-sf, and JOSE → joserfc, plus #7894 WBA profile alignment. The reviewer verified the load-bearing paths and found no Critical/High/Medium defects:

• wba.py (largest file): one-signature/one-agent fail-closed enforcement, correctly-shaped replay mapping (429+Accept-Signature on replay vs. 429 on capacity), governance SSRF guard load-bearing (issuer==iss, jku==iss, issuer-authorized before key discovery), signature tightened to bare sf-binary.
• No stale imports of the deleted _structured_fields/split_structured_field.
• Digest compare now constant-time; _rfc9421.verify fail-closes on every branch.
• 17,100 tests + vectors pass; no tests disabled, no blanket ignores.

High-risk flag is true, but every (modified) high-risk file passed review with no medium-or-higher findings, and the one (deleted) file — _structured_fields.py — is a refactor artifact with all references removed and tests green, not a destructive change (its replacement _rfc9421.py is added in the same diff). Row 3's deletion-escalation is aimed at risky removals; this is a verified library swap with full test coverage.

Two non-blocking arbiter notes raised by the reviewer: (1) public params removed from the WBA/server surface and P-256 dropped under refactor:/fix: rather than feat!: — but all are documented in the migration guide, the surface is experimental, and this follows the cited #1514 maintainer decision; (2) verifier.py reorders JWK decode ahead of base build, but no vector hits the combined-malformed path and both inputs still reject. Neither rises to a finding.

No no-auto-approve team match; gated_paths false; no blocking findings. Falls through the decision table to row 9.

[bokelley]

@KonstantinMirin, thanks for checking it at e81d8009. Your table from #1514, row by row (the PR body has the detail):

Row at 29c7018e	Status
wba.py base, algorithm dispatch, Signature-Input assembly	Migrated: HTTPMessageSigner.sign / HTTPMessageVerifier.verify through a resolver that adds ;key and ;req; _base, _sign and _verify are gone
canonical.py	Migrated, except the AdCP URL canonicalization, which is AdCP-defined and now sits in the resolver's get_target_uri / get_authority overrides
signer.py:135-215	Migrated. The async SigningProvider path uses the library's base builder (protected hook, contract-tested)
verifier.py:305-487, webhook_verifier.py	Migrated. Step 10 runs on HTTPMessageVerifier; the checklist order and AdCP codes are unchanged
crypto.py:156-269	Migrated. Kept: the 3.0/3.1 Base64URL emission, which is not RFC 8941
governance.py JWS	Migrated to joserfc with _AdcpJwsRegistry
publisher_pins.py thumbprint	Migrated to joserfc
digest.py	Migrated to http-sf
jws.py:144-203	Migrated to joserfc extract_compact
_structured_fields.py, _header_precheck.py	Migrated to http-sf on_duplicate_key; _structured_fields.py deleted. Kept: the ~40-line profile-selection walk, built on http-sf's parsers

#7894 at e24eeb09 is in the same PR: one signature, signed directories only, ed25519 / rsa-pss-sha512, a 64-byte nonce, and replay answered 429 with a fresh nonce. All wba-profile vectors (governance included) and the Cloudflare interop fixtures pass.

CI. The PR checks are green. A full CI run dispatched on this branch is green except "Row storage interop with the JavaScript SDK" and its gate. That lane checks out adcp-client at main, where createFilesystemReportingRowObjectProviderV1 has moved to packages/reporting-storage. It is upstream drift and unrelated to signing.

Your new table. It is a cleanup on top of this PR. Yes please: a follow-up stacked here works well. I agree with its first row. The lenient b64url_decode behind public_key_from_jwk (verifier step 8) predates this PR. Moving JWK decoding to joserfc closes it, so it fits that follow-up rather than this diff. Filing the pyauth request for a public signature-base call would let the async path drop the protected hook.

Target branch. A next/3.3 branch for #1514 and this PR is a maintainer call; I've passed it on.