Repository navigation
chore(release): 0.22.0 — SDK via-edge + WAL durability + ADR-068 wire - #116
Merged
Merged
Conversation
The SDK's enforcement question can now go to a local box that holds a
signed lease instead of to api.nullrun.io — opt-in via
NULLRUN_EDGE_URL, refuses to fall back to the cloud. The write-ahead
log was reworked around three loss paths the previous release did not
cover: a kill -9 while a refused event was held, a 200 batch that
hid per-item refusals, and a recovery that unlinked the WAL before
the events were provably safe. ADR-068 closes the wire: the SDK no
longer puts a price on the wire (the backend prices the cached
fraction off provider, and the SDK reports tokens because only it
knows them).
The full release notes (Migration, Security, Fixed, Changed, Tooling,
Verification, Commits) live in CHANGELOG.md under ## [0.22.0]; the
verification numbers below are the same gating signal the runbook
calls for in step 3.
ruff check src tests : All checks passed
mypy src/nullrun : 0 issues in 39 source files
(the 5 in instrumentation/auto.py reproduce
identically without these changes, same shape
as 0.21.0's 6-error baseline note)
pytest -q : 1828 passed, 12 skipped in 44.89s
(vs baseline 1694 / 1 — +134 new tests,
+11 more skipped)
scratch diff : clean
nullrun.__version__ : 0.22.0
wire-format : non-additive on /track single (cost_cents
removed, provider added when set, 422 overage
details renamed); additive on /gate
(edge_lease in the decision_source allowlist
— the gate is internal to the SDK and only
via-edge emits it)
_replay_from_wal unlinked each recovery file BEFORE calling _do_flush. The docstring claimed "Both files are removed only after a successful flush"; the code did the exact opposite. A crash between the unlink and a successful flush lost the batch outright — the events were in neither the WAL nor the buffer and the server never saw them. Since recovery only runs when the backend was unreachable at shutdown, the breaker being still open at start() is the LIKELY case, not the exotic one. Recovery is now at-least-once: no file is unlinked before its events are provably safe (accepted, or rewritten into .wal). Two more gaps on the same path: * Crash mid-send. The batch had already left _buffer and was in HTTP, so neither the buffer nor the periodic WAL held it. A .inflight file is now written before the send and cleared once the send lands. A stale one self-heals on the next successful flush. * event_id. Re-delivery is only safe because the backend dedups on it (cost_event_id_dedup PRIMARY KEY), so every event is now guaranteed an id before it reaches disk. Writing a batch no longer uses open(path, "a"): that could interleave a partial write with an existing file. tmp + fsync + os.replace, and a failed write reports False so the caller keeps its recovery file instead of deleting it. An exception escaping _do_flush also killed the background flush thread and silently stopped all delivery for the life of the process; it is now caught and re-queued. 16 tests. Verified negatively: run against the pre-fix transport.py in a clean worktree, 11 of 16 fail.
…r miscounting rejections "Any 4xx is permanent" discards good data at the worst possible moment. Right after an outage every SDK in the fleet replays its WAL at once, the rate limiter answers 429, and that rule would dead-letter the entire recovering fleet in the first second of the window. Routing is by status class now: * 408/429 transient — re-queued, NEVER dead-lettered at any attempt count. The batch is valid; the server is busy. * 401/403 — key rotated or revoked. Recoverable by operator action, so the batch stays in the WAL and the log says delivery is blocked on the key, rather than stranding the data on disk as DLQ litter. * 413 — the BATCH is too big, not the events. Halve and resend. * 400/422 — bisect. The backend validates fail-CLOSED, so one event with an empty `model` rejects the whole request; quarantining the batch would bury every valid event behind it. Halving continues to a single event, so exactly one event is lost. The halves are sent as separate batches, not pushed back onto the buffer: the buffer is flushed wholesale, so re-queuing both halves would rebuild the batch the server just refused and the split would never converge. Depth-capped so an all-bad batch cannot spin. A separate live defect, found while testing this: the rejection escaped the circuit-breaker call, which counted it as a transport failure. With max_failed_flush=10, ten consecutive rejected batches opened the circuit on a backend that was up and answering — blocking every buffered event for the whole recovery window, for a problem the breaker cannot fix. A 4xx is the server answering, and a local serialization TypeError is our own bug; neither says anything about backend health. Both are now trapped inside the call and handled after it. httpx.TransportError and BreakerTransportError still propagate, so the breaker keeps its actual job — pinned by a test that it still opens on an unreachable backend. Non-HTTP exceptions no longer return to the buffer forever. A payload that will not serialize fails identically every cycle and pins the batch ahead of the whole buffer; it now has a bounded attempt budget (NULLRUN_MAX_BATCH_ATTEMPTS, default 10) and is dead-lettered once spent. The counter is keyed by batch signature and cleared when a batch lands, so it cannot grow unbounded in a long-lived process. Append hardening for the copy-and-append path (the DLQ): a crash can leave the last line unterminated, and copying it verbatim glues the next row onto truncated JSON, corrupting an event that was perfectly valid. The torn tail is now dropped before appending. The tmp is pid-scoped and opened with "w" so a leftover from a dead process is truncated rather than appended to, and the directory is fsynced after the rename so the new name survives a power cut. 21 tests, one per 4xx class plus the breaker guards and both torn-tail cases. Full suite 1731 passed / 1 skipped.
`/track/batch` refuses individual events and still answers 200, naming
the survivors in `accepted_event_ids` and the casualties in
`rejection_details`. The old code read the status, saw 2xx, treated the
whole batch as delivered, cleared `.inflight` and dropped the refused
events — out of the buffer and out of the DLQ, with no trace anywhere.
"Data is not lost" held only for transport failures, which is a much
weaker claim than it reads as, and the status-code classification never
saw any of this because the status is 200.
Classification:
accepted gone from the retry path, `_in_flight` released
retry hint re-queued at the head, still bounded by the attempt
counter so a hint that never materialises cannot pin it
terminal parked in the DLQ on the FIRST response
unknown bounded retry, then the DLQ — never a silent drop, which
is the only failure mode nobody notices
unparseable the whole batch is treated as NOT delivered: same
bounded budget, then the DLQ under `unconfirmed_response`
Unknown reasons take the retry path rather than being treated as
terminal. Absence of `retry_after_ms` alone is not enough to park
something, because an older backend omits the field everywhere and
"no hint" would then mean "discard" for every reason it can emit. Only
`reservation_not_found` is terminal; adding to that set is safe,
forgetting an entry is not.
The write order is the load-bearing part, not the classification. Every
event must be durable — in the DLQ, or still covered by a retained
`.inflight` — BEFORE `.inflight` is cleared. Crash before the clear and
the restart replays the batch: a duplicate, which the backend absorbs
by deduping on `event_id`, and which comes back accepted because
`IdempotentReplay` maps to `Ok`. Crash the other way and the event is
in neither file with no way to reconstruct it. A duplicate is
recoverable; a silent gap is not. The test injects a crash in that
window and pins that the batch survives it.
DLQ rows are versioned (v2) and carry `reason`, `event_type`,
`first_failed_at`, `attempts` and the original payload — a replay needs
the payload, triage needs to know whether the refusal is still current.
`error` is kept in sync with `reason` so a file written before this
change still reads after an upgrade; a reader that required `v == 2`
would make the one file whose job is recovery the thing that broke.
`_retry_or_dlq` now returns whether the batch is still live, and
`_quarantine_to_dlq` no longer clears `.inflight` when only part of a
batch is parked — the partial path goes through `_write_dlq_rows`, which
only writes.
11 new tests. Full SDK suite: 1745 passed.
The write-ahead log is the SDK's only copy of an event between track() and the backend accepting it, and it was being maintained on three assumptions that no code checked: that the files are private, that nothing else writes them, and that the dead-letter queue cannot fill the disk. Files are now created 0600 rather than through open(), which applies the process umask to whatever 0666 it asks for — an event payload is a prompt, a completion and its tool arguments, and the default umask left those world-readable in /tmp, on a shared volume, and in any container several services mount. Files left by an earlier run are tightened at startup, because a file's mode survives every atomic rename: hardening only the tmp file would have left exactly the payloads most likely to be on disk (the ones from a crashed run) readable by everyone. Mutations are serialised with an advisory flock. The scope is the file mutation, never the network send — holding it across a request would serialise every worker behind one another's latency. The wait is bounded and measured against a clock, not counted in naps, because anything that returns from sleep early would otherwise spend a 5s budget in milliseconds and give up while the lock is still held. That wait is not decoration. The first implementation took the lock non-blocking and skipped the write on contention, and the Linux stand found it immediately: with four concurrent writers, the losing writer was refused on *every* attempt, because the winner was always mid-append, and its DLQ writes never landed. Nothing alerted — the events are re-queued, not lost — and nothing was delivered either. The DLQ size cap deletes nothing. At the cap the append is refused, an ERROR is logged, and the event stays on the retry path. Trimming the oldest rows would destroy the only copy of events the SDK could not deliver, silently, with no trace beyond a shorter file; a DLQ that has stopped growing is visible and alertable, a DLQ that quietly lost yesterday's refusals is neither. A permanent rejection that could not be recorded no longer clears .inflight, because that clear was the loss: the events would be in neither the DLQ nor the buffer. fcntl and directory fsync are feature-detected rather than imported, and every degradation is a named state in metrics (wal_dir_fsync, wal_lock, wal_lock_timeouts_total, dlq_bytes, dlq_overflow_total) plus one warning per aspect — once, because a degraded platform is a standing condition, not an event. `nullrun-wal` gives the DLQ a reader. Nothing replayed it, because a refusal is not something a retry fixes, which is correct and leaves the file unreadable by anyone. `status` is read-only; `replay` is a dry run unless --execute is passed, and removes a row only after the backend confirms that exact event_id — including preserving a line it could not parse, since the unparseable line is the one an operator most needs. Verified in a Linux container (a green Windows run is not proof — every POSIX assertion is skipped there): 1759 passed, 2 skipped. Then two scenarios on a named volume. kill -9 mid-flight: the persisted events come back off the volume and stay durable. gunicorn -w 4 on one shared volume: 600 appends across four processes, 600 lines, 600 distinct ids, 0 unparseable, mode 0600. docs/wal-and-dlq.md documents the guarantees, the refusals, the platform-dependent ones, and the cost of an outage past the 24h binding TTL: those events are refused with EXECUTION_NOT_BOUND and land in the DLQ, so the spend is real and the period counter did not account for it — the control center under-reports by exactly what was buffered.
A verification pass over the WAL/DLQ layer found four defects. Three are the same shape — a refusal that the operator can see in the DLQ but that the code itself no longer accounts for — and one is a retry loop that did not retry slowly. **A 429 was retried with no delay at all.** The `except httpx.HTTPStatusError` branch set `last_exc` and fell straight through to the next attempt: no backoff, no sleep, and the `Retry-After` the server had just sent ignored. A rate-limited client spent its entire retry budget in milliseconds, hammering the limit that had just answered it, then gave up. The `last_retry_after_seconds` parameter was documented, threaded through the signature, and passed by no caller, so the suite stayed green — the existing 429 test asserted the CALL COUNT, which does not depend on how long you wait between calls. The delay is now computed in one helper shared by both retry branches. Jitter on that path is one-sided ([wait, wait*(1+jitter)]): a rate limit states a floor, and spreading downward is how a 429 becomes a sustained 429. **A full DLQ let one bad event stop the whole stream.** A refused batch whose DLQ write failed was re-queued at the head of the buffer, ahead of every healthy event behind it, and re-sent every cycle forever — a terminal refusal that cannot succeed on retry. Refusals the DLQ cannot take are now held in memory (`dlq_holdover`, alertable because it counts events unrecorded RIGHT NOW, where `dlq_overflow_total` counts refusals over time) and drained as soon as there is room. A whole-batch deterministic refusal is also bisected before anything is parked, because that is what an unbatched per-event rejection looks like from out here: one bad event makes the request carrying it fail as a unit, and quarantining the batch on that evidence discarded every healthy event sharing it. The cascade spends a bounded number of requests (`NULLRUN_MAX_BISECT_REQUESTS`, default 64) rather than 2^depth — the depth guard bounds the recursion, not the wire. **`replay` deleted the operator's only record of a refused event.** It computed "did this land?" from the transport's in-memory buffer, but a refused event is in the scratch DLQ or the holdover, never the buffer. With every event refused it concluded all were confirmed, printed `sent 3 event(s); removed 3 row(s)`, and returned 0 — after the backend had just rejected all three. Both destinations are now read, and the exit code distinguishes "nothing to replay" because the DLQ was empty (0) from "nothing was confirmed" (1), so a scheduled repair cannot report success while sending nothing. **`replay` re-sent refusals that re-sending cannot fix.** `EXECUTION_NOT_BOUND` means the 24h server-side binding has expired, so the send returns the identical 422. Those rows are now withheld by default and named on stderr, with `--include-terminal` to override; recovering that class needs a ledger-only ingest the backend has to grow first, which docs/wal-and-dlq.md now says instead of implying a retry will fix it. Also: `runtime.execute` annotated `on_transport_error` as callback-only while forwarding it verbatim to a transport that has always accepted the ADR-008 strings, so the documented `"raise"` arm was un-passable under a type checker. It is named `TransportErrorHandler` now. `test_execute_network_error_raises_classified` had been disabled rather than asserting that flag — and it also mocked /api/v1/gate instead of /api/v1/execute, so it would not have passed as written. It is re-enabled, with a counterpart pinning the fail-CLOSED default that the opt-in exists to override. scripts/linux_gate.sh is the Linux gate as a repository script, since the coverage Windows cannot execute is the entire point of running it: flock tests are skipif win32, so a Windows-only run reports green with them absent. REPEAT=20 by default, because a lock bug is a timing bug and one green run is weak evidence. Verified: 1769 passed on Windows, 28 passed / 0 skipped in the Linux container, ruff and mypy clean. Every new test was run against the pre-fix source first and fails there.
The rule is in CLAUDE.md and was still broken twice, so it is checked rather than trusted. Scope is `origin/master..HEAD`, not `--all`: a whole-history grep counts trailers on the 200+ commits already published and nobody's to rewrite. On this repository the two numbers are 0 and 25 respectively, which is the whole argument for scoping it to the range — the `--all` form reports a violation on a branch that added none. Anchored at the start of a line, so a commit body discussing the rule and a CHANGELOG entry crediting a person both still pass. The trailer form is the violation; the words are not. `fetch-depth: 0` is load-bearing, not boilerplate: a shallow clone has no `origin/master` to diff against, the range would be empty, and the step would pass on every branch. A check that always passes is worse than no check, because it reads as enforcement. A `commit-msg` hook is the stronger place for this — it fails before the commit exists — and one is configured in .pre-commit-config.yaml. This is the backstop for the case a hook cannot cover: a contributor with no pre-commit installed, or a --no-commit push. One mechanism enforced one way is a mechanism enforced once, which is the number of times the convention had already failed.
must not be sat through Two durability defects, both found by asking what a `kill -9` loses. **The holdover was memory-only.** When the DLQ is at its cap a terminally-refused event goes to `_dlq_overflow` and stays off the send path — correct, because re-sending a terminal refusal produces the same refusal while occupying the head of the buffer, which is what stops one bad event starving every healthy one behind it. But it lived in RAM. `.inflight` looked like it covered this and does not: the next flush calls `_persist_inflight` with ITS batch, overwriting the file. So a process that held a refusal and then flushed again had silently overwritten the only on-disk trace of it. The memory-only holdover was itself a new loss path. Held rows now go to `<wal>.holdover` — deliberately not `.wal` (rewritten wholesale by every buffer persist) and not `.inflight` (overwritten by the next send). Recovery reads it back as a *hold*, not a replay: a restart that recovered them by re-sending would reintroduce the exact head-of-buffer starvation the hold was built to stop, on every restart, which is when an operator is most likely watching. The drain releases rows by `event_id` rather than deleting the file, because the in-memory index is bounded and can be shorter than the file. The index bound is on count and trims the index only: `dlq_holdover_index_truncated` means the gauge under-reports, which is loud and named; `dlq_holdover_persist_failures` is the one counter that means possible loss, because those rows are in memory and nowhere else. **Retry-After had a cap in the wrong direction.** It clamped with `min(server_wait, max_delay)`, which bounds the wait and also retries an hour early — exactly what the floor semantics forbid at a smaller scale. `Retry-After: 86400` produced ten retries at 30s each, spending the whole budget re-tripping the limit it was just told to respect. Past `NULLRUN_RETRY_AFTER_CEILING` the flush now stops for the cycle instead; the events are already durable, so declining to retry costs nothing. The wait became `Event.wait` rather than `time.sleep` so `stop()` can cut it short. A process that is shutting down should not sit out a rate-limit window, and that matters most when the backend is misbehaving and someone is killing the process because of it. Tests stub `_wait_on` rather than `threading.Event.wait` globally: a global patch changes every other Event.wait in the suite, including the approval-wait tests that assert on real elapsed time, and turns them into load-sensitive flakes to make the retry path fast. Also: the fsync probe is now documented as proving the syscall works and not that data survives power loss — a flush test needs a device that lies about a flush, and CI has none. And the replay contract is recorded as having arrived in fd4e581 without the behaviour its report described, since that is the second time a report outran the code here. Negative-verified: every new test fails against the pre-fix source.
/track/batch rejects a settle that lands past reserved + epsilon with 422, but the Lua overage branch has already INCRBY'd the real spend onto the period counter and the handler has already written the cost_events row. The refusal is about the reservation ceiling, not about delivery. details.reservation_recorded is the backend stating which happened. CONSUME_OVERBUDGET is in _DETERMINIC_ERROR_CODES, so the SDK read the 422 as a refusal: bisect, then _quarantine_to_dlq the singleton with "permanent rejection, will NOT be retried". For a settle the server had already booked that files a rejection that never happened, for an event it already holds a row for, and the rest of the batch dies with it — the aborted request never reached those events, so they are still owed. deliver on the marker, quarantine otherwise: * reservation_recorded is True -> DELIVERED. Drop the event from the retry path and from ._in_flight, re-queue the rest of the batch, count events_recorded_overage. No bisect: the body names the offender, so the refusal costs one send instead of log2(n). * anything else, including a body with no details at all -> unchanged quarantine path. A refusal that charged nothing must still park data. * a name we cannot see in a multi-event batch -> not provable, so let the existing cascade isolate the offender and re-enter here on the singleton. Scoped to CONSUME_OVERBUDGET alone: the marker on any other 422 grants nothing, and a recorded-marker body that names an event absent from the batch is quarantined rather than guessed at. NullRunConsumeOverbudgetError gains `recorded` (tri-state — True, False, or None for a backend that sends no marker; absence is not False) so a caller catching the single-/track error can tell "refused and charged you" from "refused and left your reservation alone", which demand opposite responses. Tests: 11, red before the fix. Three pin the unchanged quarantine paths (an unrecorded overage, a body with no details, the marker on an unrelated code) — these are the ones that catch the fail-open direction. Two mutations, both caught: dropping the whole batch instead of re-queueing the rest, and reading an absent marker as recorded. Full suite 1791 passed / 11 skipped.
The overage body's `details` block (handlers.rs, the ConsumeOverbudget 422) carries reserved_millicents, max_allowed_millicents, actual_cost_cents, soft_pass and reservation_recorded. It does not carry reserved_cents, max_allowed_cents or epsilon_cents, and never has. The dispatcher read those three, so on a parsed 422 all of them were permanently None — while runtime.py's fail-closed table told callers to reconcile the delta from exactly those three. A caller following the documentation computed None - None + None. Now reads the two millicents fields, the cents actual, and soft_pass. epsilon is deliberately NOT reconstructed: the backend does not publish its configured tolerance on this body, and deriving one by subtraction would be arithmetic dressed as evidence. The honest recovery is max_allowed_millicents - reserved_millicents, which the caller can do from the two numbers that do arrive. The *_cents params are retained for callers that construct the exception themselves, and their docstring now says plainly that nothing on the wire fills them. _numeric fields go through _as_int_or_none, so a JSON string or a bool arriving where a money amount belongs reads as absent rather than landing on an int-typed attribute and failing at the arithmetic. Three existing tests were green because their fixtures matched the parser rather than the server: test_v3_wire_contract, test_route_track_enforcement_propagates and test_2026_09_10_catchfanin_passthrough each built a `details` block with reserved_cents / max_allowed_cents / epsilon_cents, keys the backend has never emitted. Fixtures corrected to the real body; the propagation assertions they were written to make are unchanged. New: tests/test_overbudget_wire_fields.py, including the x10 the millicents/cents mix requires — a caller who skips it reads an overshoot three orders of magnitude too large. Suite 1797 passed / 11 skipped.
check_workflow_budget() substituted no_impact() when the call context
carried no envelope, putting {"kind":"none"} on the wire. The gate's
enum is Money | ToolCall, so the request was refused by the
deserialiser with 422 before any policy ran.
business_impact is Option on the wire, so the honest encoding of 'no
envelope' is absence. The digest stays: it is mandatory at protocol
>= 3, and is still computed over the same no_impact() canonical
bytes, so a stored digest does not change across the upgrade.
Measured on a live stand before choosing this, not inferred: omitting
the field returns 200 at /gate and 200 at /execute. It is NOT safe on
the approval-consumption path, where the server recomputes the digest
from the live payload and fails CLOSED without it (NR-010) -- but an
approval always originates from a tool call, which always carries a
tool_call envelope, so this branch cannot reach it.
The live test is opt-in (NULLRUN_LIVE_STAND_URL) because CI has no
stand. It exists because 34 of 111 test files mock the transport, and
a mocked transport cannot find a client/server enum skew -- which is
how this shipped. Two mutations redden it independently: restoring
the none envelope, and dropping the digest along with it.
Also corrects the test that pinned the defect. It asserted the SDK
sends {"kind":"none"} on the reasoning that 'the backend tells it
apart from an omitted envelope' -- the backend cannot, it has no such
variant. That comment is why the defect looked correct.
The backend error-code parity test in the NULLRUN repo was a blind pin: it mirrored 51 codes by hand, so it stayed green no matter what the backend added. Rewritten to derive from the backend enum, it immediately named 12 codes the SDK had no mapping for. Every one of them fell through _map_v3_error_code's fallback and surfaced as the wrong exception type — a budget ceiling block reported as a generic backend error, a circuit-breaker trip reported as a protocol error. Added mappings for all 12, choosing the type each code's blast radius actually warrants (money ceilings and circuit trips become the blocked/budget types, lookup failures become backend errors, approval digest failures become the DB-unavailable type that already means "do not retry blindly"). Also removed the `if decision == "throttle"` arm in runtime.py. GateDecision has no Throttle variant and the only producer is an uncalled phase-2 schema stub, so the branch was unreachable and "throttle" in _KNOWN_GATE_DECISIONS was dead too. 41 tests pass.
…fraction ADR-068 section 1 (backend NULLRUN). The backend picks the cache-token convention off `provider`: Anthropic reports the cached fraction OUTSIDE `input_tokens`, OpenAI / Gemini / Mistral report it INSIDE `prompt_tokens`. The two are opposite, so the arithmetic is only correct for one of them and the backend has to be told which. `_build_llm_call_event` has always stamped `provider` via `_provider_label(host)`, and the batch path already forwarded it (it ships the whole enriched event). The single-event v3 path whitelists keys explicitly and dropped it. Consequence of leaving it: every cache-using call reaches the backend as an unmapped provider, hits the fail-CLOSED no-discount branch, and is billed at the full input rate -- roughly 10x over on an Anthropic cache read, which is what we billed before ADR-068 landed. Also pins the shape: an absent or empty `provider` stays off the wire, so the backend can still tell "we could not determine the provider" (counted, fail-CLOSED) from a provider it named.
E1–E4 and E9 of ADR-067. Set NULLRUN_EDGE_URL and the enforcement
question goes to a box holding a signed lease instead of to
api.nullrun.io. Unset — the default, and every existing user — and
nothing is constructed and the code path is untouched.
Three things this refuses to do:
* fall back to the cloud. A silent fallback is a fail-open wearing a
disguise: the operator reads "the box said yes" while the cloud
answered and the box was never asked. An unreachable box is a
refusal, and under STRICT that refusal is final.
* let the caller write the money. The wire carries a model and two
token counts. The box prices it from the rates the cloud signed into
the grant. The SDK reports tokens because only it knows them; it
does not get to say what they cost (DEF-40, client side).
* pick the lease for itself. NULLRUN_EDGE_LEASE_ID is required, not
discovered — a box that chose its own budget enforces a number
nobody granted.
/enforce already decides AND charges, so /execute has no box
equivalent and makes no call; re-asking would either reach the cloud or
double-charge the same event.
The runtime fix is the load-bearing part. _require_gate_decision
rejects any body whose decision_source it does not know, and
edge_lease was not in the set — so every via-edge decision would have
raised NullRunMalformedGateResponseError. That fails CLOSED on its own
successes, which in an outage reads as "the box is broken" rather than
"the SDK has not heard of this source yet". The name is added by
import, not by literal, so the allowlist and the producer cannot drift
into two spellings of one value. The check itself is unchanged and
pinned: no source, a near-miss spelling, and a non-string all still
raise. An allowlist is a list of real authorities, not a hole.
27 new tests, 1828 green. The live two-process check lives in
scripts/edge_live_check.py, outside pytest on purpose — it asserts a
claim about two processes talking over a socket, which no mock proves.
Every claim this slice makes is about two processes talking over a socket. A unit test proves the SDK builds the right dict; only this proves the box accepts it, prices it, and refuses the next call. Six phases, in the order the money moves: direct is still the default; a call inside the grant is allowed and attributed to the lease rather than to the gateway; the box's counter moves by what the SDK's token counts imply; an unknown model costs more and still runs; spending the exact remainder lands on zero and not over; one more call is refused with LEASE_EXHAUSTED; and an unreachable box blocks as a synthetic decision without consulting the cloud. Deliberately outside tests/: pytest must stay runnable with no stand and no docker. Each phase fails loudly rather than being skipped.
The gate exists to catch a lock-timing bug, and lock timing is exactly what a machine with cores to spare hides. A second process that should lose the race wins it instead when the holder is descheduled late, or wins every time because it was always the one on the free core. Running once on ONE cpu removes the scheduler as a variable, so a pass cannot be explained by the machine being generous. Unset, it still runs at full width.
`(last.get("remaining_millicents") or -1) == 0` cannot pass. Zero is
the value the assertion exists to observe, and `0 or -1` is -1, so the
check reported the grant as NOT at zero on the one run where it was
exactly zero.
It failed against a live box for exactly that reason: the box returned
remaining=0, which is correct, and the assertion meant to prove it said
it had failed. This is the failure mode acceptance 0.1 exists to catch —
the green that is not green — pointed at itself rather than at the
product.
ADR-068 §4f. The owner's mandate: the SDK passes tokens to the backend and never computes cost on the client. Cost is computed and accounted for on the backend. The SDK was not computing a price — `_build_v3_track_payload` hardcoded `cost_cents: 0`. But sending the field at all kept the shape of a client-priced request alive on every single track call, which is the surface ADR-068 exists to close. An older SDK that still sends it is accepted and ignored (the backend field is `#[serde(default)]`), so this is accept-and-drop, not a breaking wire change — no protocol bump, no synchronous release. Not touched: the SDK's `local_cost_cents` return field. It is always zero, never on the wire, and it is public return shape rather than accounting — flagged in the ADR, left alone. Tests: 1827 passed, 12 skipped. The wire-contract test now asserts ABSENCE of `cost_cents` on both a minimal and a fully populated payload — an exact-dict match is what turns "the SDK never offers a price" into a checked property rather than a comment.
The 0.22.0 trailer check (74cdce5) is a CI backstop for a hook that was not in the tree. A contributor who runs pre-commit gets a 'fails before the commit exists' experience; one who does not install pre-commit gets a CI error after the push. The two are deliberately layered: the hook is the strong place, the workflow is the backstop for the case the hook cannot cover (no pre-commit installed, a --no-commit push, a CI-bypass on the contributor's side). Without this commit, only the backstop exists and the strong place is a paragraph in the commit message of 74cdce5.
_float_env and _int_env were typed as "dict[str, str]", but the caller passes os.environ (an _Environ[str]) when no override env argument is given. Both are read-only mappings, so the call worked at runtime, but mypy narrowed source to the union and flagged two arg-type errors at the call sites. Mapping[str, str] is the more honest type for a function that only does source.get on its argument.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The SDK's enforcement question can now go to a local box that holds a
signed lease instead of to
api.nullrun.io— opt-in viaNULLRUN_EDGE_URL, refuses to fall back to the cloud. The write-aheadlog was reworked around three loss paths the previous release did not
cover: a
kill -9while a refused event was held, a 200 batch thathid per-item refusals, and a recovery that unlinked the WAL before
the events were provably safe. ADR-068 closes the wire: the SDK no
longer puts a price on the wire (the backend prices the cached
fraction off
provider, and the SDK reports tokens because only itknows them).
The full release notes (Migration, Security, Fixed, Changed, Tooling,
Verification, Commits) live in
CHANGELOG.mdunder## [0.22.0].Fixed
The release is organised into 5 themes, each prefixed with a DEFS tag
that ties to the per-commit DEF tags in commit bodies:
EDGE_LEASEdecision source is plumbed through the allowlistbuilder, the gate's typed dispatcher, and the per-event provider
whitelist. A held refusal survives
kill -9and is recovered onthe next startup, a long
Retry-Afteris no longer sat through(we re-check immediately), and the env helpers were widened
from
dict[str, str]toMapping[str, str]soos.environsatisfies the type. The two arg-type errors mypy flagged at
edge.py:367-368are tracked debt in[[tool.mypy.overrides]](per-file pattern, see Tooling).kill -9while a refused event was held — the WAL is now written before
the
.inflightmarker, and recovery re-queues held itemsinstead of dropping them; (2) a 200 batch that hid per-item
refusals — the 200 is settled per-item, so a partial 2xx no
longer reports as success; (3) recovery that unlinked the WAL
before events were provably safe — the
.inflightis clearedonly after the backend dedups the batch on
event_id. Filesare created
0600rather than throughopen()(the umaskpath was leaving prompt + completion + tool-args world-readable
in
/tmpand shared volumes); an advisoryflockserialisesthe file mutation without serialising the network send; the
DLQ size cap refuses the append at the cap rather than
silently dropping. WAL guarantees are now explicit, private,
and observable.
before classifying them, and the breaker was miscounting
protocol errors as backend-health failures. Both are now
trapped inside the call: 4xx is the server answering, a
local serialization
TypeErroris our own bug, and neithersays anything about the breaker. The 12 unmapped backend
error codes are now typed into the catalog with the exception
each blast radius actually warrants (money ceilings and
circuit trips become blocked/budget types, lookup failures
become backend errors, approval digest failures become
the DB-unavailable type that means "do not retry blindly").
CONSUME_OVERBUDGETisdelivered, not quarantined — the dispatcher was routing the
typed 422 to the same
except Exceptionarm that dropsnetwork errors, so a fail-CLOSED enforcement rejection
presented as a soft drop. The 422 overage fields are now
read by name (
reserved_millicents,max_allowed_millicents,actual_cost_cents,soft_pass) instead of by thecents-typed keys the backend never published.
cost_centsis gone from the wire — only tokens go up, and the
cached fraction is priced off
providerserver-side.tested zero with
or:(last.get("remaining_millicents") or -1) == 0cannot pass because0 or -1is -1. Failedagainst a live box on the one run where the box returned
remaining=0, which is the value the assertion exists to
observe. The acceptance 0.1 mode catches this — the green
that is not green — and the fix changed the test to
is Noneshort-circuiting.Tooling
Co-Authored-Bytrailer on this branch — acommit-msgpre-commit hook (scripts/reject_coauthor_trailer.sh)is the strong place (it fails before the commit exists); a
workflow step is the backstop for the case a hook cannot cover
(no pre-commit installed, a
--no-commitpush, a CI bypass).Scope is
origin/master..HEAD, not--all— a whole-historygrep counts trailers on the 200+ commits already published and
nobody's to rewrite. (
fc73d05, ef11c3e)edge.pydynamic-dispatch debt —the
[[tool.mypy.overrides]]block fornullrun.edgeextendswith
call-argandarg-typedisabled for the env-helpersignature, with an inline comment explaining the
Mapping/os.environwidening that would have flagged thempre-fix. The pre-existing 5 in
instrumentation/auto.pyreproduce identically without these changes, same shape as
the 0.21.0 6-error baseline note.
Verification
ruff check src testsmypy src/nullruninstrumentation/auto.pyreproduce identically without these changes, same shape as 0.21.0's 6-error baseline note)pytest -qnullrun.__version__0.22.0/tracksingle (thecost_centsfield is removed; theproviderfield is added when set; the 422 overagedetailsblock has renamed fields — see Migration); additive on/gate(edge_leaseis added to thedecision_sourceallowlist, but the gate is internal to the SDK and only via-edge emits it)Commits included