Repository navigation
Removing the implicit 90 s model timeout leaves review runner occupancy unbounded — needs an admission/continuation bound first #2139
Description
The situation
contextual-orchestrator#1118 removed ModelClient.__init__'s implicit per-request wall-clock timeout: timeout: int = 90 at the currently vendored pin 414f2297, timeout: float | None = None at 012beaac. That removal is correct — an implicit 90-second cap on the inference path is exactly what ADR-0003's 2026-08-31 amendment forbids and what #1889/#1890/#1892 were reverted for reintroducing. It is not the problem. It is, however, load-bearing today in a way nobody planned for, and this issue is about what has to exist before it goes away.
What the 90 s cap is currently doing
Production evidence, #1884 head e85fc437, run 34732993973, sidecar stderr artifact 10310273053. The serving phase ran 02:41:17Z → 02:56:27Z (907.3 s) and ended HTTP 503. Inside that window the gateway made 15 provider attempts across two agents on two credential accounts:
10x nvidia_nim / deepseek-ai/deepseek-v4-flash-0731
5x nvidia_nim_sub / deepseek-ai/deepseek-v4-pro-0813
Measured attempt→failure durations:
90.0s 90.0s 22.0s 90.0s 90.0s
90.0s 90.0s 90.0s 90.0s 0.0s
Eight of ten land on exactly 90.0 s. That is not the providers answering — it is ModelClient's default timeout firing, over and over. Roughly 720 of the 907 seconds is the cap expiring against endpoints that never respond.
Circuit breaker in the same window: circuit_failure ×10, circuit_opened ×2, circuit_reset ×2, circuit_cleared ×1. The breaker opens at failures=3.0 threshold=3, resets 30 s later, and the ranking hands the same route straight back.
Why removing the cap makes this worse before it makes it better
At the target pin, nothing bounds a single attempt:
contextual_orchestrator_review_launcher.py:1240builds the serving client asModelClient(max_output_tokens=..., temperature=...)— notimeoutargument, so it takes the library default.noema-review.yml:261states "No job-leveltimeout-minuteshere, deliberately."
So a socket that accepts a connection and then delivers nothing has no bound below GitHub's 6-hour job ceiling. Today's failure mode — 15 minutes, bounded, with a clear 503 and a full attempt trace — becomes an indefinite runner hold with one open attempt and no verdict. That is a worse operational outcome even though the change causing it is the correct one.
What must NOT be done
Re-adding a wall-clock timeout. docs/product-goal-directive.md §8 accepts that a model path may take more than two hours and states speed is not a core consideration; #1889, #1890, and #1892 each added a 900-second cap on genuine multi-hour-hang evidence and were all reverted (#1891, #1895). The directive names the alternative explicitly: fix runner occupancy at the admission/continuation boundary instead; never convert elapsed inference time into a model-failure verdict. This issue is not a request to reopen that.
What to build instead
The distinction the reverted PRs collapsed, and which this evidence makes concrete: a model streaming tokens for two hours is healthy; a socket that has delivered zero bytes for fifteen minutes is not inference at all. The bound belongs on progress, not on total elapsed time:
- Socket-idle bound, not a deadline. Reset on every byte received. A response actively streaming is never interrupted regardless of total duration; a connection that has produced nothing since the request was written is a black hole and can be abandoned without any claim about how long the model "should" take.
- Continuation admission. Once the breaker has opened on an agent, a reset should not immediately re-offer the same route while equally-ranked alternatives have not been tried. In this run, two ready routes (
nvidia_nim/meta/llama-3.2-11b-vision-instructandnvidia_nim_sub/meta/llama-3.2-11b-vision-instruct, bothreadyat preflight) were never attempted during serving while the two deepseek routes were cycled fifteen times. - Occupancy ceiling at the job boundary, not the inference boundary. If a runner must be released, release it as an explicit occupancy decision with its own event — never as a model-failure verdict attributed to the provider.
Item 1 needs an ADR, since "is an idle-socket bound a wall-clock timeout?" is exactly the question the reverts turned on and it should be settled in writing before code.
Sequencing
#2137 advances the pin to 012beaac and therefore carries this exposure. I have flagged it there rather than merging past it. Options, for the owner to decide:
- land the occupancy bound first, then
#2137; - land
#2137and accept a window where a hung provider can hold a review runner for up to 6 hours; - land
#2137with an explicittimeout=at the launcher's serving call site — which would just reinstate the reverted cap under a different name, and I am not proposing it.
Found while root-causing #1884's noema-review failure. Refs #1915, #2000, #2137, ContextualWisdomLab/contextual-orchestrator#1118.
Activity
seonghobae commentedon Sep 13, 2026
ADR drafted as #2140 (
docs/adr/0030-review-runner-occupancy-progress-bound.md, Proposed, docs-only, 3039 tests passing).It settles the question this issue said had to be settled in writing first: an idle-socket bound is not a wall-clock deadline, because it cannot expire on a request that is making progress. That is the distinction
#1889/#1890/#1892collapsed, and leaving it to code review is how all three happened.Carries the three items from this issue as the decision — idle-socket bound reset on every byte, continuation admission so a breaker reset does not re-offer the failing agent while preflight-
readyalternatives sit untried, and runner reclamation as its own job-boundary event — plus the#1884trace as evidence and the explicit non-goals (no total-duration limit; no threshold value until time-to-first-byte is measured).One thing the ADR makes sharper than this issue did: the current 90 s cap is not merely load-bearing, it is also doing harm. The same trace has
google/gemma-4-31b-itrejected at preflight after exactly 90.0 s, which is indistinguishable from a model that needed 91 seconds. So the org holds both defects simultaneously — a cap that discards slow-but-working routes, and no bound at all on a dead socket once it is removed. That is an argument for landing the replacement rather than for keeping the cap.Item 2 (continuation admission) is independent of the timeout question entirely and is the cheaper half; it would have shortened the 907-second run on its own, since two ready routes on the same accounts were never tried once. If item 1 stays contentious, item 2 can still go.
Generated by Claude Code
seonghobae commentedon Sep 14, 2026
Fresh production evidence strengthens this issue beyond the earlier single-run trace. After protected #2163 advanced the central review sidecar to
contextual-orchestrator@767e67fbc6b881a452761f32abb69b9971b9b03b(null default model timeout), #712 measured 32 in-progress jobs and 113 queued with no self-hosted runners; 31 in-progress review jobs were still current-head work and therefore not valid cancellation targets. The longest holders were spending roughly 3.5–5+ hours inProvision contextual-orchestrator review sidecar.This separates two causal repairs that should not be collapsed into a timeout rollback:
.github#1629already has a provider-account concurrency implementation and synchronization-barrier regression so independent credential accounts can probe concurrently while same-account routes remain serial; its current branch is stale/non-mergeable and has its own no-heuristic RED, so it must be repaired/reconciled rather than copied into a competing branch;The observed queue starvation is therefore no longer hypothetical exposure from removing 90 seconds: long-running current-head review provisioning is consuming the finite hosted concurrency pool and preventing unrelated exact-head jobs from acquiring runners. Acceptance for this lane should include unchanged-head evidence that review provisioning makes bounded forward progress/reclaims occupancy without converting elapsed inference time into provider failure, plus #712 evidence that unrelated required jobs can again acquire a runner. Final provider admission/routing/TTC ownership still belongs to contextual-orchestrator#1106 and the immutable release migration tracked by #1759.
seonghobae commentedon Sep 14, 2026
four-pillars consumer-side saturation evidence, 2026-09-14T08:12Z
Measured from a consumer repository, showing the occupancy problem has now reached scheduled sentinels and fresh PRs, not just review jobs.
Organization
.githubat 08:12:54Z:ContextualWisdomLab/four-pillarsat the same moment — 11 runs queued, none started:schedule)schedule)4187b0f(CI, CodeQL, and the required-workflow set)pendingat 0sTwo consequences worth separating from ordinary latency:
mainwould go unreported and the loop's failure-issue lifecycle cannot open or close anything.502/429transport failures tracked in noema-review fails closed on provider transport errors (502/429) after gateway failover; blocks unchanged consumer PRs #2165 and from the CodeQL wake-credential defect in CodeQL dispatch terminal status publication remains unproven across repositories #1929; here no job is dispatched at all.This is consistent with the mechanism described in this issue: long-lived serving phases hold runner slots, and with no admission or continuation bound the queue drains slower than it fills, so the backlog crosses from review workloads into unrelated scheduled and PR workloads in consumer repositories.
No action requested from four-pillars' side; recording the consumer-visible blast radius as input to the admission-bound design.
seonghobae commentedon Sep 14, 2026
LineageWeave exact-head production evidence — 2026-09-15
Fresh consumer evidence from
ContextualWisdomLab/LineageWeave#1079at unchanged exact headc2923950e73c88a9f9fd932332ddd47682da124bconfirms both occupancy failure modes already owned by this issue / #2140 without requiring any LineageWeave source change.Noema — platform cancellation after sidecar provisioning never reaches readiness
Required Noema run
34815432029, job103965387845, received a hosted runner at2026-09-14T16:50:29Z. Exact-head admission, stale-trigger rejection, credential selection, GitHub App token minting, live-head validation, and target-visibility resolution all succeeded.Provision contextual-orchestrator review sidecarthen started at16:50:39Z; the sidecar vendoredcontextual-orchestrator@767e67fbc6b881a452761f32abb69b9971b9b03b, installed its pinned dependencies, selected the live OpenRouter ZDR feed, and loggedstarting review sidecar on 127.0.0.1:18080at16:52:41Z. No readiness receipt followed. At22:51:33Z, after ~6 hours of runner occupancy, GitHub emitted onlyThe operation was canceled.and killed the orphan sidecar processes. No Noema model verdict step ever ran.This is not evidence for a total model deadline: the model call was never reached. It is fresh evidence for the existing progress/occupancy distinction. Startup/provisioning needs observable forward-progress semantics and an explicit admin/platform-reclamation outcome rather than an opaque
cancelledresult. The trusted workflow itself deliberately has no job-level model timeout; this run demonstrates why startup/provisioning and model elapsed time must remain separately classified.Strix — complete scan report but fail-closed because caller saturation exceeded CO admission
Required Strix run
34815432062, job103970045852, also ran against exactc2923950e.... Trusted-source validation, CO sidecar provisioning, and Strix installation succeeded. Its uploadedstrix-reportsartifact10371857268(digestsha256:b0d69ccbdf5b4fa52c99e118a9d76e6d4251d34d7deb9341131b9ce8270dd13b) contains a normal terminal Strix report and SARIF with zero findings, but the gate correctly failed closed because concurrent Strix sub-agent calls received repeated CO503 concurrency_limit_exceeded/too many concurrent orchestration runsresponses. The report explicitly ended with provider-unavailable/failure-signal handling rather than promoting the partial zero-finding result to GREEN.CO protected behavior intentionally rejects saturated inference with
503 concurrency_limit_exceeded; therefore the consumer repair is not to weaken the gate, raise/override provider/model policy in LineageWeave, or reinterpret the clean SARIF as complete evidence. Acceptance should cover caller-side admission/backpressure relative to the released CO capacity contract so a Strix scan does not overrun the sidecar's bounded run slots, while keepingorchestrator/free, no paid/provider fallback,timeout=None, and the gateway's explicit overload semantics.These two observations strengthen the existing sequencing rather than creating a competing owner: #2170 prerequisite → non-force #1629 reconciliation for provisioning/admission → #2139/#2140 progress/occupancy semantics, with final provider admission/routing remaining in contextual-orchestrator#1106/released CO. LineageWeave #1079 remains fail-closed until fresh terminal Noema/Strix evidence exists.
seonghobae commentedon Sep 15, 2026
Owner-path convergence update from the LineageWeave #1079 evidence lane: Draft ADR PR #2140 is still based on
fb17ef556f94f673234aa557254ae52779e9a7b0and GitHub reports it non-mergeable against current protected.github/main@91be6442906c7b6b4f600272c953699708394327. I did not force-push or destructively rebase it.Current-main Draft reconstruction #2207 is open at exact
36a6755a5b699d8f43269805b6fccd5c5d75eae5, directlyahead_by=2, behind_by=0from protected main. Its ADR-0030 file has the same blob SHA as #2140 (1c133061cccb6bca9c3552cb8bdbca43b17d19f1); the original changelog evidence is carried throughCHANGELOG.d/20260915-adr-0030-review-runner-occupancy.mdso concurrent rootCHANGELOG.mdwork is not overwritten. #2207 remains Draft and #2140 remains open until current-head checks and independent review prove complete succession.No LineageWeave runtime/provider/model/timeout policy was added. #2139 remains the evidence/problem owner; #2207 only reconstructs the Proposed decision record on current protected owner state.
seonghobae commentedon Sep 15, 2026
What unbounded occupancy costs a consumer's scheduled workload, measured
This issue argues that occupancy needs an admission bound. Here is the downstream cost, measured on
ContextualWisdomLab/four-pillars, whosehourly-product-loop.ymlis a read-only quality sentinel scheduled every hour.Across the 100 most recent
scheduleevents:Six or seven of twenty-four, for five consecutive days. That is a stable regime, not a spike.
The gap distribution is sharper still:
The minimum is 1.70 hours, so not one run in the last hundred fired on its scheduled hour. The two most recent, created 2026-09-14T23:52Z and 2026-09-15T04:54Z, had still not started when I measured at 05:27Z, the older of them queued for five and a half hours.
Why the scheduled workload is the one that silently loses
A pull-request check that waits is visibly pending and someone eventually looks at it. A dropped
scheduleevent leaves nothing behind: GitHub does not queue a missed cron occurrence, so the run simply never exists. The consumer repository has no signal that its hourly contract is being met roughly a quarter of the time, and its own documentation still describes the workflow as hourly.That makes scheduled work the quietest casualty of unbounded occupancy, and it is the part an admission bound would protect most cheaply: a sentinel run is short and read-only, so it is exactly the shape of work that should not be starved by long-running review jobs.
Consumer-side measurement only. I have no visibility into the admission policy and am not proposing a specific bound.
seonghobae commentedon Sep 17, 2026
LifeOS exact production evidence — 2026-09-18
Fresh consumer evidence now reaches the platform ceiling this issue was designed to prevent.
ContextualWisdomLab/life-os#249@484652e094e4f197bfa1a9ca21211c6bd6d32aac, Required Strix run35173903681, job105121637284:Provision contextual-orchestrator Strix sidecarstarted2026-09-17T13:45:16Zand ended cancelled at19:45:48Z, about six hours later;Install StrixandRun Strix (quick)never ran, so this is not a Strix source/security verdict;strix-reportsartifact10516014728(sha256:959cbd3253e95369712ed822d60b5af6cfdd0809278a00cf12cc80fa8a399eb5) contains an empty preflight JSON and sidecar stderr. The trace shows OpenRouter free routes failing quickly with 429, then NIM attempts beginning at 13:50:40 / 13:54:58 / 13:58:41 / 14:01:59. After thenvidia_nim_sub/deepseek-ai/deepseek-v4-flash-0731attempt at 14:01:59 there is no later provider event before the job is cancelled at 19:45:48.This is stronger than the earlier hypothetical six-hour exposure: current production actually held a hosted runner until the platform/job ceiling while still inside sidecar provisioning. It also preserves the boundary in this issue: do not turn six-hour elapsed time into a provider/model failure and do not restore a total model deadline. The missing control is progress/idle-aware continuation and explicit runner-reclamation evidence.
Current protected central sidecar source still waits for
/healthzin an unboundeduntilloop as long as the sidecar PID remains alive. That loop cannot distinguish a live-but-progressing startup from a live-but-silent transport attempt; GitHub cancellation therefore becomes the first terminating authority in this exact incident. The artifact's last-event gap is useful RED evidence for the progress/idle contract, not enough evidence to invent a provider-specific timeout threshold.Canonical source/decision ownership remains here and in current-main successor #2207 (ADR-0032). LifeOS should remain fail-closed and should not copy a timeout, provider route, or central sidecar implementation.
seonghobae commentedon Sep 17, 2026
Owner-path refinement from the LifeOS Strix cancellations: existing central PR #1629 is the canonical source lane for evidence-only review admission and for removing the redundant shell-authored second
/v1/chat/completionsreplay after launcher preflight. I reconciled #1629 non-force onto current protectedmain@64aa08d7fa487deacd41c761c36277ca68cab6c9; new exact is5e7e299ab160de38af5d4b60d6704e7d5f410bf1, with fresh Runtime Quality/security/CodeQL runs pending.Important correction from fresh #211 artifact evidence: LifeOS #211 run
35177779881also spent ~6h inProvision contextual-orchestrator Strix sidecarand ended CANCELLED, butcontextual-orchestrator-preflight.jsonis empty. Its sanitized stderr shows OpenRouter 429, NIM 504, then NIM_SUB launcher/provider attempts; healthz was never reached. The earlier #249 artifact likewise had empty preflight evidence. Therefore the observed 6h incidents are pre-healthz launcher-preflight no-progress, not the shell's later duplicate chat-completions replay. #1629 may still improve the launcher/admission path and validly removes redundant post-healthz inference, but it is not yet evidence-backed as the causal fix for these incidents.The direct incident contract stays here / ADR-0032: classify and reclaim progress/idle loss separately from model elapsed time, and do not feed runner-reclaim into provider/model failure, routing, ranking, or circuit-breaker evidence.
seonghobae commentedon Sep 24, 2026
LifeOS current-head occupancy specimen — 2026-09-24
Fresh consumer evidence from
ContextualWisdomLab/life-os#304isolates the same progress/occupancy boundary without a leaf-source failure.392e890f072f4836f16160a09d2d05607ac13996overmain@193a87ef54c3fe6dcda4755bce4d6bc81e3a0297.35941341081started at2026-09-24T01:04:59Zand remainsin_progressmore than 11 hours later on that unchanged exact head.noema-reviewjob107506401923, stale-trigger rejection, reviewer credential selection, GitHub App token minting, live-head validation and target visibility all completed successfully.Provision contextual-orchestrator review sidecar. No Noema verdict preparation/publication step has started, so this is not model-quality evidence and not a provider-failure verdict.life-osrun currentlyin_progress, while the repository has 73 runs inqueuedstate. That count is repository-local evidence only; it is not an organization-wide concurrency count.This specimen therefore belongs here / #2140 for progress-aware sidecar provisioning and explicit occupancy reclamation, and to #712 for queue-health classification. It should not be repaired in LifeOS by a no-op wake commit, blind rerun, provider/model fallback, elapsed model timeout, runner-selector churn, or required-gate weakening. #304's already-proven Planning Search semantic lineage remains separate from this normal required-review gate.
Acceptance remains unchanged-head evidence that the central sidecar makes bounded forward progress or emits an explicit non-provider occupancy/reclamation outcome, followed by actual runner admission for the queued consumer evidence. No duplicate owner implementation is being created in LifeOS.
seonghobae commentedon Sep 24, 2026
2026-09-24 owner-path update from the LifeOS occupancy specimen: current-main successor #2207 has been ordinary/non-force restacked onto protected
.github/main@e6334e229581a918e2f22de18733b76fa65d7e71. New exact is31a14c821d0cdd6613c660ecd84c32b215b1a147; fresh compare is ahead-only with merge base exactly current protected main and still exactly the four intended ADR/doctoring/CHANGELOG paths. Fresh exact-head Security Scan36003800625, SAST36003800709, CodeQL36003800741, and Python Security36003800786are queued, so this is current ancestry repair rather than accepted owner GREEN. LifeOS #304 run35941341081remains current-headin_progressat the Noema sidecar provision boundary; do not classify or cancel it from elapsed time alone. The LifeOS specimen therefore continues to support #2139/#2207's progress/admission distinction, while runtime/continuation repair remains owner work rather than a LifeOS-local timeout or provider fallback.seonghobae commentedon Sep 24, 2026
2026-09-24 fresh LifeOS consumer evidence broadens the current occupancy specimen from Noema to Strix without changing the owner boundary.
Current
ContextualWisdomLab/life-osActions inventory is 3 in-progress / 62 queued. All three in-progress runs are current-head work, so elapsed time alone is not a cancellation criterion:392e890f072f4836f16160a09d2d05607ac13996: Required Noema Review35941341081, job107506401923, still inProvision contextual-orchestrator review sidecar; admission, credentials and live-head validation completed, verdict/publication have not started.24b5a2ebcebee27eb696146f04be31458c767bff: Required Strix35962913202, job107577317153, now inProvision contextual-orchestrator Strix sidecar; setup through secret gating succeeded, Strix install/scan/report steps have not started.28b5fa85625d96b8dae3512abebbac3970a7bf22: Required Strix35963315457, job107577786985, same boundary: setup through secret gating succeeded,Provision contextual-orchestrator Strix sidecarremains in progress and scanner steps have not started.This is useful because the queue-pressure symptom now reproduces simultaneously through both central review consumers while unrelated exact-head verifier/CI runs remain queued. It strengthens the existing #2139/#2207 distinction: reclaim/admit on explicit progress/idle/continuation evidence, not on total model elapsed time, and do not feed runner-reclamation into provider/model failure, routing, ranking or circuit-breaker semantics.
LifeOS should remain a consumer here: no local elapsed-time timeout, provider/model fallback, required-gate weakening, or duplicate sidecar implementation. Current-main #2207 remains the owner path; these LifeOS specimens are acceptance evidence for that lane.
seonghobae commentedon Sep 24, 2026
LifeOS current-head specimen has now reached a terminal state and strengthens this owner finding.
ContextualWisdomLab/life-os#304@392e890f072f4836f16160a09d2d05607ac13996, Required Noema run35941341081, completed cancelled at 2026-09-24T14:17:41Z. Admission, changed-scope classification, stale-trigger rejection, credential selection, GitHub App token mint, exact live-head validation, and visibility resolution all succeeded. Thenoema-reviewjob then remained inProvision contextual-orchestrator review sidecarfrom 08:16:50Z until 14:17:37Z (~6 h) and was cancelled there.Prepare Noema model verdict, bounded transport re-dispatch, evidence upload, and exact-head verdict publication were all skipped.This is not a model/provider failure verdict and should not be converted into one. The run payload proves terminal runner release without semantic review output; it does not by itself prove which administrative mechanism issued the cancellation, so keep that distinction explicit. The practical result is that the current exact LifeOS head still has no Required Noema verdict after occupying a runner for roughly six hours.
Owner implication remains the same but is now terminally reproduced: implement progress/idle-aware sidecar lifecycle and explicit occupancy/continuation semantics in the central reusable workflow/CO boundary. Do not restore an elapsed inference timeout, add a LifeOS-local timeout/provider fallback, or treat this cancellation as a valid review result. Refs LifeOS #304 and the earlier current-head occupancy evidence in this issue.