Skip to content

Removing the implicit 90 s model timeout leaves review runner occupancy unbounded — needs an admission/continuation bound first #2139

Open

Description

@seonghobae

The situation

contextual-orchestrator#1118 removed ModelClient.__init__'s implicit per-request wall-clock timeout: timeout: int = 90 at the currently vendored pin 414f2297, timeout: float | None = None at 012beaac. That removal is correct — an implicit 90-second cap on the inference path is exactly what ADR-0003's 2026-08-31 amendment forbids and what #1889/#1890/#1892 were reverted for reintroducing. It is not the problem. It is, however, load-bearing today in a way nobody planned for, and this issue is about what has to exist before it goes away.

What the 90 s cap is currently doing

Production evidence, #1884 head e85fc437, run 34732993973, sidecar stderr artifact 10310273053. The serving phase ran 02:41:17Z → 02:56:27Z (907.3 s) and ended HTTP 503. Inside that window the gateway made 15 provider attempts across two agents on two credential accounts:

10x  nvidia_nim     / deepseek-ai/deepseek-v4-flash-0731
 5x  nvidia_nim_sub / deepseek-ai/deepseek-v4-pro-0813

Measured attempt→failure durations:

  90.0s   90.0s   22.0s   90.0s   90.0s
  90.0s   90.0s   90.0s   90.0s    0.0s

Eight of ten land on exactly 90.0 s. That is not the providers answering — it is ModelClient's default timeout firing, over and over. Roughly 720 of the 907 seconds is the cap expiring against endpoints that never respond.

Circuit breaker in the same window: circuit_failure ×10, circuit_opened ×2, circuit_reset ×2, circuit_cleared ×1. The breaker opens at failures=3.0 threshold=3, resets 30 s later, and the ranking hands the same route straight back.

Why removing the cap makes this worse before it makes it better

At the target pin, nothing bounds a single attempt:

  • contextual_orchestrator_review_launcher.py:1240 builds the serving client as ModelClient(max_output_tokens=..., temperature=...) — no timeout argument, so it takes the library default.
  • noema-review.yml:261 states "No job-level timeout-minutes here, deliberately."

So a socket that accepts a connection and then delivers nothing has no bound below GitHub's 6-hour job ceiling. Today's failure mode — 15 minutes, bounded, with a clear 503 and a full attempt trace — becomes an indefinite runner hold with one open attempt and no verdict. That is a worse operational outcome even though the change causing it is the correct one.

What must NOT be done

Re-adding a wall-clock timeout. docs/product-goal-directive.md §8 accepts that a model path may take more than two hours and states speed is not a core consideration; #1889, #1890, and #1892 each added a 900-second cap on genuine multi-hour-hang evidence and were all reverted (#1891, #1895). The directive names the alternative explicitly: fix runner occupancy at the admission/continuation boundary instead; never convert elapsed inference time into a model-failure verdict. This issue is not a request to reopen that.

What to build instead

The distinction the reverted PRs collapsed, and which this evidence makes concrete: a model streaming tokens for two hours is healthy; a socket that has delivered zero bytes for fifteen minutes is not inference at all. The bound belongs on progress, not on total elapsed time:

  1. Socket-idle bound, not a deadline. Reset on every byte received. A response actively streaming is never interrupted regardless of total duration; a connection that has produced nothing since the request was written is a black hole and can be abandoned without any claim about how long the model "should" take.
  2. Continuation admission. Once the breaker has opened on an agent, a reset should not immediately re-offer the same route while equally-ranked alternatives have not been tried. In this run, two ready routes (nvidia_nim/meta/llama-3.2-11b-vision-instruct and nvidia_nim_sub/meta/llama-3.2-11b-vision-instruct, both ready at preflight) were never attempted during serving while the two deepseek routes were cycled fifteen times.
  3. Occupancy ceiling at the job boundary, not the inference boundary. If a runner must be released, release it as an explicit occupancy decision with its own event — never as a model-failure verdict attributed to the provider.

Item 1 needs an ADR, since "is an idle-socket bound a wall-clock timeout?" is exactly the question the reverts turned on and it should be settled in writing before code.

Sequencing

#2137 advances the pin to 012beaac and therefore carries this exposure. I have flagged it there rather than merging past it. Options, for the owner to decide:

  • land the occupancy bound first, then #2137;
  • land #2137 and accept a window where a hung provider can hold a review runner for up to 6 hours;
  • land #2137 with an explicit timeout= at the launcher's serving call site — which would just reinstate the reverted cap under a different name, and I am not proposing it.

Found while root-causing #1884's noema-review failure. Refs #1915, #2000, #2137, ContextualWisdomLab/contextual-orchestrator#1118.

Activity

  1. seonghobae commented on Sep 13, 2026

    @seonghobae
    ContributorAuthor

    ADR drafted as #2140 (docs/adr/0030-review-runner-occupancy-progress-bound.md, Proposed, docs-only, 3039 tests passing).

    It settles the question this issue said had to be settled in writing first: an idle-socket bound is not a wall-clock deadline, because it cannot expire on a request that is making progress. That is the distinction #1889/#1890/#1892 collapsed, and leaving it to code review is how all three happened.

    Carries the three items from this issue as the decision — idle-socket bound reset on every byte, continuation admission so a breaker reset does not re-offer the failing agent while preflight-ready alternatives sit untried, and runner reclamation as its own job-boundary event — plus the #1884 trace as evidence and the explicit non-goals (no total-duration limit; no threshold value until time-to-first-byte is measured).

    One thing the ADR makes sharper than this issue did: the current 90 s cap is not merely load-bearing, it is also doing harm. The same trace has google/gemma-4-31b-it rejected at preflight after exactly 90.0 s, which is indistinguishable from a model that needed 91 seconds. So the org holds both defects simultaneously — a cap that discards slow-but-working routes, and no bound at all on a dead socket once it is removed. That is an argument for landing the replacement rather than for keeping the cap.

    Item 2 (continuation admission) is independent of the timeout question entirely and is the cheaper half; it would have shortened the 907-second run on its own, since two ready routes on the same accounts were never tried once. If item 1 stays contentious, item 2 can still go.


    Generated by Claude Code

  2. seonghobae commented on Sep 14, 2026

    @seonghobae
    ContributorAuthor

    Fresh production evidence strengthens this issue beyond the earlier single-run trace. After protected #2163 advanced the central review sidecar to contextual-orchestrator@767e67fbc6b881a452761f32abb69b9971b9b03b (null default model timeout), #712 measured 32 in-progress jobs and 113 queued with no self-hosted runners; 31 in-progress review jobs were still current-head work and therefore not valid cancellation targets. The longest holders were spending roughly 3.5–5+ hours in Provision contextual-orchestrator review sidecar.

    This separates two causal repairs that should not be collapsed into a timeout rollback:

    1. preflight/provision startup latency: existing .github#1629 already has a provider-account concurrency implementation and synchronization-barrier regression so independent credential accounts can probe concurrently while same-account routes remain serial; its current branch is stale/non-mergeable and has its own no-heuristic RED, so it must be repaired/reconciled rather than copied into a competing branch;
    2. transport/continuation occupancy: this issue/docs(adr): bound review runner occupancy by progress, not elapsed inference time #2140 owns progress-based idle/continuation/runner-reclamation semantics. A total elapsed model deadline remains prohibited.

    The observed queue starvation is therefore no longer hypothetical exposure from removing 90 seconds: long-running current-head review provisioning is consuming the finite hosted concurrency pool and preventing unrelated exact-head jobs from acquiring runners. Acceptance for this lane should include unchanged-head evidence that review provisioning makes bounded forward progress/reclaims occupancy without converting elapsed inference time into provider failure, plus #712 evidence that unrelated required jobs can again acquire a runner. Final provider admission/routing/TTC ownership still belongs to contextual-orchestrator#1106 and the immutable release migration tracked by #1759.

  3. seonghobae commented on Sep 14, 2026

    @seonghobae
    ContributorAuthor

    four-pillars consumer-side saturation evidence, 2026-09-14T08:12Z

    Measured from a consumer repository, showing the occupancy problem has now reached scheduled sentinels and fresh PRs, not just review jobs.

    Organization .github at 08:12:54Z:

    in_progress = 23
    queued      = 100   (API page cap; actual backlog is at least this)
    

    ContextualWisdomLab/four-pillars at the same moment — 11 runs queued, none started:

    created (UTC) run state
    04:56:51 Hourly Product Quality Loop (schedule) queued 3h16m, never started
    05:29:22 Hourly NVIDIA NIM Product Development (schedule) queued 2h43m, never started
    06:34:00–06:34:02 9 runs for PR #46 head 4187b0f (CI, CodeQL, and the required-workflow set) queued 1h39m, every check still pending at 0s

    Two consequences worth separating from ordinary latency:

    1. The read-only quality sentinel has not executed since 00:28Z. The minute-17 loop is the repository's standing regression detector; while it is starved, a real regression on main would go unreported and the loop's failure-issue lifecycle cannot open or close anything.
    2. A new PR cannot obtain any verdict. PR Ignore transient governance helper cancellations #46 opened at 06:34Z and none of its checks have started, so its required checks are neither passing nor failing — they are absent. That is distinct from the 502/429 transport failures tracked in noema-review fails closed on provider transport errors (502/429) after gateway failover; blocks unchanged consumer PRs #2165 and from the CodeQL wake-credential defect in CodeQL dispatch terminal status publication remains unproven across repositories #1929; here no job is dispatched at all.

    This is consistent with the mechanism described in this issue: long-lived serving phases hold runner slots, and with no admission or continuation bound the queue drains slower than it fills, so the backlog crosses from review workloads into unrelated scheduled and PR workloads in consumer repositories.

    No action requested from four-pillars' side; recording the consumer-visible blast radius as input to the admission-bound design.

  4. seonghobae commented on Sep 14, 2026

    @seonghobae
    ContributorAuthor

    LineageWeave exact-head production evidence — 2026-09-15

    Fresh consumer evidence from ContextualWisdomLab/LineageWeave#1079 at unchanged exact head c2923950e73c88a9f9fd932332ddd47682da124b confirms both occupancy failure modes already owned by this issue / #2140 without requiring any LineageWeave source change.

    Noema — platform cancellation after sidecar provisioning never reaches readiness

    Required Noema run 34815432029, job 103965387845, received a hosted runner at 2026-09-14T16:50:29Z. Exact-head admission, stale-trigger rejection, credential selection, GitHub App token minting, live-head validation, and target-visibility resolution all succeeded. Provision contextual-orchestrator review sidecar then started at 16:50:39Z; the sidecar vendored contextual-orchestrator@767e67fbc6b881a452761f32abb69b9971b9b03b, installed its pinned dependencies, selected the live OpenRouter ZDR feed, and logged starting review sidecar on 127.0.0.1:18080 at 16:52:41Z. No readiness receipt followed. At 22:51:33Z, after ~6 hours of runner occupancy, GitHub emitted only The operation was canceled. and killed the orphan sidecar processes. No Noema model verdict step ever ran.

    This is not evidence for a total model deadline: the model call was never reached. It is fresh evidence for the existing progress/occupancy distinction. Startup/provisioning needs observable forward-progress semantics and an explicit admin/platform-reclamation outcome rather than an opaque cancelled result. The trusted workflow itself deliberately has no job-level model timeout; this run demonstrates why startup/provisioning and model elapsed time must remain separately classified.

    Strix — complete scan report but fail-closed because caller saturation exceeded CO admission

    Required Strix run 34815432062, job 103970045852, also ran against exact c2923950e.... Trusted-source validation, CO sidecar provisioning, and Strix installation succeeded. Its uploaded strix-reports artifact 10371857268 (digest sha256:b0d69ccbdf5b4fa52c99e118a9d76e6d4251d34d7deb9341131b9ce8270dd13b) contains a normal terminal Strix report and SARIF with zero findings, but the gate correctly failed closed because concurrent Strix sub-agent calls received repeated CO 503 concurrency_limit_exceeded / too many concurrent orchestration runs responses. The report explicitly ended with provider-unavailable/failure-signal handling rather than promoting the partial zero-finding result to GREEN.

    CO protected behavior intentionally rejects saturated inference with 503 concurrency_limit_exceeded; therefore the consumer repair is not to weaken the gate, raise/override provider/model policy in LineageWeave, or reinterpret the clean SARIF as complete evidence. Acceptance should cover caller-side admission/backpressure relative to the released CO capacity contract so a Strix scan does not overrun the sidecar's bounded run slots, while keeping orchestrator/free, no paid/provider fallback, timeout=None, and the gateway's explicit overload semantics.

    These two observations strengthen the existing sequencing rather than creating a competing owner: #2170 prerequisite → non-force #1629 reconciliation for provisioning/admission → #2139/#2140 progress/occupancy semantics, with final provider admission/routing remaining in contextual-orchestrator#1106/released CO. LineageWeave #1079 remains fail-closed until fresh terminal Noema/Strix evidence exists.

  5. seonghobae commented on Sep 15, 2026

    @seonghobae
    ContributorAuthor

    Owner-path convergence update from the LineageWeave #1079 evidence lane: Draft ADR PR #2140 is still based on fb17ef556f94f673234aa557254ae52779e9a7b0 and GitHub reports it non-mergeable against current protected .github/main@91be6442906c7b6b4f600272c953699708394327. I did not force-push or destructively rebase it.

    Current-main Draft reconstruction #2207 is open at exact 36a6755a5b699d8f43269805b6fccd5c5d75eae5, directly ahead_by=2, behind_by=0 from protected main. Its ADR-0030 file has the same blob SHA as #2140 (1c133061cccb6bca9c3552cb8bdbca43b17d19f1); the original changelog evidence is carried through CHANGELOG.d/20260915-adr-0030-review-runner-occupancy.md so concurrent root CHANGELOG.md work is not overwritten. #2207 remains Draft and #2140 remains open until current-head checks and independent review prove complete succession.

    No LineageWeave runtime/provider/model/timeout policy was added. #2139 remains the evidence/problem owner; #2207 only reconstructs the Proposed decision record on current protected owner state.

  6. seonghobae commented on Sep 15, 2026

    @seonghobae
    ContributorAuthor

    What unbounded occupancy costs a consumer's scheduled workload, measured

    This issue argues that occupancy needs an admission bound. Here is the downstream cost, measured on ContextualWisdomLab/four-pillars, whose hourly-product-loop.yml is a read-only quality sentinel scheduled every hour.

    Across the 100 most recent schedule events:

    day runs created runs that started scheduled
    2026-09-14 6 5 24
    2026-09-13 6 6 24
    2026-09-12 7 7 24
    2026-09-11 6 6 24
    2026-09-10 6 6 24

    Six or seven of twenty-four, for five consecutive days. That is a stable regime, not a spike.

    The gap distribution is sharper still:

    gap between consecutive scheduled starts, hours
      min 1.70   median 4.11   max 8.26   (expected 1.00)
    

    The minimum is 1.70 hours, so not one run in the last hundred fired on its scheduled hour. The two most recent, created 2026-09-14T23:52Z and 2026-09-15T04:54Z, had still not started when I measured at 05:27Z, the older of them queued for five and a half hours.

    Why the scheduled workload is the one that silently loses

    A pull-request check that waits is visibly pending and someone eventually looks at it. A dropped schedule event leaves nothing behind: GitHub does not queue a missed cron occurrence, so the run simply never exists. The consumer repository has no signal that its hourly contract is being met roughly a quarter of the time, and its own documentation still describes the workflow as hourly.

    That makes scheduled work the quietest casualty of unbounded occupancy, and it is the part an admission bound would protect most cheaply: a sentinel run is short and read-only, so it is exactly the shape of work that should not be starved by long-running review jobs.

    Consumer-side measurement only. I have no visibility into the admission policy and am not proposing a specific bound.

  7. seonghobae commented on Sep 17, 2026

    @seonghobae
    ContributorAuthor

    LifeOS exact production evidence — 2026-09-18

    Fresh consumer evidence now reaches the platform ceiling this issue was designed to prevent.

    ContextualWisdomLab/life-os#249@484652e094e4f197bfa1a9ca21211c6bd6d32aac, Required Strix run 35173903681, job 105121637284:

    • runner admission and steps 1–15 completed successfully;
    • Provision contextual-orchestrator Strix sidecar started 2026-09-17T13:45:16Z and ended cancelled at 19:45:48Z, about six hours later;
    • Install Strix and Run Strix (quick) never ran, so this is not a Strix source/security verdict;
    • the uploaded strix-reports artifact 10516014728 (sha256:959cbd3253e95369712ed822d60b5af6cfdd0809278a00cf12cc80fa8a399eb5) contains an empty preflight JSON and sidecar stderr. The trace shows OpenRouter free routes failing quickly with 429, then NIM attempts beginning at 13:50:40 / 13:54:58 / 13:58:41 / 14:01:59. After the nvidia_nim_sub/deepseek-ai/deepseek-v4-flash-0731 attempt at 14:01:59 there is no later provider event before the job is cancelled at 19:45:48.

    This is stronger than the earlier hypothetical six-hour exposure: current production actually held a hosted runner until the platform/job ceiling while still inside sidecar provisioning. It also preserves the boundary in this issue: do not turn six-hour elapsed time into a provider/model failure and do not restore a total model deadline. The missing control is progress/idle-aware continuation and explicit runner-reclamation evidence.

    Current protected central sidecar source still waits for /healthz in an unbounded until loop as long as the sidecar PID remains alive. That loop cannot distinguish a live-but-progressing startup from a live-but-silent transport attempt; GitHub cancellation therefore becomes the first terminating authority in this exact incident. The artifact's last-event gap is useful RED evidence for the progress/idle contract, not enough evidence to invent a provider-specific timeout threshold.

    Canonical source/decision ownership remains here and in current-main successor #2207 (ADR-0032). LifeOS should remain fail-closed and should not copy a timeout, provider route, or central sidecar implementation.

  8. seonghobae commented on Sep 17, 2026

    @seonghobae
    ContributorAuthor

    Owner-path refinement from the LifeOS Strix cancellations: existing central PR #1629 is the canonical source lane for evidence-only review admission and for removing the redundant shell-authored second /v1/chat/completions replay after launcher preflight. I reconciled #1629 non-force onto current protected main@64aa08d7fa487deacd41c761c36277ca68cab6c9; new exact is 5e7e299ab160de38af5d4b60d6704e7d5f410bf1, with fresh Runtime Quality/security/CodeQL runs pending.

    Important correction from fresh #211 artifact evidence: LifeOS #211 run 35177779881 also spent ~6h in Provision contextual-orchestrator Strix sidecar and ended CANCELLED, but contextual-orchestrator-preflight.json is empty. Its sanitized stderr shows OpenRouter 429, NIM 504, then NIM_SUB launcher/provider attempts; healthz was never reached. The earlier #249 artifact likewise had empty preflight evidence. Therefore the observed 6h incidents are pre-healthz launcher-preflight no-progress, not the shell's later duplicate chat-completions replay. #1629 may still improve the launcher/admission path and validly removes redundant post-healthz inference, but it is not yet evidence-backed as the causal fix for these incidents.

    The direct incident contract stays here / ADR-0032: classify and reclaim progress/idle loss separately from model elapsed time, and do not feed runner-reclaim into provider/model failure, routing, ranking, or circuit-breaker evidence.

  9. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    LifeOS current-head occupancy specimen — 2026-09-24

    Fresh consumer evidence from ContextualWisdomLab/life-os#304 isolates the same progress/occupancy boundary without a leaf-source failure.

    • PR feat(ci): central CodeQL PR workflow for org code_scanning gate #304 is still open / Draft at exact head 392e890f072f4836f16160a09d2d05607ac13996 over main@193a87ef54c3fe6dcda4755bce4d6bc81e3a0297.
    • Required Noema run 35941341081 started at 2026-09-24T01:04:59Z and remains in_progress more than 11 hours later on that unchanged exact head.
    • Admission and changed-scope jobs completed successfully. In noema-review job 107506401923, stale-trigger rejection, reviewer credential selection, GitHub App token minting, live-head validation and target visibility all completed successfully.
    • The job is still inside Provision contextual-orchestrator review sidecar. No Noema verdict preparation/publication step has started, so this is not model-quality evidence and not a provider-failure verdict.
    • A fresh repository-level read shows this is the only life-os run currently in_progress, while the repository has 73 runs in queued state. That count is repository-local evidence only; it is not an organization-wide concurrency count.

    This specimen therefore belongs here / #2140 for progress-aware sidecar provisioning and explicit occupancy reclamation, and to #712 for queue-health classification. It should not be repaired in LifeOS by a no-op wake commit, blind rerun, provider/model fallback, elapsed model timeout, runner-selector churn, or required-gate weakening. #304's already-proven Planning Search semantic lineage remains separate from this normal required-review gate.

    Acceptance remains unchanged-head evidence that the central sidecar makes bounded forward progress or emits an explicit non-provider occupancy/reclamation outcome, followed by actual runner admission for the queued consumer evidence. No duplicate owner implementation is being created in LifeOS.

  10. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    2026-09-24 owner-path update from the LifeOS occupancy specimen: current-main successor #2207 has been ordinary/non-force restacked onto protected .github/main@e6334e229581a918e2f22de18733b76fa65d7e71. New exact is 31a14c821d0cdd6613c660ecd84c32b215b1a147; fresh compare is ahead-only with merge base exactly current protected main and still exactly the four intended ADR/doctoring/CHANGELOG paths. Fresh exact-head Security Scan 36003800625, SAST 36003800709, CodeQL 36003800741, and Python Security 36003800786 are queued, so this is current ancestry repair rather than accepted owner GREEN. LifeOS #304 run 35941341081 remains current-head in_progress at the Noema sidecar provision boundary; do not classify or cancel it from elapsed time alone. The LifeOS specimen therefore continues to support #2139/#2207's progress/admission distinction, while runtime/continuation repair remains owner work rather than a LifeOS-local timeout or provider fallback.

  11. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    2026-09-24 fresh LifeOS consumer evidence broadens the current occupancy specimen from Noema to Strix without changing the owner boundary.

    Current ContextualWisdomLab/life-os Actions inventory is 3 in-progress / 62 queued. All three in-progress runs are current-head work, so elapsed time alone is not a cancellation criterion:

    This is useful because the queue-pressure symptom now reproduces simultaneously through both central review consumers while unrelated exact-head verifier/CI runs remain queued. It strengthens the existing #2139/#2207 distinction: reclaim/admit on explicit progress/idle/continuation evidence, not on total model elapsed time, and do not feed runner-reclamation into provider/model failure, routing, ranking or circuit-breaker semantics.

    LifeOS should remain a consumer here: no local elapsed-time timeout, provider/model fallback, required-gate weakening, or duplicate sidecar implementation. Current-main #2207 remains the owner path; these LifeOS specimens are acceptance evidence for that lane.

  12. seonghobae commented on Sep 24, 2026

    @seonghobae
    ContributorAuthor

    LifeOS current-head specimen has now reached a terminal state and strengthens this owner finding.

    ContextualWisdomLab/life-os#304@392e890f072f4836f16160a09d2d05607ac13996, Required Noema run 35941341081, completed cancelled at 2026-09-24T14:17:41Z. Admission, changed-scope classification, stale-trigger rejection, credential selection, GitHub App token mint, exact live-head validation, and visibility resolution all succeeded. The noema-review job then remained in Provision contextual-orchestrator review sidecar from 08:16:50Z until 14:17:37Z (~6 h) and was cancelled there. Prepare Noema model verdict, bounded transport re-dispatch, evidence upload, and exact-head verdict publication were all skipped.

    This is not a model/provider failure verdict and should not be converted into one. The run payload proves terminal runner release without semantic review output; it does not by itself prove which administrative mechanism issued the cancellation, so keep that distinction explicit. The practical result is that the current exact LifeOS head still has no Required Noema verdict after occupying a runner for roughly six hours.

    Owner implication remains the same but is now terminally reproduced: implement progress/idle-aware sidecar lifecycle and explicit occupancy/continuation semantics in the central reusable workflow/CO boundary. Do not restore an elapsed inference timeout, add a LifeOS-local timeout/provider fallback, or treat this cancellation as a valid review result. Refs LifeOS #304 and the earlier current-head occupancy evidence in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions