Skip to content

fix: harden workflow lifecycle, concurrency, and retries with structured errors - #249

Merged
liuedcson merged 2 commits into
mainfrom
refactor/workflow-lifecycle-structured-errors
Oct 14, 2026
Merged

liuedcson merged 2 commits into
mainfrom
refactor/workflow-lifecycle-structured-errors

Conversation

@liuedcson

@liuedcson liuedcson commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Features

  • Structured provider error classification. AssistantMessage.errorKind (rate_limit | quota | auth | transient | invalid_request) is derived from the HTTP status, the provider's error type/code, and transport error codes. Mid-stream errors (Anthropic SSE error, OpenAI Responses error/response.failed) carry their code via ProviderStreamError, and ProviderRetryDelayError keeps the original error as cause. Retry decisions use errorKind first and fall back to message patterns only when no status or code is available.
  • Separate 429 retry budget for sessions. retry.rateLimitMaxRetries applies only to errorKind === "rate_limit", honors Retry-After, falls back to jittered backoff (1s–30s), and can be cancelled.
  • Shared admission window for subagents and workflows. Parallel subagent batches and workflow nodes take slots from one process-wide window, capped by a CPU-derived ceiling (optionally lowered by STEP_WORKFLOW_CONCURRENCY_LIMIT). Congestion control cuts the window by 25% per 429 epoch, recovers after four successes, honors Retry-After (capped at 30s), and keeps a throttled window for 60s between back-to-back runs.
  • Large subagent batches. A subagent call accepts up to 1000 tasks and queues the tasks beyond its concurrency window.
  • Content-addressed workflow resume. resumeFromRunId replays a completed call whose prompt and options are unchanged, regardless of its position. Failed, new, and changed calls run live; identical calls each consume one cached result.
    • Once a live call starts a write tool, a shell command, or an unrecognized tool (or cannot report its tool activity), only cached calls that used no file, shell, or network tools still replay.
    • Cached results are re-validated against the call's schema; calls with a runtime (safeParse) schema always run live.
    • Replayed calls are journaled under the resumed run's own sequence numbers.
  • Workflow result reporting. Runs report partial_failed/failed with the failed nodes, callback failures as taskFailures (including nested workflows), requested vs. effective concurrency, and node timing. parallel() also accepts promise arrays.

Fixes

  • A 400 response that mentions "429" is no longer treated as a rate limit, and a 429 whose code reports exhausted quota is terminal instead of retried.
  • A provider stream that throws, ends without a terminal event, or fails during construction now finalizes the partial assistant message instead of leaving the turn unfinished; a stream that closes early is classified as transient.
  • Tool results with isError: true keep their partial output and reach the model as errors; subagent results with failed or aborted children are marked as errors.
  • Workflow shutdown cancels and drains running and queued children before publishing a terminal state, seals that state, and aborts the run scope when the tool call ends.
  • Workflow node timeouts cover execution only (not queue time). Tokens reported by a cancelled or timed-out attempt are still charged, and a runner that ignores cancellation keeps its slot until it settles.
  • Runs that return with unawaited operations fail instead of finishing silently; node failures are counted once and not reported again as callback failures.
  • Journal, progress, and live-update write failures no longer replace a node's real error, and a failed progress write no longer blocks later writes.
  • Workflow and parallel subagent children hold their admission slot until the process exits, so a child that is still shutting down or cancelling no longer overlaps the next queued task.
  • Parallel subagent batches honor STEP_WORKFLOW_CONCURRENCY_LIMIT even when no workflow is running.
  • Retry-After is preserved when the retry-delay cap rejects a wait, and an empty or non-numeric retry-after-ms is treated as absent.

Testing

  • pnpm run check
  • providers, agent-core, apps/cli, coding-agent test suites (the 3 failures in completion-check-cli.test.ts also occur on main: they need a Git worktree with a HEAD commit).

@liuedcson
liuedcson force-pushed the refactor/workflow-lifecycle-structured-errors branch from 9275efa to e996c30 Compare October 14, 2026 09:15
…red errors

Supersedes #247 as a single change. It keeps that PR's lifecycle, concurrency,
and error-propagation fixes, and replaces the parts that inferred behavior
from text.

Structured provider errors (no text matching for known failures)
- Add AssistantMessage.errorKind: rate_limit | quota | auth | transient |
  invalid_request. Providers set it in their catch blocks from the HTTP
  status, the provider's machine-readable error type/code (insufficient_quota,
  GoUsageLimitError, rate_limit_exceeded, overloaded_error, ...), and transport
  error codes (APIConnectionError, ECONNRESET, ...). Mid-stream error events
  (Anthropic SSE `error`, OpenAI Responses `error`/`response.failed`) now throw
  ProviderStreamError carrying the code, and ProviderRetryDelayError keeps the
  original 429 as its cause, so both stay classifiable.
- The agent loop classifies errors thrown by a provider stream and marks a
  stream that closes without a terminal event as transient.
- isRetryableAssistantError trusts errorKind when present; the old message
  patterns remain only for failures that arrive with no status or code.
- Session retries pick the separate 429 budget, and workflow congestion
  control throttles the admission window, only for errorKind === "rate_limit".
  A 400 that mentions "429" is no longer a rate limit, and a 429 whose code
  reports exhausted quota is terminal. The regex-based isRateLimitError and
  isTerminalQuotaError are removed. Subagent children forward the kind in
  their messages; workflow runners report it as WorkflowAgentRunResult.errorKind.

Intent is the model's call, not a regex
- Do not add #247's counted fan-out detector (COUNTED_AGENT_LAUNCH, REFUSAL,
  META_QUESTION); ultraloop-opt-in.ts stays as on main. The detector missed
  common requests ("开10个子agent分别审查" hit the "别" in 分别, "…看看怎么回事"
  hit the "怎么" question filter, and verbs like 用/让 were not listed) while
  trying to rule out negations and questions with word lists. Only the
  explicit ultraloop/ultracode keyword, the fixed opt-in phrases, and
  /ultracode session mode are recognized by the harness; the model judges
  everything else from the existing tool descriptions. No intent guidance is
  added to the system prompt.

One admission window for subagents and workflows
- Parallel subagent batches take slots from the same process-wide admission
  window as workflow nodes, and feed its AIMD control with their model-request
  events. Running both at once can no longer exceed the host ceiling, and a
  429 seen by either lowers the window for both.
- Parallel subagent batches also honor STEP_WORKFLOW_CONCURRENCY_LIMIT, so the
  configured provider limit applies without a workflow running, and an
  admitted child holds its slot until its process exits (awaitExit), so a
  child still shutting down or cancelling cannot overlap queued work.

Content-addressed workflow resume
- resumeFromRunId replays any completed call whose prompt+options hash
  matches, independent of its sequence number. A failed, inserted, or
  reordered call (pipeline stages finish in nondeterministic order) now
  re-runs only itself instead of invalidating every later call. Identical
  calls are consumed in their original order, one cached result each.
  Nothing after a malformed journal line is trusted.
- A replayed call is journaled under the resumed run's own seq, with the
  schema-cleaned value the script received, so sequence numbers stay unique
  and chained resumes see what was consumed.
- A cached result must still satisfy the call's JSON schema, otherwise the
  call runs live. Calls with a runtime (safeParse) schema never replay: such
  schemas hash to one placeholder, so two different ones are
  indistinguishable.
- The cache closes on the first write, as ZCode's amend-resume does. The
  default runner forwards each child tool_execution_start (emitted before the
  tool's permission check and execution) and reports toolActivityObserved
  only when the child exited normally, since a killed child can lose events.
  classifyWorkflowToolEffect maps tools to none (clarify_user, find_tools),
  read (file reads/searches, search_web), or mutate (file writes, shell
  commands, and any unrecognized tool such as MCP). Completed entries record
  worldAccess. Once a live call starts a mutating tool, or a live call cannot
  vouch for its tool activity, only cached entries with worldAccess === false
  still replay; entries from older journals count as having touched the
  environment. Calls issued together before any live write, such as a resumed
  fan-out where a few nodes had failed, still replay in full.
- Resume wording in the workflow tool description, its guidance, and the
  system prompt's workflow section now describes this behavior.

Kept from #247
- Workflow runtime: cancel and drain running and queued children before
  publishing a terminal state, seal terminal states, and abort the run scope
  on exit. Clamp requested concurrency to a CPU-derived ceiling (optionally
  lowered by STEP_WORKFLOW_CONCURRENCY_LIMIT) and report clamping as a notice.
  Share a process-wide admission window with a 25% cut per congestion epoch,
  recovery after four successes, a Retry-After cooldown capped at 30s, and
  60s retention of a throttled window between back-to-back workflows.
- Record parallel()/pipeline() callback failures as taskFailures (including
  nested workflows), accept promise arrays in parallel(), keep node failures
  out of the callback channel and count them once, and fail runs that return
  with unawaited operations. Report partial_failed/failed with the failed
  nodes in the final tool result.
- Node timeouts cover execution only (not queue time) and come from our own
  timer, not a substring match. Tokens a cancelled or timed-out attempt
  reports while settling are still charged. A runner that ignores
  cancellation keeps its shared slot until it settles.
- Failure recording, progress.json writes, and live-update listeners are best
  effort and cannot replace a node's real error. A failed progress write no
  longer poisons the write queue.
- Subagents: accept batches above the concurrency window (up to 1000) and
  queue the excess, wait for workflow children to exit before releasing their
  slot, mark tool results with failed or aborted children as isError, and
  report rate-limit retries and request start/success over RPC.
- Session retries: an independent 429 budget (retry.rateLimitMaxRetries) that
  honors Retry-After, falls back to jittered backoff (1s to 30s), and can be
  cancelled.
- Agent loop: honor AgentToolResult.isError. Finalize a partial assistant
  message when a provider stream throws, ends early, or fails during
  construction, without reclassifying consumer/persistence errors or emitting
  a second terminal message.
- Providers: preserve Retry-After when the delay cap rejects a wait, and treat
  an empty or non-numeric retry-after-ms as absent.
@liuedcson
liuedcson force-pushed the refactor/workflow-lifecycle-structured-errors branch from e996c30 to 4323796 Compare October 14, 2026 09:30
@liuedcson
liuedcson merged commit adcd672 into main Oct 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant