Repository navigation
fix: harden workflow lifecycle, concurrency, and retries with structured errors - #249
Merged
Merged
Conversation
liuedcson
force-pushed
the
refactor/workflow-lifecycle-structured-errors
branch
from
October 14, 2026 09:15
9275efa to
e996c30
Compare
…red errors Supersedes #247 as a single change. It keeps that PR's lifecycle, concurrency, and error-propagation fixes, and replaces the parts that inferred behavior from text. Structured provider errors (no text matching for known failures) - Add AssistantMessage.errorKind: rate_limit | quota | auth | transient | invalid_request. Providers set it in their catch blocks from the HTTP status, the provider's machine-readable error type/code (insufficient_quota, GoUsageLimitError, rate_limit_exceeded, overloaded_error, ...), and transport error codes (APIConnectionError, ECONNRESET, ...). Mid-stream error events (Anthropic SSE `error`, OpenAI Responses `error`/`response.failed`) now throw ProviderStreamError carrying the code, and ProviderRetryDelayError keeps the original 429 as its cause, so both stay classifiable. - The agent loop classifies errors thrown by a provider stream and marks a stream that closes without a terminal event as transient. - isRetryableAssistantError trusts errorKind when present; the old message patterns remain only for failures that arrive with no status or code. - Session retries pick the separate 429 budget, and workflow congestion control throttles the admission window, only for errorKind === "rate_limit". A 400 that mentions "429" is no longer a rate limit, and a 429 whose code reports exhausted quota is terminal. The regex-based isRateLimitError and isTerminalQuotaError are removed. Subagent children forward the kind in their messages; workflow runners report it as WorkflowAgentRunResult.errorKind. Intent is the model's call, not a regex - Do not add #247's counted fan-out detector (COUNTED_AGENT_LAUNCH, REFUSAL, META_QUESTION); ultraloop-opt-in.ts stays as on main. The detector missed common requests ("开10个子agent分别审查" hit the "别" in 分别, "…看看怎么回事" hit the "怎么" question filter, and verbs like 用/让 were not listed) while trying to rule out negations and questions with word lists. Only the explicit ultraloop/ultracode keyword, the fixed opt-in phrases, and /ultracode session mode are recognized by the harness; the model judges everything else from the existing tool descriptions. No intent guidance is added to the system prompt. One admission window for subagents and workflows - Parallel subagent batches take slots from the same process-wide admission window as workflow nodes, and feed its AIMD control with their model-request events. Running both at once can no longer exceed the host ceiling, and a 429 seen by either lowers the window for both. - Parallel subagent batches also honor STEP_WORKFLOW_CONCURRENCY_LIMIT, so the configured provider limit applies without a workflow running, and an admitted child holds its slot until its process exits (awaitExit), so a child still shutting down or cancelling cannot overlap queued work. Content-addressed workflow resume - resumeFromRunId replays any completed call whose prompt+options hash matches, independent of its sequence number. A failed, inserted, or reordered call (pipeline stages finish in nondeterministic order) now re-runs only itself instead of invalidating every later call. Identical calls are consumed in their original order, one cached result each. Nothing after a malformed journal line is trusted. - A replayed call is journaled under the resumed run's own seq, with the schema-cleaned value the script received, so sequence numbers stay unique and chained resumes see what was consumed. - A cached result must still satisfy the call's JSON schema, otherwise the call runs live. Calls with a runtime (safeParse) schema never replay: such schemas hash to one placeholder, so two different ones are indistinguishable. - The cache closes on the first write, as ZCode's amend-resume does. The default runner forwards each child tool_execution_start (emitted before the tool's permission check and execution) and reports toolActivityObserved only when the child exited normally, since a killed child can lose events. classifyWorkflowToolEffect maps tools to none (clarify_user, find_tools), read (file reads/searches, search_web), or mutate (file writes, shell commands, and any unrecognized tool such as MCP). Completed entries record worldAccess. Once a live call starts a mutating tool, or a live call cannot vouch for its tool activity, only cached entries with worldAccess === false still replay; entries from older journals count as having touched the environment. Calls issued together before any live write, such as a resumed fan-out where a few nodes had failed, still replay in full. - Resume wording in the workflow tool description, its guidance, and the system prompt's workflow section now describes this behavior. Kept from #247 - Workflow runtime: cancel and drain running and queued children before publishing a terminal state, seal terminal states, and abort the run scope on exit. Clamp requested concurrency to a CPU-derived ceiling (optionally lowered by STEP_WORKFLOW_CONCURRENCY_LIMIT) and report clamping as a notice. Share a process-wide admission window with a 25% cut per congestion epoch, recovery after four successes, a Retry-After cooldown capped at 30s, and 60s retention of a throttled window between back-to-back workflows. - Record parallel()/pipeline() callback failures as taskFailures (including nested workflows), accept promise arrays in parallel(), keep node failures out of the callback channel and count them once, and fail runs that return with unawaited operations. Report partial_failed/failed with the failed nodes in the final tool result. - Node timeouts cover execution only (not queue time) and come from our own timer, not a substring match. Tokens a cancelled or timed-out attempt reports while settling are still charged. A runner that ignores cancellation keeps its shared slot until it settles. - Failure recording, progress.json writes, and live-update listeners are best effort and cannot replace a node's real error. A failed progress write no longer poisons the write queue. - Subagents: accept batches above the concurrency window (up to 1000) and queue the excess, wait for workflow children to exit before releasing their slot, mark tool results with failed or aborted children as isError, and report rate-limit retries and request start/success over RPC. - Session retries: an independent 429 budget (retry.rateLimitMaxRetries) that honors Retry-After, falls back to jittered backoff (1s to 30s), and can be cancelled. - Agent loop: honor AgentToolResult.isError. Finalize a partial assistant message when a provider stream throws, ends early, or fails during construction, without reclassifying consumer/persistence errors or emitting a second terminal message. - Providers: preserve Retry-After when the delay cap rejects a wait, and treat an empty or non-numeric retry-after-ms as absent.
liuedcson
force-pushed
the
refactor/workflow-lifecycle-structured-errors
branch
from
October 14, 2026 09:30
e996c30 to
4323796
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
AssistantMessage.errorKind(rate_limit|quota|auth|transient|invalid_request) is derived from the HTTP status, the provider's error type/code, and transport error codes. Mid-stream errors (Anthropic SSEerror, OpenAI Responseserror/response.failed) carry their code viaProviderStreamError, andProviderRetryDelayErrorkeeps the original error ascause. Retry decisions useerrorKindfirst and fall back to message patterns only when no status or code is available.retry.rateLimitMaxRetriesapplies only toerrorKind === "rate_limit", honorsRetry-After, falls back to jittered backoff (1s–30s), and can be cancelled.STEP_WORKFLOW_CONCURRENCY_LIMIT). Congestion control cuts the window by 25% per 429 epoch, recovers after four successes, honorsRetry-After(capped at 30s), and keeps a throttled window for 60s between back-to-back runs.resumeFromRunIdreplays a completed call whose prompt and options are unchanged, regardless of its position. Failed, new, and changed calls run live; identical calls each consume one cached result.safeParse) schema always run live.partial_failed/failedwith the failed nodes, callback failures astaskFailures(including nested workflows), requested vs. effective concurrency, and node timing.parallel()also accepts promise arrays.Fixes
isError: truekeep their partial output and reach the model as errors; subagent results with failed or aborted children are marked as errors.STEP_WORKFLOW_CONCURRENCY_LIMITeven when no workflow is running.Retry-Afteris preserved when the retry-delay cap rejects a wait, and an empty or non-numericretry-after-msis treated as absent.Testing
pnpm run checkcompletion-check-cli.test.tsalso occur onmain: they need a Git worktree with a HEAD commit).