Repository navigation
fix: harden workflow lifecycle, concurrency, and retries with structured errors - #249
Merged
Merged
Conversation
liuedcson
force-pushed
the
refactor/workflow-lifecycle-structured-errors
branch
from
October 14, 2026 09:15
9275efa to
e996c30
Compare
…red errors Supersedes #247 as a single change. It keeps that PR's lifecycle, concurrency, and error-propagation fixes, and replaces the parts that inferred behavior from text. Structured provider errors (no text matching for known failures) - Add AssistantMessage.errorKind: rate_limit | quota | auth | transient | invalid_request. Providers set it in their catch blocks from the HTTP status, the provider's machine-readable error type/code (insufficient_quota, GoUsageLimitError, rate_limit_exceeded, overloaded_error, ...), and transport error codes (APIConnectionError, ECONNRESET, ...). Mid-stream error events (Anthropic SSE `error`, OpenAI Responses `error`/`response.failed`) now throw ProviderStreamError carrying the code, and ProviderRetryDelayError keeps the original 429 as its cause, so both stay classifiable. - The agent loop classifies errors thrown by a provider stream and marks a stream that closes without a terminal event as transient. - isRetryableAssistantError trusts errorKind when present; the old message patterns remain only for failures that arrive with no status or code. - Session retries pick the separate 429 budget, and workflow congestion control throttles the admission window, only for errorKind === "rate_limit". A 400 that mentions "429" is no longer a rate limit, and a 429 whose code reports exhausted quota is terminal. The regex-based isRateLimitError and isTerminalQuotaError are removed. Subagent children forward the kind in their messages; workflow runners report it as WorkflowAgentRunResult.errorKind. Intent is the model's call, not a regex - Do not add #247's counted fan-out detector (COUNTED_AGENT_LAUNCH, REFUSAL, META_QUESTION); ultraloop-opt-in.ts stays as on main. The detector missed common requests ("开10个子agent分别审查" hit the "别" in 分别, "…看看怎么回事" hit the "怎么" question filter, and verbs like 用/让 were not listed) while trying to rule out negations and questions with word lists. Only the explicit ultraloop/ultracode keyword, the fixed opt-in phrases, and /ultracode session mode are recognized by the harness; the model judges everything else from the existing tool descriptions. No intent guidance is added to the system prompt. One admission window for subagents and workflows - Parallel subagent batches take slots from the same process-wide admission window as workflow nodes, and feed its AIMD control with their model-request events. Running both at once can no longer exceed the host ceiling, and a 429 seen by either lowers the window for both. - Parallel subagent batches also honor STEP_WORKFLOW_CONCURRENCY_LIMIT, so the configured provider limit applies without a workflow running, and an admitted child holds its slot until its process exits (awaitExit), so a child still shutting down or cancelling cannot overlap queued work. Content-addressed workflow resume - resumeFromRunId replays any completed call whose prompt+options hash matches, independent of its sequence number. A failed, inserted, or reordered call (pipeline stages finish in nondeterministic order) now re-runs only itself instead of invalidating every later call. Identical calls are consumed in their original order, one cached result each. Nothing after a malformed journal line is trusted. - A replayed call is journaled under the resumed run's own seq, with the schema-cleaned value the script received, so sequence numbers stay unique and chained resumes see what was consumed. - A cached result must still satisfy the call's JSON schema, otherwise the call runs live. Calls with a runtime (safeParse) schema never replay: such schemas hash to one placeholder, so two different ones are indistinguishable. - The cache closes on the first write, as ZCode's amend-resume does. The default runner forwards each child tool_execution_start (emitted before the tool's permission check and execution) and reports toolActivityObserved only when the child exited normally, since a killed child can lose events. classifyWorkflowToolEffect maps tools to none (clarify_user, find_tools), read (file reads/searches, search_web), or mutate (file writes, shell commands, and any unrecognized tool such as MCP). Completed entries record worldAccess. Once a live call starts a mutating tool, or a live call cannot vouch for its tool activity, only cached entries with worldAccess === false still replay; entries from older journals count as having touched the environment. Calls issued together before any live write, such as a resumed fan-out where a few nodes had failed, still replay in full. - Resume wording in the workflow tool description, its guidance, and the system prompt's workflow section now describes this behavior. Kept from #247 - Workflow runtime: cancel and drain running and queued children before publishing a terminal state, seal terminal states, and abort the run scope on exit. Clamp requested concurrency to a CPU-derived ceiling (optionally lowered by STEP_WORKFLOW_CONCURRENCY_LIMIT) and report clamping as a notice. Share a process-wide admission window with a 25% cut per congestion epoch, recovery after four successes, a Retry-After cooldown capped at 30s, and 60s retention of a throttled window between back-to-back workflows. - Record parallel()/pipeline() callback failures as taskFailures (including nested workflows), accept promise arrays in parallel(), keep node failures out of the callback channel and count them once, and fail runs that return with unawaited operations. Report partial_failed/failed with the failed nodes in the final tool result. - Node timeouts cover execution only (not queue time) and come from our own timer, not a substring match. Tokens a cancelled or timed-out attempt reports while settling are still charged. A runner that ignores cancellation keeps its shared slot until it settles. - Failure recording, progress.json writes, and live-update listeners are best effort and cannot replace a node's real error. A failed progress write no longer poisons the write queue. - Subagents: accept batches above the concurrency window (up to 1000) and queue the excess, wait for workflow children to exit before releasing their slot, mark tool results with failed or aborted children as isError, and report rate-limit retries and request start/success over RPC. - Session retries: an independent 429 budget (retry.rateLimitMaxRetries) that honors Retry-After, falls back to jittered backoff (1s to 30s), and can be cancelled. - Agent loop: honor AgentToolResult.isError. Finalize a partial assistant message when a provider stream throws, ends early, or fails during construction, without reclassifying consumer/persistence errors or emitting a second terminal message. - Providers: preserve Retry-After when the delay cap rejects a wait, and treat an empty or non-numeric retry-after-ms as absent.
liuedcson
force-pushed
the
refactor/workflow-lifecycle-structured-errors
branch
from
October 14, 2026 09:30
e996c30 to
4323796
Compare
Hylouis233
added a commit
to Hylouis233/Step-Code
that referenced
this pull request
Oct 15, 2026
* feat(compaction): use model-managed CLM by default Keep canonical history while validating bounded edits to a persisted working view. Reuse compatible actor prefixes for maintenance and retain native compaction for failures, overflow recovery, and explicit requests. Honor explicit projection settings and disabled auto-compaction, defer unused maintenance after final answers, and cover default and native behavior in the session, CLI, persistence, and usage regressions. * fix(cli): retain explicit context projection after resource loading Apply the CLI mode after resource discovery reloads settings, so an explicit off selection takes precedence over the default CLM mode. Exercise the real CLI startup path with a local streaming endpoint and keep the config unchanged. * feat: add Full Access and simplify permission tiers Introduce Ask, Approve for Me, and Full Access with session-only or remembered risk acknowledgment. Reset legacy permission settings to Ask and enable bounded model-error continuation in both automatic tiers. Preserve explicit tool overrides and scope consent to the active session. Route /exit through the existing /quit shutdown path, remove duplicate extension registration, and cover permission migration, consent lifecycle, and exit behavior. * fix(shell): discover native Git Bash on Windows instead of WSL On Windows, a Git for Windows install in a non-default directory only puts <Git>\cmd on PATH, so shell discovery fell through to `where bash.exe` and picked C:\Windows\System32\bash.exe, silently running commands inside WSL. - Look for Git Bash in Program Files, Program Files (x86), %LOCALAPPDATA%\Programs\Git, then relative to each git.exe on PATH, then native bash.exe on PATH. Resolve symlinks and drop the where.exe dependency. - Never select the WSL launcher automatically; an explicit shellPath to it keeps its stdin command transport. - Add the Git installation's bin and usr/bin to the Git Bash child PATH only. - On Windows, pipe background command output into the log file instead of handing the file descriptor to MSYS, which exited 1 with an empty log. - Generate candidates lazily so a standard install returns without scanning PATH on every command or permission check. Fixes stepfun-ai#136 Co-authored-by: ZXJC-niusile <2632590648@qq.com> * fix(shell): use Git Bash child PATH for config value commands Config values written as `!command` were run with a direct spawnSync on the resolved shell, bypassing spawnShellChild, so on Windows they did not get the Git installation's bin and usr/bin on PATH. Export withGitBashPath and apply it there too so these commands see the same tools as run_command. * fix(skills): emit readable single-line diagnostics for SKILL.md frontmatter errors * fix(skills): make frontmatter diagnostic hints syntax-aware Inspect recovered YAML structure to avoid quoting hints for comments and block scalars. Escape parser control characters and tolerate absent error values. Cover both skill loaders and CLI diagnostic rendering. * fix: stop repeating Ultracode capability reminders (stepfun-ai#241) * fix: announce Ultracode mode only when context state changes Keep capability guidance in the workflow tool description instead of appending a tutorial to every user turn. Announce session-mode transitions after real user input and reuse the last runtime-authored state in the active context. Preserve per-turn opt-in and reconcile missing or changed mode state before provider requests. Cover provider-visible message order, unchanged turns, mode toggles, compaction, restricted profiles, and streaming revocation. * fix: keep workflow opt-in guidance accurate across tool profiles Explain unavailable workflow tools for per-turn requests and make standing consent conditional on the current tool profile. Cover restricted per-turn requests and changing tool availability without repeated state notices. * fix: avoid persisting an unchanged retry default Skip the native retry setter when automatic permission mode finds retry already enabled. This prevents startup from adding a redundant retry section to config.toml and preserves runtime-only CLI overrides. --------- Co-authored-by: liuedcson <332840681+liuedcson@users.noreply.github.com> * fix(skills): quiet startup skill conflicts for duplicated shared skill dirs Users who run both Codex and Claude Code often have the same skills copied into ~/.agents/skills and ~/.claude/skills. Each copy produced a collision entry, flooding startup even though the shadowed copy is never loaded. - Skip a colliding skill silently when its SKILL.md is byte-identical to the winner, the same way symlinked duplicates are already skipped. - Without the full resource listing, collapse remaining skill/prompt collisions into one line per (skipped dir, winning dir) pair; --verbose still lists each one. Warnings and errors are shown in full as before. * feat: persist Ultracode mode per session (stepfun-ai#245) * feat: persist Ultracode mode per session Record the latest Ultracode on/off choice in the current session journal and restore it when that session is reopened. Keep new sessions opted out and avoid global or project settings. * fix: restore Ultracode status after session reopen --------- Co-authored-by: liuedcson <332840681+liuedcson@users.noreply.github.com> * docs: add guidelines for platform and provider packages Document configuration migrations, host contracts, provider adapters, telemetry, and terminal rendering. Define dependency boundaries and module-specific validation commands. * docs: define agent runtime and coding product boundaries Describe Agent Loop, session, context projection, and compaction ownership. Keep coding-agent guidance at package scope and document tools, extensions, orchestration, and Step integrations against the current implementation. * docs: document CLI hosting and UI layer responsibilities Define startup, mode routing, process lifecycle, and protocol output boundaries. Document host-to-runtime-to-view dependencies, input handling, redraws, and focused validation. * docs: establish repository and support directory guidelines Connect module guidance through the repository architecture and ownership map. Add documentation, script, release, and plugin conventions while preserving the existing commit attribution policy and CLAUDE.md reference. * fix: harden workflow lifecycle, concurrency, and retries with structured errors (stepfun-ai#249) * fix: harden workflow lifecycle, concurrency, and retries with structured errors Supersedes stepfun-ai#247 as a single change. It keeps that PR's lifecycle, concurrency, and error-propagation fixes, and replaces the parts that inferred behavior from text. Structured provider errors (no text matching for known failures) - Add AssistantMessage.errorKind: rate_limit | quota | auth | transient | invalid_request. Providers set it in their catch blocks from the HTTP status, the provider's machine-readable error type/code (insufficient_quota, GoUsageLimitError, rate_limit_exceeded, overloaded_error, ...), and transport error codes (APIConnectionError, ECONNRESET, ...). Mid-stream error events (Anthropic SSE `error`, OpenAI Responses `error`/`response.failed`) now throw ProviderStreamError carrying the code, and ProviderRetryDelayError keeps the original 429 as its cause, so both stay classifiable. - The agent loop classifies errors thrown by a provider stream and marks a stream that closes without a terminal event as transient. - isRetryableAssistantError trusts errorKind when present; the old message patterns remain only for failures that arrive with no status or code. - Session retries pick the separate 429 budget, and workflow congestion control throttles the admission window, only for errorKind === "rate_limit". A 400 that mentions "429" is no longer a rate limit, and a 429 whose code reports exhausted quota is terminal. The regex-based isRateLimitError and isTerminalQuotaError are removed. Subagent children forward the kind in their messages; workflow runners report it as WorkflowAgentRunResult.errorKind. Intent is the model's call, not a regex - Do not add stepfun-ai#247's counted fan-out detector (COUNTED_AGENT_LAUNCH, REFUSAL, META_QUESTION); ultraloop-opt-in.ts stays as on main. The detector missed common requests ("开10个子agent分别审查" hit the "别" in 分别, "…看看怎么回事" hit the "怎么" question filter, and verbs like 用/让 were not listed) while trying to rule out negations and questions with word lists. Only the explicit ultraloop/ultracode keyword, the fixed opt-in phrases, and /ultracode session mode are recognized by the harness; the model judges everything else from the existing tool descriptions. No intent guidance is added to the system prompt. One admission window for subagents and workflows - Parallel subagent batches take slots from the same process-wide admission window as workflow nodes, and feed its AIMD control with their model-request events. Running both at once can no longer exceed the host ceiling, and a 429 seen by either lowers the window for both. - Parallel subagent batches also honor STEP_WORKFLOW_CONCURRENCY_LIMIT, so the configured provider limit applies without a workflow running, and an admitted child holds its slot until its process exits (awaitExit), so a child still shutting down or cancelling cannot overlap queued work. Content-addressed workflow resume - resumeFromRunId replays any completed call whose prompt+options hash matches, independent of its sequence number. A failed, inserted, or reordered call (pipeline stages finish in nondeterministic order) now re-runs only itself instead of invalidating every later call. Identical calls are consumed in their original order, one cached result each. Nothing after a malformed journal line is trusted. - A replayed call is journaled under the resumed run's own seq, with the schema-cleaned value the script received, so sequence numbers stay unique and chained resumes see what was consumed. - A cached result must still satisfy the call's JSON schema, otherwise the call runs live. Calls with a runtime (safeParse) schema never replay: such schemas hash to one placeholder, so two different ones are indistinguishable. - The cache closes on the first write, as ZCode's amend-resume does. The default runner forwards each child tool_execution_start (emitted before the tool's permission check and execution) and reports toolActivityObserved only when the child exited normally, since a killed child can lose events. classifyWorkflowToolEffect maps tools to none (clarify_user, find_tools), read (file reads/searches, search_web), or mutate (file writes, shell commands, and any unrecognized tool such as MCP). Completed entries record worldAccess. Once a live call starts a mutating tool, or a live call cannot vouch for its tool activity, only cached entries with worldAccess === false still replay; entries from older journals count as having touched the environment. Calls issued together before any live write, such as a resumed fan-out where a few nodes had failed, still replay in full. - Resume wording in the workflow tool description, its guidance, and the system prompt's workflow section now describes this behavior. Kept from stepfun-ai#247 - Workflow runtime: cancel and drain running and queued children before publishing a terminal state, seal terminal states, and abort the run scope on exit. Clamp requested concurrency to a CPU-derived ceiling (optionally lowered by STEP_WORKFLOW_CONCURRENCY_LIMIT) and report clamping as a notice. Share a process-wide admission window with a 25% cut per congestion epoch, recovery after four successes, a Retry-After cooldown capped at 30s, and 60s retention of a throttled window between back-to-back workflows. - Record parallel()/pipeline() callback failures as taskFailures (including nested workflows), accept promise arrays in parallel(), keep node failures out of the callback channel and count them once, and fail runs that return with unawaited operations. Report partial_failed/failed with the failed nodes in the final tool result. - Node timeouts cover execution only (not queue time) and come from our own timer, not a substring match. Tokens a cancelled or timed-out attempt reports while settling are still charged. A runner that ignores cancellation keeps its shared slot until it settles. - Failure recording, progress.json writes, and live-update listeners are best effort and cannot replace a node's real error. A failed progress write no longer poisons the write queue. - Subagents: accept batches above the concurrency window (up to 1000) and queue the excess, wait for workflow children to exit before releasing their slot, mark tool results with failed or aborted children as isError, and report rate-limit retries and request start/success over RPC. - Session retries: an independent 429 budget (retry.rateLimitMaxRetries) that honors Retry-After, falls back to jittered backoff (1s to 30s), and can be cancelled. - Agent loop: honor AgentToolResult.isError. Finalize a partial assistant message when a provider stream throws, ends early, or fails during construction, without reclassifying consumer/persistence errors or emitting a second terminal message. - Providers: preserve Retry-After when the delay cap rejects a wait, and treat an empty or non-numeric retry-after-ms as absent. * test: make subagent concurrency checks independent of host CPU count --------- Co-authored-by: liuedcson <332840681+liuedcson@users.noreply.github.com> * fix: route then_run through run_command (stepfun-ai#248) * docs: define development and testing practices * docs: map repository structure and engineering rules * docs: standardize English commits and pull request reviews * fix: provision StepPage on Windows via the published bundle provisionBuiltinPlugin() skipped win32 entirely and ensureStepPageReady() refused to install without a healthy executable, so the built-in steppage plugin could never set up steppage-mcp on Windows and every launch reported "automatic installation is currently supported on macOS and Linux" (stepfun-ai#219). Windows cannot run the POSIX installer script, so install the published bundle directly: download steppage-mcp.mjs, write a steppage-mcp.cmd wrapper next to it under ~/.local/bin, and verify the result. Executable probes run through `cmd /d /c` on Windows so both bare PATHEXT lookups and explicit .cmd files resolve on every Node runtime. The lock, Node.js >= 20 check, credential handling and failure reporting stay shared with the POSIX path; only the download-and-execute step is platform-specific. The Windows failure hint points at `step mcp prepare` instead of the POSIX-only `curl | sh` pipeline. * ci: run existing checks on the StepPage fix branch * ci: add workflow_dispatch so CI runs can be dispatched manually Restore the canonical push trigger (the branch was only added to work around the missing manual entry) and add workflow_dispatch so the CI workflow can be triggered from the Actions tab without pushing a no-op commit. * ci: retrigger checks for this PR CI had no workflow run attached to this PR. Actions is confirmed working, so this empty commit fires the pull_request synchronize event against main. * ci: retrigger PR checks Actions only started accepting event triggers after the first manual dispatch; this forces a pull_request synchronize event now that it is live. --------- Co-authored-by: youwang <61931019+MelodyVAR@users.noreply.github.com> Co-authored-by: liuedcson <332840681+liuedcson@users.noreply.github.com> Co-authored-by: edcsonzzz <liuedcson@gmail.com> Co-authored-by: xuyunfang <xuyunfang@stepfun.com> Co-authored-by: ZXJC-niusile <2632590648@qq.com> Co-authored-by: Hylouis233 <Hylouis233@users.noreply.github.com> Co-authored-by: Uking-xxx <xuklaus1@gmail.com> Co-authored-by: ZouR-Ma <2605315944@qq.com> Co-authored-by: ZouR Ma <37286405+ZouR-Ma@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
AssistantMessage.errorKind(rate_limit|quota|auth|transient|invalid_request) is derived from the HTTP status, the provider's error type/code, and transport error codes. Mid-stream errors (Anthropic SSEerror, OpenAI Responseserror/response.failed) carry their code viaProviderStreamError, andProviderRetryDelayErrorkeeps the original error ascause. Retry decisions useerrorKindfirst and fall back to message patterns only when no status or code is available.retry.rateLimitMaxRetriesapplies only toerrorKind === "rate_limit", honorsRetry-After, falls back to jittered backoff (1s–30s), and can be cancelled.STEP_WORKFLOW_CONCURRENCY_LIMIT). Congestion control cuts the window by 25% per 429 epoch, recovers after four successes, honorsRetry-After(capped at 30s), and keeps a throttled window for 60s between back-to-back runs.resumeFromRunIdreplays a completed call whose prompt and options are unchanged, regardless of its position. Failed, new, and changed calls run live; identical calls each consume one cached result.safeParse) schema always run live.partial_failed/failedwith the failed nodes, callback failures astaskFailures(including nested workflows), requested vs. effective concurrency, and node timing.parallel()also accepts promise arrays.Fixes
isError: truekeep their partial output and reach the model as errors; subagent results with failed or aborted children are marked as errors.STEP_WORKFLOW_CONCURRENCY_LIMITeven when no workflow is running.Retry-Afteris preserved when the retry-delay cap rejects a wait, and an empty or non-numericretry-after-msis treated as absent.Testing
pnpm run checkcompletion-check-cli.test.tsalso occur onmain: they need a Git worktree with a HEAD commit).