[Quinn-With-Two-Ns]

What changed?

• Track replaying mutable-side-effect invocations separately from the command queue and resolve skipped invocations after their workflow task's command events have been applied.
• Reuse the existing cancellation/cached-result transition, preserving callback workflow time without sweeping unrelated commands.
• Introduce MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY, disabled by default for its first release. Register opt-in usage at the mutableSideEffect call site and honor recorded flags during replay.
• Add cache-eviction, callback-time, flag-registration, and negative nondeterminism-detection tests. Retain only the commandless-task and callback-time history fixtures.

Skip-count calculation and marker matching are unchanged.

Why?

Refs #3163.

A skipped mutable side effect has no corresponding marker event. During full replay, its internal placeholder can remain pending when there is no later command event to trigger cleanup, and an empty RecordMarker command can reach the server.

Unconditionally notifying the pending invocation at the next task start changes the workflow time observed by its continuation and can break replay. The added timing regression covers that case; flagged cleanup instead runs at the completed task's boundary, before the next task advances time or delivers signals.

Breaking changes?

None to public APIs. The new SDK flag remains disabled by default (Enabled: pending) for rollout compatibility; tests opt in explicitly. Unflagged histories retain their existing replay behavior.

Server PR

None required.

Validation

• ./gradlew --offline spotlessApply
• Retained state-machine and mutable-side-effect tests: 14 passed.

  ./gradlew --offline :temporal-sdk:test \
    --tests io.temporal.internal.statemachines.MutableSideEffectStateMachineTest \
    --tests io.temporal.workflow.MutableSideEffectReplayTest \
    --tests io.temporal.workflow.MutableSideEffectTest

• Real-server configuration: 9 passed.

  ./gradlew --offline -PtestServer=dev-server :temporal-sdk:test \
    --tests io.temporal.workflow.MutableSideEffectReplayTest \
    --tests io.temporal.workflow.MutableSideEffectTest

• One-off compatibility validation generated 20 histories with the unmodified baseline SDK (adf0fc8) on the test and real servers. Baseline replay passed first, followed by 40 patched replay cases comparing exact callback values, order, and workflow times with opt-in off/on. The bulk corpus and generator are intentionally not included in this PR.
• Broader focused state-machine, side-effect, versioning, query, sticky-worker, and SDK-flag tests passed during development. The full repository suite has not been run.

---

Note

High Risk
Changes workflow replay and command scheduling for mutable side effects, where subtle timing mistakes cause nondeterministic replay; mitigated by a default-off SDK flag and extensive replay tests.

Overview
Fixes skipped Workflow.mutableSideEffect invocations that could leave replay placeholders pending (and emit invalid empty RecordMarker commands) after sticky cache eviction or worker restart, especially when a workflow task completes with no matching marker events.

Introduces SDK flag MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY (disabled by default for rollout). When enabled, replaying invocations are tracked separately; after all command events for a task are applied, pending skipped calls are resolved via the existing cancel/cached-result path so callbacks run at that task’s workflow time—not at the next task start, which would change Workflow.currentTimeMillis() and break replay.

Flag usage is registered at mutableSideEffect call time when opted in via initialFlags; histories without the flag keep prior timing. workflowTaskStarted also resolves pending placeholders on live tasks when the flag is active. Command iteration during task start is copied to avoid concurrent modification when recovery runs workflow code.

Adds state-machine, integration replay, callback-timing, and nondeterminism regression tests plus captured history fixtures.

^{Reviewed by Cursor Bugbot for commit 9e5c687. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9e5c687. Configure here.}

[cursor]

Later flag replay shifts skip timing

High Severity

When MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY is first recorded on a later workflow task, replay leaves earlier skipped invocations pending, then handleWorkflowTaskStarted resumes them at that later task's workflow time. The live task that wrote the flag did not re-run those continuations, so replay can emit a different command set and fail.

Additional Locations (2)

• temporal-sdk/src/main/java/io/temporal/internal/statemachines/WorkflowStateMachines.java#L387-L392
• temporal-sdk/src/main/java/io/temporal/internal/statemachines/WorkflowStateMachines.java#L1253-L1256

 

^{Reviewed by Cursor Bugbot for commit 9e5c687. Configure here.}