Repository navigation
Keep authored Nexus completion waits suspended until durable terminal delivery #116
Description
Reproduced failure
An authored Python workflow awaiting a Nexus service completes with a started operation and a null response instead of waiting for the service result.
Actual published run 38130074555 executes Python callers in both shared and tenant-b against the registered PHP service in shared. Routing, original caller identity, Avro arguments and actual PHP service input all pass. Both linked PHP service runs complete with the expected greeting. The Python callers have already completed with started and null response. The required terminal-workflow gate fails.
Frozen tuple: Server 2.5.19 at sha256:1b8621cb43340f979f4dc144aa51c32d0747aaad73f2b600130e0ee973aea352, Workflow 2.5.9, PHP SDK 2.2.7, Python SDK 2.6.0rc1, CLI 2.2.3 and Waterline 2.9.1. The official artifact retains caller results and service histories. Server #394 reports the separate input and terminal assertions.
Cause and contract
Worker._resolve_workflow_nexus_commands() currently performs a client HTTP execution, records its immediate response as RecordSideEffect, synthesizes SideEffectRecorded, and immediately resumes replay. NexusServiceCall.to_server_command() refuses serialization. An HTTP wait returning a pending operation is treated as the yielded workflow result.
The existing worker protocol already defines the leased start_service_operation command and ServiceCallStarted, ServiceCallCompleted, ServiceCallFailed and ServiceCallCancelled history. New authored calls should use that durable task boundary. A synchronous wait_for=completed call stays suspended until its terminal history arrives. Async or acceptance-only calls can resume at admission.
Backend terminal closure and caller delivery are tracked in Workflow #753. This issue owns the Python worker, serialization and replay changes needed to consume them.
Acceptance
- Emit the leased service-operation command with original caller/run identity, target namespace, Avro arguments, idempotency and supported operation options. Preserve authenticated namespace and role enforcement.
- Suspend completion waits on a nonterminal admission. Decode the committed terminal result, including runtime-managed external payloads, and deliver typed failure/cancellation at the original yield boundary.
- Keep async and acceptance-only behavior explicit. An HTTP delivery timeout must not complete a waiting workflow or replace the original operation identity.
- Preserve deterministic replay of previously recorded Nexus
SideEffectRecordedhistories. New histories use service-operation events. Reject changed recorded identity/target clearly. - Add the smallest actual replay fixture that fails on the current implementation and passes with the fix, plus focused worker command and pending-to-terminal tests.
- Re-run the actual published Python caller to PHP service fixture with the qualified backend and SDK artifacts. Include cold worker recovery at a committed delivery boundary, duplicate observation, terminal failure/cancellation, authorization refusal and oversized payloads before claiming those outcomes.
- Publish the qualified SDK fix and complete affected consumer follow-through. Source tests and successful admission alone do not close this issue.
Activity
rmcdaniel commentedon Oct 11, 2026
Both PHP source slices are merged and their main suites are green.
d41b6254952616781aa33faf13270ac8ef806f74, closes the original linked call from committed terminal history. Full source and main suites pass, including every database shard, five Laravel upgrades, 3,804 unit tests and 93.49% V2 line coverage.2f7c1765b625b447798396be0ccd88ef9274f655, creates a durable caller task and records the result at its claim boundary. Full source and main 38136128914 pass, including 3,812 unit tests and 93.49% coverage. The official embedded fixture executes the waiting caller through its actual resumed PHP result and fails on the base implementation.Neither Nexus source slice is released yet. The independent Python completion correction is SDK #116 / PR #117, merged
993419e06aadb42abb8f107963562ecff1acbb97after finalc740fde6c101c89172c49466c9a223c934ac7960passed all checks. It submits leased service commands, keeps completion waits suspended and replays terminal delivery while preserving older side-effect histories. The new ordinary fixture passes current SDK and fails the prerelease source. All 352 affected tests pass, including committed failure/cancellation causes when nullable snapshot fields are empty. Full Python 3.10/3.11/3.12, lint, package, docs, corpus and actual Server integration pass on the final head and main 38138944208.The earlier published fixture verifies routing and real Avro service input with Python 2.6.0rc1, while its terminal workflow assertion correctly fails with Started/null. That outcome remains unchanged until replacement artifacts are qualified.
Next: PHP durable authoring and the reopened HTTP argument transport fix, then qualify published terminal results, failure, cancellation, cold recovery and oversized payloads. The PHP bridge also currently hardcodes
caller_sdk_language=workflow-phpin control-plane metadata and service history. Leased Python calls must preserve their actual SDK attribution while the Server continues to supply authenticated caller/run identity. This belongs in the same cross-SDK qualification. The issue stays open through those customer outcomes.rmcdaniel commentedon Oct 11, 2026
Python SDK 2.6.0-rc.2 is published on PyPI as 2.6.0rc2, exact source
96b8cd76600bc1df91fa7c186fa0c472c09ca285. It includes the durable caller fix from #117. Calls submit leased commands and replay their admission and terminal history, including recorded failure and cancellation causes.All source and main checks pass. Publication 38146690306 verified the exact build, protected artifact handoff, PyPI metadata and supported installations. Stable 2.5.4 remains unchanged.
Next: execute PHP/Python callers against both PHP/Python workflow services with the new published Workflow/Server tuple, then qualify cold recovery, backend interruption, failure, cancellation and external payloads. This issue and Workflow #753 stay open through those results.
rmcdaniel commentedon Oct 12, 2026
The first published baseline in Server #425, run 38217295777, proves the waiting-caller cancellation gap in both cross-language directions, in same and cross namespaces. Exact immutable tuple: Server 2.6.1 / Workflow 2.5.12 / PHP SDK 2.3.3 / Python 2.6.0rc2 / CLI 2.2.3 / Waterline 2.9.1. No SDK/runtime source overlay. The normal completed, typed-failure, direct-target cancellation, caller recovery and backend-interruption gates still pass.
All four observed caller requests were accepted. Duplicates preserve their original identity and 30-second deadline. None records
CooperativeCancellationDelivered, none of the original target histories recordsCooperativeCancellationRequested, and the original calls remain Started. Both callers are still Waiting when their original result wait expires. The operation explicitly allows cancellation and propagation. PHP retains one original admission. Python records 72 and 60ServiceCallStartedevents, with advancing operation sequences but the same original call ID and linked target. This is a published prerelease Python finding, not a claim about stable 2.5.8's producer.The same-language cases were runner-blocked by duplicate operation registration, so the full eight-case cohort is incomplete. The runner now reuses its two exact operation definitions, preserving the original four outcomes. The original artifact is 11733381051, independently verified at 9,115,440 bytes and SHA256
97475a02c73c8216f849848a2ed12f3f9a63b315a09a949e692883c8f8b5bf30.Source inspection explains the missing path: both SDK cancellation-boundary mappings omit service operations, and the backend delivery-kind list omits them. Existing
cancelCall()uses terminal cancellation. The correction must cover the actual leased operation boundary, genuine target cooperation, original root identity/deadline and retained pending histories. Existing terminal APIs keep their defined semantics. Shared/native contract work remains coordinated with Server #325.rmcdaniel commentedon Oct 12, 2026
Python PR #128 is merged as
6288bd3e6577f0cd35ae923fd0502e122ee398f1. The reviewed and merged trees match. Source and fresh main checks pass, including Python 3.10–3.12, the immutable replay corpus, package/lint/portal checks and 39 ordinary HTTP integration cases. The 46 separately gated integration scenarios are skipped and are not claimed as qualified here.The correction submits no new admission while the original synchronous call is still pending. Three regressions fail against unchanged main and pass with the correction. Existing terminal/typed-failure/cancelled delivery, async/acceptance-only, external-result and changed-target checks pass.
The producer candidate remains unreleased. Stable 2.5.8 is unchanged. Full waiting-caller cancellation and retained-history qualification remain under Workflow #753, with PHP foundation PR #765 in review. This issue stays open for its remaining acceptance work.
rmcdaniel commentedon Oct 12, 2026
Python #129 is merged at 0705f3d with the identical qualified tree. Source and actual HTTP outcome covers both PHP/Python directions, Python/Python cooperation and actual killed-worker cleanup recovery. Fresh main CI is running. This issue remains open for published artifacts and its required terminal/failure/authorization/external-payload consumers. The next product step is the remaining SDK direction and a coherent qualified Nexus release cohort.
rmcdaniel commentedon Oct 13, 2026
Recovered the existing #128/#129 producer corrections at current main
24ad3e774ddeea4ce552189ffdc1afa401466ec2; no duplicate implementation is needed.The owning capability decision remains Workflow #753: protocol 1.20, both worker cancellation flags, authenticated Server discovery with its installed
workflow_runbinding, and original call/ownership/delivery/deadline preservation. There is no qualified published cooperative Nexus runtime tuple yet. Published Server 2.6.1 / Workflow 2.5.12 is the unchanged unsupported/control target.Next I will verify the recovered Python source against that actual published HTTP target: explicit Nexus cooperation refuses before registration/admission, while ordinary cooperative registration still succeeds and cleans up. No synthetic capability override or positive producer publication. The coherent installed backend/SDK cohort remains the prerequisite for the remaining #116 positive consumer gates. Stable 2.5.10 excludes the pending producer changes.
rmcdaniel commentedon Oct 13, 2026
Recovered source
24ad3e774ddeea4ce552189ffdc1afa401466ec2now passes an actual installed-source HTTP refusal/control check against unchanged published Server2.6.1 / Workflow2.5.12, imagesha256:ef043e5c582761e486a64a2ab4226476e17ae4f71682ef59d6b9ddd784d246b6.The exact unreleased source wheel is
b7cc06a86091be7c75f214b41849606a3841fdd8c4c70ca4debbfe439d238284; its existing metadata2.6.0rc2 does not identify the older published PyPI producer. Installed Client/Worker hashes were checked against the recovered source.workflow_runbinding.nexus_cooperative_cancellation_not_supported: Server must advertise its installed workflow_run bridge. Its wire contains discoveryGET only; no registration or admission occurs for that worker.Two private runner setup mistakes were corrected before the final result: tmpfs execution permissions and an assertion using the wrong registration route. Neither is a product failure. All test containers, network, volumes, dependencies and source scratch are retired after retaining the useful evidence.
The issue stays open. The coherent installed runtime cohort under Workflow #753 is still required for positive consumer qualification, publication and remaining terminal/failure/authorization/external-payload outcomes. Stable2.5.10 and existing frozen tuples remain unchanged.