Skip to content

fix(lifecycle): count reservation setup against the startup deadline - #42

Draft
kriszyp wants to merge 4 commits into
mainfrom
fix/startup-deadline-from-reservation
Draft

kriszyp wants to merge 4 commits into
mainfrom
fix/startup-deadline-from-reservation

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

The six-minute corrupt-pool quarantine merged in #32 assumed that a default start would bind its reserved address or abandon startup before recovery. startupMaxMs began after spawn, so log setup, address validation and synchronous pre-spawn work could consume unbounded extra time. On base b06bed0, a 100 ms ceiling still admitted a successful start after 150 ms of log setup (191 ms total).

❓ Your call: warranted as requested by the maintainer: retain six-minute recovery and count setup after reservation against the existing startup ceiling. Slow setup now reduces boot time.

💡 Solution

startupMaxMs now covers the pool claim through observed readiness. Pool/quarantine waits and install/fixture preparation before the claim are excluded. Expired calls refuse to spawn; otherwise the watchdog receives only the remaining budget. A five-minute ceiling with one minute of post-claim setup leaves four minutes to boot. Idle timeout resets and post-readiness monitor registration keep their existing behavior.

⚖️ Alternatives

Planning review: Framing-Verdict: chosen-approach-sound (Claude). The review confirmed the current-main gap and the need to timestamp inside the allocator, because validation, lock release and probes also happen after the claim.

  • Increase or derive quarantine: conflicts with the requested six-minute policy and still cannot cover unbounded setup.
  • Revalidate reservation ownership before spawn, or hold a canary socket during setup: does not enforce the requested reservation-to-readiness ceiling; ownership fencing remains separate work in Prevent stale loopback pool writers from overwriting address claims #33.
  • Document the pause residual or pass a duration after allocation: leaves the reproduced race or excludes work after the claim and inside script resolution.

❓ Your call: keep the strict observed-readiness boundary: readiness delivered at or after expiry fails, including when a runner stall delays a healthy child’s output. This enforces the chosen startup contract; it is not a universal address-uniqueness guarantee under host stalls or clock changes.

🔧 Changes

The internal allocator in src/loopbackAddressPool.ts returns {loopbackAddress, reservedAt}, timestamped immediately before publishing the claim; the public getNextAvailableLoopbackAddress(): Promise<string> is unchanged. src/harperLifecycle.ts carries that origin through setup, resolves the configured/default ceiling once, checks immediately before spawn and readiness, and arms the existing watchdog for the remaining budget.

Failed-start cleanup in harperLifecycle.ts stops the group and gates resource release on confirmed child exit and free ports; Windows skips an exited leader PID.

README.md documents the changed option/environment semantics, recovery limits and cleanup behavior. CONTRIBUTING.md records the internal timestamp handoff and cleanup invariant. No documentation companion is needed: this package keeps its API and internal documentation in these files.

Startup order

How does a newly allocated startup spend its ceiling?

harperLifecycle.ts:623

{ loopbackAddress, reservedAt } = ⏳ reserveLoopbackAddress()
⏳ mkdir(logDir, …)
startupDeadline = reservedAt + maxTimeoutMs
harperScript = getHarperScript(harperBinPath)
✗ if Date.now() >= startupDeadline
proc = spawn(runtime, runtimeArgs, spawnOptions)
maxTimer = ⏳ startupDeadline - Date.now()
on readiness:
  ✗ if Date.now() >= startupDeadline
  clearTimers()
  ⏳ trackedProcess.registered  // the ceiling stops at readiness
on failure:
  if proc?.pid:
    signalHarperTree(proc, 'SIGKILL')
    ⏳ killHarper(…, { graceMs: 0 })
    ✗ keep resources if exit or ports are unconfirmed
  if ownsLoopbackAddress && loopbackAddress:
    ⏳ releaseLoopbackReservation(loopbackAddress, reservedAt)
  if ownsDataRootDir:
    ⏳ rm(dataRootDir)

Product and architecture tour

Deadline contract

Before After
The ceiling began after spawn. The claim starts the ceiling; setup leaves only the remaining boot budget.

Pool waits and preparation before the claim are excluded. Readiness observed at or after expiry fails; registration can finish afterward. Reused hostnames get a fresh per-call budget.

  • Claim timestamp — Recorded inside the lock before publication.
  • Spawn deadline — Script resolution consumes the budget and expiry rejects before spawn.

Ownership and diagnostics

Scenario Failure behavior
New address and owned install Signal the process group, confirm exit and free ports, release the slot, remove the install
Caller-supplied directory Keep the directory; release a newly claimed slot
Reused hostname Fresh budget per call; keep that reservation and directory
Windows leader already exited Skip taskkill for that PID; port checks still gate cleanup
Child exit or port release unconfirmed Warn and keep reservation/install parked
Corrupted pool during cleanup Release writes nothing while quarantined
Claim at least six minutes old Leave the pool untouched; recovery may have assigned a newer same-PID claim

❓ Your call: remove owned installations after failed boot, including their local hdb.log, to avoid leaking temporary installs. Captured stdout/stderr and configured external log directories remain available. Cleanup failures warn while preserving the original startup error.

A pre-existing allocator limitation remains: validation or bind-canary failure before it returns can leave a claimed slot under the runner PID. The failure cleanup here starts once allocation returns; this PR does not change those allocator error branches.

The new internal releaseLoopbackReservation checks age under the pool lock. At six minutes or older it skips automatic release because a recovered slot can already belong to a newer start under this PID; the public release API stays unchanged.

Recovery remains bounded: direct/legacy holders, long custom ceilings, reservations held between kill and restart, wall-clock changes and stalled runners can outlast six minutes. The quarantine constant and recovery policy are unchanged.

❓ Your call: retain POSIX group cleanup after leader exit because it stops the reproduced surviving-descendant failure. If a runner pauses after reaping, the original group disappears, and its number is reused before cleanup resumes, the signal can reach an unrelated group. Portable group-identity fencing needs separate process-supervision work; skipping every exited leader would leave those descendants alive. This rare pause-and-reuse risk is accepted for this draft.

The short internal helper contracts are retained: they document the clock origin, listener ordering and private export boundary, following the repository’s existing internal-helper convention.

✅ Verification

  • npm run build, npm run check, npm test: passed on Node 26.2.0, Linux; 100 tests total, 88 passed, 12 existing platform skips.
  • test/harperStartupDeadline.test.ts: 22 passed. Covers zero/default/exact expiry, fixture cleanup, pre-claim waits, work after the claim, reduced boot budget, late readiness, script lookup, setup/spawn failures, caller/restart ownership, quarantined cleanup, leader/descendant failure, held ports, clean/nonzero exit at expiry, preservation of a newer same-PID claim, unconfirmed child exit, the Windows exited-PID guard, and chained catch delivery for pre-spawn timeout.
  • Fails-on-base: the original 20 tests ran with base lifecycle/pool source. Nineteen failed at the intended expiry/cleanup assertions; the pre-claim-wait compatibility test passed. Four additional cleanup/diagnostic cases also failed on the first reviewed implementation. The new Windows test also fails against the previous implementation at the PID-target assertion. Removing the failed-start group signal makes the descendant test fail at the intended assertion after about five seconds, rather than hang. A chained-catch regression also fails against the prior head because pre-spawn expiry previously threw before returning its Promise. Restored source then reran full gates successfully.
  • Process-boundary route: the actual lifecycle, isolated pool files/locks and filesystem with controlled clock/setup boundaries, using small fake Harper CLI processes, including a descendant listening on a private ephemeral port. Port inspection asserts the full production port list while probing that private port with a real-clock bound. A fake exited child exercises the Windows branch on Linux; native Windows process-tree behavior remains for CI. Real Harper boot/bind ordering and macOS/Windows behavior have not been exercised locally; CI runs the repository’s OS/Node matrix.
  • Existing lifecycle tests also passed, including idle progress reset and registration delays after readiness.
  • Commitlint and git diff --check: passed; the repository defines no linter/formatter.
  • CI matrix: all nine Linux/macOS/Windows × Node 22/24/26 jobs passed on head 4acac9a; commitlint also passed.

🤖 Generated by OpenAI Codex; posted via @kriszyp.

Related PRs: #32 overlaps (follow-up to merged quarantine), #33 overlaps (pool publication and release locking), #35 overlaps (startup and shutdown supervision), 1 others independent
Complexity: complicated

Review-Coverage: authored=codex; ran=cursor-composer,gemini,claude,cursor-muse; adjudicated=domain; declined=cursor-grok,cursor-kimi; rounds=4; full=2 @ 4acac9a

Review-Attention: read ~5m (decisions: do-less-alternative, startup-boundaries, reservation-age-guard, clock-source) @ 4acac9a

kriszyp and others added 4 commits October 8, 2026 17:32
Count validation and setup after an address claim against startupMaxMs, refuse expired launches, and give boot only the remaining budget. Clean up newly acquired resources after failures while preserving reused resources and unconfirmed children.

Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Reap failed process groups and verify ports before cleanup. Keep expired claims untouched after quarantine can replace them, preserve exit diagnostics, and isolate the startup tests from occupied host ports.

Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Skip taskkill once a Windows leader has exited, keep regression port waits bounded independently of the startup clock, and document the internal reservation helpers.

Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>
Return a rejected Promise before spawn so a chained catch receives the same startup timeout as the watchdog path.

Co-Authored-By: GPT-6.1 Codex <noreply@openai.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the startup timeout and address reservation logic for Harper integration tests. It introduces an absolute startup ceiling (startupMaxMs) that begins at the moment of loopback address reservation rather than after process spawning, ensuring slow setup times are accounted for. The loopback address pool now tracks reservation times (reserveLoopbackAddress), and automatic failure cleanup has been hardened to prevent releasing newer claims under the same PID after quarantine recovery. Additionally, Windows process tree signaling has been updated to avoid targeting already-exited leader PIDs, and a comprehensive test suite (harperStartupDeadline.test.ts) has been added to verify these edge cases. No review comments were provided, so there is no feedback to address.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant