Repository navigation
Reuse HTTP connections for synchronous node discovery - #162
dkropachev wants to merge 8 commits into
Conversation
Code Review by Qodo
🔴 High 1. Restart can lose background refreshes
|
Keep an owned HTTP/1 pool across /localnodes polls instead of letting urllib close every socket. Buffer and drain responses before returning connections, while preserving urllib proxy and redirect handling. Cancel leases at timeout and manager shutdown, close their sockets and prevent stalled DNS workers from connecting after close. Reopen admission on manager restart and clean up pools after failed construction. Cover HTTP/HTTPS reuse, service errors, stale connections, DNS and TLS cancellation, mutual TLS, deadlines and startup failures. Fixes #160
9b23011 to
95cd0d8
Compare
Drain unused response bodies in bounded chunks while preserving connection reuse and urllib status handling. Roll back failed manager startup for Helper callers, and order pool closure with reopening so an older stop cannot disable a restarted manager. Cover the failure paths, proxy tunnel isolation and response reuse. Explain the expected peer disconnect in the deadline regression test.
Fetcher closure now runs under the lifecycle lock, ordering it with restart; the thread join remains outside the lock.
Refresh-thread construction or startup failure now closes the reopened fetcher and clears the failed thread state, including startup through Helper.
Non-2xx response bodies are drained in bounded chunks without retaining them, preserving status and headers for urllib and allowing reuse after a complete drain. |
Expose each connecting socket through its existing HTTP connection owner before blocking in connect, so pool shutdown and deadline cancellation can interrupt it. Clear failed candidates and retain the shared deadline and post-connect cancellation checks. Cover both HTTP and HTTPS through close and deadline cancellation, checking socket cleanup and worker completion rather than caller timeout alone.
Each TCP candidate is now exposed through the owning connection before connect starts, so pool shutdown and deadline cancellation can interrupt it. Failed candidates are closed and cleared. Regression tests cover both cancellation paths for HTTP and HTTPS. |
Reopen the owned fetcher before retaining a still-live stopped refresh thread, so a concurrent start preserves manual discovery without replacing the old thread prematurely. Observe peer disconnect during dripping reads and ClientHello before handshake cancellation. Guard cleanup of unstarted test threads, and state the retry's complete-header boundary accurately.
| if self._refresh_thread is not None and self._refresh_thread.is_alive(): | ||
| return |
There was a problem hiding this comment.
🔴 High
2. Restart can lose background refreshes 🐞 Bug ≡ Correctness
SyncLiveNodesManager.start() returns when the old refresh thread is still alive, even if stop() has already set that thread's stop event. If start() runs while stop() is joining the thread, the pool reopens and manual discovery works, but the old thread exits without a replacement to perform future background refreshes.
Agent Prompt
## Issue description
A start during the stop join window reopens discovery but returns because the stopping refresh thread is still alive; no background thread replaces it.
## Fix Focus Areas
- alternator/core/live_nodes.py[397-452]
- alternator/core/live_nodes.py[543-558]
- tests/unit/test_sync_discovery_pool.py[285-341]
## Recommended Fix
When the existing thread is alive but its stop event is set, arrange for a replacement to start after that thread exits, without replacing it while it is still running. Extend the overlapping stop/start test to verify that background refresh resumes, not only that manual refresh succeeds.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Observe the starting thread's actual lifecycle-lock admission attempt before releasing the pending close. Require blocked admission so delayed thread scheduling cannot let unsynchronized closure pass the regression. Keep pool usability, peer closure and bounded cleanup checks. A controlled delay and broken-locking mutation distinguish the revised test from the legacy false positive without changing production behavior.
Pair token registration with cleanup on every exit before worker startup. If start raises after creating a worker, cancel it before releasing the registration so late DNS completion cannot connect. Pass each background generation's stop event through refresh processing so shutdown abandons the pass without reporting cluster failure or clearing valid affinity. Keep manual and genuine discovery failures unchanged. Cover startup exceptions, cancellation during ordinary and affinity discovery, and manual failure after stop.
Stop and unregister the owned manager before propagating KeyboardInterrupt or SystemExit from client or resource creation. Catching only Exception could leave the running refresh thread and pooled sockets without an owner. Preserve borrowed Helper managers and existing clients. Cover both owned factories with real socket and original-thread cleanup observations, plus borrowed ownership controls.
Drain unused bodies through a fixed readinto buffer so malformed chunk sizes cannot trigger an unbounded lower read. Only final responses may return their connections to the idle pool. Cover malformed and valid chunking, 100/103 response handling, idle capacity and disposal of duplicate leases. Keep independent manager cleanup when construction tests fail.
Summary
Synchronous
/localnodespolling previously opened a fresh TCP connection for every fetch because urllib forcedConnection: close. Retain a manager-owned HTTP/1 pool with up to 200 idle connections, one per endpoint, and drain response bodies before returning connections to the pool, including non-2xx responses.Keep urllib's proxy and redirect handling, existing TLS settings, and the total discovery deadline across DNS resolution, address fallback, and response reads. A discovery GET can retry once when a reused connection closes before response headers; both attempts share the same deadline.
Manager shutdown cancels active leases, interrupts sockets, and prevents stalled DNS workers from connecting later. Restart admits fresh requests, and failed initialization or thread/SDK construction closes discovery resources.
Fixes #160. Follow-up to closed #10; data-request host-pool sizing is handled separately in #161.