Skip to content

Give an aborted row stream's connection back when its response closes - #2735

Open
jh-RLI wants to merge 2 commits into
developfrom
feature-wf24-close-row-stream-session
Open

jh-RLI wants to merge 2 commits into
developfrom
feature-wf24-close-row-stream-session

Conversation

@jh-RLI

@jh-RLI jh-RLI commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Summary

A table download through GET /api/v0/tables/<t>/rows/ streams from a server-side cursor on a connection opened for that request. When the client stopped reading, the connection was only given back when the garbage collector next ran a full collection.

 class OEPStream(StreamingHttpResponse):
-    def __del__(self):          # waits for a full GC: the response is always in a reference cycle
-        if self.session:
-            self.session.close()
+    def close(self):            # the WSGI server calls this on completion and on abort
+        try:
+            if self.session:
+                self.session.close()
+        finally:
+            super().close()

Why __del__ never ran promptly: response.close() sends request_finished → close_caches → hasattr() on an asgiref Local, which raises and swallows an AttributeError inside a contextmanager; its traceback keeps the frame chain alive up to close(), whose self is the response. DRF adds a second cycle (view ↔ request). Same on production's versions (Django 5.2.17, asgiref 3.8.1, DRF 3.17.2).

Until then the request held its pool slot, its transaction, its snapshot and a lock on the table, idle in transaction on FETCH FORWARD 1. On production on 2026-10-21, two such connections were still open 72 minutes after their downloads had been aborted, and a DROP TABLE on such a table waits for them.

Evidence

api/tests/test_regression/test_aborted_row_stream.py, one case per form (JSON, CSV, datapackage), with gc disabled:

start GET .../rows/?form=…, read 3 chunks, close the body as a server does on an abort
assert no artificial session left, pool checked-out count back to baseline,
       no backend "idle in transaction" on FETCH
  • Before: FAILED (failures=3), each AssertionError: 0 != 1 (the session is still registered)
    After: Ran 3 tests … OK
  • manage.py test api: Ran 504 tests … OK

Merge Danger

Door: two-way

Revert the one method.

Blast Radius: small

OEPStream serves the REST rows download (all three forms), the advanced API's streamed results and advanced/cursor/fetch_* (no session there, so close() only calls Django's). A complete stream already closes its session after the last row; the second close is a no-op, as it was in __del__. An explicit advanced API session that is passed in is closed at the same point it was before, only no longer late — a late close in __del__ could close a session the client had meanwhile reused under the same id.

🤖 Generated with Claude Code

jh-RLI and others added 2 commits October 21, 2026 14:54
OEPStream closed its session in __del__. The response always sits in a
reference cycle (Django's request_finished reaches an asgiref Local
whose swallowed exception keeps the closing frame alive, and DRF's view
and request point at each other), so __del__ ran only at the next full
garbage collection. A client that stopped reading a rows download left
the connection, its transaction, its snapshot and a lock on the table
idle in transaction until then.

close() is what the WSGI server calls on completion and on abort alike,
so the session is closed there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant