Skip to content

Use a dedicated pub/sub connection under RESP3 by default - #3264

Open
mgravell wants to merge 3 commits into
mainfrom
marc/resp3-pubsub-connection
Open

mgravell wants to merge 3 commits into
mainfrom
marc/resp3-pubsub-connection

Conversation

@mgravell

@mgravell mgravell commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #3263.

Problem

Redis classifies any connection with a live subscription as a pub/sub client (flags=P), and applies the pubsub class of client-output-buffer-limit to it - 32mb 8mb 60 by default, against no limit for normal clients. #3255 (unreleased) correctly subscribed the configuration channel under RESP3, but because RESP3 shared the interactive connection with pub/sub, that made every RESP3 interactive connection a pub/sub client. A single large reply, a large MGET/HGETALL, or a burst of pipelined replies then gets the connection closed by the server, failing every in-flight command. This was never specific to the config channel: any RESP3 user who subscribed to anything was exposed; #3255 just made it the default.

Measured against Redis 8.9 with default limits:

connection flags GET 40MiB value
RESP2, unsubscribed N ok
RESP3, unsubscribed N ok
RESP3 + SUBSCRIBE P closed by server, 0 bytes
RESP3 SUBSCRIBE then UNSUBSCRIBE N ok

The whole reply is checked against the hard limit before it is written, so MGET of a 64KiB value ~600 times reproduces it just as well.

Change

  • New ConfigurationOptions.SharedSubscriptionConnection / sharedSubscriptionConnection= (also on DefaultOptionsProvider), default false. Under RESP3, pub/sub now uses a dedicated connection (negotiating RESP3 itself), exactly as under RESP2; sharing the interactive connection is opt-in. No effect under RESP2. No provider overrides the default.
  • Routing goes through ServerEndPoint.SharesSubscriptionConnection() instead of KnowOrAssumeResp3(); with sharing off, routing no longer depends on the pre-handshake protocol guess at all. Connect monitors wait for the second leg unless shared.
  • Fixed a double QUIT to the shared connection on close (Close(Subscription) resolved to the interactive bridge).
  • Docs: Resp3.md (new "Pub/sub connections" section), Configuration.md, SyncOverAsync.md - including advice to check the server's pubsub limits against the largest replies/pipelined bursts before opting in.

Tests

  • Connection-shape tests now run in both modes rather than assuming RESP3 = one connection.
  • New PubSubOutputBufferTests: reads the server's pubsub hard limit (skips if unavailable), subscribes, then MGETs past it; dedicated connection survives and stays Normal, shared RESP3 connection is killed. No server config change, so it can run in parallel.
  • MaintenanceNotificationTests: four exact delivery counts relaxed to > 0, since the test server's SendRawPush(null, ...) deliberately ignores opt-in and now also reaches the RESP3 subscription connection.

Full suite green on net10.0 locally; on net8.0 one unrelated timing flake (KeyIdleAsyncTests.TouchIdleTimeAsync, passes in isolation). net481 not run locally.

Release notes (for the GitHub release)

Under RESP3, pub/sub now uses a dedicated connection by default, as under RESP2, so RESP3 clients use two connections per server rather than one. Sharing a single connection caused the server to apply its (much tighter) pub/sub output-buffer limits to ordinary traffic, which could close the connection under large replies (#3263). To restore the previous single-connection behaviour, set sharedSubscriptionConnection=true - after checking that the server's client-output-buffer-limit pubsub settings suit your largest replies.

The server classifies any connection with a live subscription as a pub/sub
client and applies the pubsub output-buffer limits to it (32mb hard / 8mb
for 60s by default). Since #3255 subscribed the configuration channel on
the shared RESP3 interactive connection, ordinary large replies or
pipelined bursts could get that connection - and every in-flight command
on it - closed by the server.

Add ConfigurationOptions.SharedSubscriptionConnection
(sharedSubscriptionConnection=, default false): under RESP3, pub/sub now
gets its own connection as under RESP2, unless sharing is opted into.

- route via ServerEndPoint.SharesSubscriptionConnection() rather than
  KnowOrAssumeResp3(); connect monitors wait for the second leg
- send QUIT once when the connection is shared
- tests cover both modes; new PubSubOutputBufferTests reproduces the
  server-side kill with an oversized MGET, without changing server config
- docs: Resp3.md / Configuration.md explain the limits and advise checking
  the server's pubsub limits before opting in
CI saw a third socket (two Subscription sockets at the same timestamp):
ActivateServer and OnFullyEstablished both lazily created the subscription
bridge via a non-atomic `??=`, and a bridge connects as soon as it is
created, so the loser leaked an unreferenced connection. The previous
commit made OnFullyEstablished call Activate(Subscription) on every
non-shared RESP3 connect, widening a previously rare race.

- GetOrCreateBridge: CompareExchange the new bridge in; the loser is
  disposed (closing its connection)
- OnFullyEstablished only activates pub/sub for the shared-RESP3-but-got-
  RESP2 case it was written for; otherwise ActivateServer has done it

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RESP3 when using pub/sub can cause unexpected connection closures due to backlog

1 participant