Repository navigation
fetch_types: Connection killed during startup replays its 57P01 onto the first query after reconnect and an unhandled rejection #1223
Description
Versions
postgres 3.4.9 (the code is unchanged on master 411429e) · Node 24.19.0 · PostgreSQL 18.3
What happens
If a new connection's backend is terminated while the driver fetches array types (fetch_types, on by default), the driver reconnects, and the reconnect succeeds. But the first query on the new connection still fails with the old backend's error, and the same error also escapes as an unhandled rejection:
select 1 rejected: 57P01 terminating connection due to administrator command
unhandledRejection: 57P01 terminating connection due to administrator command
On Node 15+ that unhandled rejection exits the process by default. We hit it during pool warm-up after a database restart/failover, when pg_terminate_backend lands on a connection that is still starting up.
Reproduction
A TCP proxy in front of a real Postgres. The first time the driver sends the array-type query, the proxy answers with the exact ErrorResponse that pg_terminate_backend produces and closes the socket, the same as a real termination at that moment. Every later connection is passed through untouched.
import net from 'node:net'
import postgres from 'postgres'
const PGHOST = process.env.PGHOST || '127.0.0.1'
const PGPORT = Number(process.env.PGPORT || 5432)
// The ErrorResponse Postgres sends before closing a session killed by pg_terminate_backend.
function terminated() {
const field = (code, value) => Buffer.from(code + value + '\0')
const body = Buffer.concat([
field('S', 'FATAL'), field('V', 'FATAL'), field('C', '57P01'),
field('M', 'terminating connection due to administrator command'),
Buffer.from('\0')
])
const header = Buffer.alloc(5)
header.write('E')
header.writeInt32BE(body.length + 4, 1)
return Buffer.concat([header, body])
}
// A TCP proxy to Postgres that kills the first connection while it fetches array types (fetch_types).
let killed = false
const proxy = net.createServer(client => {
const server = net.connect(PGPORT, PGHOST)
client.on('data', chunk => {
if (!killed && chunk.includes('typarray')) {
killed = true
client.end(terminated())
server.destroy()
return
}
server.write(chunk)
})
server.on('data', chunk => client.writable && client.write(chunk))
client.on('error', () => server.destroy())
server.on('error', () => client.destroy())
client.on('close', () => server.destroy())
server.on('close', () => client.destroy())
})
await new Promise(r => proxy.listen(0, '127.0.0.1', r))
process.on('unhandledRejection', e => console.log('unhandledRejection:', e.code, e.message))
const sql = postgres({ host: '127.0.0.1', port: proxy.address().port, max: 1 })
try {
const [{ ok }] = await sql`select 1 as ok`
console.log('select 1 ->', ok)
} catch (e) {
console.log('select 1 rejected:', e.code, e.message)
}
console.log(await Promise.race([
sql.end({ timeout: 1 }).then(() => 'sql.end() resolved'),
new Promise(r => setTimeout(r, 3000, 'sql.end() still pending after 3s'))
]))
proxy.close()
process.exit(0)Run it against any Postgres; credentials come from the usual PGUSER / PGPASSWORD / PGDATABASE:
docker run --rm -d --name pg-startup-kill -p 127.0.0.1:5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres:18
PGUSER=postgres node repro.mjs| Output | |
|---|---|
| 3.4.9 / master | select 1 rejected: 57P01 … + unhandledRejection: 57P01 … |
| 3.4.9 + #1215 | same as above (#1215 does not cover this path) |
| 3.4.9 + the patch below | select 1 -> 1 |
Deterministic: the same result on every run.
Why
- During startup,
ReadyForQuerycallsfetchArrayTypes(), which runs its own internal query. The user's query is parked asinitial. The promise fromfetchArrayTypes()is not awaited by anyone. - The FATAL
ErrorResponsearrives while that internal query isquery, so it is stored inerrorResponseand waits for aReadyForQuerythat never comes, because the backend closes the socket. - In
closed(),if (initial) return reconnect()returns before anything clearsqueryorerrorResponse. - On the new socket, the first
ReadyForQuery(end of authentication) finds the stalequery+errorResponseand callserrored(errorResponse). That rejects the dead internal query, which nobody handles (the unhandled rejection), and then rejectsinitial(the user'sselect 1) with the old backend's 57P01.
connected() resets needsTypes, so simply discarding the stale state is enough: the new connection fetches the types again and the parked query runs normally.
Suggested fix
Clear the per-query state in the initial branch too, the same state #1215 clears on the non-initial path:
- if (initial)
+ if (initial) {
+ query = errorResponse = null
return reconnect()
+ }This is the same branch #1193 changes (for a different symptom: a clean close with a pending initial loops forever). Whatever shape the fix for #1193 takes, it would need to clear this state too.
Related, already reported
While investigating this we also hit the other connection-close bugs that are already open, and we fixed all of them together in one local patch (attached / below):
- null
socketinnextWrite()→ uncaught TypeError: nextWrite() throws an uncaughtException when a reserved connection's backend dies #1208, Error when attempting to write to a null socket #1154, TypeError: null is not an object (evaluating 'socket.write') #1066 (PRs Guard nextWrite against a closed socket - fixes #1208 #1209, Guard nextWrite against null socket after async close #1168) - stale
chunk/nextWriteTimer, and a stalequery/errorResponsereplayed after reconnect: covered by Reject transaction work after its connection closes #1215
Attachment: postgres@3.4.9.patch
Happy to open a PR for the startup-path fix (with a regression test based on the proxy above) if that helps, either standalone or on top of #1215.
Activity
[-]A connection killed during startup (fetch_types) replays its 57P01 onto the first query after reconnect, plus an unhandled rejection[/-][+]fetch_types: Connection killed during startup replays its 57P01 onto the first query after reconnect and an unhandled rejection[/+]JakeThomson commentedon Sep 29, 2026
We hit this in production on postgres 3.4.8 / Node 24, behind PgBouncer in transaction mode. When our Postgres crash-restarted, PgBouncer answered the array-type fetch with
server_login_retryand closed. The unhandled rejection took down every API process at once.The suggested fix on its own makes #1193 reachable
On the startup path, the stale replay you describe is the only thing that ever settles
initialafter a server error followed by a close. Onceclosed()clears it, a peer that keeps failing leaves the query parked, andreconnect()runs with zero delay: the loop from #1193.Against a fake PgBouncer that answers every type fetch with FATAL
08P01and closes, the suggested fix alone made 1,540 connection attempts in 3 seconds, and the query never settled. PgBouncer does exactly this for as long asserver_login_retrylasts, so the two fixes need to land together.What we're running
Your fix, plus the time-bounded reconnect from the #1193 thread (backoff, with a run budget of
connect_timeout, reset on any successful connect includingreserve()). We also kept a.catchonfetchArrayTypes()for a non-fatal failure such asstatement_timeout: the connection stays up,ReadyForQuerystill callserrored(errorResponse), and nothing else handles that promise.Results
Fake PgBouncer,
connect_timeout: 10, default backoff:08P01+ close)08P01at 9.4s57014)You offered to open a PR. I can contribute regression tests for all four cases, or happy to open the PR myself if you'd rather not.
cat-a-guerra commentedon Sep 30, 2026
@JakeThomson
Confirmed, and thanks — the one-line fix on its own does strand the query. Reproduced against a fake peer that completes the handshake and then answers every array-type fetch with
FATAL 08P01+ close,max: 1,connect_timeout: 10, postgres 3.4.9, Node 24.19.0:select 1rejected08P01in 7ms, 2 connection attemptsSo on this path the stale replay is indeed the only thing that settles
initialafter a server error followed by a close, and removing it exposes #1193 exactly as you describe. Agreed the two need to land together.Your
fetchArrayTypes().catch(noop)is a good catch too:ReadyForQueryalready settlesinitialthrougherrored(errorResponse), so the fetch's own promise has nobody waiting on it either way.The always-failing-peer script (no Postgres needed:
npm i postgres@3.4.9, thennode always-fails.mjs)Please do open the PR — that's very welcome, and your version (bound by
connect_timeoutwith the reset on any successful connect, includingreserve()) is what we'd want. Happy to review it and to contribute the proxy-based reproduction from this issue plus the always-failing-peer script above as regression tests, whichever is more useful to you.One note for anyone else landing here from a patch: we are carrying the stale-state reset alone in production, so we're taking the same bound into our local patch until a release has both.
JakeThomson commentedon Sep 30, 2026
@cat-a-guerra thanks for confirming, and for the always-failing-peer script. Opened #1241.
It's the stale-state reset from this issue plus the
connect_timeout-bounded reconnect from #1193 (reset on any successful connect, includingreserve()), and the.catchonfetchArrayTypes(). The first commit adds four tests, each of which fails on master:The full suite passes, ESM and CJS, on PostgreSQL 14 / Node 24. A review would be very welcome, thanks!
fix: replace connection lifecycle flags with explicit phases
garrytan commentedon Oct 10, 2026
Another independent reproduction, from garrytan/gbrain on postgres 3.4.9 under Bun 1.4.0. Under Bun the discarded rejection from the internal array-types query is fatal by default, so one backend terminated during the startup type fetch takes the whole process down.
Our minimal scripted-server test: the server answers startup, replies
FATAL 57P01and closes at the first statement, then serves the second connection normally. It's the first case intest/postgres-driver-startup-death.test.ts(MIT), if it's useful as an extra regression case. On 3.4.9,sql`select 1`rejects with the stale 57P01 and an unhandled rejection fires. With the stale-state reset plus a catch onfetchArrayTypes()(the approach in #1241), it resolves on the second connection.The same fix also covers #1192.
v4