[spalen0]

Closes #438 once the live pilot is complete (see Not yet done). Part of epic #437.

What this is

A continuously running worker that observes indexed TimelockEvent rows alongside the hourly timelock_alerts.py. It does not change alert delivery. The pilot sends no alerts. Its output goes to journald, $CACHE_DIR/timelock_subscriber.db, and optionally the internal errors chat (see below).

Transports (--transport both|ws|http, default both)

• WebSocket: a graphql-transport-ws live query (utils/graphql_ws.py) over the newest 50 monitored events, filtered by monitored chains and addresses. It reconnects with capped exponential backoff (1s–60s).
• HTTP reconciliation every 60s, which keeps running while the WebSocket is connected. For each chain it reads chain_metadata.latest_processed_block and fetches (checkpoint − overlap, head] in chunks of at most MAX_BLOCK_SPAN blocks. Pages are ordered by (blockNumber, logIndex, id).

State

• Per-chain checkpoints: independent per chain, never moved backwards, separate from TIMELOCK_LAST_TS.
• Event provenance: one row per Envio id with a content fingerprint, the first transport and time it was seen, and per-transport first-seen times and observation counts.
• Changes: changed (same id, new content), removed (no longer returned by a fully fetched range), restored. The old payload is kept.
• Worker log: WebSocket connects and disconnects, each reconcile's duration and per-chain counts, and ingestion timing.

Edge cases

• Duplicate snapshots and the overlapping re-scan only increment counters.
• Ranges are bounded by block number per chain, not by a timestamp cursor, so rows with equal timestamps are never skipped.
• Each cycle re-reads about 15–20 minutes of blocks per chain to catch late indexing and reorgs. The fingerprint includes blockHash, so a reorg shows as a changed row or a removed id.
• Restarts resume from the stored checkpoints. A backlog drains in bounded chunks.

Supervision

• systemd unit monitoring-timelock-subscriber: Type=notify, WatchdogSec=120, Restart=always.
• The heartbeat pings the watchdog only while reconciliation keeps completing, so a wedged worker is restarted. It also flags chains whose indexed head stopped advancing.
• event_subscriber.py health exits 1 on a stale heartbeat, a failed reconcile, a WebSocket outage, or a stalled chain.

Report

event_subscriber.py report --hours N shows:

• WebSocket coverage of HTTP-confirmed events, plus WebSocket-only and HTTP-only counts
• delay from block timestamp to first sighting per transport (p50/p95/max), and how far the WebSocket leads HTTP
• WebSocket outages and how the events inside them were recovered
• reconcile timing and errors
• a transport recommendation: both, http, or undecided

Tests

32 new tests drive the real worker, client, and SQLite inbox against a fake Envio (an in-process HTTP stand-in and a local graphql-transport-ws server). They cover:

• controlled disconnects with a backlog larger than the snapshot
• worker restarts with events indexed during downtime
• a 600-event backlog across page and chunk boundaries
• one chain failing while the others advance, and a metadata outage
• late indexing, reorgs (replaced and dropped rows), and stall detection

Each test asserts every expected event is stored exactly once. The suite passed 8 consecutive runs. ruff, ty, and the full suite (1957 passed) are clean.

Docs and ops

• protocols/timelock/README.md: pilot design.
• deploy/runbook.md: operations, controlled restart/backlog/disconnect tests, settings, and a failure-mode row.
• deploy/install.sh installs the unit, and .env.example lists the new variables.
• monitoring.yaml is unchanged because alerting behavior does not change.

Live validation (2026-10-22, updated Envio endpoint)

• Schema: TimelockEvent exposes blockHash. It is now selected, stored per event, and included in the fingerprint, so a replaced block shows as a changed row.

• Bug found and fixed: Envio's Hasura sends a keepalive ping before connection_ack. The client now answers pings during the handshake, and the fake server in the tests does the same.

• HTTP backfill (30 days): 230 events across 5 chains. The result matches an independent timestamp scan exactly, with no missing and no extra events. A rerun yields no new rows and no duplicates.

• WebSocket: connected on the first attempt and received the 50-row snapshot with no disconnects. A 5-chain reconcile takes about 1.5s.

• Restart on the same database: still 230 rows, 0 revisions, 0 changes, and every checkpoint advanced. The reconnect snapshot was absorbed as a duplicate, and SIGTERM stops the worker cleanly.

• Real new event: Aave ProposalQueued 1_26249234_585 was queued at 16:04:35 UTC while the worker was stopped. On restart it was recovered by both the WebSocket snapshot and HTTP, stored once with 0 revisions.

Review fixes

• Concurrent ingestion: observe, mark_missing, and set_checkpoint now take the SQLite write lock (BEGIN IMMEDIATE) before reading. The old code raised IntegrityError in 40 of 40 concurrent-observation rounds; the fix brings that to 0. The reconcile loop also survives unexpected exceptions.

• Backoff overflow: the exponent is capped, so 2 ** n can no longer overflow after about 1,025 failures.

• Watchdog: the reconcile deadline counts from worker start until the first cycle completes, and pings are withheld if any loop task has ended.

• Report: the connection state at the window start comes from the last earlier log entry, so an outage that began before the window is reported, clipped to it.

• Schema migration: the inbox schema is versioned. A database from the first revision gains block_hash in place, keeping all observations and checkpoints. Without this, ingestion failed with no such column: block_hash. Rows stored before blockHash was selected are upgraded silently on their next observation, not recorded as reorg changes. Tested on a copy of the live pilot database: 231 events and 5 checkpoints kept, 0 spurious changes.

• Startup notice baseline: notices only announce events from blocks after the time notices were first enabled on the inbox, a value that persists across restarts. A fresh pilot's first WebSocket snapshot no longer posts week-old events, while events indexed during a later downtime are still announced.

Errors-chat notices (opt-in)

TIMELOCK_SUBSCRIBER_NOTIFY=errors makes the pilot post plain-text diagnostics to the internal errors chat, labelled [timelock-subscriber] and stored under yearn-internal. They never go to protocol groups: notices stay off unless TELEGRAM_TOPIC_ID_ERRORS or TELEGRAM_CHAT_ID_ERRORS is set.

• 🛰️ Each new event at first sighting, with the transport that saw it first, the delay since its block, and full address and tx links. Bursts of more than 10 collapse to one line per chain. Events from before the notice baseline are never announced.
• ⚠️ WS misses: HTTP found events first while the WebSocket was connected.
• 🔀 Changed, removed, or restored rows.
• 🔌/🐢/❌ WebSocket down for more than 5 minutes, a stalled chain head, or 3 failed reconciles in a row, each followed by a ✅ recovery.
• 📊 A daily report summary whose schedule survives restarts.

Notices go through a bounded background queue that combines messages queued during a send and drops failed sends, so Telegram never delays ingestion. A notice was rendered from live data (Aave ProposalQueued, captured locally, not sent).

Not yet done (blocks closing #438)

• A real new event has been observed only through restart recovery. Live-delivery latency, with the worker running when an event lands, has not been measured yet.
• After deploying the unit: run the runbook's controlled disconnect, restart, and backlog tests on the server, collect about a week of data, and post report --hours 168 with the production transport decision. Envio fully supports hosted WebSockets only on Dedicated plans.

🤖 Generated with Claude Code