Skip to content

feat(prune): experiment with time-based retention and gradual catch-up - #25

Draft
meyer9 wants to merge 1 commit into
backport/v2.5.2/mainfrom
feat/time-based-pruning-catch-up
Draft

meyer9 wants to merge 1 commit into
backport/v2.5.2/mainfrom
feat/time-based-pruning-catch-up

Conversation

@meyer9

@meyer9 meyer9 commented Oct 13, 2026

Copy link
Copy Markdown

Opt-in experiment for the Base Denim 2s → 200ms pruning transition. Replace distance-configured history cutoffs with timestamp-based retention while keeping the existing pruning worker, scheduling, and execution limits. This is not a pruning work/time budget.

  • Start each segment at its actual checkpoint; advance the cutoff by 1.3 historical seconds per second of chain progress, capped at head time minus 31 days.
  • Persist anchors with prune checkpoints so restarts and snapshot copies preserve progress. Policy changes re-anchor rather than advancing retroactively.
  • Preserve ExEx and minimum-distance constraints; prevent backfill writes from discarding data using the superseded distance; check actual pruning progress before unwinds.
  • Defaults remain unchanged. Only distance-configured bodies, receipts, account history, and storage history use the opt-in policy.

Enable in reth.toml, alongside the existing distance modes:

[prune.segments.time_based]
retention_seconds = 2678400
catch_up_percent = 130

Snapshot trial: based on Base’s pinned base-v2.5.2.7 release branch. Testing in Base requires rebuilding with this Reth branch and enabling the config; this PR does not update Base’s dependency pin, publish binaries, or modify snapshots. Snapshot consumers need a compatible binary and the enabled config as well as database metadata/checkpoints. Seven days of excess history takes about 23 days to eliminate at 130%. Historical-time advancement is not CPU usage, and fast catch-up can still produce long prune runs. Removing the policy restores block-distance pruning and can cause a catch-up cliff.

Validation: 222 pruning/config/stages/type tests passed (one existing ignored), plus the receipt-write regression passed across storage V1/V2. The real-database transition test advances 50 new 200ms blocks while pruning six old 2s blocks per history segment, survives restarts, and converges. Node/CLI all-target compilation, targeted all-target Clippy with -D warnings, and pinned nightly formatting passed. Workspace all-features Clippy is blocked by missing LLVM 22.1. Snapshot throughput and persistence-backpressure behavior are not yet benchmarked.

Test command:

cargo test --locked -p reth-prune -p reth-prune-types -p reth-config -p reth-stages --lib --features reth-prune-types/reth-codec,reth-config/serde,reth-stages/test-utils
cargo test --locked -p reth-provider --lib test_prunable_receipts_logic

Implementation discussion

Advance per-segment cutoffs through historical time at a configurable rate,
anchored to committed progress and capped by the retention window. Keep the
existing pruning worker and execution limits unchanged.

Amp-Thread-ID: https://ampcode.com/threads/T-e489dc29-8527-4793-8a99-1c451c24fd73
Co-authored-by: Julian Meyer <julian.meyer@coinbase.com>
@meyer9 meyer9 added the enhancement New feature or request label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants