Skip to content

feat(storage): opt-in auto-GC after storage-tail compaction - #40

Open
tiensonqin wants to merge 1 commit into
mainfrom
devin/storage-auto-gc
Open

tiensonqin wants to merge 1 commit into
mainfrom
devin/storage-auto-gc

Conversation

@tiensonqin

Copy link
Copy Markdown
Contributor

Summary

Storage.collect_garbage was manual-only, so storage files only grew. This adds an opt-in auto-GC policy evaluated after every storage-tail compaction, with the default left at upstream's manual-only behavior.

API surface (smallest that covers all conn/db entry points): a storage_auto_gc : auto_gc option field on the storage record — set it via ?auto_gc on memory_storage, file_storage, Datascript_sqlite.storage, or the logseq_sqlite_storage.ml example storage. No signature changes on create_conn/init_db/conn_from_*/restore_conn/transact.

type auto_gc = { min_garbage : int; garbage_fraction : float }
val default_auto_gc : auto_gc (* { min_garbage = 1024; garbage_fraction = 0.5 } *)

Trigger points: after a compaction stores a fresh index snapshot — in Conn.transact/Conn.apply_report (via a new maybe_collect_garbage field on transact_context, keeping conn.ml policy-agnostic) and in the db-level persist_transact_tail — maybe_collect_garbage storage recomputes addresses unreachable from the new root and storage_deletes them when they are both >= min_garbage and >= garbage_fraction of all stored addresses. With storage_auto_gc = None (the default) it is a no-op; collect_garbage remains the manual path.

Also exported for explicit use: Storage.default_auto_gc, Storage.auto_collect_garbage (evaluate a policy once), Storage.maybe_collect_garbage (honor storage_auto_gc). Refactored collect_garbage onto shared live_address_set/unreachable_addresses helpers. Documented in impl/datascript.mli and docs/design.md.

Split out of #38 — independent of the storage_tail representation change there; both land cleanly in either order.

Tests

All 34 native suites pass. test_storage__test_auto_gc covers: always-tripping policy leaves only reachable addresses while default storage accumulates garbage across repeated compactions; a high min_garbage floor keeps storage manual; maybe_collect_garbage is a no-op without storage_auto_gc and honors it when set; default_auto_gc values.

Link to Devin session: https://app.devin.ai/sessions/eb0e07b23c794857b6836d8a143d1688
Open in Devin Desktop: https://app.devin.ai/desktop/session/eb0e07b23c794857b6836d8a143d1688?variant=devin
Requested by: @tiensonqin

@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

collect_garbage was manual-only, so storage files only grew. The storage
record gains storage_auto_gc : auto_gc option — set it via ?auto_gc on
memory_storage, file_storage, the SQLite storage constructors, or
logseq_sqlite_storage.ml's storage, with no signature changes on
create_conn/init_db/restore_conn.

After every tail compaction (conn transact/apply_report and
persist_transact_tail), maybe_collect_garbage re-evaluates the addresses
unreachable from the new root and deletes them when they are both
>= min_garbage and >= garbage_fraction of all stored addresses.
default_auto_gc = { min_garbage = 1024; garbage_fraction = 0.5 }.
Default None keeps upstream's manual-only behaviour.

New exports: auto_gc/default_auto_gc/auto_collect_garbage/
maybe_collect_garbage. Documented in impl/datascript.mli + docs/design.md.
test_storage__test_auto_gc covers trip/floor/no-policy behaviour.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant