Repository navigation
Codex discovery re-offer costs about 28 ms per unchanged rollout in project scope, and profile scope re-offers on every restart #4374
Description
Summary
Measured while verifying PR #4366 on the shipped CLI, with 2,010 Codex rollout files (7.9 MB) and one registered project.
- Project scope: a full re-offer of unchanged rollouts takes about 62 s, about 28 ms per file. Admission itself totals under 1 s for 4,020 file reads. The rest is spent between files, and no tracing span covers that gap.
- Profile scope: the same re-offer takes about 4.7 s, about 1.3 ms per file.
- Profile scope re-offers on every restart. It offers all 2,010 files on each daemon restart, reading 0 bytes.
masterat e2be4c3 does the same (2,010 files in one restart, 7 in another). The project scope settles and re-offers nothing.
PR #4366 deliberately triggers one re-offer of every rollout after the upgrade. With this per-file cost, that one-time pass grows with corpus size times the number of registered projects.
Next step
Instrument the project-scope discovery loop per AGENTS.md (TRACEDECAY_SPAN_TIMINGS=1) to find what the 28 ms per file is spent on. Separately, find why the profile-scope completion claim does not match after a restart.
Activity
devin-ai-integration commentedon Oct 20, 2026
Profile-scope half: fixed in #4380.
Root cause: after a restart, the consumer that registers second (usually profile) joins a hub whose shared scan has already started, so
CodexDiscoveryHub::registerputs it inReplaymode. Replay delivery started from positionNoneand never compared the consumer's durable complete frontier withindex.frontier, so every indexed rollout was redelivered and then skipped by the source ledger (0 bytes read). Registration order decides which consumer is second, which explains why one restart re-offered 2,010 files and another only 7. Reproduced with a hub-level test: 21/21 rollouts re-offered before the fix, 0 after, and a newly added rollout is still delivered.Project-scope 28 ms/file: still open, not measured yet. From reading the code, the per-file path in
run_codex(project_provider.rs) isSourceSweep::observe_file→ (sometimes)codex_rollout_project_membership→try_admit_codex_jsonl_observations_for_project_window. Only the admission reads are covered by the spans quoted in the issue. A cost of about 28 ms per file looks like a durable store commit per rollout (cursor or convergence write) rather than CPU, but this is a hypothesis. The next step is aTRACEDECAY_SPAN_TIMINGS=1daemon run with spans aroundobserve_fileand the per-file store writes.Written by Devin