Skip to content

perf(registry): skip hourly AI catalog scans when no work is missing - #668

Open
harlan-github-agent[bot] wants to merge 1 commit into
mainfrom
fix/issue-664
Open

harlan-github-agent[bot] wants to merge 1 commit into
mainfrom
fix/issue-664

Conversation

@harlan-github-agent

Copy link
Copy Markdown
Contributor

🔗 Linked issue

Closes #664

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

The d1-query-review routine measured 373 seconds and 19 million row reads per week on the hourly AI work scans, and the cost lands even when there is nothing to do: both work scans sort the entire eligible catalog by stars before LIMIT, so an idle hour pays for a full sort and returns zero rows.

The submit task now counts the missing work per kind first and only runs the ordered scan when the count is nonzero. The count keeps the CASE guard that skips trimming completed sources and has no sort step. Local fixture, 20k skills, fully idle:

== ordered select (the hourly scan being skipped) ==
  SCAN s
  CORRELATED SCALAR SUBQUERY 1
  SEARCH generated USING INDEX sqlite_autoindex_skill_generated_1 (owner=? AND repo=? AND name=? AND kind=?)
  SEARCH r USING INDEX sqlite_autoindex_repos_1 (owner=? AND repo=?)
  USE TEMP B-TREE FOR ORDER BY
== CASE-guarded count (the idle guard) ==
  SCAN s
  CORRELATED SCALAR SUBQUERY 1
  SEARCH generated USING INDEX sqlite_autoindex_skill_generated_1 (owner=? AND repo=? AND name=? AND kind=?)
  SEARCH r USING INDEX sqlite_autoindex_repos_1 (owner=? AND repo=?)

On the same fixture the count agreed with the scan in both directions: zero and empty on an idle catalog, one row each after adding a single unprocessed skill, so a zero count is a safe skip signal. Work that appears between the count and the scan waits for the next hourly run.

One thing I could not verify locally: a scoped typecheck of the scheduled task file. An ad-hoc project pulls both workers' Env declarations and reports unrelated binding errors, including on untouched lines, so that file is covered by CI's nuxt typecheck instead.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

…An idle hour sorted the whole eligible catalog per runtime kind before learning there was nothing to do. Count the missing work first and run the ordered scans only when a
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Oct 20, 2026
@harlan-github-agent

harlan-github-agent Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 READY · 93/100

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. A person still decides the merge.

  • Merge gate: Passed.
  • Review gate: Passed. No material issues.
  • CI gate: Passed.

@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. harlan-agent-ready The automated Review passed every gate on this head commit. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Oct 20, 2026

@harlan-github-agent harlan-github-agent Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harlan Agent Kit posted this automated review. AI open source policy.

Automated Review: READY.

Read the Review results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

harlan-agent-ready The automated Review passed every gate on this head commit.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hourly AI work scan reads the whole eligible catalog when idle

0 participants