Skip to content

perf: cache repository activity summaries - #1801

Open
dcoric wants to merge 11 commits into
finos:mainfrom
dcoric:perf/repository-activity-summaries
Open

dcoric wants to merge 11 commits into
finos:mainfrom
dcoric:perf/repository-activity-summaries

Conversation

@dcoric

@dcoric dcoric commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Repository pages currently scan push history to calculate counts and recent activity. Maintain summaries for the file, MongoDB, and PostgreSQL sinks instead, preserving status precedence, canonical URL matching, and latest push/pending timestamps.

File summaries update in memory. MongoDB persists revisions and retries interrupted refreshes. PostgreSQL migration 9 records audit changes transactionally and refreshes the affected summaries. Existing audits are preserved, and deployment documentation covers backfill and rebuilding derived data.

Startup migration 20261013-repository-activity backfills summaries before either server accepts requests. MongoDB checks for duplicate push IDs without deleting audits, and concurrent dashboard reads share one refresh per process while revision checks protect against stale work from other instances. Rebuilding is available through the shared database API for all three sinks.

Review actions also invalidate the relevant UI caches, and a shared helper handles numeric and serialized approval/rejection timestamps.

Related Issue

Activity portion of #1691 and PostgreSQL parity in #1794, following the merge of #1687. Audit storage and list/detail payload changes are isolated in #1795. Both PRs are needed to complete the agreed scope. The Action.diff work in #1693 and #1701 remains separate.

Validation

  • 1,698 unit tests passed; 90.86% line coverage.
  • 85 MongoDB and 89 PostgreSQL integration tests passed, including duplicate-ID migration recovery, concurrent readers, stale refresh protection, and shared rebuild behavior.
  • Six real Git push/fetch scenarios and all 18 current Cypress tests passed against PostgreSQL.
  • Manually checked approval/rejection timestamps, full diffs, and immediate count updates after rejection. A real MongoDB startup with duplicate IDs failed before either server listened, preserved both audits, and succeeded after reconciliation.
  • Verified the documented rebuild command using the installed package API.
  • Production build, TypeScript, formatting, and lint passed, with the existing tagPush Cypress warning.
  • Combined with perf: reduce duplicate push audit storage #1795 locally: 729 unit/service/processor tests, 95 MongoDB integration tests, 101 PostgreSQL integration tests, six Git push/fetch scenarios, and 18 Cypress tests passed. The sink adapter overlap has been resolved and verified locally for when the second PR is updated after the first merges.

On a local synthetic PostgreSQL history of 10,000 pushes across 10 repositories, five-run median warm reads were 2.44 ms for summaries versus 11.27 ms for the previous history query. Initial backfill took 178.51 ms. These are workload examples, not general performance guarantees.

Checklist

  • CONTRIBUTING.md reviewed; Conventional Commits used
  • FINOS CLA on file
  • Tests and deployment documentation updated
  • Unit/integration tests, lint, formatting, type checks, and build passed

No configuration schema changes.

dcoric added 5 commits October 6, 2026 12:23
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
@dcoric
dcoric requested a review from a team as a code owner October 6, 2026 10:30
@netlify

netlify Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for endearing-brigadeiros-63f9d0 ready!

Name Link
🔨 Latest commit 739bef8
🔍 Latest deploy log https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/6ae992d96a90a82c3d3c32ea
😎 Deploy Preview https://deploy-preview-1801.git-proxy.preview.finos.org
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.85057% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.62%. Comparing base (23dbf00) to head (739bef8).

Files with missing lines Patch % Lines
src/db/mongo/repoActivity.ts 97.00% 0 Missing and 3 partials ⚠️
src/db/repoActivity.ts 98.27% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1801      +/-   ##
==========================================
+ Coverage   87.05%   88.62%   +1.57%     
==========================================
  Files         113      123      +10     
  Lines        6482     6876     +394     
  Branches     1260     1305      +45     
==========================================
+ Hits         5643     6094     +451     
+ Misses        536      487      -49     
+ Partials      303      295       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
Signed-off-by: Denis Ćorić <denis85@gmail.com>
@kriswest kriswest mentioned this pull request Oct 20, 2026
14 of 21 tasks
Refresh against finos/main while retaining the repository activity summaries
and current push diff/list behavior. Add shared sink regression coverage for
detail payload preservation and list projections across review actions and
activity rebuilds.

Signed-off-by: Denis Ćorić <denis85@gmail.com>
@dcoric

dcoric commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@kriswest, a bit of context: this PR covers the repository activity optimisation from #1691. I split it out of #1795 following Juan’s suggestion to review it separately from the audit-storage changes. #1795 reduces duplicate audit data, while this PR maintains activity summaries so repository-page loads don’t need to recalculate them from the full push history.

@kriswest kriswest added this to the 2.4.0 milestone Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants