Skip to content

FOUR-33794 Fix Ellucian clients are reporting slow performance - #9106

Open
caleeli wants to merge 3 commits into
developfrom
FOUR-33794
Open

caleeli wants to merge 3 commits into
developfrom
FOUR-33794

Conversation

@caleeli

@caleeli caleeli commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Issue & Reproduction Steps

Scheduled metrics:clear ran simultaneously on every pod and tenant, spiking shared Redis.

Related Tickets & Packages

@caleeli
caleeli requested a review from nolanpro October 16, 2026 19:18
@cursor

cursor Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Operational behavior changes (metrics off by default, less frequent clears) affect observability and shared Redis; misconfiguration could hide production metrics until METRICS_ENABLED is set.

Overview
Addresses Redis spikes from metrics:clear running in lockstep across pods/tenants by making Prometheus collection opt-in and softening how clears are scheduled.

METRICS_ENABLED (default false) now controls whether metrics run at all: MetricsService lazy-initializes the registry, no-ops common write paths when disabled, and throws MetricsDisabledException on direct registry access; cache metrics wrapping in AbstractCacheFactory follows the same flag. Scheduled metrics:clear is skipped when metrics are off, moved into scheduleMetricsClear, and uses --stagger (random delay up to METRICS_CLEAR_STAGGER_MAX_SECONDS) with runInBackground / withoutOverlapping. The default clear interval changes from 10 minutes to daily (1440 minutes).

Tests enable metrics explicitly in setup and add coverage for the disabled path.

Reviewed by Cursor Bugbot for commit d6fb165. Bugbot is set up for automated code reviews on this repo. Configure here.

@decisions-sonarqube

Copy link
Copy Markdown

@caleeli

caleeli commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Default value changed from 10minutos to 24hours

METRICS_CLEAR_INTERVAL=1440

enables or disables metrics

METRICS_ENABLED=false

Max random delay (seconds) before run clear metrics

METRICS_CLEAR_STAGGER_MAX_SECONDS=60

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d6fb165. Configure here.


// 5 minutes is recommended in https://laravel.com/docs/12.x/horizon#metrics
$schedule->command('horizon:snapshot')->everyFiveMinutes();
$event->runInBackground()->withoutOverlapping($clearInterval);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stagger overlaps short clear intervals

Low Severity

withoutOverlapping expires after clear_interval minutes while --stagger may sleep for the full interval. With METRICS_CLEAR_INTERVAL=1 (or 2) and the default 60s stagger cap, the mutex can expire while the first metrics:clear is still sleeping or wiping, so the next schedule starts a second clear against the same Redis keys.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit d6fb165. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant