optimise prometheus scraping workflow

[DaanHoogland]

Description

This PR...

Fixes: #13586

1. Removes alertManager.recalculateCapacity() from PrometheusExporterImpl.updateMetrics(). This is the actual fix — it stops the exporter from forcing a full, thread-pool-churning, all-zone capacity recalculation on every single scrape. The exporter should just read CapacityDao, which AlertManagerImpl's own timer already keeps current. This alone should flatten the scrape_duration curve.
2. Reuses a single long-lived ExecutorService in AlertManagerImpl instead of creating/tearing one down on every recalculateHostCapacities()/recalculateStorageCapacities() call. Cheap, low-risk, and removes the thread-churn contributor even for the timer-driven path.

this should solve the issue, but claude suggested the following improvements as well:

3. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
4. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
5. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

if anyone wants to, these are nice to haves/good first issue ;)

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 16.47%. Comparing base (6136ac9) to head (f751359).
⚠️ Report is 22 commits behind head on 4.20.

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #13650      +/-   ##
============================================
+ Coverage     16.32%   16.47%   +0.15%     
- Complexity    13556    13743     +187     
============================================
  Files          5669     5669              
  Lines        501399   505527    +4128     
  Branches      60902    62616    +1714     
============================================
+ Hits          81847    83309    +1462     
- Misses       410390   412934    +2544     
- Partials       9162     9284     +122     

Flag	Coverage Δ
uitests	4.21% <ø> (+0.06%)	⬆️
unittests	17.35% <100.00%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
77.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[blueorangutan]

[SF] Trillian test result (tid-16835)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51997 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13650-t16835-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@NuxRo please test? @Damans227 and @shwstppr , can you review?

[Damans227]

what happens if the workers setting changes while two capacity runs overlap? looks like one can hand work to the pool that just got shut down

[Damans227]

thanks @DaanHoogland, resizing the pool in place covers the overlap case. LGTM

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
76.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud