LB: per-rule keepalive and timeouts, plus three HAProxy config fixes

[bhouse-nexthop]

Description

#12586 made the load balancer idle timeout globally configurable, as
network.loadbalancer.haproxy.idle.timeout. This expands that to per load balancer rule, and
does the same for keepalive, which until now could only be set on the network offering.

Where these settings live today:

setting	scope	set where	reaches haproxy as
keepalive	network offering	keepaliveenabled, create-time only, no UI	option httpclose in defaults
idle timeout	global	network.loadbalancer.haproxy.idle.timeout (#12574, #12586)	timeout client, timeout server in defaults

Both land in the defaults section, so they apply to every LB rule on every network using that
offering. One rule cannot differ from another on the same router.

New parameters on createLoadBalancerRule and updateLoadBalancerRule

Each is optional, and each writes into that rule's own listen section, which overrides defaults:

parameter	type	haproxy directive	applies to	unset means
keepalive	boolean	option http-keep-alive, else option httpclose	rules served in http mode	the offering's keepaliveenabled
idletimeout	long, ms	timeout client and timeout server	every rule, tcp and http	the global from #12586
keepalivetimeout	long, ms	timeout http-keep-alive	http mode, keepalive on	idletimeout applies

idletimeout keeps the global's meaning, including 0 for infinite.

What that produces, taken from the generator - an http rule with all three set, a tcp rule with
only an idle timeout, and a rule that set nothing:

listen 10_2_0_2-80
	bind 10.2.0.2:80
	mode http
	option http-keep-alive
	timeout http-keep-alive 15000
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_2-80_0 10.1.10.3:80 check

listen 10_2_0_3-3306
	bind 10.2.0.3:3306
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_3-3306_0 10.1.10.4:3306 check

listen 10_2_0_4-8080
	bind 10.2.0.4:8080
	balance roundrobin
	server 10_2_0_4-8080_0 10.1.10.5:8080 check

The three values are returned on listLoadBalancerRules, and are settable on the load balancer
rule's edit form in the UI, where blank reads as "Use default".

A negative timeout is rejected by the API. Haproxy treats one as a fatal parse error, which would
leave every rule on that router serving its previous config, so the generator drops it as well.

Stored as firewall_rule_details. No schema change.

Applies to public LB rules. Application load balancers are created through a different command and
are not covered.

What this makes possible

A rule on port 80 can now have keep-alive and X-Forwarded-For. It could not before:

keepalive	mode	connection reuse	XFF
offering flag off (the default)	mode http	no, Connection: close per response	yes
offering flag on	mode tcp	yes	no, forwardfor needs http mode
per rule, this PR	mode http	yes	yes

A rule that sets nothing keeps today's behaviour exactly, including the fall back to mode tcp
when the offering flag is on.

Also fixes three bugs in the same file

1. option forceclose is not a valid keyword

With keepalive enabled, a rule that also uses HTTP-based stickiness or SSL offload emits
no option forceclose. HAProxy has rejected that keyword since 2.0, and the system VM is Debian 12
(HAProxy 2.6):

[ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not supported any more
        since HAProxy 2.0, please just remove it, or use 'option httpclose' if absolutely
        needed.
[ALERT] config : Fatal errors found in configuration.

The whole file is rejected, so haproxy keeps serving the previous config and the rule silently
never takes effect. Replaced with option http-keep-alive.

2. One rule's settings leak into every rule generated afterwards

globalSection and defaultsSection are static String[]. The generator wraps them in
Arrays.asList(...) and calls set(), which writes through to the arrays, so the value
survives the call:

set once by	leaks as
keepalive on any rule	no option httpclose in the defaults for every later rule, any network, any account
idleTimeout > 0	that timeout becomes the default for the next rule
idleTimeout == 0	timeouts blanked permanently

The generator now copies into a fresh list, and the sections themselves became List.of(...), so a
future set() throws instead of silently corrupting every config the agent builds afterwards.
Output is unchanged for a single rule.

3. appsession was removed from HAProxy in 1.6

An LB rule with AppCookie stickiness emitted appsession, which the shipped haproxy refuses:

[ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6.
[ALERT] config : Fatal errors found in configuration.

Same blast radius as bug 1 - the file is rejected, so every rule on that router keeps serving its
previous config. The VR advertises AppCookie as a supported stickiness method, so this is
reachable from the API.

Replaced with the documented stick table equivalent:

appsession	replacement
len <n>	stick-table type string len <n> size 10k
timeout <h>	expire <h>
the cookie	stick store-response res.cook(<name>) + stick match req.cook(<name>)
request-learn	stick store-request req.cook(<name>)
prefix, mode	no equivalent, logged and ignored

prefix and mode were never applied - the directive carrying them was rejected - so no working
behaviour changes.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

HAProxyConfiguratorTest Tests run: 17, Failures: 0 and LoadBalancingRulesManagerImplTest
Tests run: 16, Failures: 0.

Each bug fix has a case that fails without it:

• keepalive with SSL offload emits no forceclose, and does emit option http-keep-alive
• a keepalive config followed by a non-keepalive one leaves no no option httpclose behind
• an idle timeout followed by an unset one falls back to the built-in default
• AppCookie stickiness emits a stick table and never appsession

The rest cover the per-rule settings:

• a rule that sets keepalive stays in http mode
• a rule's keepalive wins over the offering, in both directions
• a rule that sets nothing is byte for byte what it was before
• timeout http-keep-alive is emitted, and only when keepalive is on
• idletimeout overrides per rule on a tcp rule, leaving the defaults section alone
• a negative timeout never reaches the config, and is rejected at the API
• two rules in one config do not pick up each other's settings

Generated configs are checked against a real haproxy -c, which is where the alert text above
comes from. A file holding an AppCookie rule, an http rule with all three settings, a tcp rule
with a timeout, and a rule with a negative timeout is fatal before these changes and valid
after - the three alerts above become zero.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.70%. Comparing base (4ce5291) to head (37801b9).
⚠️ Report is 21 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##            main   #14131      +/-   ##
=========================================
- Coverage   3.71%    3.70%   -0.01%     
=========================================
  Files        487      487              
  Lines      41992    42001       +9     
  Branches    7942     7944       +2     
=========================================
  Hits        1558     1558              
- Misses     40208    40217       +9     
  Partials     226      226              

Flag	Coverage Δ
uitests	3.70% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

@bhouse-nexthop
could you make some changes for VR health check ?
similar to #12596 which fixes #12591

[bhouse-nexthop]

@weizhouapache good catch, thanks - added in the latest commit.

haproxy_check.py knew about maxconn and the global idle timeout, so the new per rule settings
could drift with nothing noticing. The health check data now carries all three, and the check
validates them in that rule's own listen section:

data	checked
ruleKeepAlive	option http-keep-alive, or option httpclose
ruleIdleTimeout	timeout client, timeout server
ruleKeepAliveTimeout	timeout http-keep-alive

The http mode test needed widening as well, otherwise a rule that sets keepalive itself stopped
being checked at all:

before   port 80 and the offering has keepalive off
after    port 80 and (the rule sets keepalive, or the offering has it off)

Compatibility: an older management server sends none of these keys, and a rule that inherits sends
them empty. Both read as nothing to check, so an updated system VM template against an older
management server still passes.

I ran the script against generated configs rather than only reading it - a matching config passes,
and each of a wrong keepalive option, a drifted timeout client and a missing
timeout http-keep-alive fails.