Repository navigation
Conversation
…achable put() used to call connect() on any send failure, which retries 10 times with a 30 s wait and a 5 s sleep: a single message could block its caller for up to ~6 minutes, and concurrent callers tore down and re-created the shared connection at the same time. put() now makes at most one short reconnection attempt, never waits for another thread that is already reconnecting, and backs off exponentially (up to 60 s) after a failed attempt. Connections are serialised by a lock and connect() is a no-op when the connection is already up.
MessageQueueHandler.emit() called the MQ producer synchronously in the thread doing the logging. When the broker became unreachable, every log call in a service could block for minutes in reconnection attempts, so request-handling threads stalled and clients timed out. The connector's own error messages also went back through the same handler. emit() now only formats the record and puts it in a bounded queue; a daemon thread creates the producer and sends the records, backing off after failures. Records are dropped (and counted, with a periodic note on stderr) when the queue is full or a record cannot be sent, records logged by the sender thread itself are not queued, and the sender is restarted in forked children. close() flushes the queue for up to 5 seconds.
MatchingDelay was checked before matching but only armed after the match, so concurrent requests could all pass the check and start many jobs within one delay. It is now a token bucket per (site, attribute, value): the Matcher reserves a token for every value that has one before querying the TaskQueueDB, excludes the others, keeps the token of the matched job and gives the rest back. Concurrent requests can therefore never share a token. An optional MatchingDelay/<Site>/Burst/<Attribute>/<Value> lets a site start up to that many jobs at once after a quiet period while keeping the configured average rate; without it the behaviour is unchanged (at least <delay> seconds between two matches). The running limits use job counts cached for 10 seconds (up to 300 s if refreshing is slow), so a site could overshoot its limit by everything matched in that window. Jobs matched by the process since the counts were taken are now added to them. Subsections that are not job attributes (CEs under RunningLimit, Burst under MatchingDelay) are no longer reported as unknown attributes.
chrisburr
force-pushed
the
fix/mq-logging-and-matching-delay
branch
from
October 6, 2026 12:49
8a6694e to
cc8ce83
Compare
chrisburr
marked this pull request as ready for review
October 7, 2026 07:59
chrisburr
requested review from
andresailer,
atsareg and
fstagni
as code owners
October 7, 2026 07:59
Member
Author
|
This seems to be working extremely nicely:
Without the burst you're limited by the rate at which pilots start. And mapping it to IO load that we were trying to smooth out: 
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BEGINRELEASENOTES
*Resources
FIX: StompMQConnector.put fails fast (one short reconnection attempt, with backoff) instead of blocking for minutes when the broker is unreachable
*Framework
FIX: MessageQueueHandler sends log records from a background thread with a bounded queue, so logging never blocks when the broker is slow or unreachable
*WorkloadManagement
NEW: MatchingDelay is a token bucket, with an optional MatchingDelay//Burst// burst size
FIX: MatchingDelay tokens are reserved before matching, so concurrent requests can no longer exceed the configured rate
FIX: running limits count the jobs matched since the cached running counts were taken, so sites no longer overshoot them
FIX: CEs and Burst subsections are no longer reported as unknown job attributes
ENDRELEASENOTES