fix(sidecar): serve long-form audio in its own lane, with per-model batch cost caps

[ShaiShmuel]

Summary

1. Before fix(sidecar): bound costly request waiting #708, the sidecar's batch former packed pending work cheapest first, with no aging (Whisper: the longest recordings wait minutes, and short clips wait for long-form audio in the same batch #585).
   • Under load a long recording kept losing to shorter requests and could time out.
   • fix(sidecar): bound costly request waiting #708 now moves the oldest request that has waited past the batch window to the front.
   • A clip batched with a long recording still waits for all of it.
2. fix(batcher): serve long-form audio in its own lane, alternating with clips #601 added a runs-alone lane to the Python batcher. This PR ports it to the sidecar.
3. On its own, the lane starved clips, because every sidecar model shared one cost cap of 16384. The Python worker uses each model's max_batch_tokens instead, which is 720000 ms of audio for Whisper large-v3-turbo and Parakeet-TDT.
4. The engine now reports each model's max_batch_tokens when the model is ready. The sidecar sizes that model's cost cap and adaptive range with the Python worker's formula.
5. At the 180000 ms adaptive floor, a clip turn could still leave a long clip over. Later turns sorted it behind newer, cheaper clips, so it was never served. Main before fix(sidecar): bound costly request waiting #708 never served it either. A clip turn now owes the clips that earlier turns left over.

Net effect:

• Long recordings stop timing out.
• While long-form audio is pending, every clip is served within a bounded number of turns, at any cost cap.
• Models on the default 16384 budget keep today's caps. That is 232 of 268 encode, score and extract profiles.

Results

These runs used one L4, with the sidecar in front of sie-server, comparing main d680237 with this branch. Both sides predate #708, and the runs were not repeated after the merge. Whisper large-v3-turbo unless noted.

load	main	this PR
8 long-form clients	5 timeouts, max 231 s	0 timeouts, max 95 s
16 clip + 4 long-form clients	0 recordings served, 4 timed out	22 served, 0 timed out
same load, clips	median 1.3 s, 2 timeouts	median 25.7 s, 0 timeouts
Parakeet-TDT, 16 clip clients	66 audio-s/s	158 audio-s/s

• The clip median under mixed load is the price of fairness.
  • Main at d680237 reached 1.3 s only by never serving the recordings.
  • The Python batcher with fix(batcher): serve long-form audio in its own lane, alternating with clips #601 gives 23.9 s on the same load.
  • Most of the remaining gap is pipeline depth 2, which this PR does not change. With depth 1 the median is 17 s.
• While a recording is pending, the merge with fix(sidecar): bound costly request waiting #708 forms the same batches as before. A property test checks this. The clip-only Parakeet row now also gets fix(sidecar): bound costly request waiting #708's promotion, and was not re-measured.
• The left-over fix did not move the live numbers. In those sessions no turn left a clip over, because the cap stayed at 720000, so the tests below are what cover it. There were 0 request errors and 0 OOMs, and no request over 30 s shared a batch.
• Overload trade-off, from a closed-loop simulation:
  • At 180000 ms, the worst clip wait drops from up to 1791 s to 30-61 s.
  • At 65536 ms with 48 clip clients, the median rises and 24-35% fewer clips beat a 120 s client timeout.
  • With the defaults, a speech model's cap adapts between 180000 and 2880000 ms. It reaches 65536 ms only in these cases:
    • SIE_BATCHER_MAX_BATCH_COST=65536, which sets every model's budget.
    • SIE_ADAPTIVE_BATCH_MAX_COST below 180000, which holds the cap there from the first batch.
    • SIE_ADAPTIVE_BATCH_MIN_COST at or below 65536, which lets the cap adapt down under load.
    • A backend that reports no budget, which starts at 16384 and adapts between 4096 and 65536.

Changes

• Runs-alone lane (scheduler/batch_former.rs, item.rs, audio_prep.rs):
  • Extract audio over 30 s runs one per batch, in arrival order, alternating with clip turns.
  • A clip turn serves the clips pending when it began, cheapest first, in batches of at most 12.
• Left-over clips:
  • A turn always owes the oldest left-over clip, plus more left-over clips, oldest first, while they fit the cap together.
  • Bound: a clip is served within the first n + 1 turns after it arrives. Here n counts it and the older clips still pending when the first of those turns ended. Later arrivals never add to n.
• With fix(sidecar): bound costly request waiting #708:
  • fix(sidecar): bound costly request waiting #708's aged promotion runs in order_for_next_batch, and applies only outside clip turns while no long recording is pending.
  • Inside a turn, what the turn owes already bounds each clip's wait. Promoting a recording or a late clip there would break the lanes.
  • Once no long recording is pending and the turn after the last one has ended, packing is main's: fix(sidecar): bound costly request waiting #708's aged promotion, then cost order.
• Per-model budget:
  • EnsureModelReadyResponse gains an optional max_batch_tokens. The Python server fills it from the worker config, and the Candle worker from the routed profile. Mixed versions keep today's behaviour.
  • The cap and adaptive range follow the budget b: floor min(max(256, b/4), b), ceiling 4b, start b.
  • Precedence: SIE_BATCHER_MAX_BATCH_COST when set, then the model's budget, then 16384.
  • SIE_ADAPTIVE_BATCH_* still override every model's range, now from its first batch. The count cap of 12 and the 720000 ms audio cap are unchanged.
• Docs: docs/architecture-guide.md and the batch former's module docs.
• Merge with main:
  • Merged main at 8ec90e1, which includes perf(sidecar): send engine-batched extract items one per RunBatch, as wide as the adapter dispatches #685 and fix(sidecar): bound costly request waiting #708.
  • fix(sidecar): bound costly request waiting #708's new test backend gains max_batch_tokens: None.
  • Whisper and Parakeet report dispatch width 1, so perf(sidecar): send engine-batched extract items one per RunBatch, as wide as the adapter dispatches #685 leaves them on this lane.

Tests

• Sidecar, 95e06bd: fmt and clippy (-D warnings) are clean.
  • The lib has 728 passed and 2 ignored, and the bin has 14 passed.
  • The integration suites pass: byte_identity, cli_help, integration_smoke, local_ingest_loopback, nats_auth and readyz_http.
• New tests:
  • The 180000 ms review case: the 30 s clip went from 0 serves in 300 batches to one every 2 turns.
  • A randomized property test over 24 seeds, each run with and without aging. It checks per-batch caps, cost order and every clip's wait bound, with caps between 4096 and 720000. While a recording stays pending, the aged runs form the same batches as the unaged ones.
  • Closed-loop latency tests at 180000, 16384 and 4096 ms.
  • With fix(sidecar): bound costly request waiting #708:
    • A recording that has waited longest still waits for the clip turn under way.
    • The turn after the last recording still serves its clips cheapest first, and the oldest clip goes first once that turn ends.
    • Outside the lanes, batches match main's packing with and without aging, over 20 seeds.
    • Removing the guard fails these tests.
• Candle worker and Python IPC tests: 200 passed and 264 passed at 469a4cb. Nothing they cover changed since.

For review

• Per-model caps are on by default, for Python parity, and operator overrides still win.
• Pipeline depth stays 2, including Helm's explicit setting. Per-model depth is a separate follow-up.
• Is a higher clip median under overload the right price for a bounded worst case?

Refs #585, #601, #708.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Team
• Run ID: d108bd1e-bbf8-41c1-a49c-a76208bd7d15

📥 Commits

Reviewing files that changed from the base of the PR and between 469a4cb and 95e06bd.

📒 Files selected for processing (8)

• packages/sie_server/src/sie_server/_ipc_test_harness.py
• packages/sie_server/src/sie_server/ipc_server.py
• packages/sie_server/src/sie_server/queue_executor.py
• packages/sie_server/tests/test_ipc_server.py
• packages/sie_server/tests/test_queue_executor.py
• packages/sie_server_sidecar/docs/architecture-guide.md
• packages/sie_server_sidecar/src/dispatcher.rs
• packages/sie_server_sidecar/src/scheduler/batch_former.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• packages/sie_server_sidecar/docs/architecture-guide.md

Included review availability: This review used your included allowance. 9 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

Walkthrough

The change adds optional model batch-cost budgets to readiness responses and uses them to configure sidecar schedulers. It also changes prepared-audio cost calculation and batch formation so long recordings run alone while recording and batchable lanes take alternating turns.

Changes

Model budgets and sidecar scheduling

Layer / File(s)	Summary
Readiness budget reporting and propagation packages/sie_server/src/sie_server/*, packages/sie_server/tests/*, packages/sie_server_rust/src/*, packages/sie_server_sidecar/src/protocol/*, packages/sie_server_sidecar/src/dispatcher.rs, packages/sie_server_sidecar/src/backend/router.rs, packages/sie_server_sidecar/docs/architecture-guide.md	Python and Rust readiness responses now include an optional max_batch_tokens value. The sidecar dispatcher passes the reported value to scheduler resolution. Tests cover ready, absent, and older wire responses.
Per-model scheduler budgets and updates packages/sie_server_sidecar/src/scheduler/{batch_config,adaptive,engine,registry}.rs, packages/sie_server_sidecar/src/lib.rs, packages/sie_server_sidecar/src/dispatcher.rs, packages/sie_server_sidecar/docs/architecture-guide.md	The registry uses a positive reported model budget unless a valid environment override pins the cost. Schedulers derive adaptive ranges from the selected budget and adopt changed reported budgets. Environment lookup is injectable.
Prepared long-recording turns packages/sie_server_sidecar/src/audio_prep.rs, packages/sie_server_sidecar/src/scheduler/{item,batch_former,registry}.rs, packages/sie_server_sidecar/docs/architecture-guide.md	Prepared extract audio uses duration-based cost. Audio above the 30-second sample threshold runs alone. Batch formation alternates runs-alone and batchable lanes, tracks turn membership, and prioritizes carried-over requests.

Sequence Diagram(s)

sequenceDiagram
  participant CandleBackend
  participant Dispatcher
  participant ProductionSchedulerRegistry
  participant Scheduler
  CandleBackend->>Dispatcher: readiness response with max_batch_tokens
  Dispatcher->>ProductionSchedulerRegistry: resolve scheduler with model budget
  ProductionSchedulerRegistry->>Scheduler: create or adopt cost budget

Loading

Suggested reviewers: krisztian-gajdar

Priority: ⬆️ High

Merge Risk: ⚪ Minimal · up to 95e06

Models can now report their own batch budgets, and long audio recordings are scheduled in their own turns alongside short clips. When a model reports no budget, the previous behavior is preserved. No concrete defect was identified, so the change looks ready to merge.

Pre-merge checks · 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 67.98% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 178 functions across 19 files. (1 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the two primary changes: a dedicated lane for long-form audio and per-model batch cost caps.

Full details: Docstring Coverage

Explanation

Docstring coverage is 67.98% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 178 functions across 19 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[svonava]

Heads-up on a sidecar change I'm about to open nearby. No action needed here; it is meant to compose with this PR.

Problem. On the sidecar path, SGLang OCR extract (LightOnOCR-3-4B, GLM-OCR) costs up to twice as much as on the worker's direct path. The scheduler packs up to 12 pages into one RunBatch, a batch answers only when its slowest page finishes, and pipeline depth 2 plus the 4-slot IPC pool cap the pages in flight. #675 fixed the same thing on the direct path.

Planned change:

• Python reports the worker's dispatch width as an optional max_concurrent_dispatch on ModelDescriptor. It goes on the descriptor, not on EnsureModelReadyResponse, so it does not touch your max_batch_tokens field.
• In dispatcher.rs, for a model whose width is above 1:
  • the drain loop sends each formed extract batch as single-item RunBatch calls;
  • the model's pipeline depth becomes max(depth, width).
• ipc_client.rs grows the slot pool by that width.
• Nothing under scheduler/ changes (batch_former, registry, engine, adaptive, item, batch_config).

Overlap with this PR. Only dispatcher.rs is shared. My hunks are in the drain loop and the readiness handling, not in resolve_scheduler, and the test-only EnsureModelReadyResponse literals don't change.

After both land. The split could move onto your runs-alone lane: an extract item of an engine-batched model would report runs_alone(). That needs the item to know its model's width, so I've kept the split in the dispatcher for now.

[huronat]

Reviewed 469a4cb. No blocking issue found.

The carry-over reservation addresses the previously reported starvation case at the 180000 audio-cost floor: over 300 extracted batches, the 30-second clip is now served 50 times, with short clips and recordings continuing to make progress. Independently ran the sidecar library suite: 721 passed, 2 ignored. Required CI is green.

I support per-model batch cost budgets by default, retaining explicit operator overrides and the existing safety caps. Keep configurable per-model/profile pipeline depth as a separate follow-up; this PR leaves depth 2 unchanged, including Helm's explicit setting. When composing this with #708, preserve the clip-turn reservation and validate the combined scheduler behavior.

Non-blocking documentation correction: the low-cap overload trade-off can also be reached through SIE_ADAPTIVE_BATCH_MAX_COST, even when the backend reports a 720000 budget. GPU performance results were not independently rerun in this review.

Codex review, posted at Attila's request.

[ShaiShmuel]

@huronat #708 landed first, so this branch now merges main at 8ec90e1 and composes the two. The push dismissed your approval. Here is what changed since 469a4cb:

• c8aa483 is the merge. fix(sidecar): bound costly request waiting #708's aged promotion moves into order_for_next_batch and applies only outside clip turns, while no long recording is pending. Inside a turn, the clip-turn reservation is unchanged. fix(sidecar): bound costly request waiting #708's test backend gains max_batch_tokens: None.
• 6933300 adds tests for both sides of that line. Both lane property tests now run with aging on and off. While a recording stays pending, the aged runs form the same batches as before. Removing the guard fails these tests.
• 95e06bd updates the docs, including your SIE_ADAPTIVE_BATCH_MAX_COST note.

We also ran a closed-loop simulation of the combined behaviour, which is not committed:

• It found no lane or reservation violation in any load.
• With recordings always pending, the batches are identical to 469a4cb's.
• Between turns, fix(sidecar): bound costly request waiting #708 now bounds the wait of a costly clip that cost order alone kept passing over.