Skip to content

fix(native-sidecar): return freed memory to the OS after VM disposal - #2074

Open
eersnington wants to merge 1 commit into
mainfrom
fix/native-heap-release
Open

eersnington wants to merge 1 commit into
mainfrom
fix/native-heap-release

Conversation

@eersnington

@eersnington eersnington commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Fixes #1902.

The sidecar's memory only grew as VMs came and went: glibc keeps pages freed by a disposed VM's V8 isolates mapped in its arenas.

  • After VM disposal, the sidecar runs one coalesced malloc_trim(0) per second on the existing blocking executor, off the hot path and without holding locks.
  • Linux with glibc only. Other targets compile it as a no-op.
  • resource-limits.mdx documents sidecar memory and the warm isolate pool cost (about 20 MiB per isolate, AGENTOS_V8_WARM_ISOLATES).
One shared sidecar main This PR
RSS after 300 VMs (8 concurrent; JS, Python, shell each) ~2,080 MiB ~481 MiB
RSS after 40 one-shot Python VMs 346 MiB ~190 MiB
Guest loopback read of 10 MB ~0.45 s ~0.45 s
Allocation-heavy guest JS loop 40 ms 40 ms
Probe evaluate p99 while other VMs churn unchanged unchanged

An earlier revision also pinned glibc's M_MMAP_THRESHOLD at 128 KiB. It made allocation-heavy guest code and loopback TCP 3 to 6 times slower, for little extra memory, so this revision removes it.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

📟 Pip-Boy · ea97e53

📦 Packages

Skipped: this PR doesn't change any published package.

🧹 Rust dependencies · cargo deny + cargo shear

check new in this PR fixed existing on base
advisories ✅ 0 0 25
licenses ✅ 0 0 0
sources ✅ 0 0 0
bans ✅ 0 0 0
unused ✅ 0 0 16

🦀 Binaries · publish Dockerfiles, --release

Base: main@9413c84, the merge base.

artifact npm @latest base this PR Δ vs base
agentos-native-sidecar darwin-arm64 120.1 MB 120.1 MB 120.1 MB +912 B (+0.0%)
agentos-native-sidecar darwin-x64 129.6 MB 129.6 MB 129.6 MB −2.8 KB (−0.0%)
agentos-native-sidecar linux-arm64-gnu 140.8 MB 140.8 MB 140.8 MB −7.9 KB (−0.0%)
agentos-native-sidecar linux-x64-gnu 143.6 MB 143.7 MB 143.8 MB +50.0 KB (+0.0%)

⏱ Timing

job wall time slowest step
changes 31s checkout · 28s
zizmor 9s checkout · 2s
hygiene 1m04s checkout · 31s
native · linux-x64-gnu 14m04s Build with the publish Dockerfile · 13m35s
native · darwin-arm64 9m00s Build with the publish Dockerfile · 8m44s
native · darwin-x64 10m36s Build with the publish Dockerfile · 10m14s
native · linux-arm64-gnu 10m14s Build with the publish Dockerfile · 9m55s
⏭ self-test — skipped
⏭ footprint — skipped
report 22s setup-node · 6s
🔭 Full CI trace for this commit

One OpenTelemetry trace of CI and Pip-Boy for ea97e53: every job and step, with the sizes above attached. Run after CI finishes.

pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha ea97e53fb1ffee2c9affdbaa0e1ef7618a2c33ff

Add --otlp <url> to send it to any OTLP/HTTP collector.

Agent instruction: From a rivet-dev/agentos checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.

glibc keeps pages freed by a disposed VM's V8 isolates mapped in its arenas, so a long-lived sidecar's RSS only grows. After VM disposal the sidecar now runs one coalesced malloc_trim(0) per second on the shared blocking executor, off the hot path. Linux with glibc only; other targets compile it as a no-op.
@eersnington
eersnington force-pushed the fix/native-heap-release branch from e348728 to ea97e53 Compare October 21, 2026 16:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One-shot Pyodide executions retain a high sidecar memory watermark after VM disposal

1 participant