Repository navigation
docs(flash): load volume-backed models via VolumeCache - #899
Merged
Merged
Conversation
The Flash storage page showed loading weights straight from /runpod-volume, which reads them over the network on every cold start. Add a section that keeps the HF cache on container disk and wraps the load in VolumeCache, and point the best-practices bullet at it.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
There was a problem hiding this comment.
🟢 Approval recommended
The focused documentation changes align with existing VolumeCache guidance, with no unresolved findings.
0 open findings
What changed in this PR
Documents how Flash endpoints can use VolumeCache to load model weights from local disk rather than directly from network storage.
Changes:
- Adds a model-loading example with SDK requirements and cache caveats.
- Links the pattern from production best practices.
| File | Description |
|---|---|
| flash/configuration/storage.mdx | Adds the VolumeCache example and operational guidance. |
| flash/configuration/best-practices.mdx | Links to local-disk model-loading guidance. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
justinwlin
self-requested a review
October 15, 2026 16:29
justinwlin
approved these changes
Oct 15, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Flash storage page's model example loads the weights straight from
/runpod-volume/models/.... The best-practices page says to "use network volumes for large models". Neither mentionsVolumeCache, so every cold start reads the weights over the network mount.In a forced-cold benchmark on Flash (deanq/flash-model-cache-bench, RTX 4090, EU-RO-1, medians), loading in place from the volume was about 2.4× slower than hydrating with
VolumeCacheand loading locally. Qwen2.5-3B took 17.5s vs 7.2s, and 7B took 37.6s vs 15.5s.What changed
flash/configuration/storage.mdx: new Load large models from local disk section after the existing access example. It has a class-form@Endpointthat setsHF_HUB_CACHEto container disk, wraps the load inVolumeCache, and addsrunpod>=1.12.0todependencies. It also covers the gotchas from the VolumeCache page: errors fall back silently by default, symlinks aren't copied so Hub access must stay on, and the volume must be sized for the mirror.flash/configuration/best-practices.mdx: the network-volume bullet links to the new section.The existing access example stays, since reading arbitrary files from the volume is still valid.
Checks
validate-tooltips.jsandvalidate-redirect-sources.jspass.Model Store
For a Hugging Face model on a GPU endpoint, the guidance now points to Model Store (cached models) before
VolumeCache. It needs no volume or handler code, and the download isn't billed. Flash users attach the model afterflash deploywithrunpodctl serverless update <endpoint-id> --model-reference …and setHF_HUB_CACHE=/runpod-volume/huggingface-cache/hub.Model Store isn't framed as the default. In the forced-cold benchmark it had the fastest load once a worker was running (2.3s / 14.2s for 3B / 7B), but the slowest whole cold request (106s / 81s), because the worker waited 69–99s for a host that already held the model. So the guidance says to time the whole cold request, and to use the volume with
VolumeCachewhen that wait is too long.Verified live on 2026-10-14 (no jobs sent, so no workers ran): attached a model to a runpod-flash 1.20.1 endpoint with runpodctl 2.14.0, then ran
flash deploywith a changed env. GraphQLmodelReferencesstill held the reference afterward, and the env change landed.runpodctl serverless getdoesn't printmodelReferences; theupdateoutput does.Live check
Live-verified on 2026-10-14. I deployed the documented pattern with runpod-flash 1.20.1: a class
@EndpointwithHF_HUB_CACHE=/opt/hf-cache,runpod>=1.12.0independencies, and the load wrapped inVolumeCache. It ran on a 24 GB GPU in EU-RO-1 with a 20 GB volume and Qwen2.5-0.5B-Instruct. I forced a real cold start between the two calls, then deleted the endpoint and volume.runpodimportedfrom_pretrainedz2ev…/app/runpod(bundled)n479…/app/runpod(bundled)The
runpodthat Flash bundles fromdependenciestakes precedence over the image's SDK. The 147 B on the second cold start is the Hub metadata call that re-linkssnapshots/, which confirms the "keep Hub access on" rule.