fix(serverless): VolumeCache copies files on volumes that reject utime/chmod

[deanq]

Bug

VolumeCache._copy_file used shutil.copy2, which calls copystat (utime, then chmod) after copying the data. FUSE object stores such as Global Store reject both with EPERM, so the entire copy raised. The except OSError logged it at debug level and returned False.

As a result, every file of 256 KiB or more, which covers all model weights, silently dropped out of the mirror. The manifest correctly left them out, so each cold start re-downloaded the model and nothing ever reported an error. I found this while benchmarking VolumeCache on Global Store (deanq/flash-model-cache-bench, finding 4).

Fix

• Copy the data with shutil.copyfile, then carry the mtime (os.utime) and mode (shutil.copymode) over separately, each best-effort with a debug log.
• Large files are reconciled against the manifest's recorded source metadata (size, mtime) in both directions, not against the mirror file's own mtime. When the volume rejects utime, mirror files carry copy-time mtimes, and comparing those could skip a same-size update on sync or overwrite a newer container file on hydrate (review feedback).
  • Sync: copy when the mirror file is missing or a different size, or when the source differs from its prior manifest entry. With no prior entry, only an exact size+mtime match counts as current, so a lost manifest self-heals by recopying.
  • Hydrate: copy when the container file doesn't satisfy the manifest entry (_meta_satisfied_by_local), and stamp the manifest's source mtime on the restored file so the next sync is a no-op.
• Only the permission bits are lost, and only on a filesystem that refuses chmod.
• Real copy failures are now logged with log.warn rather than log.debug, so a cache that is silently not caching shows up in worker logs.

Tests

The new tests failed before the fix and pass after it:

• test_big_file_round_trips_when_volume_rejects_metadata_ops: sync, delete, hydrate with os.utime/os.chmod raising EPERM. Before the fix, 0 files were restored.
• test_sync_converges_when_volume_rejects_metadata_ops: a second sync re-copies nothing.
• test_copy_failure_is_logged_as_warning.

Two more tests:

• test_copy_file_preserves_mtime_and_mode guards the normal-filesystem behavior (mtime and 0o700 preserved).
• The existing test_copy_file_returns_false_and_cleans_tmp_on_failure now patches copyfile.

Review regression tests, simulating a volume that rejects utime/chmod only under the volume path:

• test_sync_recopies_same_size_update_when_volume_rejects_utime (failed before)
• test_hydrate_keeps_newer_local_file_when_volume_rejects_utime (failed before)
• test_hydrate_restores_source_mtime_so_next_sync_is_noop (failed before)
• test_sync_recopies_when_mirror_file_is_missing (guard)

Full suite: uv run pytest, 1478 passed.

Not addressed

On Global Store, the benchmark also saw 7B hydrates come back about half-complete even with this copy fix. The cause wasn't identified, so this PR doesn't claim to fix it. Symlinks under the HF cache's snapshots/ aren't mirrored either, by design.

[Copilot]

🟡 Changes recommended

Copy-time timestamps can hide source updates and cause hydration to restore stale bytes or overwrite newer local data.

1 open finding

• Use manifest source metadata for reliable mtime sync and hydration · New

What changed in this PR

Updates VolumeCache to copy files on volumes that reject timestamp or permission changes.

Changes:

• Separates data copying from best-effort metadata preservation.
• Logs copy failures as warnings.
• Adds metadata-rejection, convergence, and preservation tests.

File	Description
tests/​test_serverless/​test_utils/​test_rp_volume_cache.py	Tests metadata failures, round trips, convergence, and warning logs.
runpod/​serverless/​utils/​rp_volume_cache.py	Makes metadata copying best-effort and raises copy-failure log visibility.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[promptless]

Promptless documentation updates

• Note VolumeCache copy warnings and permission limits updates the Volume cache page to say that failed copies log warnings, and that cached files lose their permission bits on volumes that reject chmod.