[6.x] Improve term/entry stache warming performance

[edalzell]

On our largest site, stache refreshing took 7-8 mins, which is too close to Forge's 10 mins deployment limit.

I proposed a 2 pass approach to Claude to see if it was feasible and this code results in over a 2 min decrease on my machine, from ~7:30 to ~5:15.

[jasonvarga]

The two-pass warm approach is sound and the perf win is real, but I found two correctness regressions and a test gap that should be addressed before merge. Left inline comments on the specific spots.

No tests added. This touches core stache:warm behavior and adds a fast/cold-path split with non-trivial cache-key assumptions. A test asserting term/entry associations + value indexes match the old single-pass output — covering (a) a max_items: 1 taxonomy field and (b) an on-the-fly/fileless term — would have caught both inline issues below.

[edalzell]

Thanks, both were real. Rebased on 6.x and addressed:

• max_items: 1: Associations now reads the entries store's $handle, site and id indexes via index() and wraps scalar values. No more hand-built cache keys or separate cold path; outside warming the indexes build on demand, same cost as the old query.
• Terms\Value: pass 1 now only takes Value indexes that don't override getItems(). Everything else (incl. all Terms\Value subclasses) builds in pass 2 as before.

Added warm tests for a max_items: 1 field and an on-the-fly term; both failed before the fix.

[jasonvarga]

Thanks for the rebase — I've re-reviewed at 476d590. Both of my previous Critical findings are properly fixed, and the two new tests cover them well. Arr::wrap() handles the max_items: 1 shape, and the ReflectionMethod discriminator is a much better check than method_exists — I traced it through every index class in core and it classifies all of them correctly (Terms\Site in particular, which resolves to Terms\Value via same-namespace lookup, correctly stays in Pass 2). CI is green and there are no conflicts.

Four things left, all Warnings rather than blockers-in-principle — but I'd like them addressed before this goes in, since two of them are silent failures.

One broader note on the approach: the two-pass split is sound, and I confirmed the perf mechanism actually fires — CollectionEntriesStore::storeIndexes() already merges $collection->taxonomies()->map->handle(), so the taxonomy value indexes are always resolved and always built in Pass 1, even on a cold cache. So the ordering does what the docblocks claim.

Inline comments below.

Changes made

[edalzell]

Benchmarked this against #15639 (merged today) on two real sites, locally: full stache:warm --exclude=revisions,assets after cache:clear, file watcher off, 2 runs each. Zakat: 5302 entries, 4 sites, file cache. Passivehouse: 3086 entries, redis cache. "6x" is db5ff547c (includes #15639), "pre15639" is its parent.

site	variant	seconds	peak MB
passivehouse	pre15639	33.6, 33.9	569
passivehouse	pre15639 + this PR	33.2, 34.8	596
passivehouse	6x	21.0, 21.1	569
passivehouse	6x + this PR	23.7, 22.9	596
zakat	pre15639	44.5, 46.7	821
zakat	pre15639 + this PR	47.5, 53.7	797
zakat	6x	36.3, 36.8	821
zakat	6x + this PR	40.6, 41.0	796

#15639 is where the win is. This PR adds nothing on top of it and is slightly slower. A per-store breakdown (Passivehouse, pre15639) shows the terms store at 37.6s of a 39.7s warm: the associations index loads every entry, and that per-entry cost is what #15639 fixed.

Harness

stache-perf.sh symlinks vendor/statamic/cms to a checkout, times N warms, restores the vendor dir on exit.

#!/usr/bin/env bash
# Usage: stache-perf.sh <site-dir> <variant-name> <cms-checkout-path|installed> <runs> <results.csv>
# Times `please stache:warm --exclude=revisions,assets` after a full cache clear, N times,
# with vendor/statamic/cms temporarily symlinked to the given checkout.
set -euo pipefail
SITE=$1; VARIANT=$2; CMS=$3; RUNS=$4; OUT=$5
VENDOR="$SITE/vendor/statamic/cms"

restore() {
  if [ -L "$VENDOR" ]; then rm "$VENDOR"; fi
  if [ -d "$VENDOR.orig" ] && [ ! -e "$VENDOR" ]; then mv "$VENDOR.orig" "$VENDOR"; fi
}
trap restore EXIT

if [ "$CMS" != "installed" ]; then
  [ -L "$VENDOR" ] && rm "$VENDOR"
  [ -d "$VENDOR" ] && mv "$VENDOR" "$VENDOR.orig"
  ln -s "$CMS" "$VENDOR"
fi

export STATAMIC_STACHE_WATCHER=false STATAMIC_STACHE_FILE_WATCHER=false DEBUGBAR_ENABLED=false
cd "$SITE"
VERSION=$(php -d memory_limit=-1 please --version 2>/dev/null | tail -1)
echo "[$VARIANT] $VERSION src=$(readlink "$VENDOR" || echo installed) blueprint-md5=$(md5 -q "$VENDOR/src/Fields/Blueprint.php")"

for i in $(seq 1 "$RUNS"); do
  php artisan cache:clear -q
  php -d memory_limit=-1 please stache:clear -q
  START=$(python3 -c 'import time;print(time.time())')
  /usr/bin/time -l php -d memory_limit=-1 please stache:warm --exclude=revisions,assets 2> "$OUT.time" >/dev/null
  END=$(python3 -c 'import time;print(time.time())')
  SECS=$(python3 -c "print(round($END-$START,1))")
  RSS=$(awk '/maximum resident set size/{printf "%d", $1/1048576}' "$OUT.time")
  echo "$(basename "$SITE"),$VARIANT,$i,$SECS,$RSS" | tee -a "$OUT"
done

breakdown.php warms each store on its own and prints time/memory per store (diagnostic only; ordering differs from a real warm).

<?php
// Usage: php -d memory_limit=-1 breakdown.php <site-dir>
// Clears the cache and warms each Stache store one by one, in Stache::stores() order,
// printing the time and memory each one takes. Diagnostic only: stores warmed this way
// don't go through Stache::warm(), so cross-store ordering effects differ from a real warm.
$site = rtrim($argv[1], '/');
chdir($site);
require $site.'/vendor/autoload.php';
$app = require $site.'/bootstrap/app.php';
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();

use Statamic\Facades\Stache;

Illuminate\Support\Facades\Cache::flush();
Stache::clear();

$total = microtime(true);
printf("%-14s %8s %8s %6s\n", 'store', 'seconds', 'mem_mb', 'files');
foreach (Stache::stores()->except(['revisions', 'assets']) as $key => $store) {
    $t = microtime(true);
    $store->warm();
    printf("%-14s %8.1f %8d %6d\n", $key, microtime(true) - $t, memory_get_peak_usage(true) / 1048576, $store->paths()->count());
}
printf("%-14s %8.1f %8d\n", 'TOTAL', microtime(true) - $total, memory_get_peak_usage(true) / 1048576);

[edalzell]

#15639 is faster. This is not needed anymore.