fix(prune-cache): stop the aside sweeper depending on timing it cannot see

Review findings on #2. The first is the one that mattered: the sweeper this
PR added had the shape the PR exists to remove. Pass A renames a candidate
aside; pass B's sweep sees an aside with no readers and reclaims it; A then
finds a reader and restores. `rm -rf` traverses fd-relative, so the rename
does not stop it and A can republish a half-emptied tree under a live cache
name. `capacity: 1` bounds it today, which is exactly the kind of reason
this PR was written to stop relying on.

The unlink itself was never the problem — the ordering proof covers it under
any interleaving, since the aside name only exists after the evicting pass's
rename. What was missing is that an aside with no readers is indistinguishable
from one a pass has just created and not yet decided about. The sweeper now
leaves an aside alone until it has settled (EVICTION_ASIDE_SETTLE_SECONDS,
default 60), which separates the two without having to identify the pass that
created it — a PID is meaningless across the job containers these passes run
in, and recycles. Read from ctime, not mtime: rename(2) updates the first and
leaves the second at whenever the cache was last written, which is the signal
list_by_lru wants and the wrong one here.

That is a bound, not a construction, and both the code comment and the README
now say which of the two properties is which instead of asserting the broader
one.

Also from the review: a pass that declined every dead cache it found no
longer signs off with "no dead-branch caches found", and evict_dir no longer
promises a later reclamation of an aside that is already gone.

Scenario 14 covers the settle window against the script's own default, with
nothing faked — the directory really was set aside a moment ago. Scenario 4
gains the summary assertion. 31 -> 37 assertions; each new gate verified red
by defeating it alone in a scratch copy.

README.md

@@ -182,9 +182,14 @@ not collapsed into one:
   rename cannot resolve the path at all and starts cold instead. A cache
   claimed inside that window is put back under its own name, and one whose
   name a concurrent seed has taken in the meantime is left aside and reclaimed
-  by a later pass once its readers drain. Until this was structural it was
+  by a later pass once its readers drain. That later pass leaves an aside
-  merely policy — snapshots belong to protected refs, protected refs are never
+  directory alone until it has been set aside for a minute — not for the
-  eviction candidates — which is a property held by vigilance rather than by
+  unlink's sake, which the ordering proof above already covers, but so that a
+  pass still deciding about one is never mistaken for a pass that died holding
+  it. That settle window is a bound rather than a construction, and it is the
+  only part of this that is. Until the rest of it was structural it was merely
+  policy — snapshots belong to protected refs, protected refs are never
+  eviction candidates — a property held by vigilance rather than by
   construction.
 - **Every other way the source can change mid-clone is detected, not
   prevented.** A `seed-fallback-dir` pointing at a directory something else
@@ -199,9 +204,9 @@ not collapsed into one:
   residual is capped at one deferred generation per publisher ref, and its
   real cost is close to inode count rather than byte count, since the
   artifacts are hardlinked to whatever cloned them. A declined eviction is
-  bounded the same way: the cache stays, either under its own name or aside
+  never unlinked under the job that claimed it; what it costs meanwhile is
-  awaiting the next pass, and is never unlinked under the job that claimed
+  disk, normally as the cache restored under its own name and otherwise as one
-  it.
+  set aside for a later pass to reclaim.
 
 **Eviction** runs three passes: caches for branches that no longer exist on
 origin are removed unconditionally; then, only if free space is under the

scripts/prune-cache-selftest.sh

@@ -13,7 +13,9 @@
 #      not age, is what decides pass 1.
 #   3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
 #      their caches are the oldest on disk and would rank first for LRU.
-#   4. LOCKED CACHE PROTECTED even when dead, old, and under pressure.
+#   4. LOCKED CACHE PROTECTED even when dead, old, and under pressure — and
+#      the pass's closing summary agrees with the decline it just logged,
+#      rather than reporting that it found nothing.
 #   5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
 #      STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
 #      permanently.
@@ -38,6 +40,10 @@
 #  13. A DEFERRED EVICTION IS RECLAIMED, but not while its reader is live.
 #      Nothing else globs a dotted name, so an unswept one is disk lost for
 #      good on the volume whose whole problem is disk.
+#  14. AND NOT WHILE ANOTHER PASS MAY STILL BE EVICTING IT. An aside with no
+#      readers is indistinguishable from one a concurrent pass has just
+#      renamed and not yet decided about; reclaiming that one lets `rm -rf`
+#      empty a tree its owner may still restore under a live cache name.
 set -euo pipefail
 script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$script_dir/cache-lib.sh"
@@ -82,8 +88,14 @@ reset_cache() {
   mk "snapshot-$DEAD" '2030-01-01'
   mk "target-$OWN"   '2025-01-01'
 }
+# run_prune <df-override> [settle-seconds]
+#
+# The settle window is only set when a scenario asks for it, so every other
+# scenario — scenario 14 above all — runs against the script's own default
+# rather than against a value this file chose.
 run_prune() {
   local free="${1:-}"
+  if [ -n "${2:-}" ]; then export EVICTION_ASIDE_SETTLE_SECONDS="$2"; else unset EVICTION_ASIDE_SETTLE_SECONDS; fi
   CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
     bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
     || { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
@@ -133,6 +145,17 @@ date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
 run_prune "1000000 1000"
 assert_kept "$root/target-$DEAD" "locked cache survives both passes"
 assert_log "held open by" "lock reported in the log"
+# With the locked one the only dead cache left, the pass has declined every
+# dead cache it found — at which point "no dead-branch caches found" is a
+# false summary of the decline logged two lines above it.
+rm -rf "$root/snapshot-$DEAD"
+run_prune "1000000 900000"
+assert_kept "$root/target-$DEAD" "still not evicted when it is the only dead cache"
+assert_log "none pruned this pass" "a pass that declined every dead cache reports that"
+if grep -q "no dead-branch caches found" "$scratch/log"; then
+  fail "the closing summary contradicts the decline logged above it"
+fi
+ok "the summary does not claim it found nothing"
 
 echo
 echo "=== 5: a stale lock is not honoured forever ==="
@@ -225,13 +248,33 @@ reset_cache
 aside="$root/.evicting-target-$DEAD-9999"
 mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
 date +%s > "$root/.reading-target-$DEAD-job1"
-run_prune "1000000 900000"
+# The settle window (scenario 14) gates this sweep first and would decide both
+# runs on its own. A directory's ctime is what that window reads and cannot be
+# backdated the way `touch -d` backdates an mtime, so the window is moved out
+# of the way rather than the directory aged into it.
+run_prune "1000000 900000" 0
 assert_kept "$aside" "a deferred eviction is not reclaimed while a job is still reading it"
 assert_log "deferring its reclamation again" "the continued deferral is reported"
 rm -f "$root/.reading-target-$DEAD-job1"
-run_prune "1000000 900000"
+run_prune "1000000 900000" 0
 assert_gone "$aside" "a deferred eviction is reclaimed once its reader is gone"
 assert_log "reclaiming deferred eviction" "the reclamation is reported"
 
+echo
+echo "=== 14: an aside another pass may still be evicting is left alone ==="
+reset_cache
+aside="$root/.evicting-target-$DEAD-9999"
+mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
+# Deliberately no reader marker: the reader gate would pass this straight
+# through, which is the whole point. An aside with no readers is exactly what
+# a pass that has just renamed one aside and not yet decided about it looks
+# like, and `rm -rf` traverses fd-relative — so reclaiming it out from under
+# that pass lets it republish a half-emptied tree under a live cache name.
+# Nothing here fakes an age: the directory really was set aside a moment ago.
+run_prune "1000000 900000"
+assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
+assert_kept "$aside/blob" "and is left intact, not part-way emptied"
+assert_log "may still be evicting it" "the deferral gives its actual reason"
+
 echo
 echo "prune-cache-selftest: ${pass_count} assertions passed"

scripts/prune-cache.sh

@@ -9,6 +9,10 @@
 #                       abandoned (default 7200)
 #   CACHE_LIVENESS      "false"/"0" to skip the liveness pass entirely
 #   CACHE_DF_OVERRIDE   "<total_kb> <free_kb>", for the selftest
+#   EVICTION_ASIDE_SETTLE_SECONDS
+#                       how long a directory renamed aside for eviction is
+#                       left alone before another pass may reclaim it
+#                       (default 60)
 #
 # Three passes, in order:
 #
@@ -82,6 +86,15 @@ OWN_DIR="${2:?}"
 PROTECTED_REFS="${3:-}"
 MIN_FREE_PCT="${4:-10}"
 STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}"
+# An aside directory is in flight for one rename plus one marker glob —
+# milliseconds. Anything older belongs to a pass that died between the two, so
+# an age is what separates "another pass is mid-eviction" from "a leftover",
+# and it separates them without having to identify the pass that created it: a
+# PID is meaningless across the job containers these passes run in, and
+# recycles. Three orders of magnitude of headroom over the operation it covers,
+# and short enough that a genuine leftover is reclaimed by the next run rather
+# than lingering while the volume is under pressure.
+EVICTION_ASIDE_SETTLE_SECONDS="${EVICTION_ASIDE_SETTLE_SECONDS:-60}"
 
 declare -A protected_ns=()
 for ref in $PROTECTED_REFS; do
@@ -172,8 +185,10 @@ evict_dir() {
   if is_locked "$aside" "$name"; then
     if mv -T "$aside" "$dir" 2>/dev/null; then
       echo "  ${name}: claimed by a job while its eviction was in flight — restored, not evicted"
+    elif [ -d "$aside" ]; then
+      echo "::warning::prune: ${name} was claimed mid-eviction and its own name is taken again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain"
     else
-      echo "::warning::prune: ${name} was claimed mid-eviction and its own name is occupied again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain"
+      echo "::warning::prune: ${name} was claimed mid-eviction and is already gone — another pass reclaimed it after its readers drained"
     fi
     return 1
   fi
@@ -204,17 +219,41 @@ list_by_lru() {
 
 # Deferred reclamations from an earlier pass: a directory renamed aside for
 # eviction that could not be unlinked, because a job claimed it inside the
-# window and its own name was occupied again before it could be restored — or
+# window and its own name was taken again before it could be restored — or
 # whose run was killed between the rename and the unlink. Nothing below would
 # ever see one: every pass globs target-*/snapshot-*, which a dotted name does
 # not match. Left unswept it is permanently unreclaimable disk on the one
 # volume whose entire problem is disk.
 #
-# Safe under a concurrent pass mid-eviction for the reason evict_dir is: the
+# TWO DISTINCT PROPERTIES HOLD HERE, and neither implies the other.
-# aside name only exists after that pass's rename, so a consumer that resolved
+#
-# the directory published its marker before the count below.
+# No clone can be truncated by the unlink below, by construction: the aside
+# name only comes into existence after the evicting pass's rename, so a
+# consumer that resolved the directory published its marker strictly before
+# that rename and therefore before the count below — it cannot be missed. A
+# consumer that arrives later cannot resolve the path at all. This holds under
+# any interleaving and needs no settle window.
+#
+# No pass mid-eviction is mistaken for a leftover, by bound rather than by
+# construction, and this is what the settle window is for. Without it a
+# concurrent pass could unlink an aside its owner is still deciding about, and
+# `rm -rf` traverses fd-relative: the owner's restore can then republish a
+# half-emptied tree under a live cache name. What the window guarantees is that
+# the two cannot be confused within it. What it does not guarantee is the
+# pathological case beyond it — an evicting pass suspended past the window and
+# then resumed finds its aside reclaimed and fails its restore, saying so.
+now=$(date +%s)
 for aside in "$ROOT"/.evicting-*; do
   [ -d "$aside" ] || continue
+  # Change time, not modification time. `mv` leaves a directory's mtime alone
+  # (a cache last written days ago keeps a days-old mtime, which is what
+  # list_by_lru wants and exactly the wrong signal here) but rename(2) does
+  # update ctime, so %Z is when this directory was set aside and %Y is not.
+  aside_age=$(( now - $(stat -c '%Z' "$aside" 2>/dev/null || echo "$now") ))
+  if [ "$aside_age" -lt "$EVICTION_ASIDE_SETTLE_SECONDS" ]; then
+    echo "prune: $(basename "$aside") was set aside ${aside_age}s ago — another pass may still be evicting it, leaving it alone"
+    continue
+  fi
   aside_name=$(basename "$aside"); aside_name="${aside_name#.evicting-}"; aside_name="${aside_name%-*}"
   if [ "$(live_reader_count "$ROOT" "$aside_name")" -gt 0 ]; then
     echo "prune: $(basename "$aside") is still being read — deferring its reclamation again"
@@ -250,19 +289,29 @@ fi
 
 if [ "$LIVENESS_AVAILABLE" = "1" ]; then
   pruned_any=0
+  # Tracked separately so the line below cannot contradict the decline lines
+  # above it: "none pruned" and "none found" are different outcomes, and a
+  # pass that declined every dead cache it found has not found none.
+  spared_any=0
   for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do
     [ -d "$dir" ] || continue
     name=$(basename "$dir")
     is_protected "$dir" && continue
     [ -n "${live_ns[$name]:-}" ] && continue
-    is_locked "$dir" && continue
+    is_locked "$dir" && { spared_any=1; continue; }
     dir_gb=$(usage_gb "$dir")
-    evict_dir "$dir" || continue
+    evict_dir "$dir" || { spared_any=1; continue; }
     echo "::warning::pruned dead-branch cache ${name} (${dir_gb} GB) — no matching branch on origin"
     summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — branch no longer exists on origin"
     pruned_any=1
   done
-  [ "$pruned_any" = "1" ] || echo "no dead-branch caches found"
+  if [ "$pruned_any" = "0" ]; then
+    if [ "$spared_any" = "1" ]; then
+      echo "every dead-branch cache found is still in use — none pruned this pass"
+    else
+      echo "no dead-branch caches found"
+    fi
+  fi
 else
   echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped"
 fi