fix(prune-cache): stop the aside sweeper depending on timing it cannot see

Review findings on #2. The first is the one that mattered: the sweeper this
PR added had the shape the PR exists to remove. Pass A renames a candidate
aside; pass B's sweep sees an aside with no readers and reclaims it; A then
finds a reader and restores. `rm -rf` traverses fd-relative, so the rename
does not stop it and A can republish a half-emptied tree under a live cache
name. `capacity: 1` bounds it today, which is exactly the kind of reason
this PR was written to stop relying on.

The unlink itself was never the problem — the ordering proof covers it under
any interleaving, since the aside name only exists after the evicting pass's
rename. What was missing is that an aside with no readers is indistinguishable
from one a pass has just created and not yet decided about. The sweeper now
leaves an aside alone until it has settled (EVICTION_ASIDE_SETTLE_SECONDS,
default 60), which separates the two without having to identify the pass that
created it — a PID is meaningless across the job containers these passes run
in, and recycles. Read from ctime, not mtime: rename(2) updates the first and
leaves the second at whenever the cache was last written, which is the signal
list_by_lru wants and the wrong one here.

That is a bound, not a construction, and both the code comment and the README
now say which of the two properties is which instead of asserting the broader
one.

Also from the review: a pass that declined every dead cache it found no
longer signs off with "no dead-branch caches found", and evict_dir no longer
promises a later reclamation of an aside that is already gone.

Scenario 14 covers the settle window against the script's own default, with
nothing faked — the directory really was set aside a moment ago. Scenario 4
gains the summary assertion. 31 -> 37 assertions; each new gate verified red
by defeating it alone in a scratch copy.
This commit is contained in:
2026-08-23 23:30:56 -05:00
parent c6a3fa6d97
commit 65f0782233
3 changed files with 114 additions and 17 deletions
+46 -3
View File
@@ -13,7 +13,9 @@
# not age, is what decides pass 1.
# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
# their caches are the oldest on disk and would rank first for LRU.
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure.
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure — and
# the pass's closing summary agrees with the decline it just logged,
# rather than reporting that it found nothing.
# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
# permanently.
@@ -38,6 +40,10 @@
# 13. A DEFERRED EVICTION IS RECLAIMED, but not while its reader is live.
# Nothing else globs a dotted name, so an unswept one is disk lost for
# good on the volume whose whole problem is disk.
# 14. AND NOT WHILE ANOTHER PASS MAY STILL BE EVICTING IT. An aside with no
# readers is indistinguishable from one a concurrent pass has just
# renamed and not yet decided about; reclaiming that one lets `rm -rf`
# empty a tree its owner may still restore under a live cache name.
set -euo pipefail
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$script_dir/cache-lib.sh"
@@ -82,8 +88,14 @@ reset_cache() {
mk "snapshot-$DEAD" '2030-01-01'
mk "target-$OWN" '2025-01-01'
}
# run_prune <df-override> [settle-seconds]
#
# The settle window is only set when a scenario asks for it, so every other
# scenario — scenario 14 above all — runs against the script's own default
# rather than against a value this file chose.
run_prune() {
local free="${1:-}"
if [ -n "${2:-}" ]; then export EVICTION_ASIDE_SETTLE_SECONDS="$2"; else unset EVICTION_ASIDE_SETTLE_SECONDS; fi
CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
|| { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
@@ -133,6 +145,17 @@ date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
run_prune "1000000 1000"
assert_kept "$root/target-$DEAD" "locked cache survives both passes"
assert_log "held open by" "lock reported in the log"
# With the locked one the only dead cache left, the pass has declined every
# dead cache it found — at which point "no dead-branch caches found" is a
# false summary of the decline logged two lines above it.
rm -rf "$root/snapshot-$DEAD"
run_prune "1000000 900000"
assert_kept "$root/target-$DEAD" "still not evicted when it is the only dead cache"
assert_log "none pruned this pass" "a pass that declined every dead cache reports that"
if grep -q "no dead-branch caches found" "$scratch/log"; then
fail "the closing summary contradicts the decline logged above it"
fi
ok "the summary does not claim it found nothing"
echo
echo "=== 5: a stale lock is not honoured forever ==="
@@ -225,13 +248,33 @@ reset_cache
aside="$root/.evicting-target-$DEAD-9999"
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
date +%s > "$root/.reading-target-$DEAD-job1"
run_prune "1000000 900000"
# The settle window (scenario 14) gates this sweep first and would decide both
# runs on its own. A directory's ctime is what that window reads and cannot be
# backdated the way `touch -d` backdates an mtime, so the window is moved out
# of the way rather than the directory aged into it.
run_prune "1000000 900000" 0
assert_kept "$aside" "a deferred eviction is not reclaimed while a job is still reading it"
assert_log "deferring its reclamation again" "the continued deferral is reported"
rm -f "$root/.reading-target-$DEAD-job1"
run_prune "1000000 900000"
run_prune "1000000 900000" 0
assert_gone "$aside" "a deferred eviction is reclaimed once its reader is gone"
assert_log "reclaiming deferred eviction" "the reclamation is reported"
echo
echo "=== 14: an aside another pass may still be evicting is left alone ==="
reset_cache
aside="$root/.evicting-target-$DEAD-9999"
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
# Deliberately no reader marker: the reader gate would pass this straight
# through, which is the whole point. An aside with no readers is exactly what
# a pass that has just renamed one aside and not yet decided about it looks
# like, and `rm -rf` traverses fd-relative — so reclaiming it out from under
# that pass lets it republish a half-emptied tree under a live cache name.
# Nothing here fakes an age: the directory really was set aside a moment ago.
run_prune "1000000 900000"
assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
assert_kept "$aside/blob" "and is left intact, not part-way emptied"
assert_log "may still be evicting it" "the deferral gives its actual reason"
echo
echo "prune-cache-selftest: ${pass_count} assertions passed"