claude
65f0782233
Review findings on #2. The first is the one that mattered: the sweeper this PR added had the shape the PR exists to remove. Pass A renames a candidate aside; pass B's sweep sees an aside with no readers and reclaims it; A then finds a reader and restores. `rm -rf` traverses fd-relative, so the rename does not stop it and A can republish a half-emptied tree under a live cache name. `capacity: 1` bounds it today, which is exactly the kind of reason this PR was written to stop relying on. The unlink itself was never the problem — the ordering proof covers it under any interleaving, since the aside name only exists after the evicting pass's rename. What was missing is that an aside with no readers is indistinguishable from one a pass has just created and not yet decided about. The sweeper now leaves an aside alone until it has settled (EVICTION_ASIDE_SETTLE_SECONDS, default 60), which separates the two without having to identify the pass that created it — a PID is meaningless across the job containers these passes run in, and recycles. Read from ctime, not mtime: rename(2) updates the first and leaves the second at whenever the cache was last written, which is the signal list_by_lru wants and the wrong one here. That is a bound, not a construction, and both the code comment and the README now say which of the two properties is which instead of asserting the broader one. Also from the review: a pass that declined every dead cache it found no longer signs off with "no dead-branch caches found", and evict_dir no longer promises a later reclamation of an aside that is already gone. Scenario 14 covers the settle window against the script's own default, with nothing faked — the directory really was set aside a moment ago. Scenario 4 gains the summary assertion. 31 -> 37 assertions; each new gate verified red by defeating it alone in a scratch copy.
281 lines
14 KiB
Bash
Executable File
281 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for prune-cache.sh. Builds a real scratch git repo standing
|
|
# in for `origin` and a real scratch directory standing in for the cache root,
|
|
# then runs the ACTUAL script against both — not a simulation of its logic.
|
|
#
|
|
# What each scenario demonstrates, and why the controls matter as much as the
|
|
# fixes (a scenario that always passes proves nothing):
|
|
#
|
|
# 1. DEAD BRANCH PRUNED, not gated on disk pressure — a cache whose branch
|
|
# no longer exists on origin is removed even with plenty of free space.
|
|
# Waiting for pressure to notice means paying for dead caches until then.
|
|
# 2. LIVE BRANCH SURVIVES despite being OLDER than the dead one — liveness,
|
|
# not age, is what decides pass 1.
|
|
# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
|
|
# their caches are the oldest on disk and would rank first for LRU.
|
|
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure — and
|
|
# the pass's closing summary agrees with the decline it just logged,
|
|
# rather than reporting that it found nothing.
|
|
# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
|
|
# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
|
|
# permanently.
|
|
# 6. LIVENESS UNAVAILABLE FAILS SAFE — origin unreachable: a genuinely dead
|
|
# cache is NOT pruned, the log says so plainly, and the pressure fallback
|
|
# still works independently. "Unavailable" degrades to pressure-only, not
|
|
# to no eviction at all.
|
|
# 7. TARGET DIRS EVICTED BEFORE SNAPSHOTS — the ordering that differs from
|
|
# the obvious one. A snapshot is hardlinked to the caches cloned from it,
|
|
# so evicting it frees almost nothing while costing every future PR its
|
|
# warm start.
|
|
# 8. SELF-CLEAR REPORTS LOUDLY to the job summary, not just a log warning.
|
|
# 9. OWN CACHE NEVER EVICTED by a sibling pass.
|
|
# 10. SCOPED TO THE CACHE ROOT — a decoy outside it (standing in for another
|
|
# project's volume) is never touched.
|
|
# 11. A LIVE READER MARKER PROTECTS A CACHE the same way a lock file does — a
|
|
# directory somebody is hardlink-cloning this instant is not a candidate,
|
|
# however dead and however tight the disk.
|
|
# 12. AND SO DOES ONE PUBLISHED INSIDE THE CHECK-TO-UNLINK WINDOW, which is
|
|
# the property a check-then-delete eviction does NOT have. This is the
|
|
# one that fails against the pre-fix script.
|
|
# 13. A DEFERRED EVICTION IS RECLAIMED, but not while its reader is live.
|
|
# Nothing else globs a dotted name, so an unswept one is disk lost for
|
|
# good on the volume whose whole problem is disk.
|
|
# 14. AND NOT WHILE ANOTHER PASS MAY STILL BE EVICTING IT. An aside with no
|
|
# readers is indistinguishable from one a concurrent pass has just
|
|
# renamed and not yet decided about; reclaiming that one lets `rm -rf`
|
|
# empty a tree its owner may still restore under a live cache name.
|
|
set -euo pipefail
|
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
. "$script_dir/cache-lib.sh"
|
|
prune="$script_dir/prune-cache.sh"
|
|
|
|
scratch=$(mktemp -d)
|
|
trap 'rm -rf "$scratch"' EXIT
|
|
pass_count=0
|
|
fail() { echo "ASSERTION FAILED: $*" >&2; [ -n "${1:-}" ] && [ -f "$scratch/log" ] && { echo "--- log ---" >&2; cat "$scratch/log" >&2; }; exit 1; }
|
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
assert_gone() { [ -e "$1" ] && fail "expected gone: $1 ($2)"; ok "$2"; }
|
|
assert_kept() { [ -e "$1" ] || fail "expected kept: $1 ($2)"; ok "$2"; }
|
|
assert_log() { grep -q -- "$1" "$scratch/log" || fail "expected in log: $1 ($2)"; ok "$2"; }
|
|
|
|
echo "=== building a scratch origin with real branches ==="
|
|
origin="$scratch/origin.git"; git init -q --bare "$origin"
|
|
work="$scratch/work"; git init -q "$work"
|
|
(
|
|
cd "$work"
|
|
git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init
|
|
git branch -M main
|
|
git checkout -q -b dev; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m dev
|
|
git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m live
|
|
git remote add origin "$origin"
|
|
git push -q origin main dev feat/live
|
|
)
|
|
cd "$work"
|
|
|
|
MAIN=$(cache_key main); DEV=$(cache_key dev); LIVE=$(cache_key feat/live)
|
|
DEAD=$(cache_key feat/dead); OWN=$(cache_key feat/own)
|
|
|
|
root="$scratch/cache"
|
|
mk() { mkdir -p "$root/$1"; head -c 4096 /dev/zero > "$root/$1/blob"; touch -d "$2" "$root/$1/.cache-last-used"; }
|
|
reset_cache() {
|
|
rm -rf "$root"; mkdir -p "$root"
|
|
mk "target-$MAIN" '2020-01-01'
|
|
mk "snapshot-$MAIN" '2020-01-01'
|
|
mk "target-$DEV" '2020-01-01'
|
|
mk "snapshot-$DEV" '2020-01-01'
|
|
mk "target-$LIVE" '2020-01-02' # older than the dead one, deliberately
|
|
mk "target-$DEAD" '2030-01-01' # newest on disk, but its branch is gone
|
|
mk "snapshot-$DEAD" '2030-01-01'
|
|
mk "target-$OWN" '2025-01-01'
|
|
}
|
|
# run_prune <df-override> [settle-seconds]
|
|
#
|
|
# The settle window is only set when a scenario asks for it, so every other
|
|
# scenario — scenario 14 above all — runs against the script's own default
|
|
# rather than against a value this file chose.
|
|
run_prune() {
|
|
local free="${1:-}"
|
|
if [ -n "${2:-}" ]; then export EVICTION_ASIDE_SETTLE_SECONDS="$2"; else unset EVICTION_ASIDE_SETTLE_SECONDS; fi
|
|
CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
|
|
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
|
|
}
|
|
|
|
echo
|
|
echo "=== 1/2: dead pruned unconditionally; older-but-live survives ==="
|
|
reset_cache
|
|
run_prune "1000000 900000" # 90% free: no pressure at all
|
|
assert_gone "$root/target-$DEAD" "dead branch's target dir pruned with no disk pressure"
|
|
assert_gone "$root/snapshot-$DEAD" "dead branch's snapshot pruned too"
|
|
assert_kept "$root/target-$LIVE" "live branch survives despite an older marker than the dead one"
|
|
assert_log "no matching branch on origin" "eviction reason reported"
|
|
|
|
echo
|
|
echo "=== 3: protected refs never evicted under forced pressure ==="
|
|
reset_cache
|
|
run_prune "1000000 1000" # 0.1% free
|
|
assert_kept "$root/target-$DEV" "dev's target dir survives disk pressure"
|
|
assert_kept "$root/snapshot-$DEV" "dev's snapshot survives disk pressure"
|
|
assert_kept "$root/target-$MAIN" "main's target dir survives disk pressure"
|
|
assert_kept "$root/snapshot-$MAIN" "main's snapshot survives disk pressure"
|
|
|
|
echo
|
|
echo "=== 9: own cache never evicted by a sibling pass ==="
|
|
assert_kept "$root/target-$OWN" "this run's own cache survives"
|
|
|
|
echo
|
|
echo "=== 7: target dirs evicted before snapshots ==="
|
|
reset_cache
|
|
# Only the live branch is evictable; give it both a target dir and a snapshot
|
|
# with identical markers so ordering, not age, decides.
|
|
mk "snapshot-$LIVE" '2020-01-02'
|
|
# The df override is a fixed reading, so the pressure loop drains everything
|
|
# evictable — which is what makes the ORDER the observable property here, not
|
|
# what survives. Assert the eviction order directly from the log.
|
|
run_prune "1000000 1000"
|
|
order=$(grep -o "evicted \(target\|snapshot\)-$LIVE" "$scratch/log" | sed "s/evicted //")
|
|
[ "$(printf '%s\n' "$order" | head -1)" = "target-$LIVE" ] \
|
|
|| fail "expected target-$LIVE to be evicted before snapshot-$LIVE, got: $order"
|
|
ok "target dirs are evicted before snapshots"
|
|
|
|
echo
|
|
echo "=== 4: a fresh lock protects a dead, old, under-pressure cache ==="
|
|
reset_cache
|
|
date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$root/target-$DEAD" "locked cache survives both passes"
|
|
assert_log "held open by" "lock reported in the log"
|
|
# With the locked one the only dead cache left, the pass has declined every
|
|
# dead cache it found — at which point "no dead-branch caches found" is a
|
|
# false summary of the decline logged two lines above it.
|
|
rm -rf "$root/snapshot-$DEAD"
|
|
run_prune "1000000 900000"
|
|
assert_kept "$root/target-$DEAD" "still not evicted when it is the only dead cache"
|
|
assert_log "none pruned this pass" "a pass that declined every dead cache reports that"
|
|
if grep -q "no dead-branch caches found" "$scratch/log"; then
|
|
fail "the closing summary contradicts the decline logged above it"
|
|
fi
|
|
ok "the summary does not claim it found nothing"
|
|
|
|
echo
|
|
echo "=== 5: a stale lock is not honoured forever ==="
|
|
reset_cache
|
|
echo 0 > "$root/target-$DEAD/.ci-lock-ci-1"
|
|
touch -d '2000-01-01' "$root/target-$DEAD/.ci-lock-ci-1"
|
|
run_prune "1000000 900000"
|
|
assert_gone "$root/target-$DEAD" "cache with an abandoned lock is evicted"
|
|
assert_log "treating as abandoned" "abandoned lock reported in the log"
|
|
|
|
echo
|
|
echo "=== 6: liveness unavailable fails safe, pressure fallback still works ==="
|
|
reset_cache
|
|
(
|
|
cd "$work" && git remote set-url origin "$scratch/nonexistent.git"
|
|
)
|
|
run_prune "1000000 900000" # no pressure
|
|
assert_kept "$root/target-$DEAD" "dead cache NOT pruned when liveness is unavailable"
|
|
assert_log "treating as UNAVAILABLE" "unavailability reported plainly, not folded into 'no branches'"
|
|
run_prune "1000000 1000" # now with pressure
|
|
if [ -e "$root/target-$DEAD" ] && [ -e "$root/target-$LIVE" ]; then
|
|
fail "pressure fallback did nothing when liveness was unavailable"
|
|
fi
|
|
ok "pressure fallback still evicts when liveness is unavailable"
|
|
(cd "$work" && git remote set-url origin "$origin")
|
|
|
|
echo
|
|
echo "=== 8: self-clear reports to the job summary ==="
|
|
reset_cache
|
|
rm -rf "$root/target-$DEAD" "$root/snapshot-$DEAD" "$root/target-$LIVE"
|
|
: > "$scratch/summary"
|
|
run_prune "1000000 1000" # nothing evictable left but the run's own cache
|
|
assert_log "clearing own" "self-clear reported in the log"
|
|
grep -q 'self-clear' "$scratch/summary" || fail "self-clear missing from the job summary"
|
|
ok "self-clear reported to the job summary, not only the log"
|
|
[ -d "$root/target-$OWN" ] || fail "self-clear left the own directory missing"
|
|
[ -z "$(ls -A "$root/target-$OWN")" ] || fail "self-clear did not actually empty the directory"
|
|
ok "own cache wiped and recreated empty"
|
|
|
|
echo
|
|
echo "=== 10: scoped to the cache root ==="
|
|
reset_cache
|
|
decoy="$scratch/other-project"; mkdir -p "$decoy/target-$DEAD"; touch "$decoy/target-$DEAD/blob"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$decoy/target-$DEAD" "a cache outside the cache root is never touched"
|
|
|
|
echo
|
|
echo "=== 11: a live reader marker protects a cache, like a lock file does ==="
|
|
reset_cache
|
|
date +%s > "$root/.reading-target-$DEAD-job1"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$root/target-$DEAD" "a cache being hardlink-cloned right now survives both passes"
|
|
assert_log "currently cloning it" "the reader is named in the log, not silently honoured"
|
|
rm -f "$root/.reading-target-$DEAD-job1"
|
|
|
|
echo
|
|
echo "=== 12: a reader marker published INSIDE the check-to-unlink window ==="
|
|
reset_cache
|
|
# A consumer publishes its marker whenever it starts a clone, which can be at
|
|
# any instant — including after the pass has checked for markers and before it
|
|
# unlinks. That window is real time, not a theoretical interleaving: `du -sk`
|
|
# on a multi-GB cache runs for seconds, and the pass runs one on every
|
|
# candidate. It is reproduced deterministically here by making that very `du`
|
|
# publish the marker, which places it strictly after the check and strictly
|
|
# before the unlink — exactly where a check-then-delete eviction cannot see
|
|
# it. The candidate must still be standing afterwards, with its contents
|
|
# intact and nothing left renamed aside.
|
|
mkdir -p "$scratch/bin"
|
|
real_du=$(command -v du)
|
|
cat > "$scratch/bin/du" <<EOF
|
|
#!/usr/bin/env bash
|
|
for arg; do
|
|
case "\$arg" in */target-$DEAD) date +%s > "$root/.reading-target-$DEAD-racer" ;; esac
|
|
done
|
|
exec "$real_du" "\$@"
|
|
EOF
|
|
chmod +x "$scratch/bin/du"
|
|
( PATH="$scratch/bin:$PATH"; run_prune "1000000 900000" )
|
|
[ -e "$root/.reading-target-$DEAD-racer" ] || fail "the racing marker was never published — scenario 12 proves nothing"
|
|
assert_kept "$root/target-$DEAD" "a cache claimed inside the eviction window is not unlinked"
|
|
assert_kept "$root/target-$DEAD/blob" "the reprieved cache still has its contents"
|
|
assert_log "restored, not evicted" "the reprieve is reported, not silent"
|
|
[ -z "$(ls -d "$root"/.evicting-* 2>/dev/null)" ] || fail "an aside directory was left behind after the reprieve"
|
|
ok "nothing left renamed aside once the eviction is declined"
|
|
rm -f "$root/.reading-target-$DEAD-racer"
|
|
|
|
echo
|
|
echo "=== 13: a deferred eviction is reclaimed, but not under a live reader ==="
|
|
reset_cache
|
|
aside="$root/.evicting-target-$DEAD-9999"
|
|
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
|
|
date +%s > "$root/.reading-target-$DEAD-job1"
|
|
# The settle window (scenario 14) gates this sweep first and would decide both
|
|
# runs on its own. A directory's ctime is what that window reads and cannot be
|
|
# backdated the way `touch -d` backdates an mtime, so the window is moved out
|
|
# of the way rather than the directory aged into it.
|
|
run_prune "1000000 900000" 0
|
|
assert_kept "$aside" "a deferred eviction is not reclaimed while a job is still reading it"
|
|
assert_log "deferring its reclamation again" "the continued deferral is reported"
|
|
rm -f "$root/.reading-target-$DEAD-job1"
|
|
run_prune "1000000 900000" 0
|
|
assert_gone "$aside" "a deferred eviction is reclaimed once its reader is gone"
|
|
assert_log "reclaiming deferred eviction" "the reclamation is reported"
|
|
|
|
echo
|
|
echo "=== 14: an aside another pass may still be evicting is left alone ==="
|
|
reset_cache
|
|
aside="$root/.evicting-target-$DEAD-9999"
|
|
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
|
|
# Deliberately no reader marker: the reader gate would pass this straight
|
|
# through, which is the whole point. An aside with no readers is exactly what
|
|
# a pass that has just renamed one aside and not yet decided about it looks
|
|
# like, and `rm -rf` traverses fd-relative — so reclaiming it out from under
|
|
# that pass lets it republish a half-emptied tree under a live cache name.
|
|
# Nothing here fakes an age: the directory really was set aside a moment ago.
|
|
run_prune "1000000 900000"
|
|
assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
|
|
assert_kept "$aside/blob" "and is left intact, not part-way emptied"
|
|
assert_log "may still be evicting it" "the deferral gives its actual reason"
|
|
|
|
echo
|
|
echo "prune-cache-selftest: ${pass_count} assertions passed"
|