claude
b0d63c807b
Review nits on #2, both cosmetic. The sentence added last round left its paragraph at 127 characters in a file that otherwise wraps comments at 78-79 — the rewrap after the insert simply did not happen. `awk 'length($0)>80'` over both files now reports one comment line, the pre-existing 93-character usage string at :4. Scenario 14's header called its fixture "the only shape that can tell the two timestamps apart". The property it needs is old mtime with fresh ctime; an old directory renamed a moment ago is the production instance of that, not the only construction of it. "Production's shape" was already carrying the argument. No behaviour change and no assertion change.
291 lines
14 KiB
Bash
Executable File
291 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for prune-cache.sh. Builds a real scratch git repo standing
|
|
# in for `origin` and a real scratch directory standing in for the cache root,
|
|
# then runs the ACTUAL script against both — not a simulation of its logic.
|
|
#
|
|
# What each scenario demonstrates, and why the controls matter as much as the
|
|
# fixes (a scenario that always passes proves nothing):
|
|
#
|
|
# 1. DEAD BRANCH PRUNED, not gated on disk pressure — a cache whose branch
|
|
# no longer exists on origin is removed even with plenty of free space.
|
|
# Waiting for pressure to notice means paying for dead caches until then.
|
|
# 2. LIVE BRANCH SURVIVES despite being OLDER than the dead one — liveness,
|
|
# not age, is what decides pass 1.
|
|
# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
|
|
# their caches are the oldest on disk and would rank first for LRU.
|
|
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure — and
|
|
# the pass's closing summary agrees with the decline it just logged,
|
|
# rather than reporting that it found nothing.
|
|
# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
|
|
# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
|
|
# permanently.
|
|
# 6. LIVENESS UNAVAILABLE FAILS SAFE — origin unreachable: a genuinely dead
|
|
# cache is NOT pruned, the log says so plainly, and the pressure fallback
|
|
# still works independently. "Unavailable" degrades to pressure-only, not
|
|
# to no eviction at all.
|
|
# 7. TARGET DIRS EVICTED BEFORE SNAPSHOTS — the ordering that differs from
|
|
# the obvious one. A snapshot is hardlinked to the caches cloned from it,
|
|
# so evicting it frees almost nothing while costing every future PR its
|
|
# warm start.
|
|
# 8. SELF-CLEAR REPORTS LOUDLY to the job summary, not just a log warning.
|
|
# 9. OWN CACHE NEVER EVICTED by a sibling pass.
|
|
# 10. SCOPED TO THE CACHE ROOT — a decoy outside it (standing in for another
|
|
# project's volume) is never touched.
|
|
# 11. A LIVE READER MARKER PROTECTS A CACHE the same way a lock file does — a
|
|
# directory somebody is hardlink-cloning this instant is not a candidate,
|
|
# however dead and however tight the disk.
|
|
# 12. AND SO DOES ONE PUBLISHED INSIDE THE CHECK-TO-UNLINK WINDOW, which is
|
|
# the property a check-then-delete eviction does NOT have. This is the
|
|
# one that fails against the pre-fix script.
|
|
# 13. A DEFERRED EVICTION IS RECLAIMED, but not while its reader is live.
|
|
# Nothing else globs a dotted name, so an unswept one is disk lost for
|
|
# good on the volume whose whole problem is disk.
|
|
# 14. AND NOT WHILE ANOTHER PASS MAY STILL BE EVICTING IT. An aside with no
|
|
# readers is indistinguishable from one a concurrent pass has just
|
|
# renamed and not yet decided about; reclaiming that one lets `rm -rf`
|
|
# empty a tree its owner may still restore under a live cache name. Its
|
|
# fixture is an OLD directory renamed a moment ago — production's shape,
|
|
# and what lets it tell the two timestamps apart.
|
|
set -euo pipefail
|
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
. "$script_dir/cache-lib.sh"
|
|
prune="$script_dir/prune-cache.sh"
|
|
|
|
scratch=$(mktemp -d)
|
|
trap 'rm -rf "$scratch"' EXIT
|
|
pass_count=0
|
|
fail() { echo "ASSERTION FAILED: $*" >&2; [ -n "${1:-}" ] && [ -f "$scratch/log" ] && { echo "--- log ---" >&2; cat "$scratch/log" >&2; }; exit 1; }
|
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
assert_gone() { [ -e "$1" ] && fail "expected gone: $1 ($2)"; ok "$2"; }
|
|
assert_kept() { [ -e "$1" ] || fail "expected kept: $1 ($2)"; ok "$2"; }
|
|
assert_log() { grep -q -- "$1" "$scratch/log" || fail "expected in log: $1 ($2)"; ok "$2"; }
|
|
|
|
echo "=== building a scratch origin with real branches ==="
|
|
origin="$scratch/origin.git"; git init -q --bare "$origin"
|
|
work="$scratch/work"; git init -q "$work"
|
|
(
|
|
cd "$work"
|
|
git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init
|
|
git branch -M main
|
|
git checkout -q -b dev; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m dev
|
|
git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m live
|
|
git remote add origin "$origin"
|
|
git push -q origin main dev feat/live
|
|
)
|
|
cd "$work"
|
|
|
|
MAIN=$(cache_key main); DEV=$(cache_key dev); LIVE=$(cache_key feat/live)
|
|
DEAD=$(cache_key feat/dead); OWN=$(cache_key feat/own)
|
|
|
|
root="$scratch/cache"
|
|
mk() { mkdir -p "$root/$1"; head -c 4096 /dev/zero > "$root/$1/blob"; touch -d "$2" "$root/$1/.cache-last-used"; }
|
|
reset_cache() {
|
|
rm -rf "$root"; mkdir -p "$root"
|
|
mk "target-$MAIN" '2020-01-01'
|
|
mk "snapshot-$MAIN" '2020-01-01'
|
|
mk "target-$DEV" '2020-01-01'
|
|
mk "snapshot-$DEV" '2020-01-01'
|
|
mk "target-$LIVE" '2020-01-02' # older than the dead one, deliberately
|
|
mk "target-$DEAD" '2030-01-01' # newest on disk, but its branch is gone
|
|
mk "snapshot-$DEAD" '2030-01-01'
|
|
mk "target-$OWN" '2025-01-01'
|
|
}
|
|
# run_prune <df-override> [settle-seconds]
|
|
#
|
|
# The settle window is only set when a scenario asks for it, so every other
|
|
# scenario — scenario 14 above all — runs against the script's own default
|
|
# rather than against a value this file chose.
|
|
run_prune() {
|
|
local free="${1:-}"
|
|
if [ -n "${2:-}" ]; then export EVICTION_ASIDE_SETTLE_SECONDS="$2"; else unset EVICTION_ASIDE_SETTLE_SECONDS; fi
|
|
CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
|
|
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
|
|
}
|
|
|
|
echo
|
|
echo "=== 1/2: dead pruned unconditionally; older-but-live survives ==="
|
|
reset_cache
|
|
run_prune "1000000 900000" # 90% free: no pressure at all
|
|
assert_gone "$root/target-$DEAD" "dead branch's target dir pruned with no disk pressure"
|
|
assert_gone "$root/snapshot-$DEAD" "dead branch's snapshot pruned too"
|
|
assert_kept "$root/target-$LIVE" "live branch survives despite an older marker than the dead one"
|
|
assert_log "no matching branch on origin" "eviction reason reported"
|
|
|
|
echo
|
|
echo "=== 3: protected refs never evicted under forced pressure ==="
|
|
reset_cache
|
|
run_prune "1000000 1000" # 0.1% free
|
|
assert_kept "$root/target-$DEV" "dev's target dir survives disk pressure"
|
|
assert_kept "$root/snapshot-$DEV" "dev's snapshot survives disk pressure"
|
|
assert_kept "$root/target-$MAIN" "main's target dir survives disk pressure"
|
|
assert_kept "$root/snapshot-$MAIN" "main's snapshot survives disk pressure"
|
|
|
|
echo
|
|
echo "=== 9: own cache never evicted by a sibling pass ==="
|
|
assert_kept "$root/target-$OWN" "this run's own cache survives"
|
|
|
|
echo
|
|
echo "=== 7: target dirs evicted before snapshots ==="
|
|
reset_cache
|
|
# Only the live branch is evictable; give it both a target dir and a snapshot
|
|
# with identical markers so ordering, not age, decides.
|
|
mk "snapshot-$LIVE" '2020-01-02'
|
|
# The df override is a fixed reading, so the pressure loop drains everything
|
|
# evictable — which is what makes the ORDER the observable property here, not
|
|
# what survives. Assert the eviction order directly from the log.
|
|
run_prune "1000000 1000"
|
|
order=$(grep -o "evicted \(target\|snapshot\)-$LIVE" "$scratch/log" | sed "s/evicted //")
|
|
[ "$(printf '%s\n' "$order" | head -1)" = "target-$LIVE" ] \
|
|
|| fail "expected target-$LIVE to be evicted before snapshot-$LIVE, got: $order"
|
|
ok "target dirs are evicted before snapshots"
|
|
|
|
echo
|
|
echo "=== 4: a fresh lock protects a dead, old, under-pressure cache ==="
|
|
reset_cache
|
|
date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$root/target-$DEAD" "locked cache survives both passes"
|
|
assert_log "held open by" "lock reported in the log"
|
|
# With the locked one the only dead cache left, the pass has declined every
|
|
# dead cache it found — at which point "no dead-branch caches found" is a
|
|
# false summary of the decline logged two lines above it.
|
|
rm -rf "$root/snapshot-$DEAD"
|
|
run_prune "1000000 900000"
|
|
assert_kept "$root/target-$DEAD" "still not evicted when it is the only dead cache"
|
|
assert_log "none pruned this pass" "a pass that declined every dead cache reports that"
|
|
if grep -q "no dead-branch caches found" "$scratch/log"; then
|
|
fail "the closing summary contradicts the decline logged above it"
|
|
fi
|
|
ok "the summary does not claim it found nothing"
|
|
|
|
echo
|
|
echo "=== 5: a stale lock is not honoured forever ==="
|
|
reset_cache
|
|
echo 0 > "$root/target-$DEAD/.ci-lock-ci-1"
|
|
touch -d '2000-01-01' "$root/target-$DEAD/.ci-lock-ci-1"
|
|
run_prune "1000000 900000"
|
|
assert_gone "$root/target-$DEAD" "cache with an abandoned lock is evicted"
|
|
assert_log "treating as abandoned" "abandoned lock reported in the log"
|
|
|
|
echo
|
|
echo "=== 6: liveness unavailable fails safe, pressure fallback still works ==="
|
|
reset_cache
|
|
(
|
|
cd "$work" && git remote set-url origin "$scratch/nonexistent.git"
|
|
)
|
|
run_prune "1000000 900000" # no pressure
|
|
assert_kept "$root/target-$DEAD" "dead cache NOT pruned when liveness is unavailable"
|
|
assert_log "treating as UNAVAILABLE" "unavailability reported plainly, not folded into 'no branches'"
|
|
run_prune "1000000 1000" # now with pressure
|
|
if [ -e "$root/target-$DEAD" ] && [ -e "$root/target-$LIVE" ]; then
|
|
fail "pressure fallback did nothing when liveness was unavailable"
|
|
fi
|
|
ok "pressure fallback still evicts when liveness is unavailable"
|
|
(cd "$work" && git remote set-url origin "$origin")
|
|
|
|
echo
|
|
echo "=== 8: self-clear reports to the job summary ==="
|
|
reset_cache
|
|
rm -rf "$root/target-$DEAD" "$root/snapshot-$DEAD" "$root/target-$LIVE"
|
|
: > "$scratch/summary"
|
|
run_prune "1000000 1000" # nothing evictable left but the run's own cache
|
|
assert_log "clearing own" "self-clear reported in the log"
|
|
grep -q 'self-clear' "$scratch/summary" || fail "self-clear missing from the job summary"
|
|
ok "self-clear reported to the job summary, not only the log"
|
|
[ -d "$root/target-$OWN" ] || fail "self-clear left the own directory missing"
|
|
[ -z "$(ls -A "$root/target-$OWN")" ] || fail "self-clear did not actually empty the directory"
|
|
ok "own cache wiped and recreated empty"
|
|
|
|
echo
|
|
echo "=== 10: scoped to the cache root ==="
|
|
reset_cache
|
|
decoy="$scratch/other-project"; mkdir -p "$decoy/target-$DEAD"; touch "$decoy/target-$DEAD/blob"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$decoy/target-$DEAD" "a cache outside the cache root is never touched"
|
|
|
|
echo
|
|
echo "=== 11: a live reader marker protects a cache, like a lock file does ==="
|
|
reset_cache
|
|
date +%s > "$root/.reading-target-$DEAD-job1"
|
|
run_prune "1000000 1000"
|
|
assert_kept "$root/target-$DEAD" "a cache being hardlink-cloned right now survives both passes"
|
|
assert_log "currently cloning it" "the reader is named in the log, not silently honoured"
|
|
rm -f "$root/.reading-target-$DEAD-job1"
|
|
|
|
echo
|
|
echo "=== 12: a reader marker published INSIDE the check-to-unlink window ==="
|
|
reset_cache
|
|
# A consumer publishes its marker whenever it starts a clone, which can be at
|
|
# any instant — including after the pass has checked for markers and before it
|
|
# unlinks. That window is real time, not a theoretical interleaving: `du -sk`
|
|
# on a multi-GB cache runs for seconds, and the pass runs one on every
|
|
# candidate. It is reproduced deterministically here by making that very `du`
|
|
# publish the marker, which places it strictly after the check and strictly
|
|
# before the unlink — exactly where a check-then-delete eviction cannot see
|
|
# it. The candidate must still be standing afterwards, with its contents
|
|
# intact and nothing left renamed aside.
|
|
mkdir -p "$scratch/bin"
|
|
real_du=$(command -v du)
|
|
cat > "$scratch/bin/du" <<EOF
|
|
#!/usr/bin/env bash
|
|
for arg; do
|
|
case "\$arg" in */target-$DEAD) date +%s > "$root/.reading-target-$DEAD-racer" ;; esac
|
|
done
|
|
exec "$real_du" "\$@"
|
|
EOF
|
|
chmod +x "$scratch/bin/du"
|
|
( PATH="$scratch/bin:$PATH"; run_prune "1000000 900000" )
|
|
[ -e "$root/.reading-target-$DEAD-racer" ] || fail "the racing marker was never published — scenario 12 proves nothing"
|
|
assert_kept "$root/target-$DEAD" "a cache claimed inside the eviction window is not unlinked"
|
|
assert_kept "$root/target-$DEAD/blob" "the reprieved cache still has its contents"
|
|
assert_log "restored, not evicted" "the reprieve is reported, not silent"
|
|
[ -z "$(ls -d "$root"/.evicting-* 2>/dev/null)" ] || fail "an aside directory was left behind after the reprieve"
|
|
ok "nothing left renamed aside once the eviction is declined"
|
|
rm -f "$root/.reading-target-$DEAD-racer"
|
|
|
|
echo
|
|
echo "=== 13: a deferred eviction is reclaimed, but not under a live reader ==="
|
|
reset_cache
|
|
aside="$root/.evicting-target-$DEAD-9999"
|
|
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
|
|
date +%s > "$root/.reading-target-$DEAD-job1"
|
|
# The settle window (scenario 14) gates this sweep first and would decide both
|
|
# runs on its own. A directory's ctime is what that window reads and cannot be
|
|
# backdated the way `touch -d` backdates an mtime, so the window is moved out
|
|
# of the way rather than the directory aged into it.
|
|
run_prune "1000000 900000" 0
|
|
assert_kept "$aside" "a deferred eviction is not reclaimed while a job is still reading it"
|
|
assert_log "deferring its reclamation again" "the continued deferral is reported"
|
|
rm -f "$root/.reading-target-$DEAD-job1"
|
|
run_prune "1000000 900000" 0
|
|
assert_gone "$aside" "a deferred eviction is reclaimed once its reader is gone"
|
|
assert_log "reclaiming deferred eviction" "the reclamation is reported"
|
|
|
|
echo
|
|
echo "=== 14: an aside another pass may still be evicting is left alone ==="
|
|
reset_cache
|
|
aside="$root/.evicting-target-$DEAD-9999"
|
|
# Built the way the pass builds one — an old cache directory renamed a moment
|
|
# ago — because the two ages that describe it disagree, and which of them the
|
|
# window reads is the whole mechanism. `mkdir`-ing the aside directly would
|
|
# give it a fresh mtime as well as a fresh ctime, and a fixture whose two
|
|
# clocks agree cannot tell %Z from %Y: the settle window would read the wrong
|
|
# one, never fire for any real aside, and this scenario would not notice.
|
|
victim="$root/target-$DEAD-victim"
|
|
mkdir -p "$victim"; head -c 4096 /dev/zero > "$victim/blob"
|
|
touch -d '2020-01-01' "$victim" # an old cache, which is every cache
|
|
mv -T "$victim" "$aside" # set aside a moment ago
|
|
# Deliberately no reader marker: the reader gate would pass this straight
|
|
# through, which is the whole point. An aside with no readers is exactly what
|
|
# a pass that has just renamed one aside and not yet decided about it looks
|
|
# like, and `rm -rf` traverses fd-relative — so reclaiming it out from under
|
|
# that pass lets it republish a half-emptied tree under a live cache name.
|
|
run_prune "1000000 900000"
|
|
assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
|
|
assert_kept "$aside/blob" "and is left intact, not part-way emptied"
|
|
assert_log "may still be evicting it" "the deferral gives its actual reason"
|
|
|
|
echo
|
|
echo "prune-cache-selftest: ${pass_count} assertions passed"
|