fix(prune-cache): stop the aside sweeper depending on timing it cannot see
Review findings on #2. The first is the one that mattered: the sweeper this PR added had the shape the PR exists to remove. Pass A renames a candidate aside; pass B's sweep sees an aside with no readers and reclaims it; A then finds a reader and restores. `rm -rf` traverses fd-relative, so the rename does not stop it and A can republish a half-emptied tree under a live cache name. `capacity: 1` bounds it today, which is exactly the kind of reason this PR was written to stop relying on. The unlink itself was never the problem — the ordering proof covers it under any interleaving, since the aside name only exists after the evicting pass's rename. What was missing is that an aside with no readers is indistinguishable from one a pass has just created and not yet decided about. The sweeper now leaves an aside alone until it has settled (EVICTION_ASIDE_SETTLE_SECONDS, default 60), which separates the two without having to identify the pass that created it — a PID is meaningless across the job containers these passes run in, and recycles. Read from ctime, not mtime: rename(2) updates the first and leaves the second at whenever the cache was last written, which is the signal list_by_lru wants and the wrong one here. That is a bound, not a construction, and both the code comment and the README now say which of the two properties is which instead of asserting the broader one. Also from the review: a pass that declined every dead cache it found no longer signs off with "no dead-branch caches found", and evict_dir no longer promises a later reclamation of an aside that is already gone. Scenario 14 covers the settle window against the script's own default, with nothing faked — the directory really was set aside a moment ago. Scenario 4 gains the summary assertion. 31 -> 37 assertions; each new gate verified red by defeating it alone in a scratch copy.
This commit is contained in:
@@ -13,7 +13,9 @@
|
||||
# not age, is what decides pass 1.
|
||||
# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
|
||||
# their caches are the oldest on disk and would rank first for LRU.
|
||||
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure.
|
||||
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure — and
|
||||
# the pass's closing summary agrees with the decline it just logged,
|
||||
# rather than reporting that it found nothing.
|
||||
# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
|
||||
# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
|
||||
# permanently.
|
||||
@@ -38,6 +40,10 @@
|
||||
# 13. A DEFERRED EVICTION IS RECLAIMED, but not while its reader is live.
|
||||
# Nothing else globs a dotted name, so an unswept one is disk lost for
|
||||
# good on the volume whose whole problem is disk.
|
||||
# 14. AND NOT WHILE ANOTHER PASS MAY STILL BE EVICTING IT. An aside with no
|
||||
# readers is indistinguishable from one a concurrent pass has just
|
||||
# renamed and not yet decided about; reclaiming that one lets `rm -rf`
|
||||
# empty a tree its owner may still restore under a live cache name.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
@@ -82,8 +88,14 @@ reset_cache() {
|
||||
mk "snapshot-$DEAD" '2030-01-01'
|
||||
mk "target-$OWN" '2025-01-01'
|
||||
}
|
||||
# run_prune <df-override> [settle-seconds]
|
||||
#
|
||||
# The settle window is only set when a scenario asks for it, so every other
|
||||
# scenario — scenario 14 above all — runs against the script's own default
|
||||
# rather than against a value this file chose.
|
||||
run_prune() {
|
||||
local free="${1:-}"
|
||||
if [ -n "${2:-}" ]; then export EVICTION_ASIDE_SETTLE_SECONDS="$2"; else unset EVICTION_ASIDE_SETTLE_SECONDS; fi
|
||||
CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
|
||||
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
|
||||
|| { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
|
||||
@@ -133,6 +145,17 @@ date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
|
||||
run_prune "1000000 1000"
|
||||
assert_kept "$root/target-$DEAD" "locked cache survives both passes"
|
||||
assert_log "held open by" "lock reported in the log"
|
||||
# With the locked one the only dead cache left, the pass has declined every
|
||||
# dead cache it found — at which point "no dead-branch caches found" is a
|
||||
# false summary of the decline logged two lines above it.
|
||||
rm -rf "$root/snapshot-$DEAD"
|
||||
run_prune "1000000 900000"
|
||||
assert_kept "$root/target-$DEAD" "still not evicted when it is the only dead cache"
|
||||
assert_log "none pruned this pass" "a pass that declined every dead cache reports that"
|
||||
if grep -q "no dead-branch caches found" "$scratch/log"; then
|
||||
fail "the closing summary contradicts the decline logged above it"
|
||||
fi
|
||||
ok "the summary does not claim it found nothing"
|
||||
|
||||
echo
|
||||
echo "=== 5: a stale lock is not honoured forever ==="
|
||||
@@ -225,13 +248,33 @@ reset_cache
|
||||
aside="$root/.evicting-target-$DEAD-9999"
|
||||
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
|
||||
date +%s > "$root/.reading-target-$DEAD-job1"
|
||||
run_prune "1000000 900000"
|
||||
# The settle window (scenario 14) gates this sweep first and would decide both
|
||||
# runs on its own. A directory's ctime is what that window reads and cannot be
|
||||
# backdated the way `touch -d` backdates an mtime, so the window is moved out
|
||||
# of the way rather than the directory aged into it.
|
||||
run_prune "1000000 900000" 0
|
||||
assert_kept "$aside" "a deferred eviction is not reclaimed while a job is still reading it"
|
||||
assert_log "deferring its reclamation again" "the continued deferral is reported"
|
||||
rm -f "$root/.reading-target-$DEAD-job1"
|
||||
run_prune "1000000 900000"
|
||||
run_prune "1000000 900000" 0
|
||||
assert_gone "$aside" "a deferred eviction is reclaimed once its reader is gone"
|
||||
assert_log "reclaiming deferred eviction" "the reclamation is reported"
|
||||
|
||||
echo
|
||||
echo "=== 14: an aside another pass may still be evicting is left alone ==="
|
||||
reset_cache
|
||||
aside="$root/.evicting-target-$DEAD-9999"
|
||||
mkdir -p "$aside"; head -c 4096 /dev/zero > "$aside/blob"
|
||||
# Deliberately no reader marker: the reader gate would pass this straight
|
||||
# through, which is the whole point. An aside with no readers is exactly what
|
||||
# a pass that has just renamed one aside and not yet decided about it looks
|
||||
# like, and `rm -rf` traverses fd-relative — so reclaiming it out from under
|
||||
# that pass lets it republish a half-emptied tree under a live cache name.
|
||||
# Nothing here fakes an age: the directory really was set aside a moment ago.
|
||||
run_prune "1000000 900000"
|
||||
assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
|
||||
assert_kept "$aside/blob" "and is left intact, not part-way emptied"
|
||||
assert_log "may still be evicting it" "the deferral gives its actual reason"
|
||||
|
||||
echo
|
||||
echo "prune-cache-selftest: ${pass_count} assertions passed"
|
||||
|
||||
Reference in New Issue
Block a user