fix(prune): reclaim merged branches, and size the volume for the clone
Two assumptions in the eviction pass did not hold on this forge, and between them a volume filled up three times in three days with nothing reclaimed automatically. Both are replaced here; the pass also moves ahead of the seed, which is the only order in which its work can help the run performing it. LIVENESS. Pass 1 evicted a cache only when its branch was gone from origin. Gitea keeps a PR's branch after the merge unless the repo opts into delete-on-merge, and zemyna does not — so ls-remote reports fifty merged branches and the signal fires for none of them. A second signal is added beside it: a branch still on origin whose tip is an ancestor of a protected branch's tip holds no commit that branch does not, so its cache will never be read again and goes in the same unconditional pass. Ancestry is answered from the commits in the job's own checkout, so the answer "cannot tell" exists and stays distinct from "not merged" at both granularities. A shallow checkout withholds the signal entirely, since a missing object is its normal case rather than evidence. A single branch whose tip is not in the checkout is kept, with a warning naming it. A squash or rebase merge leaves no ancestry and reads as live until the branch is deleted. All three are missed reclamations, which cost disk; the other direction costs a branch its cache mid-build. HEADROOM. Passes 2 and 3 gated on a percentage of the volume, which cannot express the failure they have to prevent: a clone runs out of disk while unsharing its mutable paths, and how much that needs is a property of the snapshot rather than of the disk. Staging failed at 34 G free and passed at 74 G, so a 10% floor — 19 G here — never fired first. The requirement is now measured per run off the very source the seed will read, the pass evicts oldest-first until it is met and stops there, and falling short of it fails with the shortfall and every directory it kept, rather than letting the seed fail seconds later against a staging path that names none of that. min-free-percent survives as an additional floor, defaulting to 0, and falling short of that one is still a warning and a self-clear. ORDERING. The prune step ran after the seed, so each run freed space for the next one. It now runs between resolve and seed. Two things that makes newly reachable are closed: the source about to be cloned is excluded from every pass by name, and a concurrent job's target dir already carries its lock from the instant it appears under its final name, so nothing is seen unlocked that is in use. Red-proven: sixteen assertions across the four new scenarios fail against the pre-fix scripts, including the zemyna layout evicting nothing where it should evict exactly one directory, and the headroom scenario exiting 0 where it should exit 1. Refs #20. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JXMQCJ5Eg5f9G9cfYzyh4Z
This commit is contained in:
Claude Fable 5.1
@@ -46,10 +46,36 @@
|
||||
# empty a tree its owner may still restore under a live cache name. Its
|
||||
# fixture is an OLD directory renamed a moment ago — production's shape,
|
||||
# and what lets it tell the two timestamps apart.
|
||||
# 15. A MERGED-BUT-UNDELETED BRANCH IS DEAD TOO. This forge keeps branches
|
||||
# after merge, so `ls-remote` reports them forever and scenario 1's
|
||||
# signal never fires for them — which is how three 40 GB caches sat on a
|
||||
# full volume until somebody removed them by hand (gitdan-actions#20). A
|
||||
# branch whose tip is an ancestor of a protected branch's tip is pruned
|
||||
# like a deleted one; an unmerged branch beside it is not.
|
||||
# 16. AND "CANNOT TELL" IS STILL NOT DEATH, at both granularities: a branch
|
||||
# whose tip is not in this checkout is kept with a warning naming it,
|
||||
# and a shallow checkout — where a missing object is the normal case —
|
||||
# withholds the whole signal rather than reading it as "nothing merged".
|
||||
# The deleted-branch signal keeps working in both.
|
||||
# 17. THE FREE-SPACE REQUIREMENT IS MEASURED OFF THE SOURCE, not taken as a
|
||||
# percentage of the volume: the pass evicts until the clone the seed is
|
||||
# about to make fits, and stops there rather than draining the volume.
|
||||
# When it cannot get there it FAILS, naming the shortfall and every
|
||||
# directory it kept instead — because the seed would otherwise fail
|
||||
# seconds later against a staging path that names nothing.
|
||||
# 18. AND ON THE LAYOUT THAT PRODUCED THE BUG: three equal-sized caches, one
|
||||
# of them a merged-but-undeleted branch's, with disk to spare. Exactly
|
||||
# that one goes. Equal sizes and no pressure are the point — nothing but
|
||||
# the merge state can be what decides.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
prune="$script_dir/prune-cache.sh"
|
||||
# Two scenarios below put a stub of a real tool on PATH for one command.
|
||||
# Captured once, here, rather than read back at each of those sites: a `$PATH`
|
||||
# read after the first of them is indistinguishable, to a static check, from
|
||||
# reading the modification the subshell lost.
|
||||
outer_path="$PATH"
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
@@ -235,7 +261,7 @@ done
|
||||
exec "$real_du" "\$@"
|
||||
EOF
|
||||
chmod +x "$scratch/bin/du"
|
||||
( PATH="$scratch/bin:$PATH"; run_prune "1000000 900000" )
|
||||
( PATH="$scratch/bin:$outer_path"; run_prune "1000000 900000" )
|
||||
[ -e "$root/.reading-target-$DEAD-racer" ] || fail "the racing marker was never published — scenario 12 proves nothing"
|
||||
assert_kept "$root/target-$DEAD" "a cache claimed inside the eviction window is not unlinked"
|
||||
assert_kept "$root/target-$DEAD/blob" "the reprieved cache still has its contents"
|
||||
@@ -286,5 +312,172 @@ assert_kept "$aside" "an aside younger than the settle window is not reclaimed"
|
||||
assert_kept "$aside/blob" "and is left intact, not part-way emptied"
|
||||
assert_log "may still be evicting it" "the deferral gives its actual reason"
|
||||
|
||||
echo
|
||||
echo "=== 15: a merged-but-undeleted branch is dead too ==="
|
||||
# This forge keeps a PR's branch after the merge, so `ls-remote` reports it
|
||||
# forever. Built the way that happens: a branch merged into dev with a merge
|
||||
# commit, still pushed, beside one branched at the same point and NOT merged.
|
||||
git="git -c user.email=t@t -c user.name=t -c commit.gpgsign=false"
|
||||
(
|
||||
cd "$work"
|
||||
git checkout -q dev
|
||||
git checkout -q -b feat/merged
|
||||
$git commit -q --allow-empty -m merged
|
||||
git checkout -q dev
|
||||
$git merge -q --no-ff feat/merged -m "merge feat/merged"
|
||||
git checkout -q -b feat/unmerged
|
||||
$git commit -q --allow-empty -m unmerged
|
||||
git checkout -q dev
|
||||
git push -q origin dev feat/merged feat/unmerged
|
||||
)
|
||||
MERGED=$(cache_key feat/merged); UNMERGED=$(cache_key feat/unmerged)
|
||||
reset_cache
|
||||
mk "target-$MERGED" '2030-01-01'
|
||||
mk "snapshot-$MERGED" '2030-01-01'
|
||||
mk "target-$UNMERGED" '2020-01-01' # older, deliberately: merge state decides, not age
|
||||
run_prune "1000000 900000" # 90% free: no pressure at all
|
||||
assert_log "merged-branch detection anchored on" "the pass says what it anchored ancestry on"
|
||||
assert_gone "$root/target-$MERGED" "a merged branch's cache is pruned though its branch is still on origin"
|
||||
assert_gone "$root/snapshot-$MERGED" "and so is its snapshot"
|
||||
assert_kept "$root/target-$UNMERGED" "an unmerged branch's cache survives, though it is the older of the two"
|
||||
assert_log "merged into dev" "the eviction names the branch it was merged into"
|
||||
|
||||
echo
|
||||
echo "=== 16: 'cannot tell' is not death, per branch and per checkout ==="
|
||||
# A branch whose tip this checkout has never seen. Pushed from a second clone,
|
||||
# so `ls-remote` reports a SHA that `$work` holds no object for — which is
|
||||
# what "cannot determine" actually looks like, rather than a stubbed failure.
|
||||
other="$scratch/other"; git clone -q "$origin" "$other"
|
||||
(
|
||||
cd "$other"
|
||||
git checkout -q -b feat/elsewhere origin/dev
|
||||
git -c user.email=t@t -c user.name=t -c commit.gpgsign=false commit -q --allow-empty -m elsewhere
|
||||
git push -q origin feat/elsewhere
|
||||
)
|
||||
ELSEWHERE=$(cache_key feat/elsewhere)
|
||||
reset_cache
|
||||
mk "target-$ELSEWHERE" '2030-01-01'
|
||||
mk "target-$MERGED" '2030-01-01'
|
||||
run_prune "1000000 900000"
|
||||
assert_kept "$root/target-$ELSEWHERE" "a branch whose tip is not in this checkout is kept, not classified dead"
|
||||
assert_log "cannot tell merged from live" "and the undecidable branch is named, not silently skipped"
|
||||
assert_gone "$root/target-$MERGED" "while a branch it CAN decide is still pruned in the same pass"
|
||||
|
||||
# A shallow checkout, where a missing object is the ordinary case rather than
|
||||
# a signal — so the whole merged half is withheld. The deleted-branch half is
|
||||
# unaffected, which is what keeps this a narrowing rather than an outage.
|
||||
shallow="$scratch/shallow"; git clone -q --depth 1 -b dev "file://$origin" "$shallow"
|
||||
[ "$(git -C "$shallow" rev-parse --is-shallow-repository)" = true ] \
|
||||
|| fail "the fixture clone is not shallow — scenario 16's second half proves nothing"
|
||||
reset_cache
|
||||
mk "target-$MERGED" '2030-01-01'
|
||||
(
|
||||
cd "$shallow"
|
||||
CACHE_DF_OVERRIDE="1000000 900000" GITHUB_STEP_SUMMARY="$scratch/summary" \
|
||||
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10
|
||||
) > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "prune-cache.sh exited non-zero in a shallow checkout"; }
|
||||
assert_log "checkout is shallow" "a shallow checkout withholds the merged signal and says why"
|
||||
assert_kept "$root/target-$MERGED" "and keeps a merged branch's cache rather than guessing"
|
||||
assert_gone "$root/target-$DEAD" "while the deleted-branch signal still fires"
|
||||
|
||||
echo
|
||||
echo "=== 17: the free-space requirement is measured off the clone's source ==="
|
||||
# A `df` that answers from the cache root's actual size, because the property
|
||||
# under test is that the pass STOPS once the requirement is met — which a
|
||||
# fixed CACHE_DF_OVERRIDE cannot express, since evicting never changes it.
|
||||
mkdir -p "$scratch/bin17"
|
||||
real_du=$(command -v du)
|
||||
build_seed_fixture() {
|
||||
rm -rf "$root"; mkdir -p "$root"
|
||||
# The source the seed is about to clone. 8 MB of dep-info, which
|
||||
# unshare_mutable_paths has to real-copy, beside 16 MB of .rlib that it
|
||||
# leaves hardlinked — so a requirement derived from the SIZE of the source
|
||||
# would be three times the one derived from its mutable set.
|
||||
mkdir -p "$root/snapshot-$DEV/debug/.fingerprint/unit" "$root/snapshot-$DEV/debug/deps"
|
||||
head -c $((8 * 1024 * 1024)) /dev/zero > "$root/snapshot-$DEV/debug/.fingerprint/unit/dep-lib"
|
||||
head -c $((16 * 1024 * 1024)) /dev/zero > "$root/snapshot-$DEV/debug/deps/libx.rlib"
|
||||
touch -d '2020-01-01' "$root/snapshot-$DEV/.cache-last-used"
|
||||
# Three live, unmerged branches' caches of 4 MB each, oldest first.
|
||||
local i=0
|
||||
for b in a b c; do
|
||||
i=$((i + 1))
|
||||
mkdir -p "$root/target-$(cache_key "feat/$b")"
|
||||
head -c $((4 * 1024 * 1024)) /dev/zero > "$root/target-$(cache_key "feat/$b")/blob"
|
||||
touch -d "202${i}-01-01" "$root/target-$(cache_key "feat/$b")/.cache-last-used"
|
||||
done
|
||||
# A volume with 2 MB to spare: under the requirement, over nothing else.
|
||||
cap=$(( $($real_du -sk "$root" | awk '{print $1}') + 2048 ))
|
||||
cat > "$scratch/bin17/df" <<DFEOF
|
||||
#!/usr/bin/env bash
|
||||
used=\$($real_du -sk "$root" | awk '{print \$1}')
|
||||
echo "Filesystem 1024-blocks Used Available Capacity Mounted-on"
|
||||
echo "fake $cap \$used \$(( $cap - used )) 50% $root"
|
||||
DFEOF
|
||||
chmod +x "$scratch/bin17/df"
|
||||
}
|
||||
(
|
||||
cd "$work"
|
||||
for b in a b c; do
|
||||
git checkout -q dev
|
||||
git checkout -q -b "feat/$b"
|
||||
git -c user.email=t@t -c user.name=t -c commit.gpgsign=false commit -q --allow-empty -m "$b"
|
||||
done
|
||||
git checkout -q dev
|
||||
git push -q origin feat/a feat/b feat/c
|
||||
)
|
||||
# run_seeded_prune <own-ref> <base-ref> — the form cargo-cache/action.yml uses:
|
||||
# the same pass, told what the seed step it now runs ahead of will clone. The
|
||||
# headroom knobs are pinned so the arithmetic is the fixture's, not the
|
||||
# defaults' (whose 2 GiB floor would dwarf any fixture on a test host).
|
||||
seeded_rc=0
|
||||
run_seeded_prune() {
|
||||
seeded_rc=0
|
||||
PATH="$scratch/bin17:$outer_path" \
|
||||
CACHE_CLONE_HEADROOM_PERCENT=100 CACHE_CLONE_HEADROOM_FLOOR_KB=1024 \
|
||||
GITHUB_STEP_SUMMARY="$scratch/summary" \
|
||||
bash "$prune" "$root" "$root/target-$(cache_key "$1")" "dev main" 0 \
|
||||
"$(cache_key "$1")" "$(cache_key "$2")" "" \
|
||||
> "$scratch/log" 2>&1 || seeded_rc=$?
|
||||
}
|
||||
|
||||
build_seed_fixture
|
||||
run_seeded_prune feat/own dev
|
||||
[ "$seeded_rc" = 0 ] || { cat "$scratch/log"; fail "prune-cache.sh exited ${seeded_rc} with the requirement satisfiable"; }
|
||||
assert_log "measured from its mutable set" "the requirement says where it came from"
|
||||
assert_gone "$root/target-$(cache_key feat/a)" "the oldest cache is evicted to make room for the clone"
|
||||
assert_gone "$root/target-$(cache_key feat/b)" "and the next oldest, because one was not enough"
|
||||
assert_kept "$root/target-$(cache_key feat/c)" "and the pass STOPS there rather than draining the volume"
|
||||
assert_kept "$root/snapshot-$DEV" "the source the seed is about to clone is never a candidate"
|
||||
|
||||
# Nothing eligible: every sibling is held open by a running job. The pass
|
||||
# cannot reach the requirement, and the seed that follows would fail against a
|
||||
# staging path naming none of this.
|
||||
build_seed_fixture
|
||||
for b in a b c; do date +%s > "$root/target-$(cache_key "feat/$b")/.ci-lock-ci-1"; done
|
||||
run_seeded_prune feat/own dev
|
||||
[ "$seeded_rc" = 1 ] || { cat "$scratch/log"; fail "expected exit 1 when the clone cannot fit, got ${seeded_rc}" ; }
|
||||
ok "a clone that cannot be made to fit fails the pass rather than the seed"
|
||||
assert_log "short by" "the failure names the shortfall"
|
||||
assert_log "held open by a running job" "and what was kept instead of it, with the reason"
|
||||
assert_kept "$root/target-$(cache_key feat/a)" "a locked cache is still not evicted, however tight the disk"
|
||||
|
||||
echo
|
||||
echo "=== 18: the layout that produced the bug ==="
|
||||
# zemyna's volume on 2026-09-07: the base branch's snapshot and target dir,
|
||||
# plus one target dir for a branch merged the day before and never deleted.
|
||||
# Equal sizes and 90% free, so neither age nor pressure nor size can be what
|
||||
# decides — only the merge state.
|
||||
rm -rf "$root"; mkdir -p "$root"
|
||||
mk "snapshot-$DEV" '2026-09-01'
|
||||
mk "target-$DEV" '2026-09-01'
|
||||
mk "target-$MERGED" '2026-09-06'
|
||||
run_prune "1000000 900000"
|
||||
assert_kept "$root/snapshot-$DEV" "the base snapshot stays"
|
||||
assert_kept "$root/target-$DEV" "and the base target dir stays"
|
||||
assert_gone "$root/target-$MERGED" "and the merged-but-undeleted branch's cache is the one reclaimed"
|
||||
[ "$(grep -c 'pruned dead-branch cache' "$scratch/log")" = 1 ] \
|
||||
|| { cat "$scratch/log"; fail "expected exactly one eviction on the zemyna layout"; }
|
||||
ok "exactly one directory is evicted, and it is that one"
|
||||
|
||||
echo
|
||||
echo "prune-cache-selftest: ${pass_count} assertions passed"
|
||||
|
||||
Reference in New Issue
Block a user