fix(prune): stop protecting publisher target dirs under disk pressure
CI / shellcheck + selftests (pull_request) Successful in 1m48s
CI / shellcheck + selftests (pull_request) Successful in 1m48s
A publisher branch's target-<ref> was protected identically to its snapshot-<ref>, so it was never a pressure-pass candidate however old and however tight the disk. With one branch's target dir permanently resident alongside its snapshot, a second branch had no room to seed, and every PR that night needed a hand eviction between runs (daniel/gitdan-actions#24). A publisher's target dir is a convenience cache its own next run reseeds from the snapshot, so losing it under pressure is cheap; nothing downstream depends on it surviving. Only the snapshot stays protected in the pressure and self-clear passes. Unprotecting the target dir outright surfaced a second bug the fix would otherwise have shipped: a branch's tip is trivially an ancestor of itself, so once a protected ref's target dir was no longer skipped before reaching the merged-branch check, pass 1 read it as "merged into itself" and deleted it unconditionally on every run, independent of disk pressure. is_protected_from_liveness keeps a protected ref's target dir out of pass 1 alone, so it stays an ordinary pressure-pass candidate without ever reaching that check. Scenario 3b in the selftest red-proves this against the unprotect-only version of the fix. Also corrects the header's inode-sharing claim, measured false on the live volume by daniel/zemyna#1073: publish-snapshot.sh unshares every executable after its cp -al, and executables are most of the tree by bytes, so a snapshot eviction is a real, large disk cost rather than the near-free one the old text described — the target-before-snapshot ordering still holds, now for the warm-start reason alone plus that cost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UkSjXXtU6JYcN2vPntWhfb
This commit is contained in:
+54
-23
@@ -67,21 +67,32 @@
|
||||
# physics, not an arbitrary GB number.
|
||||
#
|
||||
# EVICTION ORDER, and why it is the reverse of the obvious one: within the
|
||||
# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones.
|
||||
# A snapshot is a hardlink clone of a live target dir and of every consumer
|
||||
# cloned from it, so removing it frees almost no real bytes — its inodes stay
|
||||
# alive through those other links — while costing every future PR its warm
|
||||
# start. Evicting snapshots first would be nearly pure loss. Target
|
||||
# directories are where a branch's own divergent artifacts actually live, so
|
||||
# they are what freeing space means.
|
||||
# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones
|
||||
# (a publisher's own target dir included — see PROTECTED below). A publisher
|
||||
# branch's target dir is a convenience cache that its own next run reseeds
|
||||
# from the snapshot, so losing it is cheap; evicting the snapshot instead
|
||||
# forces every subsequent PR to start cold. Measured on the live volume
|
||||
# (daniel/zemyna#1073), that cold start is not the near-free move it looks
|
||||
# like either: a snapshot shares almost nothing with the target dirs cloned
|
||||
# from it, because publish-snapshot.sh unshares every executable after its
|
||||
# `cp -al` (cargo and the linker rewrite binaries in place, so a shared
|
||||
# original would corrupt under them), and executables are the great majority
|
||||
# of the tree by bytes. So a snapshot eviction is a real, large disk cost as
|
||||
# well as a cold-start one — reserved for last because both costs are larger
|
||||
# than a target dir's.
|
||||
#
|
||||
# Two exclusions every pass respects:
|
||||
#
|
||||
# PROTECTED — the publisher branches' target and snapshot directories, and
|
||||
# this run's own target dir, are never candidates in any pass. Evicting a
|
||||
# publisher's snapshot doesn't free real disk (every open PR's clone keeps
|
||||
# the data alive) but does force every subsequent PR to start cold, which is
|
||||
# the entire benefit this scheme exists to deliver.
|
||||
# PROTECTED — a publisher branch's SNAPSHOT directory, and this run's own
|
||||
# target dir, are never candidates in the pressure or self-clear passes. A
|
||||
# publisher branch's own TARGET dir is not protected there: it is an
|
||||
# ordinary pressure-pass candidate, evicted oldest-first like any other,
|
||||
# because nothing downstream depends on it surviving — the publisher's own
|
||||
# next run reseeds it from the snapshot. It IS excluded from pass 1 alone
|
||||
# (is_protected_from_liveness): a branch's tip is trivially an ancestor of
|
||||
# itself, so without this exclusion the merged-branch signal would read a
|
||||
# publisher's own target dir as "merged into itself" and pass 1 —
|
||||
# unconditional, not gated on pressure — would delete it every run.
|
||||
#
|
||||
# LOCKED — a directory carrying a .ci-lock-* marker younger than
|
||||
# STALE_LOCK_SECONDS is held open by a running job, or named by a live
|
||||
@@ -180,32 +191,52 @@ else
|
||||
fi
|
||||
|
||||
declare -A protected_ns=()
|
||||
# A protected ref's own target dir is excluded from pass 1 ONLY (see
|
||||
# is_protected_from_liveness below), never from the pressure pass. It must
|
||||
# stay out of pass 1 for a reason that has nothing to do with disk: a
|
||||
# protected ref's tip is trivially an ancestor of itself, so without this
|
||||
# is_merged_dead would read dev's own target dir as "merged into dev" and
|
||||
# pass 1 — unconditional, not gated on pressure — would delete it on every
|
||||
# single run.
|
||||
declare -A protected_target_ns=()
|
||||
for ref in $PROTECTED_REFS; do
|
||||
suffix=$(cache_key "$ref")
|
||||
protected_ns["target-${suffix}"]=1
|
||||
protected_ns["snapshot-${suffix}"]=1
|
||||
protected_target_ns["target-${suffix}"]=1
|
||||
done
|
||||
|
||||
# Prints why <dir> is off limits to every pass, or nothing when it is a
|
||||
# candidate. The reason is not decoration: it is what the failure report at
|
||||
# the bottom lists against each directory it kept while running out of space.
|
||||
# Prints why <dir> is off limits to the pressure/self-clear passes, or
|
||||
# nothing when it is a candidate there. The reason is not decoration: it is
|
||||
# what the failure report at the bottom lists against each directory it kept
|
||||
# while running out of space.
|
||||
#
|
||||
# SEED_SRC is the third exclusion and the one this script did not used to need.
|
||||
# The prune ran after the seed, so the source had already been cloned and the
|
||||
# reader marker over it was gone; running BEFORE the seed puts the directory
|
||||
# this run is about to read squarely in the candidate set, and pass 1 would
|
||||
# take it the moment its branch merged.
|
||||
# SEED_SRC is the third exclusion and the one this script did not used to
|
||||
# need. The prune ran after the seed, so the source had already been cloned
|
||||
# and the reader marker over it was gone; running BEFORE the seed puts the
|
||||
# directory this run is about to read squarely in the candidate set, and
|
||||
# pass 1 would take it the moment its branch merged.
|
||||
protected_reason() {
|
||||
local dir="$1" name
|
||||
name=$(basename "$dir")
|
||||
[ "$dir" = "$OWN_DIR" ] && { printf 'this run own cache'; return 0; }
|
||||
[ -n "$SEED_SRC" ] && [ "$dir" = "$SEED_SRC" ] && { printf 'the source this run is about to clone'; return 0; }
|
||||
[ -n "${protected_ns[$name]:-}" ] && { printf 'a protected branch cache'; return 0; }
|
||||
[ -n "${protected_ns[$name]:-}" ] && { printf 'a protected branch snapshot'; return 0; }
|
||||
return 1
|
||||
}
|
||||
|
||||
is_protected() { protected_reason "$1" >/dev/null; }
|
||||
|
||||
# Pass 1 (liveness) only: also excludes a protected ref's own target dir, for
|
||||
# the self-ancestor reason above protected_target_ns documents. The pressure
|
||||
# pass does not call this — is_protected is what it uses, via protected_reason
|
||||
# directly.
|
||||
is_protected_from_liveness() {
|
||||
local dir="$1" name
|
||||
is_protected "$dir" && return 0
|
||||
name=$(basename "$dir")
|
||||
[ -n "${protected_target_ns[$name]:-}" ]
|
||||
}
|
||||
|
||||
# is_locked <dir> [name]
|
||||
#
|
||||
# `name` is the directory's own name for reporting and for the reader-marker
|
||||
@@ -480,7 +511,7 @@ if [ "$LIVENESS_AVAILABLE" = "1" ]; then
|
||||
for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do
|
||||
[ -d "$dir" ] || continue
|
||||
name=$(basename "$dir")
|
||||
is_protected "$dir" && continue
|
||||
is_protected_from_liveness "$dir" && continue
|
||||
if [ -z "${live_ns[$name]:-}" ]; then
|
||||
why="no matching branch on origin"
|
||||
why_summary="branch no longer exists on origin"
|
||||
|
||||
Reference in New Issue
Block a user