Files
gitdan-actions/scripts/prune-cache.sh
T
claudeandClaude Fable 5.1 07ba53ca79 fix(prune): reclaim merged branches, and size the volume for the clone
Two assumptions in the eviction pass did not hold on this forge, and
between them a volume filled up three times in three days with nothing
reclaimed automatically. Both are replaced here; the pass also moves
ahead of the seed, which is the only order in which its work can help
the run performing it.

LIVENESS. Pass 1 evicted a cache only when its branch was gone from
origin. Gitea keeps a PR's branch after the merge unless the repo opts
into delete-on-merge, and zemyna does not — so ls-remote reports fifty
merged branches and the signal fires for none of them. A second signal
is added beside it: a branch still on origin whose tip is an ancestor of
a protected branch's tip holds no commit that branch does not, so its
cache will never be read again and goes in the same unconditional pass.

Ancestry is answered from the commits in the job's own checkout, so the
answer "cannot tell" exists and stays distinct from "not merged" at both
granularities. A shallow checkout withholds the signal entirely, since a
missing object is its normal case rather than evidence. A single branch
whose tip is not in the checkout is kept, with a warning naming it. A
squash or rebase merge leaves no ancestry and reads as live until the
branch is deleted. All three are missed reclamations, which cost disk;
the other direction costs a branch its cache mid-build.

HEADROOM. Passes 2 and 3 gated on a percentage of the volume, which
cannot express the failure they have to prevent: a clone runs out of
disk while unsharing its mutable paths, and how much that needs is a
property of the snapshot rather than of the disk. Staging failed at 34 G
free and passed at 74 G, so a 10% floor — 19 G here — never fired first.
The requirement is now measured per run off the very source the seed
will read, the pass evicts oldest-first until it is met and stops there,
and falling short of it fails with the shortfall and every directory it
kept, rather than letting the seed fail seconds later against a staging
path that names none of that. min-free-percent survives as an additional
floor, defaulting to 0, and falling short of that one is still a warning
and a self-clear.

ORDERING. The prune step ran after the seed, so each run freed space for
the next one. It now runs between resolve and seed. Two things that
makes newly reachable are closed: the source about to be cloned is
excluded from every pass by name, and a concurrent job's target dir
already carries its lock from the instant it appears under its final
name, so nothing is seen unlocked that is in use.

Red-proven: sixteen assertions across the four new scenarios fail
against the pre-fix scripts, including the zemyna layout evicting
nothing where it should evict exactly one directory, and the headroom
scenario exiting 0 where it should exit 1.

Refs #20.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JXMQCJ5Eg5f9G9cfYzyh4Z
2026-09-07 00:21:12 -05:00

609 lines
29 KiB
Bash
Executable File

#!/usr/bin/env bash
# Eviction for the per-ref cache directories on the persistent volume.
#
# Usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> \
# <min-free-percent> [own-key] [base-key] [fallback-dir]
# protected-branches space-separated raw refs (e.g. "dev main")
# min-free-percent a FLOOR, not the gate — 0 to rely on the derived
# requirement alone (the default)
# own-key, base-key, fallback-dir
# the same three the seed step resolves its source from.
# Given them, this pass sizes the volume for the clone
# that step is about to make; without them it has only
# the percentage floor, and says so.
#
# Optional environment:
# STALE_LOCK_SECONDS age past which a .ci-lock-* marker is treated as
# abandoned (default 7200)
# CACHE_LIVENESS "false"/"0" to skip the liveness pass entirely
# CACHE_DF_OVERRIDE "<total_kb> <free_kb>", for the selftest
# EVICTION_ASIDE_SETTLE_SECONDS
# how long a directory renamed aside for eviction is
# left alone before another pass may reclaim it
# (default 60)
#
# Three passes, in order:
#
# 1. LIVENESS — every target-*/snapshot-* directory whose branch is DEAD is
# removed UNCONDITIONALLY, not gated on free space. A directory for a
# branch nothing will build again is pure loss; waiting for disk pressure
# to notice means paying for it until then. Two signals make a branch
# dead, and the second exists because the first alone is inert on a forge
# that keeps branches after merge (gitdan-actions#20):
#
# DELETED — the branch is no longer on origin at all.
# MERGED — the branch is still on origin, but its tip is an ancestor
# of a protected branch's tip, so every commit it holds is
# already on the branch its cache would be re-cloned from.
#
# Skipped entirely, loudly, if the signal itself is unavailable —
# "couldn't determine" is never folded into "dead", for either signal
# and at either granularity: a checkout that cannot answer ancestry at
# all skips the merged half, and a single branch whose tip is not in the
# checkout is kept with a warning naming it.
# 2. PRESSURE — if free space is under the requirement, evict remaining
# (now necessarily live) directories oldest-first until it clears. The
# requirement is what the seed step is about to need to clone its source,
# MEASURED off that source (see clone_headroom_kb in cache-lib.sh), and a
# percentage floor only if one is configured. A percentage cannot express
# this: the failure it has to prevent is a clone running out of disk
# part-way through unsharing its mutable paths, and how much that needs
# is a property of the snapshot, not of the volume.
# 3. SELF-CLEAR — if pass 2 still isn't enough for the percentage floor,
# wipe this run's own target dir and pay a cold rebuild, reported to the
# job summary as well as the log, because a warning on a green run is
# what lets a silently 4x-slower job go unnoticed. It is not a way out of
# the derived requirement: a run that has an own target dir to wipe is a
# run whose seed reuses it and clones nothing, so that requirement is
# zero. Falling short of a NON-ZERO derived requirement fails the job
# here, naming the shortfall and what was kept instead of it — the seed
# would otherwise fail seconds later against a half-unshared staging
# tree, which is the failure this pass exists to pre-empt.
#
# Reactive-only, with no hard cap on cache size: a workspace's natural working
# set is what it is, and bounding the footprint preemptively means wiping
# useful content before it is actually causing host pressure. The bound is
# physics, not an arbitrary GB number.
#
# EVICTION ORDER, and why it is the reverse of the obvious one: within the
# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones.
# A snapshot is a hardlink clone of a live target dir and of every consumer
# cloned from it, so removing it frees almost no real bytes — its inodes stay
# alive through those other links — while costing every future PR its warm
# start. Evicting snapshots first would be nearly pure loss. Target
# directories are where a branch's own divergent artifacts actually live, so
# they are what freeing space means.
#
# Two exclusions every pass respects:
#
# PROTECTED — the publisher branches' target and snapshot directories, and
# this run's own target dir, are never candidates in any pass. Evicting a
# publisher's snapshot doesn't free real disk (every open PR's clone keeps
# the data alive) but does force every subsequent PR to start cold, which is
# the entire benefit this scheme exists to deliver.
#
# LOCKED — a directory carrying a .ci-lock-* marker younger than
# STALE_LOCK_SECONDS is held open by a running job, or named by a live
# .reading-<dir>-* marker (a job is hardlink-cloning it this instant), is
# skipped by every pass, however dead and however tight the disk. This is
# what makes eviction safe on a runner with more than one execution slot.
# An older marker is treated as abandoned and logged as such, so an
# actually-still-running job that somehow exceeds the threshold is visible
# in the log rather than silently losing its cache mid-build.
#
# That exclusion is decided TWICE per eviction — once as the cheap filter
# that keeps a held directory out of the pass at all, and once after the
# directory has been renamed aside, which is the decision the unlink
# actually rests on. See evict_dir.
#
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
# timeout. A directory name cannot be inverted back to a branch name (the
# sanitiser is lossy and the disambiguating suffix is a one-way hash), so this
# goes the other direction: it recomputes the expected directory names for
# every branch origin reports, using cache-lib.sh's OWN cache_key function —
# the same one the seed step used to create them. Reusing that function rather
# than reimplementing a lookalike is what makes the classification sound; any
# drift between two spellings would silently misclassify every directory.
#
# The merged half reads the TIP SHA out of that same `ls-remote` output and
# asks `git merge-base --is-ancestor` against each protected branch's tip,
# using the objects in this job's own checkout. Ancestry is only decidable
# where the objects are there to decide it, so the answer "I cannot tell"
# exists and is distinct from "not merged" everywhere it can arise:
#
# - a shallow checkout makes a MISSING object prove nothing, so the whole
# signal is withheld rather than read as "no branch is merged";
# - a protected tip that is not in the checkout is not used as an anchor;
# - a branch tip that is not in the checkout is kept, loudly.
#
# A squash or rebase merge leaves no ancestor relationship at all, so its
# branch reads as live here. That is a missed reclamation, not a wrong one,
# and the deleted-branch signal still covers it once the branch is removed.
#
# Only ever globs inside <cache-root>. Another project's volume is a different
# Docker named volume and is not mounted in this container at all, so "stays
# scoped to this repo's cache" holds structurally, not by convention.
set -euo pipefail
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
ROOT="${1:?usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> <min-free-percent> [own-key] [base-key] [fallback-dir]}"
OWN_DIR="${2:?}"
PROTECTED_REFS="${3:-}"
MIN_FREE_PCT="${4:-0}"
OWN_KEY="${5:-}"
BASE_KEY="${6:-}"
FALLBACK="${7:-}"
# Mirrored by daniel/gitdan's ci-cache-reclaim.sh, whose copy must be >= this
# one — raising this without raising theirs first lets that script treat a lock
# this side still honours as abandoned. Same direction and same reasoning as
# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh.
STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}"
# An aside directory is in flight for one rename plus one marker glob —
# milliseconds. Anything older belongs to a pass that died between the two, so
# an age is what separates "another pass is mid-eviction" from "a leftover",
# and it separates them without having to identify the pass that created it.
# The `$$` in an aside's name was that pass's PID inside its own job
# container, so testing it with `kill -0` from a different one is not
# unreliable, it is meaningless — and PIDs recycle besides. Three orders of
# magnitude of headroom over the operation it covers, and short enough that a
# genuine leftover is reclaimed by the next run rather than lingering while
# the volume is under pressure.
EVICTION_ASIDE_SETTLE_SECONDS="${EVICTION_ASIDE_SETTLE_SECONDS:-60}"
# What the seed step is about to do, resolved through the same function that
# step resolves it with (cache-lib.sh's seed_source_candidates). Empty when it
# will clone nothing at all: its own target dir already exists and it reuses
# it, or no source exists and it starts cold. Either way the derived
# requirement is zero, because nothing is about to be copied.
#
# gb() is for reporting only. Every comparison below is in KB, because
# `read_df` reports KB and rounding a threshold to a tenth of a GB either
# passes a run that cannot fit or evicts a cache the run did not need.
gb() { awk -v k="${1:-0}" 'BEGIN { printf "%.1f", k / 1048576 }'; }
SEED_SRC=""
CLONE_KB=0
if [ -n "$OWN_KEY" ]; then
SEED_SRC=$(seed_clone_source "$ROOT" "$OWN_KEY" "$BASE_KEY" "$FALLBACK")
if [ -n "$SEED_SRC" ]; then
# A full walk of the source, and the reason this pass moved ahead of the
# seed rather than staying where it was: the number is only useful before
# the clone it describes.
CLONE_KB=$(clone_headroom_kb "$SEED_SRC")
echo "clone requirement: seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB free (measured from its mutable set)"
else
echo "clone requirement: none — this run reuses its own cache or starts cold, so nothing will be copied"
fi
else
echo "clone requirement: not derivable — no cache key was passed to this pass; the ${MIN_FREE_PCT}% floor is the only gate"
fi
declare -A protected_ns=()
for ref in $PROTECTED_REFS; do
suffix=$(cache_key "$ref")
protected_ns["target-${suffix}"]=1
protected_ns["snapshot-${suffix}"]=1
done
# Prints why <dir> is off limits to every pass, or nothing when it is a
# candidate. The reason is not decoration: it is what the failure report at
# the bottom lists against each directory it kept while running out of space.
#
# SEED_SRC is the third exclusion and the one this script did not used to need.
# The prune ran after the seed, so the source had already been cloned and the
# reader marker over it was gone; running BEFORE the seed puts the directory
# this run is about to read squarely in the candidate set, and pass 1 would
# take it the moment its branch merged.
protected_reason() {
local dir="$1" name
name=$(basename "$dir")
[ "$dir" = "$OWN_DIR" ] && { printf 'this run own cache'; return 0; }
[ -n "$SEED_SRC" ] && [ "$dir" = "$SEED_SRC" ] && { printf 'the source this run is about to clone'; return 0; }
[ -n "${protected_ns[$name]:-}" ] && { printf 'a protected branch cache'; return 0; }
return 1
}
is_protected() { protected_reason "$1" >/dev/null; }
# is_locked <dir> [name]
#
# `name` is the directory's own name for reporting and for the reader-marker
# lookup, which matters when <dir> has been renamed aside for eviction: the
# markers a consumer publishes are keyed on the name it resolved, not on
# whatever the eviction pass has since called the directory.
is_locked() {
local dir="$1" name="${2:-$(basename "$1")}" now lock_file lock_age locked=1 readers
now=$(date +%s)
# A directory being hardlink-cloned right now carries no .ci-lock-* of its
# own — a snapshot has its locks stripped by construction — so the reader
# markers are the only signal that unlinking it would truncate somebody's
# in-flight clone. Today no reachable configuration prunes a snapshot (only
# protected refs publish them, and protected refs are excluded from every
# pass), which makes this guard redundant *by policy* — evict_dir's second
# look is what stops policy being the reason it is safe.
readers=$(live_reader_count "$ROOT" "$name")
if [ "$readers" -gt 0 ]; then
echo " ${name}: ${readers} job(s) currently cloning it — not a candidate"
locked=0
fi
for lock_file in "$dir"/.ci-lock-*; do
[ -e "$lock_file" ] || continue
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then
echo " ${name}: held open by $(basename "$lock_file") (${lock_age}s old)"
locked=0
else
echo " ${name}: ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned"
fi
done
return "$locked"
}
# evict_dir <dir>
#
# Unlinks <dir>, or declines to and says why. Returns 0 only if the directory
# is actually gone.
#
# The caller has already established that <dir> is a candidate, which is not
# the same as establishing that it is still one at the instant of the unlink:
# a consumer publishes its reader marker whenever it starts a clone, and the
# `du` that measures the directory between those two points runs for seconds
# on a multi-GB tree. Unlinking under a live clone truncates it silently —
# `cp -al` never reports a subtree that was removed before it read the
# parent's listing (see cache-lib.sh's reader-marker section).
#
# So the directory is renamed aside first and only then re-examined, which is
# what makes the second look conclusive rather than merely closer to the
# unlink. It is publish-snapshot.sh's rotation, and it rests on the same
# ordering proof: a consumer publishes its marker BEFORE it resolves the
# source path, so one that resolved this directory did so before the rename
# and therefore published its marker before the scan below, which happens
# strictly after that rename. A consumer arriving after the rename cannot
# resolve the path at all and falls through to its own cold-start path — the
# same safe degrade the publisher's swap window already produces.
#
# The rename disturbs nothing already in flight: it unlinks no entry and
# leaves the source inode unchanged, which is precisely why the publish side
# can rotate a snapshot out from under a live reader. A declined eviction
# therefore costs a deferred eviction and nothing else.
#
# `.evicting-<name>-<pid>` is a contract name shared with daniel/gitdan's
# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh).
# The sweep at the top of this script reclaims these on every pass, so the only
# one that reaches the arbiter belongs to a repo whose workflow has stopped
# running — which is exactly the case no in-workflow pass can reach.
evict_dir() {
local dir="$1" name aside
name=$(basename "$dir")
aside="${ROOT}/.evicting-${name}-$$"
mv -T "$dir" "$aside" 2>/dev/null || {
echo " ${name}: could not be set aside for eviction — skipped this pass"
return 1
}
if is_locked "$aside" "$name"; then
if mv -T "$aside" "$dir" 2>/dev/null; then
echo " ${name}: claimed by a job while its eviction was in flight — restored, not evicted"
elif [ -d "$aside" ]; then
echo "::warning::prune: ${name} was claimed mid-eviction and its own name is taken again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain"
else
echo "::warning::prune: ${name} was claimed mid-eviction and is already gone — another pass reclaimed it after its readers drained"
fi
return 1
fi
rm -rf "$aside"
return 0
}
# Directories oldest-first, target-* before snapshot-* (see the header).
# The sort key is a rank digit followed by a zero-padded mtime, so the two
# groups sort as blocks rather than interleaving by age. `.cache-last-used`
# is the marker every run touches; a run that hits the cache for every crate
# may write nothing at all inside the tree, which would make a
# just-used directory look stale without it. Fall back to the directory's
# own mtime when the marker is missing (a partially-written directory from an
# interrupted run is still a valid, if less precise, "last touched" signal).
list_by_lru() {
local rank d ts
for rank in 0:target 1:snapshot; do
for d in "$ROOT"/"${rank#*:}"-*; do
[ -d "$d" ] || continue
if [ -e "$d/.cache-last-used" ]; then ts=$(stat -c '%Y' "$d/.cache-last-used" 2>/dev/null || echo 0)
else ts=$(stat -c '%Y' "$d" 2>/dev/null || echo 0); fi
printf '%s%012d\t%s\n' "${rank%%:*}" "$ts" "$d"
done
done | sort | cut -f2-
}
# Deferred reclamations from an earlier pass: a directory renamed aside for
# eviction that could not be unlinked, because a job claimed it inside the
# window and its own name was taken again before it could be restored — or
# whose run was killed between the rename and the unlink. Nothing below would
# ever see one: every pass globs target-*/snapshot-*, which a dotted name does
# not match. Left unswept it is permanently unreclaimable disk on the one
# volume whose entire problem is disk.
#
# TWO DISTINCT PROPERTIES HOLD HERE, and neither implies the other.
#
# No clone can be truncated by the unlink below, by construction: the aside
# name only comes into existence after the evicting pass's rename, so a
# consumer that resolved the directory published its marker strictly before
# that rename and therefore before the count below — it cannot be missed. A
# consumer that arrives later cannot resolve the path at all. This holds under
# any interleaving and needs no settle window.
#
# No pass mid-eviction is mistaken for a leftover, by bound rather than by
# construction, and this is what the settle window is for. Without it a
# concurrent pass could unlink an aside its owner is still deciding about, and
# `rm -rf` traverses fd-relative: the owner's restore can then republish a
# half-emptied tree under a live cache name. What the window guarantees is that
# the two cannot be confused within it. What it does not guarantee is the
# pathological case beyond it — an evicting pass suspended past the window and
# then resumed finds its aside reclaimed and fails its restore, saying so.
now=$(date +%s)
for aside in "$ROOT"/.evicting-*; do
[ -d "$aside" ] || continue
# Change time, not modification time. `mv` leaves a directory's mtime alone
# (a cache last written days ago keeps a days-old mtime, which is what
# list_by_lru wants and exactly the wrong signal here) but rename(2) does
# update ctime, so %Z is when this directory was set aside and %Y is not.
aside_age=$(( now - $(stat -c '%Z' "$aside" 2>/dev/null || echo "$now") ))
if [ "$aside_age" -lt "$EVICTION_ASIDE_SETTLE_SECONDS" ]; then
echo "prune: $(basename "$aside") was set aside ${aside_age}s ago — another pass may still be evicting it, leaving it alone"
continue
fi
aside_name=$(basename "$aside"); aside_name="${aside_name#.evicting-}"; aside_name="${aside_name%-*}"
if [ "$(live_reader_count "$ROOT" "$aside_name")" -gt 0 ]; then
echo "prune: $(basename "$aside") is still being read — deferring its reclamation again"
continue
fi
echo "prune: reclaiming deferred eviction $(basename "$aside")"
rm -rf "$aside"
done
echo "=== pass 1: liveness (unconditional, not gated on free space) ==="
declare -A live_ns=()
# The tip SHA origin reports for the branch each directory name belongs to.
# Same output, same loop, one field over — reading it from a second `git` call
# would be reading a different instant.
declare -A live_tip=()
declare -A remote_tip_of=()
LIVENESS_AVAILABLE=0
LIVENESS_REASON=""
if [ "${CACHE_LIVENESS:-true}" = "0" ] || [ "${CACHE_LIVENESS:-true}" = "false" ]; then
LIVENESS_REASON="disabled via CACHE_LIVENESS=${CACHE_LIVENESS}"
elif remote_heads=$(timeout 20 git ls-remote --heads origin 2>&1); then
LIVENESS_AVAILABLE=1
branch_count=0
while IFS= read -r line; do
[ -n "$line" ] || continue
case "$line" in *refs/heads/*) ;; *) continue ;; esac
branch="${line#*refs/heads/}"
[ -n "$branch" ] || continue
sha="${line%%[[:space:]]*}"
suffix=$(cache_key "$branch")
live_ns["target-${suffix}"]=1
live_ns["snapshot-${suffix}"]=1
live_tip["target-${suffix}"]="$sha"
live_tip["snapshot-${suffix}"]="$sha"
remote_tip_of["$branch"]="$sha"
branch_count=$((branch_count + 1))
done <<< "$remote_heads"
echo "liveness: ${branch_count} live branches on origin"
else
LIVENESS_REASON="git ls-remote --heads origin failed or timed out"
fi
# The merged half of pass 1, and whether this checkout can answer it at all.
# Every branch of this decision that ends in "no" ends in the signal being
# WITHHELD, never in a directory being classified dead by default.
MERGED_AVAILABLE=0
MERGED_REASON=""
PROTECTED_TIPS=()
declare -A merged_verdict=()
declare -A merged_into=()
MERGED_INTO=""
if [ "$LIVENESS_AVAILABLE" = "1" ]; then
if ! git rev-parse --git-dir >/dev/null 2>&1; then
MERGED_REASON="not inside a git checkout"
elif [ "$(git rev-parse --is-shallow-repository 2>/dev/null || echo unknown)" != "false" ]; then
# In a shallow clone an absent commit is the normal case, so `--is-ancestor`
# answers about the graph that was fetched rather than the one that exists.
MERGED_REASON="the checkout is shallow, so a commit missing from it says nothing about ancestry"
else
for ref in $PROTECTED_REFS; do
tip="${remote_tip_of[$ref]:-}"
[ -n "$tip" ] || continue
git cat-file -e "${tip}^{commit}" 2>/dev/null || continue
PROTECTED_TIPS+=("${ref}:${tip}")
done
if [ "${#PROTECTED_TIPS[@]}" -gt 0 ]; then
MERGED_AVAILABLE=1
echo "liveness: merged-branch detection anchored on ${PROTECTED_TIPS[*]%%:*}"
else
MERGED_REASON="none of the protected branch tips (${PROTECTED_REFS:-none configured}) is present in this checkout"
fi
fi
[ "$MERGED_AVAILABLE" = "1" ] || \
echo "::warning::liveness: ${MERGED_REASON} — merged-but-undeleted branches keep their caches this run"
fi
# is_merged_dead <dir-name>
#
# True when the branch this directory belongs to is still on origin but every
# commit it holds is already on a protected branch — a merged PR whose branch
# the forge did not delete, which is the case zemyna hits on every merge and
# which the deleted-branch signal above can never see.
#
# Memoised per tip because target-<key> and snapshot-<key> share one branch,
# and because the "cannot tell" warning belongs to the branch rather than to
# each of its directories.
is_merged_dead() {
local name="$1" tip="${live_tip[$1]:-}" entry ref psha
MERGED_INTO=""
[ "$MERGED_AVAILABLE" = "1" ] || return 1
[ -n "$tip" ] || return 1
case "${merged_verdict[$tip]:-}" in
dead) MERGED_INTO="${merged_into[$tip]}"; return 0 ;;
live|unknown) return 1 ;;
esac
if ! git cat-file -e "${tip}^{commit}" 2>/dev/null; then
merged_verdict["$tip"]=unknown
echo "::warning::prune: ${name}: its branch tip ${tip} is not in this checkout — cannot tell merged from live, keeping it"
return 1
fi
for entry in "${PROTECTED_TIPS[@]}"; do
ref="${entry%%:*}"; psha="${entry#*:}"
if git merge-base --is-ancestor "$tip" "$psha" 2>/dev/null; then
merged_verdict["$tip"]=dead
merged_into["$tip"]="$ref"
MERGED_INTO="$ref"
return 0
fi
done
merged_verdict["$tip"]=live
return 1
}
if [ "$LIVENESS_AVAILABLE" = "1" ]; then
pruned_any=0
# Tracked separately so the line below cannot contradict the decline lines
# above it: "none pruned" and "none found" are different outcomes, and a
# pass that declined every dead cache it found has not found none.
spared_any=0
for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do
[ -d "$dir" ] || continue
name=$(basename "$dir")
is_protected "$dir" && continue
if [ -z "${live_ns[$name]:-}" ]; then
why="no matching branch on origin"
why_summary="branch no longer exists on origin"
elif is_merged_dead "$name"; then
why="merged into ${MERGED_INTO}, whose tip already contains its every commit"
why_summary="merged into \`${MERGED_INTO}\`"
else
continue
fi
is_locked "$dir" && { spared_any=1; continue; }
dir_gb=$(usage_gb "$dir")
evict_dir "$dir" || { spared_any=1; continue; }
echo "::warning::pruned dead-branch cache ${name} (${dir_gb} GB) — ${why}"
summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — ${why_summary}"
pruned_any=1
done
if [ "$pruned_any" = "0" ]; then
if [ "$spared_any" = "1" ]; then
echo "every dead-branch cache found is still in use — none pruned this pass"
else
echo "no dead-branch caches found"
fi
fi
else
echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped"
fi
echo
echo "=== pass 2/3: disk pressure ==="
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
PCT_KB=$(( TOTAL_KB * MIN_FREE_PCT / 100 ))
# The two floors, and which of them governs. They are kept apart all the way
# down rather than collapsed here, because falling short of them means
# different things: the derived one predicts that the very next step cannot
# finish, and the percentage one is a hygiene target for the volume.
REQUIRED_KB="$CLONE_KB"
GOVERNS="the clone this run is about to make"
if [ "$PCT_KB" -gt "$REQUIRED_KB" ]; then
REQUIRED_KB="$PCT_KB"
GOVERNS="the ${MIN_FREE_PCT}% floor"
fi
echo "required: $(gb "$REQUIRED_KB") GB free — ${GOVERNS} (clone $(gb "$CLONE_KB") GB, floor $(gb "$PCT_KB") GB)"
own_report() {
if [ -d "$OWN_DIR" ]; then
echo "cache: $(basename "$OWN_DIR") $(usage_gb "$OWN_DIR") GB | $(report_df host "$1" "$2")"
else
# Ordinary now that this pass runs ahead of the seed: on a branch's first
# run of the day the directory does not exist yet, and reporting 0.0 GB
# for it would read as an emptied cache.
echo "cache: $(basename "$OWN_DIR") not seeded yet | $(report_df host "$1" "$2")"
fi
}
if [ "$FREE_KB" -ge "$REQUIRED_KB" ]; then
own_report "$FREE_KB" "$TOTAL_KB"
exit 0
fi
echo "::warning::$(report_df disk "$FREE_KB" "$TOTAL_KB") < the $(gb "$REQUIRED_KB") GB this run requires"
# What survived the pass, and why, in the order the pass considered them. Read
# only by the failure report at the bottom: a run that cannot fit its clone is
# a run whose log has to answer "then what is all that space?" without anyone
# having to reconstruct the pass by hand.
KEPT=()
# A plain loop over a pre-materialised, pre-sorted list rather than a live
# `find | while` pipeline, so `rm -rf` inside the loop cannot make a running
# directory walk re-observe its own deletions.
mapfile -t LRU < <(list_by_lru)
for dir in "${LRU[@]}"; do
[ -d "$dir" ] || continue
if reason=$(protected_reason "$dir"); then
KEPT+=("$(basename "$dir")${reason}")
continue
fi
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
[ "$FREE_KB" -ge "$REQUIRED_KB" ] && break
if is_locked "$dir"; then
KEPT+=("$(basename "$dir") — held open by a running job")
continue
fi
dir_gb=$(usage_gb "$dir")
if ! evict_dir "$dir"; then
KEPT+=("$(basename "$dir") — claimed by a job while its eviction was in flight")
continue
fi
echo "::warning::evicted $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)"
summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)"
done
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
# Falling short of the DERIVED requirement is a failure, not a warning. The
# seed step is next, it will clone that source, and it will run out of disk
# part-way through unsharing the clone's mutable paths — reported against a
# staging path, with nothing in the message about which cache was holding the
# space. Failing here says that instead.
#
# There is nothing to self-clear on this path and it is not skipped in error:
# a non-zero requirement means the seed is about to CLONE, which means this
# run has no own target dir to wipe (seed_clone_source returns nothing when it
# does), so pass 3 has no candidate. See the header.
if [ "$FREE_KB" -lt "$CLONE_KB" ]; then
echo "::error::prune: $(gb "$FREE_KB") GB free after evicting every eligible cache, but seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB — short by $(gb "$(( CLONE_KB - FREE_KB ))") GB"
summary_line "- **out of disk**: seeding from \`$(basename "$SEED_SRC")\` needs $(gb "$CLONE_KB") GB, $(gb "$FREE_KB") GB free"
echo "prune: kept, and why:"
for entry in ${KEPT[@]+"${KEPT[@]}"}; do echo " ${entry}"; done
[ "${#KEPT[@]}" -gt 0 ] || echo " (nothing — the volume holds no cache directories at all)"
exit 1
fi
if [ "$FREE_KB" -lt "$PCT_KB" ]; then
OWN_GB=$(usage_gb "$OWN_DIR")
echo "::warning::still under threshold after evicting every eligible sibling; clearing own $(basename "$OWN_DIR") (was ${OWN_GB} GB) — this run pays a cold rebuild"
summary_line "- **self-clear**: \`$(basename "$OWN_DIR")\` (was ${OWN_GB} GB) wiped — this run pays a cold rebuild"
# Recreated empty rather than left absent, and that is what keeps this path
# out of the requirement above: the seed reuses an own target dir that
# exists, whatever is in it, so a self-cleared run clones nothing and needs
# no headroom. Leaving it absent would send that run to the base snapshot
# instead, needing a clone this pass has just spent its last eligible bytes
# not sizing for.
rm -rf "$OWN_DIR"
mkdir -p "$OWN_DIR"
else
echo "$(report_df post-eviction "$FREE_KB" "$TOTAL_KB") — sibling eviction recovered enough space"
fi