CI / shellcheck + selftests (pull_request) Successful in 1m27s
`daniel/zemyna` enabled `default_delete_branch_after_merge` after this branch was written, so the deleted-branch signal will fire there on future merges. That makes the setting worth recommending — it is the cheapest case for this scheme, decidable from `ls-remote` with no checkout, no objects and no walk — and it does not make the ancestry signal redundant. Three things it leaves behind, now enumerated in the README's eviction section rather than implied: every branch merged before the setting was turned on, of which zemyna carried 48 and which nothing retroactively deletes; every merge whose deletion the forge declines or is never asked to make, since it is best-effort and silent and an API merge without the flag never asks; and every repo that has not enabled it, which is the default. Two present-tense claims about one repo's configuration are reworded into the conditions they were standing in for, in prune-cache.sh's header and beside is_merged_dead, plus the two in the selftest that asserted the forge keeps branches rather than describing the fixture. No behaviour change; the suite is green and unchanged at 61 assertions. Refs #20. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JXMQCJ5Eg5f9G9cfYzyh4Z
613 lines
30 KiB
Bash
Executable File
613 lines
30 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Eviction for the per-ref cache directories on the persistent volume.
|
|
#
|
|
# Usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> \
|
|
# <min-free-percent> [own-key] [base-key] [fallback-dir]
|
|
# protected-branches space-separated raw refs (e.g. "dev main")
|
|
# min-free-percent a FLOOR, not the gate — 0 to rely on the derived
|
|
# requirement alone (the default)
|
|
# own-key, base-key, fallback-dir
|
|
# the same three the seed step resolves its source from.
|
|
# Given them, this pass sizes the volume for the clone
|
|
# that step is about to make; without them it has only
|
|
# the percentage floor, and says so.
|
|
#
|
|
# Optional environment:
|
|
# STALE_LOCK_SECONDS age past which a .ci-lock-* marker is treated as
|
|
# abandoned (default 7200)
|
|
# CACHE_LIVENESS "false"/"0" to skip the liveness pass entirely
|
|
# CACHE_DF_OVERRIDE "<total_kb> <free_kb>", for the selftest
|
|
# EVICTION_ASIDE_SETTLE_SECONDS
|
|
# how long a directory renamed aside for eviction is
|
|
# left alone before another pass may reclaim it
|
|
# (default 60)
|
|
#
|
|
# Three passes, in order:
|
|
#
|
|
# 1. LIVENESS — every target-*/snapshot-* directory whose branch is DEAD is
|
|
# removed UNCONDITIONALLY, not gated on free space. A directory for a
|
|
# branch nothing will build again is pure loss; waiting for disk pressure
|
|
# to notice means paying for it until then. Two signals make a branch
|
|
# dead, and the second exists because the first alone is inert wherever
|
|
# a merged branch stays on origin — the default, and still the outcome
|
|
# whenever delete-on-merge declines or is not asked (gitdan-actions#20):
|
|
#
|
|
# DELETED — the branch is no longer on origin at all.
|
|
# MERGED — the branch is still on origin, but its tip is an ancestor
|
|
# of a protected branch's tip, so every commit it holds is
|
|
# already on the branch its cache would be re-cloned from.
|
|
#
|
|
# Skipped entirely, loudly, if the signal itself is unavailable —
|
|
# "couldn't determine" is never folded into "dead", for either signal
|
|
# and at either granularity: a checkout that cannot answer ancestry at
|
|
# all skips the merged half, and a single branch whose tip is not in the
|
|
# checkout is kept with a warning naming it.
|
|
# 2. PRESSURE — if free space is under the requirement, evict remaining
|
|
# (now necessarily live) directories oldest-first until it clears. The
|
|
# requirement is what the seed step is about to need to clone its source,
|
|
# MEASURED off that source (see clone_headroom_kb in cache-lib.sh), and a
|
|
# percentage floor only if one is configured. A percentage cannot express
|
|
# this: the failure it has to prevent is a clone running out of disk
|
|
# part-way through unsharing its mutable paths, and how much that needs
|
|
# is a property of the snapshot, not of the volume.
|
|
# 3. SELF-CLEAR — if pass 2 still isn't enough for the percentage floor,
|
|
# wipe this run's own target dir and pay a cold rebuild, reported to the
|
|
# job summary as well as the log, because a warning on a green run is
|
|
# what lets a silently 4x-slower job go unnoticed. It is not a way out of
|
|
# the derived requirement: a run that has an own target dir to wipe is a
|
|
# run whose seed reuses it and clones nothing, so that requirement is
|
|
# zero. Falling short of a NON-ZERO derived requirement fails the job
|
|
# here, naming the shortfall and what was kept instead of it — the seed
|
|
# would otherwise fail seconds later against a half-unshared staging
|
|
# tree, which is the failure this pass exists to pre-empt.
|
|
#
|
|
# Reactive-only, with no hard cap on cache size: a workspace's natural working
|
|
# set is what it is, and bounding the footprint preemptively means wiping
|
|
# useful content before it is actually causing host pressure. The bound is
|
|
# physics, not an arbitrary GB number.
|
|
#
|
|
# EVICTION ORDER, and why it is the reverse of the obvious one: within the
|
|
# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones.
|
|
# A snapshot is a hardlink clone of a live target dir and of every consumer
|
|
# cloned from it, so removing it frees almost no real bytes — its inodes stay
|
|
# alive through those other links — while costing every future PR its warm
|
|
# start. Evicting snapshots first would be nearly pure loss. Target
|
|
# directories are where a branch's own divergent artifacts actually live, so
|
|
# they are what freeing space means.
|
|
#
|
|
# Two exclusions every pass respects:
|
|
#
|
|
# PROTECTED — the publisher branches' target and snapshot directories, and
|
|
# this run's own target dir, are never candidates in any pass. Evicting a
|
|
# publisher's snapshot doesn't free real disk (every open PR's clone keeps
|
|
# the data alive) but does force every subsequent PR to start cold, which is
|
|
# the entire benefit this scheme exists to deliver.
|
|
#
|
|
# LOCKED — a directory carrying a .ci-lock-* marker younger than
|
|
# STALE_LOCK_SECONDS is held open by a running job, or named by a live
|
|
# .reading-<dir>-* marker (a job is hardlink-cloning it this instant), is
|
|
# skipped by every pass, however dead and however tight the disk. This is
|
|
# what makes eviction safe on a runner with more than one execution slot.
|
|
# An older marker is treated as abandoned and logged as such, so an
|
|
# actually-still-running job that somehow exceeds the threshold is visible
|
|
# in the log rather than silently losing its cache mid-build.
|
|
#
|
|
# That exclusion is decided TWICE per eviction — once as the cheap filter
|
|
# that keeps a held directory out of the pass at all, and once after the
|
|
# directory has been renamed aside, which is the decision the unlink
|
|
# actually rests on. See evict_dir.
|
|
#
|
|
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
|
|
# timeout. A directory name cannot be inverted back to a branch name (the
|
|
# sanitiser is lossy and the disambiguating suffix is a one-way hash), so this
|
|
# goes the other direction: it recomputes the expected directory names for
|
|
# every branch origin reports, using cache-lib.sh's OWN cache_key function —
|
|
# the same one the seed step used to create them. Reusing that function rather
|
|
# than reimplementing a lookalike is what makes the classification sound; any
|
|
# drift between two spellings would silently misclassify every directory.
|
|
#
|
|
# The merged half reads the TIP SHA out of that same `ls-remote` output and
|
|
# asks `git merge-base --is-ancestor` against each protected branch's tip,
|
|
# using the objects in this job's own checkout. Ancestry is only decidable
|
|
# where the objects are there to decide it, so the answer "I cannot tell"
|
|
# exists and is distinct from "not merged" everywhere it can arise:
|
|
#
|
|
# - a shallow checkout makes a MISSING object prove nothing, so the whole
|
|
# signal is withheld rather than read as "no branch is merged";
|
|
# - a protected tip that is not in the checkout is not used as an anchor;
|
|
# - a branch tip that is not in the checkout is kept, loudly.
|
|
#
|
|
# A squash or rebase merge leaves no ancestor relationship at all, so its
|
|
# branch reads as live here. That is a missed reclamation, not a wrong one,
|
|
# and the deleted-branch signal still covers it once the branch is removed.
|
|
#
|
|
# Only ever globs inside <cache-root>. Another project's volume is a different
|
|
# Docker named volume and is not mounted in this container at all, so "stays
|
|
# scoped to this repo's cache" holds structurally, not by convention.
|
|
set -euo pipefail
|
|
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
|
|
|
ROOT="${1:?usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> <min-free-percent> [own-key] [base-key] [fallback-dir]}"
|
|
OWN_DIR="${2:?}"
|
|
PROTECTED_REFS="${3:-}"
|
|
MIN_FREE_PCT="${4:-0}"
|
|
OWN_KEY="${5:-}"
|
|
BASE_KEY="${6:-}"
|
|
FALLBACK="${7:-}"
|
|
# Mirrored by daniel/gitdan's ci-cache-reclaim.sh, whose copy must be >= this
|
|
# one — raising this without raising theirs first lets that script treat a lock
|
|
# this side still honours as abandoned. Same direction and same reasoning as
|
|
# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh.
|
|
STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}"
|
|
# An aside directory is in flight for one rename plus one marker glob —
|
|
# milliseconds. Anything older belongs to a pass that died between the two, so
|
|
# an age is what separates "another pass is mid-eviction" from "a leftover",
|
|
# and it separates them without having to identify the pass that created it.
|
|
# The `$$` in an aside's name was that pass's PID inside its own job
|
|
# container, so testing it with `kill -0` from a different one is not
|
|
# unreliable, it is meaningless — and PIDs recycle besides. Three orders of
|
|
# magnitude of headroom over the operation it covers, and short enough that a
|
|
# genuine leftover is reclaimed by the next run rather than lingering while
|
|
# the volume is under pressure.
|
|
EVICTION_ASIDE_SETTLE_SECONDS="${EVICTION_ASIDE_SETTLE_SECONDS:-60}"
|
|
|
|
# What the seed step is about to do, resolved through the same function that
|
|
# step resolves it with (cache-lib.sh's seed_source_candidates). Empty when it
|
|
# will clone nothing at all: its own target dir already exists and it reuses
|
|
# it, or no source exists and it starts cold. Either way the derived
|
|
# requirement is zero, because nothing is about to be copied.
|
|
#
|
|
# gb() is for reporting only. Every comparison below is in KB, because
|
|
# `read_df` reports KB and rounding a threshold to a tenth of a GB either
|
|
# passes a run that cannot fit or evicts a cache the run did not need.
|
|
gb() { awk -v k="${1:-0}" 'BEGIN { printf "%.1f", k / 1048576 }'; }
|
|
|
|
SEED_SRC=""
|
|
CLONE_KB=0
|
|
if [ -n "$OWN_KEY" ]; then
|
|
SEED_SRC=$(seed_clone_source "$ROOT" "$OWN_KEY" "$BASE_KEY" "$FALLBACK")
|
|
if [ -n "$SEED_SRC" ]; then
|
|
# A full walk of the source, and the reason this pass moved ahead of the
|
|
# seed rather than staying where it was: the number is only useful before
|
|
# the clone it describes.
|
|
CLONE_KB=$(clone_headroom_kb "$SEED_SRC")
|
|
echo "clone requirement: seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB free (measured from its mutable set)"
|
|
else
|
|
echo "clone requirement: none — this run reuses its own cache or starts cold, so nothing will be copied"
|
|
fi
|
|
else
|
|
echo "clone requirement: not derivable — no cache key was passed to this pass; the ${MIN_FREE_PCT}% floor is the only gate"
|
|
fi
|
|
|
|
declare -A protected_ns=()
|
|
for ref in $PROTECTED_REFS; do
|
|
suffix=$(cache_key "$ref")
|
|
protected_ns["target-${suffix}"]=1
|
|
protected_ns["snapshot-${suffix}"]=1
|
|
done
|
|
|
|
# Prints why <dir> is off limits to every pass, or nothing when it is a
|
|
# candidate. The reason is not decoration: it is what the failure report at
|
|
# the bottom lists against each directory it kept while running out of space.
|
|
#
|
|
# SEED_SRC is the third exclusion and the one this script did not used to need.
|
|
# The prune ran after the seed, so the source had already been cloned and the
|
|
# reader marker over it was gone; running BEFORE the seed puts the directory
|
|
# this run is about to read squarely in the candidate set, and pass 1 would
|
|
# take it the moment its branch merged.
|
|
protected_reason() {
|
|
local dir="$1" name
|
|
name=$(basename "$dir")
|
|
[ "$dir" = "$OWN_DIR" ] && { printf 'this run own cache'; return 0; }
|
|
[ -n "$SEED_SRC" ] && [ "$dir" = "$SEED_SRC" ] && { printf 'the source this run is about to clone'; return 0; }
|
|
[ -n "${protected_ns[$name]:-}" ] && { printf 'a protected branch cache'; return 0; }
|
|
return 1
|
|
}
|
|
|
|
is_protected() { protected_reason "$1" >/dev/null; }
|
|
|
|
# is_locked <dir> [name]
|
|
#
|
|
# `name` is the directory's own name for reporting and for the reader-marker
|
|
# lookup, which matters when <dir> has been renamed aside for eviction: the
|
|
# markers a consumer publishes are keyed on the name it resolved, not on
|
|
# whatever the eviction pass has since called the directory.
|
|
is_locked() {
|
|
local dir="$1" name="${2:-$(basename "$1")}" now lock_file lock_age locked=1 readers
|
|
now=$(date +%s)
|
|
# A directory being hardlink-cloned right now carries no .ci-lock-* of its
|
|
# own — a snapshot has its locks stripped by construction — so the reader
|
|
# markers are the only signal that unlinking it would truncate somebody's
|
|
# in-flight clone. Today no reachable configuration prunes a snapshot (only
|
|
# protected refs publish them, and protected refs are excluded from every
|
|
# pass), which makes this guard redundant *by policy* — evict_dir's second
|
|
# look is what stops policy being the reason it is safe.
|
|
readers=$(live_reader_count "$ROOT" "$name")
|
|
if [ "$readers" -gt 0 ]; then
|
|
echo " ${name}: ${readers} job(s) currently cloning it — not a candidate"
|
|
locked=0
|
|
fi
|
|
for lock_file in "$dir"/.ci-lock-*; do
|
|
[ -e "$lock_file" ] || continue
|
|
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
|
|
if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then
|
|
echo " ${name}: held open by $(basename "$lock_file") (${lock_age}s old)"
|
|
locked=0
|
|
else
|
|
echo " ${name}: ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned"
|
|
fi
|
|
done
|
|
return "$locked"
|
|
}
|
|
|
|
# evict_dir <dir>
|
|
#
|
|
# Unlinks <dir>, or declines to and says why. Returns 0 only if the directory
|
|
# is actually gone.
|
|
#
|
|
# The caller has already established that <dir> is a candidate, which is not
|
|
# the same as establishing that it is still one at the instant of the unlink:
|
|
# a consumer publishes its reader marker whenever it starts a clone, and the
|
|
# `du` that measures the directory between those two points runs for seconds
|
|
# on a multi-GB tree. Unlinking under a live clone truncates it silently —
|
|
# `cp -al` never reports a subtree that was removed before it read the
|
|
# parent's listing (see cache-lib.sh's reader-marker section).
|
|
#
|
|
# So the directory is renamed aside first and only then re-examined, which is
|
|
# what makes the second look conclusive rather than merely closer to the
|
|
# unlink. It is publish-snapshot.sh's rotation, and it rests on the same
|
|
# ordering proof: a consumer publishes its marker BEFORE it resolves the
|
|
# source path, so one that resolved this directory did so before the rename
|
|
# and therefore published its marker before the scan below, which happens
|
|
# strictly after that rename. A consumer arriving after the rename cannot
|
|
# resolve the path at all and falls through to its own cold-start path — the
|
|
# same safe degrade the publisher's swap window already produces.
|
|
#
|
|
# The rename disturbs nothing already in flight: it unlinks no entry and
|
|
# leaves the source inode unchanged, which is precisely why the publish side
|
|
# can rotate a snapshot out from under a live reader. A declined eviction
|
|
# therefore costs a deferred eviction and nothing else.
|
|
#
|
|
# `.evicting-<name>-<pid>` is a contract name shared with daniel/gitdan's
|
|
# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh).
|
|
# The sweep at the top of this script reclaims these on every pass, so the only
|
|
# one that reaches the arbiter belongs to a repo whose workflow has stopped
|
|
# running — which is exactly the case no in-workflow pass can reach.
|
|
evict_dir() {
|
|
local dir="$1" name aside
|
|
name=$(basename "$dir")
|
|
aside="${ROOT}/.evicting-${name}-$$"
|
|
|
|
mv -T "$dir" "$aside" 2>/dev/null || {
|
|
echo " ${name}: could not be set aside for eviction — skipped this pass"
|
|
return 1
|
|
}
|
|
|
|
if is_locked "$aside" "$name"; then
|
|
if mv -T "$aside" "$dir" 2>/dev/null; then
|
|
echo " ${name}: claimed by a job while its eviction was in flight — restored, not evicted"
|
|
elif [ -d "$aside" ]; then
|
|
echo "::warning::prune: ${name} was claimed mid-eviction and its own name is taken again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain"
|
|
else
|
|
echo "::warning::prune: ${name} was claimed mid-eviction and is already gone — another pass reclaimed it after its readers drained"
|
|
fi
|
|
return 1
|
|
fi
|
|
|
|
rm -rf "$aside"
|
|
return 0
|
|
}
|
|
|
|
# Directories oldest-first, target-* before snapshot-* (see the header).
|
|
# The sort key is a rank digit followed by a zero-padded mtime, so the two
|
|
# groups sort as blocks rather than interleaving by age. `.cache-last-used`
|
|
# is the marker every run touches; a run that hits the cache for every crate
|
|
# may write nothing at all inside the tree, which would make a
|
|
# just-used directory look stale without it. Fall back to the directory's
|
|
# own mtime when the marker is missing (a partially-written directory from an
|
|
# interrupted run is still a valid, if less precise, "last touched" signal).
|
|
list_by_lru() {
|
|
local rank d ts
|
|
for rank in 0:target 1:snapshot; do
|
|
for d in "$ROOT"/"${rank#*:}"-*; do
|
|
[ -d "$d" ] || continue
|
|
if [ -e "$d/.cache-last-used" ]; then ts=$(stat -c '%Y' "$d/.cache-last-used" 2>/dev/null || echo 0)
|
|
else ts=$(stat -c '%Y' "$d" 2>/dev/null || echo 0); fi
|
|
printf '%s%012d\t%s\n' "${rank%%:*}" "$ts" "$d"
|
|
done
|
|
done | sort | cut -f2-
|
|
}
|
|
|
|
# Deferred reclamations from an earlier pass: a directory renamed aside for
|
|
# eviction that could not be unlinked, because a job claimed it inside the
|
|
# window and its own name was taken again before it could be restored — or
|
|
# whose run was killed between the rename and the unlink. Nothing below would
|
|
# ever see one: every pass globs target-*/snapshot-*, which a dotted name does
|
|
# not match. Left unswept it is permanently unreclaimable disk on the one
|
|
# volume whose entire problem is disk.
|
|
#
|
|
# TWO DISTINCT PROPERTIES HOLD HERE, and neither implies the other.
|
|
#
|
|
# No clone can be truncated by the unlink below, by construction: the aside
|
|
# name only comes into existence after the evicting pass's rename, so a
|
|
# consumer that resolved the directory published its marker strictly before
|
|
# that rename and therefore before the count below — it cannot be missed. A
|
|
# consumer that arrives later cannot resolve the path at all. This holds under
|
|
# any interleaving and needs no settle window.
|
|
#
|
|
# No pass mid-eviction is mistaken for a leftover, by bound rather than by
|
|
# construction, and this is what the settle window is for. Without it a
|
|
# concurrent pass could unlink an aside its owner is still deciding about, and
|
|
# `rm -rf` traverses fd-relative: the owner's restore can then republish a
|
|
# half-emptied tree under a live cache name. What the window guarantees is that
|
|
# the two cannot be confused within it. What it does not guarantee is the
|
|
# pathological case beyond it — an evicting pass suspended past the window and
|
|
# then resumed finds its aside reclaimed and fails its restore, saying so.
|
|
now=$(date +%s)
|
|
for aside in "$ROOT"/.evicting-*; do
|
|
[ -d "$aside" ] || continue
|
|
# Change time, not modification time. `mv` leaves a directory's mtime alone
|
|
# (a cache last written days ago keeps a days-old mtime, which is what
|
|
# list_by_lru wants and exactly the wrong signal here) but rename(2) does
|
|
# update ctime, so %Z is when this directory was set aside and %Y is not.
|
|
aside_age=$(( now - $(stat -c '%Z' "$aside" 2>/dev/null || echo "$now") ))
|
|
if [ "$aside_age" -lt "$EVICTION_ASIDE_SETTLE_SECONDS" ]; then
|
|
echo "prune: $(basename "$aside") was set aside ${aside_age}s ago — another pass may still be evicting it, leaving it alone"
|
|
continue
|
|
fi
|
|
aside_name=$(basename "$aside"); aside_name="${aside_name#.evicting-}"; aside_name="${aside_name%-*}"
|
|
if [ "$(live_reader_count "$ROOT" "$aside_name")" -gt 0 ]; then
|
|
echo "prune: $(basename "$aside") is still being read — deferring its reclamation again"
|
|
continue
|
|
fi
|
|
echo "prune: reclaiming deferred eviction $(basename "$aside")"
|
|
rm -rf "$aside"
|
|
done
|
|
|
|
echo "=== pass 1: liveness (unconditional, not gated on free space) ==="
|
|
declare -A live_ns=()
|
|
# The tip SHA origin reports for the branch each directory name belongs to.
|
|
# Same output, same loop, one field over — reading it from a second `git` call
|
|
# would be reading a different instant.
|
|
declare -A live_tip=()
|
|
declare -A remote_tip_of=()
|
|
LIVENESS_AVAILABLE=0
|
|
LIVENESS_REASON=""
|
|
if [ "${CACHE_LIVENESS:-true}" = "0" ] || [ "${CACHE_LIVENESS:-true}" = "false" ]; then
|
|
LIVENESS_REASON="disabled via CACHE_LIVENESS=${CACHE_LIVENESS}"
|
|
elif remote_heads=$(timeout 20 git ls-remote --heads origin 2>&1); then
|
|
LIVENESS_AVAILABLE=1
|
|
branch_count=0
|
|
while IFS= read -r line; do
|
|
[ -n "$line" ] || continue
|
|
case "$line" in *refs/heads/*) ;; *) continue ;; esac
|
|
branch="${line#*refs/heads/}"
|
|
[ -n "$branch" ] || continue
|
|
sha="${line%%[[:space:]]*}"
|
|
suffix=$(cache_key "$branch")
|
|
live_ns["target-${suffix}"]=1
|
|
live_ns["snapshot-${suffix}"]=1
|
|
live_tip["target-${suffix}"]="$sha"
|
|
live_tip["snapshot-${suffix}"]="$sha"
|
|
remote_tip_of["$branch"]="$sha"
|
|
branch_count=$((branch_count + 1))
|
|
done <<< "$remote_heads"
|
|
echo "liveness: ${branch_count} live branches on origin"
|
|
else
|
|
LIVENESS_REASON="git ls-remote --heads origin failed or timed out"
|
|
fi
|
|
|
|
# The merged half of pass 1, and whether this checkout can answer it at all.
|
|
# Every branch of this decision that ends in "no" ends in the signal being
|
|
# WITHHELD, never in a directory being classified dead by default.
|
|
MERGED_AVAILABLE=0
|
|
MERGED_REASON=""
|
|
PROTECTED_TIPS=()
|
|
declare -A merged_verdict=()
|
|
declare -A merged_into=()
|
|
MERGED_INTO=""
|
|
if [ "$LIVENESS_AVAILABLE" = "1" ]; then
|
|
if ! git rev-parse --git-dir >/dev/null 2>&1; then
|
|
MERGED_REASON="not inside a git checkout"
|
|
elif [ "$(git rev-parse --is-shallow-repository 2>/dev/null || echo unknown)" != "false" ]; then
|
|
# In a shallow clone an absent commit is the normal case, so `--is-ancestor`
|
|
# answers about the graph that was fetched rather than the one that exists.
|
|
MERGED_REASON="the checkout is shallow, so a commit missing from it says nothing about ancestry"
|
|
else
|
|
for ref in $PROTECTED_REFS; do
|
|
tip="${remote_tip_of[$ref]:-}"
|
|
[ -n "$tip" ] || continue
|
|
git cat-file -e "${tip}^{commit}" 2>/dev/null || continue
|
|
PROTECTED_TIPS+=("${ref}:${tip}")
|
|
done
|
|
if [ "${#PROTECTED_TIPS[@]}" -gt 0 ]; then
|
|
MERGED_AVAILABLE=1
|
|
echo "liveness: merged-branch detection anchored on ${PROTECTED_TIPS[*]%%:*}"
|
|
else
|
|
MERGED_REASON="none of the protected branch tips (${PROTECTED_REFS:-none configured}) is present in this checkout"
|
|
fi
|
|
fi
|
|
[ "$MERGED_AVAILABLE" = "1" ] || \
|
|
echo "::warning::liveness: ${MERGED_REASON} — merged-but-undeleted branches keep their caches this run"
|
|
fi
|
|
|
|
# is_merged_dead <dir-name>
|
|
#
|
|
# True when the branch this directory belongs to is still on origin but every
|
|
# commit it holds is already on a protected branch — a merged PR whose branch
|
|
# the forge did not delete, which the deleted-branch signal above can never
|
|
# see. Enabling delete-on-merge narrows this to the branches merged before it
|
|
# was enabled, the ones its deletion declines (protected, or used by another
|
|
# open PR), and the merges that never ask (an API merge without the flag);
|
|
# see README's eviction section.
|
|
#
|
|
# Memoised per tip because target-<key> and snapshot-<key> share one branch,
|
|
# and because the "cannot tell" warning belongs to the branch rather than to
|
|
# each of its directories.
|
|
is_merged_dead() {
|
|
local name="$1" tip="${live_tip[$1]:-}" entry ref psha
|
|
MERGED_INTO=""
|
|
[ "$MERGED_AVAILABLE" = "1" ] || return 1
|
|
[ -n "$tip" ] || return 1
|
|
case "${merged_verdict[$tip]:-}" in
|
|
dead) MERGED_INTO="${merged_into[$tip]}"; return 0 ;;
|
|
live|unknown) return 1 ;;
|
|
esac
|
|
if ! git cat-file -e "${tip}^{commit}" 2>/dev/null; then
|
|
merged_verdict["$tip"]=unknown
|
|
echo "::warning::prune: ${name}: its branch tip ${tip} is not in this checkout — cannot tell merged from live, keeping it"
|
|
return 1
|
|
fi
|
|
for entry in "${PROTECTED_TIPS[@]}"; do
|
|
ref="${entry%%:*}"; psha="${entry#*:}"
|
|
if git merge-base --is-ancestor "$tip" "$psha" 2>/dev/null; then
|
|
merged_verdict["$tip"]=dead
|
|
merged_into["$tip"]="$ref"
|
|
MERGED_INTO="$ref"
|
|
return 0
|
|
fi
|
|
done
|
|
merged_verdict["$tip"]=live
|
|
return 1
|
|
}
|
|
|
|
if [ "$LIVENESS_AVAILABLE" = "1" ]; then
|
|
pruned_any=0
|
|
# Tracked separately so the line below cannot contradict the decline lines
|
|
# above it: "none pruned" and "none found" are different outcomes, and a
|
|
# pass that declined every dead cache it found has not found none.
|
|
spared_any=0
|
|
for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do
|
|
[ -d "$dir" ] || continue
|
|
name=$(basename "$dir")
|
|
is_protected "$dir" && continue
|
|
if [ -z "${live_ns[$name]:-}" ]; then
|
|
why="no matching branch on origin"
|
|
why_summary="branch no longer exists on origin"
|
|
elif is_merged_dead "$name"; then
|
|
why="merged into ${MERGED_INTO}, whose tip already contains its every commit"
|
|
why_summary="merged into \`${MERGED_INTO}\`"
|
|
else
|
|
continue
|
|
fi
|
|
is_locked "$dir" && { spared_any=1; continue; }
|
|
dir_gb=$(usage_gb "$dir")
|
|
evict_dir "$dir" || { spared_any=1; continue; }
|
|
echo "::warning::pruned dead-branch cache ${name} (${dir_gb} GB) — ${why}"
|
|
summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — ${why_summary}"
|
|
pruned_any=1
|
|
done
|
|
if [ "$pruned_any" = "0" ]; then
|
|
if [ "$spared_any" = "1" ]; then
|
|
echo "every dead-branch cache found is still in use — none pruned this pass"
|
|
else
|
|
echo "no dead-branch caches found"
|
|
fi
|
|
fi
|
|
else
|
|
echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped"
|
|
fi
|
|
|
|
echo
|
|
echo "=== pass 2/3: disk pressure ==="
|
|
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
|
PCT_KB=$(( TOTAL_KB * MIN_FREE_PCT / 100 ))
|
|
|
|
# The two floors, and which of them governs. They are kept apart all the way
|
|
# down rather than collapsed here, because falling short of them means
|
|
# different things: the derived one predicts that the very next step cannot
|
|
# finish, and the percentage one is a hygiene target for the volume.
|
|
REQUIRED_KB="$CLONE_KB"
|
|
GOVERNS="the clone this run is about to make"
|
|
if [ "$PCT_KB" -gt "$REQUIRED_KB" ]; then
|
|
REQUIRED_KB="$PCT_KB"
|
|
GOVERNS="the ${MIN_FREE_PCT}% floor"
|
|
fi
|
|
echo "required: $(gb "$REQUIRED_KB") GB free — ${GOVERNS} (clone $(gb "$CLONE_KB") GB, floor $(gb "$PCT_KB") GB)"
|
|
|
|
own_report() {
|
|
if [ -d "$OWN_DIR" ]; then
|
|
echo "cache: $(basename "$OWN_DIR") $(usage_gb "$OWN_DIR") GB | $(report_df host "$1" "$2")"
|
|
else
|
|
# Ordinary now that this pass runs ahead of the seed: on a branch's first
|
|
# run of the day the directory does not exist yet, and reporting 0.0 GB
|
|
# for it would read as an emptied cache.
|
|
echo "cache: $(basename "$OWN_DIR") not seeded yet | $(report_df host "$1" "$2")"
|
|
fi
|
|
}
|
|
|
|
if [ "$FREE_KB" -ge "$REQUIRED_KB" ]; then
|
|
own_report "$FREE_KB" "$TOTAL_KB"
|
|
exit 0
|
|
fi
|
|
|
|
echo "::warning::$(report_df disk "$FREE_KB" "$TOTAL_KB") < the $(gb "$REQUIRED_KB") GB this run requires"
|
|
|
|
# What survived the pass, and why, in the order the pass considered them. Read
|
|
# only by the failure report at the bottom: a run that cannot fit its clone is
|
|
# a run whose log has to answer "then what is all that space?" without anyone
|
|
# having to reconstruct the pass by hand.
|
|
KEPT=()
|
|
|
|
# A plain loop over a pre-materialised, pre-sorted list rather than a live
|
|
# `find | while` pipeline, so `rm -rf` inside the loop cannot make a running
|
|
# directory walk re-observe its own deletions.
|
|
mapfile -t LRU < <(list_by_lru)
|
|
for dir in "${LRU[@]}"; do
|
|
[ -d "$dir" ] || continue
|
|
if reason=$(protected_reason "$dir"); then
|
|
KEPT+=("$(basename "$dir") — ${reason}")
|
|
continue
|
|
fi
|
|
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
|
[ "$FREE_KB" -ge "$REQUIRED_KB" ] && break
|
|
if is_locked "$dir"; then
|
|
KEPT+=("$(basename "$dir") — held open by a running job")
|
|
continue
|
|
fi
|
|
dir_gb=$(usage_gb "$dir")
|
|
if ! evict_dir "$dir"; then
|
|
KEPT+=("$(basename "$dir") — claimed by a job while its eviction was in flight")
|
|
continue
|
|
fi
|
|
echo "::warning::evicted $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)"
|
|
summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)"
|
|
done
|
|
|
|
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
|
|
|
# Falling short of the DERIVED requirement is a failure, not a warning. The
|
|
# seed step is next, it will clone that source, and it will run out of disk
|
|
# part-way through unsharing the clone's mutable paths — reported against a
|
|
# staging path, with nothing in the message about which cache was holding the
|
|
# space. Failing here says that instead.
|
|
#
|
|
# There is nothing to self-clear on this path and it is not skipped in error:
|
|
# a non-zero requirement means the seed is about to CLONE, which means this
|
|
# run has no own target dir to wipe (seed_clone_source returns nothing when it
|
|
# does), so pass 3 has no candidate. See the header.
|
|
if [ "$FREE_KB" -lt "$CLONE_KB" ]; then
|
|
echo "::error::prune: $(gb "$FREE_KB") GB free after evicting every eligible cache, but seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB — short by $(gb "$(( CLONE_KB - FREE_KB ))") GB"
|
|
summary_line "- **out of disk**: seeding from \`$(basename "$SEED_SRC")\` needs $(gb "$CLONE_KB") GB, $(gb "$FREE_KB") GB free"
|
|
echo "prune: kept, and why:"
|
|
for entry in ${KEPT[@]+"${KEPT[@]}"}; do echo " ${entry}"; done
|
|
[ "${#KEPT[@]}" -gt 0 ] || echo " (nothing — the volume holds no cache directories at all)"
|
|
exit 1
|
|
fi
|
|
|
|
if [ "$FREE_KB" -lt "$PCT_KB" ]; then
|
|
OWN_GB=$(usage_gb "$OWN_DIR")
|
|
echo "::warning::still under threshold after evicting every eligible sibling; clearing own $(basename "$OWN_DIR") (was ${OWN_GB} GB) — this run pays a cold rebuild"
|
|
summary_line "- **self-clear**: \`$(basename "$OWN_DIR")\` (was ${OWN_GB} GB) wiped — this run pays a cold rebuild"
|
|
# Recreated empty rather than left absent, and that is what keeps this path
|
|
# out of the requirement above: the seed reuses an own target dir that
|
|
# exists, whatever is in it, so a self-cleared run clones nothing and needs
|
|
# no headroom. Leaving it absent would send that run to the base snapshot
|
|
# instead, needing a clone this pass has just spent its last eligible bytes
|
|
# not sizing for.
|
|
rm -rf "$OWN_DIR"
|
|
mkdir -p "$OWN_DIR"
|
|
else
|
|
echo "$(report_df post-eviction "$FREE_KB" "$TOTAL_KB") — sibling eviction recovered enough space"
|
|
fi
|