#!/usr/bin/env bash # Eviction for the per-ref cache directories on the persistent volume. # # Usage: prune-cache.sh \ # [own-key] [base-key] [fallback-dir] # protected-branches space-separated raw refs (e.g. "dev main") # min-free-percent a FLOOR, not the gate — 0 to rely on the derived # requirement alone (the default) # own-key, base-key, fallback-dir # the same three the seed step resolves its source from. # Given them, this pass sizes the volume for the clone # that step is about to make; without them it has only # the percentage floor, and says so. # # Optional environment: # STALE_LOCK_SECONDS age past which a .ci-lock-* marker is treated as # abandoned (default 7200) # CACHE_LIVENESS "false"/"0" to skip the liveness pass entirely # CACHE_DF_OVERRIDE " ", for the selftest # EVICTION_ASIDE_SETTLE_SECONDS # how long a directory renamed aside for eviction is # left alone before another pass may reclaim it # (default 60) # # Three passes, in order: # # 1. LIVENESS — every target-*/snapshot-* directory whose branch is DEAD is # removed UNCONDITIONALLY, not gated on free space. A directory for a # branch nothing will build again is pure loss; waiting for disk pressure # to notice means paying for it until then. Two signals make a branch # dead, and the second exists because the first alone is inert on a forge # that keeps branches after merge (gitdan-actions#20): # # DELETED — the branch is no longer on origin at all. # MERGED — the branch is still on origin, but its tip is an ancestor # of a protected branch's tip, so every commit it holds is # already on the branch its cache would be re-cloned from. # # Skipped entirely, loudly, if the signal itself is unavailable — # "couldn't determine" is never folded into "dead", for either signal # and at either granularity: a checkout that cannot answer ancestry at # all skips the merged half, and a single branch whose tip is not in the # checkout is kept with a warning naming it. # 2. PRESSURE — if free space is under the requirement, evict remaining # (now necessarily live) directories oldest-first until it clears. The # requirement is what the seed step is about to need to clone its source, # MEASURED off that source (see clone_headroom_kb in cache-lib.sh), and a # percentage floor only if one is configured. A percentage cannot express # this: the failure it has to prevent is a clone running out of disk # part-way through unsharing its mutable paths, and how much that needs # is a property of the snapshot, not of the volume. # 3. SELF-CLEAR — if pass 2 still isn't enough for the percentage floor, # wipe this run's own target dir and pay a cold rebuild, reported to the # job summary as well as the log, because a warning on a green run is # what lets a silently 4x-slower job go unnoticed. It is not a way out of # the derived requirement: a run that has an own target dir to wipe is a # run whose seed reuses it and clones nothing, so that requirement is # zero. Falling short of a NON-ZERO derived requirement fails the job # here, naming the shortfall and what was kept instead of it — the seed # would otherwise fail seconds later against a half-unshared staging # tree, which is the failure this pass exists to pre-empt. # # Reactive-only, with no hard cap on cache size: a workspace's natural working # set is what it is, and bounding the footprint preemptively means wiping # useful content before it is actually causing host pressure. The bound is # physics, not an arbitrary GB number. # # EVICTION ORDER, and why it is the reverse of the obvious one: within the # pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones. # A snapshot is a hardlink clone of a live target dir and of every consumer # cloned from it, so removing it frees almost no real bytes — its inodes stay # alive through those other links — while costing every future PR its warm # start. Evicting snapshots first would be nearly pure loss. Target # directories are where a branch's own divergent artifacts actually live, so # they are what freeing space means. # # Two exclusions every pass respects: # # PROTECTED — the publisher branches' target and snapshot directories, and # this run's own target dir, are never candidates in any pass. Evicting a # publisher's snapshot doesn't free real disk (every open PR's clone keeps # the data alive) but does force every subsequent PR to start cold, which is # the entire benefit this scheme exists to deliver. # # LOCKED — a directory carrying a .ci-lock-* marker younger than # STALE_LOCK_SECONDS is held open by a running job, or named by a live # .reading--* marker (a job is hardlink-cloning it this instant), is # skipped by every pass, however dead and however tight the disk. This is # what makes eviction safe on a runner with more than one execution slot. # An older marker is treated as abandoned and logged as such, so an # actually-still-running job that somehow exceeds the threshold is visible # in the log rather than silently losing its cache mid-build. # # That exclusion is decided TWICE per eviction — once as the cheap filter # that keeps a held directory out of the pass at all, and once after the # directory has been renamed aside, which is the decision the unlink # actually rests on. See evict_dir. # # Liveness is resolved by `git ls-remote --heads origin`, wrapped in a # timeout. A directory name cannot be inverted back to a branch name (the # sanitiser is lossy and the disambiguating suffix is a one-way hash), so this # goes the other direction: it recomputes the expected directory names for # every branch origin reports, using cache-lib.sh's OWN cache_key function — # the same one the seed step used to create them. Reusing that function rather # than reimplementing a lookalike is what makes the classification sound; any # drift between two spellings would silently misclassify every directory. # # The merged half reads the TIP SHA out of that same `ls-remote` output and # asks `git merge-base --is-ancestor` against each protected branch's tip, # using the objects in this job's own checkout. Ancestry is only decidable # where the objects are there to decide it, so the answer "I cannot tell" # exists and is distinct from "not merged" everywhere it can arise: # # - a shallow checkout makes a MISSING object prove nothing, so the whole # signal is withheld rather than read as "no branch is merged"; # - a protected tip that is not in the checkout is not used as an anchor; # - a branch tip that is not in the checkout is kept, loudly. # # A squash or rebase merge leaves no ancestor relationship at all, so its # branch reads as live here. That is a missed reclamation, not a wrong one, # and the deleted-branch signal still covers it once the branch is removed. # # Only ever globs inside . Another project's volume is a different # Docker named volume and is not mounted in this container at all, so "stays # scoped to this repo's cache" holds structurally, not by convention. set -euo pipefail . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" ROOT="${1:?usage: prune-cache.sh [own-key] [base-key] [fallback-dir]}" OWN_DIR="${2:?}" PROTECTED_REFS="${3:-}" MIN_FREE_PCT="${4:-0}" OWN_KEY="${5:-}" BASE_KEY="${6:-}" FALLBACK="${7:-}" # Mirrored by daniel/gitdan's ci-cache-reclaim.sh, whose copy must be >= this # one — raising this without raising theirs first lets that script treat a lock # this side still honours as abandoned. Same direction and same reasoning as # CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh. STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}" # An aside directory is in flight for one rename plus one marker glob — # milliseconds. Anything older belongs to a pass that died between the two, so # an age is what separates "another pass is mid-eviction" from "a leftover", # and it separates them without having to identify the pass that created it. # The `$$` in an aside's name was that pass's PID inside its own job # container, so testing it with `kill -0` from a different one is not # unreliable, it is meaningless — and PIDs recycle besides. Three orders of # magnitude of headroom over the operation it covers, and short enough that a # genuine leftover is reclaimed by the next run rather than lingering while # the volume is under pressure. EVICTION_ASIDE_SETTLE_SECONDS="${EVICTION_ASIDE_SETTLE_SECONDS:-60}" # What the seed step is about to do, resolved through the same function that # step resolves it with (cache-lib.sh's seed_source_candidates). Empty when it # will clone nothing at all: its own target dir already exists and it reuses # it, or no source exists and it starts cold. Either way the derived # requirement is zero, because nothing is about to be copied. # # gb() is for reporting only. Every comparison below is in KB, because # `read_df` reports KB and rounding a threshold to a tenth of a GB either # passes a run that cannot fit or evicts a cache the run did not need. gb() { awk -v k="${1:-0}" 'BEGIN { printf "%.1f", k / 1048576 }'; } SEED_SRC="" CLONE_KB=0 if [ -n "$OWN_KEY" ]; then SEED_SRC=$(seed_clone_source "$ROOT" "$OWN_KEY" "$BASE_KEY" "$FALLBACK") if [ -n "$SEED_SRC" ]; then # A full walk of the source, and the reason this pass moved ahead of the # seed rather than staying where it was: the number is only useful before # the clone it describes. CLONE_KB=$(clone_headroom_kb "$SEED_SRC") echo "clone requirement: seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB free (measured from its mutable set)" else echo "clone requirement: none — this run reuses its own cache or starts cold, so nothing will be copied" fi else echo "clone requirement: not derivable — no cache key was passed to this pass; the ${MIN_FREE_PCT}% floor is the only gate" fi declare -A protected_ns=() for ref in $PROTECTED_REFS; do suffix=$(cache_key "$ref") protected_ns["target-${suffix}"]=1 protected_ns["snapshot-${suffix}"]=1 done # Prints why is off limits to every pass, or nothing when it is a # candidate. The reason is not decoration: it is what the failure report at # the bottom lists against each directory it kept while running out of space. # # SEED_SRC is the third exclusion and the one this script did not used to need. # The prune ran after the seed, so the source had already been cloned and the # reader marker over it was gone; running BEFORE the seed puts the directory # this run is about to read squarely in the candidate set, and pass 1 would # take it the moment its branch merged. protected_reason() { local dir="$1" name name=$(basename "$dir") [ "$dir" = "$OWN_DIR" ] && { printf 'this run own cache'; return 0; } [ -n "$SEED_SRC" ] && [ "$dir" = "$SEED_SRC" ] && { printf 'the source this run is about to clone'; return 0; } [ -n "${protected_ns[$name]:-}" ] && { printf 'a protected branch cache'; return 0; } return 1 } is_protected() { protected_reason "$1" >/dev/null; } # is_locked [name] # # `name` is the directory's own name for reporting and for the reader-marker # lookup, which matters when has been renamed aside for eviction: the # markers a consumer publishes are keyed on the name it resolved, not on # whatever the eviction pass has since called the directory. is_locked() { local dir="$1" name="${2:-$(basename "$1")}" now lock_file lock_age locked=1 readers now=$(date +%s) # A directory being hardlink-cloned right now carries no .ci-lock-* of its # own — a snapshot has its locks stripped by construction — so the reader # markers are the only signal that unlinking it would truncate somebody's # in-flight clone. Today no reachable configuration prunes a snapshot (only # protected refs publish them, and protected refs are excluded from every # pass), which makes this guard redundant *by policy* — evict_dir's second # look is what stops policy being the reason it is safe. readers=$(live_reader_count "$ROOT" "$name") if [ "$readers" -gt 0 ]; then echo " ${name}: ${readers} job(s) currently cloning it — not a candidate" locked=0 fi for lock_file in "$dir"/.ci-lock-*; do [ -e "$lock_file" ] || continue lock_age=$(( now - $(stat -c '%Y' "$lock_file") )) if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then echo " ${name}: held open by $(basename "$lock_file") (${lock_age}s old)" locked=0 else echo " ${name}: ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned" fi done return "$locked" } # evict_dir # # Unlinks , or declines to and says why. Returns 0 only if the directory # is actually gone. # # The caller has already established that is a candidate, which is not # the same as establishing that it is still one at the instant of the unlink: # a consumer publishes its reader marker whenever it starts a clone, and the # `du` that measures the directory between those two points runs for seconds # on a multi-GB tree. Unlinking under a live clone truncates it silently — # `cp -al` never reports a subtree that was removed before it read the # parent's listing (see cache-lib.sh's reader-marker section). # # So the directory is renamed aside first and only then re-examined, which is # what makes the second look conclusive rather than merely closer to the # unlink. It is publish-snapshot.sh's rotation, and it rests on the same # ordering proof: a consumer publishes its marker BEFORE it resolves the # source path, so one that resolved this directory did so before the rename # and therefore published its marker before the scan below, which happens # strictly after that rename. A consumer arriving after the rename cannot # resolve the path at all and falls through to its own cold-start path — the # same safe degrade the publisher's swap window already produces. # # The rename disturbs nothing already in flight: it unlinks no entry and # leaves the source inode unchanged, which is precisely why the publish side # can rotate a snapshot out from under a live reader. A declined eviction # therefore costs a deferred eviction and nothing else. # # `.evicting--` is a contract name shared with daniel/gitdan's # host-level arbiter (see the cross-repo contract at the top of cache-lib.sh). # The sweep at the top of this script reclaims these on every pass, so the only # one that reaches the arbiter belongs to a repo whose workflow has stopped # running — which is exactly the case no in-workflow pass can reach. evict_dir() { local dir="$1" name aside name=$(basename "$dir") aside="${ROOT}/.evicting-${name}-$$" mv -T "$dir" "$aside" 2>/dev/null || { echo " ${name}: could not be set aside for eviction — skipped this pass" return 1 } if is_locked "$aside" "$name"; then if mv -T "$aside" "$dir" 2>/dev/null; then echo " ${name}: claimed by a job while its eviction was in flight — restored, not evicted" elif [ -d "$aside" ]; then echo "::warning::prune: ${name} was claimed mid-eviction and its own name is taken again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain" else echo "::warning::prune: ${name} was claimed mid-eviction and is already gone — another pass reclaimed it after its readers drained" fi return 1 fi rm -rf "$aside" return 0 } # Directories oldest-first, target-* before snapshot-* (see the header). # The sort key is a rank digit followed by a zero-padded mtime, so the two # groups sort as blocks rather than interleaving by age. `.cache-last-used` # is the marker every run touches; a run that hits the cache for every crate # may write nothing at all inside the tree, which would make a # just-used directory look stale without it. Fall back to the directory's # own mtime when the marker is missing (a partially-written directory from an # interrupted run is still a valid, if less precise, "last touched" signal). list_by_lru() { local rank d ts for rank in 0:target 1:snapshot; do for d in "$ROOT"/"${rank#*:}"-*; do [ -d "$d" ] || continue if [ -e "$d/.cache-last-used" ]; then ts=$(stat -c '%Y' "$d/.cache-last-used" 2>/dev/null || echo 0) else ts=$(stat -c '%Y' "$d" 2>/dev/null || echo 0); fi printf '%s%012d\t%s\n' "${rank%%:*}" "$ts" "$d" done done | sort | cut -f2- } # Deferred reclamations from an earlier pass: a directory renamed aside for # eviction that could not be unlinked, because a job claimed it inside the # window and its own name was taken again before it could be restored — or # whose run was killed between the rename and the unlink. Nothing below would # ever see one: every pass globs target-*/snapshot-*, which a dotted name does # not match. Left unswept it is permanently unreclaimable disk on the one # volume whose entire problem is disk. # # TWO DISTINCT PROPERTIES HOLD HERE, and neither implies the other. # # No clone can be truncated by the unlink below, by construction: the aside # name only comes into existence after the evicting pass's rename, so a # consumer that resolved the directory published its marker strictly before # that rename and therefore before the count below — it cannot be missed. A # consumer that arrives later cannot resolve the path at all. This holds under # any interleaving and needs no settle window. # # No pass mid-eviction is mistaken for a leftover, by bound rather than by # construction, and this is what the settle window is for. Without it a # concurrent pass could unlink an aside its owner is still deciding about, and # `rm -rf` traverses fd-relative: the owner's restore can then republish a # half-emptied tree under a live cache name. What the window guarantees is that # the two cannot be confused within it. What it does not guarantee is the # pathological case beyond it — an evicting pass suspended past the window and # then resumed finds its aside reclaimed and fails its restore, saying so. now=$(date +%s) for aside in "$ROOT"/.evicting-*; do [ -d "$aside" ] || continue # Change time, not modification time. `mv` leaves a directory's mtime alone # (a cache last written days ago keeps a days-old mtime, which is what # list_by_lru wants and exactly the wrong signal here) but rename(2) does # update ctime, so %Z is when this directory was set aside and %Y is not. aside_age=$(( now - $(stat -c '%Z' "$aside" 2>/dev/null || echo "$now") )) if [ "$aside_age" -lt "$EVICTION_ASIDE_SETTLE_SECONDS" ]; then echo "prune: $(basename "$aside") was set aside ${aside_age}s ago — another pass may still be evicting it, leaving it alone" continue fi aside_name=$(basename "$aside"); aside_name="${aside_name#.evicting-}"; aside_name="${aside_name%-*}" if [ "$(live_reader_count "$ROOT" "$aside_name")" -gt 0 ]; then echo "prune: $(basename "$aside") is still being read — deferring its reclamation again" continue fi echo "prune: reclaiming deferred eviction $(basename "$aside")" rm -rf "$aside" done echo "=== pass 1: liveness (unconditional, not gated on free space) ===" declare -A live_ns=() # The tip SHA origin reports for the branch each directory name belongs to. # Same output, same loop, one field over — reading it from a second `git` call # would be reading a different instant. declare -A live_tip=() declare -A remote_tip_of=() LIVENESS_AVAILABLE=0 LIVENESS_REASON="" if [ "${CACHE_LIVENESS:-true}" = "0" ] || [ "${CACHE_LIVENESS:-true}" = "false" ]; then LIVENESS_REASON="disabled via CACHE_LIVENESS=${CACHE_LIVENESS}" elif remote_heads=$(timeout 20 git ls-remote --heads origin 2>&1); then LIVENESS_AVAILABLE=1 branch_count=0 while IFS= read -r line; do [ -n "$line" ] || continue case "$line" in *refs/heads/*) ;; *) continue ;; esac branch="${line#*refs/heads/}" [ -n "$branch" ] || continue sha="${line%%[[:space:]]*}" suffix=$(cache_key "$branch") live_ns["target-${suffix}"]=1 live_ns["snapshot-${suffix}"]=1 live_tip["target-${suffix}"]="$sha" live_tip["snapshot-${suffix}"]="$sha" remote_tip_of["$branch"]="$sha" branch_count=$((branch_count + 1)) done <<< "$remote_heads" echo "liveness: ${branch_count} live branches on origin" else LIVENESS_REASON="git ls-remote --heads origin failed or timed out" fi # The merged half of pass 1, and whether this checkout can answer it at all. # Every branch of this decision that ends in "no" ends in the signal being # WITHHELD, never in a directory being classified dead by default. MERGED_AVAILABLE=0 MERGED_REASON="" PROTECTED_TIPS=() declare -A merged_verdict=() declare -A merged_into=() MERGED_INTO="" if [ "$LIVENESS_AVAILABLE" = "1" ]; then if ! git rev-parse --git-dir >/dev/null 2>&1; then MERGED_REASON="not inside a git checkout" elif [ "$(git rev-parse --is-shallow-repository 2>/dev/null || echo unknown)" != "false" ]; then # In a shallow clone an absent commit is the normal case, so `--is-ancestor` # answers about the graph that was fetched rather than the one that exists. MERGED_REASON="the checkout is shallow, so a commit missing from it says nothing about ancestry" else for ref in $PROTECTED_REFS; do tip="${remote_tip_of[$ref]:-}" [ -n "$tip" ] || continue git cat-file -e "${tip}^{commit}" 2>/dev/null || continue PROTECTED_TIPS+=("${ref}:${tip}") done if [ "${#PROTECTED_TIPS[@]}" -gt 0 ]; then MERGED_AVAILABLE=1 echo "liveness: merged-branch detection anchored on ${PROTECTED_TIPS[*]%%:*}" else MERGED_REASON="none of the protected branch tips (${PROTECTED_REFS:-none configured}) is present in this checkout" fi fi [ "$MERGED_AVAILABLE" = "1" ] || \ echo "::warning::liveness: ${MERGED_REASON} — merged-but-undeleted branches keep their caches this run" fi # is_merged_dead # # True when the branch this directory belongs to is still on origin but every # commit it holds is already on a protected branch — a merged PR whose branch # the forge did not delete, which is the case zemyna hits on every merge and # which the deleted-branch signal above can never see. # # Memoised per tip because target- and snapshot- share one branch, # and because the "cannot tell" warning belongs to the branch rather than to # each of its directories. is_merged_dead() { local name="$1" tip="${live_tip[$1]:-}" entry ref psha MERGED_INTO="" [ "$MERGED_AVAILABLE" = "1" ] || return 1 [ -n "$tip" ] || return 1 case "${merged_verdict[$tip]:-}" in dead) MERGED_INTO="${merged_into[$tip]}"; return 0 ;; live|unknown) return 1 ;; esac if ! git cat-file -e "${tip}^{commit}" 2>/dev/null; then merged_verdict["$tip"]=unknown echo "::warning::prune: ${name}: its branch tip ${tip} is not in this checkout — cannot tell merged from live, keeping it" return 1 fi for entry in "${PROTECTED_TIPS[@]}"; do ref="${entry%%:*}"; psha="${entry#*:}" if git merge-base --is-ancestor "$tip" "$psha" 2>/dev/null; then merged_verdict["$tip"]=dead merged_into["$tip"]="$ref" MERGED_INTO="$ref" return 0 fi done merged_verdict["$tip"]=live return 1 } if [ "$LIVENESS_AVAILABLE" = "1" ]; then pruned_any=0 # Tracked separately so the line below cannot contradict the decline lines # above it: "none pruned" and "none found" are different outcomes, and a # pass that declined every dead cache it found has not found none. spared_any=0 for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do [ -d "$dir" ] || continue name=$(basename "$dir") is_protected "$dir" && continue if [ -z "${live_ns[$name]:-}" ]; then why="no matching branch on origin" why_summary="branch no longer exists on origin" elif is_merged_dead "$name"; then why="merged into ${MERGED_INTO}, whose tip already contains its every commit" why_summary="merged into \`${MERGED_INTO}\`" else continue fi is_locked "$dir" && { spared_any=1; continue; } dir_gb=$(usage_gb "$dir") evict_dir "$dir" || { spared_any=1; continue; } echo "::warning::pruned dead-branch cache ${name} (${dir_gb} GB) — ${why}" summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — ${why_summary}" pruned_any=1 done if [ "$pruned_any" = "0" ]; then if [ "$spared_any" = "1" ]; then echo "every dead-branch cache found is still in use — none pruned this pass" else echo "no dead-branch caches found" fi fi else echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped" fi echo echo "=== pass 2/3: disk pressure ===" read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" PCT_KB=$(( TOTAL_KB * MIN_FREE_PCT / 100 )) # The two floors, and which of them governs. They are kept apart all the way # down rather than collapsed here, because falling short of them means # different things: the derived one predicts that the very next step cannot # finish, and the percentage one is a hygiene target for the volume. REQUIRED_KB="$CLONE_KB" GOVERNS="the clone this run is about to make" if [ "$PCT_KB" -gt "$REQUIRED_KB" ]; then REQUIRED_KB="$PCT_KB" GOVERNS="the ${MIN_FREE_PCT}% floor" fi echo "required: $(gb "$REQUIRED_KB") GB free — ${GOVERNS} (clone $(gb "$CLONE_KB") GB, floor $(gb "$PCT_KB") GB)" own_report() { if [ -d "$OWN_DIR" ]; then echo "cache: $(basename "$OWN_DIR") $(usage_gb "$OWN_DIR") GB | $(report_df host "$1" "$2")" else # Ordinary now that this pass runs ahead of the seed: on a branch's first # run of the day the directory does not exist yet, and reporting 0.0 GB # for it would read as an emptied cache. echo "cache: $(basename "$OWN_DIR") not seeded yet | $(report_df host "$1" "$2")" fi } if [ "$FREE_KB" -ge "$REQUIRED_KB" ]; then own_report "$FREE_KB" "$TOTAL_KB" exit 0 fi echo "::warning::$(report_df disk "$FREE_KB" "$TOTAL_KB") < the $(gb "$REQUIRED_KB") GB this run requires" # What survived the pass, and why, in the order the pass considered them. Read # only by the failure report at the bottom: a run that cannot fit its clone is # a run whose log has to answer "then what is all that space?" without anyone # having to reconstruct the pass by hand. KEPT=() # A plain loop over a pre-materialised, pre-sorted list rather than a live # `find | while` pipeline, so `rm -rf` inside the loop cannot make a running # directory walk re-observe its own deletions. mapfile -t LRU < <(list_by_lru) for dir in "${LRU[@]}"; do [ -d "$dir" ] || continue if reason=$(protected_reason "$dir"); then KEPT+=("$(basename "$dir") — ${reason}") continue fi read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" [ "$FREE_KB" -ge "$REQUIRED_KB" ] && break if is_locked "$dir"; then KEPT+=("$(basename "$dir") — held open by a running job") continue fi dir_gb=$(usage_gb "$dir") if ! evict_dir "$dir"; then KEPT+=("$(basename "$dir") — claimed by a job while its eviction was in flight") continue fi echo "::warning::evicted $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)" summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)" done read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" # Falling short of the DERIVED requirement is a failure, not a warning. The # seed step is next, it will clone that source, and it will run out of disk # part-way through unsharing the clone's mutable paths — reported against a # staging path, with nothing in the message about which cache was holding the # space. Failing here says that instead. # # There is nothing to self-clear on this path and it is not skipped in error: # a non-zero requirement means the seed is about to CLONE, which means this # run has no own target dir to wipe (seed_clone_source returns nothing when it # does), so pass 3 has no candidate. See the header. if [ "$FREE_KB" -lt "$CLONE_KB" ]; then echo "::error::prune: $(gb "$FREE_KB") GB free after evicting every eligible cache, but seeding from $(basename "$SEED_SRC") needs $(gb "$CLONE_KB") GB — short by $(gb "$(( CLONE_KB - FREE_KB ))") GB" summary_line "- **out of disk**: seeding from \`$(basename "$SEED_SRC")\` needs $(gb "$CLONE_KB") GB, $(gb "$FREE_KB") GB free" echo "prune: kept, and why:" for entry in ${KEPT[@]+"${KEPT[@]}"}; do echo " ${entry}"; done [ "${#KEPT[@]}" -gt 0 ] || echo " (nothing — the volume holds no cache directories at all)" exit 1 fi if [ "$FREE_KB" -lt "$PCT_KB" ]; then OWN_GB=$(usage_gb "$OWN_DIR") echo "::warning::still under threshold after evicting every eligible sibling; clearing own $(basename "$OWN_DIR") (was ${OWN_GB} GB) — this run pays a cold rebuild" summary_line "- **self-clear**: \`$(basename "$OWN_DIR")\` (was ${OWN_GB} GB) wiped — this run pays a cold rebuild" # Recreated empty rather than left absent, and that is what keeps this path # out of the requirement above: the seed reuses an own target dir that # exists, whatever is in it, so a self-cleared run clones nothing and needs # no headroom. Leaving it absent would send that run to the base snapshot # instead, needing a clone this pass has just spent its last eligible bytes # not sizing for. rm -rf "$OWN_DIR" mkdir -p "$OWN_DIR" else echo "$(report_df post-eviction "$FREE_KB" "$TOTAL_KB") — sibling eviction recovered enough space" fi