daniel/gitdan-actions
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(prune-cache): close the reader-marker check-then-delete window

Browse Source 
Both eviction sites checked for a consumer's `.reading-` marker and then,
seconds later — `usage_gb` runs `du -sk` over a multi-GB tree between the
two — unlinked the directory. A consumer that started a clone inside that
gap had its source removed mid-walk, which `cp -al` does not report: a
subtree unlinked before its parent is listed is silently omitted.

Unreachable today, and only by policy: snapshots belong to protected refs
and protected refs never reach the marker check. `cargo-cache` and
`cargo-cache-publish` take that ref list as two independent inputs, so a
workflow listing a publisher in one and not the other arms this with no
code change at all.

Closed structurally, with publish-snapshot.sh's rotation rather than a new
mechanism: the candidate is renamed aside and only then re-examined, so the
scan the unlink rests on happens strictly after the rename. A consumer that
resolved the directory published its marker before that scan and cannot be
missed; one arriving after cannot resolve the path and starts cold, the same
degrade the publisher's swap window already produces. Renaming disturbs no
clone in flight — no entry is unlinked and the inode is unchanged — so a
declined eviction costs a deferred eviction and nothing else. A reprieved
cache is put back under its own name; one whose name a concurrent seed has
retaken is left aside and swept by a later pass once its readers drain,
since nothing else globs a dotted name.

prune-cache-selftest gains three scenarios (21 -> 31 assertions). Scenario
12 is the one that bites: the `du` the pass runs on its candidate publishes
the marker, placing it strictly after the check and strictly before the
unlink. Against check-then-delete, 1-11 pass and 12 fails; breaking only
the second look and leaving the rename fails it too.
This commit is contained in:
claude
2026-08-23 23:09:35 -05:00
parent 3b2dec6a50
commit 88ab063b64
3 changed files with 177 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/prune-cache.sh
+94 -11
View File
@@ -57,6 +57,11 @@
# actually-still-running job that somehow exceeds the threshold is visible
# in the log rather than silently losing its cache mid-build.
#
# That exclusion is decided TWICE per eviction — once as the cheap filter
# that keeps a held directory out of the pass at all, and once after the
# directory has been renamed aside, which is the decision the unlink
# actually rests on. See evict_dir.
#
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
# timeout. A directory name cannot be inverted back to a branch name (the
# sanitiser is lossy and the disambiguating suffix is a one-way hash), so this
@@ -93,34 +98,90 @@ is_protected() {
return 1
}
# is_locked <dir> [name]
#
# `name` is the directory's own name for reporting and for the reader-marker
# lookup, which matters when <dir> has been renamed aside for eviction: the
# markers a consumer publishes are keyed on the name it resolved, not on
# whatever the eviction pass has since called the directory.
is_locked() {
local dir="$1" now lock_file lock_age locked=1 readers
local dir="$1" name="${2:-$(basename "$1")}" now lock_file lock_age locked=1 readers
now=$(date +%s)
# A directory being hardlink-cloned right now carries no .ci-lock-* of its
# own — a snapshot has its locks stripped by construction — so the reader
# markers are the only signal that unlinking it would truncate somebody's
# in-flight clone. Today no reachable configuration prunes a snapshot (only
# protected refs publish them, and protected refs are excluded from every
# pass), which makes this guard redundant *by policy*. It is here so that
# stops being the reason it is safe.
readers=$(live_reader_count "$ROOT" "$(basename "$dir")")
# pass), which makes this guard redundant *by policy* — evict_dir's second
# look is what stops policy being the reason it is safe.
readers=$(live_reader_count "$ROOT" "$name")
if [ "$readers" -gt 0 ]; then
echo " $(basename "$dir"): ${readers} job(s) currently cloning it — not a candidate"
echo " ${name}: ${readers} job(s) currently cloning it — not a candidate"
locked=0
fi
for lock_file in "$dir"/.ci-lock-*; do
[ -e "$lock_file" ] || continue
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then
echo " $(basename "$dir"): held open by $(basename "$lock_file") (${lock_age}s old)"
echo " ${name}: held open by $(basename "$lock_file") (${lock_age}s old)"
locked=0
else
echo " $(basename "$dir"): ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned"
echo " ${name}: ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned"
fi
done
return "$locked"
}
# evict_dir <dir>
#
# Unlinks <dir>, or declines to and says why. Returns 0 only if the directory
# is actually gone.
#
# The caller has already established that <dir> is a candidate, which is not
# the same as establishing that it is still one at the instant of the unlink:
# a consumer publishes its reader marker whenever it starts a clone, and the
# `du` that measures the directory between those two points runs for seconds
# on a multi-GB tree. Unlinking under a live clone truncates it silently —
# `cp -al` never reports a subtree that was removed before it read the
# parent's listing (see cache-lib.sh's reader-marker section).
#
# So the directory is renamed aside first and only then re-examined, which is
# what makes the second look conclusive rather than merely closer to the
# unlink. It is publish-snapshot.sh's rotation, and it rests on the same
# ordering proof: a consumer publishes its marker BEFORE it resolves the
# source path, so one that resolved this directory did so before the rename
# and therefore published its marker before the scan below, which happens
# strictly after that rename. A consumer arriving after the rename cannot
# resolve the path at all and falls through to its own cold-start path — the
# same safe degrade the publisher's swap window already produces.
#
# The rename disturbs nothing already in flight: it unlinks no entry and
# leaves the source inode unchanged, which is precisely why the publish side
# can rotate a snapshot out from under a live reader. A declined eviction
# therefore costs a deferred eviction and nothing else.
evict_dir() {
local dir="$1" name aside
name=$(basename "$dir")
aside="${ROOT}/.evicting-${name}-$$"
mv -T "$dir" "$aside" 2>/dev/null || {
echo " ${name}: could not be set aside for eviction — skipped this pass"
return 1
}
if is_locked "$aside" "$name"; then
if mv -T "$aside" "$dir" 2>/dev/null; then
echo " ${name}: claimed by a job while its eviction was in flight — restored, not evicted"
else
echo "::warning::prune: ${name} was claimed mid-eviction and its own name is occupied again — leaving $(basename "$aside") for a later pass to reclaim once its readers drain"
fi
return 1
fi
rm -rf "$aside"
return 0
}
# Directories oldest-first, target-* before snapshot-* (see the header).
# The sort key is a rank digit followed by a zero-padded mtime, so the two
# groups sort as blocks rather than interleaving by age. `.cache-last-used`
@@ -141,6 +202,28 @@ list_by_lru() {
done | sort | cut -f2-
}
# Deferred reclamations from an earlier pass: a directory renamed aside for
# eviction that could not be unlinked, because a job claimed it inside the
# window and its own name was occupied again before it could be restored — or
# whose run was killed between the rename and the unlink. Nothing below would
# ever see one: every pass globs target-*/snapshot-*, which a dotted name does
# not match. Left unswept it is permanently unreclaimable disk on the one
# volume whose entire problem is disk.
#
# Safe under a concurrent pass mid-eviction for the reason evict_dir is: the
# aside name only exists after that pass's rename, so a consumer that resolved
# the directory published its marker before the count below.
for aside in "$ROOT"/.evicting-*; do
[ -d "$aside" ] || continue
aside_name=$(basename "$aside"); aside_name="${aside_name#.evicting-}"; aside_name="${aside_name%-*}"
if [ "$(live_reader_count "$ROOT" "$aside_name")" -gt 0 ]; then
echo "prune: $(basename "$aside") is still being read — deferring its reclamation again"
continue
fi
echo "prune: reclaiming deferred eviction $(basename "$aside")"
rm -rf "$aside"
done
echo "=== pass 1: liveness (unconditional, not gated on free space) ==="
declare -A live_ns=()
LIVENESS_AVAILABLE=0
@@ -174,9 +257,9 @@ if [ "$LIVENESS_AVAILABLE" = "1" ]; then
[ -n "${live_ns[$name]:-}" ] && continue
is_locked "$dir" && continue
dir_gb=$(usage_gb "$dir")
echo "::warning::pruning dead-branch cache ${name} (${dir_gb} GB) — no matching branch on origin"
evict_dir "$dir" || continue
echo "::warning::pruned dead-branch cache ${name} (${dir_gb} GB) — no matching branch on origin"
summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — branch no longer exists on origin"
rm -rf "$dir"
pruned_any=1
done
[ "$pruned_any" = "1" ] || echo "no dead-branch caches found"
@@ -207,9 +290,9 @@ for dir in "${LRU[@]}"; do
[ "$FREE_KB" -ge "$THRESHOLD_KB" ] && break
is_locked "$dir" && continue
dir_gb=$(usage_gb "$dir")
echo "::warning::evicting $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)"
evict_dir "$dir" || continue
echo "::warning::evicted $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)"
summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)"
rm -rf "$dir"
done
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"