fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
Claude Opus 5
+224
-13
@@ -113,22 +113,31 @@ strip_cargo_locks() {
|
||||
# only ever run this against a staging directory nothing else can see yet
|
||||
# (see hardlink_clone_into's contract), so the brief window where the path is
|
||||
# absent is not observable.
|
||||
#
|
||||
# Returns non-zero if the copy or either rename failed. That status is
|
||||
# load-bearing: a failed unshare leaves the staging tree still aliasing its
|
||||
# source, which is the exact corruption `unshare_mutable_paths` exists to
|
||||
# prevent, so it must abort the clone rather than be swallowed.
|
||||
unshare_subtree() {
|
||||
local d="$1" tmp
|
||||
[ -d "$d" ] || return 0
|
||||
tmp="${d}.unshare.$$"
|
||||
rm -rf "$tmp"
|
||||
cp -a "$d" "$tmp"
|
||||
rm -rf "$d"
|
||||
mv -T "$tmp" "$d"
|
||||
cp -a "$d" "$tmp" || { rm -rf "$tmp"; return 1; }
|
||||
rm -rf "$d" || { rm -rf "$tmp"; return 1; }
|
||||
mv -T "$tmp" "$d" || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
_unshare_files() {
|
||||
# `-links +1` restricts the work to files that are actually shared, which
|
||||
# makes this idempotent and near-free on an already-unshared tree.
|
||||
#
|
||||
# The inner shell propagates a failure of any individual copy-and-rename out
|
||||
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
||||
# partially-unshared tree is reported rather than silently accepted.
|
||||
find "$@" -links +1 -print0 2>/dev/null |
|
||||
xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _
|
||||
return 0
|
||||
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
||||
}
|
||||
|
||||
# THE load-bearing function of this whole design.
|
||||
@@ -184,33 +193,235 @@ unshare_mutable_paths() {
|
||||
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
||||
for d in "${dirs[@]}"; do
|
||||
[ -n "$d" ] || continue
|
||||
unshare_subtree "$d"
|
||||
unshare_subtree "$d" || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare ${d}" >&2
|
||||
return 1
|
||||
}
|
||||
done
|
||||
_unshare_files "$root" -type f -name '*.d'
|
||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json'
|
||||
_unshare_files "$root" -type f -name '*.d' || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reader markers: the consume side's half of the seed-vs-republish interlock
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# A hardlink clone reads its source over many seconds. The publish side
|
||||
# rotates a snapshot with two renames and then unlinks the generation it
|
||||
# rotated away — and unlinking entries out from under an in-flight directory
|
||||
# walk is what produces a SILENTLY truncated clone: `cp -al` reports the
|
||||
# entries it manages to stat, and simply never sees a subdirectory that was
|
||||
# unlinked before it read the parent's listing. Exit status alone does not
|
||||
# catch that case.
|
||||
#
|
||||
# So the two sides interlock through a marker file, and the ordering is what
|
||||
# makes it sound rather than probabilistic:
|
||||
#
|
||||
# Consumer: create .reading-<snap>-<tag> -> stat <snap> -> cp -al
|
||||
# Publisher: mv <snap> aside -> mv new into place -> scan for markers
|
||||
# -> unlink the rotated-away generation
|
||||
#
|
||||
# If a consumer's `stat` resolved to the OLD generation, that stat happened
|
||||
# before the publisher's first rename, so its marker — created strictly
|
||||
# earlier still — was already on disk before the publisher's scan, which
|
||||
# happens strictly after that rename. The publisher therefore cannot miss it.
|
||||
# A consumer that creates its marker after the scan necessarily resolves the
|
||||
# path to the NEW generation, which is not the one being unlinked.
|
||||
#
|
||||
# The wait is bounded (CACHE_READ_GRACE_SECONDS). Exceeding it does not force
|
||||
# the unlink: reclamation of that generation is DEFERRED to a later publish
|
||||
# instead. The residual is therefore disk, not correctness.
|
||||
CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}"
|
||||
# A marker older than this belongs to a job the runner killed before it could
|
||||
# clean up. Honouring one forever would let a crashed job pin an entire
|
||||
# snapshot generation on disk permanently.
|
||||
CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}"
|
||||
|
||||
reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; }
|
||||
|
||||
reader_lock_acquire() {
|
||||
date +%s > "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
|
||||
return 0
|
||||
}
|
||||
|
||||
reader_lock_release() {
|
||||
rm -f "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
|
||||
return 0
|
||||
}
|
||||
|
||||
# Prints the number of live readers of <source-name> under <marker-root>, and
|
||||
# sweeps markers past the staleness threshold as it goes.
|
||||
live_reader_count() {
|
||||
local root="$1" name="$2" now marker age n=0
|
||||
now=$(date +%s)
|
||||
for marker in "$root"/.reading-"$name"-*; do
|
||||
[ -e "$marker" ] || continue
|
||||
age=$(( now - $(stat -c '%Y' "$marker" 2>/dev/null || echo "$now") ))
|
||||
if [ "$age" -lt "$CACHE_READ_STALE_SECONDS" ]; then
|
||||
n=$((n + 1))
|
||||
else
|
||||
echo "readers: sweeping stale marker $(basename "$marker") (${age}s old > ${CACHE_READ_STALE_SECONDS}s)" >&2
|
||||
rm -f "$marker" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
printf '%s' "$n"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Blocks until nothing is reading <source-name>, or until the grace period
|
||||
# expires. Returns 0 when drained, 1 on timeout — the caller decides what to
|
||||
# do with a timeout, and in this codebase that decision is always "defer the
|
||||
# unlink", never "unlink anyway".
|
||||
wait_for_readers() {
|
||||
local root="$1" name="$2" grace="${3:-$CACHE_READ_GRACE_SECONDS}" deadline n waited=0
|
||||
deadline=$(( $(date +%s) + grace ))
|
||||
while :; do
|
||||
n=$(live_reader_count "$root" "$name")
|
||||
[ "$n" -eq 0 ] && {
|
||||
[ "$waited" -gt 0 ] && echo "readers: ${name} drained after ${waited}s"
|
||||
return 0
|
||||
}
|
||||
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||
echo "readers: ${n} job(s) still reading ${name} after ${grace}s" >&2
|
||||
return 1
|
||||
fi
|
||||
[ "$waited" = 0 ] && echo "readers: waiting for ${n} in-flight clone(s) of ${name} (grace ${grace}s)"
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The clone itself
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Number of times a torn clone is retried before the caller is failed. A tear
|
||||
# means the source changed identity or lost entries mid-walk, which is a
|
||||
# transient condition by definition — the publisher that caused it has already
|
||||
# put a complete new generation at the same path — so one retry almost always
|
||||
# suffices; the rest are headroom.
|
||||
CACHE_CLONE_ATTEMPTS="${CACHE_CLONE_ATTEMPTS:-4}"
|
||||
|
||||
_tree_entries() {
|
||||
local n
|
||||
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
|
||||
printf '%s' "$n"
|
||||
return 0
|
||||
}
|
||||
|
||||
_dir_inode() {
|
||||
stat -c '%i' "$1" 2>/dev/null || printf 'missing'
|
||||
return 0
|
||||
}
|
||||
|
||||
write_cache_lock() {
|
||||
local dir="$1" id="$2"
|
||||
[ -n "$id" ] || return 0
|
||||
[ -d "$dir" ] || return 0
|
||||
date +%s > "${dir}/.ci-lock-${id}" 2>/dev/null || true
|
||||
return 0
|
||||
}
|
||||
|
||||
# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
|
||||
# with a single atomic rename.
|
||||
#
|
||||
# hardlink_clone_into <src> <dst> <tag> [lock-id]
|
||||
#
|
||||
# The staging + rename is what closes the concurrent-seed race structurally
|
||||
# rather than by runner topology: a second job sharing this cache key either
|
||||
# sees DST absent (and stages its own clone, losing the rename harmlessly) or
|
||||
# sees it complete. There is no observable half-populated state, because a
|
||||
# directory rename is atomic and DST is never written through.
|
||||
#
|
||||
# The rename is necessary but NOT sufficient, and that gap is what this
|
||||
# function's retry loop closes. An atomic rename of a TRUNCATED tree publishes
|
||||
# a truncated tree atomically. Three things can truncate one:
|
||||
#
|
||||
# * `cp -al` failing partway (a source entry vanished after readdir listed
|
||||
# it) — caught by checking its exit status, which is why that status is
|
||||
# read into a variable here rather than left to an ambient `set -e` the
|
||||
# CALL SITES suppress anyway by invoking this function as a condition;
|
||||
# * `cp -al` succeeding while having silently never seen a subtree that was
|
||||
# unlinked before it read the parent's listing — caught only by the entry
|
||||
# count, since there is no error to report;
|
||||
# * the source being replaced wholesale mid-walk, so the clone splices two
|
||||
# generations — caught by comparing the source directory's inode before
|
||||
# and after.
|
||||
#
|
||||
# All three are verified on every attempt and a failing one restarts the
|
||||
# clone; a tree that fails the last attempt is deleted and reported, never
|
||||
# renamed into place. Combined with the reader marker (held across the copy,
|
||||
# which is what stops the publish side unlinking underneath it in the first
|
||||
# place), a partial tree cannot reach DST.
|
||||
#
|
||||
# `lock-id`, when given, writes this job's cache lock INTO the staging tree so
|
||||
# the directory already carries it the instant it appears under its final
|
||||
# name. Acquiring the lock after the rename would leave a freshly seeded
|
||||
# directory momentarily unlocked and therefore evictable by a concurrent job's
|
||||
# prune pass.
|
||||
#
|
||||
# Returns 0 if this caller's clone won the rename, 1 if another caller got
|
||||
# there first (the staging copy is discarded; DST is already valid).
|
||||
# there first (the staging copy is discarded; DST is already valid), and 2 if
|
||||
# the source could not be cloned consistently at all.
|
||||
hardlink_clone_into() {
|
||||
local src="$1" dst="$2" tag="$3" tmp parent
|
||||
local src="$1" dst="$2" tag="$3" lock_id="${4:-}"
|
||||
local parent tmp src_name attempt cp_rc n_before n_after i_before i_after
|
||||
|
||||
if [ ! -d "$src" ]; then
|
||||
echo "::error::clone: source ${src} does not exist" >&2
|
||||
return 2
|
||||
fi
|
||||
|
||||
parent=$(dirname "$dst")
|
||||
src_name=$(basename "$src")
|
||||
tmp="${parent}/.stage-${tag}"
|
||||
rm -rf "$tmp"
|
||||
cp -al "$src" "$tmp"
|
||||
|
||||
attempt=1
|
||||
while : ; do
|
||||
rm -rf "$tmp"
|
||||
# Marker first, then the identity read, then the copy — see the ordering
|
||||
# proof in the reader-marker section above; swapping the first two lines
|
||||
# is what would reintroduce the race.
|
||||
reader_lock_acquire "$parent" "$src_name" "$tag"
|
||||
i_before=$(_dir_inode "$src")
|
||||
n_before=$(_tree_entries "$src")
|
||||
cp_rc=0
|
||||
cp -al "$src" "$tmp" || cp_rc=$?
|
||||
n_after=$(_tree_entries "$tmp")
|
||||
i_after=$(_dir_inode "$src")
|
||||
reader_lock_release "$parent" "$src_name" "$tag"
|
||||
|
||||
if [ "$cp_rc" -eq 0 ] && [ "$i_before" != missing ] && [ "$i_before" = "$i_after" ] \
|
||||
&& [ "$n_after" -eq "$n_before" ]; then
|
||||
break
|
||||
fi
|
||||
|
||||
echo "::warning::clone: attempt ${attempt}/${CACHE_CLONE_ATTEMPTS} of ${src_name} was torn (cp rc=${cp_rc}, ${n_after}/${n_before} entries, source inode ${i_before} -> ${i_after}) — discarding and retrying" >&2
|
||||
rm -rf "$tmp"
|
||||
if [ "$attempt" -ge "$CACHE_CLONE_ATTEMPTS" ]; then
|
||||
echo "::error::clone: ${src} could not be read consistently in ${CACHE_CLONE_ATTEMPTS} attempts — refusing to publish a partial tree at ${dst}" >&2
|
||||
return 2
|
||||
fi
|
||||
attempt=$((attempt + 1))
|
||||
sleep 1
|
||||
done
|
||||
|
||||
strip_cargo_locks "$tmp"
|
||||
rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
|
||||
unshare_mutable_paths "$tmp"
|
||||
if ! unshare_mutable_paths "$tmp"; then
|
||||
echo "::error::clone: could not privately own the mutable paths of ${dst} — discarding the staging tree rather than publishing one that aliases ${src}" >&2
|
||||
rm -rf "$tmp"
|
||||
return 2
|
||||
fi
|
||||
write_cache_lock "$tmp" "$lock_id"
|
||||
|
||||
if mv -T "$tmp" "$dst" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user