fix(cargo-cache): close the seed-vs-republish race the design claimed to close

The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
2026-08-23 16:40:21 -05:00
co-authored by Claude Opus 5
parent 248af3061e
commit f57e2a6013
9 changed files with 562 additions and 68 deletions
+49 -9
View File
@@ -130,14 +130,49 @@ the staging path into `target-<own>`.
**A push to a protected branch** has no base to layer over. It builds in its
own directory and, if the build goes green, republishes it as
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
one in. Consumers only ever observe a complete snapshot or none at all.
one in, then reclaim the old one once nothing is still reading it. Consumers
only ever observe a complete snapshot or none at all.
**Concurrency.** Two jobs sharing one cache key each stage under their own tag
and race on one atomic rename; the loser discards its staging copy. There is
no window in which a partially-populated directory is visible under the final
name — which means this does not depend on the runner having a single
execution slot. Two jobs then building in the same directory is Cargo's own
`.cargo-lock` territory, which is what that lock is for.
**Concurrency, on the destination.** Two jobs sharing one cache key each stage
under their own tag and race on one atomic rename; the loser discards its
staging copy. There is no window in which a partially-populated directory is
visible under the final name. Two jobs then building in the same directory is
Cargo's own `.cargo-lock` territory, which is what that lock is for.
**Concurrency, on the source.** The atomic rename is necessary and not
sufficient, because renaming a *truncated* tree publishes a truncated tree
atomically. A clone reads its source over many seconds, and a publisher
rotating that source unlinks the generation being read — at which point
`cp -al` can silently omit a subtree it never saw, and report success. Two
mechanisms, both required:
- **The publisher does not unlink under a reader.** A consumer publishes a
`.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
the publisher scans for markers *after* its first rename. A consumer holding
the old generation therefore published its marker before that scan and
cannot be missed, and one that arrives after the scan necessarily resolves
to the new generation. The publisher waits for readers to drain
(`read-grace-seconds`, default 300) and, if they do not, **defers** the
reclamation rather than forcing it — the old generation stays on disk and is
swept by a later publish.
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
exit status, the source directory's inode before and after (a wholesale
replacement mid-walk would otherwise splice two generations), and the entry
count (the only signal for a subtree unlinked before its parent was listed —
there is no error to read). A tree that fails any of the three is deleted
and the clone retried; one that fails the last attempt fails the job. A
partial tree never reaches the final name.
**What this does and does not guarantee.** *Correctness* is closed by
construction: no combination of publish and seed timing produces a target
directory holding part of one generation, and a source that cannot be read
consistently fails the job loudly instead of seeding a truncated cache.
*Disk reclamation* is bounded, not immediate: a consumer slower than the grace
period leaves one extra snapshot generation of directory entries on the volume
until the next publish of that snapshot sweeps it. That residual is capped at
one deferred generation per publisher ref, and its real cost is close to the
inode count rather than the byte count, since the artifacts are hardlinked to
whatever cloned them.
**Eviction** runs three passes: caches for branches that no longer exist on
origin are removed unconditionally; then, only if free space is under the
@@ -194,6 +229,8 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
@@ -251,14 +288,17 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
| suite | covers |
|---|---|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree |
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
Every suite runs the actual script, not a reimplementation of its logic, and
every fix scenario is paired with a control that reproduces the bug — a
scenario that passes either way proves nothing.
scenario that passes either way proves nothing. The concurrency scenarios race
real processes rather than mocking the interleaving, and gate the interfering
step on *observed* progress of the step it interferes with, so the window is
hit deterministically instead of on a fast machine's coin flip.
The action YAML holds no logic beyond wiring; everything testable lives in
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
+24 -5
View File
@@ -1,8 +1,8 @@
name: 'Cargo cache (publish)'
description: >-
Records this run''s build watermark and, on a publisher branch, atomically
Records this run's build watermark and, on a publisher branch, atomically
republishes its target directory as the immutable snapshot that other
branches'' caches are hardlink-cloned from.
branches' caches are hardlink-cloned from.
author: 'gitdan'
inputs:
@@ -19,9 +19,9 @@ inputs:
mode:
description: >-
publish — record the watermark, publish a snapshot if eligible,
release this job''s cache lock (the normal call, after a
release this job's cache lock (the normal call, after a
green build).
release-lock — release this job''s cache lock and do nothing else. Use
release-lock — release this job's cache lock and do nothing else. Use
in a final `if: always()` step so a failed run does not
leave a lock behind for the staleness grace period.
required: false
@@ -37,9 +37,25 @@ inputs:
because its ref is not the reference branch even when it targets one.
required: false
default: 'push'
read-grace-seconds:
description: >-
How long the snapshot swap waits for in-flight consumers to finish
cloning the generation it is replacing before reclaiming it. On timeout
the old generation is LEFT ON DISK and swept by a later publish — the
unlink is never forced, because unlinking a tree a consumer is walking
is what silently truncates that consumer's clone.
required: false
default: '300'
reader-stale-seconds:
description: >-
Age past which a consumer's read marker is treated as abandoned by a
job the runner killed. Without it one crashed job would pin a snapshot
generation on disk permanently.
required: false
default: '7200'
record-watermark:
description: >-
Record this run''s HEAD as the build watermark for this target dir.
Record this run's HEAD as the build watermark for this target dir.
True for PR runs too, not just publishers: a feature branch accumulates
its own build history across several pushes and needs its own watermark.
required: false
@@ -109,6 +125,9 @@ runs:
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
shell: bash
env:
CACHE_READ_GRACE_SECONDS: ${{ inputs.read-grace-seconds }}
CACHE_READ_STALE_SECONDS: ${{ inputs.reader-stale-seconds }}
run: |
set -euo pipefail
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
+17 -10
View File
@@ -22,9 +22,9 @@ inputs:
default: '10'
restore-mtimes:
description: >-
Restore every tracked file''s mtime from git history. Requires a
Restore every tracked file's mtime from git history. Requires a
full-history checkout (fetch-depth: 0). Set to false only if the build
does not use Cargo''s mtime-based freshness at all.
does not use Cargo's mtime-based freshness at all.
required: false
default: 'true'
prune:
@@ -53,7 +53,7 @@ inputs:
default: ''
watermark-file:
description: >-
Name of this job''s build-watermark file inside the target dir. MUST be
Name of this job's build-watermark file inside the target dir. MUST be
distinct per job when two jobs share one cache key. Defaults to
.ci-watermark-<job>-sha.
required: false
@@ -140,7 +140,13 @@ runs:
# Seeds this ref's target dir from the base's published snapshot. See
# scripts/seed-target-dir.sh — the staging-then-atomic-rename is what
# makes concurrent jobs sharing one cache key safe by construction rather
# than by the runner happening to have a single execution slot.
# than by the runner happening to have a single execution slot, and the
# clone's own consistency check plus the publish side's reader interlock
# are what make it safe against the base republishing MID-CLONE.
#
# The lock id is passed here as well as acquired in the next step: the
# seed writes it into the staging tree, so the directory carries a lock
# the instant it appears under its final name rather than a step later.
- id: seed
shell: bash
run: |
@@ -151,13 +157,14 @@ runs:
"${{ steps.resolve.outputs.base-key }}" \
"${{ inputs.cache-root }}" \
"${{ github.job }}-${{ github.run_id }}-$$" \
"${{ inputs.seed-fallback-dir }}"
"${{ inputs.seed-fallback-dir }}" \
"${{ steps.resolve.outputs.lock-id }}"
# Marks the directory as held open, so any job's prune pass (this one
# included) skips it, and stamps the LRU marker. The marker is touched
# unconditionally every run: a run that hits the cache for every crate may
# write nothing at all inside the tree, which would make a just-used
# directory look stale to the eviction pass.
# Re-stamps the lock the seed step already wrote (acquiring is idempotent
# — it rewrites the timestamp) and stamps the LRU marker. The marker is
# touched unconditionally every run: a run that hits the cache for every
# crate may write nothing at all inside the tree, which would make a
# just-used directory look stale to the eviction pass.
- shell: bash
run: |
set -euo pipefail
+224 -13
View File
@@ -113,22 +113,31 @@ strip_cargo_locks() {
# only ever run this against a staging directory nothing else can see yet
# (see hardlink_clone_into's contract), so the brief window where the path is
# absent is not observable.
#
# Returns non-zero if the copy or either rename failed. That status is
# load-bearing: a failed unshare leaves the staging tree still aliasing its
# source, which is the exact corruption `unshare_mutable_paths` exists to
# prevent, so it must abort the clone rather than be swallowed.
unshare_subtree() {
local d="$1" tmp
[ -d "$d" ] || return 0
tmp="${d}.unshare.$$"
rm -rf "$tmp"
cp -a "$d" "$tmp"
rm -rf "$d"
mv -T "$tmp" "$d"
cp -a "$d" "$tmp" || { rm -rf "$tmp"; return 1; }
rm -rf "$d" || { rm -rf "$tmp"; return 1; }
mv -T "$tmp" "$d" || return 1
return 0
}
_unshare_files() {
# `-links +1` restricts the work to files that are actually shared, which
# makes this idempotent and near-free on an already-unshared tree.
#
# The inner shell propagates a failure of any individual copy-and-rename out
# through xargs (which exits 123 if any invocation exits 1-125), so a
# partially-unshared tree is reported rather than silently accepted.
find "$@" -links +1 -print0 2>/dev/null |
xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _
return 0
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
}
# THE load-bearing function of this whole design.
@@ -184,33 +193,235 @@ unshare_mutable_paths() {
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
for d in "${dirs[@]}"; do
[ -n "$d" ] || continue
unshare_subtree "$d"
unshare_subtree "$d" || {
echo "::error::unshare_mutable_paths: failed to unshare ${d}" >&2
return 1
}
done
_unshare_files "$root" -type f -name '*.d'
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json'
_unshare_files "$root" -type f -name '*.d' || {
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
return 1
}
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
return 1
}
return 0
}
# ---------------------------------------------------------------------------
# Reader markers: the consume side's half of the seed-vs-republish interlock
# ---------------------------------------------------------------------------
#
# A hardlink clone reads its source over many seconds. The publish side
# rotates a snapshot with two renames and then unlinks the generation it
# rotated away — and unlinking entries out from under an in-flight directory
# walk is what produces a SILENTLY truncated clone: `cp -al` reports the
# entries it manages to stat, and simply never sees a subdirectory that was
# unlinked before it read the parent's listing. Exit status alone does not
# catch that case.
#
# So the two sides interlock through a marker file, and the ordering is what
# makes it sound rather than probabilistic:
#
# Consumer: create .reading-<snap>-<tag> -> stat <snap> -> cp -al
# Publisher: mv <snap> aside -> mv new into place -> scan for markers
# -> unlink the rotated-away generation
#
# If a consumer's `stat` resolved to the OLD generation, that stat happened
# before the publisher's first rename, so its marker — created strictly
# earlier still — was already on disk before the publisher's scan, which
# happens strictly after that rename. The publisher therefore cannot miss it.
# A consumer that creates its marker after the scan necessarily resolves the
# path to the NEW generation, which is not the one being unlinked.
#
# The wait is bounded (CACHE_READ_GRACE_SECONDS). Exceeding it does not force
# the unlink: reclamation of that generation is DEFERRED to a later publish
# instead. The residual is therefore disk, not correctness.
CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}"
# A marker older than this belongs to a job the runner killed before it could
# clean up. Honouring one forever would let a crashed job pin an entire
# snapshot generation on disk permanently.
CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}"
reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; }
reader_lock_acquire() {
date +%s > "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
return 0
}
reader_lock_release() {
rm -f "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
return 0
}
# Prints the number of live readers of <source-name> under <marker-root>, and
# sweeps markers past the staleness threshold as it goes.
live_reader_count() {
local root="$1" name="$2" now marker age n=0
now=$(date +%s)
for marker in "$root"/.reading-"$name"-*; do
[ -e "$marker" ] || continue
age=$(( now - $(stat -c '%Y' "$marker" 2>/dev/null || echo "$now") ))
if [ "$age" -lt "$CACHE_READ_STALE_SECONDS" ]; then
n=$((n + 1))
else
echo "readers: sweeping stale marker $(basename "$marker") (${age}s old > ${CACHE_READ_STALE_SECONDS}s)" >&2
rm -f "$marker" 2>/dev/null || true
fi
done
printf '%s' "$n"
return 0
}
# Blocks until nothing is reading <source-name>, or until the grace period
# expires. Returns 0 when drained, 1 on timeout — the caller decides what to
# do with a timeout, and in this codebase that decision is always "defer the
# unlink", never "unlink anyway".
wait_for_readers() {
local root="$1" name="$2" grace="${3:-$CACHE_READ_GRACE_SECONDS}" deadline n waited=0
deadline=$(( $(date +%s) + grace ))
while :; do
n=$(live_reader_count "$root" "$name")
[ "$n" -eq 0 ] && {
[ "$waited" -gt 0 ] && echo "readers: ${name} drained after ${waited}s"
return 0
}
if [ "$(date +%s)" -ge "$deadline" ]; then
echo "readers: ${n} job(s) still reading ${name} after ${grace}s" >&2
return 1
fi
[ "$waited" = 0 ] && echo "readers: waiting for ${n} in-flight clone(s) of ${name} (grace ${grace}s)"
sleep 1
waited=$((waited + 1))
done
}
# ---------------------------------------------------------------------------
# The clone itself
# ---------------------------------------------------------------------------
# Number of times a torn clone is retried before the caller is failed. A tear
# means the source changed identity or lost entries mid-walk, which is a
# transient condition by definition — the publisher that caused it has already
# put a complete new generation at the same path — so one retry almost always
# suffices; the rest are headroom.
CACHE_CLONE_ATTEMPTS="${CACHE_CLONE_ATTEMPTS:-4}"
_tree_entries() {
local n
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
printf '%s' "$n"
return 0
}
_dir_inode() {
stat -c '%i' "$1" 2>/dev/null || printf 'missing'
return 0
}
write_cache_lock() {
local dir="$1" id="$2"
[ -n "$id" ] || return 0
[ -d "$dir" ] || return 0
date +%s > "${dir}/.ci-lock-${id}" 2>/dev/null || true
return 0
}
# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
# with a single atomic rename.
#
# hardlink_clone_into <src> <dst> <tag> [lock-id]
#
# The staging + rename is what closes the concurrent-seed race structurally
# rather than by runner topology: a second job sharing this cache key either
# sees DST absent (and stages its own clone, losing the rename harmlessly) or
# sees it complete. There is no observable half-populated state, because a
# directory rename is atomic and DST is never written through.
#
# The rename is necessary but NOT sufficient, and that gap is what this
# function's retry loop closes. An atomic rename of a TRUNCATED tree publishes
# a truncated tree atomically. Three things can truncate one:
#
# * `cp -al` failing partway (a source entry vanished after readdir listed
# it) — caught by checking its exit status, which is why that status is
# read into a variable here rather than left to an ambient `set -e` the
# CALL SITES suppress anyway by invoking this function as a condition;
# * `cp -al` succeeding while having silently never seen a subtree that was
# unlinked before it read the parent's listing — caught only by the entry
# count, since there is no error to report;
# * the source being replaced wholesale mid-walk, so the clone splices two
# generations — caught by comparing the source directory's inode before
# and after.
#
# All three are verified on every attempt and a failing one restarts the
# clone; a tree that fails the last attempt is deleted and reported, never
# renamed into place. Combined with the reader marker (held across the copy,
# which is what stops the publish side unlinking underneath it in the first
# place), a partial tree cannot reach DST.
#
# `lock-id`, when given, writes this job's cache lock INTO the staging tree so
# the directory already carries it the instant it appears under its final
# name. Acquiring the lock after the rename would leave a freshly seeded
# directory momentarily unlocked and therefore evictable by a concurrent job's
# prune pass.
#
# Returns 0 if this caller's clone won the rename, 1 if another caller got
# there first (the staging copy is discarded; DST is already valid).
# there first (the staging copy is discarded; DST is already valid), and 2 if
# the source could not be cloned consistently at all.
hardlink_clone_into() {
local src="$1" dst="$2" tag="$3" tmp parent
local src="$1" dst="$2" tag="$3" lock_id="${4:-}"
local parent tmp src_name attempt cp_rc n_before n_after i_before i_after
if [ ! -d "$src" ]; then
echo "::error::clone: source ${src} does not exist" >&2
return 2
fi
parent=$(dirname "$dst")
src_name=$(basename "$src")
tmp="${parent}/.stage-${tag}"
rm -rf "$tmp"
cp -al "$src" "$tmp"
attempt=1
while : ; do
rm -rf "$tmp"
# Marker first, then the identity read, then the copy — see the ordering
# proof in the reader-marker section above; swapping the first two lines
# is what would reintroduce the race.
reader_lock_acquire "$parent" "$src_name" "$tag"
i_before=$(_dir_inode "$src")
n_before=$(_tree_entries "$src")
cp_rc=0
cp -al "$src" "$tmp" || cp_rc=$?
n_after=$(_tree_entries "$tmp")
i_after=$(_dir_inode "$src")
reader_lock_release "$parent" "$src_name" "$tag"
if [ "$cp_rc" -eq 0 ] && [ "$i_before" != missing ] && [ "$i_before" = "$i_after" ] \
&& [ "$n_after" -eq "$n_before" ]; then
break
fi
echo "::warning::clone: attempt ${attempt}/${CACHE_CLONE_ATTEMPTS} of ${src_name} was torn (cp rc=${cp_rc}, ${n_after}/${n_before} entries, source inode ${i_before} -> ${i_after}) — discarding and retrying" >&2
rm -rf "$tmp"
if [ "$attempt" -ge "$CACHE_CLONE_ATTEMPTS" ]; then
echo "::error::clone: ${src} could not be read consistently in ${CACHE_CLONE_ATTEMPTS} attempts — refusing to publish a partial tree at ${dst}" >&2
return 2
fi
attempt=$((attempt + 1))
sleep 1
done
strip_cargo_locks "$tmp"
rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
unshare_mutable_paths "$tmp"
if ! unshare_mutable_paths "$tmp"; then
echo "::error::clone: could not privately own the mutable paths of ${dst} — discarding the staging tree rather than publishing one that aliases ${src}" >&2
rm -rf "$tmp"
return 2
fi
write_cache_lock "$tmp" "$lock_id"
if mv -T "$tmp" "$dst" 2>/dev/null; then
return 0
fi
+20 -7
View File
@@ -49,12 +49,13 @@
# the entire benefit this scheme exists to deliver.
#
# LOCKED — a directory carrying a .ci-lock-* marker younger than
# STALE_LOCK_SECONDS is held open by a running job and is skipped by every
# pass, however dead and however tight the disk. This is what makes eviction
# safe on a runner with more than one execution slot. An older marker is
# treated as abandoned and logged as such, so an actually-still-running job
# that somehow exceeds the threshold is visible in the log rather than
# silently losing its cache mid-build.
# STALE_LOCK_SECONDS is held open by a running job, or named by a live
# .reading-<dir>-* marker (a job is hardlink-cloning it this instant), is
# skipped by every pass, however dead and however tight the disk. This is
# what makes eviction safe on a runner with more than one execution slot.
# An older marker is treated as abandoned and logged as such, so an
# actually-still-running job that somehow exceeds the threshold is visible
# in the log rather than silently losing its cache mid-build.
#
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
# timeout. A directory name cannot be inverted back to a branch name (the
@@ -93,8 +94,20 @@ is_protected() {
}
is_locked() {
local dir="$1" now lock_file lock_age locked=1
local dir="$1" now lock_file lock_age locked=1 readers
now=$(date +%s)
# A directory being hardlink-cloned right now carries no .ci-lock-* of its
# own — a snapshot has its locks stripped by construction — so the reader
# markers are the only signal that unlinking it would truncate somebody's
# in-flight clone. Today no reachable configuration prunes a snapshot (only
# protected refs publish them, and protected refs are excluded from every
# pass), which makes this guard redundant *by policy*. It is here so that
# stops being the reason it is safe.
readers=$(live_reader_count "$ROOT" "$(basename "$dir")")
if [ "$readers" -gt 0 ]; then
echo " $(basename "$dir"): ${readers} job(s) currently cloning it — not a candidate"
locked=0
fi
for lock_file in "$dir"/.ci-lock-*; do
[ -e "$lock_file" ] || continue
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
+50 -7
View File
@@ -23,11 +23,23 @@
# own cold-start path — a safe degrade that self-heals on its next run, not
# corruption.
#
# `rm -rf` on the old snapshot removes directory entries only. Any consumer
# that already hardlink-cloned from it keeps every inode alive through its own
# links, so a republish never pulls data out from under a running job — it
# just stops new consumers from seeing the old generation. Disk is reclaimed
# when the last clone referencing those inodes is itself evicted.
# `rm -rf` on the old snapshot removes directory entries only, so a consumer
# that has ALREADY FINISHED cloning from it keeps every inode alive through
# its own links. That is the easy half, and on its own it is not enough: a
# consumer still WALKING the old generation has its entries unlinked out from
# under it, and `cp -al` does not report a subtree that was removed before it
# read the parent's listing. That is a silently truncated clone — the failure
# mode this script's own selftest (scenario 8) reproduces against the
# unguarded version.
#
# So the unlink is interlocked with the consume side rather than
# unconditional: after the swap, this script waits for every in-flight reader
# of this snapshot to drain (see the reader-marker ordering proof in
# cache-lib.sh) and only then reclaims the rotated-away generation. If the
# grace period expires first, reclamation is DEFERRED — the directory is left
# under `.publish-old-<key>-<tag>` and swept by a later publish once no reader
# holds it. The residual of a very slow consumer is therefore one extra
# generation of directory entries on disk, never a torn clone.
set -euo pipefail
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
@@ -36,7 +48,11 @@ ROOT="${2:?}"; TAG="${3:?}"
SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
OLD="${ROOT}/.publish-old-${TAG}"
# Keyed by cache key as well as tag, so a deferred generation can be matched
# back to the snapshot whose readers must drain before it is safe to reclaim.
OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}"
SNAP_NAME=$(basename "$DST")
GRACE="${CACHE_READ_GRACE_SECONDS}"
if [ ! -d "$SRC" ]; then
echo "publish: no target dir at ${SRC} — nothing to snapshot"
@@ -47,6 +63,21 @@ fi
# concurrently running job's staging directory is never touched.
rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
# Deferred reclamation from an earlier publish of THIS snapshot whose readers
# had not drained in time. Safe to sweep now only if nothing is reading the
# snapshot at all: a reader holds the snapshot path, not the deferred name, so
# "no readers of snapshot-<key>" is the condition that makes every deferred
# generation of it unreachable. Over-conservative by design — a reader of the
# CURRENT generation also defers the sweep to the next publish, which costs a
# directory listing, not correctness.
if [ "$(live_reader_count "$ROOT" "$SNAP_NAME")" -eq 0 ]; then
for stale_old in "${ROOT}/.publish-old-${OWN_KEY}-"*; do
[ -d "$stale_old" ] || continue
echo "publish: reclaiming deferred snapshot generation $(basename "$stale_old")"
rm -rf "$stale_old"
done
fi
start=$(date +%s)
# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
# of every file Cargo rewrites in place (unshare_mutable_paths, called inside
@@ -63,7 +94,19 @@ hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {
if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
mv -T "$TMP_DST" "$DST"
rm -rf "$OLD"
# The scan below happens strictly after the rename above, which is what makes
# it impossible for a consumer holding the OLD generation to be missed: such a
# consumer resolved the path before that rename, and published its marker
# before that. See cache-lib.sh's reader-marker section.
if [ -d "$OLD" ]; then
if wait_for_readers "$ROOT" "$SNAP_NAME" "$GRACE"; then
rm -rf "$OLD"
else
echo "::warning::publish: a consumer is still cloning the previous ${SNAP_NAME} after ${GRACE}s — deferring reclamation of $(basename "$OLD") rather than unlinking a tree being read"
summary_line "- deferred reclaiming the previous \`${SNAP_NAME}\` generation (a consumer is still cloning it); it will be swept by a later publish"
fi
fi
echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"
+2 -1
View File
@@ -170,7 +170,8 @@
# the next run's diff base stays at the last GREEN build — over-inclusive
# (it may re-stamp files that a failed run partially rebuilt anyway) but
# never under-inclusive, which is the only direction that matters for
# soundness.set -euo pipefail
# soundness.
set -euo pipefail
repo_root=$(git rev-parse --show-toplevel)
cd "$repo_root"
+132 -1
View File
@@ -29,6 +29,21 @@
# at no point is a partially-populated directory visible under the final
# name. This is the property that replaces "the runner only has one job
# slot" with an actual guarantee.
# 8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the
# one that actually mattered: a consumer hardlink-cloning a snapshot
# while the publisher of that snapshot rotates it and unlinks the
# generation being read. Two seeds racing on a DESTINATION is a different
# race from a seed racing a publisher on its SOURCE, and only the second
# one can truncate a tree. Against the unguarded version this scenario
# reproduces a silent partial clone reported as success — 20,328 of
# 48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
# 9. DEFERRED RECLAMATION — when a consumer is STILL reading after the grace
# period, the publisher leaves the rotated-away generation on disk rather
# than unlinking a tree under an in-flight walk, and a later publish
# sweeps it once the reader is gone. The residual is disk, not a torn
# clone.
# 10. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct
# status instead of renaming whatever it managed to produce into place.
set -euo pipefail
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$script_dir/cache-lib.sh"
@@ -56,7 +71,32 @@ make_tree() {
: > "$d/debug/.cargo-lock"
}
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
# Always succeeds and always prints a number: an absent directory is "0
# entries so far", which is the normal state at the top of the progress poll
# below, not an error worth aborting the suite over.
tree_entries() {
local n
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
printf '%s\n' "$n"
return 0
}
# A tree wide enough that a hardlink clone of it takes long enough to be
# caught mid-walk. The race under test is a real interleaving, not a mocked
# one, so the fixture has to be big enough for the window to exist: a
# four-file tree clones in microseconds and no scheduling could ever land
# inside it. Built by cloning one small template directory N times, which is N
# forks rather than N*M file creations.
make_wide_tree() {
local d="$1" marker="$2" ndirs="$3" i
mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x"
for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done
for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done
rm -rf "$d/debug/deps/.tmpl"
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
}
BASE_KEY=$(cache_key dev)
OWN_KEY=$(cache_key feat/thing)
@@ -127,5 +167,96 @@ if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
fi
ok "neither racing job fell through to a cold start"
echo
echo "=== 8: seeding while the base republishes the snapshot underneath it ==="
ROT=$(cache_key feat/rotate)
rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY"
# Generation 1 is wide (the consumer will still be walking it when the swap
# happens); the generation replacing it is small, so the publisher's own
# staging clone does not itself outlast the consumer's.
make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800
make_wide_tree "$root/target-$BASE_KEY" gen2 8
gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) &
seed_pid=$!
# Rotate only once the clone is demonstrably mid-walk. Gating on observed
# progress rather than on a sleep is what makes the interleaving reproducible
# instead of a coin flip that passes on a fast machine for the wrong reason.
threshold=$(( gen1_entries / 5 ))
progress=0
deadline=$(( $(date +%s) + 60 ))
while :; do
progress=$(tree_entries "$root/.stage-jobRot")
if [ "$progress" -ge "$threshold" ]; then break; fi
if ! kill -0 "$seed_pid" 2>/dev/null; then
fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)"
fi
if [ "$(date +%s)" -ge "$deadline" ]; then
fail "the staging clone never reached ${threshold} of ${gen1_entries} entries"
fi
done
ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries"
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \
|| { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; }
wait "$seed_pid"
rot_dir="$root/target-$ROT"
[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; }
ok "the seed completed"
# THE assertion. Before the guard, this is where it failed: the seed reported
# `cloned in 1s` and exit 0 while target-<rot> held less than half the entries
# of the snapshot it claimed to have cloned. Comparing against the snapshot as
# it stands NOW is the right bar either way — a clone that raced the rotation
# must end up holding one complete generation, and a consumer caught mid-walk
# re-reads, so that generation is the new one.
snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
rot_entries=$(tree_entries "$rot_dir")
[ "$rot_entries" -eq "$snap_entries" ] \
|| fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)"
ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)"
assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two"
grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read"
ok "the torn read was detected and reported, not swallowed"
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)"
ok "no staging, reader-marker or deferred-generation scratch left behind"
echo
echo "=== 9: a reader that outlasts the grace period defers reclamation ==="
# A synthetic reader marker stands in for a consumer whose clone is slower
# than the grace period. Driving that with a real slow consumer would make the
# test's runtime the thing under test; the marker is the whole contract
# between the two sides, so holding one IS being a reader.
SNAP_NAME="snapshot-$BASE_KEY"
date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
make_wide_tree "$root/target-$BASE_KEY" gen3 4
CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubDefer > "$scratch/logDefer" 2>&1 \
|| { tail -40 "$scratch/logDefer"; fail "publish-snapshot.sh exited non-zero"; }
assert_content "$root/$SNAP_NAME/debug/.fingerprint/x/dep-lib-x" gen3 "the new generation was published regardless"
deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${BASE_KEY}-*" -print -quit)
[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
ok "the rotated-away generation was left on disk rather than unlinked under a reader"
grep -q 'deferring reclamation' "$scratch/logDefer" || fail "the deferral was not reported"
ok "the deferral is reported as a warning, not silent"
rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
make_wide_tree "$root/target-$BASE_KEY" gen4 4
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubSweep > "$scratch/logSweep" 2>&1 \
|| { tail -40 "$scratch/logSweep"; fail "publish-snapshot.sh exited non-zero"; }
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|| fail "the deferred generation was never reclaimed"
ok "a later publish reclaims the deferred generation once no reader holds it"
echo
echo "=== 10: a source that cannot be read fails loudly ==="
rc=0
hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$?
[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}"
ok "an unreadable source returns the distinct hard-failure status"
assert_absent "$root/target-nosuch" "nothing was renamed into place"
echo
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
+44 -15
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# Consume side: make this run's own target dir exist, warm, and private.
#
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir]
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir] [lock-id]
# own-key cache key for this run's own ref
# base-key cache key for the ref to layer over ("" for a run whose own
# ref IS a reference branch)
@@ -10,6 +10,14 @@
# directory so two concurrent jobs can never collide on it
# fallback-dir optional absolute path to seed from when no snapshot exists
# (a legacy flat cache dir during a migration, typically)
# lock-id optional cache-lock id. When given, the lock marker is
# written on EVERY path out of this script — including into
# the staging tree before its rename — so the directory is
# never observable under its final name without a lock. The
# action acquires the lock in a later step too; that step is
# idempotent, and this closes the window before it runs, where
# a concurrent job's prune pass could evict a directory that
# exists but is not yet held.
#
# The design in one paragraph: a PR branch's first run hardlink-clones the
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
@@ -27,17 +35,18 @@
set -euo pipefail
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
if [ $# -lt 4 ] || [ $# -gt 5 ]; then
echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2
if [ $# -lt 4 ] || [ $# -gt 6 ]; then
echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2
exit 1
fi
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}"
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
mkdir -p "$ROOT"
if [ -d "$OWN_DIR" ]; then
write_cache_lock "$OWN_DIR" "$LOCK_ID"
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
exit 0
@@ -62,20 +71,40 @@ for entry in "${CANDIDATES[@]}"; do
[ -d "$src" ] || continue
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
start=$(date +%s)
if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then
echo "seed: cloned in $(( $(date +%s) - start ))s"
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
else
# Another job sharing this cache key won the rename while we were
# cloning. Its directory is complete (the rename is the publish step), so
# there is nothing to do but use it — and nothing was ever observable in
# a half-seeded state.
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
fi
# The status is captured and dispatched on explicitly. Calling this as a
# bare `if` condition — which is what this script used to do — suppresses
# `set -e` for the whole call, so a hard clone failure could not abort the
# seed even in principle; the three outcomes are genuinely distinct and each
# needs its own handling.
clone_rc=0
hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$?
case "$clone_rc" in
0)
echo "seed: cloned in $(( $(date +%s) - start ))s"
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
;;
1)
# Another job sharing this cache key won the rename while we were
# cloning. Its directory is complete (the rename is the publish step),
# so there is nothing to do but use it — and nothing was ever
# observable in a half-seeded state.
write_cache_lock "$OWN_DIR" "$LOCK_ID"
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
;;
*)
# A source that could not be read consistently. Failing the job is the
# only safe answer: the alternative that used to happen here was
# seeding a truncated tree and reporting success, which hands Cargo a
# directory whose fingerprints and artifacts disagree.
echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2
exit 1
;;
esac
exit 0
done
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
mkdir -p "$OWN_DIR"
write_cache_lock "$OWN_DIR" "$LOCK_ID"