fix(cargo-cache): close the seed-vs-republish race the design claimed to close

The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L

scripts/cache-lib.sh

@@ -113,22 +113,31 @@ strip_cargo_locks() {
 # only ever run this against a staging directory nothing else can see yet
 # (see hardlink_clone_into's contract), so the brief window where the path is
 # absent is not observable.
+#
+# Returns non-zero if the copy or either rename failed. That status is
+# load-bearing: a failed unshare leaves the staging tree still aliasing its
+# source, which is the exact corruption `unshare_mutable_paths` exists to
+# prevent, so it must abort the clone rather than be swallowed.
 unshare_subtree() {
   local d="$1" tmp
   [ -d "$d" ] || return 0
   tmp="${d}.unshare.$$"
   rm -rf "$tmp"
-  cp -a "$d" "$tmp"
-  rm -rf "$d"
-  mv -T "$tmp" "$d"
+  cp -a "$d" "$tmp" || { rm -rf "$tmp"; return 1; }
+  rm -rf "$d" || { rm -rf "$tmp"; return 1; }
+  mv -T "$tmp" "$d" || return 1
+  return 0
 }
 
 _unshare_files() {
   # `-links +1` restricts the work to files that are actually shared, which
   # makes this idempotent and near-free on an already-unshared tree.
+  #
+  # The inner shell propagates a failure of any individual copy-and-rename out
+  # through xargs (which exits 123 if any invocation exits 1-125), so a
+  # partially-unshared tree is reported rather than silently accepted.
   find "$@" -links +1 -print0 2>/dev/null |
-    xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _
-  return 0
+    xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
 }
 
 # THE load-bearing function of this whole design.
@@ -184,33 +193,235 @@ unshare_mutable_paths() {
   mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
   for d in "${dirs[@]}"; do
     [ -n "$d" ] || continue
-    unshare_subtree "$d"
+    unshare_subtree "$d" || {
+      echo "::error::unshare_mutable_paths: failed to unshare ${d}" >&2
+      return 1
+    }
   done
-  _unshare_files "$root" -type f -name '*.d'
-  _unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json'
+  _unshare_files "$root" -type f -name '*.d' || {
+    echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
+    return 1
+  }
+  _unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
+    echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
+    return 1
+  }
+  return 0
+}
+
+# ---------------------------------------------------------------------------
+# Reader markers: the consume side's half of the seed-vs-republish interlock
+# ---------------------------------------------------------------------------
+#
+# A hardlink clone reads its source over many seconds. The publish side
+# rotates a snapshot with two renames and then unlinks the generation it
+# rotated away — and unlinking entries out from under an in-flight directory
+# walk is what produces a SILENTLY truncated clone: `cp -al` reports the
+# entries it manages to stat, and simply never sees a subdirectory that was
+# unlinked before it read the parent's listing. Exit status alone does not
+# catch that case.
+#
+# So the two sides interlock through a marker file, and the ordering is what
+# makes it sound rather than probabilistic:
+#
+#   Consumer:  create .reading-<snap>-<tag>  ->  stat <snap>  ->  cp -al
+#   Publisher: mv <snap> aside  ->  mv new into place  ->  scan for markers
+#              ->  unlink the rotated-away generation
+#
+# If a consumer's `stat` resolved to the OLD generation, that stat happened
+# before the publisher's first rename, so its marker — created strictly
+# earlier still — was already on disk before the publisher's scan, which
+# happens strictly after that rename. The publisher therefore cannot miss it.
+# A consumer that creates its marker after the scan necessarily resolves the
+# path to the NEW generation, which is not the one being unlinked.
+#
+# The wait is bounded (CACHE_READ_GRACE_SECONDS). Exceeding it does not force
+# the unlink: reclamation of that generation is DEFERRED to a later publish
+# instead. The residual is therefore disk, not correctness.
+CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}"
+# A marker older than this belongs to a job the runner killed before it could
+# clean up. Honouring one forever would let a crashed job pin an entire
+# snapshot generation on disk permanently.
+CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}"
+
+reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; }
+
+reader_lock_acquire() {
+  date +%s > "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
+  return 0
+}
+
+reader_lock_release() {
+  rm -f "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
+  return 0
+}
+
+# Prints the number of live readers of <source-name> under <marker-root>, and
+# sweeps markers past the staleness threshold as it goes.
+live_reader_count() {
+  local root="$1" name="$2" now marker age n=0
+  now=$(date +%s)
+  for marker in "$root"/.reading-"$name"-*; do
+    [ -e "$marker" ] || continue
+    age=$(( now - $(stat -c '%Y' "$marker" 2>/dev/null || echo "$now") ))
+    if [ "$age" -lt "$CACHE_READ_STALE_SECONDS" ]; then
+      n=$((n + 1))
+    else
+      echo "readers: sweeping stale marker $(basename "$marker") (${age}s old > ${CACHE_READ_STALE_SECONDS}s)" >&2
+      rm -f "$marker" 2>/dev/null || true
+    fi
+  done
+  printf '%s' "$n"
+  return 0
+}
+
+# Blocks until nothing is reading <source-name>, or until the grace period
+# expires. Returns 0 when drained, 1 on timeout — the caller decides what to
+# do with a timeout, and in this codebase that decision is always "defer the
+# unlink", never "unlink anyway".
+wait_for_readers() {
+  local root="$1" name="$2" grace="${3:-$CACHE_READ_GRACE_SECONDS}" deadline n waited=0
+  deadline=$(( $(date +%s) + grace ))
+  while :; do
+    n=$(live_reader_count "$root" "$name")
+    [ "$n" -eq 0 ] && {
+      [ "$waited" -gt 0 ] && echo "readers: ${name} drained after ${waited}s"
+      return 0
+    }
+    if [ "$(date +%s)" -ge "$deadline" ]; then
+      echo "readers: ${n} job(s) still reading ${name} after ${grace}s" >&2
+      return 1
+    fi
+    [ "$waited" = 0 ] && echo "readers: waiting for ${n} in-flight clone(s) of ${name} (grace ${grace}s)"
+    sleep 1
+    waited=$((waited + 1))
+  done
+}
+
+# ---------------------------------------------------------------------------
+# The clone itself
+# ---------------------------------------------------------------------------
+
+# Number of times a torn clone is retried before the caller is failed. A tear
+# means the source changed identity or lost entries mid-walk, which is a
+# transient condition by definition — the publisher that caused it has already
+# put a complete new generation at the same path — so one retry almost always
+# suffices; the rest are headroom.
+CACHE_CLONE_ATTEMPTS="${CACHE_CLONE_ATTEMPTS:-4}"
+
+_tree_entries() {
+  local n
+  n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
+  printf '%s' "$n"
+  return 0
+}
+
+_dir_inode() {
+  stat -c '%i' "$1" 2>/dev/null || printf 'missing'
+  return 0
+}
+
+write_cache_lock() {
+  local dir="$1" id="$2"
+  [ -n "$id" ] || return 0
+  [ -d "$dir" ] || return 0
+  date +%s > "${dir}/.ci-lock-${id}" 2>/dev/null || true
   return 0
 }
 
 # Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
 # with a single atomic rename.
 #
+#   hardlink_clone_into <src> <dst> <tag> [lock-id]
+#
 # The staging + rename is what closes the concurrent-seed race structurally
 # rather than by runner topology: a second job sharing this cache key either
 # sees DST absent (and stages its own clone, losing the rename harmlessly) or
 # sees it complete. There is no observable half-populated state, because a
 # directory rename is atomic and DST is never written through.
 #
+# The rename is necessary but NOT sufficient, and that gap is what this
+# function's retry loop closes. An atomic rename of a TRUNCATED tree publishes
+# a truncated tree atomically. Three things can truncate one:
+#
+#   * `cp -al` failing partway (a source entry vanished after readdir listed
+#     it) — caught by checking its exit status, which is why that status is
+#     read into a variable here rather than left to an ambient `set -e` the
+#     CALL SITES suppress anyway by invoking this function as a condition;
+#   * `cp -al` succeeding while having silently never seen a subtree that was
+#     unlinked before it read the parent's listing — caught only by the entry
+#     count, since there is no error to report;
+#   * the source being replaced wholesale mid-walk, so the clone splices two
+#     generations — caught by comparing the source directory's inode before
+#     and after.
+#
+# All three are verified on every attempt and a failing one restarts the
+# clone; a tree that fails the last attempt is deleted and reported, never
+# renamed into place. Combined with the reader marker (held across the copy,
+# which is what stops the publish side unlinking underneath it in the first
+# place), a partial tree cannot reach DST.
+#
+# `lock-id`, when given, writes this job's cache lock INTO the staging tree so
+# the directory already carries it the instant it appears under its final
+# name. Acquiring the lock after the rename would leave a freshly seeded
+# directory momentarily unlocked and therefore evictable by a concurrent job's
+# prune pass.
+#
 # Returns 0 if this caller's clone won the rename, 1 if another caller got
-# there first (the staging copy is discarded; DST is already valid).
+# there first (the staging copy is discarded; DST is already valid), and 2 if
+# the source could not be cloned consistently at all.
 hardlink_clone_into() {
-  local src="$1" dst="$2" tag="$3" tmp parent
+  local src="$1" dst="$2" tag="$3" lock_id="${4:-}"
+  local parent tmp src_name attempt cp_rc n_before n_after i_before i_after
+
+  if [ ! -d "$src" ]; then
+    echo "::error::clone: source ${src} does not exist" >&2
+    return 2
+  fi
+
   parent=$(dirname "$dst")
+  src_name=$(basename "$src")
   tmp="${parent}/.stage-${tag}"
-  rm -rf "$tmp"
-  cp -al "$src" "$tmp"
+
+  attempt=1
+  while : ; do
+    rm -rf "$tmp"
+    # Marker first, then the identity read, then the copy — see the ordering
+    # proof in the reader-marker section above; swapping the first two lines
+    # is what would reintroduce the race.
+    reader_lock_acquire "$parent" "$src_name" "$tag"
+    i_before=$(_dir_inode "$src")
+    n_before=$(_tree_entries "$src")
+    cp_rc=0
+    cp -al "$src" "$tmp" || cp_rc=$?
+    n_after=$(_tree_entries "$tmp")
+    i_after=$(_dir_inode "$src")
+    reader_lock_release "$parent" "$src_name" "$tag"
+
+    if [ "$cp_rc" -eq 0 ] && [ "$i_before" != missing ] && [ "$i_before" = "$i_after" ] \
+       && [ "$n_after" -eq "$n_before" ]; then
+      break
+    fi
+
+    echo "::warning::clone: attempt ${attempt}/${CACHE_CLONE_ATTEMPTS} of ${src_name} was torn (cp rc=${cp_rc}, ${n_after}/${n_before} entries, source inode ${i_before} -> ${i_after}) — discarding and retrying" >&2
+    rm -rf "$tmp"
+    if [ "$attempt" -ge "$CACHE_CLONE_ATTEMPTS" ]; then
+      echo "::error::clone: ${src} could not be read consistently in ${CACHE_CLONE_ATTEMPTS} attempts — refusing to publish a partial tree at ${dst}" >&2
+      return 2
+    fi
+    attempt=$((attempt + 1))
+    sleep 1
+  done
+
   strip_cargo_locks "$tmp"
   rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
-  unshare_mutable_paths "$tmp"
+  if ! unshare_mutable_paths "$tmp"; then
+    echo "::error::clone: could not privately own the mutable paths of ${dst} — discarding the staging tree rather than publishing one that aliases ${src}" >&2
+    rm -rf "$tmp"
+    return 2
+  fi
+  write_cache_lock "$tmp" "$lock_id"
+
   if mv -T "$tmp" "$dst" 2>/dev/null; then
     return 0
   fi

scripts/prune-cache.sh

@@ -49,12 +49,13 @@
 #   the entire benefit this scheme exists to deliver.
 #
 #   LOCKED — a directory carrying a .ci-lock-* marker younger than
-#   STALE_LOCK_SECONDS is held open by a running job and is skipped by every
-#   pass, however dead and however tight the disk. This is what makes eviction
-#   safe on a runner with more than one execution slot. An older marker is
-#   treated as abandoned and logged as such, so an actually-still-running job
-#   that somehow exceeds the threshold is visible in the log rather than
-#   silently losing its cache mid-build.
+#   STALE_LOCK_SECONDS is held open by a running job, or named by a live
+#   .reading-<dir>-* marker (a job is hardlink-cloning it this instant), is
+#   skipped by every pass, however dead and however tight the disk. This is
+#   what makes eviction safe on a runner with more than one execution slot.
+#   An older marker is treated as abandoned and logged as such, so an
+#   actually-still-running job that somehow exceeds the threshold is visible
+#   in the log rather than silently losing its cache mid-build.
 #
 # Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
 # timeout. A directory name cannot be inverted back to a branch name (the
@@ -93,8 +94,20 @@ is_protected() {
 }
 
 is_locked() {
-  local dir="$1" now lock_file lock_age locked=1
+  local dir="$1" now lock_file lock_age locked=1 readers
   now=$(date +%s)
+  # A directory being hardlink-cloned right now carries no .ci-lock-* of its
+  # own — a snapshot has its locks stripped by construction — so the reader
+  # markers are the only signal that unlinking it would truncate somebody's
+  # in-flight clone. Today no reachable configuration prunes a snapshot (only
+  # protected refs publish them, and protected refs are excluded from every
+  # pass), which makes this guard redundant *by policy*. It is here so that
+  # stops being the reason it is safe.
+  readers=$(live_reader_count "$ROOT" "$(basename "$dir")")
+  if [ "$readers" -gt 0 ]; then
+    echo "  $(basename "$dir"): ${readers} job(s) currently cloning it — not a candidate"
+    locked=0
+  fi
   for lock_file in "$dir"/.ci-lock-*; do
     [ -e "$lock_file" ] || continue
     lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))

scripts/publish-snapshot.sh

@@ -23,11 +23,23 @@
 # own cold-start path — a safe degrade that self-heals on its next run, not
 # corruption.
 #
-# `rm -rf` on the old snapshot removes directory entries only. Any consumer
-# that already hardlink-cloned from it keeps every inode alive through its own
-# links, so a republish never pulls data out from under a running job — it
-# just stops new consumers from seeing the old generation. Disk is reclaimed
-# when the last clone referencing those inodes is itself evicted.
+# `rm -rf` on the old snapshot removes directory entries only, so a consumer
+# that has ALREADY FINISHED cloning from it keeps every inode alive through
+# its own links. That is the easy half, and on its own it is not enough: a
+# consumer still WALKING the old generation has its entries unlinked out from
+# under it, and `cp -al` does not report a subtree that was removed before it
+# read the parent's listing. That is a silently truncated clone — the failure
+# mode this script's own selftest (scenario 8) reproduces against the
+# unguarded version.
+#
+# So the unlink is interlocked with the consume side rather than
+# unconditional: after the swap, this script waits for every in-flight reader
+# of this snapshot to drain (see the reader-marker ordering proof in
+# cache-lib.sh) and only then reclaims the rotated-away generation. If the
+# grace period expires first, reclamation is DEFERRED — the directory is left
+# under `.publish-old-<key>-<tag>` and swept by a later publish once no reader
+# holds it. The residual of a very slow consumer is therefore one extra
+# generation of directory entries on disk, never a torn clone.
 set -euo pipefail
 . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
 
@@ -36,7 +48,11 @@ ROOT="${2:?}"; TAG="${3:?}"
 
 SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
 DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
-OLD="${ROOT}/.publish-old-${TAG}"
+# Keyed by cache key as well as tag, so a deferred generation can be matched
+# back to the snapshot whose readers must drain before it is safe to reclaim.
+OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}"
+SNAP_NAME=$(basename "$DST")
+GRACE="${CACHE_READ_GRACE_SECONDS}"
 
 if [ ! -d "$SRC" ]; then
   echo "publish: no target dir at ${SRC} — nothing to snapshot"
@@ -47,6 +63,21 @@ fi
 # concurrently running job's staging directory is never touched.
 rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
 
+# Deferred reclamation from an earlier publish of THIS snapshot whose readers
+# had not drained in time. Safe to sweep now only if nothing is reading the
+# snapshot at all: a reader holds the snapshot path, not the deferred name, so
+# "no readers of snapshot-<key>" is the condition that makes every deferred
+# generation of it unreachable. Over-conservative by design — a reader of the
+# CURRENT generation also defers the sweep to the next publish, which costs a
+# directory listing, not correctness.
+if [ "$(live_reader_count "$ROOT" "$SNAP_NAME")" -eq 0 ]; then
+  for stale_old in "${ROOT}/.publish-old-${OWN_KEY}-"*; do
+    [ -d "$stale_old" ] || continue
+    echo "publish: reclaiming deferred snapshot generation $(basename "$stale_old")"
+    rm -rf "$stale_old"
+  done
+fi
+
 start=$(date +%s)
 # The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
 # of every file Cargo rewrites in place (unshare_mutable_paths, called inside
@@ -63,7 +94,19 @@ hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {
 
 if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
 mv -T "$TMP_DST" "$DST"
-rm -rf "$OLD"
+
+# The scan below happens strictly after the rename above, which is what makes
+# it impossible for a consumer holding the OLD generation to be missed: such a
+# consumer resolved the path before that rename, and published its marker
+# before that. See cache-lib.sh's reader-marker section.
+if [ -d "$OLD" ]; then
+  if wait_for_readers "$ROOT" "$SNAP_NAME" "$GRACE"; then
+    rm -rf "$OLD"
+  else
+    echo "::warning::publish: a consumer is still cloning the previous ${SNAP_NAME} after ${GRACE}s — deferring reclamation of $(basename "$OLD") rather than unlinking a tree being read"
+    summary_line "- deferred reclaiming the previous \`${SNAP_NAME}\` generation (a consumer is still cloning it); it will be swept by a later publish"
+  fi
+fi
 
 echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
 summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"

scripts/restore-mtimes.sh

@@ -170,7 +170,8 @@
 # the next run's diff base stays at the last GREEN build — over-inclusive
 # (it may re-stamp files that a failed run partially rebuilt anyway) but
 # never under-inclusive, which is the only direction that matters for
-# soundness.set -euo pipefail
+# soundness.
+set -euo pipefail
 
 repo_root=$(git rev-parse --show-toplevel)
 cd "$repo_root"

scripts/seed-target-dir-selftest.sh

@@ -29,6 +29,21 @@
 #      at no point is a partially-populated directory visible under the final
 #      name. This is the property that replaces "the runner only has one job
 #      slot" with an actual guarantee.
+#   8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the
+#      one that actually mattered: a consumer hardlink-cloning a snapshot
+#      while the publisher of that snapshot rotates it and unlinks the
+#      generation being read. Two seeds racing on a DESTINATION is a different
+#      race from a seed racing a publisher on its SOURCE, and only the second
+#      one can truncate a tree. Against the unguarded version this scenario
+#      reproduces a silent partial clone reported as success — 20,328 of
+#      48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
+#   9. DEFERRED RECLAMATION — when a consumer is STILL reading after the grace
+#      period, the publisher leaves the rotated-away generation on disk rather
+#      than unlinking a tree under an in-flight walk, and a later publish
+#      sweeps it once the reader is gone. The residual is disk, not a torn
+#      clone.
+#  10. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct
+#      status instead of renaming whatever it managed to produce into place.
 set -euo pipefail
 script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$script_dir/cache-lib.sh"
@@ -56,7 +71,32 @@ make_tree() {
   : > "$d/debug/.cargo-lock"
 }
 
-seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
+seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
+
+# Always succeeds and always prints a number: an absent directory is "0
+# entries so far", which is the normal state at the top of the progress poll
+# below, not an error worth aborting the suite over.
+tree_entries() {
+  local n
+  n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
+  printf '%s\n' "$n"
+  return 0
+}
+
+# A tree wide enough that a hardlink clone of it takes long enough to be
+# caught mid-walk. The race under test is a real interleaving, not a mocked
+# one, so the fixture has to be big enough for the window to exist: a
+# four-file tree clones in microseconds and no scheduling could ever land
+# inside it. Built by cloning one small template directory N times, which is N
+# forks rather than N*M file creations.
+make_wide_tree() {
+  local d="$1" marker="$2" ndirs="$3" i
+  mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x"
+  for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done
+  for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done
+  rm -rf "$d/debug/deps/.tmpl"
+  echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
+}
 
 BASE_KEY=$(cache_key dev)
 OWN_KEY=$(cache_key feat/thing)
@@ -127,5 +167,96 @@ if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
 fi
 ok "neither racing job fell through to a cold start"
 
+echo
+echo "=== 8: seeding while the base republishes the snapshot underneath it ==="
+ROT=$(cache_key feat/rotate)
+rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY"
+# Generation 1 is wide (the consumer will still be walking it when the swap
+# happens); the generation replacing it is small, so the publisher's own
+# staging clone does not itself outlast the consumer's.
+make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800
+make_wide_tree "$root/target-$BASE_KEY" gen2 8
+gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
+
+( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) &
+seed_pid=$!
+
+# Rotate only once the clone is demonstrably mid-walk. Gating on observed
+# progress rather than on a sleep is what makes the interleaving reproducible
+# instead of a coin flip that passes on a fast machine for the wrong reason.
+threshold=$(( gen1_entries / 5 ))
+progress=0
+deadline=$(( $(date +%s) + 60 ))
+while :; do
+  progress=$(tree_entries "$root/.stage-jobRot")
+  if [ "$progress" -ge "$threshold" ]; then break; fi
+  if ! kill -0 "$seed_pid" 2>/dev/null; then
+    fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)"
+  fi
+  if [ "$(date +%s)" -ge "$deadline" ]; then
+    fail "the staging clone never reached ${threshold} of ${gen1_entries} entries"
+  fi
+done
+ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries"
+
+bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \
+  || { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; }
+wait "$seed_pid"
+
+rot_dir="$root/target-$ROT"
+[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; }
+ok "the seed completed"
+# THE assertion. Before the guard, this is where it failed: the seed reported
+# `cloned in 1s` and exit 0 while target-<rot> held less than half the entries
+# of the snapshot it claimed to have cloned. Comparing against the snapshot as
+# it stands NOW is the right bar either way — a clone that raced the rotation
+# must end up holding one complete generation, and a consumer caught mid-walk
+# re-reads, so that generation is the new one.
+snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
+rot_entries=$(tree_entries "$rot_dir")
+[ "$rot_entries" -eq "$snap_entries" ] \
+  || fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)"
+ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)"
+assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two"
+grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read"
+ok "the torn read was detected and reported, not swallowed"
+[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \
+  || fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)"
+ok "no staging, reader-marker or deferred-generation scratch left behind"
+
+echo
+echo "=== 9: a reader that outlasts the grace period defers reclamation ==="
+# A synthetic reader marker stands in for a consumer whose clone is slower
+# than the grace period. Driving that with a real slow consumer would make the
+# test's runtime the thing under test; the marker is the whole contract
+# between the two sides, so holding one IS being a reader.
+SNAP_NAME="snapshot-$BASE_KEY"
+date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
+make_wide_tree "$root/target-$BASE_KEY" gen3 4
+CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubDefer > "$scratch/logDefer" 2>&1 \
+  || { tail -40 "$scratch/logDefer"; fail "publish-snapshot.sh exited non-zero"; }
+assert_content "$root/$SNAP_NAME/debug/.fingerprint/x/dep-lib-x" gen3 "the new generation was published regardless"
+deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${BASE_KEY}-*" -print -quit)
+[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
+ok "the rotated-away generation was left on disk rather than unlinked under a reader"
+grep -q 'deferring reclamation' "$scratch/logDefer" || fail "the deferral was not reported"
+ok "the deferral is reported as a warning, not silent"
+
+rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
+make_wide_tree "$root/target-$BASE_KEY" gen4 4
+bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubSweep > "$scratch/logSweep" 2>&1 \
+  || { tail -40 "$scratch/logSweep"; fail "publish-snapshot.sh exited non-zero"; }
+[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
+  || fail "the deferred generation was never reclaimed"
+ok "a later publish reclaims the deferred generation once no reader holds it"
+
+echo
+echo "=== 10: a source that cannot be read fails loudly ==="
+rc=0
+hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$?
+[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}"
+ok "an unreadable source returns the distinct hard-failure status"
+assert_absent "$root/target-nosuch" "nothing was renamed into place"
+
 echo
 echo "seed-target-dir-selftest: ${pass_count} assertions passed"

scripts/seed-target-dir.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # Consume side: make this run's own target dir exist, warm, and private.
 #
-# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir]
+# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir] [lock-id]
 #   own-key       cache key for this run's own ref
 #   base-key      cache key for the ref to layer over ("" for a run whose own
 #                 ref IS a reference branch)
@@ -10,6 +10,14 @@
 #                 directory so two concurrent jobs can never collide on it
 #   fallback-dir  optional absolute path to seed from when no snapshot exists
 #                 (a legacy flat cache dir during a migration, typically)
+#   lock-id       optional cache-lock id. When given, the lock marker is
+#                 written on EVERY path out of this script — including into
+#                 the staging tree before its rename — so the directory is
+#                 never observable under its final name without a lock. The
+#                 action acquires the lock in a later step too; that step is
+#                 idempotent, and this closes the window before it runs, where
+#                 a concurrent job's prune pass could evict a directory that
+#                 exists but is not yet held.
 #
 # The design in one paragraph: a PR branch's first run hardlink-clones the
 # base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
@@ -27,17 +35,18 @@
 set -euo pipefail
 . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
 
-if [ $# -lt 4 ] || [ $# -gt 5 ]; then
-  echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2
+if [ $# -lt 4 ] || [ $# -gt 6 ]; then
+  echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2
   exit 1
 fi
 
-OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"
+OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}"
 OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
 
 mkdir -p "$ROOT"
 
 if [ -d "$OWN_DIR" ]; then
+  write_cache_lock "$OWN_DIR" "$LOCK_ID"
   echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
   echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
   exit 0
@@ -62,20 +71,40 @@ for entry in "${CANDIDATES[@]}"; do
   [ -d "$src" ] || continue
   echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
   start=$(date +%s)
-  if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then
-    echo "seed: cloned in $(( $(date +%s) - start ))s"
-    echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
-  else
-    # Another job sharing this cache key won the rename while we were
-    # cloning. Its directory is complete (the rename is the publish step), so
-    # there is nothing to do but use it — and nothing was ever observable in
-    # a half-seeded state.
-    echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
-    echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
-  fi
+  # The status is captured and dispatched on explicitly. Calling this as a
+  # bare `if` condition — which is what this script used to do — suppresses
+  # `set -e` for the whole call, so a hard clone failure could not abort the
+  # seed even in principle; the three outcomes are genuinely distinct and each
+  # needs its own handling.
+  clone_rc=0
+  hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$?
+  case "$clone_rc" in
+    0)
+      echo "seed: cloned in $(( $(date +%s) - start ))s"
+      echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
+      ;;
+    1)
+      # Another job sharing this cache key won the rename while we were
+      # cloning. Its directory is complete (the rename is the publish step),
+      # so there is nothing to do but use it — and nothing was ever
+      # observable in a half-seeded state.
+      write_cache_lock "$OWN_DIR" "$LOCK_ID"
+      echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
+      echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
+      ;;
+    *)
+      # A source that could not be read consistently. Failing the job is the
+      # only safe answer: the alternative that used to happen here was
+      # seeding a truncated tree and reporting success, which hands Cargo a
+      # directory whose fingerprints and artifacts disagree.
+      echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2
+      exit 1
+      ;;
+  esac
   exit 0
 done
 
 echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
 echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
 mkdir -p "$OWN_DIR"
+write_cache_lock "$OWN_DIR" "$LOCK_ID"