daniel/gitdan-actions
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cargo-cache): close the seed-vs-republish race the design claimed to close

Browse Source 
The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
claude
2026-08-23 16:40:21 -05:00
co-authored by Claude Opus 5
parent 248af3061e
commit f57e2a6013
9 changed files with 562 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/seed-target-dir.sh
+44 -15
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
# Consume side: make this run's own target dir exist, warm, and private.
#
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir]
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir] [lock-id]
# own-key cache key for this run's own ref
# base-key cache key for the ref to layer over ("" for a run whose own
# ref IS a reference branch)
@@ -10,6 +10,14 @@
# directory so two concurrent jobs can never collide on it
# fallback-dir optional absolute path to seed from when no snapshot exists
# (a legacy flat cache dir during a migration, typically)
# lock-id optional cache-lock id. When given, the lock marker is
# written on EVERY path out of this script — including into
# the staging tree before its rename — so the directory is
# never observable under its final name without a lock. The
# action acquires the lock in a later step too; that step is
# idempotent, and this closes the window before it runs, where
# a concurrent job's prune pass could evict a directory that
# exists but is not yet held.
#
# The design in one paragraph: a PR branch's first run hardlink-clones the
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
@@ -27,17 +35,18 @@
set -euo pipefail
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
if [ $# -lt 4 ] || [ $# -gt 5 ]; then
echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2
if [ $# -lt 4 ] || [ $# -gt 6 ]; then
echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2
exit 1
fi
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}"
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
mkdir -p "$ROOT"
if [ -d "$OWN_DIR" ]; then
write_cache_lock "$OWN_DIR" "$LOCK_ID"
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
exit 0
@@ -62,20 +71,40 @@ for entry in "${CANDIDATES[@]}"; do
[ -d "$src" ] || continue
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
start=$(date +%s)
if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then
echo "seed: cloned in $(( $(date +%s) - start ))s"
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
else
# Another job sharing this cache key won the rename while we were
# cloning. Its directory is complete (the rename is the publish step), so
# there is nothing to do but use it — and nothing was ever observable in
# a half-seeded state.
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
fi
# The status is captured and dispatched on explicitly. Calling this as a
# bare `if` condition — which is what this script used to do — suppresses
# `set -e` for the whole call, so a hard clone failure could not abort the
# seed even in principle; the three outcomes are genuinely distinct and each
# needs its own handling.
clone_rc=0
hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$?
case "$clone_rc" in
0)
echo "seed: cloned in $(( $(date +%s) - start ))s"
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
;;
1)
# Another job sharing this cache key won the rename while we were
# cloning. Its directory is complete (the rename is the publish step),
# so there is nothing to do but use it — and nothing was ever
# observable in a half-seeded state.
write_cache_lock "$OWN_DIR" "$LOCK_ID"
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
;;
*)
# A source that could not be read consistently. Failing the job is the
# only safe answer: the alternative that used to happen here was
# seeding a truncated tree and reporting success, which hands Cargo a
# directory whose fingerprints and artifacts disagree.
echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2
exit 1
;;
esac
exit 0
done
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
mkdir -p "$OWN_DIR"
write_cache_lock "$OWN_DIR" "$LOCK_ID"