fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
Claude Opus 5
@@ -29,6 +29,21 @@
|
||||
# at no point is a partially-populated directory visible under the final
|
||||
# name. This is the property that replaces "the runner only has one job
|
||||
# slot" with an actual guarantee.
|
||||
# 8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the
|
||||
# one that actually mattered: a consumer hardlink-cloning a snapshot
|
||||
# while the publisher of that snapshot rotates it and unlinks the
|
||||
# generation being read. Two seeds racing on a DESTINATION is a different
|
||||
# race from a seed racing a publisher on its SOURCE, and only the second
|
||||
# one can truncate a tree. Against the unguarded version this scenario
|
||||
# reproduces a silent partial clone reported as success — 20,328 of
|
||||
# 48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
|
||||
# 9. DEFERRED RECLAMATION — when a consumer is STILL reading after the grace
|
||||
# period, the publisher leaves the rotated-away generation on disk rather
|
||||
# than unlinking a tree under an in-flight walk, and a later publish
|
||||
# sweeps it once the reader is gone. The residual is disk, not a torn
|
||||
# clone.
|
||||
# 10. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct
|
||||
# status instead of renaming whatever it managed to produce into place.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
@@ -56,7 +71,32 @@ make_tree() {
|
||||
: > "$d/debug/.cargo-lock"
|
||||
}
|
||||
|
||||
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
||||
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
||||
|
||||
# Always succeeds and always prints a number: an absent directory is "0
|
||||
# entries so far", which is the normal state at the top of the progress poll
|
||||
# below, not an error worth aborting the suite over.
|
||||
tree_entries() {
|
||||
local n
|
||||
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
|
||||
printf '%s\n' "$n"
|
||||
return 0
|
||||
}
|
||||
|
||||
# A tree wide enough that a hardlink clone of it takes long enough to be
|
||||
# caught mid-walk. The race under test is a real interleaving, not a mocked
|
||||
# one, so the fixture has to be big enough for the window to exist: a
|
||||
# four-file tree clones in microseconds and no scheduling could ever land
|
||||
# inside it. Built by cloning one small template directory N times, which is N
|
||||
# forks rather than N*M file creations.
|
||||
make_wide_tree() {
|
||||
local d="$1" marker="$2" ndirs="$3" i
|
||||
mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x"
|
||||
for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done
|
||||
for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done
|
||||
rm -rf "$d/debug/deps/.tmpl"
|
||||
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
||||
}
|
||||
|
||||
BASE_KEY=$(cache_key dev)
|
||||
OWN_KEY=$(cache_key feat/thing)
|
||||
@@ -127,5 +167,96 @@ if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
|
||||
fi
|
||||
ok "neither racing job fell through to a cold start"
|
||||
|
||||
echo
|
||||
echo "=== 8: seeding while the base republishes the snapshot underneath it ==="
|
||||
ROT=$(cache_key feat/rotate)
|
||||
rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY"
|
||||
# Generation 1 is wide (the consumer will still be walking it when the swap
|
||||
# happens); the generation replacing it is small, so the publisher's own
|
||||
# staging clone does not itself outlast the consumer's.
|
||||
make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800
|
||||
make_wide_tree "$root/target-$BASE_KEY" gen2 8
|
||||
gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
||||
|
||||
( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) &
|
||||
seed_pid=$!
|
||||
|
||||
# Rotate only once the clone is demonstrably mid-walk. Gating on observed
|
||||
# progress rather than on a sleep is what makes the interleaving reproducible
|
||||
# instead of a coin flip that passes on a fast machine for the wrong reason.
|
||||
threshold=$(( gen1_entries / 5 ))
|
||||
progress=0
|
||||
deadline=$(( $(date +%s) + 60 ))
|
||||
while :; do
|
||||
progress=$(tree_entries "$root/.stage-jobRot")
|
||||
if [ "$progress" -ge "$threshold" ]; then break; fi
|
||||
if ! kill -0 "$seed_pid" 2>/dev/null; then
|
||||
fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)"
|
||||
fi
|
||||
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||
fail "the staging clone never reached ${threshold} of ${gen1_entries} entries"
|
||||
fi
|
||||
done
|
||||
ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries"
|
||||
|
||||
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \
|
||||
|| { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; }
|
||||
wait "$seed_pid"
|
||||
|
||||
rot_dir="$root/target-$ROT"
|
||||
[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; }
|
||||
ok "the seed completed"
|
||||
# THE assertion. Before the guard, this is where it failed: the seed reported
|
||||
# `cloned in 1s` and exit 0 while target-<rot> held less than half the entries
|
||||
# of the snapshot it claimed to have cloned. Comparing against the snapshot as
|
||||
# it stands NOW is the right bar either way — a clone that raced the rotation
|
||||
# must end up holding one complete generation, and a consumer caught mid-walk
|
||||
# re-reads, so that generation is the new one.
|
||||
snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
||||
rot_entries=$(tree_entries "$rot_dir")
|
||||
[ "$rot_entries" -eq "$snap_entries" ] \
|
||||
|| fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)"
|
||||
ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)"
|
||||
assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two"
|
||||
grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read"
|
||||
ok "the torn read was detected and reported, not swallowed"
|
||||
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \
|
||||
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)"
|
||||
ok "no staging, reader-marker or deferred-generation scratch left behind"
|
||||
|
||||
echo
|
||||
echo "=== 9: a reader that outlasts the grace period defers reclamation ==="
|
||||
# A synthetic reader marker stands in for a consumer whose clone is slower
|
||||
# than the grace period. Driving that with a real slow consumer would make the
|
||||
# test's runtime the thing under test; the marker is the whole contract
|
||||
# between the two sides, so holding one IS being a reader.
|
||||
SNAP_NAME="snapshot-$BASE_KEY"
|
||||
date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
|
||||
make_wide_tree "$root/target-$BASE_KEY" gen3 4
|
||||
CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubDefer > "$scratch/logDefer" 2>&1 \
|
||||
|| { tail -40 "$scratch/logDefer"; fail "publish-snapshot.sh exited non-zero"; }
|
||||
assert_content "$root/$SNAP_NAME/debug/.fingerprint/x/dep-lib-x" gen3 "the new generation was published regardless"
|
||||
deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${BASE_KEY}-*" -print -quit)
|
||||
[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
|
||||
ok "the rotated-away generation was left on disk rather than unlinked under a reader"
|
||||
grep -q 'deferring reclamation' "$scratch/logDefer" || fail "the deferral was not reported"
|
||||
ok "the deferral is reported as a warning, not silent"
|
||||
|
||||
rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
|
||||
make_wide_tree "$root/target-$BASE_KEY" gen4 4
|
||||
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubSweep > "$scratch/logSweep" 2>&1 \
|
||||
|| { tail -40 "$scratch/logSweep"; fail "publish-snapshot.sh exited non-zero"; }
|
||||
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
||||
|| fail "the deferred generation was never reclaimed"
|
||||
ok "a later publish reclaims the deferred generation once no reader holds it"
|
||||
|
||||
echo
|
||||
echo "=== 10: a source that cannot be read fails loudly ==="
|
||||
rc=0
|
||||
hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$?
|
||||
[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}"
|
||||
ok "an unreadable source returns the distinct hard-failure status"
|
||||
assert_absent "$root/target-nosuch" "nothing was renamed into place"
|
||||
|
||||
echo
|
||||
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
||||
|
||||
Reference in New Issue
Block a user