claudeandClaude Opus 5
719475831b
Follow-up tof57e2a6, addressing the reviewer's sharpest question: is the race "closed by construction", or merely detected and retried? The honest answer is "both, on different paths", and the README said only the first half. * README now states three claims separately instead of collapsing them: a publisher rotating a snapshot cannot tear a clone of it (by construction — the marker ordering prevents the unlink, and the consumer's verification is a redundant second check on that path); every OTHER way the source can change mid-clone is detected, not prevented (the eviction pass's reader check is check-then-delete, and a seed-fallback-dir has no interlock at all — there, verification plus a bounded retry and a loud failure is the whole guard); and disk reclamation is bounded rather than immediate. Overclaiming this property once was the finding; overclaiming it twice would be worse. * publish-snapshot-selftest.sh now covers the PUBLISHER's half of the race, where a reader of the swap belongs. Scenario 3 only ever covered a consumer that had already FINISHED cloning — safe for free, since its own hardlinks keep the inodes alive. New scenario 6 covers a reader still in flight past the grace period (the generation is left on disk, the deferral is warned about, and an earlier consumer is still unaffected); scenario 7 covers the sweep, so "we defer instead of forcing" cannot quietly become a disk leak. Red-proven against 248af306's scripts: ASSERTION FAILED: the previous generation was unlinked while a reader still held it The consumer's half stays in seed-target-dir-selftest.sh scenario 8, which still red-proves at 16693 of 48805 entries against the same scripts. * seed-target-dir-selftest.sh now asserts what happens when the retries are EXHAUSTED, not just what hardlink_clone_into returns: an unreadable source makes the seed script exit non-zero, name the reason, leave no target dir, and — the one that matters — not fall through to its cold-start branch. A corrupt-cache bug degrading into an invisible 4x-slower CI job is the failure mode worth pinning down. Skipped when running as root, where mode bits deny nothing. * usage_kb: a directory we cannot read measured as the empty string, which was then spliced into usage_gb's awk program and made it a syntax error at the exact moment something was already going wrong. Now measures 0. Verification: `bash scripts/selftest.sh` — 5 suites, exit 0, 82 assertions (was 75 afterf57e2a6, 63 before). shellcheck over scripts/: no new findings. Measured the cost the reviewer asked about, on ext4, warm cache, 78,554 entries: `cp -al` 3126 ms against 44 ms for one `find | wc -l`. Two counts per attempt is ~2.8% on top of the clone. Not measured on the CI runner's volume. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
266 lines
14 KiB
Bash
Executable File
266 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for seed-target-dir.sh: which source a run seeds from, and
|
|
# what happens when two jobs sharing one cache key seed at the same time.
|
|
#
|
|
# Runs the ACTUAL script against a real scratch cache directory with fake
|
|
# target trees standing in for cargo output — no compiler needed, so this is
|
|
# the fast half of the suite. hardlink-clone-selftest.sh covers the parts that
|
|
# need a real build.
|
|
#
|
|
# What each scenario demonstrates:
|
|
#
|
|
# 1. BASE SNAPSHOT PREFERRED — a PR whose base has published a snapshot
|
|
# seeds from it, and the seeded directory really is a hardlink clone
|
|
# (shared inodes), not a copy.
|
|
# 2. OWN DIR WINS — a second run of the same ref reuses what is already
|
|
# there and does not re-seed over its own work.
|
|
# 3. OWN SNAPSHOT AS SELF-RESTORE — a publisher whose live target dir was
|
|
# evicted restores from the snapshot it last published, instead of
|
|
# rebuilding cold.
|
|
# 4. FALLBACK DIR — with no snapshot at all, an explicitly configured
|
|
# fallback (a pre-existing flat cache, during a migration) is used.
|
|
# 5. COLD — with nothing available, the directory is created empty rather
|
|
# than the script failing.
|
|
# 6. LOCK FILES STRIPPED — Cargo's in-place-flock'd lock files never
|
|
# survive a clone, because a shared lock inode would make two branches
|
|
# contend on one mutex.
|
|
# 7. CONCURRENT SEED IS ATOMIC — two seeds racing on one cache key: the
|
|
# loser discards its staging copy and uses the winner's directory, and
|
|
# at no point is a partially-populated directory visible under the final
|
|
# name. This is the property that replaces "the runner only has one job
|
|
# slot" with an actual guarantee.
|
|
# 8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the
|
|
# one that actually mattered: a consumer hardlink-cloning a snapshot
|
|
# while the publisher of that snapshot rotates it and unlinks the
|
|
# generation being read. Two seeds racing on a DESTINATION is a different
|
|
# race from a seed racing a publisher on its SOURCE, and only the second
|
|
# one can truncate a tree. Against the unguarded version this scenario
|
|
# reproduces a silent partial clone reported as success — 20,328 of
|
|
# 48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
|
|
# 9. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct status
|
|
# instead of renaming whatever it managed to produce into place, and the
|
|
# seed SCRIPT turns that status into a failed job rather than a silent
|
|
# cold build. Retries are what make a torn read survivable; exhausting
|
|
# them must not degrade into "start cold and rebuild everything", which
|
|
# would turn a corrupt-cache bug into an invisible 4x-slower CI job.
|
|
# (The publisher's half of the rotation race — deferring reclamation
|
|
# while a reader is still in flight — lives in
|
|
# publish-snapshot-selftest.sh, next to the swap it modifies.)
|
|
set -euo pipefail
|
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
. "$script_dir/cache-lib.sh"
|
|
|
|
scratch=$(mktemp -d)
|
|
trap 'rm -rf "$scratch"' EXIT
|
|
root="$scratch/cache"
|
|
mkdir -p "$root"
|
|
pass_count=0
|
|
|
|
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
assert_file() { [ -e "$1" ] || fail "expected $1 to exist ($2)"; ok "$2"; }
|
|
assert_absent() { [ -e "$1" ] && fail "expected $1 to be gone ($2)"; ok "$2"; }
|
|
assert_content() { [ "$(cat "$1")" = "$2" ] || fail "expected '$2' in $1, got '$(cat "$1")' ($3)"; ok "$3"; }
|
|
|
|
# A plausible target tree: a big shared artifact, a mutable fingerprint, a
|
|
# build-script output, and a lock file.
|
|
make_tree() {
|
|
local d="$1" marker="$2"
|
|
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x" "$d/debug/build/x/out"
|
|
echo "$marker" > "$d/debug/deps/libx.rlib"
|
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
|
echo "$marker" > "$d/debug/build/x/out/gen.txt"
|
|
: > "$d/debug/.cargo-lock"
|
|
}
|
|
|
|
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
|
|
|
# Always succeeds and always prints a number: an absent directory is "0
|
|
# entries so far", which is the normal state at the top of the progress poll
|
|
# below, not an error worth aborting the suite over.
|
|
tree_entries() {
|
|
local n
|
|
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
|
|
printf '%s\n' "$n"
|
|
return 0
|
|
}
|
|
|
|
# A tree wide enough that a hardlink clone of it takes long enough to be
|
|
# caught mid-walk. The race under test is a real interleaving, not a mocked
|
|
# one, so the fixture has to be big enough for the window to exist: a
|
|
# four-file tree clones in microseconds and no scheduling could ever land
|
|
# inside it. Built by cloning one small template directory N times, which is N
|
|
# forks rather than N*M file creations.
|
|
make_wide_tree() {
|
|
local d="$1" marker="$2" ndirs="$3" i
|
|
mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x"
|
|
for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done
|
|
for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done
|
|
rm -rf "$d/debug/deps/.tmpl"
|
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
|
}
|
|
|
|
BASE_KEY=$(cache_key dev)
|
|
OWN_KEY=$(cache_key feat/thing)
|
|
|
|
echo "=== 1: base snapshot preferred, and cloned by hardlink ==="
|
|
make_tree "$root/snapshot-$BASE_KEY" base-content
|
|
seed "$OWN_KEY" "$BASE_KEY" "$root" job1
|
|
own="$root/target-$OWN_KEY"
|
|
assert_content "$own/debug/deps/libx.rlib" base-content "seeded from the base snapshot"
|
|
[ "$(stat -c '%i' "$own/debug/deps/libx.rlib")" = "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/deps/libx.rlib")" ] \
|
|
|| fail "artifact was copied, not hardlinked"
|
|
ok "artifact shares an inode with the snapshot (hardlink clone, not a copy)"
|
|
[ "$(stat -c '%i' "$own/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/.fingerprint/x/dep-lib-x")" ] \
|
|
|| fail "fingerprint still shares an inode with the snapshot"
|
|
ok "fingerprint is privately owned (unshare_mutable_paths ran)"
|
|
|
|
echo
|
|
echo "=== 6: Cargo lock files never survive a clone ==="
|
|
assert_absent "$own/debug/.cargo-lock" "cloned .cargo-lock removed"
|
|
|
|
echo
|
|
echo "=== 2: an existing own dir is reused, never re-seeded over ==="
|
|
echo own-work > "$own/debug/deps/libx.rlib"
|
|
seed "$OWN_KEY" "$BASE_KEY" "$root" job1
|
|
assert_content "$own/debug/deps/libx.rlib" own-work "own directory reused as-is"
|
|
grep -q 'reusing this ref' "$scratch/log" || fail "expected the reuse path in the log"
|
|
ok "reuse is reported in the log"
|
|
|
|
echo
|
|
echo "=== 3: a publisher restores from its own snapshot after eviction ==="
|
|
make_tree "$root/snapshot-$BASE_KEY" published-dev
|
|
seed "$BASE_KEY" "" "$root" job1
|
|
assert_content "$root/target-$BASE_KEY/debug/deps/libx.rlib" published-dev "publisher self-restored from its own snapshot"
|
|
|
|
echo
|
|
echo "=== 4: explicit fallback dir when no snapshot exists ==="
|
|
OTHER=$(cache_key feat/other)
|
|
make_tree "$scratch/legacy-flat" legacy
|
|
seed "$OTHER" "$(cache_key nosuch)" "$root" job1 "$scratch/legacy-flat"
|
|
assert_content "$root/target-$OTHER/debug/deps/libx.rlib" legacy "seeded from the fallback dir"
|
|
|
|
echo
|
|
echo "=== 5: cold start when nothing is available ==="
|
|
COLD=$(cache_key feat/cold)
|
|
seed "$COLD" "$(cache_key nosuch)" "$root" job1
|
|
[ -d "$root/target-$COLD" ] || fail "cold start did not create the directory"
|
|
[ -z "$(ls -A "$root/target-$COLD")" ] || fail "cold start directory is not empty"
|
|
ok "cold start creates an empty directory rather than failing"
|
|
|
|
echo
|
|
echo "=== 7: two jobs racing on one cache key ==="
|
|
RACE=$(cache_key feat/race)
|
|
make_tree "$root/snapshot-$BASE_KEY" race-source
|
|
# Both jobs seed concurrently from the same snapshot into the same key. Each
|
|
# stages under its own tag, so the only interaction is the final rename.
|
|
( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobA > "$scratch/logA" 2>&1 ) &
|
|
( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobB > "$scratch/logB" 2>&1 ) &
|
|
wait
|
|
race_dir="$root/target-$RACE"
|
|
assert_content "$race_dir/debug/deps/libx.rlib" race-source "the surviving directory is complete"
|
|
[ -z "$(find "$root" -maxdepth 1 -name '.stage-*' -print -quit)" ] || fail "a staging directory was left behind"
|
|
ok "no staging directory survived the race"
|
|
# Exactly one job may claim it seeded; the other must report either the
|
|
# concurrent-peer path or a plain reuse (if it started after the winner
|
|
# finished). Neither may report a cold start.
|
|
if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
|
|
cat "$scratch/logA" "$scratch/logB"; fail "a racing job reported a cold start"
|
|
fi
|
|
ok "neither racing job fell through to a cold start"
|
|
|
|
echo
|
|
echo "=== 8: seeding while the base republishes the snapshot underneath it ==="
|
|
ROT=$(cache_key feat/rotate)
|
|
rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY"
|
|
# Generation 1 is wide (the consumer will still be walking it when the swap
|
|
# happens); the generation replacing it is small, so the publisher's own
|
|
# staging clone does not itself outlast the consumer's.
|
|
make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800
|
|
make_wide_tree "$root/target-$BASE_KEY" gen2 8
|
|
gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
|
|
|
( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) &
|
|
seed_pid=$!
|
|
|
|
# Rotate only once the clone is demonstrably mid-walk. Gating on observed
|
|
# progress rather than on a sleep is what makes the interleaving reproducible
|
|
# instead of a coin flip that passes on a fast machine for the wrong reason.
|
|
threshold=$(( gen1_entries / 5 ))
|
|
progress=0
|
|
deadline=$(( $(date +%s) + 60 ))
|
|
while :; do
|
|
progress=$(tree_entries "$root/.stage-jobRot")
|
|
if [ "$progress" -ge "$threshold" ]; then break; fi
|
|
if ! kill -0 "$seed_pid" 2>/dev/null; then
|
|
fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)"
|
|
fi
|
|
if [ "$(date +%s)" -ge "$deadline" ]; then
|
|
fail "the staging clone never reached ${threshold} of ${gen1_entries} entries"
|
|
fi
|
|
done
|
|
ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries"
|
|
|
|
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \
|
|
|| { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; }
|
|
wait "$seed_pid"
|
|
|
|
rot_dir="$root/target-$ROT"
|
|
[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; }
|
|
ok "the seed completed"
|
|
# THE assertion. Before the guard, this is where it failed: the seed reported
|
|
# `cloned in 1s` and exit 0 while target-<rot> held less than half the entries
|
|
# of the snapshot it claimed to have cloned. Comparing against the snapshot as
|
|
# it stands NOW is the right bar either way — a clone that raced the rotation
|
|
# must end up holding one complete generation, and a consumer caught mid-walk
|
|
# re-reads, so that generation is the new one.
|
|
snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
|
rot_entries=$(tree_entries "$rot_dir")
|
|
[ "$rot_entries" -eq "$snap_entries" ] \
|
|
|| fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)"
|
|
ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)"
|
|
assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two"
|
|
grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read"
|
|
ok "the torn read was detected and reported, not swallowed"
|
|
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \
|
|
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)"
|
|
ok "no staging, reader-marker or deferred-generation scratch left behind"
|
|
|
|
echo
|
|
echo "=== 9: a source that cannot be read fails loudly ==="
|
|
rc=0
|
|
hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$?
|
|
[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}"
|
|
ok "an unreadable source returns the distinct hard-failure status"
|
|
assert_absent "$root/target-nosuch" "nothing was renamed into place"
|
|
|
|
# The status only matters if the script acts on it. A source that exists but
|
|
# cannot be read exercises the whole path: retries exhaust, the function
|
|
# returns 2, and seed-target-dir.sh must exit non-zero rather than falling
|
|
# through to its cold-start branch.
|
|
if [ "$(id -u)" = "0" ]; then
|
|
echo "SKIP: running as root — mode bits do not deny access"
|
|
else
|
|
UNREADABLE=$(cache_key feat/unreadable)
|
|
VICTIM=$(cache_key feat/victim)
|
|
make_tree "$root/snapshot-$UNREADABLE" locked-away
|
|
chmod 000 "$root/snapshot-$UNREADABLE"
|
|
rc=0
|
|
CACHE_CLONE_ATTEMPTS=2 bash "$script_dir/seed-target-dir.sh" \
|
|
"$VICTIM" "$UNREADABLE" "$root" jobUnread > "$scratch/logUnread" 2>&1 || rc=$?
|
|
chmod 755 "$root/snapshot-$UNREADABLE"
|
|
[ "$rc" -ne 0 ] || { tail -20 "$scratch/logUnread"; fail "the seed reported success against a source it could not read"; }
|
|
ok "the seed exits non-zero when its source cannot be cloned"
|
|
grep -q 'refusing to build against a partial cache' "$scratch/logUnread" \
|
|
|| { tail -20 "$scratch/logUnread"; fail "the failure was not reported as such"; }
|
|
ok "the failure names the reason rather than exiting silently"
|
|
if grep -q 'starts cold' "$scratch/logUnread"; then
|
|
tail -20 "$scratch/logUnread"; fail "an unreadable source degraded into a silent cold build"
|
|
fi
|
|
ok "it does not degrade into a silent cold build"
|
|
assert_absent "$root/target-$VICTIM" "no target dir was left behind by the failed seed"
|
|
fi
|
|
|
|
echo
|
|
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|