claudeandClaude Opus 5
3b2dec6a50
The staleness path decides whether a publisher may reclaim disk, so getting it
wrong means an abandoned marker pins a snapshot generation forever — the exact
outcome the bound exists to prevent. It was previously covered only by
analogy to prune-cache.sh's .ci-lock-* staleness, which is not the bar.
New publish-snapshot-selftest.sh scenario 8 asserts both directions against
the SAME backdated marker, which is what separates "honours the bound" from
"ignores anything that looks old":
* under CACHE_READ_STALE_SECONDS=86400 a three-hour-old marker is left alone
and still defers reclamation, exactly as a live reader does;
* under the 7200s default the same marker is swept, reported as swept, and the
generation it was pinning — plus the one deferred by the first half — is
reclaimed.
Backdated with `touch -d`, not slept for; the suite stays fast.
Red-proven by mutation rather than against the pre-fix scripts, since the
whole mechanism is new there and "it does not exist yet" proves nothing about
the threshold logic. Mutating live_reader_count's bound test to `true` (never
sweep) fails scenario 8:
ASSERTION FAILED: the stale marker was not reported as swept
and to `false` (sweep everything, bound ignored) fails scenario 6 instead,
which is the right blast radius — ignoring the bound means unlinking under a
LIVE reader:
ASSERTION FAILED: the previous generation was unlinked while a reader
still held it
Also documents the entry-count check's measured cost in the README: on ext4
with a warm cache over 78,554 entries, 44 ms per metadata walk against 3,126 ms
for the `cp -al` it guards — about 2.8%. Not a perf-claiming change; the number
is there so the next reader does not have to wonder.
The reviewer's other nit — scenarios 8/9 of the seed suite exercising
publish-snapshot.sh's interlock — was already covered by the cross-reference
in this file's header, so no move.
Verification: `bash scripts/selftest.sh` — 5 suites, exit 0, 88 assertions
(82 before this commit, 63 at baseline). shellcheck: no new findings.
Refs: daniel/gitdan#11, zemyna#911
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
211 lines
11 KiB
Bash
Executable File
211 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for publish-snapshot.sh: the atomic swap, and the two
|
|
# properties the swap exists to guarantee.
|
|
#
|
|
# 1. FIRST PUBLISH — with no prior snapshot, the target dir is published and
|
|
# is a hardlink clone of it (cheap), with the mutable metadata privately
|
|
# owned (sound: the publisher's NEXT build must not be able to mutate the
|
|
# snapshot it just published).
|
|
# 2. REPUBLISH REPLACES — a second publish replaces the snapshot's content
|
|
# rather than merging into it, and leaves no scratch directories behind.
|
|
# 3. A LIVE CONSUMER SURVIVES A REPUBLISH — a branch that already cloned the
|
|
# previous generation keeps reading its own consistent copy. Removing the
|
|
# old snapshot unlinks directory entries; the inodes stay alive through
|
|
# the consumer's own links. This is why a republish can never pull data
|
|
# out from under a running job.
|
|
# 4. NO TARGET DIR — publishing when there is nothing to publish is a no-op,
|
|
# not a failure.
|
|
# 5. LOCKS AND MARKERS DO NOT RIDE ALONG — the publishing job's own cache
|
|
# lock is still held while this runs, and must not be baked into the
|
|
# snapshot: a lock timestamped at this run's start would look fresh to
|
|
# the prune pass on every branch later seeded from it.
|
|
# 6. DEFERRED RECLAMATION — the publisher's half of the seed-vs-rotation
|
|
# race. Scenario 3 above covers a consumer that has ALREADY FINISHED
|
|
# cloning; that one is safe for free, because its own hardlinks keep the
|
|
# inodes alive. A consumer still WALKING the old generation is the case
|
|
# that actually tears, and unlinking underneath it is what produced a
|
|
# silently truncated clone. So when a reader is still in flight past the
|
|
# grace period, the swap leaves the rotated-away generation on disk
|
|
# instead of unlinking it.
|
|
# 7. AND THE SWEEP — a deferred generation is not leaked: the next publish
|
|
# of that snapshot reclaims it once no reader holds it. Without this the
|
|
# "we defer instead of forcing" answer would just be a disk leak with
|
|
# better manners.
|
|
# 8. AN ABANDONED MARKER DOES NOT PIN A GENERATION FOREVER — and the
|
|
# staleness bound is genuinely consulted rather than old markers being
|
|
# unconditionally ignored. Scenario 6 covers a marker that is cleanly
|
|
# released; this is the other exit from a reader's lifetime, the one a
|
|
# killed job takes. Both halves are asserted against the SAME backdated
|
|
# marker, which is what separates "honours the bound" from "ignores
|
|
# anything that looks old": under a bound wide enough to still cover it,
|
|
# that marker must still defer.
|
|
#
|
|
# The consumer's half of the same race — a seed catching a rotation mid-clone
|
|
# — is in seed-target-dir-selftest.sh scenario 8.
|
|
set -euo pipefail
|
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
. "$script_dir/cache-lib.sh"
|
|
|
|
scratch=$(mktemp -d)
|
|
trap 'rm -rf "$scratch"' EXIT
|
|
root="$scratch/cache"; mkdir -p "$root"
|
|
pass_count=0
|
|
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
|
|
# Cargo and rustc REPLACE an artifact (write elsewhere, rename over the path)
|
|
# rather than truncating it in place, which is exactly why a snapshot may
|
|
# share artifact inodes with the live target dir it was cloned from. The
|
|
# fixtures here have to model that faithfully — a plain `>` redirect truncates
|
|
# in place and would write straight through the shared inode into the
|
|
# snapshot and every consumer, which is a property of the test fixture, not of
|
|
# a real build. hardlink-clone-selftest.sh is what verifies the real thing
|
|
# against a real compiler.
|
|
replace_file() {
|
|
printf '%s\n' "$2" > "$1.new"
|
|
mv -f "$1.new" "$1"
|
|
}
|
|
|
|
make_tree() {
|
|
local d="$1" marker="$2"
|
|
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x"
|
|
echo "$marker" > "$d/debug/deps/libx.rlib"
|
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
|
: > "$d/debug/.cargo-lock"
|
|
}
|
|
|
|
KEY=$(cache_key dev)
|
|
TGT="$root/target-$KEY"
|
|
SNAP="$root/snapshot-$KEY"
|
|
publish() { bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" "$1" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }; }
|
|
|
|
echo "=== 4: nothing to publish is a no-op ==="
|
|
publish job1
|
|
[ -d "$SNAP" ] && fail "published a snapshot with no target dir present"
|
|
grep -q 'nothing to snapshot' "$scratch/log" || fail "expected a 'nothing to snapshot' line"
|
|
ok "no target dir: no-op, reported plainly"
|
|
|
|
echo
|
|
echo "=== 1: first publish ==="
|
|
make_tree "$TGT" gen1
|
|
date +%s > "$TGT/.ci-lock-ci-42"
|
|
touch "$TGT/.cache-last-used"
|
|
publish job1
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen1" ] || fail "snapshot content wrong"
|
|
ok "snapshot published"
|
|
[ "$(stat -c '%i' "$SNAP/debug/deps/libx.rlib")" = "$(stat -c '%i' "$TGT/debug/deps/libx.rlib")" ] \
|
|
|| fail "snapshot artifact was copied, not hardlinked"
|
|
ok "snapshot shares artifact inodes with the target dir (cheap)"
|
|
[ "$(stat -c '%i' "$SNAP/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$TGT/debug/.fingerprint/x/dep-lib-x")" ] \
|
|
|| fail "snapshot fingerprint still aliases the live target dir"
|
|
ok "snapshot owns its mutable metadata (publisher's next build cannot corrupt it)"
|
|
|
|
echo
|
|
echo "=== 5: locks and LRU markers do not ride along ==="
|
|
[ -e "$SNAP/.ci-lock-ci-42" ] && fail "the publishing job's lock was baked into the snapshot"
|
|
ok "cache lock not published"
|
|
[ -e "$SNAP/.cache-last-used" ] && fail "the LRU marker was baked into the snapshot"
|
|
ok "LRU marker not published"
|
|
[ -e "$SNAP/debug/.cargo-lock" ] && fail "a Cargo lock file was published"
|
|
ok "Cargo lock file not published"
|
|
|
|
echo
|
|
echo "=== 3: a consumer that cloned generation 1 ==="
|
|
CONSUMER="$root/target-$(cache_key feat/consumer)"
|
|
hardlink_clone_into "$SNAP" "$CONSUMER" consumer-tag || fail "consumer clone failed"
|
|
ok "consumer cloned generation 1"
|
|
|
|
echo
|
|
echo "=== 2: republish replaces, leaves no scratch behind ==="
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen2
|
|
# The fingerprint IS written in place by Cargo — and the snapshot owns its own
|
|
# copy precisely so that write cannot reach it. Truncating in place here is
|
|
# the faithful model.
|
|
echo gen2 > "$TGT/debug/.fingerprint/x/dep-lib-x"
|
|
publish job1
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen2" ] || fail "republish did not replace the snapshot"
|
|
ok "republished snapshot carries generation 2"
|
|
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.publish-*' \) -print)
|
|
[ -z "$leftovers" ] || fail "scratch directories left behind: $leftovers"
|
|
ok "no scratch directories left behind"
|
|
[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \
|
|
|| fail "the consumer's clone changed under it when the snapshot was replaced"
|
|
ok "the live consumer still reads its own consistent generation-1 copy"
|
|
|
|
echo
|
|
echo "=== 6: a reader still in flight defers reclamation ==="
|
|
# A synthetic reader marker stands in for a consumer whose clone outlasts the
|
|
# grace period. Racing a real slow consumer would make the suite's runtime the
|
|
# thing under test; the marker IS the entire contract between the two sides,
|
|
# so holding one is being a reader.
|
|
SNAP_NAME=$(basename "$SNAP")
|
|
date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen3
|
|
CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobDefer \
|
|
> "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen3" ] || fail "the new generation was not published"
|
|
ok "the new generation is published even while a reader holds the old one"
|
|
deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${KEY}-*" -print -quit)
|
|
[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
|
|
ok "the rotated-away generation is left on disk rather than unlinked under a reader"
|
|
grep -q 'deferring reclamation' "$scratch/log" || fail "the deferral was not reported"
|
|
ok "the deferral is surfaced as a warning, not silent"
|
|
[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \
|
|
|| fail "the earlier consumer's clone changed under it"
|
|
ok "the generation-1 consumer is still unaffected"
|
|
|
|
echo
|
|
echo "=== 7: a later publish sweeps the deferred generation ==="
|
|
rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen4
|
|
publish jobSweep
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen4" ] || fail "the fourth generation was not published"
|
|
ok "publishing continues normally after a deferral"
|
|
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
|
|| fail "the deferred generation was never reclaimed — this is a disk leak"
|
|
ok "the deferred generation is reclaimed once no reader holds it"
|
|
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print -quit)" ] \
|
|
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print)"
|
|
ok "no staging or reader-marker scratch left behind"
|
|
|
|
echo
|
|
echo "=== 8: an abandoned reader marker is swept, and the bound is honoured ==="
|
|
# A reader whose job the runner killed never reaches its release. Backdated
|
|
# rather than slept for: the default bound is two hours, and a suite that
|
|
# waited it out would not be a suite anyone runs.
|
|
CRASHED="$root/.reading-${SNAP_NAME}-crashed"
|
|
date +%s > "$CRASHED"
|
|
touch -d '3 hours ago' "$CRASHED"
|
|
|
|
# First, the negative control. The same three-hour-old marker, under a bound
|
|
# wide enough to still cover it, must defer exactly as a live one does — if
|
|
# this passed only because the marker looked old, the sweep below would prove
|
|
# nothing about the bound.
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen5
|
|
CACHE_READ_STALE_SECONDS=86400 CACHE_READ_GRACE_SECONDS=1 \
|
|
bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobWideBound > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
[ -e "$CRASHED" ] || fail "a marker inside the staleness bound was swept anyway"
|
|
ok "a marker inside the staleness bound is left alone"
|
|
[ -n "$(find "$root" -maxdepth 1 -name ".publish-old-${KEY}-*" -print -quit)" ] \
|
|
|| fail "the generation was reclaimed despite a marker inside the bound"
|
|
ok "and still defers reclamation, exactly as a live reader does"
|
|
|
|
# Now the same marker against the default bound it is genuinely past.
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen6
|
|
CACHE_READ_GRACE_SECONDS=1 \
|
|
bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobStaleSweep > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
grep -q 'sweeping stale marker' "$scratch/log" || { cat "$scratch/log"; fail "the stale marker was not reported as swept"; }
|
|
ok "a marker past the bound is swept, and says so"
|
|
[ -e "$CRASHED" ] && fail "the stale marker survived the sweep"
|
|
ok "the abandoned marker is gone"
|
|
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
|
|| fail "an abandoned marker pinned a generation past its staleness bound"
|
|
ok "the generation it was pinning — and the one deferred earlier — are reclaimed"
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen6" ] || fail "the current generation is wrong"
|
|
ok "publishing is otherwise unaffected"
|
|
|
|
echo
|
|
echo "publish-snapshot-selftest: ${pass_count} assertions passed"
|