claude
5551e994da
README's Development section stated the repo's methodology for writing concurrency scenarios as "gate the interfering step on observed progress of the step it interferes with, so the window is hit deterministically". That described the progress poll scenario 8 used, which this branch removes — and the property it claims is precisely what issue #3 records as false: observing that a walk has started says nothing about where it will be when the interference lands. Left standing it would tell the next contributor to build the next scenario the way this one had to be rewritten. Replaced with what the suites do: stub, on PATH, a command the code under test calls at a known point, so placement is a fact rather than a scheduling outcome; assert the stub fired; and assert which guard caught the fault where more than one could. Also: the seed suite's table row now names both tear modes, and publish-snapshot-selftest.sh's cross-reference points at 8a and 8b rather than a scenario 8 that no longer exists. (publish-snapshot.sh's similar mislabel predates this branch and is left alone.) The stub directory and the real-cp lookup move up next to seed_with_stub, so 8b no longer depends on setup buried in 8a's block and either scenario can be run or mutated alone.
211 lines
11 KiB
Bash
Executable File
211 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Regression test for publish-snapshot.sh: the atomic swap, and the two
|
|
# properties the swap exists to guarantee.
|
|
#
|
|
# 1. FIRST PUBLISH — with no prior snapshot, the target dir is published and
|
|
# is a hardlink clone of it (cheap), with the mutable metadata privately
|
|
# owned (sound: the publisher's NEXT build must not be able to mutate the
|
|
# snapshot it just published).
|
|
# 2. REPUBLISH REPLACES — a second publish replaces the snapshot's content
|
|
# rather than merging into it, and leaves no scratch directories behind.
|
|
# 3. A LIVE CONSUMER SURVIVES A REPUBLISH — a branch that already cloned the
|
|
# previous generation keeps reading its own consistent copy. Removing the
|
|
# old snapshot unlinks directory entries; the inodes stay alive through
|
|
# the consumer's own links. This is why a republish can never pull data
|
|
# out from under a running job.
|
|
# 4. NO TARGET DIR — publishing when there is nothing to publish is a no-op,
|
|
# not a failure.
|
|
# 5. LOCKS AND MARKERS DO NOT RIDE ALONG — the publishing job's own cache
|
|
# lock is still held while this runs, and must not be baked into the
|
|
# snapshot: a lock timestamped at this run's start would look fresh to
|
|
# the prune pass on every branch later seeded from it.
|
|
# 6. DEFERRED RECLAMATION — the publisher's half of the seed-vs-rotation
|
|
# race. Scenario 3 above covers a consumer that has ALREADY FINISHED
|
|
# cloning; that one is safe for free, because its own hardlinks keep the
|
|
# inodes alive. A consumer still WALKING the old generation is the case
|
|
# that actually tears, and unlinking underneath it is what produced a
|
|
# silently truncated clone. So when a reader is still in flight past the
|
|
# grace period, the swap leaves the rotated-away generation on disk
|
|
# instead of unlinking it.
|
|
# 7. AND THE SWEEP — a deferred generation is not leaked: the next publish
|
|
# of that snapshot reclaims it once no reader holds it. Without this the
|
|
# "we defer instead of forcing" answer would just be a disk leak with
|
|
# better manners.
|
|
# 8. AN ABANDONED MARKER DOES NOT PIN A GENERATION FOREVER — and the
|
|
# staleness bound is genuinely consulted rather than old markers being
|
|
# unconditionally ignored. Scenario 6 covers a marker that is cleanly
|
|
# released; this is the other exit from a reader's lifetime, the one a
|
|
# killed job takes. Both halves are asserted against the SAME backdated
|
|
# marker, which is what separates "honours the bound" from "ignores
|
|
# anything that looks old": under a bound wide enough to still cover it,
|
|
# that marker must still defer.
|
|
#
|
|
# The consumer's half of the same race — a seed catching a rotation mid-clone
|
|
# — is in seed-target-dir-selftest.sh scenarios 8a and 8b.
|
|
set -euo pipefail
|
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
. "$script_dir/cache-lib.sh"
|
|
|
|
scratch=$(mktemp -d)
|
|
trap 'rm -rf "$scratch"' EXIT
|
|
root="$scratch/cache"; mkdir -p "$root"
|
|
pass_count=0
|
|
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
|
|
# Cargo and rustc REPLACE an artifact (write elsewhere, rename over the path)
|
|
# rather than truncating it in place, which is exactly why a snapshot may
|
|
# share artifact inodes with the live target dir it was cloned from. The
|
|
# fixtures here have to model that faithfully — a plain `>` redirect truncates
|
|
# in place and would write straight through the shared inode into the
|
|
# snapshot and every consumer, which is a property of the test fixture, not of
|
|
# a real build. hardlink-clone-selftest.sh is what verifies the real thing
|
|
# against a real compiler.
|
|
replace_file() {
|
|
printf '%s\n' "$2" > "$1.new"
|
|
mv -f "$1.new" "$1"
|
|
}
|
|
|
|
make_tree() {
|
|
local d="$1" marker="$2"
|
|
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x"
|
|
echo "$marker" > "$d/debug/deps/libx.rlib"
|
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
|
: > "$d/debug/.cargo-lock"
|
|
}
|
|
|
|
KEY=$(cache_key dev)
|
|
TGT="$root/target-$KEY"
|
|
SNAP="$root/snapshot-$KEY"
|
|
publish() { bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" "$1" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }; }
|
|
|
|
echo "=== 4: nothing to publish is a no-op ==="
|
|
publish job1
|
|
[ -d "$SNAP" ] && fail "published a snapshot with no target dir present"
|
|
grep -q 'nothing to snapshot' "$scratch/log" || fail "expected a 'nothing to snapshot' line"
|
|
ok "no target dir: no-op, reported plainly"
|
|
|
|
echo
|
|
echo "=== 1: first publish ==="
|
|
make_tree "$TGT" gen1
|
|
date +%s > "$TGT/.ci-lock-ci-42"
|
|
touch "$TGT/.cache-last-used"
|
|
publish job1
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen1" ] || fail "snapshot content wrong"
|
|
ok "snapshot published"
|
|
[ "$(stat -c '%i' "$SNAP/debug/deps/libx.rlib")" = "$(stat -c '%i' "$TGT/debug/deps/libx.rlib")" ] \
|
|
|| fail "snapshot artifact was copied, not hardlinked"
|
|
ok "snapshot shares artifact inodes with the target dir (cheap)"
|
|
[ "$(stat -c '%i' "$SNAP/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$TGT/debug/.fingerprint/x/dep-lib-x")" ] \
|
|
|| fail "snapshot fingerprint still aliases the live target dir"
|
|
ok "snapshot owns its mutable metadata (publisher's next build cannot corrupt it)"
|
|
|
|
echo
|
|
echo "=== 5: locks and LRU markers do not ride along ==="
|
|
[ -e "$SNAP/.ci-lock-ci-42" ] && fail "the publishing job's lock was baked into the snapshot"
|
|
ok "cache lock not published"
|
|
[ -e "$SNAP/.cache-last-used" ] && fail "the LRU marker was baked into the snapshot"
|
|
ok "LRU marker not published"
|
|
[ -e "$SNAP/debug/.cargo-lock" ] && fail "a Cargo lock file was published"
|
|
ok "Cargo lock file not published"
|
|
|
|
echo
|
|
echo "=== 3: a consumer that cloned generation 1 ==="
|
|
CONSUMER="$root/target-$(cache_key feat/consumer)"
|
|
hardlink_clone_into "$SNAP" "$CONSUMER" consumer-tag || fail "consumer clone failed"
|
|
ok "consumer cloned generation 1"
|
|
|
|
echo
|
|
echo "=== 2: republish replaces, leaves no scratch behind ==="
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen2
|
|
# The fingerprint IS written in place by Cargo — and the snapshot owns its own
|
|
# copy precisely so that write cannot reach it. Truncating in place here is
|
|
# the faithful model.
|
|
echo gen2 > "$TGT/debug/.fingerprint/x/dep-lib-x"
|
|
publish job1
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen2" ] || fail "republish did not replace the snapshot"
|
|
ok "republished snapshot carries generation 2"
|
|
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.publish-*' \) -print)
|
|
[ -z "$leftovers" ] || fail "scratch directories left behind: $leftovers"
|
|
ok "no scratch directories left behind"
|
|
[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \
|
|
|| fail "the consumer's clone changed under it when the snapshot was replaced"
|
|
ok "the live consumer still reads its own consistent generation-1 copy"
|
|
|
|
echo
|
|
echo "=== 6: a reader still in flight defers reclamation ==="
|
|
# A synthetic reader marker stands in for a consumer whose clone outlasts the
|
|
# grace period. Racing a real slow consumer would make the suite's runtime the
|
|
# thing under test; the marker IS the entire contract between the two sides,
|
|
# so holding one is being a reader.
|
|
SNAP_NAME=$(basename "$SNAP")
|
|
date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen3
|
|
CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobDefer \
|
|
> "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen3" ] || fail "the new generation was not published"
|
|
ok "the new generation is published even while a reader holds the old one"
|
|
deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${KEY}-*" -print -quit)
|
|
[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
|
|
ok "the rotated-away generation is left on disk rather than unlinked under a reader"
|
|
grep -q 'deferring reclamation' "$scratch/log" || fail "the deferral was not reported"
|
|
ok "the deferral is surfaced as a warning, not silent"
|
|
[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \
|
|
|| fail "the earlier consumer's clone changed under it"
|
|
ok "the generation-1 consumer is still unaffected"
|
|
|
|
echo
|
|
echo "=== 7: a later publish sweeps the deferred generation ==="
|
|
rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen4
|
|
publish jobSweep
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen4" ] || fail "the fourth generation was not published"
|
|
ok "publishing continues normally after a deferral"
|
|
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
|
|| fail "the deferred generation was never reclaimed — this is a disk leak"
|
|
ok "the deferred generation is reclaimed once no reader holds it"
|
|
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print -quit)" ] \
|
|
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print)"
|
|
ok "no staging or reader-marker scratch left behind"
|
|
|
|
echo
|
|
echo "=== 8: an abandoned reader marker is swept, and the bound is honoured ==="
|
|
# A reader whose job the runner killed never reaches its release. Backdated
|
|
# rather than slept for: the default bound is two hours, and a suite that
|
|
# waited it out would not be a suite anyone runs.
|
|
CRASHED="$root/.reading-${SNAP_NAME}-crashed"
|
|
date +%s > "$CRASHED"
|
|
touch -d '3 hours ago' "$CRASHED"
|
|
|
|
# First, the negative control. The same three-hour-old marker, under a bound
|
|
# wide enough to still cover it, must defer exactly as a live one does — if
|
|
# this passed only because the marker looked old, the sweep below would prove
|
|
# nothing about the bound.
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen5
|
|
CACHE_READ_STALE_SECONDS=86400 CACHE_READ_GRACE_SECONDS=1 \
|
|
bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobWideBound > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
[ -e "$CRASHED" ] || fail "a marker inside the staleness bound was swept anyway"
|
|
ok "a marker inside the staleness bound is left alone"
|
|
[ -n "$(find "$root" -maxdepth 1 -name ".publish-old-${KEY}-*" -print -quit)" ] \
|
|
|| fail "the generation was reclaimed despite a marker inside the bound"
|
|
ok "and still defers reclamation, exactly as a live reader does"
|
|
|
|
# Now the same marker against the default bound it is genuinely past.
|
|
replace_file "$TGT/debug/deps/libx.rlib" gen6
|
|
CACHE_READ_GRACE_SECONDS=1 \
|
|
bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" jobStaleSweep > "$scratch/log" 2>&1 \
|
|
|| { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }
|
|
grep -q 'sweeping stale marker' "$scratch/log" || { cat "$scratch/log"; fail "the stale marker was not reported as swept"; }
|
|
ok "a marker past the bound is swept, and says so"
|
|
[ -e "$CRASHED" ] && fail "the stale marker survived the sweep"
|
|
ok "the abandoned marker is gone"
|
|
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
|
|| fail "an abandoned marker pinned a generation past its staleness bound"
|
|
ok "the generation it was pinning — and the one deferred earlier — are reclaimed"
|
|
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen6" ] || fail "the current generation is wrong"
|
|
ok "publishing is otherwise unaffected"
|
|
|
|
echo
|
|
echo "publish-snapshot-selftest: ${pass_count} assertions passed"
|