test(seed): pin reader_lock_acquire in hardlink_clone_into (#12) #12

Merged
claude merged 2 commits from fix/reader-lock-pin into main 2026-08-24 23:05:18 +00:00
2 changed files with 109 additions and 5 deletions
+19 -5
View File
@@ -422,7 +422,7 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
| suite | covers | | suite | covers |
|---|---| |---|---|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. | | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published | | `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone | | `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` | | `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. | | `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
@@ -442,7 +442,7 @@ key and asserts only what must be true whichever of them wins the rename.
which places the interference inside the window rather than hoping it lands which places the interference inside the window rather than hoping it lands
there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
measurement publishes a reader marker strictly between its check and its measurement publishes a reader marker strictly between its check and its
unlink. `seed-target-dir-selftest.sh` uses the shape four times over, on two unlink. `seed-target-dir-selftest.sh` uses the shape five times over, on two
different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
clone is what rotates the snapshot underneath it, loses a subtree of its own clone is what rotates the snapshot underneath it, loses a subtree of its own
source, or reports a failure over a tree that is in fact whole — each strictly source, or reports a failure over a tree that is in fact whole — each strictly
@@ -453,9 +453,14 @@ could not be READ rather than on one that changed, and the only way to make
that the sole witness is to fail the identity reads while the copy between that the sole witness is to fail the identity reads while the copy between
them succeeds. 8a also starts a second real process — the actual them succeeds. 8a also starts a second real process — the actual
`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the `publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
concurrency is incidental to the determinism. Every stub asserts that it concurrency is incidental to the determinism. Scenario 11 reuses 8a's exact
fired, because a scenario whose interference silently did not happen passes stub and the same forced rotation, but reads a different witness: not the
for the wrong reason. consumer's own checks, but a line in the *publisher's* log reporting that it
waited on the reader marker this consumer's clone wrote — `reader_lock_acquire`
is exercised by 8a already, but nothing asserts it actually fired until this
scenario reads that line back. Every stub asserts that it fired, because a
scenario whose interference silently did not happen passes for the wrong
reason.
**A synthetic stand-in for the other side, where that artefact *is* the **A synthetic stand-in for the other side, where that artefact *is* the
contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
@@ -487,6 +492,15 @@ how both `[ "$cp_rc" -eq 0 ]` and `[ "$i_before" != missing ]` sat unpinned
(issue #5) while looking well covered. Isolating a check means constructing (issue #5) while looking well covered. Isolating a check means constructing
the state only it can see, not the state that trips several at once. the state only it can see, not the state that trips several at once.
Scenario 11 applies the same discipline to a witness outside the clone
entirely: not which of several checks inside `hardlink_clone_into` caught a
fault, but whether `reader_lock_acquire`'s marker was observed by anything
outside it at all. The consumer's own log and exit status are silent either
way — a run with the marker deleted still succeeds — so what is asserted is
one line in the *publisher's* log reporting that it waited. Deleting
`reader_lock_acquire` (issue #10) leaves that line unwritten without failing
anything else in the suite.
The action YAML holds no logic beyond wiring; everything testable lives in The action YAML holds no logic beyond wiring; everything testable lives in
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and `scripts/`. A composite action needs `shell: bash` on every `run:` step, and
the actions reach their shared scripts through the actions reach their shared scripts through
+90
View File
@@ -74,6 +74,17 @@
# onto one set of fingerprints. That is the silent stale-reuse bug the # onto one set of fingerprints. That is the silent stale-reuse bug the
# whole scheme exists to prevent, so the failure has to abort the clone # whole scheme exists to prevent, so the failure has to abort the clone
# rather than be swallowed. # rather than be swallowed.
# 11. THE CONSUMER'S MARKER ACTUALLY STOPS THE PUBLISHER — reader_lock_acquire
# is exercised (not faked, unlike publish-snapshot-selftest.sh's scenario
# 6) by a real hardlink_clone_into racing a real, concurrent
# publish-snapshot.sh republish of the exact snapshot being cloned. Pins
# that the publisher OBSERVABLY WAITS on this consumer's marker — its own
# log reports entering the drain wait — rather than only that the run
# succeeds, which stayed green with the marker call deleted (issue #10).
# Kept as its own scenario, not folded into 8a, because 8a already pins
# exactly one property (the identity check) for exactly one mutant, and
# the suite's one-scenario-one-mutant diagonal across 8a to 8d and 10 is
# deliberate.
set -euo pipefail set -euo pipefail
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$script_dir/cache-lib.sh" . "$script_dir/cache-lib.sh"
@@ -635,5 +646,84 @@ leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \
ok "no staging or reader-marker scratch left behind" ok "no staging or reader-marker scratch left behind"
rm -f "$scratch/bin/cp" rm -f "$scratch/bin/cp"
echo
echo "=== 11: the consumer's marker actually stops the publisher ==="
# hardlink_clone_into's reader_lock_acquire call is exercised here, not
# faked. publish-snapshot-selftest.sh's scenario 6 stands a hand-written
# marker file in for "a consumer whose clone outlasts the grace period" — a
# deliberate simplification that does not need this consumer's clone code at
# all, so it cannot tell reader_lock_acquire apart from no marker existing.
# This scenario forces the two real scripts to race on the same snapshot: a
# consumer hardlink-cloning it, and a publisher republishing it out from
# under that clone, exactly as 8a does to force the identity check — but what
# is pinned here is not the consumer's response to the rotation (8a's
# property), it is that the PUBLISHER, reading the SAME marker this
# consumer's clone wrote, is observed to have entered its drain wait. Deleting
# reader_lock_acquire (issue #10) leaves the marker never written: the
# publisher's live_reader_count sees zero readers on its first check and
# proceeds straight to reclaiming the rotated generation, silently — the run
# still succeeds, and nothing about its own outcome says so, only the absence
# of a line in the publisher's log.
INTLK=$(cache_key feat/interlock-consumer)
INTLK_BASE=$(cache_key release/5)
TAG_INTLK=jobIntlk
rm -rf "$root/snapshot-$INTLK_BASE" "$root/target-$INTLK_BASE"
make_tree "$root/snapshot-$INTLK_BASE" intlk-gen1
make_tree "$root/target-$INTLK_BASE" intlk-gen2
intlk_gen1_inode=$(stat -c '%i' "$root/snapshot-$INTLK_BASE")
cat > "$scratch/bin/cp" <<EOF
#!/usr/bin/env bash
# Fires once, only on the consumer's own top-level hardlink clone — identified
# by its destination, the seed's private staging path. Every other cp in the
# process tree (the unshare copies, and the publisher's own staging clone)
# falls through to the real one.
if [ "\${@: -1}" = "$root/.stage-$TAG_INTLK" ] && [ ! -e "$scratch/firedIntlk" ]; then
: > "$scratch/firedIntlk"
rc=0; "$real_cp" "\$@" || rc=\$?
# Concurrently: a real, second publish of the exact snapshot this consumer
# is cloning — the republish that, without the marker this consumer's clone
# holds, would reclaim the generation out from under it.
( CACHE_READ_GRACE_SECONDS=10 bash "$script_dir/publish-snapshot.sh" "$INTLK_BASE" "$root" pubIntlk \
> "$scratch/logPubIntlk" 2>&1
echo \$? > "$scratch/rcPubIntlk" ) &
# Hand control back only once the swap is on disk, so the identity read
# immediately after this cp is guaranteed to resolve to the new generation
# — the same technique 8a uses to force the interleaving rather than hope
# for it. reader_lock_release does not run until AFTER this script exits,
# so the marker stays live for the publisher's whole swap-and-scan.
deadline=\$(( \$(date +%s) + 60 ))
while [ "\$(stat -c '%i' "$root/snapshot-$INTLK_BASE" 2>/dev/null)" = "$intlk_gen1_inode" ]; do
[ "\$(date +%s)" -lt "\$deadline" ] || { echo "stub cp: the publisher never swapped the snapshot" >&2; exit 92; }
sleep 0.05
done
exit \$rc
fi
exec "$real_cp" "\$@"
EOF
chmod +x "$scratch/bin/cp"
rcIntlk=0
seed_with_stub "$INTLK" "$INTLK_BASE" "$root" "$TAG_INTLK" > "$scratch/logIntlk" 2>&1 || rcIntlk=$?
[ -e "$scratch/firedIntlk" ] \
|| fail "the stubbed cp never fired: the consumer never raced the publisher, so this scenario proves nothing"
ok "the consumer's clone raced a real, concurrent republish of its own source"
wait_for_file "$scratch/rcPubIntlk" "the publisher never finished"
[ "$(cat "$scratch/rcPubIntlk")" = "0" ] || { tail -40 "$scratch/logPubIntlk"; fail "publish-snapshot.sh exited non-zero"; }
[ "$rcIntlk" = "0" ] || { tail -40 "$scratch/logIntlk"; fail "the seed exited non-zero"; }
ok "both sides of the race completed"
# The property under test: not that the run succeeded, but that the publisher
# itself reports having found a live reader and waited on it. This is silent
# in the consumer's own log and in the run's exit status alike — only the
# publisher's log carries it.
grep -q "readers: waiting for 1 in-flight clone(s) of snapshot-${INTLK_BASE}" "$scratch/logPubIntlk" \
|| { tail -40 "$scratch/logPubIntlk"; fail "the publisher never reported waiting on the consumer's reader marker — the interlock did not observably engage"; }
ok "the publisher observably waited on the consumer's own reader marker before reclaiming the rotated snapshot generation"
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)
[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
ok "no staging, reader-marker or deferred-generation scratch left behind"
rm -f "$scratch/bin/cp"
echo echo
echo "seed-target-dir-selftest: ${pass_count} assertions passed" echo "seed-target-dir-selftest: ${pass_count} assertions passed"