From bd60b430e06c1d2f6762d6770288dda2b9c7ca96 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 13:24:53 -0500 Subject: [PATCH] test(seed): pin the two unguarded terms of the torn-clone condition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hardlink_clone_into's torn-clone detection is a four-term condition, and a mutation sweep found two of the four unpinned: removing either `[ "$cp_rc" -eq 0 ]` (issue #5) or `[ "$i_before" != missing ]` left all five suites green. They were unpinned for the same reason — they mask each other. A source that vanishes mid-clone reads as `missing` at both ends AND fails `cp -al`, so with both terms present either one catches it and neither is individually necessary. Isolating them needs a state each term alone can see: 8c cp reports failure over a tree that is in fact whole. Neither inference sees anything — 0 entries short, one unchanged inode — so the exit status is the only witness. Forced with the PATH stub 8a/8b already use, on the consumer's own top-level clone. 8d both identity reads fail while the copy succeeds. `_dir_inode` folds every stat failure into the string `missing`, so two failed reads compare equal TO EACH OTHER; without the sentinel term the tree is published on the strength of two errors. Stubs `stat` narrowly — only the `%i` reads of this clone's own source — because taking the source away would fail `cp -al` too and pin 8c's property over again. The sweep also found `_unshare_files`'s xargs status unpinned, which is the guard that stops a staging tree whose dep-info files still point at the SOURCE's inodes from being renamed into place — not a tear, so all four clone checks pass it, and exactly the silent cross-branch stale-reuse the scheme exists to prevent. Scenario 10 pins it by refusing the dep-info unshares and asserting the clone discards rather than publishes. assert_tear gains an `unreadable` identity expectation, and its `same` case now demands a READ identity rather than two equal strings — `missing` equals `missing`, which is the exact confusion 8d exists to pin. Each of the five pinning scenarios was run in isolation against each mutation; the result is a clean diagonal, so every scenario fails only for its own term. Closes #5 --- README.md | 2 +- scripts/seed-target-dir-selftest.sh | 238 ++++++++++++++++++++++++++-- 2 files changed, 228 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 45d826b..941eee6 100644 --- a/README.md +++ b/README.md @@ -361,7 +361,7 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler | suite | covers | |---|---| | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. | -| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed whose source is rotated — or silently loses a subtree — underneath its clone**: the two ways a hardlink clone tears | +| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published | | `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone | | `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` | | `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. | diff --git a/scripts/seed-target-dir-selftest.sh b/scripts/seed-target-dir-selftest.sh index be3c313..a7f026e 100755 --- a/scripts/seed-target-dir-selftest.sh +++ b/scripts/seed-target-dir-selftest.sh @@ -44,8 +44,17 @@ # never changes. Against the unguarded version this is a partial tree # renamed into place and reported as a success — 20,328 of 48,805 # entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot. -# Both scenarios force their interleaving rather than racing for it, and -# each asserts WHICH check caught the tear, so neither stays green if +# 8c. AND A COPY THAT SAYS SO ITSELF — the third witness, and the only one +# the tool volunteers: `cp -al` exiting non-zero over a tree that both +# other checks read as whole. Pins that a copy's own failure report is +# never overruled by two inferences that saw nothing. +# 8d. AND AN IDENTITY THAT COULD NOT BE READ AT ALL — the fourth. A failed +# identity read is reported as the string `missing`, so two of them +# compare equal to each other; without the term that rejects the +# sentinel, a clone whose source could not be identified at either end +# is published on the strength of two errors. +# All four scenarios force their interleaving rather than racing for it, +# and each asserts WHICH check caught the tear, so none stays green if # the check it exercises is removed. # 9. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct status # instead of renaming whatever it managed to produce into place, and the @@ -56,6 +65,13 @@ # (The publisher's half of the rotation race — deferring reclamation # while a reader is still in flight — lives in # publish-snapshot-selftest.sh, next to the swap it modifies.) +# 10. A TREE THAT CANNOT BE PRIVATELY OWNED IS DISCARDED — the other way a +# clone must refuse to publish, and the one that is not about tearing at +# all: if unsharing the mutable paths fails, the staging tree still +# aliases its source, so renaming it into place would wire two branches +# onto one set of fingerprints. That is the silent stale-reuse bug the +# whole scheme exists to prevent, so the failure has to abort the clone +# rather than be swallowed. set -euo pipefail script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$script_dir/cache-lib.sh" @@ -92,6 +108,7 @@ seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { t # that needs them, so either scenario can be run, moved or mutated alone. mkdir -p "$scratch/bin" real_cp=$(command -v cp) +real_stat=$(command -v stat) seed_with_stub() { PATH="$scratch/bin:$PATH" bash "$script_dir/seed-target-dir.sh" "$@"; } # Always succeeds and always prints a number: a directory that does not exist @@ -115,17 +132,20 @@ wait_for_file() { done } -# Reads the clone's own torn-read report back and asserts WHICH of its three -# checks fired: the copy's exit status, the entry count, or the source's -# identity. Scenarios 8a and 8b each force exactly one of the three, and a -# scenario that only asserted "some tear was reported" would stay green if -# the check it exercises were deleted and one of the others happened to fire -# in its place. +# Reads the clone's own torn-read report back and asserts WHICH of its four +# checks fired: the copy's exit status, the entry count, the source's identity, +# or that identity being unreadable at all. Scenarios 8a to 8d each force +# exactly one of the four, and a scenario that only asserted "some tear was +# reported" would stay green if the check it exercises were deleted and one of +# the others happened to fire in its place. # -# assert_tear +# assert_tear assert_tear() { local log="$1" want_rc="$2" want_count="$3" want_inode="$4" msg="$5" line - local re='cp rc=([0-9]+), ([0-9]+)/([0-9]+) entries, source inode ([0-9]+) -> ([0-9]+)' + # The identity fields are matched as `|missing` rather than as + # "anything up to the space", so a report whose inode field is neither is a + # loud parse failure instead of a silently-compared string. + local re='cp rc=([0-9]+), ([0-9]+)/([0-9]+) entries, source inode ([0-9]+|missing) -> ([0-9]+|missing)' # `|| line=""` rather than a bare assignment: no match makes grep exit 1, # which under `set -e` would abort the suite with no message at all — the # exact case this assertion exists to report. @@ -139,8 +159,14 @@ assert_tear() { *) fail "assert_tear: bad entry-count expectation '${want_count}'" ;; esac case "$want_inode" in - same) [ "${BASH_REMATCH[4]}" = "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to hold: ${line}" ;; + # `same` demands a READ identity that held, not merely two equal strings: + # two failed reads are both `missing` and would otherwise satisfy it, which + # is the exact confusion scenario 8d exists to pin. + same) [ "${BASH_REMATCH[4]}" != missing ] || fail "expected the source's identity to be readable: ${line}" + [ "${BASH_REMATCH[4]}" = "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to hold: ${line}" ;; differs) [ "${BASH_REMATCH[4]}" != "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to change: ${line}" ;; + unreadable) [ "${BASH_REMATCH[4]}" = missing ] && [ "${BASH_REMATCH[5]}" = missing ] \ + || fail "expected neither identity read to have resolved: ${line}" ;; *) fail "assert_tear: bad identity expectation '${want_inode}'" ;; esac ok "$msg (${line})" @@ -373,6 +399,134 @@ leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \ [ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}" ok "no staging or reader-marker scratch left behind" rm -f "$scratch/bin/cp" + +echo +echo "=== 8c: a copy that reports failure over a tree that looks whole ===" +# The third witness, and the only one the tool volunteers rather than leaving +# to be inferred: `cp -al` exiting non-zero. What this pins is that a copy's +# own failure report is never overruled by the other two checks agreeing that +# the tree looks intact. +# +# Both of those are inferences with blind spots. The entry count is +# `find | wc -l` over two trees, so a failure below a directory NEITHER walk +# could enumerate moves it not at all, and an entry lost against one gained +# cancels out. The inode comparison sees only a source replaced wholesale. A +# `cp -a` that links every entry and still fails — it could not preserve a +# directory's ownership, say — is invisible to both, and that is the shape +# forced here: the report reads 0 entries short and one unchanged inode, so +# the exit status is the only witness there is. +# +# Forced with the PATH stub the suite already uses for 8a and 8b, on the +# consumer's own top-level clone: the whole tree really is copied, and then +# the failure is reported, once. The retry finds nothing wrong, so the +# scenario also pins that the seed RECOVERS rather than merely refusing — +# a status that is honoured but not survivable would fail every job whose +# runner hiccuped once. +CPSTAT=$(cache_key feat/cp-status) +CPSTAT_BASE=$(cache_key release/2) +make_wide_tree "$root/snapshot-$CPSTAT_BASE" cpstatgen 4 +cpstat_entries=$(tree_entries "$root/snapshot-$CPSTAT_BASE") + +cat > "$scratch/bin/cp" < "$scratch/cp_reported_failure" + "$real_cp" "\$@" + echo "cp: failed to preserve ownership for '\${@: -1}': Operation not permitted" >&2 + exit 1 +fi +exec "$real_cp" "\$@" +EOF +chmod +x "$scratch/bin/cp" + +rcCpStat=0 +seed_with_stub "$CPSTAT" "$CPSTAT_BASE" "$root" jobCpStat > "$scratch/logCpStat" 2>&1 || rcCpStat=$? +[ -e "$scratch/cp_reported_failure" ] \ + || fail "the stubbed cp never fired: no copy reported failure, so this scenario proves nothing" +ok "a copy reported failure over a staging tree that was in fact complete" +[ "$rcCpStat" = "0" ] || { tail -40 "$scratch/logCpStat"; fail "the seed exited non-zero"; } +ok "the seed completed" +# `same` on both inferences is the whole point: neither of them saw anything, +# so deleting the exit-status check leaves nothing to report and this +# assertion is the one that goes red. +assert_tear "$scratch/logCpStat" 1 same same "the failure was caught by the copy's exit status alone" + +cpstat_dir="$root/target-$CPSTAT" +[ "$(tree_entries "$cpstat_dir")" -eq "$cpstat_entries" ] \ + || fail "the seeded tree is short: $(tree_entries "$cpstat_dir") entries against the source's ${cpstat_entries}" +ok "the seed re-cloned and holds every entry the source has (${cpstat_entries})" +assert_content "$cpstat_dir/debug/.fingerprint/x/dep-lib-x" cpstatgen "the seeded tree is the source's content" +leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print) +[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}" +ok "no staging or reader-marker scratch left behind" +rm -f "$scratch/bin/cp" + +echo +echo "=== 8d: an identity that could not be read at either end ===" +# The fourth check, `[ "$i_before" != missing ]`, and the one whose absence is +# hardest to see. `_dir_inode` folds every stat failure into the string +# `missing`, so two FAILED identity reads compare equal TO EACH OTHER: drop +# this check and a clone whose source could not be identified before or after +# the walk satisfies `i_before = i_after` on the strength of two errors, and +# is renamed into place having proven nothing about the tree it holds. +# +# It cannot be forced by taking the source away. A source that is really gone +# fails `cp -al` too, so the exit status would fire in this check's place and +# the scenario would pin 8c's property over again. What isolates it is the +# identity read failing while the copy SUCCEEDS — a transient stat error over +# a source that is otherwise perfectly readable — so the stub goes on `stat` +# rather than on the tree, and narrowly: only the `%i` reads of this clone's +# own source, which is the sole caller of `stat -c '%i'` on this path. The +# `%Y` reads the reader markers do are left alone. +# +# BOTH reads of the attempt have to fail. If only one did, the surviving one +# would differ from `missing` and the identity COMPARISON would become the +# witness instead — 8a's property, not this one. The assertion that the stub +# fired is therefore a count rather than a flag. +IDENT=$(cache_key feat/identity-unreadable) +IDENT_BASE=$(cache_key release/3) +make_wide_tree "$root/snapshot-$IDENT_BASE" identgen 4 +ident_src="$root/snapshot-$IDENT_BASE" +ident_entries=$(tree_entries "$ident_src") + +cat > "$scratch/bin/stat" </dev/null || echo 0)" -lt 2 ]; then + printf 'x' >> "$scratch/identity_reads_failed" + echo "stat: cannot statx '\$3': Input/output error" >&2 + exit 1 +fi +exec "$real_stat" "\$@" +EOF +chmod +x "$scratch/bin/stat" + +rcIdent=0 +seed_with_stub "$IDENT" "$IDENT_BASE" "$root" jobIdent > "$scratch/logIdent" 2>&1 || rcIdent=$? +ident_failures=$(wc -c < "$scratch/identity_reads_failed" 2>/dev/null || echo 0) +[ "$ident_failures" -eq 2 ] \ + || fail "expected both identity reads of one attempt to fail, got ${ident_failures} — a single failure would make the identity comparison the witness instead, which is scenario 8a's property" +ok "neither identity read of the first attempt resolved" +[ "$rcIdent" = "0" ] || { tail -40 "$scratch/logIdent"; fail "the seed exited non-zero"; } +ok "the seed completed" +assert_tear "$scratch/logIdent" 0 same unreadable "the unreadable identity was caught by the sentinel check alone" + +ident_dir="$root/target-$IDENT" +[ "$(tree_entries "$ident_dir")" -eq "$ident_entries" ] \ + || fail "the seeded tree is short: $(tree_entries "$ident_dir") entries against the source's ${ident_entries}" +ok "the seed re-cloned once the identity was readable again (${ident_entries} entries)" +assert_content "$ident_dir/debug/.fingerprint/x/dep-lib-x" identgen "the seeded tree is the source's content" +leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print) +[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}" +ok "no staging or reader-marker scratch left behind" +rm -f "$scratch/bin/stat" + echo echo "=== 9: a source that cannot be read fails loudly ===" rc=0 @@ -408,5 +562,67 @@ else assert_absent "$root/target-$VICTIM" "no target dir was left behind by the failed seed" fi +echo +echo "=== 10: a staging tree that cannot be privately owned is never published ===" +# unshare_mutable_paths turns the hardlinked copies of Cargo's mutable +# metadata back into private inodes, and its failure status is load-bearing in +# a way no other check covers: a staging tree whose dep-info files still point +# at the SOURCE's inodes is not torn — every entry is present and the source +# never moved — so all four torn-clone checks pass it. Publish it anyway and +# this branch's build rewrites the base branch's dep-info, which is the silent +# stale-artifact reuse documented at length in cache-lib.sh. The only thing +# standing between that tree and DST is the status being propagated. +# +# Forced with the same PATH stub shape as 8a to 8d, on the narrowest possible +# target: `_unshare_files` copies each shared file as `cp -p -- +# .unshare.`, so refusing exactly the `.d` copies leaves the directory +# unshares (`cp -a`) and the clone itself (`cp -al`) untouched, and the +# failure that reaches the clone is unambiguously this one. The dep-info file +# has to live outside .fingerprint/ and build/, or it would be inside a +# subtree already replaced wholesale and never reach the per-file pass. +ALIAS=$(cache_key feat/unshare-fails) +ALIAS_BASE=$(cache_key release/4) +make_tree "$root/snapshot-$ALIAS_BASE" aliasgen +alias_dep="$root/snapshot-$ALIAS_BASE/debug/deps/libx.d" +echo aliasgen > "$alias_dep" + +cat > "$scratch/bin/cp" < "$scratch/depinfo_unshare_refused" + echo "cp: cannot create regular file '\$4': No space left on device" >&2 + exit 1 +fi +exec "$real_cp" "\$@" +EOF +chmod +x "$scratch/bin/cp" + +rcAlias=0 +seed_with_stub "$ALIAS" "$ALIAS_BASE" "$root" jobAlias > "$scratch/logAlias" 2>&1 || rcAlias=$? +[ -e "$scratch/depinfo_unshare_refused" ] \ + || fail "the stubbed cp never fired: no unshare was refused, so this scenario proves nothing" +ok "a dep-info unshare was refused inside the clone" +[ "$rcAlias" != "0" ] || { tail -40 "$scratch/logAlias"; fail "the seed reported success over a staging tree that still aliased its source"; } +ok "the seed exits non-zero when the staging tree cannot be privately owned" +# Which failure aborted the clone is asserted, not assumed: without this the +# scenario would stay green if the clone had failed for any other reason — +# the same wrong-reason pass that let a mutation survive scenario 9 (issue #5). +grep -q 'failed to unshare dep-info files' "$scratch/logAlias" \ + || { tail -20 "$scratch/logAlias"; fail "the dep-info unshare failure was not the one reported"; } +ok "the refused unshare is what was reported" +grep -q 'could not privately own the mutable paths' "$scratch/logAlias" \ + || { tail -20 "$scratch/logAlias"; fail "the clone did not report why it refused to publish"; } +ok "the clone names aliasing as its reason for refusing" +assert_absent "$root/target-$ALIAS" "nothing was renamed into place" +[ "$(stat -c '%h' "$alias_dep")" = "1" ] \ + || fail "the source's dep-info file is still hardlinked from somewhere: $(stat -c '%h' "$alias_dep") links" +ok "the discarded staging tree took its hardlinks to the source with it" +leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print) +[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}" +ok "no staging or reader-marker scratch left behind" +rm -f "$scratch/bin/cp" + echo echo "seed-target-dir-selftest: ${pass_count} assertions passed"