README.md

@@ -361,7 +361,7 @@ bash scripts/selftest.sh --fast   # fixture-only suites, no compiler
 | suite | covers |
 |---|---|
 | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
-| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed whose source is rotated — or silently loses a subtree — underneath its clone**: the two ways a hardlink clone tears |
+| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published |
 | `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
 | `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
 | `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
@@ -381,13 +381,20 @@ key and asserts only what must be true whichever of them wins the rename.
 which places the interference inside the window rather than hoping it lands
 there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
 measurement publishes a reader marker strictly between its check and its
-unlink; `seed-target-dir-selftest.sh` scenarios 8a and 8b stub `cp`, so the
+unlink. `seed-target-dir-selftest.sh` uses the shape four times over, on two
-consumer's own clone is what rotates the snapshot underneath it, or what
+different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
-loses a subtree of its own source, strictly inside the identity window. 8a
+clone is what rotates the snapshot underneath it, loses a subtree of its own
-does start a second real process — the actual `publish-snapshot.sh` — but the
+source, or reports a failure over a tree that is in fact whole — each strictly
-stub is what fixes where its swap lands; the concurrency is incidental to the
+inside the window that clone's checks cover; scenario 10 stubs `cp` one level
-determinism. Every stub asserts that it fired, because a scenario whose
+down instead, refusing the per-file copies that unshare the mutable paths.
-interference silently did not happen passes for the wrong reason.
+Scenario 8d stubs `stat`, because the check it pins fires on an identity that
+could not be READ rather than on one that changed, and the only way to make
+that the sole witness is to fail the identity reads while the copy between
+them succeeds. 8a also starts a second real process — the actual
+`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
+concurrency is incidental to the determinism. Every stub asserts that it
+fired, because a scenario whose interference silently did not happen passes
+for the wrong reason.
 
 **A synthetic stand-in for the other side, where that artefact *is* the
 contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
@@ -404,9 +411,20 @@ does it any more.
 
 Where more than one guard could catch a fault, a scenario should assert
 *which* one did — otherwise deleting the guard under test leaves the suite
-green because a sibling fires in its place. Scenarios 8a and 8b of the seed
+green because a sibling fires in its place. Scenarios 8a to 8d and 10 of the
-suite do; scenario 9 of the same suite does not yet, which is why a mutation
+seed suite do, and each is reddened by exactly one mutation of the clone's
-survives it (issue #5).
+checks. Scenario 9 does not, and a mutation still survives it: with its source
+unreadable, `cp -al` leaves the staging directory at mode `000`, so deleting
+the exit-status check does not change the outcome — the unshare pass aborts
+the clone instead, and the assertion is satisfied down a path it was not
+written for.
+
+That is the standing hazard here, and it is not hypothetical. Two guards that
+can each catch the same fault mask each other, so **neither** is individually
+necessary and no fixture built around that fault can pin either one — which is
+how both `[ "$cp_rc" -eq 0 ]` and `[ "$i_before" != missing ]` sat unpinned
+(issue #5) while looking well covered. Isolating a check means constructing
+the state only it can see, not the state that trips several at once.
 
 The action YAML holds no logic beyond wiring; everything testable lives in
 `scripts/`. A composite action needs `shell: bash` on every `run:` step, and

scripts/seed-target-dir-selftest.sh

@@ -44,8 +44,19 @@
 #      never changes. Against the unguarded version this is a partial tree
 #      renamed into place and reported as a success — 20,328 of 48,805
 #      entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
-#      Both scenarios force their interleaving rather than racing for it, and
+#  8c. AND A COPY THAT SAYS SO ITSELF — the third witness, and the only one
-#      each asserts WHICH check caught the tear, so neither stays green if
+#      the tool volunteers: `cp -al` exiting non-zero over a tree that both
+#      other checks read as whole. Pins that a copy's own failure report is
+#      never overruled by two inferences that saw nothing.
+#  8d. AND AN IDENTITY THAT COULD NOT BE READ AT ALL — the fourth. A failed
+#      identity read is reported as the string `missing`, so two of them
+#      compare equal to each other; without the term that rejects the
+#      sentinel, a clone whose source could not be identified at either end
+#      is published on the strength of two errors. A regression guard on a
+#      defensive term rather than a reproduction of a reachable state — see
+#      the scenario's own comment.
+#      All four scenarios force their interleaving rather than racing for it,
+#      and each asserts WHICH check caught the tear, so none stays green if
 #      the check it exercises is removed.
 #   9. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct status
 #      instead of renaming whatever it managed to produce into place, and the
@@ -56,6 +67,13 @@
 #      (The publisher's half of the rotation race — deferring reclamation
 #      while a reader is still in flight — lives in
 #      publish-snapshot-selftest.sh, next to the swap it modifies.)
+#  10. A TREE THAT CANNOT BE PRIVATELY OWNED IS DISCARDED — the other way a
+#      clone must refuse to publish, and the one that is not about tearing at
+#      all: if unsharing the mutable paths fails, the staging tree still
+#      aliases its source, so renaming it into place would wire two branches
+#      onto one set of fingerprints. That is the silent stale-reuse bug the
+#      whole scheme exists to prevent, so the failure has to abort the clone
+#      rather than be swallowed.
 set -euo pipefail
 script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$script_dir/cache-lib.sh"
@@ -92,6 +110,7 @@ seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { t
 # that needs them, so either scenario can be run, moved or mutated alone.
 mkdir -p "$scratch/bin"
 real_cp=$(command -v cp)
+real_stat=$(command -v stat)
 seed_with_stub() { PATH="$scratch/bin:$PATH" bash "$script_dir/seed-target-dir.sh" "$@"; }
 
 # Always succeeds and always prints a number: a directory that does not exist
@@ -115,17 +134,20 @@ wait_for_file() {
   done
 }
 
-# Reads the clone's own torn-read report back and asserts WHICH of its three
+# Reads the clone's own torn-read report back and asserts WHICH of its four
-# checks fired: the copy's exit status, the entry count, or the source's
+# checks fired: the copy's exit status, the entry count, the source's identity,
-# identity. Scenarios 8a and 8b each force exactly one of the three, and a
+# or that identity being unreadable at all. Scenarios 8a to 8d each force
-# scenario that only asserted "some tear was reported" would stay green if
+# exactly one of the four, and a scenario that only asserted "some tear was
-# the check it exercises were deleted and one of the others happened to fire
+# reported" would stay green if the check it exercises were deleted and one of
-# in its place.
+# the others happened to fire in its place.
 #
-#   assert_tear <log> <cp-status> <same|short> <same|differs> <message>
+#   assert_tear <log> <cp-status> <same|short> <same|differs|unreadable> <msg>
 assert_tear() {
   local log="$1" want_rc="$2" want_count="$3" want_inode="$4" msg="$5" line
-  local re='cp rc=([0-9]+), ([0-9]+)/([0-9]+) entries, source inode ([0-9]+) -> ([0-9]+)'
+  # The identity fields are matched as `<digits>|missing` rather than as
+  # "anything up to the space", so a report whose inode field is neither is a
+  # loud parse failure instead of a silently-compared string.
+  local re='cp rc=([0-9]+), ([0-9]+)/([0-9]+) entries, source inode ([0-9]+|missing) -> ([0-9]+|missing)'
   # `|| line=""` rather than a bare assignment: no match makes grep exit 1,
   # which under `set -e` would abort the suite with no message at all — the
   # exact case this assertion exists to report.
@@ -139,8 +161,14 @@ assert_tear() {
     *) fail "assert_tear: bad entry-count expectation '${want_count}'" ;;
   esac
   case "$want_inode" in
-    same)    [ "${BASH_REMATCH[4]}" = "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to hold: ${line}" ;;
+    # `same` demands a READ identity that held, not merely two equal strings:
+    # two failed reads are both `missing` and would otherwise satisfy it, which
+    # is the exact confusion scenario 8d exists to pin.
+    same)    [ "${BASH_REMATCH[4]}" != missing ] || fail "expected the source's identity to be readable: ${line}"
+             [ "${BASH_REMATCH[4]}" = "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to hold: ${line}" ;;
     differs) [ "${BASH_REMATCH[4]}" != "${BASH_REMATCH[5]}" ] || fail "expected the source's identity to change: ${line}" ;;
+    unreadable) [ "${BASH_REMATCH[4]}" = missing ] && [ "${BASH_REMATCH[5]}" = missing ] \
+             || fail "expected neither identity read to have resolved: ${line}" ;;
     *) fail "assert_tear: bad identity expectation '${want_inode}'" ;;
   esac
   ok "$msg (${line})"
@@ -373,6 +401,143 @@ leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \
 [ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
 ok "no staging or reader-marker scratch left behind"
 rm -f "$scratch/bin/cp"
+
+echo
+echo "=== 8c: a copy that reports failure over a tree that looks whole ==="
+# The third witness, and the only one the tool volunteers rather than leaving
+# to be inferred: `cp -al` exiting non-zero. What this pins is that a copy's
+# own failure report is never overruled by the other two checks agreeing that
+# the tree looks intact.
+#
+# Both of those are inferences with blind spots. The entry count is
+# `find | wc -l` over two trees, so a failure below a directory NEITHER walk
+# could enumerate moves it not at all, and an entry lost against one gained
+# cancels out. The inode comparison sees only a source replaced wholesale. A
+# `cp -a` that links every entry and still fails — it could not preserve a
+# directory's ownership, say — is invisible to both, and that is the shape
+# forced here: the report reads 0 entries short and one unchanged inode, so
+# the exit status is the only witness there is.
+#
+# Forced with the PATH stub the suite already uses for 8a and 8b, on the
+# consumer's own top-level clone: the whole tree really is copied, and then
+# the failure is reported, once. The retry finds nothing wrong, so the
+# scenario also pins that the seed RECOVERS rather than merely refusing —
+# a status that is honoured but not survivable would fail every job whose
+# runner hiccuped once.
+CPSTAT=$(cache_key feat/cp-status)
+CPSTAT_BASE=$(cache_key release/2)
+make_wide_tree "$root/snapshot-$CPSTAT_BASE" cpstatgen 4
+cpstat_entries=$(tree_entries "$root/snapshot-$CPSTAT_BASE")
+
+cat > "$scratch/bin/cp" <<EOF
+#!/usr/bin/env bash
+# Fires once, and only on the consumer's own top-level hardlink clone. The
+# real copy runs to completion first: the point is a status nothing else can
+# see, so the staging tree it reports on must be whole.
+if [ "\${@: -1}" = "$root/.stage-jobCpStat" ] && [ ! -e "$scratch/cp_reported_failure" ]; then
+  : > "$scratch/cp_reported_failure"
+  "$real_cp" "\$@"
+  echo "cp: failed to preserve ownership for '\${@: -1}': Operation not permitted" >&2
+  exit 1
+fi
+exec "$real_cp" "\$@"
+EOF
+chmod +x "$scratch/bin/cp"
+
+rcCpStat=0
+seed_with_stub "$CPSTAT" "$CPSTAT_BASE" "$root" jobCpStat > "$scratch/logCpStat" 2>&1 || rcCpStat=$?
+[ -e "$scratch/cp_reported_failure" ] \
+  || fail "the stubbed cp never fired: no copy reported failure, so this scenario proves nothing"
+ok "a copy reported failure over a staging tree that was in fact complete"
+[ "$rcCpStat" = "0" ] || { tail -40 "$scratch/logCpStat"; fail "the seed exited non-zero"; }
+ok "the seed completed"
+# `same` on both inferences is the whole point: neither of them saw anything,
+# so deleting the exit-status check leaves nothing to report and this
+# assertion is the one that goes red.
+assert_tear "$scratch/logCpStat" 1 same same "the failure was caught by the copy's exit status alone"
+
+cpstat_dir="$root/target-$CPSTAT"
+[ "$(tree_entries "$cpstat_dir")" -eq "$cpstat_entries" ] \
+  || fail "the seeded tree is short: $(tree_entries "$cpstat_dir") entries against the source's ${cpstat_entries}"
+ok "the seed re-cloned and holds every entry the source has (${cpstat_entries})"
+assert_content "$cpstat_dir/debug/.fingerprint/x/dep-lib-x" cpstatgen "the seeded tree is the source's content"
+leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print)
+[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
+ok "no staging or reader-marker scratch left behind"
+rm -f "$scratch/bin/cp"
+
+echo
+echo "=== 8d: an identity that could not be read at either end ==="
+# The fourth check, `[ "$i_before" != missing ]`, and the one whose absence is
+# hardest to see. `_dir_inode` folds every stat failure into the string
+# `missing`, so two FAILED identity reads compare equal TO EACH OTHER: drop
+# this check and a clone whose source could not be identified before or after
+# the walk satisfies `i_before = i_after` on the strength of two errors, and
+# is renamed into place having proven nothing about the tree it holds.
+#
+# It cannot be forced by taking the source away. A source that is really gone
+# fails `cp -al` too, so the exit status would fire in this check's place and
+# the scenario would pin 8c's property over again. What isolates it is the
+# identity read failing while the copy SUCCEEDS — a transient stat error over
+# a source that is otherwise perfectly readable — so the stub goes on `stat`
+# rather than on the tree, and narrowly: only the `%i` reads of this clone's
+# own source, which is the sole caller of `stat -c '%i'` on this path. The
+# `%Y` reads the reader markers do are left alone.
+#
+# BOTH reads of the attempt have to fail. If only one did, the surviving one
+# would differ from `missing` and the identity COMPARISON would become the
+# witness instead — 8a's property, not this one. The assertion that the stub
+# fired is therefore a count rather than a flag.
+#
+# So be clear about what this scenario is. It is NOT a reproduction of a state
+# a CI job reaches: every route to it is closed off — a rotation hands the
+# witness to 8a, and a source that is genuinely gone fails `cp -al` and hands
+# it to 8c. It is a REGRESSION GUARD ON A DEFENSIVE TERM, and the thing it
+# defends against is a sentinel comparing equal to itself, which is a property
+# of the code rather than of the filesystem. That is worth pinning precisely
+# because nothing else can reach it: a term no fixture exercises is the one a
+# refactor drops without argument.
+IDENT=$(cache_key feat/identity-unreadable)
+IDENT_BASE=$(cache_key release/3)
+make_wide_tree "$root/snapshot-$IDENT_BASE" identgen 4
+ident_src="$root/snapshot-$IDENT_BASE"
+ident_entries=$(tree_entries "$ident_src")
+
+cat > "$scratch/bin/stat" <<EOF
+#!/usr/bin/env bash
+# Fails the first two identity reads of this clone's source — the pair that
+# opens and closes one attempt's identity window — and passes everything else
+# through, including the reader markers' own \`%Y\` reads.
+if [ "\$1" = "-c" ] && [ "\$2" = "%i" ] && [ "\$3" = "$ident_src" ] \
+   && [ "\$(wc -c < "$scratch/identity_reads_failed" 2>/dev/null || echo 0)" -lt 2 ]; then
+  printf 'x' >> "$scratch/identity_reads_failed"
+  echo "stat: cannot statx '\$3': Input/output error" >&2
+  exit 1
+fi
+exec "$real_stat" "\$@"
+EOF
+chmod +x "$scratch/bin/stat"
+
+rcIdent=0
+seed_with_stub "$IDENT" "$IDENT_BASE" "$root" jobIdent > "$scratch/logIdent" 2>&1 || rcIdent=$?
+ident_failures=$(wc -c < "$scratch/identity_reads_failed" 2>/dev/null || echo 0)
+[ "$ident_failures" -eq 2 ] \
+  || fail "expected both identity reads of one attempt to fail, got ${ident_failures} — a single failure would make the identity comparison the witness instead, which is scenario 8a's property"
+ok "neither identity read of the first attempt resolved"
+[ "$rcIdent" = "0" ] || { tail -40 "$scratch/logIdent"; fail "the seed exited non-zero"; }
+ok "the seed completed"
+assert_tear "$scratch/logIdent" 0 same unreadable "the unreadable identity was caught by the sentinel check alone"
+
+ident_dir="$root/target-$IDENT"
+[ "$(tree_entries "$ident_dir")" -eq "$ident_entries" ] \
+  || fail "the seeded tree is short: $(tree_entries "$ident_dir") entries against the source's ${ident_entries}"
+ok "the seed re-cloned once the identity was readable again (${ident_entries} entries)"
+assert_content "$ident_dir/debug/.fingerprint/x/dep-lib-x" identgen "the seeded tree is the source's content"
+leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print)
+[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
+ok "no staging or reader-marker scratch left behind"
+rm -f "$scratch/bin/stat"
+
 echo
 echo "=== 9: a source that cannot be read fails loudly ==="
 rc=0
@@ -408,5 +573,67 @@ else
   assert_absent "$root/target-$VICTIM" "no target dir was left behind by the failed seed"
 fi
 
+echo
+echo "=== 10: a staging tree that cannot be privately owned is never published ==="
+# unshare_mutable_paths turns the hardlinked copies of Cargo's mutable
+# metadata back into private inodes, and its failure status is load-bearing in
+# a way no other check covers: a staging tree whose dep-info files still point
+# at the SOURCE's inodes is not torn — every entry is present and the source
+# never moved — so all four torn-clone checks pass it. Publish it anyway and
+# this branch's build rewrites the base branch's dep-info, which is the silent
+# stale-artifact reuse documented at length in cache-lib.sh. The only thing
+# standing between that tree and DST is the status being propagated.
+#
+# Forced with the same PATH stub shape as 8a to 8d, on the narrowest possible
+# target: `_unshare_files` copies each shared file as `cp -p -- <f>
+# <f>.unshare.<pid>`, so refusing exactly the `.d` copies leaves the directory
+# unshares (`cp -a`) and the clone itself (`cp -al`) untouched, and the
+# failure that reaches the clone is unambiguously this one. The dep-info file
+# has to live outside .fingerprint/ and build/, or it would be inside a
+# subtree already replaced wholesale and never reach the per-file pass.
+ALIAS=$(cache_key feat/unshare-fails)
+ALIAS_BASE=$(cache_key release/4)
+make_tree "$root/snapshot-$ALIAS_BASE" aliasgen
+alias_dep="$root/snapshot-$ALIAS_BASE/debug/deps/libx.d"
+echo aliasgen > "$alias_dep"
+
+cat > "$scratch/bin/cp" <<EOF
+#!/usr/bin/env bash
+# Refuses the dep-info unshares and nothing else. Modelled on the failure that
+# actually happens here — the cache volume filling up mid-clone.
+if [ "\$1" = "-p" ] && [ "\$2" = "--" ] && [ "\${3%.d}" != "\$3" ]; then
+  : > "$scratch/depinfo_unshare_refused"
+  echo "cp: cannot create regular file '\$4': No space left on device" >&2
+  exit 1
+fi
+exec "$real_cp" "\$@"
+EOF
+chmod +x "$scratch/bin/cp"
+
+rcAlias=0
+seed_with_stub "$ALIAS" "$ALIAS_BASE" "$root" jobAlias > "$scratch/logAlias" 2>&1 || rcAlias=$?
+[ -e "$scratch/depinfo_unshare_refused" ] \
+  || fail "the stubbed cp never fired: no unshare was refused, so this scenario proves nothing"
+ok "a dep-info unshare was refused inside the clone"
+[ "$rcAlias" != "0" ] || { tail -40 "$scratch/logAlias"; fail "the seed reported success over a staging tree that still aliased its source"; }
+ok "the seed exits non-zero when the staging tree cannot be privately owned"
+# Which failure aborted the clone is asserted, not assumed: without this the
+# scenario would stay green if the clone had failed for any other reason —
+# the same wrong-reason pass that let a mutation survive scenario 9 (issue #5).
+grep -q 'failed to unshare dep-info files' "$scratch/logAlias" \
+  || { tail -20 "$scratch/logAlias"; fail "the dep-info unshare failure was not the one reported"; }
+ok "the refused unshare is what was reported"
+grep -q 'could not privately own the mutable paths' "$scratch/logAlias" \
+  || { tail -20 "$scratch/logAlias"; fail "the clone did not report why it refused to publish"; }
+ok "the clone names aliasing as its reason for refusing"
+assert_absent "$root/target-$ALIAS" "nothing was renamed into place"
+[ "$(stat -c '%h' "$alias_dep")" = "1" ] \
+  || fail "the source's dep-info file is still hardlinked from somewhere: $(stat -c '%h' "$alias_dep") links"
+ok "the discarded staging tree took its hardlinks to the source with it"
+leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \) -print)
+[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
+ok "no staging or reader-marker scratch left behind"
+rm -f "$scratch/bin/cp"
+
 echo
 echo "seed-target-dir-selftest: ${pass_count} assertions passed"