test(hardlink): move the probe's INACTIVE answer off bash's error codes
CI / shellcheck + selftests (pull_request) Successful in 1m21s

Review finding on #13, and the last change to this function. A typo'd variable
name on a line that HAS its `|| exit 2` guard — `$CONTNET_B` for `$CONTENT_B`
— fails in EXPANSION, before the command runs, so neither the guard nor the
ERR trap ever sees it. Under `set -u` that exits 1, which was the code meaning
"measured, and the answer is mtime". Not a live defect: the probe references
only set variables today, and the 1 gitdan-ci reports is a genuine measurement.

The defect is that the answer codes and the error codes overlapped at all.
Three rounds on this function each found a narrower way for a shell-generated
1 to be read as a measurement — an unguarded command, then a guarded line
whose guard could not fire — and each was closed by narrowing the failure
surface, which is a game with no last move.

INACTIVE is now 3, and anything that is not 0 or 3 is NOT MEASURED. Bash
generates 1, 2, 126, 127 and 128+n for its own errors and never 3, so "not an
answer" is decided by a property of the shell rather than by an enumeration of
the ways a step can go wrong. A step added later without a guard, or with a
guard that cannot fire, lands on NOT MEASURED by construction.

The guards and the trap stay, with their job restated: reporting rather than
correctness. They put a failed step on 2 with both probe logs printed instead
of on some incidental status — nicer to debug, and the same destination either
way. The `set +e` bracket at the call site keeps its own note, since a `set -e`
that cannot fire is the kind of thing a reader assumes works.

The not-measured message now quotes the actual exit code, so the two ways to
reach it are distinguishable in a log without reading this file.

Red-proven with the reviewer's own mutation, A/B: with INACTIVE at 1 the typo
reports `mode: off — resolves freshness by mtime`, a false measurement; at 3 it
reports `mode: unmeasured — the probe exited 1`. All four paths re-verified —
unmodified 4 assertions; env stripped inside the probe, measured INACTIVE with
the mtime reason; a broken probe build, unmeasured via a guard; an unguarded
`false`, unmeasured via the trap.
This commit is contained in:
2026-08-26 14:40:27 -05:00
parent 5b46986cd7
commit e94c46f2ec
2 changed files with 44 additions and 28 deletions
+1 -1
View File
@@ -562,7 +562,7 @@ change here reaches all of them at once. That is what the gate is for.
| suite | covers | | suite | covers |
|---|---| |---|---|
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard | | `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly that actually resolves freshness by content for its last scenario**: the source's-next-build check reasons about content rather than mtime, so under mtime freshness it would assert a bug. Whether the toolchain does is settled by experiment on a throwaway crate, not by asking it — 1.100.0-nightly accepts `-Z checksum-freshness` and rebuilds on mtime anyway. The experiment reports **three** outcomes, not two: active, measured-inactive, and *not measured* when a probe build failed. The scenario is skipped for the last two alike, but a failure to measure is never reported as a measurement. The control also reports which mutation families the running Cargo exhibits — a note, not an assertion, since that set moves upstream. | | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly that actually resolves freshness by content for its last scenario**: the source's-next-build check reasons about content rather than mtime, so under mtime freshness it would assert a bug. Whether the toolchain does is settled by experiment on a throwaway crate, not by asking it — 1.100.0-nightly accepts `-Z checksum-freshness` and rebuilds on mtime anyway. The experiment reports **three** outcomes, not two: active, measured-inactive, and *not measured*. Its answer codes are `0` and `3`, deliberately clear of every status bash generates for its own errors — so nothing that goes wrong inside the probe, including an expansion failure no guard can catch, can be read as an answer. The scenario is skipped for the last two alike, but a failure to measure is never reported as a measurement. The control also reports which mutation families the running Cargo exhibits — a note, not an assertion, since that set moves upstream. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot | | `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone | | `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` | | `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
+43 -27
View File
@@ -121,43 +121,57 @@ CONTENT_B='pub fn f() -> u32 { 22222 } pub fn g() -> u32 { 7 }'
# #
# 0 content freshness measured ACTIVE — both builds ran, the backdated # 0 content freshness measured ACTIVE — both builds ran, the backdated
# rebuild recompiled # rebuild recompiled
# 1 measured INACTIVE — both builds ran, the backdated # 3 measured INACTIVE — both builds ran, the backdated
# rebuild reported Fresh # rebuild reported Fresh
# 2 NOT MEASURED — a probe step failed; nothing was # anything else NOT MEASURED — nothing was learned about the
# learned about the toolchain # toolchain
# #
# Two mechanisms, and BOTH are needed. Every step that could fail for a reason # THE ANSWER CODES ARE 0 AND 3, AND THE GAP IS THE MECHANISM. Bash produces 1
# other than the experiment's own outcome exits 2 explicitly; the subshell also # for an ordinary command failure, 2 for a usage error, 126/127 for a command
# arms `set -e` with an ERR trap that maps any unguarded failure onto 2, so a # it could not run, 128+n for a signal, and — this is the one that matters —
# step added later without a guard lands on "not measured" rather than on an # 1 for an unbound-variable or other EXPANSION failure, which happens before
# answer. # the command runs and is therefore invisible to a `||` guard and to an ERR
# trap alike. It never produces 3. So "not an answer code" is decided by a
# property of the shell rather than by an enumeration of the ways a step can
# go wrong, and a step added later without a guard, or with a guard that
# cannot fire, lands on NOT MEASURED by construction.
# #
# The `set +e` around the call site is what makes the second mechanism work, # That is the whole reason INACTIVE is not 1. It was, and three review rounds
# and it is not decoration. A command on the left of `||` — or in an `if` # on this function each found a narrower way for a shell-generated 1 to be read
# condition — runs with errexit suppressed, and that suppression propagates # as a measurement — an unguarded command, then a typo'd variable name on a
# into a subshell and is NOT undone by a `set -e` inside it (verified on bash # line that HAS its guard. Each was closed by narrowing the failure surface,
# 5.3: an unguarded `false` there falls through to `exit 0` and reports # which is a game with no last move. Moving the answer off the codes bash can
# ACTIVE). Calling with errexit disarmed at the site is the only form that # generate ends it instead: there is no longer a mutation that turns an error
# lets the subshell re-arm it. The ERR trap is then required on top, because a # into an answer, only mutations that turn an error into a different error.
# bare `set -e` abort exits with the FAILING COMMAND's status — `false` gives
# 1, which is precisely the value that means "measured, and the answer is
# mtime". Belt and braces here buys a wrong answer; belt, braces and a trap
# buys "not measured".
# #
# WHY 2 SKIPS RATHER THAN FAILS. The scenario it gates is the only thing in # The guards below stay, and so does the trap, but their job is now reporting
# rather than correctness: they make a failed step land on 2 with its logs
# printed instead of on some incidental status, which is nicer to debug and
# lands in the same place either way.
#
# One piece of that reporting layer is load-bearing and not obvious. A command
# on the left of `||` — or in an `if` condition — runs with errexit suppressed,
# and that suppression propagates into a subshell and is NOT undone by a
# `set -e` inside it (measured on bash 5.3: an unguarded `false` there falls
# through to `exit 0`). Calling with errexit disarmed at the site is the only
# form that lets the subshell re-arm it; hence the `set +e` bracket. The ERR
# trap is then required on top, because a bare `set -e` abort exits with the
# FAILING COMMAND's status, and `false` gives 1.
#
# WHY NOT-MEASURED SKIPS RATHER THAN FAILS. The scenario it gates is the only thing in
# this suite that depends on freshness mode; everything else still runs and # this suite that depends on freshness mode; everything else still runs and
# still catches real regressions. Failing instead would turn a statement about # still catches real regressions. Failing instead would turn a statement about
# one machine's toolchain into a red gate reading "the hardlink scheme is # one machine's toolchain into a red gate reading "the hardlink scheme is
# broken" across the three repos consuming this action — the same category # broken" across the three repos consuming this action — the same category
# error the three-state split exists to prevent, one level up. What would # error the three-state split exists to prevent, one level up. What would
# change the answer is 2 becoming the everyday CI outcome; it is not (gitdan-ci # change the answer is not-measured becoming the everyday CI outcome; it is not
# reports 1, by measurement). # (gitdan-ci reports a measured INACTIVE, by measurement).
CHECKSUM_MODE="off" CHECKSUM_MODE="off"
CHECKSUM_REASON="no nightly on PATH accepting -Z checksum-freshness" CHECKSUM_REASON="no nightly on PATH accepting -Z checksum-freshness"
CARGO_BIN=(cargo) CARGO_BIN=(cargo)
checksum_freshness_probe() { checksum_freshness_probe() {
local d="$scratch/freshness-probe" t="$scratch/freshness-probe-target" local d="$scratch/freshness-probe" t="$scratch/freshness-probe-target"
mkcrate "$d" || return 2 mkcrate "$d" || return 2 # 2 is simply "not 0 and not 3"; see the header
( (
set -e set -e
trap 'exit 2' ERR trap 'exit 2' ERR
@@ -167,7 +181,9 @@ checksum_freshness_probe() {
printf '%s\n' "$CONTENT_B" > src/lib.rs || exit 2 printf '%s\n' "$CONTENT_B" > src/lib.rs || exit 2
touch -d '@1000000000' src/lib.rs || exit 2 touch -d '@1000000000' src/lib.rs || exit 2
CARGO_TARGET_DIR="$t" cargo +nightly build -v > "$scratch/freshness-probe.log" 2>&1 || exit 2 CARGO_TARGET_DIR="$t" cargo +nightly build -v > "$scratch/freshness-probe.log" 2>&1 || exit 2
if grep -qE '^\s+Fresh probe' "$scratch/freshness-probe.log"; then exit 1; fi # 3, not 1: see the header. This is the only statement in the subshell that
# may report a measurement, and it is the only one that may exit 3.
if grep -qE '^\s+Fresh probe' "$scratch/freshness-probe.log"; then exit 3; fi
exit 0 exit 0
) )
} }
@@ -186,18 +202,18 @@ if cargo +nightly -Z checksum-freshness locate-project > /dev/null 2>&1; then
CHECKSUM_MODE="on" CHECKSUM_MODE="on"
CHECKSUM_REASON="" CHECKSUM_REASON=""
;; ;;
1) 3)
unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS
CHECKSUM_REASON="this nightly accepts -Z checksum-freshness but resolves freshness by mtime" CHECKSUM_REASON="this nightly accepts -Z checksum-freshness but resolves freshness by mtime"
;; ;;
*) *)
unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS
CHECKSUM_MODE="unmeasured" CHECKSUM_MODE="unmeasured"
CHECKSUM_REASON="a probe build failed, so this was NOT MEASURED — this toolchain may or may not resolve freshness by content" CHECKSUM_REASON="the probe exited ${probe_rc}, which is not one of its answer codes, so this was NOT MEASURED — this toolchain may or may not resolve freshness by content"
# Loud, because the cost is silently lost coverage on a machine that # Loud, because the cost is silently lost coverage on a machine that
# might have had it. The suite continues: everything else it asserts is # might have had it. The suite continues: everything else it asserts is
# independent of freshness mode. # independent of freshness mode.
echo "::warning::hardlink-clone-selftest: could not measure whether this toolchain resolves freshness by content — a probe build failed. This is a failure to measure, not a finding about Cargo." echo "::warning::hardlink-clone-selftest: could not measure whether this toolchain resolves freshness by content — the probe exited ${probe_rc}. This is a failure to measure, not a finding about Cargo."
tail -n 15 "$scratch/freshness-probe-warm.log" "$scratch/freshness-probe.log" 2>/dev/null | sed 's/^/ /' >&2 || true tail -n 15 "$scratch/freshness-probe-warm.log" "$scratch/freshness-probe.log" 2>/dev/null | sed 's/^/ /' >&2 || true
;; ;;
esac esac