fix(cache): separate build directories for same-ref jobs, and a CI gate #13

Merged

daniel merged 9 commits from fix/lock-contention into main 2026-08-26 21:38:35 +00:00

Conversation 4 Commits 9 Files Changed 11

+857 -47

9 Commits

Author	SHA1	Message	Date
claude	e94c46f2ec	test(hardlink): move the probe's INACTIVE answer off bash's error codes ... Review finding on #13, and the last change to this function. A typo'd variable name on a line that HAS its `|| exit 2` guard — `$CONTNET_B` for `$CONTENT_B` — fails in EXPANSION, before the command runs, so neither the guard nor the ERR trap ever sees it. Under `set -u` that exits 1, which was the code meaning "measured, and the answer is mtime". Not a live defect: the probe references only set variables today, and the 1 gitdan-ci reports is a genuine measurement. The defect is that the answer codes and the error codes overlapped at all. Three rounds on this function each found a narrower way for a shell-generated 1 to be read as a measurement — an unguarded command, then a guarded line whose guard could not fire — and each was closed by narrowing the failure surface, which is a game with no last move. INACTIVE is now 3, and anything that is not 0 or 3 is NOT MEASURED. Bash generates 1, 2, 126, 127 and 128+n for its own errors and never 3, so "not an answer" is decided by a property of the shell rather than by an enumeration of the ways a step can go wrong. A step added later without a guard, or with a guard that cannot fire, lands on NOT MEASURED by construction. The guards and the trap stay, with their job restated: reporting rather than correctness. They put a failed step on 2 with both probe logs printed instead of on some incidental status — nicer to debug, and the same destination either way. The `set +e` bracket at the call site keeps its own note, since a `set -e` that cannot fire is the kind of thing a reader assumes works. The not-measured message now quotes the actual exit code, so the two ways to reach it are distinguishable in a log without reading this file. Red-proven with the reviewer's own mutation, A/B: with INACTIVE at 1 the typo reports `mode: off — resolves freshness by mtime`, a false measurement; at 3 it reports `mode: unmeasured — the probe exited 1`. All four paths re-verified — unmodified 4 assertions; env stripped inside the probe, measured INACTIVE with the mtime reason; a broken probe build, unmeasured via a guard; an unguarded `false`, unmeasured via the trap.	2026-08-26 14:40:27 -05:00
claude	5b46986cd7	test(hardlink): make the errexit invariant real, and stop gpg deciding a gate ... Two review findings on #13. THE STATED INVARIANT WAS NOT IN FORCE. The header claimed a `set -e` abort could never be mistaken for the `1` that means "measured, and the answer is mtime". It could not fire at all: `checksum_freshness_probe || probe_rc=$?` runs the left side with errexit suppressed, and that suppression propagates into the subshell. An unguarded step there fell through to `exit 0` and reported ACTIVE — a fourth outcome the header said was impossible. The guard audit was complete, so nothing was broken; the wrong mechanism had the credit, in a comment inviting the next editor to add an unguarded step and rely on it. The suggested fix — `set -e` as the first line of the subshell — does not work, and measured on bash 5.3 it makes things worse rather than not-better: set -e inside a subshell called via `||` or `if` still falls through: rc=0 set -e inside, called with errexit off at the site aborts, but with the FAILING COMMAND's status — `false` gives 1, which is exactly the value that means "mtime" So both halves are needed and neither is decoration: `set +e` around the call site, so the subshell can arm its own errexit at all, and `trap 'exit 2' ERR` inside it, so an abort lands on "not measured" instead of on an answer. The explicit `|| exit 2` guards stay as the first line of defence. All of that is now written down where the false claim was. Red-proven with one unguarded `false` between the `touch` and the second build, on a box where freshness is live: without the fix, mode `on` and 4 assertions — the reviewer's fourth outcome, reproduced; with it, mode `unmeasured`, the warning, 3 assertions; unmodified, mode `on` and 4. Recorded in the same block, since the next person to hit a `2` will ask: why it skips rather than fails. The gated scenario is the only thing here that depends on freshness mode, and failing would turn a statement about one machine into a red gate reading "the hardlink scheme is broken" across three consuming repos. What would change the answer is `2` becoming the everyday CI outcome, and run 2583 measures that it is not — gitdan-ci reports `1`. GPG. The two suites that commit in scratch repos inherited the developer's GLOBAL `commit.gpgsign`, so whether this gate passes could depend on their gpg agent — seen as a red `restore-mtimes-selftest.sh` caused by a full disk breaking gpg, in a suite with nothing to say about either. Pinned off locally, in the throwaway repos only: `git config commit.gpgsign false` beside the identity the scratch repo already sets, and `-c commit.gpgsign=false` on prune-cache-selftest's three commits, matching its existing `-c` style. Red-proven under a GIT_CONFIG_GLOBAL with gpgsign on and a nonexistent gpg program: without it `fatal: failed to write commit object`, with it both suites pass.	2026-08-26 14:28:22 -05:00
claude	5a90d5c829	test(hardlink): let the freshness probe report that it could not measure ... Review finding on #13. `checksum_freshness_active()` returned non-zero for any reason — including its own `cargo build` failing — and the caller's `else` branch then announced "this nightly accepts -Z checksum-freshness but still resolves freshness by mtime" regardless. On a machine where content freshness IS live, breaking only the probe's build produced that sentence, which is false, and silently dropped the scenario, which is lost coverage. Reachable, not theoretical: it needs a half-installed toolchain, which is exactly the state a probe exists to notice. The whole point of the preceding commit was to settle this by experiment rather than by asking, and an experiment that cannot tell a negative result from a failed measurement is not settling it. "This toolchain resolves freshness by mtime" is a statement about Cargo; "a probe build failed" is a statement about this machine. They are not interchangeable. Three outcomes now, carried in an exit code: 0 measured ACTIVE both builds ran, the backdated rebuild recompiled 1 measured INACTIVE both builds ran, the backdated rebuild was Fresh 2 NOT MEASURED a probe step failed; nothing was learned Every step that could fail for a reason other than the experiment's own outcome exits 2 explicitly, so a `set -e` abort cannot be mistaken for the 1 that means "measured, and the answer is mtime". Outcome 2 emits a ::warning:: saying in those words that this is a failure to measure and not a finding about Cargo, and prints the tail of both probe logs. The scenario still skips for 1 and 2 alike — everything else the suite asserts is independent of freshness mode — but the reason is now carried to the skip line, so the two are distinguishable at a glance. Verified all three states on a machine where freshness is live: unmodified, mode `on` and 4 assertions; probe build broken with a bogus flag, mode `unmeasured` with the warning and 3 assertions, and no claim about mtime; CARGO_UNSTABLE_CHECKSUM_FRESHNESS stripped inside the probe — CI's condition — mode `off` naming mtime, and 3 assertions.	2026-08-26 13:34:42 -05:00
claude	eb3f0bd09b	docs(ci): say what the nightly step actually buys today ... The step's comment claimed nightly was not optional. CI's own first green run falsified that: 1.100.0-nightly accepts -Z checksum-freshness and resolves freshness by mtime regardless, so the suite's probe reports it and skips the scenario either way. The step stays — twenty seconds, and the coverage returns by itself the day upstream restores the behaviour — but the comment now says that rather than the opposite. The open question about what upstream actually did is daniel/gitdan#62.	2026-08-26 12:57:15 -05:00
claude	6a80423bd0	test(hardlink): settle checksum freshness by experiment, not by asking ... Third toolchain probe in this branch, and the first one that asks the question the suite actually depends on. The two before it each let the suite assert a property the toolchain did not have: cargo +nightly -V answers "did a proxy called with +nightly exit 0". `-V` short-circuits before `-Z` is parsed at all. cargo +nightly -Z checksum-freshness answers "is the flag still accepted". locate-project 1.100.0-nightly (2026-08-25) accepts it and resolves freshness by mtime anyway, which is how CI reached the final scenario and failed there. The final scenario depends on exactly one property: that changed content with an OLDER mtime rebuilds. Under mtime freshness the correct answer is Fresh, so under mtime freshness that scenario asserts a bug — which is what CI reported. So the probe performs that experiment, on its own crate and its own target dir, with no clone anywhere near it. That separation is also what keeps it a control rather than a restatement: the probe establishes that the toolchain rebuilds on content, the scenario establishes that a hardlink clone did not take that away. Verified both ways locally: with a real content-freshness nightly, mode on and 4 assertions; with the env var stripped inside the probe — the runner's condition, faithfully — the probe says so by name, mode goes off, the scenario is skipped loudly and the remaining 3 assertions pass.	2026-08-26 12:54:24 -05:00
claude	5e8e773f78	test(hardlink): report the mutation families, don't assert one of them ... CI's nightly is 1.100.0-nightly (2026-08-25); the machine this suite was written on had 1.96.0-nightly (2026-02-24). On the newer one the control's `cp -al` clone mutates only the build/ and *.d families — upstream appears to have stopped rewriting `.fingerprint/*/dep-*` in place under checksum freshness — so the suite went red on the *absence* of a hazard. That is the wrong shape for a gate. The control's job is to prove the hazard exists at all, which a non-empty mutated set already does; naming one family as mandatory makes the suite red whenever upstream stops doing something we never wanted it to do, and red in the CONTROL, where a failure reads as "the hazard is gone" rather than "upstream changed". It is now a note either way. Nothing is given up. The fix scenario asserts the source is byte-identical after a full rebuild in the clone, which covers every family the running Cargo has, named or not — and the checksum-freshness scenario after it tests the stale-reuse hazard directly. The dep-* line only ever documented which family was in play. `unshare_mutable_paths` keeps unsharing dep-* regardless, and its measurement block now records both observations with their versions: 22 MB of a 6.9 GB tree against a failure mode that is a wrong answer rather than a slow one.	2026-08-26 12:50:42 -05:00
claude	3f97d3d1e7	fix(selftest): make the two compiler-backed suites survive a CI runner ... The new gate's first run went red on both suites that drive a real Cargo. Neither failure was a defect in what they test; both were assumptions about the machine, which had only ever been a dev box with a nightly installed and no colour forcing. Fixed here rather than waived — turning a gate on is what obliges fixing what it finds. COLOUR. Every assertion in restore-mtimes-selftest.sh, and one in hardlink-clone-selftest.sh, reads cargo's own words out of a build log (`Compiling libdep`, `Fresh probe`). gitdan-ci's runner image forces colour, so cargo wrote `Compiling\e[0m libdep` into the log and `grep -q "Compiling libdep"` stopped matching. The suite then reported the opposite of what happened — the failure printed the log, and the log plainly said `Compiling libdep`. Both suites now pin CARGO_TERM_COLOR=never, which is the format their assertions are written against. Red-proven: with the export removed and CARGO_TERM_COLOR=always, restore-mtimes-selftest reproduces the CI failure verbatim ("expected to find: Compiling libdep"); with it, 14/14 pass under the same forced colour. THE NIGHTLY PROBE ASKED THE WRONG QUESTION. `cargo +nightly -V` answers "did a cargo proxy called with +nightly exit 0", which is not "will this build have checksum freshness": `-V` short-circuits before `-Z` is validated at all. So the probe said "on" on a runner where the flag was not in effect, and the suite asserted the checksum-freshness mutation family against a build that never had it — failing in the CONTROL, where a failure reads as "the hazard is gone" rather than "the toolchain is wrong". It now probes the capability: `cargo +nightly -Z checksum-freshness locate-project`, the narrowest command that actually parses the flag. It rejects the stable channel and an unknown flag name alike, needs no network and builds nothing. Red-proven: the old channel probe, pointed at a cargo without the flag, reproduces the CI failure exactly. AND THE OFF PATH DID NOT WORK EITHER. The suite's header claimed that without checksum freshness "the test still covers the build/ and *.d families". It did not: the final scenario backdates the source to 2001 and asserts the rebuild is not Fresh, which is checksum-freshness-only reasoning. Under Cargo's ordinary mtime freshness a 2001 source IS older than the artifact and Fresh is the correct answer, so the scenario asserted a bug. It is now gated on the mode and skipped loudly, like the control's dep-* assertion already was: 5 assertions with a nightly, 3 without. CI installs a nightly as well as stable (nightly first, so stable stays the default) — that hazard is the one this whole scheme exists to close, and a CI that skips it is checking the cheap half.	2026-08-26 12:46:03 -05:00
claude	fb7a788c90	feat(cache): give same-ref jobs separate build directories via cache-lineage ... A cache key names a REF. What a target directory holds is the product of a ref and a build configuration, and emowheel builds the same ref twice on every push — once for the host, once for wasm32, in two jobs that start together. Keyed on the ref alone, `cargo-cache@v1` handed both the same CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second job sat on "Blocking waiting for file lock on build directory" for the length of the first while holding a runner capacity slot, so a third repo's queued job waited behind a job doing nothing. `cache-lineage` is that second dimension. It names ONE directory level under the cache root: <cache-root>/target-<key> no lineage (unchanged) <cache-root>/<lineage>/target-<key> a lineage Nesting, not a suffix on the key, and that is the whole design decision. `prune-cache.sh`'s liveness pass classifies a directory by recomputing `target-<cache_key(branch)>` for every branch on origin and evicting whatever does not match — a `target-<key>-wasm32` matches nothing, so it would be classified dead and evicted unconditionally on every run. daniel/gitdan's host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls out of that too, so those caches would never be reclaim candidates and a whole lineage would go missing from the shared disk budget. Nesting leaves both matchers reading exactly the names they already read, one level down — which is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own suite pins the depth-2 case). Every interacting part, checked rather than assumed: - SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in a lineage layers over THAT lineage's base snapshot. Asserted. - PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root. The publish action now takes the root from the `CARGO_CACHE_ROOT` the consume step exported, and CHECKS its own inputs against it — a publish step left at the default while its consume step nested would otherwise republish a different lineage's live target dir over that lineage's snapshot, on every push, silently. `mode: release-lock` is exempt: it releases a lock on `$CARGO_TARGET_DIR` and never touches a root. - WATERMARK: per target dir, so it follows the lineage. Unchanged. - PRUNE and LIVENESS: scoped to the root they are given, so a pass in one lineage neither evicts nor sees a sibling's caches, or the flat layout's. Liveness keeps resolving real branch names, which is what a key suffix would have broken. - ci_cache_reclaim: verified by dry-run against a fixture in this layout — all six nested and flat dirs collected as candidates, protection resolved correctly on the nested ones, and a `.stage-` stranded inside the lineage found by the leftover sweep. Refused lineage names are refused at resolve time, each rejection naming the reader that imposes it: a path separator (the arbiter's depth budget), a Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix (the leftover-naming contract). None of these fails visibly on its own — each produces a working directory that some pass silently stops seeing. Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna and emowheel's `ci` job keep the exact directories they have on the volume. New suite `cache-root-selftest.sh` (19 assertions), red-proven against three deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled validator, and a `verify` that never rejects a mismatch.	2026-08-26 12:32:21 -05:00
claude	f76789358d	ci(gitea): gate scripts/ with shellcheck and the selftests ... This repo had eight scripts, six selftest suites and nothing that ran any of them. It is a composite action consumed at `@v1` — a moving tag — by three repos' CI, so a bad edit here reaches zemyna, emowheel and lublub at once and is discovered by whichever of them builds next. One job: `shellcheck -x --source-path=scripts scripts/*.sh`, then `bash scripts/selftest.sh`. `-x` follows the `. cache-lib.sh` every script sources, which is where most of the logic being checked lives; without it shellcheck reports SC1091 and analyses each file with a hole in it. The full suite, not `--fast`: `hardlink-clone-selftest.sh` and `restore-mtimes-selftest.sh` are the two that drive a real Cargo rather than a fixture, and selftest.sh's own header says `--fast` is for iterating, not for signing off a change. So the job installs a stable toolchain. The scratch workspaces they build use path dependencies only, so nothing reaches crates.io, and the job references no credentials at all. Modelled on daniel/gitdan's workflow, including the two clauses of the draft-skip guard (`github.event_name != 'pull_request' || !github.event.pull_request.draft` — the first is what stops the second from also skipping pushes to `main`, where there is no `pull_request` context) and the per-commit concurrency group for `push`. No `container.volumes:` entry, deliberately: the suites want a cold target dir every run, since "did this rebuild?" is exactly what restore-mtimes-selftest asks. This job takes no share of the shared CI cache disk budget. Turning the gate on surfaced three pre-existing findings, all fixed here rather than suppressed or waived: - `restore-mtimes-selftest.sh` SC2038: `find | xargs touch` -> `-print0 | xargs -0`. - `hardlink-clone-selftest.sh` SC2295: `${f#$base_fix/}` -> `${f#"$base_fix"/}`. - `cache-lib.sh` SC2016: a per-line disable with the rationale. The single quotes are load-bearing — the program is for the inner shell, where `$f` is its loop variable, `$rc` its accumulator and `$$` its pid — so this is the documented-false-positive case, not a stopgap.	2026-08-26 12:31:50 -05:00