fix(cache): separate build directories for same-ref jobs, and a CI gate #13
@@ -66,6 +66,18 @@ jobs:
|
|||||||
#
|
#
|
||||||
# The scratch workspaces use path dependencies only, so nothing here
|
# The scratch workspaces use path dependencies only, so nothing here
|
||||||
# reaches crates.io.
|
# reaches crates.io.
|
||||||
|
# Nightly first, stable second, so stable ends up the default and
|
||||||
|
# nightly is only reachable through an explicit `+nightly`.
|
||||||
|
#
|
||||||
|
# Nightly is not optional here. `hardlink-clone-selftest.sh` gates its
|
||||||
|
# two strongest assertions on `-Z checksum-freshness` — the mode where
|
||||||
|
# Cargo's dep-info file carries per-source checksums and is rewritten in
|
||||||
|
# place, which is the mutation that turns a hardlink clone into SILENT
|
||||||
|
# stale-artifact reuse rather than a slow build. Without a nightly the
|
||||||
|
# suite still runs, and skips exactly the hazard this whole scheme exists
|
||||||
|
# to close.
|
||||||
|
- name: Install Rust nightly
|
||||||
|
uses: dtolnay/rust-toolchain@nightly
|
||||||
- name: Install Rust toolchain
|
- name: Install Rust toolchain
|
||||||
uses: dtolnay/rust-toolchain@stable
|
uses: dtolnay/rust-toolchain@stable
|
||||||
|
|
||||||
|
|||||||
@@ -544,7 +544,10 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
|||||||
|
|
||||||
Both run in CI — `.gitea/workflows/ci.yaml`, one job, on pushes to `main` and
|
Both run in CI — `.gitea/workflows/ci.yaml`, one job, on pushes to `main` and
|
||||||
on PRs that were non-draft when the run was created. It installs shellcheck
|
on PRs that were non-draft when the run was created. It installs shellcheck
|
||||||
and a stable Rust toolchain and references no credentials; the scratch
|
and both a stable and a nightly Rust toolchain (nightly for
|
||||||
|
`-Z checksum-freshness`, without which `hardlink-clone-selftest.sh` skips the
|
||||||
|
two assertions that cover the silent-stale-reuse hazard) and references no
|
||||||
|
credentials; the scratch
|
||||||
workspaces the compiler-backed suites build use path dependencies only, so
|
workspaces the compiler-backed suites build use path dependencies only, so
|
||||||
nothing reaches crates.io. It runs the full suite rather than `--fast`,
|
nothing reaches crates.io. It runs the full suite rather than `--fast`,
|
||||||
because the two compiler-backed suites are the ones that check this scheme
|
because the two compiler-backed suites are the ones that check this scheme
|
||||||
@@ -559,7 +562,7 @@ change here reaches all of them at once. That is what the gate is for.
|
|||||||
| suite | covers |
|
| suite | covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
||||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly for two of its five assertions**: the checksum-freshness dep-info file is the mutation that turns a hardlink clone into silent stale-artifact reuse, and it only exists under `-Z checksum-freshness`. Without one those two are skipped, loudly. |
|
||||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
||||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
||||||
|
|||||||
@@ -73,11 +73,33 @@ mkcrate "$crate_dir"
|
|||||||
cd "$crate_dir"
|
cd "$crate_dir"
|
||||||
|
|
||||||
export CARGO_INCREMENTAL=0
|
export CARGO_INCREMENTAL=0
|
||||||
|
|
||||||
|
# Every assertion below reads cargo's own words out of a build log
|
||||||
|
# (`Compiling libdep`, `Fresh probe`). A CI image that forces colour splices an
|
||||||
|
# ANSI reset between the status word and the crate name, at which point every
|
||||||
|
# one of those greps silently stops matching and the suite reports the
|
||||||
|
# opposite of what happened — observed on gitdan-ci's runner image, where
|
||||||
|
# scenario 2 failed while the log it printed plainly showed `Compiling libdep`.
|
||||||
|
# Pin the format the assertions are written against.
|
||||||
|
export CARGO_TERM_COLOR=never
|
||||||
# Checksum freshness is where the worst failure lives (the dep-* file carries
|
# Checksum freshness is where the worst failure lives (the dep-* file carries
|
||||||
# per-source checksums and is rewritten in place). Only available on nightly;
|
# per-source checksums and is rewritten in place). Only available on nightly;
|
||||||
# without it the test still covers the build/ and *.d families.
|
# without it the test still covers the build/ and *.d families.
|
||||||
CHECKSUM_MODE="off"
|
CHECKSUM_MODE="off"
|
||||||
if cargo +nightly -V >/dev/null 2>&1; then
|
# Probe the CAPABILITY, not the channel. `cargo +nightly -V` answers "did a
|
||||||
|
# cargo proxy called with +nightly exit 0", which is a different question from
|
||||||
|
# "will this build have checksum freshness" — `-V` short-circuits before `-Z`
|
||||||
|
# is validated at all, so that probe says yes on any cargo that resolves the
|
||||||
|
# name, including one whose nightly has since moved the flag. The scenario
|
||||||
|
# below then asserts the checksum-freshness mutation family against a build
|
||||||
|
# that never had it, and fails in the CONTROL, where a failure reads as "the
|
||||||
|
# hazard is gone" rather than "the toolchain is wrong". Observed the first
|
||||||
|
# time this suite ran on gitdan-ci.
|
||||||
|
#
|
||||||
|
# `-Z <flag> locate-project` is the narrowest command that actually parses the
|
||||||
|
# flag: it rejects the stable channel and an unknown flag name alike, needs no
|
||||||
|
# network, and builds nothing.
|
||||||
|
if cargo +nightly -Z checksum-freshness locate-project >/dev/null 2>&1; then
|
||||||
export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true
|
export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true
|
||||||
CARGO_BIN=(cargo +nightly)
|
CARGO_BIN=(cargo +nightly)
|
||||||
CHECKSUM_MODE="on"
|
CHECKSUM_MODE="on"
|
||||||
@@ -161,18 +183,32 @@ fi
|
|||||||
ok "no file in the source changed after a full rebuild in the clone"
|
ok "no file in the source changed after a full rebuild in the clone"
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "=== the whole point: the source's next build is still correct ==="
|
if [ "$CHECKSUM_MODE" = "on" ]; then
|
||||||
# The source's cache holds artifacts built from CONTENT_A. Advance the source
|
echo "=== the whole point: the source's next build is still correct ==="
|
||||||
# to CONTENT_B (as a merge would) and rebuild in it. If the clone had
|
# The source's cache holds artifacts built from CONTENT_A. Advance the
|
||||||
# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib.
|
# source to CONTENT_B (as a merge would) and rebuild in it. If the clone had
|
||||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib.
|
||||||
touch -d '@1000000000' src/lib.rs
|
#
|
||||||
log="$scratch/rebuild.log"
|
# CHECKSUM-FRESHNESS ONLY, and the backdated mtime is why. Under checksum
|
||||||
CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; }
|
# freshness the dep-info file's per-source checksums decide, so a 2001
|
||||||
if grep -qE '^\s+Fresh probe' "$log"; then
|
# timestamp on changed content must still rebuild — the assertion below.
|
||||||
fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse"
|
# Under Cargo's ordinary MTIME freshness the same timestamp means the source
|
||||||
|
# is older than the artifact, and reporting Fresh is the correct answer;
|
||||||
|
# asserting otherwise asserts a bug. This scenario was written against a
|
||||||
|
# machine with a nightly installed and, run without one, failed on that
|
||||||
|
# correct answer.
|
||||||
|
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||||
|
touch -d '@1000000000' src/lib.rs
|
||||||
|
log="$scratch/rebuild.log"
|
||||||
|
CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; }
|
||||||
|
if grep -qE '^\s+Fresh probe' "$log"; then
|
||||||
|
fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse"
|
||||||
|
fi
|
||||||
|
ok "source correctly rebuilt its crate after advancing to the clone's content"
|
||||||
|
else
|
||||||
|
echo "=== skipped: the source's-next-build scenario needs checksum freshness ==="
|
||||||
|
echo " (a nightly cargo accepting -Z checksum-freshness; see the probe above)"
|
||||||
fi
|
fi
|
||||||
ok "source correctly rebuilt its crate after advancing to the clone's content"
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
||||||
|
|||||||
@@ -75,6 +75,15 @@
|
|||||||
# Asserts BOTH jobs correctly recompile the dependency and succeed.
|
# Asserts BOTH jobs correctly recompile the dependency and succeed.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Every assertion below reads cargo's own words out of a build log
|
||||||
|
# (`Compiling libdep`, `Fresh probe`). A CI image that forces colour splices an
|
||||||
|
# ANSI reset between the status word and the crate name, at which point every
|
||||||
|
# one of those greps silently stops matching and the suite reports the
|
||||||
|
# opposite of what happened — observed on gitdan-ci's runner image, where
|
||||||
|
# scenario 2 failed while the log it printed plainly showed `Compiling libdep`.
|
||||||
|
# Pin the format the assertions are written against.
|
||||||
|
export CARGO_TERM_COLOR=never
|
||||||
|
|
||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
restore_mtimes="$script_dir/restore-mtimes.sh"
|
restore_mtimes="$script_dir/restore-mtimes.sh"
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user