fix(hardlink): name the mutable set directly, under both build-dir layouts
CI / shellcheck + selftests (pull_request) Skipped

`unshare_mutable_paths` selected `.fingerprint` and `build` directories. Under
Cargo's build-dir layout v2 the first clause matches nothing and the second
matches the whole tree, because v2 regroups artifacts under `build/` alongside
the metadata. Measured on one scratch crate: 39.3% of the tree real-copied
under v1, 99.996% under v2.

The selection now names the mutable set rather than the container it used to
live in: fingerprint directories under either spelling, layout v2's `run/`
directories, layout v1's loose build-script run metadata, and the `out`
directories that are a build script's OUT_DIR rather than a compile unit's
artifact directory. The two are told apart structurally, by Cargo's record of
the build-script execution sitting beside the OUT_DIR and nowhere else.

Verifying that turned up a second, layout-independent hazard: a linked
executable is written through whatever inode is already at its path, so a
`cargo test --no-run` inside a `cp -al` clone rewrites the source's own test
binary. Reproduced on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly and
1.100.0-nightly, under both layouts. Every executable is now real-copied;
`.rlib`, `.rmeta` and `incremental/` are what stay shared.

hardlink-clone-selftest.sh gains two file-only layout fixtures that pin the
partition in both directions without a compiler, and a live scenario that
relinks a test binary.
This commit is contained in:
2026-08-27 14:12:01 -05:00
parent 4114996954
commit e5b26a9368
4 changed files with 398 additions and 52 deletions
+279 -10
View File
@@ -28,8 +28,6 @@ set -euo pipefail
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$script_dir/cache-lib.sh"
command -v cargo >/dev/null || { echo "SKIP: no cargo on PATH"; exit 0; }
scratch=$(mktemp -d)
trap 'rm -rf "$scratch"' EXIT
pass_count=0
@@ -59,6 +57,14 @@ version = "0.1.0"
edition = "2021"
[workspace]
TOML
# A binary as well as a library, because the two are written differently and
# only one of them is safe to share: rustc writes an rlib to a temporary and
# renames it into place, while the LINKER writes an executable through the
# existing inode. Without a bin target this suite never relinks anything and
# cannot see that difference.
cat > "$dir/src/main.rs" <<'RS'
fn main() { println!("{}", probe::f()); }
RS
cat > "$dir/build.rs" <<'RS'
use std::{env, fs, path::PathBuf};
fn main() {
@@ -70,6 +76,208 @@ fn main() {
RS
}
# ---------------------------------------------------------------------------
# Both build-dir layouts, without a compiler
# ---------------------------------------------------------------------------
#
# Every other scenario in this file runs whichever layout the installed Cargo
# happens to write, so on any one machine it exercises exactly ONE of the two —
# and gitdan-ci's is v1. The two fixtures below reproduce both directory shapes
# from files alone, clone them through the real `hardlink_clone_into()`, and
# assert file by file which side of the partition each one lands on.
#
# Shapes taken from a scratch crate (serde + serde_json + regex, plus a build
# script) built on 2026-08-27: cargo 1.98.0-nightly (a335d47ff 2026-06-26)
# writes v1, cargo 1.100.0-nightly (e8cb624d5 2026-08-22) writes v2.
#
# v2 is where the partition is easy to get wrong, and the fixtures are built to
# say so: a build script's OUT_DIR and a compile unit's rlib are BOTH a
# directory called `out`, one directory apart, and they need opposite
# treatment.
mkfile() { mkdir -p "$(dirname "$1")"; printf '%s' "$2" > "$1"; }
# Big enough that the byte-fraction assertion below measures something.
artifact_bytes=$(head -c 4096 /dev/zero | tr '\0' 'A')
# Spec lines are `<shared|private>|<path relative to the tree root>`.
assert_partition() {
local label="$1" src="$2" spec="$3"
local clone="${src}-clone" want rel si ci total=0 copied=0 sz
hardlink_clone_into "$src" "$clone" "selftest-$label" \
|| fail "$label: hardlink_clone_into refused the destination"
while IFS='|' read -r want rel; do
[ -n "${rel:-}" ] || continue
[ -e "$clone/$rel" ] || fail "$label: $rel is missing from the clone"
si=$(stat -c '%i' "$src/$rel"); ci=$(stat -c '%i' "$clone/$rel")
case "$want" in
shared)
[ "$si" = "$ci" ] \
|| fail "$label: $rel was real-copied, but it is an artifact and must stay shared" ;;
private)
[ "$si" != "$ci" ] \
|| fail "$label: $rel still shares an inode with the source, so a build in the clone can rewrite it" ;;
*) fail "$label: unknown spec verb '$want'" ;;
esac
done <<< "$spec"
ok "$label: every file landed on the right side of the partition"
# The cost model, asserted rather than assumed. Selecting too much is not a
# correctness bug, which is exactly why nothing caught layout v2 taking the
# selection from a fifth of the tree to 99.998% of it (gitdan-actions#14):
# a hardlink clone that real-copies everything is a `cp -a` with extra steps.
# The bound is loose on purpose. It is not a budget — the honest figure moves
# with how much of a tree is linker output, and these fixtures are mostly
# that by construction — it is a floor under "still a hardlink clone at all".
while IFS= read -r rel; do
# `.cargo-*lock*` is stripped from every clone by design, so it has no
# counterpart to compare against.
[ -e "$clone/$rel" ] || continue
sz=$(stat -c '%s' "$src/$rel")
total=$((total + sz))
[ "$(stat -c '%i' "$src/$rel")" = "$(stat -c '%i' "$clone/$rel")" ] || copied=$((copied + sz))
done < <(cd "$src" && find . -type f -printf '%P\n')
[ "$total" -gt 0 ] || fail "$label: fixture has no bytes to measure"
[ $((copied * 100 / total)) -lt 90 ] \
|| fail "$label: the clone real-copied $((copied * 100 / total))% of its source's bytes — the hardlink saving is gone"
ok "$label: clone real-copies $((copied * 100 / total))% of ${total} B (${copied} B), the rest is shared"
}
# Layout v2: no `.fingerprint`, no `deps`. Everything regroups per build unit
# under `build/<pkg>/<hash>/{fingerprint,out,run}`, artifacts included — which
# is what took `-name build` from "the metadata" to "the whole tree".
#
# THE THREE UNIT KINDS ARE THE POINT. `1bf...` is a compile unit: its `out`
# holds the rlib. `08c...` is the build script's own compile unit: its `out`
# holds the build-script binary. `da9...` is the build-script RUN unit: its
# `out` IS the OUT_DIR, and it is the only one of the three whose `out` is
# mutable. The `run/` directory beside it is the structural difference.
v2="$scratch/layout-v2"
mkfile "$v2/CACHEDIR.TAG" 'Signature: 8a477f597d28d172'
mkfile "$v2/.rustc_info.json" '{"rustc_fingerprint":1}'
mkfile "$v2/debug/.cargo-lock" ''
mkfile "$v2/debug/libprobe.rlib" "$artifact_bytes"
mkfile "$v2/debug/libprobe.d" '/probe/src/lib.rs:'
for f in dep-lib-probe lib-probe lib-probe.json invoked.timestamp; do
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/fingerprint/$f" "$f"
done
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rlib" "$artifact_bytes"
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rmeta" "$artifact_bytes"
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/probe-1bf5493368dce3cd.d" '/probe/src/lib.rs:'
for f in build-script-build-script-build build-script-build-script-build.json \
dep-build-script-build-script-build invoked.timestamp; do
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/fingerprint/$f" "$f"
done
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build" "$artifact_bytes"
chmod +x "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build"
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build.d" '/probe/build.rs:'
# A test binary: the same `out` directory as the rlib above, and the largest
# thing in a real tree that a linker writes.
for f in dep-test-lib-probe test-lib-probe test-lib-probe.json invoked.timestamp; do
mkfile "$v2/debug/build/probe/6a091d813b2be60d/fingerprint/$f" "$f"
done
mkfile "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d" "$artifact_bytes"
chmod +x "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d"
mkfile "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d.d" '/probe/src/lib.rs:'
for f in run-build-script-build-script-build run-build-script-build-script-build.json; do
mkfile "$v2/debug/build/probe/da96cf45111f80dd/fingerprint/$f" "$f"
done
mkfile "$v2/debug/build/probe/da96cf45111f80dd/out/gen.txt" 'generated from 24 bytes'
for f in invoked.timestamp root-output stdout stderr; do
mkfile "$v2/debug/build/probe/da96cf45111f80dd/run/$f" "$f"
done
mkfile "$v2/debug/incremental/probe-abc/s-xyz/dep-graph.bin" "$artifact_bytes"
assert_partition "layout v2" "$v2" "$(cat <<'SPEC'
private|.rustc_info.json
private|debug/libprobe.d
private|debug/build/probe/1bf5493368dce3cd/fingerprint/dep-lib-probe
private|debug/build/probe/1bf5493368dce3cd/fingerprint/lib-probe
private|debug/build/probe/1bf5493368dce3cd/fingerprint/lib-probe.json
private|debug/build/probe/1bf5493368dce3cd/fingerprint/invoked.timestamp
private|debug/build/probe/1bf5493368dce3cd/out/probe-1bf5493368dce3cd.d
private|debug/build/probe/08c7dda6eacd6dca/fingerprint/dep-build-script-build-script-build
private|debug/build/probe/08c7dda6eacd6dca/fingerprint/invoked.timestamp
private|debug/build/probe/08c7dda6eacd6dca/out/build_script_build.d
private|debug/build/probe/da96cf45111f80dd/fingerprint/run-build-script-build-script-build
private|debug/build/probe/da96cf45111f80dd/out/gen.txt
private|debug/build/probe/da96cf45111f80dd/run/root-output
private|debug/build/probe/da96cf45111f80dd/run/stdout
private|debug/build/probe/da96cf45111f80dd/run/stderr
private|debug/build/probe/da96cf45111f80dd/run/invoked.timestamp
shared|debug/libprobe.rlib
shared|debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rlib
shared|debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rmeta
private|debug/build/probe/08c7dda6eacd6dca/out/build_script_build
private|debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d
private|debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d.d
private|debug/build/probe/6a091d813b2be60d/fingerprint/dep-test-lib-probe
shared|debug/incremental/probe-abc/s-xyz/dep-graph.bin
SPEC
)"
# Layout v1: one `.fingerprint` and one `deps` per profile; `build/<pkg>-<hash>`
# holds the build script's compiled binary in one unit directory and its run
# metadata plus OUT_DIR in another.
v1="$scratch/layout-v1"
mkfile "$v1/CACHEDIR.TAG" 'Signature: 8a477f597d28d172'
mkfile "$v1/.rustc_info.json" '{"rustc_fingerprint":1}'
mkfile "$v1/debug/.cargo-lock" ''
mkfile "$v1/debug/libprobe.rlib" "$artifact_bytes"
mkfile "$v1/debug/libprobe.d" '/probe/src/lib.rs:'
mkfile "$v1/debug/deps/libprobe-1bf5493368dce3cd.rlib" "$artifact_bytes"
mkfile "$v1/debug/deps/libprobe-1bf5493368dce3cd.rmeta" "$artifact_bytes"
mkfile "$v1/debug/deps/probe-1bf5493368dce3cd.d" '/probe/src/lib.rs:'
# A test binary, which layout v1 leaves in `deps/` beside the rlibs.
mkfile "$v1/debug/deps/probe-6a091d813b2be60d" "$artifact_bytes"
chmod +x "$v1/debug/deps/probe-6a091d813b2be60d"
mkfile "$v1/debug/deps/probe-6a091d813b2be60d.d" '/probe/src/lib.rs:'
for f in dep-lib-probe lib-probe lib-probe.json invoked.timestamp; do
mkfile "$v1/debug/.fingerprint/probe-1bf5493368dce3cd/$f" "$f"
done
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build-script-build" "$artifact_bytes"
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca" "$artifact_bytes"
chmod +x "$v1/debug/build/probe-08c7dda6eacd6dca/build-script-build" \
"$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca"
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca.d" '/probe/build.rs:'
for f in invoked.timestamp output root-output stderr; do
mkfile "$v1/debug/build/probe-da96cf45111f80dd/$f" "$f"
done
mkfile "$v1/debug/build/probe-da96cf45111f80dd/out/gen.txt" 'generated from 24 bytes'
mkfile "$v1/debug/incremental/probe-abc/s-xyz/dep-graph.bin" "$artifact_bytes"
assert_partition "layout v1" "$v1" "$(cat <<'SPEC'
private|.rustc_info.json
private|debug/libprobe.d
private|debug/deps/probe-1bf5493368dce3cd.d
private|debug/.fingerprint/probe-1bf5493368dce3cd/dep-lib-probe
private|debug/.fingerprint/probe-1bf5493368dce3cd/lib-probe
private|debug/.fingerprint/probe-1bf5493368dce3cd/lib-probe.json
private|debug/.fingerprint/probe-1bf5493368dce3cd/invoked.timestamp
private|debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca.d
private|debug/deps/probe-6a091d813b2be60d
private|debug/deps/probe-6a091d813b2be60d.d
private|debug/build/probe-da96cf45111f80dd/invoked.timestamp
private|debug/build/probe-da96cf45111f80dd/output
private|debug/build/probe-da96cf45111f80dd/root-output
private|debug/build/probe-da96cf45111f80dd/stderr
private|debug/build/probe-da96cf45111f80dd/out/gen.txt
shared|debug/libprobe.rlib
shared|debug/deps/libprobe-1bf5493368dce3cd.rlib
shared|debug/deps/libprobe-1bf5493368dce3cd.rmeta
private|debug/build/probe-08c7dda6eacd6dca/build-script-build
private|debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca
shared|debug/incremental/probe-abc/s-xyz/dep-graph.bin
SPEC
)"
echo
command -v cargo >/dev/null || {
echo "SKIP: no cargo on PATH — the fixture scenarios above ran, the live-Cargo ones cannot"
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
exit 0
}
crate_dir="$scratch/probe"
mkcrate "$crate_dir"
cd "$crate_dir"
@@ -304,27 +512,44 @@ build_base "$base_fix"
before=$(snapshot_tree "$base_fix")
hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_into reported the destination already existed"
# The clone's contract, asserted before anything builds in it: artifacts
# share inodes (that is what makes the clone near-free), and every file Cargo
# rewrites in place does not (that is what makes it sound). Checking after a
# rebuild would prove nothing — the rebuild replaces those files anyway.
shared=0; unshared=0
# The clone's contract, asserted before anything builds in it and asserted in
# BOTH directions: every file Cargo rewrites in place is privately owned (that
# is what makes the clone sound), and every artifact still shares its inode
# (that is what makes it near-free). Checking after a rebuild would prove
# nothing — the rebuild replaces those files anyway.
#
# The mutable-family patterns cover both layouts: `.fingerprint/` is v1's,
# `fingerprint/` and `run/` are v2's, and `gen.txt` is this crate's build
# script's OUT_DIR product, which under v2 sits in a directory called `out`
# beside sibling units whose `out` holds artifacts.
shared=0; unshared=0; shared_bytes=0; copied_bytes=0
while IFS= read -r f; do
rel="${f#"$base_fix"/}"
[ -e "$clone_fix/$rel" ] || continue
sz=$(stat -c '%s' "$f")
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
case "$rel" in
*/.fingerprint/*|*/build/*|*.d|.rustc_info.json)
*/.fingerprint/*|*/fingerprint/*|*/run/*|*/out/gen.txt|*/output|*/root-output|*/stderr|*/invoked.timestamp|*.d|.rustc_info.json)
fail "mutable path still shares an inode with the source: $rel" ;;
esac
shared=$((shared + 1))
shared=$((shared + 1)); shared_bytes=$((shared_bytes + sz))
else
unshared=$((unshared + 1))
case "$rel" in
*.rlib|*.rmeta)
fail "artifact was real-copied rather than shared: $rel" ;;
esac
unshared=$((unshared + 1)); copied_bytes=$((copied_bytes + sz))
fi
done < <(find "$base_fix" -type f)
[ "$shared" -gt 0 ] || fail "nothing is shared — the clone degenerated into a full copy"
[ "$unshared" -gt 0 ] || fail "nothing was unshared — unshare_mutable_paths did not run"
total_bytes=$((shared_bytes + copied_bytes))
ok "fresh clone shares ${shared} artifact files and privately owns ${unshared} mutable ones"
# Reported, not asserted. This crate has no dependencies, so almost all of its
# bytes are the two executables — a ratio that says nothing about a real tree.
# The fixtures above are where the cost model is gated, because there the
# composition is fixed.
echo " note: this clone real-copies $((copied_bytes * 100 / total_bytes))% of ${total_bytes} B"
printf '%s\n' "$CONTENT_B" > src/lib.rs
CARGO_TARGET_DIR="$clone_fix" "${CARGO_BIN[@]}" build -q
@@ -337,6 +562,50 @@ if [ -n "$fix_mutated" ]; then
fi
ok "no file in the source changed after a full rebuild in the clone"
echo
echo "=== a linked TEST binary, which nothing uplifts and nothing replaces ==="
# The one artifact family that is NOT safe to share, and the reason
# `unshare_mutable_paths` privately owns every executable. rustc writes an
# rlib to a temporary and renames it in; the LINKER writes an executable
# through whatever inode is already at the path. Cargo re-creates the path
# first when it also has to uplift the result — a bin target's
# `deps/<bin>-<hash>` has a hardlink twin at `<profile>/<bin>` — which is why
# bin targets look safe. A test binary has no twin, nothing re-creates it, and
# the write lands on the source's inode.
#
# Measured 2026-08-27 on cargo 1.98.0-nightly (a335d47ff, layout v1) and
# 1.100.0-nightly (e8cb624d5, layout v2): under BOTH layouts a `cargo test
# --no-run` inside a raw `cp -al` clone rewrote the SOURCE's own test binary.
base_exe="$scratch/base-exe"; clone_exe_ctl="$scratch/clone-exe-ctl"; clone_exe="$scratch/clone-exe"
printf '%s\n' "$CONTENT_A" > src/lib.rs
CARGO_TARGET_DIR="$base_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
before=$(snapshot_tree "$base_exe")
cp -al "$base_exe" "$clone_exe_ctl"
strip_cargo_locks "$clone_exe_ctl"
printf '%s\n' "$CONTENT_B" > src/lib.rs
CARGO_TARGET_DIR="$clone_exe_ctl" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
exe_ctl_mutated=$(mutated_paths "$before" "$(snapshot_tree "$base_exe")")
[ -n "$exe_ctl_mutated" ] \
|| fail "control: a raw cp -al clone of a test build no longer mutates the source — the test can no longer tell fixed from broken"
ok "raw cp -al clone of a test build mutates the source ($(printf '%s\n' "$exe_ctl_mutated" | wc -l) paths)"
printf '%s\n' "$exe_ctl_mutated" | sed 's/^/ /'
# Rebuild the base from CONTENT_A so it is warm and consistent again, then do
# the same thing through the real clone.
printf '%s\n' "$CONTENT_A" > src/lib.rs
CARGO_TARGET_DIR="$base_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
before=$(snapshot_tree "$base_exe")
hardlink_clone_into "$base_exe" "$clone_exe" "selftest-exe" \
|| fail "hardlink_clone_into refused the destination"
printf '%s\n' "$CONTENT_B" > src/lib.rs
CARGO_TARGET_DIR="$clone_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
exe_mutated=$(mutated_paths "$before" "$(snapshot_tree "$base_exe")")
if [ -n "$exe_mutated" ]; then
printf '%s\n' "$exe_mutated" | sed 's/^/ /' >&2
fail "a test build in the clone mutated the source through a shared inode"
fi
ok "no file in the source changed after a full test build in the clone"
echo
if [ "$CHECKSUM_MODE" = "on" ]; then
echo "=== the whole point: the source's next build is still correct ==="