fix(hardlink): name the mutable set directly, under both build-dir layouts
CI / shellcheck + selftests (pull_request) Skipped

`unshare_mutable_paths` selected `.fingerprint` and `build` directories. Under
Cargo's build-dir layout v2 the first clause matches nothing and the second
matches the whole tree, because v2 regroups artifacts under `build/` alongside
the metadata. Measured on one scratch crate: 39.3% of the tree real-copied
under v1, 99.996% under v2.

The selection now names the mutable set rather than the container it used to
live in: fingerprint directories under either spelling, layout v2's `run/`
directories, layout v1's loose build-script run metadata, and the `out`
directories that are a build script's OUT_DIR rather than a compile unit's
artifact directory. The two are told apart structurally, by Cargo's record of
the build-script execution sitting beside the OUT_DIR and nowhere else.

Verifying that turned up a second, layout-independent hazard: a linked
executable is written through whatever inode is already at its path, so a
`cargo test --no-run` inside a `cp -al` clone rewrites the source's own test
binary. Reproduced on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly and
1.100.0-nightly, under both layouts. Every executable is now real-copied;
`.rlib`, `.rmeta` and `incremental/` are what stay shared.

hardlink-clone-selftest.sh gains two file-only layout fixtures that pin the
partition in both directions without a compiler, and a live scenario that
relinks a test binary.
This commit is contained in:
2026-08-27 14:12:01 -05:00
parent 4114996954
commit e5b26a9368
4 changed files with 398 additions and 52 deletions
+86 -28
View File
@@ -320,14 +320,13 @@ _unshare_files() {
# THE load-bearing function of this whole design.
#
# A hardlink clone is only safe if every write the clone's build performs
# lands on a NEW inode, leaving the source's data untouched. That is true for
# compilation artifacts — rustc and the linker replace `deps/*.rlib`,
# `*.rmeta`, and binaries rather than truncating them in place — and it is
# NOT true for the metadata Cargo and build scripts write with a plain
# truncating write. Measured directly (Linux, ext4, cargo 1.9x nightly:
# `cp -al` a warm target dir, change a source file, build in the clone, diff
# the source) the following files in the SOURCE were mutated through the
# shared inode:
# lands on a NEW inode, leaving the source's data untouched. That is true of
# rustc's own outputs — it writes an `.rlib` or `.rmeta` to a temporary and
# renames it into place — and it is NOT true of the metadata Cargo and build
# scripts write with a plain truncating write, nor of anything the LINKER
# produces. Measured directly (Linux, ext4: `cp -al` a warm target dir, change
# a source file, build in the clone, diff the source) the following files in
# the SOURCE were mutated through the shared inode:
#
# <profile>/.fingerprint/<unit>/dep-<target> (build-dir layout v1) — or,
# <profile>/build/<pkg>/<hash>/fingerprint/dep-<target>
@@ -347,6 +346,20 @@ _unshare_files() {
# plain fs::write)
# <profile>/deps/*.d, <profile>/*.d (Cargo's post-processed
# dep-info)
# <profile>/deps/<test>-<hash> (a linked TEST binary; under
# layout v2,
# `build/<pkg>/<hash>/out/`)
#
# THE LINKED-OUTPUT CASE IS NOT LAYOUT-SPECIFIC AND WAS NOT PART OF THIS
# FUNCTION UNTIL 2026-08-27 (gitdan-actions#14). A `cargo test --no-run` inside
# a raw `cp -al` clone rewrote the source's own test binary in place on cargo
# 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly (layout v1) and 1.100.0-nightly
# (layout v2) alike. Cargo re-creates the path first when it also has to uplift
# the result — a bin target's `deps/<bin>-<hash>` has a hardlink twin at
# `<profile>/<bin>` — and other crate shapes relinked to a fresh inode for
# reasons this measurement did not pin down. Since the safe cases could not be
# enumerated, every executable is treated as mutable; `.rlib`, `.rmeta` and
# `incremental/` are what stay shared, and they are the bytes worth sharing.
#
# The checksum-freshness case is not a cosmetic one. Reproduced end to end:
# branch B clones base's cache, builds its own content, and thereby rewrites
@@ -390,40 +403,61 @@ _unshare_files() {
# included. Bracketed locally: cargo 1.97.1 and 1.98.0-nightly write v1,
# 1.100.0-nightly writes v2.
#
# The `-name .fingerprint` clause below therefore matches nothing under a v2
# Cargo, and the dep-info file is covered only because the `-name build` clause
# happens to swallow its new home. That is belt-and-braces by accident, not by
# design — and the same accident makes this function real-copy essentially the
# whole tree, because the artifacts moved under `build/` too. Measured on one
# scratch crate (serde + serde_json + regex), same sources both ways:
# Until 2026-08-27 the selection below was `-name .fingerprint -o -name build`,
# which under a v2 Cargo matched nothing on its first clause and the entire
# tree on its second, because the artifacts moved under `build/` too. The guard
# held by accident and the saving did not: on one scratch crate (serde +
# serde_json + regex plus a build script), same sources both ways —
#
# cargo 1.97.1 (layout v1) 27.0 MB unshared of 126.7 MB 21.3%
# 1.100.0-nightly (layout v2) 105.9 MB unshared of 105.9 MB — 99.998%
# cargo 1.98.0-nightly (layout v1) 64.9 MB unshared of 165.1 MB — 39.3%
# 1.100.0-nightly (layout v2) 110.4 MB unshared of 110.4 MB — 99.996%
#
# So the guard still holds and the saving does not. Deliberately NOT fixed
# here: adjusting the selection is a change to what gets hardlinked on every
# consumer, which wants its own change and its own review, and the deadline is
# cargo 1.100.0 stable on 2026-11-12. Tracked as gitdan-actions#14.
#
# The historical v1 figure this block used to quote stands as measured: on a
# 6.9 GB Bevy workspace target dir the unshared set was .fingerprint 22 MB +
# build/ 237 MB + a handful of dep-info files — about 3.7% of the tree, against
# 100% for a plain `cp -a`. It describes layout v1 only.
# The selection now names the mutable set directly rather than by the container
# it used to live in, so it holds under both layouts; the same crate measures
# 45.3% (v1) and 38.4% (v2), both dominated by the linked-output rule above
# rather than by the layout. On a real 5.5 GB Bevy target dir the whole change
# moves the real-copied share from 9.0% to 14.1%.
#
# `incremental/` is deliberately left shared: rustc writes each incremental
# session to a fresh `s-*-working` directory and finalises it with a rename,
# and garbage-collects old sessions by unlinking directory entries — neither
# of which mutates a shared inode. CI should still set CARGO_INCREMENTAL=0,
# for size rather than correctness.
# The directories `unshare_mutable_paths` replaces, under either layout.
#
# All four names are pruned, so nothing selected here can contain anything else
# selected here and the caller never unshares a subtree twice.
#
# `out` is the one that needs deciding rather than naming, and it is the whole
# difficulty of layout v2: a compile unit's rlib and a build script's OUT_DIR
# are both a directory called `out`, one directory apart, and they need
# opposite treatment. The discriminator is structural — Cargo records a build
# script's execution beside its OUT_DIR and nowhere else, as `run/root-output`
# under v2 and as a loose `root-output` under v1 — so a unit directory holding
# one of those is a RUN unit and its `out` is the OUT_DIR. A unit directory
# without one is a compile unit, and its `out` holds the artifact whose sharing
# is the entire point of the clone.
_mutable_dirs() {
local root="$1" d unit
while IFS= read -r d; do
if [ "${d##*/}" = out ]; then
unit="${d%/out}"
[ -d "$unit/run" ] || [ -e "$unit/root-output" ] || continue
fi
printf '%s\n' "$d"
done < <(find "$root" -type d \
\( -name .fingerprint -o -name fingerprint -o -name run -o -name out \) \
-prune -print 2>/dev/null)
}
unshare_mutable_paths() {
local root="$1" d
[ -d "$root" ] || return 0
# The list is materialised in full before anything is replaced: each
# replacement deletes and recreates a directory, and a live `find` walk over
# a tree being mutated underneath it is a needless hazard. `-prune` keeps a
# match's own contents out of the list.
# a tree being mutated underneath it is a needless hazard.
local -a dirs=()
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
mapfile -t dirs < <(_mutable_dirs "$root")
for d in "${dirs[@]}"; do
[ -n "$d" ] || continue
unshare_subtree "$d" || {
@@ -435,6 +469,30 @@ unshare_mutable_paths() {
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
return 1
}
# Layout v1's build-script run metadata, which v2 groups under `run/` and v1
# leaves loose in the run unit's directory. `invoked.timestamp` is empty and
# carries its meaning in its mtime, which a shared inode carries too.
_unshare_files "$root" -type f \
\( -name output -o -name root-output -o -name stderr -o -name invoked.timestamp \) || {
echo "::error::unshare_mutable_paths: failed to unshare build-script run metadata under ${root}" >&2
return 1
}
# Linked outputs. Unlike an rlib or an rmeta — which rustc writes to a
# temporary and renames into place — an executable or shared object is
# written by the LINKER, and the linker writes THROUGH an existing inode
# whenever the path it is given has no other hard link. Measured 2026-08-27
# on cargo 1.98.0-nightly (a335d47ff, layout v1) and 1.100.0-nightly
# (e8cb624d5, layout v2), mold and the default linker alike: a `cargo test`
# binary in a `cp -al` clone rewrote the SOURCE's copy of itself in place,
# under both layouts.
#
# The executable bit is the discriminator because it is the linker's own
# output that is at risk, not the directory it happens to land in — `.rlib`,
# `.rmeta` and `incremental/` stay shared and they are the bytes that matter.
_unshare_files "$root" -type f -perm -u+x || {
echo "::error::unshare_mutable_paths: failed to unshare linked outputs under ${root}" >&2
return 1
}
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
return 1