fix(hardlink): name the mutable set directly, under both build-dir layouts
CI / shellcheck + selftests (pull_request) Skipped
CI / shellcheck + selftests (pull_request) Skipped
`unshare_mutable_paths` selected `.fingerprint` and `build` directories. Under Cargo's build-dir layout v2 the first clause matches nothing and the second matches the whole tree, because v2 regroups artifacts under `build/` alongside the metadata. Measured on one scratch crate: 39.3% of the tree real-copied under v1, 99.996% under v2. The selection now names the mutable set rather than the container it used to live in: fingerprint directories under either spelling, layout v2's `run/` directories, layout v1's loose build-script run metadata, and the `out` directories that are a build script's OUT_DIR rather than a compile unit's artifact directory. The two are told apart structurally, by Cargo's record of the build-script execution sitting beside the OUT_DIR and nowhere else. Verifying that turned up a second, layout-independent hazard: a linked executable is written through whatever inode is already at its path, so a `cargo test --no-run` inside a `cp -al` clone rewrites the source's own test binary. Reproduced on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly and 1.100.0-nightly, under both layouts. Every executable is now real-copied; `.rlib`, `.rmeta` and `incremental/` are what stay shared. hardlink-clone-selftest.sh gains two file-only layout fixtures that pin the partition in both directions without a compiler, and a live scenario that relinks a test binary.
This commit is contained in:
+86
-28
@@ -320,14 +320,13 @@ _unshare_files() {
|
||||
# THE load-bearing function of this whole design.
|
||||
#
|
||||
# A hardlink clone is only safe if every write the clone's build performs
|
||||
# lands on a NEW inode, leaving the source's data untouched. That is true for
|
||||
# compilation artifacts — rustc and the linker replace `deps/*.rlib`,
|
||||
# `*.rmeta`, and binaries rather than truncating them in place — and it is
|
||||
# NOT true for the metadata Cargo and build scripts write with a plain
|
||||
# truncating write. Measured directly (Linux, ext4, cargo 1.9x nightly:
|
||||
# `cp -al` a warm target dir, change a source file, build in the clone, diff
|
||||
# the source) the following files in the SOURCE were mutated through the
|
||||
# shared inode:
|
||||
# lands on a NEW inode, leaving the source's data untouched. That is true of
|
||||
# rustc's own outputs — it writes an `.rlib` or `.rmeta` to a temporary and
|
||||
# renames it into place — and it is NOT true of the metadata Cargo and build
|
||||
# scripts write with a plain truncating write, nor of anything the LINKER
|
||||
# produces. Measured directly (Linux, ext4: `cp -al` a warm target dir, change
|
||||
# a source file, build in the clone, diff the source) the following files in
|
||||
# the SOURCE were mutated through the shared inode:
|
||||
#
|
||||
# <profile>/.fingerprint/<unit>/dep-<target> (build-dir layout v1) — or,
|
||||
# <profile>/build/<pkg>/<hash>/fingerprint/dep-<target>
|
||||
@@ -347,6 +346,20 @@ _unshare_files() {
|
||||
# plain fs::write)
|
||||
# <profile>/deps/*.d, <profile>/*.d (Cargo's post-processed
|
||||
# dep-info)
|
||||
# <profile>/deps/<test>-<hash> (a linked TEST binary; under
|
||||
# layout v2,
|
||||
# `build/<pkg>/<hash>/out/`)
|
||||
#
|
||||
# THE LINKED-OUTPUT CASE IS NOT LAYOUT-SPECIFIC AND WAS NOT PART OF THIS
|
||||
# FUNCTION UNTIL 2026-08-27 (gitdan-actions#14). A `cargo test --no-run` inside
|
||||
# a raw `cp -al` clone rewrote the source's own test binary in place on cargo
|
||||
# 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly (layout v1) and 1.100.0-nightly
|
||||
# (layout v2) alike. Cargo re-creates the path first when it also has to uplift
|
||||
# the result — a bin target's `deps/<bin>-<hash>` has a hardlink twin at
|
||||
# `<profile>/<bin>` — and other crate shapes relinked to a fresh inode for
|
||||
# reasons this measurement did not pin down. Since the safe cases could not be
|
||||
# enumerated, every executable is treated as mutable; `.rlib`, `.rmeta` and
|
||||
# `incremental/` are what stay shared, and they are the bytes worth sharing.
|
||||
#
|
||||
# The checksum-freshness case is not a cosmetic one. Reproduced end to end:
|
||||
# branch B clones base's cache, builds its own content, and thereby rewrites
|
||||
@@ -390,40 +403,61 @@ _unshare_files() {
|
||||
# included. Bracketed locally: cargo 1.97.1 and 1.98.0-nightly write v1,
|
||||
# 1.100.0-nightly writes v2.
|
||||
#
|
||||
# The `-name .fingerprint` clause below therefore matches nothing under a v2
|
||||
# Cargo, and the dep-info file is covered only because the `-name build` clause
|
||||
# happens to swallow its new home. That is belt-and-braces by accident, not by
|
||||
# design — and the same accident makes this function real-copy essentially the
|
||||
# whole tree, because the artifacts moved under `build/` too. Measured on one
|
||||
# scratch crate (serde + serde_json + regex), same sources both ways:
|
||||
# Until 2026-08-27 the selection below was `-name .fingerprint -o -name build`,
|
||||
# which under a v2 Cargo matched nothing on its first clause and the entire
|
||||
# tree on its second, because the artifacts moved under `build/` too. The guard
|
||||
# held by accident and the saving did not: on one scratch crate (serde +
|
||||
# serde_json + regex plus a build script), same sources both ways —
|
||||
#
|
||||
# cargo 1.97.1 (layout v1) 27.0 MB unshared of 126.7 MB — 21.3%
|
||||
# 1.100.0-nightly (layout v2) 105.9 MB unshared of 105.9 MB — 99.998%
|
||||
# cargo 1.98.0-nightly (layout v1) 64.9 MB unshared of 165.1 MB — 39.3%
|
||||
# 1.100.0-nightly (layout v2) 110.4 MB unshared of 110.4 MB — 99.996%
|
||||
#
|
||||
# So the guard still holds and the saving does not. Deliberately NOT fixed
|
||||
# here: adjusting the selection is a change to what gets hardlinked on every
|
||||
# consumer, which wants its own change and its own review, and the deadline is
|
||||
# cargo 1.100.0 stable on 2026-11-12. Tracked as gitdan-actions#14.
|
||||
#
|
||||
# The historical v1 figure this block used to quote stands as measured: on a
|
||||
# 6.9 GB Bevy workspace target dir the unshared set was .fingerprint 22 MB +
|
||||
# build/ 237 MB + a handful of dep-info files — about 3.7% of the tree, against
|
||||
# 100% for a plain `cp -a`. It describes layout v1 only.
|
||||
# The selection now names the mutable set directly rather than by the container
|
||||
# it used to live in, so it holds under both layouts; the same crate measures
|
||||
# 45.3% (v1) and 38.4% (v2), both dominated by the linked-output rule above
|
||||
# rather than by the layout. On a real 5.5 GB Bevy target dir the whole change
|
||||
# moves the real-copied share from 9.0% to 14.1%.
|
||||
#
|
||||
# `incremental/` is deliberately left shared: rustc writes each incremental
|
||||
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
||||
# and garbage-collects old sessions by unlinking directory entries — neither
|
||||
# of which mutates a shared inode. CI should still set CARGO_INCREMENTAL=0,
|
||||
# for size rather than correctness.
|
||||
# The directories `unshare_mutable_paths` replaces, under either layout.
|
||||
#
|
||||
# All four names are pruned, so nothing selected here can contain anything else
|
||||
# selected here and the caller never unshares a subtree twice.
|
||||
#
|
||||
# `out` is the one that needs deciding rather than naming, and it is the whole
|
||||
# difficulty of layout v2: a compile unit's rlib and a build script's OUT_DIR
|
||||
# are both a directory called `out`, one directory apart, and they need
|
||||
# opposite treatment. The discriminator is structural — Cargo records a build
|
||||
# script's execution beside its OUT_DIR and nowhere else, as `run/root-output`
|
||||
# under v2 and as a loose `root-output` under v1 — so a unit directory holding
|
||||
# one of those is a RUN unit and its `out` is the OUT_DIR. A unit directory
|
||||
# without one is a compile unit, and its `out` holds the artifact whose sharing
|
||||
# is the entire point of the clone.
|
||||
_mutable_dirs() {
|
||||
local root="$1" d unit
|
||||
while IFS= read -r d; do
|
||||
if [ "${d##*/}" = out ]; then
|
||||
unit="${d%/out}"
|
||||
[ -d "$unit/run" ] || [ -e "$unit/root-output" ] || continue
|
||||
fi
|
||||
printf '%s\n' "$d"
|
||||
done < <(find "$root" -type d \
|
||||
\( -name .fingerprint -o -name fingerprint -o -name run -o -name out \) \
|
||||
-prune -print 2>/dev/null)
|
||||
}
|
||||
|
||||
unshare_mutable_paths() {
|
||||
local root="$1" d
|
||||
[ -d "$root" ] || return 0
|
||||
# The list is materialised in full before anything is replaced: each
|
||||
# replacement deletes and recreates a directory, and a live `find` walk over
|
||||
# a tree being mutated underneath it is a needless hazard. `-prune` keeps a
|
||||
# match's own contents out of the list.
|
||||
# a tree being mutated underneath it is a needless hazard.
|
||||
local -a dirs=()
|
||||
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
||||
mapfile -t dirs < <(_mutable_dirs "$root")
|
||||
for d in "${dirs[@]}"; do
|
||||
[ -n "$d" ] || continue
|
||||
unshare_subtree "$d" || {
|
||||
@@ -435,6 +469,30 @@ unshare_mutable_paths() {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
# Layout v1's build-script run metadata, which v2 groups under `run/` and v1
|
||||
# leaves loose in the run unit's directory. `invoked.timestamp` is empty and
|
||||
# carries its meaning in its mtime, which a shared inode carries too.
|
||||
_unshare_files "$root" -type f \
|
||||
\( -name output -o -name root-output -o -name stderr -o -name invoked.timestamp \) || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare build-script run metadata under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
# Linked outputs. Unlike an rlib or an rmeta — which rustc writes to a
|
||||
# temporary and renames into place — an executable or shared object is
|
||||
# written by the LINKER, and the linker writes THROUGH an existing inode
|
||||
# whenever the path it is given has no other hard link. Measured 2026-08-27
|
||||
# on cargo 1.98.0-nightly (a335d47ff, layout v1) and 1.100.0-nightly
|
||||
# (e8cb624d5, layout v2), mold and the default linker alike: a `cargo test`
|
||||
# binary in a `cp -al` clone rewrote the SOURCE's copy of itself in place,
|
||||
# under both layouts.
|
||||
#
|
||||
# The executable bit is the discriminator because it is the linker's own
|
||||
# output that is at risk, not the directory it happens to land in — `.rlib`,
|
||||
# `.rmeta` and `incremental/` stay shared and they are the bytes that matter.
|
||||
_unshare_files "$root" -type f -perm -u+x || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare linked outputs under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
||||
return 1
|
||||
|
||||
Reference in New Issue
Block a user