fix(hardlink): name the mutable set directly, under both build-dir layouts #16
@@ -29,7 +29,7 @@ on the runner having a single execution slot.
|
|||||||
|
|
||||||
One thing neither project had, and the reason the clone is not a plain
|
One thing neither project had, and the reason the clone is not a plain
|
||||||
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
||||||
cloned from.** Cargo writes real artifacts by replacing them, but writes its
|
cloned from.** rustc renames its own outputs into place, but Cargo writes its
|
||||||
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
||||||
write, straight through the shared inode. When Cargo resolves freshness by
|
write, straight through the shared inode. When Cargo resolves freshness by
|
||||||
content the file that gets corrupted is its dep-info fingerprint —
|
content the file that gets corrupted is its dep-info fingerprint —
|
||||||
@@ -37,17 +37,55 @@ content the file that gets corrupted is its dep-info fingerprint —
|
|||||||
`build/<pkg>/<hash>/fingerprint/dep-<target>` under Cargo's build-dir layout v2
|
`build/<pkg>/<hash>/fingerprint/dep-<target>` under Cargo's build-dir layout v2
|
||||||
— which holds the per-source checksums that decide freshness, and the failure
|
— which holds the per-source checksums that decide freshness, and the failure
|
||||||
is silent stale-artifact reuse rather than a slow build.
|
is silent stale-artifact reuse rather than a slow build.
|
||||||
`scripts/hardlink-clone-selftest.sh` reproduces it as an explicit control and
|
|
||||||
asserts the fix. The fix is to hardlink the artifacts (the GB) and real-copy
|
|
||||||
the metadata (the MB) — about 3.7% of a Bevy-sized target directory, against
|
|
||||||
100% for a full copy.
|
|
||||||
|
|
||||||
That ratio is a **layout-v1** figure and does not survive layout v2, which
|
One more family joins them, and it is not metadata: **anything the linker
|
||||||
regroups artifacts under `build/` alongside the metadata and so drags nearly
|
writes**. rustc writes an `.rlib` or `.rmeta` to a temporary and renames it
|
||||||
the whole tree into the real-copy set. Measured 2026-08-26 at 99.998% on a
|
into place, but a linked executable is written through whatever inode is
|
||||||
scratch crate. v2 is the nightly default and stabilises in cargo 1.100.0 on
|
already at its path — so a `cargo test --no-run` in a raw `cp -al` clone
|
||||||
2026-11-12; the correctness guarantee above is unaffected, the saving is not.
|
rewrites the source's own test binary. Measured 2026-08-27 on cargo 1.93.1
|
||||||
Tracked as [#14](https://gitdan.com/daniel/gitdan-actions/issues/14).
|
stable, 1.96.0-nightly, 1.98.0-nightly and 1.100.0-nightly, under both
|
||||||
|
build-dir layouts.
|
||||||
|
|
||||||
|
`scripts/hardlink-clone-selftest.sh` reproduces both as explicit controls and
|
||||||
|
asserts the fix. The fix is to hardlink what rustc renames into place — the
|
||||||
|
`.rlib`, `.rmeta` and `incremental/` bulk — and real-copy the metadata and the
|
||||||
|
linker outputs.
|
||||||
|
|
||||||
|
**The selection names that set directly, under either build-dir layout.**
|
||||||
|
Layout v2 regroups everything per build unit under
|
||||||
|
`build/<pkg>/<hash>/{fingerprint,out,run}/`, artifacts included, so there is no
|
||||||
|
`.fingerprint` and no `deps` to key off and `build/` is no longer a proxy for
|
||||||
|
"metadata" — it is the whole tree. The one place the two layouts genuinely
|
||||||
|
differ is that under v2 a build script's `OUT_DIR` and a compile unit's rlib
|
||||||
|
are both a directory called `out`.
|
||||||
|
|
||||||
|
**Ambiguity there resolves toward unsharing**, because over-unsharing costs
|
||||||
|
bytes and under-unsharing costs corruption. An `out` directory stays shared
|
||||||
|
only when two independent signals agree it is a compile unit's: it holds an
|
||||||
|
`.rlib`/`.rmeta` of its own, *and* its unit carries no record of a build-script
|
||||||
|
execution beside it (`run/` under v2, a loose `root-output` under v1). The
|
||||||
|
execution record alone is not enough — Cargo writes it only after the script
|
||||||
|
succeeds, so a build script that populates `OUT_DIR` and then fails leaves a
|
||||||
|
unit that reads as a compile unit. v2 is the nightly default and stabilises in
|
||||||
|
cargo 1.100.0 on 2026-11-12.
|
||||||
|
|
||||||
|
### What it costs
|
||||||
|
|
||||||
|
Real-copied share of a 5.5 GB Bevy target directory, before and after the
|
||||||
|
linker-output rule landed:
|
||||||
|
|
||||||
|
| tree | before | after |
|
||||||
|
|---|---|---|
|
||||||
|
| **excluding `incremental/`** — the figure to plan against, since the quick-start below sets `CARGO_INCREMENTAL: 0` | **36.4%** | **57.0%** |
|
||||||
|
| whole tree, `incremental/` included (a local dev checkout, not CI) | 9.0% | 14.1% |
|
||||||
|
|
||||||
|
The first row is the one a CI consumer gets. The increase is the linker-output
|
||||||
|
rule, not the layout work: on a scratch crate the layout fix alone takes v2
|
||||||
|
from 99.996% to 0.2%.
|
||||||
|
|
||||||
|
The copy is paid per clone and does not amortise — a fresh `cp -al` leaves
|
||||||
|
every file with `nlink >= 2`, so the `-links +1` filter cannot skip anything —
|
||||||
|
and a clone happens twice per job, once seeding and once publishing.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -578,7 +616,7 @@ change here reaches all of them at once. That is what the gate is for.
|
|||||||
| suite | covers |
|
| suite | covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
||||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly that actually resolves freshness by content for its last scenario**: the source's-next-build check reasons about content rather than mtime, so under mtime freshness it would assert a bug. Whether the toolchain does is settled by experiment on a throwaway crate, not by asking it — accepting `-Z checksum-freshness` stopped implying it on 2026-08-22, when cargo PR #17382 demoted the flag to a gate and gave `build.fingerprint` (default `mtime`) the choice; the suite now exports both and 1.100.0-nightly measures ACTIVE again. The experiment reports **three** outcomes, not two: active, measured-inactive, and *not measured*. Its answer codes are `0` and `3`, deliberately clear of every status bash generates for its own errors — so nothing that goes wrong inside the probe, including an expansion failure no guard can catch, can be read as an answer. The scenario is skipped for the last two alike, but a failure to measure is never reported as a measurement. The control also reports which mutation families the running Cargo exhibits — a note, not an assertion, since that set moves upstream. |
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly that actually resolves freshness by content for its last scenario**: the source's-next-build check reasons about content rather than mtime, so under mtime freshness it would assert a bug. Whether the toolchain does is settled by experiment on a throwaway crate, not by asking it — accepting `-Z checksum-freshness` stopped implying it on 2026-08-22, when cargo PR #17382 demoted the flag to a gate and gave `build.fingerprint` (default `mtime`) the choice; the suite now exports both and 1.100.0-nightly measures ACTIVE again. The experiment reports **three** outcomes, not two: active, measured-inactive, and *not measured*. Its answer codes are `0` and `3`, deliberately clear of every status bash generates for its own errors — so nothing that goes wrong inside the probe, including an expansion failure no guard can catch, can be read as an answer. The scenario is skipped for the last two alike, but a failure to measure is never reported as a measurement. The control also reports which mutation families the running Cargo exhibits — a note, not an assertion, since that set moves upstream. It also pins the SELECTION itself against both of Cargo's build-dir layouts, from two file-only fixtures that need no compiler — so the layout the installed Cargo does not happen to write is still covered — and asserts the partition in both directions: every file Cargo rewrites in place is privately owned, and every `.rlib`/`.rmeta` still shares its inode. The second half is what the old suite never checked beyond `shared > 0`, and it is what a layout change silently inverts. |
|
||||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
||||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
||||||
|
|||||||
+133
-28
@@ -317,17 +317,83 @@ _unshare_files() {
|
|||||||
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# True when a directory holds a compiled library artifact of its own.
|
||||||
|
#
|
||||||
|
# The glob is left unquoted and unmatched-glob-safe on purpose: with nullglob
|
||||||
|
# off an unmatched pattern stays literal and the `-e` test fails, which is the
|
||||||
|
# answer wanted.
|
||||||
|
_holds_compiled_artifact() {
|
||||||
|
local f
|
||||||
|
for f in "$1"/*.rlib "$1"/*.rmeta; do
|
||||||
|
[ -e "$f" ] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# The directories `unshare_mutable_paths` replaces, under either layout.
|
||||||
|
#
|
||||||
|
# All four names are pruned, so nothing selected here can contain anything else
|
||||||
|
# selected here and the caller never unshares a subtree twice.
|
||||||
|
#
|
||||||
|
# `out` is the one that needs deciding rather than naming, and it is the whole
|
||||||
|
# difficulty of layout v2: a compile unit's rlib and a build script's OUT_DIR
|
||||||
|
# are both a directory called `out`, one directory apart, and they need
|
||||||
|
# opposite treatment.
|
||||||
|
#
|
||||||
|
# AMBIGUITY RESOLVES TOWARD UNSHARING, and that direction is the rule rather
|
||||||
|
# than a default: over-unsharing costs bytes, under-unsharing costs corruption.
|
||||||
|
# So an `out` directory is left shared only when TWO independent signals agree
|
||||||
|
# it is a compile unit's artifact directory, and either one missing is enough
|
||||||
|
# to real-copy it:
|
||||||
|
#
|
||||||
|
# 1. it holds an `.rlib`/`.rmeta` of its own — the artifact whose sharing is
|
||||||
|
# the entire point of the clone; and
|
||||||
|
# 2. its unit directory has no record of a build-script execution beside it
|
||||||
|
# (`run/` under layout v2, a loose `root-output` under v1).
|
||||||
|
#
|
||||||
|
# Signal 2 alone was the first cut of this and it is NOT sufficient, because
|
||||||
|
# Cargo writes `root-output` only AFTER the script exits successfully. A build
|
||||||
|
# script that populates `OUT_DIR` and then FAILS leaves a unit with no record
|
||||||
|
# at all, which reads as "compile unit" — and the old `-name build` selection
|
||||||
|
# covered that state by real-copying `build/` wholesale, so trusting signal 2
|
||||||
|
# alone was a regression against it. Reproduced on cargo 1.93.1 stable: the
|
||||||
|
# clone's build wrote through the shared inode into the source's OUT_DIR.
|
||||||
|
# Signal 1 closes it, because a failed build script's OUT_DIR holds no rlib.
|
||||||
|
#
|
||||||
|
# The residual is a build script that writes a file NAMED `*.rlib`/`*.rmeta`
|
||||||
|
# into `OUT_DIR` and has never once succeeded. Nothing bounds that away; it is
|
||||||
|
# simply far narrower than what it replaces.
|
||||||
|
#
|
||||||
|
# Requiring signal 1 also means a bin, test or build-script COMPILE unit's
|
||||||
|
# `out` is real-copied rather than shared — at no cost in bytes, since
|
||||||
|
# everything in one is an executable or a `*.d`, and both are privately owned
|
||||||
|
# by the file rules below either way.
|
||||||
|
_mutable_dirs() {
|
||||||
|
local root="$1" d unit
|
||||||
|
while IFS= read -r d; do
|
||||||
|
if [ "${d##*/}" = out ]; then
|
||||||
|
unit="${d%/out}"
|
||||||
|
if ! [ -d "$unit/run" ] && ! [ -e "$unit/root-output" ] \
|
||||||
|
&& _holds_compiled_artifact "$d"; then
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
printf '%s\n' "$d"
|
||||||
|
done < <(find "$root" -type d \
|
||||||
|
\( -name .fingerprint -o -name fingerprint -o -name run -o -name out \) \
|
||||||
|
-prune -print 2>/dev/null)
|
||||||
|
}
|
||||||
|
|
||||||
# THE load-bearing function of this whole design.
|
# THE load-bearing function of this whole design.
|
||||||
#
|
#
|
||||||
# A hardlink clone is only safe if every write the clone's build performs
|
# A hardlink clone is only safe if every write the clone's build performs
|
||||||
# lands on a NEW inode, leaving the source's data untouched. That is true for
|
# lands on a NEW inode, leaving the source's data untouched. That is true of
|
||||||
# compilation artifacts — rustc and the linker replace `deps/*.rlib`,
|
# rustc's own outputs — it writes an `.rlib` or `.rmeta` to a temporary and
|
||||||
# `*.rmeta`, and binaries rather than truncating them in place — and it is
|
# renames it into place — and it is NOT true of the metadata Cargo and build
|
||||||
# NOT true for the metadata Cargo and build scripts write with a plain
|
# scripts write with a plain truncating write, nor of anything the LINKER
|
||||||
# truncating write. Measured directly (Linux, ext4, cargo 1.9x nightly:
|
# produces. Measured directly (Linux, ext4: `cp -al` a warm target dir, change
|
||||||
# `cp -al` a warm target dir, change a source file, build in the clone, diff
|
# a source file, build in the clone, diff the source) the following files in
|
||||||
# the source) the following files in the SOURCE were mutated through the
|
# the SOURCE were mutated through the shared inode:
|
||||||
# shared inode:
|
|
||||||
#
|
#
|
||||||
# <profile>/.fingerprint/<unit>/dep-<target> (build-dir layout v1) — or,
|
# <profile>/.fingerprint/<unit>/dep-<target> (build-dir layout v1) — or,
|
||||||
# <profile>/build/<pkg>/<hash>/fingerprint/dep-<target>
|
# <profile>/build/<pkg>/<hash>/fingerprint/dep-<target>
|
||||||
@@ -347,6 +413,20 @@ _unshare_files() {
|
|||||||
# plain fs::write)
|
# plain fs::write)
|
||||||
# <profile>/deps/*.d, <profile>/*.d (Cargo's post-processed
|
# <profile>/deps/*.d, <profile>/*.d (Cargo's post-processed
|
||||||
# dep-info)
|
# dep-info)
|
||||||
|
# <profile>/deps/<test>-<hash> (a linked TEST binary; under
|
||||||
|
# layout v2,
|
||||||
|
# `build/<pkg>/<hash>/out/`)
|
||||||
|
#
|
||||||
|
# THE LINKED-OUTPUT CASE IS NOT LAYOUT-SPECIFIC AND WAS NOT PART OF THIS
|
||||||
|
# FUNCTION UNTIL 2026-08-27 (gitdan-actions#14). A `cargo test --no-run` inside
|
||||||
|
# a raw `cp -al` clone rewrote the source's own test binary in place on cargo
|
||||||
|
# 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly (layout v1) and 1.100.0-nightly
|
||||||
|
# (layout v2) alike. Cargo re-creates the path first when it also has to uplift
|
||||||
|
# the result — a bin target's `deps/<bin>-<hash>` has a hardlink twin at
|
||||||
|
# `<profile>/<bin>` — and other crate shapes relinked to a fresh inode for
|
||||||
|
# reasons this measurement did not pin down. Since the safe cases could not be
|
||||||
|
# enumerated, every executable is treated as mutable; `.rlib`, `.rmeta` and
|
||||||
|
# `incremental/` are what stay shared, and they are the bytes worth sharing.
|
||||||
#
|
#
|
||||||
# The checksum-freshness case is not a cosmetic one. Reproduced end to end:
|
# The checksum-freshness case is not a cosmetic one. Reproduced end to end:
|
||||||
# branch B clones base's cache, builds its own content, and thereby rewrites
|
# branch B clones base's cache, builds its own content, and thereby rewrites
|
||||||
@@ -390,25 +470,20 @@ _unshare_files() {
|
|||||||
# included. Bracketed locally: cargo 1.97.1 and 1.98.0-nightly write v1,
|
# included. Bracketed locally: cargo 1.97.1 and 1.98.0-nightly write v1,
|
||||||
# 1.100.0-nightly writes v2.
|
# 1.100.0-nightly writes v2.
|
||||||
#
|
#
|
||||||
# The `-name .fingerprint` clause below therefore matches nothing under a v2
|
# Until 2026-08-27 the selection was `-name .fingerprint -o -name build`,
|
||||||
# Cargo, and the dep-info file is covered only because the `-name build` clause
|
# which under a v2 Cargo matched nothing on its first clause and the entire
|
||||||
# happens to swallow its new home. That is belt-and-braces by accident, not by
|
# tree on its second, because the artifacts moved under `build/` too. The guard
|
||||||
# design — and the same accident makes this function real-copy essentially the
|
# held by accident and the saving did not: on one scratch crate (serde +
|
||||||
# whole tree, because the artifacts moved under `build/` too. Measured on one
|
# serde_json + regex plus a build script), same sources both ways —
|
||||||
# scratch crate (serde + serde_json + regex), same sources both ways:
|
|
||||||
#
|
#
|
||||||
# cargo 1.97.1 (layout v1) 27.0 MB unshared of 126.7 MB — 21.3%
|
# cargo 1.98.0-nightly (layout v1) 64.9 MB unshared of 165.1 MB — 39.3%
|
||||||
# 1.100.0-nightly (layout v2) 105.9 MB unshared of 105.9 MB — 99.998%
|
# 1.100.0-nightly (layout v2) 110.4 MB unshared of 110.4 MB — 99.996%
|
||||||
#
|
#
|
||||||
# So the guard still holds and the saving does not. Deliberately NOT fixed
|
# The selection now names the mutable set directly rather than by the container
|
||||||
# here: adjusting the selection is a change to what gets hardlinked on every
|
# it used to live in, so it holds under both layouts; the same crate measures
|
||||||
# consumer, which wants its own change and its own review, and the deadline is
|
# 45.3% (v1) and 38.4% (v2), both dominated by the linked-output rule above
|
||||||
# cargo 1.100.0 stable on 2026-11-12. Tracked as gitdan-actions#14.
|
# rather than by the layout. On a real 5.5 GB Bevy target dir the whole change
|
||||||
#
|
# moves the real-copied share from 9.0% to 14.1%.
|
||||||
# The historical v1 figure this block used to quote stands as measured: on a
|
|
||||||
# 6.9 GB Bevy workspace target dir the unshared set was .fingerprint 22 MB +
|
|
||||||
# build/ 237 MB + a handful of dep-info files — about 3.7% of the tree, against
|
|
||||||
# 100% for a plain `cp -a`. It describes layout v1 only.
|
|
||||||
#
|
#
|
||||||
# `incremental/` is deliberately left shared: rustc writes each incremental
|
# `incremental/` is deliberately left shared: rustc writes each incremental
|
||||||
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
||||||
@@ -420,10 +495,9 @@ unshare_mutable_paths() {
|
|||||||
[ -d "$root" ] || return 0
|
[ -d "$root" ] || return 0
|
||||||
# The list is materialised in full before anything is replaced: each
|
# The list is materialised in full before anything is replaced: each
|
||||||
# replacement deletes and recreates a directory, and a live `find` walk over
|
# replacement deletes and recreates a directory, and a live `find` walk over
|
||||||
# a tree being mutated underneath it is a needless hazard. `-prune` keeps a
|
# a tree being mutated underneath it is a needless hazard.
|
||||||
# match's own contents out of the list.
|
|
||||||
local -a dirs=()
|
local -a dirs=()
|
||||||
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
mapfile -t dirs < <(_mutable_dirs "$root")
|
||||||
for d in "${dirs[@]}"; do
|
for d in "${dirs[@]}"; do
|
||||||
[ -n "$d" ] || continue
|
[ -n "$d" ] || continue
|
||||||
unshare_subtree "$d" || {
|
unshare_subtree "$d" || {
|
||||||
@@ -435,6 +509,37 @@ unshare_mutable_paths() {
|
|||||||
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
# Layout v1's build-script run metadata, which v2 groups under `run/` and v1
|
||||||
|
# leaves loose in the run unit's directory. `invoked.timestamp` is empty and
|
||||||
|
# carries its meaning in its mtime, which a shared inode carries too.
|
||||||
|
_unshare_files "$root" -type f \
|
||||||
|
\( -name output -o -name root-output -o -name stderr -o -name invoked.timestamp \) || {
|
||||||
|
echo "::error::unshare_mutable_paths: failed to unshare build-script run metadata under ${root}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
# Linked outputs. Unlike an rlib or an rmeta — which rustc writes to a
|
||||||
|
# temporary and renames into place — an executable or shared object is
|
||||||
|
# written by the LINKER, and the linker writes THROUGH an existing inode.
|
||||||
|
# Measured 2026-08-27 on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly
|
||||||
|
# (layout v1) and 1.100.0-nightly (e8cb624d5, layout v2), mold and the
|
||||||
|
# default linker alike: a `cargo test --no-run` binary in a `cp -al` clone
|
||||||
|
# rewrote the SOURCE's copy of itself in place, under both layouts.
|
||||||
|
#
|
||||||
|
# What separates that from the executables measured INTACT is not
|
||||||
|
# established. Every intact case observed was one Cargo has to re-create
|
||||||
|
# anyway to maintain an uplift hardlink — a bin target's
|
||||||
|
# `deps/<bin>-<hash>`, twinned at `<profile>/<bin>`. Whether the twin is the
|
||||||
|
# mechanism or a correlate of it was not determined, and the rule below does
|
||||||
|
# not depend on the answer: exempting twinned executables would recover no
|
||||||
|
# bytes this function newly copies. See gitdan-actions#17.
|
||||||
|
#
|
||||||
|
# The executable bit is the discriminator because it is the linker's own
|
||||||
|
# output that is at risk, not the directory it happens to land in — `.rlib`,
|
||||||
|
# `.rmeta` and `incremental/` stay shared and they are the bytes that matter.
|
||||||
|
_unshare_files "$root" -type f -perm -u+x || {
|
||||||
|
echo "::error::unshare_mutable_paths: failed to unshare linked outputs under ${root}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
||||||
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
||||||
return 1
|
return 1
|
||||||
|
|||||||
@@ -28,8 +28,6 @@ set -euo pipefail
|
|||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
. "$script_dir/cache-lib.sh"
|
. "$script_dir/cache-lib.sh"
|
||||||
|
|
||||||
command -v cargo >/dev/null || { echo "SKIP: no cargo on PATH"; exit 0; }
|
|
||||||
|
|
||||||
scratch=$(mktemp -d)
|
scratch=$(mktemp -d)
|
||||||
trap 'rm -rf "$scratch"' EXIT
|
trap 'rm -rf "$scratch"' EXIT
|
||||||
pass_count=0
|
pass_count=0
|
||||||
@@ -59,6 +57,14 @@ version = "0.1.0"
|
|||||||
edition = "2021"
|
edition = "2021"
|
||||||
[workspace]
|
[workspace]
|
||||||
TOML
|
TOML
|
||||||
|
# A binary as well as a library, because the two are written differently and
|
||||||
|
# only one of them is safe to share: rustc writes an rlib to a temporary and
|
||||||
|
# renames it into place, while the LINKER writes an executable through the
|
||||||
|
# existing inode. Without a bin target this suite never relinks anything and
|
||||||
|
# cannot see that difference.
|
||||||
|
cat > "$dir/src/main.rs" <<'RS'
|
||||||
|
fn main() { println!("{}", probe::f()); }
|
||||||
|
RS
|
||||||
cat > "$dir/build.rs" <<'RS'
|
cat > "$dir/build.rs" <<'RS'
|
||||||
use std::{env, fs, path::PathBuf};
|
use std::{env, fs, path::PathBuf};
|
||||||
fn main() {
|
fn main() {
|
||||||
@@ -70,6 +76,219 @@ fn main() {
|
|||||||
RS
|
RS
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Both build-dir layouts, without a compiler
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# Every other scenario in this file runs whichever layout the installed Cargo
|
||||||
|
# happens to write, so on any one machine it exercises exactly ONE of the two —
|
||||||
|
# and gitdan-ci's is v1. The two fixtures below reproduce both directory shapes
|
||||||
|
# from files alone, clone them through the real `hardlink_clone_into()`, and
|
||||||
|
# assert file by file which side of the partition each one lands on.
|
||||||
|
#
|
||||||
|
# Shapes taken from a scratch crate (serde + serde_json + regex, plus a build
|
||||||
|
# script) built on 2026-08-27: cargo 1.98.0-nightly (a335d47ff 2026-06-26)
|
||||||
|
# writes v1, cargo 1.100.0-nightly (e8cb624d5 2026-08-22) writes v2.
|
||||||
|
#
|
||||||
|
# v2 is where the partition is easy to get wrong, and the fixtures are built to
|
||||||
|
# say so: a build script's OUT_DIR and a compile unit's rlib are BOTH a
|
||||||
|
# directory called `out`, one directory apart, and they need opposite
|
||||||
|
# treatment.
|
||||||
|
mkfile() { mkdir -p "$(dirname "$1")"; printf '%s' "$2" > "$1"; }
|
||||||
|
|
||||||
|
# Big enough that the byte-fraction assertion below measures something.
|
||||||
|
artifact_bytes=$(head -c 4096 /dev/zero | tr '\0' 'A')
|
||||||
|
|
||||||
|
# Spec lines are `<shared|private>|<path relative to the tree root>`.
|
||||||
|
assert_partition() {
|
||||||
|
local label="$1" src="$2" spec="$3"
|
||||||
|
local clone="${src}-clone" want rel si ci total=0 copied=0 sz
|
||||||
|
hardlink_clone_into "$src" "$clone" "selftest-$label" \
|
||||||
|
|| fail "$label: hardlink_clone_into refused the destination"
|
||||||
|
while IFS='|' read -r want rel; do
|
||||||
|
[ -n "${rel:-}" ] || continue
|
||||||
|
[ -e "$clone/$rel" ] || fail "$label: $rel is missing from the clone"
|
||||||
|
si=$(stat -c '%i' "$src/$rel"); ci=$(stat -c '%i' "$clone/$rel")
|
||||||
|
case "$want" in
|
||||||
|
shared)
|
||||||
|
[ "$si" = "$ci" ] \
|
||||||
|
|| fail "$label: $rel was real-copied, but it is an artifact and must stay shared" ;;
|
||||||
|
private)
|
||||||
|
[ "$si" != "$ci" ] \
|
||||||
|
|| fail "$label: $rel still shares an inode with the source, so a build in the clone can rewrite it" ;;
|
||||||
|
*) fail "$label: unknown spec verb '$want'" ;;
|
||||||
|
esac
|
||||||
|
done <<< "$spec"
|
||||||
|
ok "$label: every file landed on the right side of the partition"
|
||||||
|
|
||||||
|
# The cost model, asserted rather than assumed. Selecting too much is not a
|
||||||
|
# correctness bug, which is exactly why nothing caught layout v2 taking the
|
||||||
|
# selection from 39.3% of one scratch crate's tree to 99.996% of it
|
||||||
|
# (gitdan-actions#14):
|
||||||
|
# a hardlink clone that real-copies everything is a `cp -a` with extra steps.
|
||||||
|
# The bound is loose on purpose. It is not a budget — the honest figure moves
|
||||||
|
# with how much of a tree is linker output, and these fixtures are mostly
|
||||||
|
# that by construction — it is a floor under "still a hardlink clone at all".
|
||||||
|
while IFS= read -r rel; do
|
||||||
|
# `.cargo-*lock*` is stripped from every clone by design, so it has no
|
||||||
|
# counterpart to compare against.
|
||||||
|
[ -e "$clone/$rel" ] || continue
|
||||||
|
sz=$(stat -c '%s' "$src/$rel")
|
||||||
|
total=$((total + sz))
|
||||||
|
[ "$(stat -c '%i' "$src/$rel")" = "$(stat -c '%i' "$clone/$rel")" ] || copied=$((copied + sz))
|
||||||
|
done < <(cd "$src" && find . -type f -printf '%P\n')
|
||||||
|
[ "$total" -gt 0 ] || fail "$label: fixture has no bytes to measure"
|
||||||
|
[ $((copied * 100 / total)) -lt 90 ] \
|
||||||
|
|| fail "$label: the clone real-copied $((copied * 100 / total))% of its source's bytes — the hardlink saving is gone"
|
||||||
|
ok "$label: clone real-copies $((copied * 100 / total))% of ${total} B (${copied} B), the rest is shared"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Layout v2: no `.fingerprint`, no `deps`. Everything regroups per build unit
|
||||||
|
# under `build/<pkg>/<hash>/{fingerprint,out,run}`, artifacts included — which
|
||||||
|
# is what took `-name build` from "the metadata" to "the whole tree".
|
||||||
|
#
|
||||||
|
# THE THREE UNIT KINDS ARE THE POINT. `1bf...` is a compile unit: its `out`
|
||||||
|
# holds the rlib. `08c...` is the build script's own compile unit: its `out`
|
||||||
|
# holds the build-script binary. `da9...` is the build-script RUN unit: its
|
||||||
|
# `out` IS the OUT_DIR, and it is the only one of the three whose `out` is
|
||||||
|
# mutable. The `run/` directory beside it is the structural difference.
|
||||||
|
v2="$scratch/layout-v2"
|
||||||
|
mkfile "$v2/CACHEDIR.TAG" 'Signature: 8a477f597d28d172'
|
||||||
|
mkfile "$v2/.rustc_info.json" '{"rustc_fingerprint":1}'
|
||||||
|
mkfile "$v2/debug/.cargo-lock" ''
|
||||||
|
mkfile "$v2/debug/libprobe.rlib" "$artifact_bytes"
|
||||||
|
mkfile "$v2/debug/libprobe.d" '/probe/src/lib.rs:'
|
||||||
|
for f in dep-lib-probe lib-probe lib-probe.json invoked.timestamp; do
|
||||||
|
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/fingerprint/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rlib" "$artifact_bytes"
|
||||||
|
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rmeta" "$artifact_bytes"
|
||||||
|
mkfile "$v2/debug/build/probe/1bf5493368dce3cd/out/probe-1bf5493368dce3cd.d" '/probe/src/lib.rs:'
|
||||||
|
for f in build-script-build-script-build build-script-build-script-build.json \
|
||||||
|
dep-build-script-build-script-build invoked.timestamp; do
|
||||||
|
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/fingerprint/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build" "$artifact_bytes"
|
||||||
|
chmod +x "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build"
|
||||||
|
mkfile "$v2/debug/build/probe/08c7dda6eacd6dca/out/build_script_build.d" '/probe/build.rs:'
|
||||||
|
# A test binary: the same `out` directory as the rlib above, and the largest
|
||||||
|
# thing in a real tree that a linker writes.
|
||||||
|
for f in dep-test-lib-probe test-lib-probe test-lib-probe.json invoked.timestamp; do
|
||||||
|
mkfile "$v2/debug/build/probe/6a091d813b2be60d/fingerprint/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d" "$artifact_bytes"
|
||||||
|
chmod +x "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d"
|
||||||
|
mkfile "$v2/debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d.d" '/probe/src/lib.rs:'
|
||||||
|
for f in run-build-script-build-script-build run-build-script-build-script-build.json; do
|
||||||
|
mkfile "$v2/debug/build/probe/da96cf45111f80dd/fingerprint/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v2/debug/build/probe/da96cf45111f80dd/out/gen.txt" 'generated from 24 bytes'
|
||||||
|
for f in invoked.timestamp root-output stdout stderr; do
|
||||||
|
mkfile "$v2/debug/build/probe/da96cf45111f80dd/run/$f" "$f"
|
||||||
|
done
|
||||||
|
# A build script that wrote into OUT_DIR and then FAILED: Cargo records the
|
||||||
|
# run only on success, so this unit has `out/` populated and no `run/` at all.
|
||||||
|
# Reading "no execution record" as "compile unit" left this shared, which is
|
||||||
|
# the one state the old `-name build` selection covered and the first cut of
|
||||||
|
# this one did not.
|
||||||
|
mkfile "$v2/debug/build/probe/f00ded00f00ded00/out/gen.txt" 'half-written'
|
||||||
|
mkfile "$v2/debug/incremental/probe-abc/s-xyz/dep-graph.bin" "$artifact_bytes"
|
||||||
|
|
||||||
|
assert_partition "layout v2" "$v2" "$(cat <<'SPEC'
|
||||||
|
private|.rustc_info.json
|
||||||
|
private|debug/libprobe.d
|
||||||
|
private|debug/build/probe/1bf5493368dce3cd/fingerprint/dep-lib-probe
|
||||||
|
private|debug/build/probe/1bf5493368dce3cd/fingerprint/lib-probe
|
||||||
|
private|debug/build/probe/1bf5493368dce3cd/fingerprint/lib-probe.json
|
||||||
|
private|debug/build/probe/1bf5493368dce3cd/fingerprint/invoked.timestamp
|
||||||
|
private|debug/build/probe/1bf5493368dce3cd/out/probe-1bf5493368dce3cd.d
|
||||||
|
private|debug/build/probe/08c7dda6eacd6dca/fingerprint/dep-build-script-build-script-build
|
||||||
|
private|debug/build/probe/08c7dda6eacd6dca/fingerprint/invoked.timestamp
|
||||||
|
private|debug/build/probe/08c7dda6eacd6dca/out/build_script_build.d
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/fingerprint/run-build-script-build-script-build
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/out/gen.txt
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/run/root-output
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/run/stdout
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/run/stderr
|
||||||
|
private|debug/build/probe/da96cf45111f80dd/run/invoked.timestamp
|
||||||
|
private|debug/build/probe/f00ded00f00ded00/out/gen.txt
|
||||||
|
shared|debug/libprobe.rlib
|
||||||
|
shared|debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rlib
|
||||||
|
shared|debug/build/probe/1bf5493368dce3cd/out/libprobe-1bf5493368dce3cd.rmeta
|
||||||
|
private|debug/build/probe/08c7dda6eacd6dca/out/build_script_build
|
||||||
|
private|debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d
|
||||||
|
private|debug/build/probe/6a091d813b2be60d/out/probe-6a091d813b2be60d.d
|
||||||
|
private|debug/build/probe/6a091d813b2be60d/fingerprint/dep-test-lib-probe
|
||||||
|
shared|debug/incremental/probe-abc/s-xyz/dep-graph.bin
|
||||||
|
SPEC
|
||||||
|
)"
|
||||||
|
# Layout v1: one `.fingerprint` and one `deps` per profile; `build/<pkg>-<hash>`
|
||||||
|
# holds the build script's compiled binary in one unit directory and its run
|
||||||
|
# metadata plus OUT_DIR in another.
|
||||||
|
v1="$scratch/layout-v1"
|
||||||
|
mkfile "$v1/CACHEDIR.TAG" 'Signature: 8a477f597d28d172'
|
||||||
|
mkfile "$v1/.rustc_info.json" '{"rustc_fingerprint":1}'
|
||||||
|
mkfile "$v1/debug/.cargo-lock" ''
|
||||||
|
mkfile "$v1/debug/libprobe.rlib" "$artifact_bytes"
|
||||||
|
mkfile "$v1/debug/libprobe.d" '/probe/src/lib.rs:'
|
||||||
|
mkfile "$v1/debug/deps/libprobe-1bf5493368dce3cd.rlib" "$artifact_bytes"
|
||||||
|
mkfile "$v1/debug/deps/libprobe-1bf5493368dce3cd.rmeta" "$artifact_bytes"
|
||||||
|
mkfile "$v1/debug/deps/probe-1bf5493368dce3cd.d" '/probe/src/lib.rs:'
|
||||||
|
# A test binary, which layout v1 leaves in `deps/` beside the rlibs.
|
||||||
|
mkfile "$v1/debug/deps/probe-6a091d813b2be60d" "$artifact_bytes"
|
||||||
|
chmod +x "$v1/debug/deps/probe-6a091d813b2be60d"
|
||||||
|
mkfile "$v1/debug/deps/probe-6a091d813b2be60d.d" '/probe/src/lib.rs:'
|
||||||
|
for f in dep-lib-probe lib-probe lib-probe.json invoked.timestamp; do
|
||||||
|
mkfile "$v1/debug/.fingerprint/probe-1bf5493368dce3cd/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build-script-build" "$artifact_bytes"
|
||||||
|
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca" "$artifact_bytes"
|
||||||
|
chmod +x "$v1/debug/build/probe-08c7dda6eacd6dca/build-script-build" \
|
||||||
|
"$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca"
|
||||||
|
mkfile "$v1/debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca.d" '/probe/build.rs:'
|
||||||
|
for f in invoked.timestamp output root-output stderr; do
|
||||||
|
mkfile "$v1/debug/build/probe-da96cf45111f80dd/$f" "$f"
|
||||||
|
done
|
||||||
|
mkfile "$v1/debug/build/probe-da96cf45111f80dd/out/gen.txt" 'generated from 24 bytes'
|
||||||
|
# The same never-succeeded build script under layout v1.
|
||||||
|
mkfile "$v1/debug/build/probe-f00ded00f00ded00/out/gen.txt" 'half-written'
|
||||||
|
mkfile "$v1/debug/incremental/probe-abc/s-xyz/dep-graph.bin" "$artifact_bytes"
|
||||||
|
|
||||||
|
assert_partition "layout v1" "$v1" "$(cat <<'SPEC'
|
||||||
|
private|.rustc_info.json
|
||||||
|
private|debug/libprobe.d
|
||||||
|
private|debug/deps/probe-1bf5493368dce3cd.d
|
||||||
|
private|debug/.fingerprint/probe-1bf5493368dce3cd/dep-lib-probe
|
||||||
|
private|debug/.fingerprint/probe-1bf5493368dce3cd/lib-probe
|
||||||
|
private|debug/.fingerprint/probe-1bf5493368dce3cd/lib-probe.json
|
||||||
|
private|debug/.fingerprint/probe-1bf5493368dce3cd/invoked.timestamp
|
||||||
|
private|debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca.d
|
||||||
|
private|debug/deps/probe-6a091d813b2be60d
|
||||||
|
private|debug/deps/probe-6a091d813b2be60d.d
|
||||||
|
private|debug/build/probe-da96cf45111f80dd/invoked.timestamp
|
||||||
|
private|debug/build/probe-da96cf45111f80dd/output
|
||||||
|
private|debug/build/probe-da96cf45111f80dd/root-output
|
||||||
|
private|debug/build/probe-da96cf45111f80dd/stderr
|
||||||
|
private|debug/build/probe-da96cf45111f80dd/out/gen.txt
|
||||||
|
private|debug/build/probe-f00ded00f00ded00/out/gen.txt
|
||||||
|
shared|debug/libprobe.rlib
|
||||||
|
shared|debug/deps/libprobe-1bf5493368dce3cd.rlib
|
||||||
|
shared|debug/deps/libprobe-1bf5493368dce3cd.rmeta
|
||||||
|
private|debug/build/probe-08c7dda6eacd6dca/build-script-build
|
||||||
|
private|debug/build/probe-08c7dda6eacd6dca/build_script_build-08c7dda6eacd6dca
|
||||||
|
shared|debug/incremental/probe-abc/s-xyz/dep-graph.bin
|
||||||
|
SPEC
|
||||||
|
)"
|
||||||
|
|
||||||
|
|
||||||
|
echo
|
||||||
|
command -v cargo >/dev/null || {
|
||||||
|
echo "SKIP: no cargo on PATH — the fixture scenarios above ran, the live-Cargo ones cannot"
|
||||||
|
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
crate_dir="$scratch/probe"
|
crate_dir="$scratch/probe"
|
||||||
mkcrate "$crate_dir"
|
mkcrate "$crate_dir"
|
||||||
cd "$crate_dir"
|
cd "$crate_dir"
|
||||||
@@ -304,27 +523,44 @@ build_base "$base_fix"
|
|||||||
before=$(snapshot_tree "$base_fix")
|
before=$(snapshot_tree "$base_fix")
|
||||||
hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_into reported the destination already existed"
|
hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_into reported the destination already existed"
|
||||||
|
|
||||||
# The clone's contract, asserted before anything builds in it: artifacts
|
# The clone's contract, asserted before anything builds in it and asserted in
|
||||||
# share inodes (that is what makes the clone near-free), and every file Cargo
|
# BOTH directions: every file Cargo rewrites in place is privately owned (that
|
||||||
# rewrites in place does not (that is what makes it sound). Checking after a
|
# is what makes the clone sound), and every artifact still shares its inode
|
||||||
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
# (that is what makes it near-free). Checking after a rebuild would prove
|
||||||
shared=0; unshared=0
|
# nothing — the rebuild replaces those files anyway.
|
||||||
|
#
|
||||||
|
# The mutable-family patterns cover both layouts: `.fingerprint/` is v1's,
|
||||||
|
# `fingerprint/` and `run/` are v2's, and `gen.txt` is this crate's build
|
||||||
|
# script's OUT_DIR product, which under v2 sits in a directory called `out`
|
||||||
|
# beside sibling units whose `out` holds artifacts.
|
||||||
|
shared=0; unshared=0; shared_bytes=0; copied_bytes=0
|
||||||
while IFS= read -r f; do
|
while IFS= read -r f; do
|
||||||
rel="${f#"$base_fix"/}"
|
rel="${f#"$base_fix"/}"
|
||||||
[ -e "$clone_fix/$rel" ] || continue
|
[ -e "$clone_fix/$rel" ] || continue
|
||||||
|
sz=$(stat -c '%s' "$f")
|
||||||
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
||||||
case "$rel" in
|
case "$rel" in
|
||||||
*/.fingerprint/*|*/build/*|*.d|.rustc_info.json)
|
*/.fingerprint/*|*/fingerprint/*|*/run/*|*/out/gen.txt|*/output|*/root-output|*/stderr|*/invoked.timestamp|*.d|.rustc_info.json)
|
||||||
fail "mutable path still shares an inode with the source: $rel" ;;
|
fail "mutable path still shares an inode with the source: $rel" ;;
|
||||||
esac
|
esac
|
||||||
shared=$((shared + 1))
|
shared=$((shared + 1)); shared_bytes=$((shared_bytes + sz))
|
||||||
else
|
else
|
||||||
unshared=$((unshared + 1))
|
case "$rel" in
|
||||||
|
*.rlib|*.rmeta)
|
||||||
|
fail "artifact was real-copied rather than shared: $rel" ;;
|
||||||
|
esac
|
||||||
|
unshared=$((unshared + 1)); copied_bytes=$((copied_bytes + sz))
|
||||||
fi
|
fi
|
||||||
done < <(find "$base_fix" -type f)
|
done < <(find "$base_fix" -type f)
|
||||||
[ "$shared" -gt 0 ] || fail "nothing is shared — the clone degenerated into a full copy"
|
[ "$shared" -gt 0 ] || fail "nothing is shared — the clone degenerated into a full copy"
|
||||||
[ "$unshared" -gt 0 ] || fail "nothing was unshared — unshare_mutable_paths did not run"
|
[ "$unshared" -gt 0 ] || fail "nothing was unshared — unshare_mutable_paths did not run"
|
||||||
|
total_bytes=$((shared_bytes + copied_bytes))
|
||||||
ok "fresh clone shares ${shared} artifact files and privately owns ${unshared} mutable ones"
|
ok "fresh clone shares ${shared} artifact files and privately owns ${unshared} mutable ones"
|
||||||
|
# Reported, not asserted. This crate has no dependencies, so almost all of its
|
||||||
|
# bytes are the two executables — a ratio that says nothing about a real tree.
|
||||||
|
# The fixtures above are where the cost model is gated, because there the
|
||||||
|
# composition is fixed.
|
||||||
|
echo " note: this clone real-copies $((copied_bytes * 100 / total_bytes))% of ${total_bytes} B"
|
||||||
|
|
||||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||||
CARGO_TARGET_DIR="$clone_fix" "${CARGO_BIN[@]}" build -q
|
CARGO_TARGET_DIR="$clone_fix" "${CARGO_BIN[@]}" build -q
|
||||||
@@ -337,6 +573,91 @@ if [ -n "$fix_mutated" ]; then
|
|||||||
fi
|
fi
|
||||||
ok "no file in the source changed after a full rebuild in the clone"
|
ok "no file in the source changed after a full rebuild in the clone"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== a linked TEST binary, which nothing uplifts and nothing replaces ==="
|
||||||
|
# The one artifact family that is NOT safe to share, and the reason
|
||||||
|
# `unshare_mutable_paths` privately owns every executable. rustc writes an
|
||||||
|
# rlib to a temporary and renames it in; the LINKER writes an executable
|
||||||
|
# through whatever inode is already at the path.
|
||||||
|
#
|
||||||
|
# THE CRATE SHAPE IS LOAD-BEARING AND WAS WRONG ONCE. An earlier cut of this
|
||||||
|
# scenario reused the lib+bin probe crate above, whose test binaries relink to
|
||||||
|
# a FRESH inode — a shape gitdan-actions#17 records as measured safe. Both
|
||||||
|
# halves then passed green against the unfixed selection, on the strength of
|
||||||
|
# dep-info mutations the previous scenario already covers, and the scenario
|
||||||
|
# pinned nothing. A bin-only crate with a unit test does exhibit the rewrite,
|
||||||
|
# on cargo 1.93.1 stable and on 1.98.0-nightly and 1.100.0-nightly, so that is
|
||||||
|
# what this builds. The shape is chosen by measurement rather than derived:
|
||||||
|
# what separates a rewritten executable from an intact one is not established,
|
||||||
|
# so the only crate shape this scenario may rest on is one observed to exhibit
|
||||||
|
# the rewrite.
|
||||||
|
mkbincrate() {
|
||||||
|
local dir="$1" marker="$2"
|
||||||
|
mkdir -p "$dir/src"
|
||||||
|
cat > "$dir/Cargo.toml" <<'TOML'
|
||||||
|
[package]
|
||||||
|
name = "binprobe"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2021"
|
||||||
|
[workspace]
|
||||||
|
TOML
|
||||||
|
cat > "$dir/src/main.rs" <<RS
|
||||||
|
fn main() { println!("${marker}"); }
|
||||||
|
#[cfg(test)]
|
||||||
|
mod t { #[test] fn a() { assert_eq!("${marker}".len() > 0, true); } }
|
||||||
|
RS
|
||||||
|
}
|
||||||
|
|
||||||
|
bin_dir="$scratch/binprobe"
|
||||||
|
mkbincrate "$bin_dir" MARKER_AAAA
|
||||||
|
cd "$bin_dir"
|
||||||
|
base_exe="$scratch/base-exe"; clone_exe_ctl="$scratch/clone-exe-ctl"; clone_exe="$scratch/clone-exe"
|
||||||
|
|
||||||
|
# Only the executables are read here. The families the other scenarios cover
|
||||||
|
# would satisfy a "something changed" assertion on their own, which is exactly
|
||||||
|
# how the earlier cut of this passed while pinning nothing.
|
||||||
|
source_exe_digest() {
|
||||||
|
(cd "$1" && find . -type f -executable -print0 | sort -z | xargs -0 -r sha1sum) 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
CARGO_TARGET_DIR="$base_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1 \
|
||||||
|
|| fail "the bin-only probe crate failed to build"
|
||||||
|
before=$(source_exe_digest "$base_exe")
|
||||||
|
cp -al "$base_exe" "$clone_exe_ctl"
|
||||||
|
strip_cargo_locks "$clone_exe_ctl"
|
||||||
|
mkbincrate "$bin_dir" MARKER_BBBB
|
||||||
|
CARGO_TARGET_DIR="$clone_exe_ctl" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
|
||||||
|
exe_ctl_mutated=$(mutated_paths "$before" "$(source_exe_digest "$base_exe")")
|
||||||
|
|
||||||
|
# THREE OUTCOMES, as the freshness probe above has, and for the same reason: a
|
||||||
|
# scenario that cannot tell "the fix works" from "the hazard never fired" is
|
||||||
|
# not a gate. If this Cargo does not rewrite the source's test binary, the
|
||||||
|
# assertion below would pass for a toolchain reason rather than a code one, so
|
||||||
|
# it is skipped LOUDLY instead of passing quietly.
|
||||||
|
if [ -z "$exe_ctl_mutated" ]; then
|
||||||
|
echo "::warning::hardlink-clone-selftest: this toolchain did not rewrite the source's test binary through a raw cp -al clone, so the linked-output scenario proves nothing here and was SKIPPED. That is a statement about this Cargo, not about unshare_mutable_paths."
|
||||||
|
else
|
||||||
|
ok "control: a raw cp -al clone rewrites the source's own linked test binary"
|
||||||
|
printf '%s\n' "$exe_ctl_mutated" | sed 's/^/ /'
|
||||||
|
|
||||||
|
# Rebuild the base from the original marker so it is warm and consistent
|
||||||
|
# again, then do the same thing through the real clone.
|
||||||
|
mkbincrate "$bin_dir" MARKER_AAAA
|
||||||
|
CARGO_TARGET_DIR="$base_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
|
||||||
|
before=$(snapshot_tree "$base_exe")
|
||||||
|
hardlink_clone_into "$base_exe" "$clone_exe" "selftest-exe" \
|
||||||
|
|| fail "hardlink_clone_into refused the destination"
|
||||||
|
mkbincrate "$bin_dir" MARKER_BBBB
|
||||||
|
CARGO_TARGET_DIR="$clone_exe" "${CARGO_BIN[@]}" test --no-run -q > /dev/null 2>&1
|
||||||
|
exe_mutated=$(mutated_paths "$before" "$(snapshot_tree "$base_exe")")
|
||||||
|
if [ -n "$exe_mutated" ]; then
|
||||||
|
printf '%s\n' "$exe_mutated" | sed 's/^/ /' >&2
|
||||||
|
fail "a test build in the clone mutated the source through a shared inode"
|
||||||
|
fi
|
||||||
|
ok "no file in the source changed after a full test build in the clone"
|
||||||
|
fi
|
||||||
|
cd "$crate_dir"
|
||||||
|
|
||||||
echo
|
echo
|
||||||
if [ "$CHECKSUM_MODE" = "on" ]; then
|
if [ "$CHECKSUM_MODE" = "on" ]; then
|
||||||
echo "=== the whole point: the source's next build is still correct ==="
|
echo "=== the whole point: the source's next build is still correct ==="
|
||||||
|
|||||||
@@ -53,9 +53,11 @@ pass_count=0
|
|||||||
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
||||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||||
|
|
||||||
# Cargo and rustc REPLACE an artifact (write elsewhere, rename over the path)
|
# rustc REPLACES an `.rlib`/`.rmeta` (writes elsewhere, renames over the path)
|
||||||
# rather than truncating it in place, which is exactly why a snapshot may
|
# rather than truncating it in place, which is exactly why a snapshot may
|
||||||
# share artifact inodes with the live target dir it was cloned from. The
|
# share those inodes with the live target dir it was cloned from. Linker
|
||||||
|
# outputs are the exception and are real-copied instead — see
|
||||||
|
# `unshare_mutable_paths` in cache-lib.sh. The
|
||||||
# fixtures here have to model that faithfully — a plain `>` redirect truncates
|
# fixtures here have to model that faithfully — a plain `>` redirect truncates
|
||||||
# in place and would write straight through the shared inode into the
|
# in place and would write straight through the shared inode into the
|
||||||
# snapshot and every consumer, which is a property of the test fixture, not of
|
# snapshot and every consumer, which is a property of the test fixture, not of
|
||||||
|
|||||||
Reference in New Issue
Block a user