refactor(cache): name the mutable set once, and measure it
The set of paths a hardlink clone has to real-copy — dep-info, build-script metadata, linked outputs, the pruned directories that hold them — was spelled out inline in unshare_mutable_paths, in four find invocations. Nothing else needed it, so one spelling was enough. Something else needs it now: the prune has to know what a clone will cost before it happens, and a sizer with its own copy of the predicates would drift from the copier silently and in the dangerous direction — an under-measured clone is one that starts and runs out of disk halfway through unsharing. So the directory names become one array and the file rules one dispatcher, applied by a callback per side, with each rule's rationale moved to the rule rather than left at the old call site. mutable_set_kb measures that set off a snapshot, skipping the subtrees already measured whole so nothing is counted twice; clone_headroom_kb scales it by a hand-written margin and floor for what the measurement cannot see (cp -al materialising every directory for real, and the unshare holding one subtree twice at its peak). Both residuals are named where the function is, in both directions. seed_source_candidates moves the seed's source-preference list into cache-lib for the same reason: the prune ahead of it has to resolve the same source the seed will clone, and two agreeing derivations are one edit away from disagreeing. No behaviour change — the copier applies the same rules to the same tree, verified by the hardlink-clone suite's inode partition in both directions. Refs #20. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JXMQCJ5Eg5f9G9cfYzyh4Z
This commit is contained in:
Claude Fable 5.1
+187
-40
@@ -330,6 +330,62 @@ _holds_compiled_artifact() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# The directory names that select a mutable subtree, as one find predicate.
|
||||
#
|
||||
# Named once because THREE readers have to agree on it: the selection in
|
||||
# _mutable_dirs, the prune that skips those subtrees when it sizes the file
|
||||
# rules, and anything later that measures what a clone will cost. Two
|
||||
# spellings of this list would size a different tree than the one copied, and
|
||||
# the direction that fails is silent — an under-measured clone runs out of
|
||||
# disk mid-unshare, which is gitdan-actions#20.
|
||||
_MUTABLE_DIR_NAMES=( -name .fingerprint -o -name fingerprint -o -name run -o -name out )
|
||||
|
||||
# _mutable_file_rules <fn>
|
||||
#
|
||||
# The FILE half of the mutable set, applied one rule at a time:
|
||||
#
|
||||
# <fn> <label> <maxdepth|-> <find-predicate...>
|
||||
#
|
||||
# Same reason as the array above — `unshare_mutable_paths` copies these and
|
||||
# `mutable_set_kb` measures them, and a rule that exists in only one of the
|
||||
# two is exactly the under-estimate the headroom gate cannot survive. The
|
||||
# maxdepth is a separate field because GNU find wants it ahead of every other
|
||||
# predicate, so it cannot live inside the predicate vector.
|
||||
#
|
||||
# `-type f` is each caller's to add: the sizer needs it inside the `-o`
|
||||
# alternation it builds, the copier ahead of it.
|
||||
_mutable_file_rules() {
|
||||
local fn="$1"
|
||||
"$fn" 'dep-info files' - -name '*.d' || return 1
|
||||
# Layout v1's build-script run metadata, which v2 groups under `run/` and v1
|
||||
# leaves loose in the run unit's directory. `invoked.timestamp` is empty and
|
||||
# carries its meaning in its mtime, which a shared inode carries too.
|
||||
"$fn" 'build-script run metadata' - \
|
||||
\( -name output -o -name root-output -o -name stderr -o -name invoked.timestamp \) || return 1
|
||||
# Linked outputs. Unlike an rlib or an rmeta — which rustc writes to a
|
||||
# temporary and renames into place — an executable or shared object is
|
||||
# written by the LINKER, and the linker writes THROUGH an existing inode.
|
||||
# Measured 2026-08-27 on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly
|
||||
# (layout v1) and 1.100.0-nightly (e8cb624d5, layout v2), mold and the
|
||||
# default linker alike: a `cargo test --no-run` binary in a `cp -al` clone
|
||||
# rewrote the SOURCE's copy of itself in place, under both layouts.
|
||||
#
|
||||
# What separates that from the executables measured INTACT is not
|
||||
# established. Every intact case observed was one Cargo has to re-create
|
||||
# anyway to maintain an uplift hardlink — a bin target's
|
||||
# `deps/<bin>-<hash>`, twinned at `<profile>/<bin>`. Whether the twin is the
|
||||
# mechanism or a correlate of it was not determined, and the rule below does
|
||||
# not depend on the answer: exempting twinned executables would recover no
|
||||
# bytes this function newly copies. See gitdan-actions#17.
|
||||
#
|
||||
# The executable bit is the discriminator because it is the linker's own
|
||||
# output that is at risk, not the directory it happens to land in — `.rlib`,
|
||||
# `.rmeta` and `incremental/` stay shared and they are the bytes that matter.
|
||||
"$fn" 'linked outputs' - -perm -u+x || return 1
|
||||
"$fn" '.rustc_info.json' 3 -name '.rustc_info.json' || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
# The directories `unshare_mutable_paths` replaces, under either layout.
|
||||
#
|
||||
# All four names are pruned, so nothing selected here can contain anything else
|
||||
@@ -380,10 +436,24 @@ _mutable_dirs() {
|
||||
fi
|
||||
printf '%s\n' "$d"
|
||||
done < <(find "$root" -type d \
|
||||
\( -name .fingerprint -o -name fingerprint -o -name run -o -name out \) \
|
||||
\( "${_MUTABLE_DIR_NAMES[@]}" \) \
|
||||
-prune -print 2>/dev/null)
|
||||
}
|
||||
|
||||
# _mutable_file_rules' callback for the copying side. The root travels in a
|
||||
# variable rather than an argument because the callback's own signature is the
|
||||
# rule's, and every rule has to reach the same tree.
|
||||
_unshare_one_rule() {
|
||||
local label="$1" maxdepth="$2"; shift 2
|
||||
local -a depth=()
|
||||
[ "$maxdepth" = - ] || depth=(-maxdepth "$maxdepth")
|
||||
_unshare_files "$_MUTABLE_ROOT" ${depth[@]+"${depth[@]}"} -type f "$@" || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare ${label} under ${_MUTABLE_ROOT}" >&2
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
# THE load-bearing function of this whole design.
|
||||
#
|
||||
# A hardlink clone is only safe if every write the clone's build performs
|
||||
@@ -493,6 +563,7 @@ _mutable_dirs() {
|
||||
unshare_mutable_paths() {
|
||||
local root="$1" d
|
||||
[ -d "$root" ] || return 0
|
||||
_MUTABLE_ROOT="$root"
|
||||
# The list is materialised in full before anything is replaced: each
|
||||
# replacement deletes and recreates a directory, and a live `find` walk over
|
||||
# a tree being mutated underneath it is a needless hazard.
|
||||
@@ -505,45 +576,121 @@ unshare_mutable_paths() {
|
||||
return 1
|
||||
}
|
||||
done
|
||||
_unshare_files "$root" -type f -name '*.d' || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
# Layout v1's build-script run metadata, which v2 groups under `run/` and v1
|
||||
# leaves loose in the run unit's directory. `invoked.timestamp` is empty and
|
||||
# carries its meaning in its mtime, which a shared inode carries too.
|
||||
_unshare_files "$root" -type f \
|
||||
\( -name output -o -name root-output -o -name stderr -o -name invoked.timestamp \) || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare build-script run metadata under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
# Linked outputs. Unlike an rlib or an rmeta — which rustc writes to a
|
||||
# temporary and renames into place — an executable or shared object is
|
||||
# written by the LINKER, and the linker writes THROUGH an existing inode.
|
||||
# Measured 2026-08-27 on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly
|
||||
# (layout v1) and 1.100.0-nightly (e8cb624d5, layout v2), mold and the
|
||||
# default linker alike: a `cargo test --no-run` binary in a `cp -al` clone
|
||||
# rewrote the SOURCE's copy of itself in place, under both layouts.
|
||||
#
|
||||
# What separates that from the executables measured INTACT is not
|
||||
# established. Every intact case observed was one Cargo has to re-create
|
||||
# anyway to maintain an uplift hardlink — a bin target's
|
||||
# `deps/<bin>-<hash>`, twinned at `<profile>/<bin>`. Whether the twin is the
|
||||
# mechanism or a correlate of it was not determined, and the rule below does
|
||||
# not depend on the answer: exempting twinned executables would recover no
|
||||
# bytes this function newly copies. See gitdan-actions#17.
|
||||
#
|
||||
# The executable bit is the discriminator because it is the linker's own
|
||||
# output that is at risk, not the directory it happens to land in — `.rlib`,
|
||||
# `.rmeta` and `incremental/` stay shared and they are the bytes that matter.
|
||||
_unshare_files "$root" -type f -perm -u+x || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare linked outputs under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
||||
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
||||
return 1
|
||||
}
|
||||
_mutable_file_rules _unshare_one_rule || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# What the seed will clone, and what that clone costs in disk
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# The sources seed-target-dir.sh considers, most specific first, as
|
||||
# `<dir>:<label>` lines.
|
||||
#
|
||||
# Read by the seed, which clones the first one that exists, and by the prune
|
||||
# that has to size the volume for that clone BEFORE it happens. One derivation
|
||||
# rather than two agreeing ones: a prune that sizes a different tree than the
|
||||
# seed clones is measuring nothing, and nothing downstream would say so.
|
||||
seed_source_candidates() {
|
||||
local root="$1" own_key="$2" base_key="$3" fallback="${4:-}"
|
||||
[ -n "$base_key" ] && printf '%s:base snapshot\n' "$(snapshot_dir_for "$root" "$base_key")"
|
||||
printf '%s:own snapshot\n' "$(snapshot_dir_for "$root" "$own_key")"
|
||||
[ -n "$fallback" ] && printf '%s:fallback dir\n' "$fallback"
|
||||
return 0
|
||||
}
|
||||
|
||||
# The directory the seed will actually hardlink-clone on this run, or nothing
|
||||
# at all when it will not clone: its own target dir already exists (the seed
|
||||
# reuses it and returns before the candidate list is consulted), or no
|
||||
# candidate exists (it starts cold).
|
||||
seed_clone_source() {
|
||||
local root="$1" own_key="$2" base_key="$3" fallback="${4:-}" entry src
|
||||
[ -d "$(target_dir_for "$root" "$own_key")" ] && return 0
|
||||
while IFS= read -r entry; do
|
||||
src="${entry%%:*}"
|
||||
if [ -d "$src" ]; then printf '%s' "$src"; return 0; fi
|
||||
done < <(seed_source_candidates "$root" "$own_key" "$base_key" "$fallback")
|
||||
return 0
|
||||
}
|
||||
|
||||
# _mutable_file_rules' callback for the measuring side.
|
||||
#
|
||||
# One find per rule, skipping the subtrees `_mutable_dirs` already selects
|
||||
# whole — those are measured by the `du` in mutable_set_kb, and counting a
|
||||
# file twice would inflate the requirement into evicting caches nothing
|
||||
# needed. `%k` is allocated 1K blocks, the same unit `du -sk` reports, so the
|
||||
# two halves add.
|
||||
_size_one_rule() {
|
||||
local maxdepth="$2"; shift 2
|
||||
local -a depth=()
|
||||
[ "$maxdepth" = - ] || depth=(-maxdepth "$maxdepth")
|
||||
find "$_MUTABLE_ROOT" ${depth[@]+"${depth[@]}"} \
|
||||
\( "${_MUTABLE_DIR_NAMES[@]}" \) -prune -o \
|
||||
-type f \( "$@" \) -printf '%k\n' 2>/dev/null
|
||||
return 0
|
||||
}
|
||||
|
||||
# The kilobytes `unshare_mutable_paths` will really-copy out of <dir> — the
|
||||
# part of a hardlink clone that costs new disk, as opposed to the `.rlib`,
|
||||
# `.rmeta` and `incremental/` bytes that stay shared with the source.
|
||||
#
|
||||
# Measured off the SAME enumeration the copier uses (`_mutable_dirs` and
|
||||
# `_mutable_file_rules`), which is the only thing that makes this a
|
||||
# measurement rather than an estimate.
|
||||
#
|
||||
# Two residuals, both named because neither is bounded away:
|
||||
#
|
||||
# OVER by any file matching two rules at once — an executable named
|
||||
# `output`, say. Rare, and small.
|
||||
# UNDER by the `*.d` files inside an `out` directory that _mutable_dirs
|
||||
# leaves SHARED (the compile-unit case: it holds an .rlib and has no
|
||||
# build-script record beside it). Such a directory holds a library artifact
|
||||
# by definition, so what is missed is dep-info, not executables. Also under
|
||||
# by a `.rustc_info.json` deeper than the copier's own maxdepth, which is
|
||||
# kilobytes.
|
||||
#
|
||||
# The margin in clone_headroom_kb is what covers the under-count; it is not
|
||||
# there to make the measurement optional.
|
||||
mutable_set_kb() {
|
||||
local root="$1"
|
||||
[ -d "$root" ] || { printf '0'; return 0; }
|
||||
_MUTABLE_ROOT="$root"
|
||||
{
|
||||
_mutable_dirs "$root" | tr '\n' '\0' | xargs -0 -r du -sk 2>/dev/null | awk '{print $1}'
|
||||
_mutable_file_rules _size_one_rule
|
||||
} | awk '{s += $1} END { printf "%d", s + 0 }'
|
||||
return 0
|
||||
}
|
||||
|
||||
# How much free space the seed needs on the volume before it clones <dir>.
|
||||
#
|
||||
# CACHE_CLONE_HEADROOM_PERCENT scales the measured mutable set;
|
||||
# CACHE_CLONE_HEADROOM_FLOOR_KB is added on top. BOTH DEFAULTS ARE
|
||||
# HAND-WRITTEN — nothing measures them, and they are separate because they
|
||||
# cover different things:
|
||||
#
|
||||
# The percentage covers what scales with the tree: `unshare_subtree` stages
|
||||
# each mutable directory through a sibling copy before dropping the shared
|
||||
# original, so at its peak one subtree is held twice, and the under-count
|
||||
# named on mutable_set_kb scales with the tree too.
|
||||
# The floor covers what does not: `cp -al` materialises every DIRECTORY for
|
||||
# real (only files are linked), and a Bevy-sized target dir has hundreds of
|
||||
# thousands of them.
|
||||
#
|
||||
# Both are overridable, and the direction of error is deliberate. Over-asking
|
||||
# evicts a cache that would have fitted, costing one branch a cold start;
|
||||
# under-asking lets the clone start and run out of disk halfway through the
|
||||
# unshare, which fails the job with an error naming a staging path — the
|
||||
# failure gitdan-actions#20 is filed about.
|
||||
CACHE_CLONE_HEADROOM_PERCENT="${CACHE_CLONE_HEADROOM_PERCENT:-150}"
|
||||
CACHE_CLONE_HEADROOM_FLOOR_KB="${CACHE_CLONE_HEADROOM_FLOOR_KB:-2097152}"
|
||||
|
||||
clone_headroom_kb() {
|
||||
local src="${1:-}" kb
|
||||
if [ -z "$src" ] || [ ! -d "$src" ]; then printf '0'; return 0; fi
|
||||
kb=$(mutable_set_kb "$src")
|
||||
awk -v k="$kb" -v pct="$CACHE_CLONE_HEADROOM_PERCENT" -v floor="$CACHE_CLONE_HEADROOM_FLOOR_KB" \
|
||||
'BEGIN { printf "%d", (k * pct / 100) + floor }'
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user