claudeandClaude Fable 5.1
a960c8f91b
The set of paths a hardlink clone has to real-copy — dep-info, build-script metadata, linked outputs, the pruned directories that hold them — was spelled out inline in unshare_mutable_paths, in four find invocations. Nothing else needed it, so one spelling was enough. Something else needs it now: the prune has to know what a clone will cost before it happens, and a sizer with its own copy of the predicates would drift from the copier silently and in the dangerous direction — an under-measured clone is one that starts and runs out of disk halfway through unsharing. So the directory names become one array and the file rules one dispatcher, applied by a callback per side, with each rule's rationale moved to the rule rather than left at the old call site. mutable_set_kb measures that set off a snapshot, skipping the subtrees already measured whole so nothing is counted twice; clone_headroom_kb scales it by a hand-written margin and floor for what the measurement cannot see (cp -al materialising every directory for real, and the unshare holding one subtree twice at its peak). Both residuals are named where the function is, in both directions. seed_source_candidates moves the seed's source-preference list into cache-lib for the same reason: the prune ahead of it has to resolve the same source the seed will clone, and two agreeing derivations are one edit away from disagreeing. No behaviour change — the copier applies the same rules to the same tree, verified by the hardlink-clone suite's inode partition in both directions. Refs #20. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JXMQCJ5Eg5f9G9cfYzyh4Z
112 lines
5.5 KiB
Bash
Executable File
112 lines
5.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Consume side: make this run's own target dir exist, warm, and private.
|
|
#
|
|
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir] [lock-id]
|
|
# own-key cache key for this run's own ref
|
|
# base-key cache key for the ref to layer over ("" for a run whose own
|
|
# ref IS a reference branch)
|
|
# cache-root mount point of the persistent volume
|
|
# tag a per-job-per-run unique string, used to name the staging
|
|
# directory so two concurrent jobs can never collide on it
|
|
# fallback-dir optional absolute path to seed from when no snapshot exists
|
|
# (a legacy flat cache dir during a migration, typically)
|
|
# lock-id optional cache-lock id. When given, the lock marker is
|
|
# written on EVERY path out of this script — including into
|
|
# the staging tree before its rename — so the directory is
|
|
# never observable under its final name without a lock. The
|
|
# action acquires the lock in a later step too; that step is
|
|
# idempotent, and this closes the window before it runs, where
|
|
# a concurrent job's prune pass could evict a directory that
|
|
# exists but is not yet held.
|
|
#
|
|
# The design in one paragraph: a PR branch's first run hardlink-clones the
|
|
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
|
|
# time proportional to inode count rather than data volume — on ext4, with no
|
|
# reflink support, that is the only way to make "layer over the base" cheap.
|
|
# Snapshot rather than the base's live target dir, because a live directory is
|
|
# being written by its own job while a consumer reads it, and a torn read
|
|
# pairs one build's fingerprint with another build's artifact — which degrades
|
|
# to a WRONG reuse, not to a safe miss. Neither half is novel; the combination
|
|
# is what makes the scheme both cheap and sound, instead of one or the other.
|
|
#
|
|
# Everything about correctness under concurrency lives in
|
|
# hardlink_clone_into() and unshare_mutable_paths() in cache-lib.sh — read
|
|
# those before changing anything here.
|
|
set -euo pipefail
|
|
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
|
|
|
if [ $# -lt 4 ] || [ $# -gt 6 ]; then
|
|
echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2
|
|
exit 1
|
|
fi
|
|
|
|
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}"
|
|
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
|
|
|
|
mkdir -p "$ROOT"
|
|
|
|
if [ -d "$OWN_DIR" ]; then
|
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|
|
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
|
|
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
|
|
exit 0
|
|
fi
|
|
|
|
# Source preference, most specific first:
|
|
#
|
|
# 1. the base ref's snapshot — the ordinary PR case, and the whole point.
|
|
# 2. this ref's OWN snapshot — a reference branch whose live target dir was
|
|
# evicted under disk pressure can restore itself from the last snapshot
|
|
# it published, instead of paying a cold rebuild. Free, given the
|
|
# snapshot already exists.
|
|
# 3. an explicit fallback directory — migration off a pre-existing flat
|
|
# cache, so the first run under this scheme isn't a needless cold build.
|
|
#
|
|
# The list itself lives in cache-lib.sh because prune-cache.sh reads it too:
|
|
# it runs ahead of this step and has to free enough disk for the clone below,
|
|
# which means resolving the same source this loop will pick.
|
|
mapfile -t CANDIDATES < <(seed_source_candidates "$ROOT" "$OWN_KEY" "$BASE_KEY" "$FALLBACK")
|
|
|
|
for entry in "${CANDIDATES[@]}"; do
|
|
src="${entry%%:*}"; label="${entry#*:}"
|
|
[ -d "$src" ] || continue
|
|
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
|
|
start=$(date +%s)
|
|
# The status is captured and dispatched on explicitly. Calling this as a
|
|
# bare `if` condition — which is what this script used to do — suppresses
|
|
# `set -e` for the whole call, so a hard clone failure could not abort the
|
|
# seed even in principle; the three outcomes are genuinely distinct and each
|
|
# needs its own handling.
|
|
clone_rc=0
|
|
hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$?
|
|
case "$clone_rc" in
|
|
0)
|
|
echo "seed: cloned in $(( $(date +%s) - start ))s"
|
|
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
|
|
;;
|
|
1)
|
|
# Another job sharing this cache key won the rename while we were
|
|
# cloning. Its directory is complete (the rename is the publish step),
|
|
# so there is nothing to do but use it — and nothing was ever
|
|
# observable in a half-seeded state.
|
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|
|
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
|
|
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
|
|
;;
|
|
*)
|
|
# A source that could not be read consistently. Failing the job is the
|
|
# only safe answer: the alternative that used to happen here was
|
|
# seeding a truncated tree and reporting success, which hands Cargo a
|
|
# directory whose fingerprints and artifacts disagree.
|
|
echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
exit 0
|
|
done
|
|
|
|
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
|
|
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
|
|
mkdir -p "$OWN_DIR"
|
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|