feat(cargo-cache): hardlink-clone a per-ref Cargo cache from a published snapshot
Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Publish side: atomically republish this reference branch's live target dir
|
||||
# as the immutable snapshot other branches seed from.
|
||||
#
|
||||
# Usage: publish-snapshot.sh <own-key> <cache-root> <tag>
|
||||
#
|
||||
# Run only after the build has already succeeded, and only on a branch that is
|
||||
# a publisher (see the cargo-cache-publish action, which makes that decision).
|
||||
# A red build must never overwrite a known-good snapshot: the caller's step
|
||||
# ordering is the whole gate here, so keep this step last.
|
||||
#
|
||||
# The swap is two renames, not one, because POSIX rename() can only replace an
|
||||
# EMPTY directory and a snapshot from a prior publish is not one:
|
||||
#
|
||||
# 1. stage the new snapshot at .stage-<tag> (copy time is off every
|
||||
# consumer's hot path — nothing reads a staging path);
|
||||
# 2. rename the current snapshot aside to .publish-old-<tag>, if present;
|
||||
# 3. rename the staged snapshot into place.
|
||||
#
|
||||
# Step 3 is a single atomic rename onto a path now guaranteed absent, so it
|
||||
# can never partially overwrite a live snapshot. A consumer landing in the
|
||||
# window between 2 and 3 sees the snapshot as absent and falls through to its
|
||||
# own cold-start path — a safe degrade that self-heals on its next run, not
|
||||
# corruption.
|
||||
#
|
||||
# `rm -rf` on the old snapshot removes directory entries only. Any consumer
|
||||
# that already hardlink-cloned from it keeps every inode alive through its own
|
||||
# links, so a republish never pulls data out from under a running job — it
|
||||
# just stops new consumers from seeing the old generation. Disk is reclaimed
|
||||
# when the last clone referencing those inodes is itself evicted.
|
||||
set -euo pipefail
|
||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||
|
||||
OWN_KEY="${1:?usage: publish-snapshot.sh <own-key> <cache-root> <tag>}"
|
||||
ROOT="${2:?}"; TAG="${3:?}"
|
||||
|
||||
SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
|
||||
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
|
||||
OLD="${ROOT}/.publish-old-${TAG}"
|
||||
|
||||
if [ ! -d "$SRC" ]; then
|
||||
echo "publish: no target dir at ${SRC} — nothing to snapshot"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Leftovers from a run cancelled mid-swap. Scoped to this job's own tag so a
|
||||
# concurrently running job's staging directory is never touched.
|
||||
rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
|
||||
|
||||
start=$(date +%s)
|
||||
# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
|
||||
# of every file Cargo rewrites in place (unshare_mutable_paths, called inside
|
||||
# hardlink_clone_into). Without that, this branch's NEXT build would mutate
|
||||
# the snapshot it just published — the same aliasing hazard the consume side
|
||||
# closes, pointing the other way. The staging path is not the final name, so
|
||||
# `hardlink_clone_into`'s rename lands on DST only after OLD is out of the way.
|
||||
TMP_DST="${ROOT}/.publish-new-${TAG}"
|
||||
rm -rf "$TMP_DST"
|
||||
hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {
|
||||
echo "::error::publish: failed to stage a snapshot of ${SRC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
|
||||
mv -T "$TMP_DST" "$DST"
|
||||
rm -rf "$OLD"
|
||||
|
||||
echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
|
||||
summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"
|
||||
Reference in New Issue
Block a user