feat(cargo-cache): hardlink-clone a per-ref Cargo cache from a published snapshot
Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,131 @@
|
||||
name: 'Cargo cache (publish)'
|
||||
description: >-
|
||||
Records this run''s build watermark and, on a publisher branch, atomically
|
||||
republishes its target directory as the immutable snapshot that other
|
||||
branches'' caches are hardlink-cloned from.
|
||||
author: 'gitdan'
|
||||
|
||||
inputs:
|
||||
cache-root:
|
||||
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
||||
required: false
|
||||
default: '/cache'
|
||||
protected-branches:
|
||||
description: >-
|
||||
Space-separated refs that publish snapshots. A run whose own ref is not
|
||||
in this list records its watermark and skips publishing.
|
||||
required: false
|
||||
default: 'dev main'
|
||||
mode:
|
||||
description: >-
|
||||
publish — record the watermark, publish a snapshot if eligible,
|
||||
release this job''s cache lock (the normal call, after a
|
||||
green build).
|
||||
release-lock — release this job''s cache lock and do nothing else. Use
|
||||
in a final `if: always()` step so a failed run does not
|
||||
leave a lock behind for the staleness grace period.
|
||||
required: false
|
||||
default: 'publish'
|
||||
own-ref:
|
||||
description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.'
|
||||
required: false
|
||||
default: ''
|
||||
publish-on-events:
|
||||
description: >-
|
||||
Space-separated event names on which a publisher branch actually
|
||||
publishes. Defaults to `push` — a pull_request run never publishes,
|
||||
because its ref is not the reference branch even when it targets one.
|
||||
required: false
|
||||
default: 'push'
|
||||
record-watermark:
|
||||
description: >-
|
||||
Record this run''s HEAD as the build watermark for this target dir.
|
||||
True for PR runs too, not just publishers: a feature branch accumulates
|
||||
its own build history across several pushes and needs its own watermark.
|
||||
required: false
|
||||
default: 'true'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
# Everything here reads the environment the consume action exported, so a
|
||||
# workflow that forgets to run cargo-cache first fails loudly here rather
|
||||
# than silently publishing a snapshot of the wrong directory.
|
||||
- id: resolve
|
||||
shell: bash
|
||||
env:
|
||||
PROTECTED_BRANCHES: ${{ inputs.protected-branches }}
|
||||
PUBLISH_ON_EVENTS: ${{ inputs.publish-on-events }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# In release-lock mode this action is called from an `if: always()`
|
||||
# step, which can run after a failure that happened before the
|
||||
# cargo-cache action ever executed. Missing environment there means
|
||||
# "there is no lock to release", not an error worth failing the job a
|
||||
# second time over.
|
||||
if [ -z "${CARGO_CACHE_SCRIPTS:-}" ] || [ -z "${CARGO_TARGET_DIR:-}" ]; then
|
||||
if [ "${{ inputs.mode }}" = "release-lock" ]; then
|
||||
echo "cargo-cache-publish: no cache environment in this job — nothing to release"
|
||||
echo "publish=no" >> "$GITHUB_OUTPUT"
|
||||
echo "active=no" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "::error::cargo-cache-publish: run the cargo-cache action earlier in this job"
|
||||
exit 1
|
||||
fi
|
||||
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
||||
echo "active=yes" >> "$GITHUB_OUTPUT"
|
||||
|
||||
OWN_REF="${{ inputs.own-ref }}"
|
||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||
|
||||
# A publisher is a branch other branches layer over. Two conditions,
|
||||
# both required: its ref is in protected-branches, AND the event is one
|
||||
# where this ref really is the reference branch. A pull_request run
|
||||
# from `dev` into `main` has own-ref `dev` and would otherwise publish
|
||||
# a snapshot of a merge-preview build — which is not what `dev` is.
|
||||
PUBLISH=no
|
||||
for ref in $PROTECTED_BRANCHES; do
|
||||
[ "$ref" = "$OWN_REF" ] || continue
|
||||
for ev in $PUBLISH_ON_EVENTS; do
|
||||
[ "$ev" = "${{ github.event_name }}" ] && PUBLISH=yes
|
||||
done
|
||||
done
|
||||
echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT"
|
||||
echo "own-ref=${OWN_REF}" >> "$GITHUB_OUTPUT"
|
||||
echo "publisher check: ref '${OWN_REF}', event '${{ github.event_name }}' -> publish=${PUBLISH}"
|
||||
|
||||
# Ordered before the snapshot publish so a snapshot always carries a
|
||||
# watermark at least as new as the build it holds. Both steps sit after
|
||||
# the consuming job's build steps, so a run that fails an earlier gate
|
||||
# never reaches either: the watermark stays at the last GREEN build and a
|
||||
# red build can never overwrite a known-good snapshot.
|
||||
- if: ${{ steps.resolve.outputs.active == 'yes' && inputs.mode == 'publish' && inputs.record-watermark == 'true' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
bash "${CARGO_CACHE_SCRIPTS}/record-watermark.sh" \
|
||||
"$CARGO_TARGET_DIR" "${CI_WATERMARK_FILE:-.ci-watermark-sha}"
|
||||
|
||||
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
||||
"${{ github.job }}-${{ github.run_id }}-$$"
|
||||
|
||||
# Released in both modes. In `publish` mode this is the normal end-of-job
|
||||
# release; the separate `release-lock` call exists for `if: always()`, so a
|
||||
# failed run does not leave its lock sitting until the staleness grace
|
||||
# period expires.
|
||||
- if: ${{ steps.resolve.outputs.active == 'yes' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${CARGO_CACHE_LOCK_ID:-}" ]; then
|
||||
echo "cargo-cache-publish: no lock id in the environment — nothing to release"
|
||||
exit 0
|
||||
fi
|
||||
bash "${CARGO_CACHE_SCRIPTS}/cache-lock.sh" release \
|
||||
"$CARGO_TARGET_DIR" "${CARGO_CACHE_LOCK_ID}"
|
||||
Reference in New Issue
Block a user