Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
132 lines
5.8 KiB
YAML
132 lines
5.8 KiB
YAML
name: 'Cargo cache (publish)'
|
|
description: >-
|
|
Records this run''s build watermark and, on a publisher branch, atomically
|
|
republishes its target directory as the immutable snapshot that other
|
|
branches'' caches are hardlink-cloned from.
|
|
author: 'gitdan'
|
|
|
|
inputs:
|
|
cache-root:
|
|
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
|
required: false
|
|
default: '/cache'
|
|
protected-branches:
|
|
description: >-
|
|
Space-separated refs that publish snapshots. A run whose own ref is not
|
|
in this list records its watermark and skips publishing.
|
|
required: false
|
|
default: 'dev main'
|
|
mode:
|
|
description: >-
|
|
publish — record the watermark, publish a snapshot if eligible,
|
|
release this job''s cache lock (the normal call, after a
|
|
green build).
|
|
release-lock — release this job''s cache lock and do nothing else. Use
|
|
in a final `if: always()` step so a failed run does not
|
|
leave a lock behind for the staleness grace period.
|
|
required: false
|
|
default: 'publish'
|
|
own-ref:
|
|
description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.'
|
|
required: false
|
|
default: ''
|
|
publish-on-events:
|
|
description: >-
|
|
Space-separated event names on which a publisher branch actually
|
|
publishes. Defaults to `push` — a pull_request run never publishes,
|
|
because its ref is not the reference branch even when it targets one.
|
|
required: false
|
|
default: 'push'
|
|
record-watermark:
|
|
description: >-
|
|
Record this run''s HEAD as the build watermark for this target dir.
|
|
True for PR runs too, not just publishers: a feature branch accumulates
|
|
its own build history across several pushes and needs its own watermark.
|
|
required: false
|
|
default: 'true'
|
|
|
|
runs:
|
|
using: 'composite'
|
|
steps:
|
|
# Everything here reads the environment the consume action exported, so a
|
|
# workflow that forgets to run cargo-cache first fails loudly here rather
|
|
# than silently publishing a snapshot of the wrong directory.
|
|
- id: resolve
|
|
shell: bash
|
|
env:
|
|
PROTECTED_BRANCHES: ${{ inputs.protected-branches }}
|
|
PUBLISH_ON_EVENTS: ${{ inputs.publish-on-events }}
|
|
run: |
|
|
set -euo pipefail
|
|
# In release-lock mode this action is called from an `if: always()`
|
|
# step, which can run after a failure that happened before the
|
|
# cargo-cache action ever executed. Missing environment there means
|
|
# "there is no lock to release", not an error worth failing the job a
|
|
# second time over.
|
|
if [ -z "${CARGO_CACHE_SCRIPTS:-}" ] || [ -z "${CARGO_TARGET_DIR:-}" ]; then
|
|
if [ "${{ inputs.mode }}" = "release-lock" ]; then
|
|
echo "cargo-cache-publish: no cache environment in this job — nothing to release"
|
|
echo "publish=no" >> "$GITHUB_OUTPUT"
|
|
echo "active=no" >> "$GITHUB_OUTPUT"
|
|
exit 0
|
|
fi
|
|
echo "::error::cargo-cache-publish: run the cargo-cache action earlier in this job"
|
|
exit 1
|
|
fi
|
|
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
|
echo "active=yes" >> "$GITHUB_OUTPUT"
|
|
|
|
OWN_REF="${{ inputs.own-ref }}"
|
|
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
|
|
|
# A publisher is a branch other branches layer over. Two conditions,
|
|
# both required: its ref is in protected-branches, AND the event is one
|
|
# where this ref really is the reference branch. A pull_request run
|
|
# from `dev` into `main` has own-ref `dev` and would otherwise publish
|
|
# a snapshot of a merge-preview build — which is not what `dev` is.
|
|
PUBLISH=no
|
|
for ref in $PROTECTED_BRANCHES; do
|
|
[ "$ref" = "$OWN_REF" ] || continue
|
|
for ev in $PUBLISH_ON_EVENTS; do
|
|
[ "$ev" = "${{ github.event_name }}" ] && PUBLISH=yes
|
|
done
|
|
done
|
|
echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT"
|
|
echo "own-ref=${OWN_REF}" >> "$GITHUB_OUTPUT"
|
|
echo "publisher check: ref '${OWN_REF}', event '${{ github.event_name }}' -> publish=${PUBLISH}"
|
|
|
|
# Ordered before the snapshot publish so a snapshot always carries a
|
|
# watermark at least as new as the build it holds. Both steps sit after
|
|
# the consuming job's build steps, so a run that fails an earlier gate
|
|
# never reaches either: the watermark stays at the last GREEN build and a
|
|
# red build can never overwrite a known-good snapshot.
|
|
- if: ${{ steps.resolve.outputs.active == 'yes' && inputs.mode == 'publish' && inputs.record-watermark == 'true' }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
bash "${CARGO_CACHE_SCRIPTS}/record-watermark.sh" \
|
|
"$CARGO_TARGET_DIR" "${CI_WATERMARK_FILE:-.ci-watermark-sha}"
|
|
|
|
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
|
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
|
"${{ github.job }}-${{ github.run_id }}-$$"
|
|
|
|
# Released in both modes. In `publish` mode this is the normal end-of-job
|
|
# release; the separate `release-lock` call exists for `if: always()`, so a
|
|
# failed run does not leave its lock sitting until the staleness grace
|
|
# period expires.
|
|
- if: ${{ steps.resolve.outputs.active == 'yes' }}
|
|
shell: bash
|
|
run: |
|
|
set -euo pipefail
|
|
if [ -z "${CARGO_CACHE_LOCK_ID:-}" ]; then
|
|
echo "cargo-cache-publish: no lock id in the environment — nothing to release"
|
|
exit 0
|
|
fi
|
|
bash "${CARGO_CACHE_SCRIPTS}/cache-lock.sh" release \
|
|
"$CARGO_TARGET_DIR" "${CARGO_CACHE_LOCK_ID}"
|