feat(cache): give same-ref jobs separate build directories via cache-lineage
CI / shellcheck + selftests (pull_request) Failing after 1m19s
CI / shellcheck + selftests (pull_request) Failing after 1m19s
A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.
`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:
<cache-root>/target-<key> no lineage (unchanged)
<cache-root>/<lineage>/target-<key> a lineage
Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).
Every interacting part, checked rather than assumed:
- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
The publish action now takes the root from the `CARGO_CACHE_ROOT` the
consume step exported, and CHECKS its own inputs against it — a publish step
left at the default while its consume step nested would otherwise republish
a different lineage's live target dir over that lineage's snapshot, on every
push, silently. `mode: release-lock` is exempt: it releases a lock on
`$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
lineage neither evicts nor sees a sibling's caches, or the flat layout's.
Liveness keeps resolving real branch names, which is what a key suffix would
have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
all six nested and flat dirs collected as candidates, protection resolved
correctly on the nested ones, and a `.stage-` stranded inside the lineage
found by the leftover sweep.
Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.
Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.
New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
This commit is contained in:
@@ -10,6 +10,15 @@ inputs:
|
||||
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
||||
required: false
|
||||
default: '/cache'
|
||||
cache-lineage:
|
||||
description: >-
|
||||
Must match the cargo-cache step in this job. Checked rather than
|
||||
assumed: this action derives both ends of the snapshot swap from its own
|
||||
cache-root, so a publish step left at the default while its consume step
|
||||
nested would republish a DIFFERENT lineage's live target dir over that
|
||||
lineage's snapshot, on every push, silently. A mismatch fails the step.
|
||||
required: false
|
||||
default: ''
|
||||
protected-branches:
|
||||
description: >-
|
||||
Space-separated refs that publish snapshots. A run whose own ref is not
|
||||
@@ -67,6 +76,11 @@ runs:
|
||||
# Everything here reads the environment the consume action exported, so a
|
||||
# workflow that forgets to run cargo-cache first fails loudly here rather
|
||||
# than silently publishing a snapshot of the wrong directory.
|
||||
#
|
||||
# The cache root is taken from that same environment for the same reason,
|
||||
# and this action's own cache-root/cache-lineage inputs are checked against
|
||||
# it rather than used. The two have always had to agree; until a lineage
|
||||
# existed they always did, because nobody overrode the default.
|
||||
- id: resolve
|
||||
shell: bash
|
||||
env:
|
||||
@@ -92,6 +106,18 @@ runs:
|
||||
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
||||
echo "active=yes" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Only in publish mode. The `release-lock` call is an `if: always()`
|
||||
# step that consuming workflows invoke with `mode:` and nothing else —
|
||||
# it releases $CARGO_CACHE_LOCK_ID on $CARGO_TARGET_DIR and never
|
||||
# touches a cache root at all, so holding its default inputs to the
|
||||
# consume step's would fail the cleanup step of every job that sets a
|
||||
# lineage, for a value it does not use.
|
||||
if [ "${{ inputs.mode }}" = "publish" ]; then
|
||||
: "${CARGO_CACHE_ROOT:?cargo-cache-publish: CARGO_CACHE_ROOT not set by the cargo-cache action}"
|
||||
bash "${CARGO_CACHE_SCRIPTS}/cache-root.sh" verify \
|
||||
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}" "$CARGO_CACHE_ROOT"
|
||||
fi
|
||||
|
||||
OWN_REF="${{ inputs.own-ref }}"
|
||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||
|
||||
@@ -131,7 +157,7 @@ runs:
|
||||
run: |
|
||||
set -euo pipefail
|
||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
||||
"$CARGO_CACHE_KEY" "$CARGO_CACHE_ROOT" \
|
||||
"${{ github.job }}-${{ github.run_id }}-$$"
|
||||
|
||||
# Released in both modes. In `publish` mode this is the normal end-of-job
|
||||
|
||||
Reference in New Issue
Block a user