feat(cache): give same-ref jobs separate build directories via cache-lineage
CI / shellcheck + selftests (pull_request) Failing after 1m19s

A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.

`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:

    <cache-root>/target-<key>              no lineage (unchanged)
    <cache-root>/<lineage>/target-<key>    a lineage

Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).

Every interacting part, checked rather than assumed:

- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
  a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
  The publish action now takes the root from the `CARGO_CACHE_ROOT` the
  consume step exported, and CHECKS its own inputs against it — a publish step
  left at the default while its consume step nested would otherwise republish
  a different lineage's live target dir over that lineage's snapshot, on every
  push, silently. `mode: release-lock` is exempt: it releases a lock on
  `$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
  lineage neither evicts nor sees a sibling's caches, or the flat layout's.
  Liveness keeps resolving real branch names, which is what a key suffix would
  have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
  all six nested and flat dirs collected as candidates, protection resolved
  correctly on the nested ones, and a `.stage-` stranded inside the lineage
  found by the leftover sweep.

Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.

Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.

New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
This commit is contained in:
2026-08-26 12:32:21 -05:00
parent f76789358d
commit fb7a788c90
7 changed files with 522 additions and 16 deletions
+100
View File
@@ -114,6 +114,106 @@ cache_key() {
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
# ---------------------------------------------------------------------------
# Cache lineages
# ---------------------------------------------------------------------------
#
# A cache key names a REF. What a target directory holds is the product of a
# ref and a BUILD CONFIGURATION, and the two are not the same thing: emowheel
# builds the same ref twice on every push, once for the host and once for
# wasm32, in two jobs that run concurrently. Keyed on the ref alone both
# resolve to one CARGO_TARGET_DIR, and Cargo's build-directory lock is
# exclusive — so the second job sits on `Blocking waiting for file lock on
# build directory` for the length of the first, holding a runner capacity slot
# while doing nothing (daniel/gitdan#60).
#
# A lineage is that second dimension, and it is expressed as ONE DIRECTORY
# LEVEL above the per-ref directories rather than as a suffix on the key:
#
# <cache-root>/target-<key> no lineage (the flat layout)
# <cache-root>/<lineage>/target-<key> a lineage
#
# Nesting rather than suffixing is what keeps every existing reader correct
# without teaching any of them a new name shape. `prune-cache.sh` resolves
# liveness by recomputing `target-<cache_key(branch)>` for every branch on
# origin and evicting whatever does not match — a suffixed `target-<key>-wasm32`
# matches nothing, so it would be classified dead and unconditionally evicted
# on every single run. The host-level arbiter in daniel/gitdan reads the same
# shape (its BRANCH_DIR_RE), and a suffixed name falls out of it too: not
# evicted there, but never a candidate either, so a whole lineage becomes
# invisible to the global disk budget. Nesting leaves both matchers reading
# exactly the names they already read, one directory deeper.
#
# ONE LEVEL, AND NOT TWO. The arbiter walks a volume to CI_CACHE_MAX_DEPTH,
# which is 2 — `_data/target-<key>` and `_data/<lineage>/target-<key>`. It is
# kept tight there on purpose (a deeper walk starts meeting Cargo's own
# `incremental/<crate>-<hash>` directories, which match the same name shape and
# must never be evicted individually), so a lineage is a single path component
# and validate_cache_lineage refuses one containing a slash.
# Directory names daniel/gitdan's ci-cache-reclaim.sh refuses to descend into
# (its CI_CACHE_NODESCEND_NAMES). A lineage named one of these puts its whole
# subtree outside the global arbiter's reach: the caches accumulate and the one
# script whose job is the shared disk budget cannot see them.
CACHE_LINEAGE_RESERVED_NAMES="debug release deps incremental build .fingerprint tmp examples doc"
# The shape that same script reads as a per-branch cache directory (its
# BRANCH_DIR_RE). A lineage matching it is taken for a cache dir in its own
# right — never descended into, and an eviction candidate whole, which is the
# entire lineage rather than one ref's share of it.
CACHE_LINEAGE_BRANCH_DIR_RE='^.+-[0-9a-f]{7,40}$'
# Every rejection below names the reader that imposes it, because that is the
# only way the constraint survives: none of these is a filesystem limit, and a
# name that trips one produces no error anywhere — it produces a lineage that
# silently stops being pruned, or silently stops being reclaimed.
validate_cache_lineage() {
local lineage="$1" reserved
[ -n "$lineage" ] || return 0
case "$lineage" in
*/*)
echo "::error::cache lineage '${lineage}' must be a single path component: the host-level arbiter walks a cache volume to depth 2, so <cache-root>/<lineage>/target-<key> is as deep as a cache directory may sit and still be reclaimable" >&2
return 1
;;
.*)
echo "::error::cache lineage '${lineage}' must not start with a dot: every dot-prefixed entry under a cache root belongs to the leftover-naming contract (see the top of this file), and a lineage is not garbage to be reclaimed" >&2
return 1
;;
target-* | snapshot-*)
echo "::error::cache lineage '${lineage}' must not start with 'target-' or 'snapshot-': prune-cache.sh globs both prefixes at the cache root, so the lineage directory itself would become an eviction candidate" >&2
return 1
;;
*[!A-Za-z0-9._-]*)
echo "::error::cache lineage '${lineage}' may contain only [A-Za-z0-9._-] — the same charset cache_key() sanitises a ref down to" >&2
return 1
;;
esac
for reserved in $CACHE_LINEAGE_RESERVED_NAMES; do
if [ "$lineage" = "$reserved" ]; then
echo "::error::cache lineage '${lineage}' is one of the Cargo directory names daniel/gitdan's ci-cache-reclaim.sh never descends into (CI_CACHE_NODESCEND_NAMES) — every cache under it would be invisible to the host-level disk budget" >&2
return 1
fi
done
if [[ $lineage =~ $CACHE_LINEAGE_BRANCH_DIR_RE ]]; then
echo "::error::cache lineage '${lineage}' ends in a hex suffix, which is the shape daniel/gitdan's ci-cache-reclaim.sh reads as a per-branch cache directory — it would treat the lineage directory as one cache and evict the whole thing" >&2
return 1
fi
return 0
}
# The cache root a lineage's directories actually live under. An empty lineage
# resolves to the cache root unchanged, byte for byte: that is what makes this
# a no-op for every consumer that does not set one, rather than a migration.
cache_root_for() {
local root="$1" lineage="${2:-}"
validate_cache_lineage "$lineage" || return 1
printf '%s%s' "$root" "${lineage:+/$lineage}"
}
# ---------------------------------------------------------------------------
# Disk accounting
# ---------------------------------------------------------------------------