feat(cache): give same-ref jobs separate build directories via cache-lineage
CI / shellcheck + selftests (pull_request) Failing after 1m19s

A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.

`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:

    <cache-root>/target-<key>              no lineage (unchanged)
    <cache-root>/<lineage>/target-<key>    a lineage

Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).

Every interacting part, checked rather than assumed:

- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
  a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
  The publish action now takes the root from the `CARGO_CACHE_ROOT` the
  consume step exported, and CHECKS its own inputs against it — a publish step
  left at the default while its consume step nested would otherwise republish
  a different lineage's live target dir over that lineage's snapshot, on every
  push, silently. `mode: release-lock` is exempt: it releases a lock on
  `$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
  lineage neither evicts nor sees a sibling's caches, or the flat layout's.
  Liveness keeps resolving real branch names, which is what a key suffix would
  have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
  all six nested and flat dirs collected as candidates, protection resolved
  correctly on the nested ones, and a `.stage-` stranded inside the lineage
  found by the leftover sweep.

Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.

Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.

New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
This commit is contained in:
2026-08-26 12:32:21 -05:00
parent f76789358d
commit fb7a788c90
7 changed files with 522 additions and 16 deletions
+101 -8
View File
@@ -317,6 +317,39 @@ spelling of some new dot-prefixed name — a name it reads for a decision but
never reclaims — that is exactly this category, and it needs the matching
`DEPEND_*` entry on gitdan's side before it ships, not after.
### The directory LAYOUT is part of that contract as well
Names are one half; where they sit is the other. gitdan's arbiter walks a
volume's `_data` tree to `CI_CACHE_MAX_DEPTH`, which is **2** — deliberately
tight, because a deeper walk starts meeting Cargo's own
`incremental/<crate>-<hash>` directories, which match the same name shape it
uses to recognise a cache dir and must never be evicted individually. So:
```
_data/target-<key> depth 1 — no lineage
_data/<lineage>/target-<key> depth 2 — a lineage
_data/<a>/<b>/target-<key> depth 3 — INVISIBLE to the arbiter
```
That budget is the whole reason `cache-lineage` is one path component and not
a path. Nesting deeper is not an error anywhere: the caches work, the
in-workflow prune pass keeps managing them, and the one script whose job is
the shared disk budget across every repo simply never sees them again.
It is also why the fix for daniel/gitdan#60 nests rather than suffixing the
cache key. A `target-<key>-<lineage>` name would be read as dead by
`prune-cache.sh`'s liveness pass — which classifies by recomputing
`target-<cache_key(branch)>` for every branch on origin — and evicted
unconditionally on every run; and it falls out of the arbiter's own
`BRANCH_DIR_RE` too, so the same directories would never be candidates there
either. Nesting leaves both matchers reading exactly the names they already
read, one level down.
One known rough edge, on gitdan's side and cosmetic: that script logs an
eviction as `<volume>/<basename>`, so a nested `target-<key>` and a flat one
of the same key are indistinguishable in its output. It evicts the right
directory; the line just doesn't say which.
---
## Inputs
@@ -326,6 +359,7 @@ never reclaims — that is exactly this category, and it needs the matching
| input | default | meaning |
|---|---|---|
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
| `cache-lineage` | *(empty)* | one directory level under `cache-root`, for a second job building the same ref for a different target or profile — see [Multiple jobs in one workflow](#multiple-jobs-in-one-workflow) |
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
| `min-free-percent` | `10` | prune when free space drops below this |
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
@@ -334,7 +368,7 @@ never reclaims — that is exactly this category, and it needs the matching
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one target directory; the default already does |
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
@@ -350,6 +384,7 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
| input | default | meaning |
|---|---|---|
| `cache-root` | `/cache` | must match the consume action |
| `cache-lineage` | *(empty)* | must match the consume action; a mismatch fails the step rather than publishing the wrong tree |
| `protected-branches` | `dev main` | refs that publish snapshots |
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
@@ -366,13 +401,70 @@ of a merge-preview build, which is not what `dev` is.
## Multiple jobs in one workflow
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
each need their **own** watermark file. A shared one breaks the moment two
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
and job B then reads that just-advanced value, computes an empty diff, and
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
already gives each job its own; only override `watermark-file` if you also
override `lock-id`, and then keep both distinct per job.
Two jobs building the same ref — a `ci` job and a `wasm` job, say — are two
consumers of one cache key, and the cache key alone is not enough to keep them
apart.
**Give each its own lineage.** A cache key names a *ref*; what a target
directory holds is the product of a ref and a build configuration. Left to the
key alone, both jobs export the same `CARGO_TARGET_DIR`, and Cargo's
build-directory lock is exclusive — so on a runner with more than one slot the
second job sits on `Blocking waiting for file lock on build directory` for the
length of the first, occupying a capacity slot while doing nothing
(daniel/gitdan#60). `cache-lineage` is that second dimension:
```yaml
- name: Restore the Cargo cache
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
with:
cache-lineage: wasm32 # the `ci` job sets none
# ... build steps ...
- name: Record watermark, publish cache snapshot
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
with:
cache-lineage: wasm32 # the SAME value, or the step fails
```
A lineage nests one directory level under the cache root
(`<cache-root>/<lineage>/target-<key>`), so each lineage gets its own target
dirs, its own snapshots, and its own prune pass. Everything else works as it
already did, one level down: a PR branch in a lineage layers over **that
lineage's** base snapshot, the publisher branch publishes into it, and a prune
pass run inside it never sees a sibling lineage's caches.
Setting no lineage resolves to the cache root unchanged, byte for byte, so a
workflow that does not use one keeps the exact directories it already has on
the volume.
**Both actions need the same value.** `cargo-cache-publish` derives both ends
of the snapshot swap from its own `cache-root`, so a publish step left at the
default while its consume step nested would republish a *different* lineage's
live target dir over that lineage's snapshot, on every push, with nothing in
the log to say so. The publish action therefore compares its own inputs
against the `CARGO_CACHE_ROOT` the consume step exported and fails the step on
a mismatch. (The `mode: release-lock` call is exempt: it releases a lock on
`$CARGO_TARGET_DIR` and never touches a cache root, so it takes no lineage.)
**Some lineage names are refused.** A lineage is one path component, drawn
from `[A-Za-z0-9._-]`, and several otherwise-reasonable names are rejected at
resolve time because a *reader elsewhere* would stop seeing the caches
underneath them: a Cargo profile name (`debug`, `release`, `doc`, …) is one
gitdan's arbiter never descends into, a `target-`/`snapshot-` prefix makes the
lineage directory itself an eviction candidate for this repo's own prune pass,
and a hex-suffixed name is read by that arbiter as a per-branch cache dir in
its own right. `validate_cache_lineage()` in `scripts/cache-lib.sh` states each
rejection with the reader that imposes it.
**Watermarks are still per job.** Two jobs in one lineage — or one job before
lineages were introduced — each need their **own** watermark file. A shared one
breaks the moment two jobs run in sequence within one trigger: job A advances
the watermark to HEAD, and job B then reads that just-advanced value, computes
an empty diff, and loses the merge protection entirely. The default
(`.ci-watermark-<job>-sha`) already gives each job its own; only override
`watermark-file` if you also override `lock-id`, and then keep both distinct
per job.
---
@@ -466,6 +558,7 @@ change here reaches all of them at once. That is what the gate is for.
| suite | covers |
|---|---|
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |