feat(cache): give same-ref jobs separate build directories via cache-lineage
CI / shellcheck + selftests (pull_request) Failing after 1m19s
CI / shellcheck + selftests (pull_request) Failing after 1m19s
A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.
`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:
<cache-root>/target-<key> no lineage (unchanged)
<cache-root>/<lineage>/target-<key> a lineage
Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).
Every interacting part, checked rather than assumed:
- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
The publish action now takes the root from the `CARGO_CACHE_ROOT` the
consume step exported, and CHECKS its own inputs against it — a publish step
left at the default while its consume step nested would otherwise republish
a different lineage's live target dir over that lineage's snapshot, on every
push, silently. `mode: release-lock` is exempt: it releases a lock on
`$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
lineage neither evicts nor sees a sibling's caches, or the flat layout's.
Liveness keeps resolving real branch names, which is what a key suffix would
have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
all six nested and flat dirs collected as candidates, protection resolved
correctly on the nested ones, and a `.stage-` stranded inside the lineage
found by the leftover sweep.
Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.
Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.
New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
This commit is contained in:
@@ -317,6 +317,39 @@ spelling of some new dot-prefixed name — a name it reads for a decision but
|
||||
never reclaims — that is exactly this category, and it needs the matching
|
||||
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
||||
|
||||
### The directory LAYOUT is part of that contract as well
|
||||
|
||||
Names are one half; where they sit is the other. gitdan's arbiter walks a
|
||||
volume's `_data` tree to `CI_CACHE_MAX_DEPTH`, which is **2** — deliberately
|
||||
tight, because a deeper walk starts meeting Cargo's own
|
||||
`incremental/<crate>-<hash>` directories, which match the same name shape it
|
||||
uses to recognise a cache dir and must never be evicted individually. So:
|
||||
|
||||
```
|
||||
_data/target-<key> depth 1 — no lineage
|
||||
_data/<lineage>/target-<key> depth 2 — a lineage
|
||||
_data/<a>/<b>/target-<key> depth 3 — INVISIBLE to the arbiter
|
||||
```
|
||||
|
||||
That budget is the whole reason `cache-lineage` is one path component and not
|
||||
a path. Nesting deeper is not an error anywhere: the caches work, the
|
||||
in-workflow prune pass keeps managing them, and the one script whose job is
|
||||
the shared disk budget across every repo simply never sees them again.
|
||||
|
||||
It is also why the fix for daniel/gitdan#60 nests rather than suffixing the
|
||||
cache key. A `target-<key>-<lineage>` name would be read as dead by
|
||||
`prune-cache.sh`'s liveness pass — which classifies by recomputing
|
||||
`target-<cache_key(branch)>` for every branch on origin — and evicted
|
||||
unconditionally on every run; and it falls out of the arbiter's own
|
||||
`BRANCH_DIR_RE` too, so the same directories would never be candidates there
|
||||
either. Nesting leaves both matchers reading exactly the names they already
|
||||
read, one level down.
|
||||
|
||||
One known rough edge, on gitdan's side and cosmetic: that script logs an
|
||||
eviction as `<volume>/<basename>`, so a nested `target-<key>` and a flat one
|
||||
of the same key are indistinguishable in its output. It evicts the right
|
||||
directory; the line just doesn't say which.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
@@ -326,6 +359,7 @@ never reclaims — that is exactly this category, and it needs the matching
|
||||
| input | default | meaning |
|
||||
|---|---|---|
|
||||
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
||||
| `cache-lineage` | *(empty)* | one directory level under `cache-root`, for a second job building the same ref for a different target or profile — see [Multiple jobs in one workflow](#multiple-jobs-in-one-workflow) |
|
||||
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
||||
| `min-free-percent` | `10` | prune when free space drops below this |
|
||||
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
||||
@@ -334,7 +368,7 @@ never reclaims — that is exactly this category, and it needs the matching
|
||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
||||
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
||||
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
|
||||
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one target directory; the default already does |
|
||||
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
||||
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
||||
|
||||
@@ -350,6 +384,7 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
||||
| input | default | meaning |
|
||||
|---|---|---|
|
||||
| `cache-root` | `/cache` | must match the consume action |
|
||||
| `cache-lineage` | *(empty)* | must match the consume action; a mismatch fails the step rather than publishing the wrong tree |
|
||||
| `protected-branches` | `dev main` | refs that publish snapshots |
|
||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||
@@ -366,13 +401,70 @@ of a merge-preview build, which is not what `dev` is.
|
||||
|
||||
## Multiple jobs in one workflow
|
||||
|
||||
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
|
||||
each need their **own** watermark file. A shared one breaks the moment two
|
||||
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
|
||||
and job B then reads that just-advanced value, computes an empty diff, and
|
||||
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
|
||||
already gives each job its own; only override `watermark-file` if you also
|
||||
override `lock-id`, and then keep both distinct per job.
|
||||
Two jobs building the same ref — a `ci` job and a `wasm` job, say — are two
|
||||
consumers of one cache key, and the cache key alone is not enough to keep them
|
||||
apart.
|
||||
|
||||
**Give each its own lineage.** A cache key names a *ref*; what a target
|
||||
directory holds is the product of a ref and a build configuration. Left to the
|
||||
key alone, both jobs export the same `CARGO_TARGET_DIR`, and Cargo's
|
||||
build-directory lock is exclusive — so on a runner with more than one slot the
|
||||
second job sits on `Blocking waiting for file lock on build directory` for the
|
||||
length of the first, occupying a capacity slot while doing nothing
|
||||
(daniel/gitdan#60). `cache-lineage` is that second dimension:
|
||||
|
||||
```yaml
|
||||
- name: Restore the Cargo cache
|
||||
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
|
||||
with:
|
||||
cache-lineage: wasm32 # the `ci` job sets none
|
||||
|
||||
# ... build steps ...
|
||||
|
||||
- name: Record watermark, publish cache snapshot
|
||||
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
||||
with:
|
||||
cache-lineage: wasm32 # the SAME value, or the step fails
|
||||
```
|
||||
|
||||
A lineage nests one directory level under the cache root
|
||||
(`<cache-root>/<lineage>/target-<key>`), so each lineage gets its own target
|
||||
dirs, its own snapshots, and its own prune pass. Everything else works as it
|
||||
already did, one level down: a PR branch in a lineage layers over **that
|
||||
lineage's** base snapshot, the publisher branch publishes into it, and a prune
|
||||
pass run inside it never sees a sibling lineage's caches.
|
||||
|
||||
Setting no lineage resolves to the cache root unchanged, byte for byte, so a
|
||||
workflow that does not use one keeps the exact directories it already has on
|
||||
the volume.
|
||||
|
||||
**Both actions need the same value.** `cargo-cache-publish` derives both ends
|
||||
of the snapshot swap from its own `cache-root`, so a publish step left at the
|
||||
default while its consume step nested would republish a *different* lineage's
|
||||
live target dir over that lineage's snapshot, on every push, with nothing in
|
||||
the log to say so. The publish action therefore compares its own inputs
|
||||
against the `CARGO_CACHE_ROOT` the consume step exported and fails the step on
|
||||
a mismatch. (The `mode: release-lock` call is exempt: it releases a lock on
|
||||
`$CARGO_TARGET_DIR` and never touches a cache root, so it takes no lineage.)
|
||||
|
||||
**Some lineage names are refused.** A lineage is one path component, drawn
|
||||
from `[A-Za-z0-9._-]`, and several otherwise-reasonable names are rejected at
|
||||
resolve time because a *reader elsewhere* would stop seeing the caches
|
||||
underneath them: a Cargo profile name (`debug`, `release`, `doc`, …) is one
|
||||
gitdan's arbiter never descends into, a `target-`/`snapshot-` prefix makes the
|
||||
lineage directory itself an eviction candidate for this repo's own prune pass,
|
||||
and a hex-suffixed name is read by that arbiter as a per-branch cache dir in
|
||||
its own right. `validate_cache_lineage()` in `scripts/cache-lib.sh` states each
|
||||
rejection with the reader that imposes it.
|
||||
|
||||
**Watermarks are still per job.** Two jobs in one lineage — or one job before
|
||||
lineages were introduced — each need their **own** watermark file. A shared one
|
||||
breaks the moment two jobs run in sequence within one trigger: job A advances
|
||||
the watermark to HEAD, and job B then reads that just-advanced value, computes
|
||||
an empty diff, and loses the merge protection entirely. The default
|
||||
(`.ci-watermark-<job>-sha`) already gives each job its own; only override
|
||||
`watermark-file` if you also override `lock-id`, and then keep both distinct
|
||||
per job.
|
||||
|
||||
---
|
||||
|
||||
@@ -466,6 +558,7 @@ change here reaches all of them at once. That is what the gate is for.
|
||||
|
||||
| suite | covers |
|
||||
|---|---|
|
||||
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||
|
||||
Reference in New Issue
Block a user