From fb7a788c90e7d5204edb037c32a4d5ee9cad8ca4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 12:32:21 -0500 Subject: [PATCH] feat(cache): give same-ref jobs separate build directories via cache-lineage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A cache key names a REF. What a target directory holds is the product of a ref and a build configuration, and emowheel builds the same ref twice on every push — once for the host, once for wasm32, in two jobs that start together. Keyed on the ref alone, `cargo-cache@v1` handed both the same CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second job sat on "Blocking waiting for file lock on build directory" for the length of the first while holding a runner capacity slot, so a third repo's queued job waited behind a job doing nothing. `cache-lineage` is that second dimension. It names ONE directory level under the cache root: /target- no lineage (unchanged) //target- a lineage Nesting, not a suffix on the key, and that is the whole design decision. `prune-cache.sh`'s liveness pass classifies a directory by recomputing `target-` for every branch on origin and evicting whatever does not match — a `target--wasm32` matches nothing, so it would be classified dead and evicted unconditionally on every run. daniel/gitdan's host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls out of that too, so those caches would never be reclaim candidates and a whole lineage would go missing from the shared disk budget. Nesting leaves both matchers reading exactly the names they already read, one level down — which is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own suite pins the depth-2 case). Every interacting part, checked rather than assumed: - SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in a lineage layers over THAT lineage's base snapshot. Asserted. - PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root. The publish action now takes the root from the `CARGO_CACHE_ROOT` the consume step exported, and CHECKS its own inputs against it — a publish step left at the default while its consume step nested would otherwise republish a different lineage's live target dir over that lineage's snapshot, on every push, silently. `mode: release-lock` is exempt: it releases a lock on `$CARGO_TARGET_DIR` and never touches a root. - WATERMARK: per target dir, so it follows the lineage. Unchanged. - PRUNE and LIVENESS: scoped to the root they are given, so a pass in one lineage neither evicts nor sees a sibling's caches, or the flat layout's. Liveness keeps resolving real branch names, which is what a key suffix would have broken. - ci_cache_reclaim: verified by dry-run against a fixture in this layout — all six nested and flat dirs collected as candidates, protection resolved correctly on the nested ones, and a `.stage-` stranded inside the lineage found by the leftover sweep. Refused lineage names are refused at resolve time, each rejection naming the reader that imposes it: a path separator (the arbiter's depth budget), a Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix (the leftover-naming contract). None of these fails visibly on its own — each produces a working directory that some pass silently stops seeing. Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna and emowheel's `ci` job keep the exact directories they have on the volume. New suite `cache-root-selftest.sh` (19 assertions), red-proven against three deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled validator, and a `verify` that never rejects a mismatch. --- README.md | 109 ++++++++++++++++-- cargo-cache-publish/action.yml | 28 ++++- cargo-cache/action.yml | 44 +++++++- scripts/cache-lib.sh | 100 +++++++++++++++++ scripts/cache-root-selftest.sh | 194 +++++++++++++++++++++++++++++++++ scripts/cache-root.sh | 61 +++++++++++ scripts/selftest.sh | 2 +- 7 files changed, 522 insertions(+), 16 deletions(-) create mode 100755 scripts/cache-root-selftest.sh create mode 100755 scripts/cache-root.sh diff --git a/README.md b/README.md index 7a11859..671f9f5 100644 --- a/README.md +++ b/README.md @@ -317,6 +317,39 @@ spelling of some new dot-prefixed name — a name it reads for a decision but never reclaims — that is exactly this category, and it needs the matching `DEPEND_*` entry on gitdan's side before it ships, not after. +### The directory LAYOUT is part of that contract as well + +Names are one half; where they sit is the other. gitdan's arbiter walks a +volume's `_data` tree to `CI_CACHE_MAX_DEPTH`, which is **2** — deliberately +tight, because a deeper walk starts meeting Cargo's own +`incremental/-` directories, which match the same name shape it +uses to recognise a cache dir and must never be evicted individually. So: + +``` +_data/target- depth 1 — no lineage +_data//target- depth 2 — a lineage +_data///target- depth 3 — INVISIBLE to the arbiter +``` + +That budget is the whole reason `cache-lineage` is one path component and not +a path. Nesting deeper is not an error anywhere: the caches work, the +in-workflow prune pass keeps managing them, and the one script whose job is +the shared disk budget across every repo simply never sees them again. + +It is also why the fix for daniel/gitdan#60 nests rather than suffixing the +cache key. A `target--` name would be read as dead by +`prune-cache.sh`'s liveness pass — which classifies by recomputing +`target-` for every branch on origin — and evicted +unconditionally on every run; and it falls out of the arbiter's own +`BRANCH_DIR_RE` too, so the same directories would never be candidates there +either. Nesting leaves both matchers reading exactly the names they already +read, one level down. + +One known rough edge, on gitdan's side and cosmetic: that script logs an +eviction as `/`, so a nested `target-` and a flat one +of the same key are indistinguishable in its output. It evicts the right +directory; the line just doesn't say which. + --- ## Inputs @@ -326,6 +359,7 @@ never reclaims — that is exactly this category, and it needs the matching | input | default | meaning | |---|---|---| | `cache-root` | `/cache` | mount point of the persistent volume inside the job container | +| `cache-lineage` | *(empty)* | one directory level under `cache-root`, for a second job building the same ref for a different target or profile — see [Multiple jobs in one workflow](#multiple-jobs-in-one-workflow) | | `protected-branches` | `dev main` | refs that publish snapshots and are never evicted | | `min-free-percent` | `10` | prune when free space drops below this | | `restore-mtimes` | `true` | restore tracked-file mtimes from git history | @@ -334,7 +368,7 @@ never reclaims — that is exactly this category, and it needs the matching | `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` | | `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) | | `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache | -| `watermark-file` | `.ci-watermark--sha` | must differ per job when two jobs share one cache key | +| `watermark-file` | `.ci-watermark--sha` | must differ per job when two jobs share one target directory; the default already does | | `lock-id` | `-` | identifies this job's cache lock | | `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned | @@ -350,6 +384,7 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`, | input | default | meaning | |---|---|---| | `cache-root` | `/cache` | must match the consume action | +| `cache-lineage` | *(empty)* | must match the consume action; a mismatch fails the step rather than publishing the wrong tree | | `protected-branches` | `dev main` | refs that publish snapshots | | `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step | | `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` | @@ -366,13 +401,70 @@ of a merge-preview build, which is not what `dev` is. ## Multiple jobs in one workflow -Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say) -each need their **own** watermark file. A shared one breaks the moment two -jobs run in sequence within one trigger: job A advances the watermark to HEAD, -and job B then reads that just-advanced value, computes an empty diff, and -loses the merge protection entirely. The default (`.ci-watermark--sha`) -already gives each job its own; only override `watermark-file` if you also -override `lock-id`, and then keep both distinct per job. +Two jobs building the same ref — a `ci` job and a `wasm` job, say — are two +consumers of one cache key, and the cache key alone is not enough to keep them +apart. + +**Give each its own lineage.** A cache key names a *ref*; what a target +directory holds is the product of a ref and a build configuration. Left to the +key alone, both jobs export the same `CARGO_TARGET_DIR`, and Cargo's +build-directory lock is exclusive — so on a runner with more than one slot the +second job sits on `Blocking waiting for file lock on build directory` for the +length of the first, occupying a capacity slot while doing nothing +(daniel/gitdan#60). `cache-lineage` is that second dimension: + +```yaml + - name: Restore the Cargo cache + uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1 + with: + cache-lineage: wasm32 # the `ci` job sets none + + # ... build steps ... + + - name: Record watermark, publish cache snapshot + uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1 + with: + cache-lineage: wasm32 # the SAME value, or the step fails +``` + +A lineage nests one directory level under the cache root +(`//target-`), so each lineage gets its own target +dirs, its own snapshots, and its own prune pass. Everything else works as it +already did, one level down: a PR branch in a lineage layers over **that +lineage's** base snapshot, the publisher branch publishes into it, and a prune +pass run inside it never sees a sibling lineage's caches. + +Setting no lineage resolves to the cache root unchanged, byte for byte, so a +workflow that does not use one keeps the exact directories it already has on +the volume. + +**Both actions need the same value.** `cargo-cache-publish` derives both ends +of the snapshot swap from its own `cache-root`, so a publish step left at the +default while its consume step nested would republish a *different* lineage's +live target dir over that lineage's snapshot, on every push, with nothing in +the log to say so. The publish action therefore compares its own inputs +against the `CARGO_CACHE_ROOT` the consume step exported and fails the step on +a mismatch. (The `mode: release-lock` call is exempt: it releases a lock on +`$CARGO_TARGET_DIR` and never touches a cache root, so it takes no lineage.) + +**Some lineage names are refused.** A lineage is one path component, drawn +from `[A-Za-z0-9._-]`, and several otherwise-reasonable names are rejected at +resolve time because a *reader elsewhere* would stop seeing the caches +underneath them: a Cargo profile name (`debug`, `release`, `doc`, …) is one +gitdan's arbiter never descends into, a `target-`/`snapshot-` prefix makes the +lineage directory itself an eviction candidate for this repo's own prune pass, +and a hex-suffixed name is read by that arbiter as a per-branch cache dir in +its own right. `validate_cache_lineage()` in `scripts/cache-lib.sh` states each +rejection with the reader that imposes it. + +**Watermarks are still per job.** Two jobs in one lineage — or one job before +lineages were introduced — each need their **own** watermark file. A shared one +breaks the moment two jobs run in sequence within one trigger: job A advances +the watermark to HEAD, and job B then reads that just-advanced value, computes +an empty diff, and loses the merge protection entirely. The default +(`.ci-watermark--sha`) already gives each job its own; only override +`watermark-file` if you also override `lock-id`, and then keep both distinct +per job. --- @@ -466,6 +558,7 @@ change here reaches all of them at once. That is what the gate is for. | suite | covers | |---|---| +| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard | | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. | | `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot | | `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone | diff --git a/cargo-cache-publish/action.yml b/cargo-cache-publish/action.yml index 86729ed..a4b94d0 100644 --- a/cargo-cache-publish/action.yml +++ b/cargo-cache-publish/action.yml @@ -10,6 +10,15 @@ inputs: description: 'Mount point of the persistent cache volume. Must match the consume action.' required: false default: '/cache' + cache-lineage: + description: >- + Must match the cargo-cache step in this job. Checked rather than + assumed: this action derives both ends of the snapshot swap from its own + cache-root, so a publish step left at the default while its consume step + nested would republish a DIFFERENT lineage's live target dir over that + lineage's snapshot, on every push, silently. A mismatch fails the step. + required: false + default: '' protected-branches: description: >- Space-separated refs that publish snapshots. A run whose own ref is not @@ -67,6 +76,11 @@ runs: # Everything here reads the environment the consume action exported, so a # workflow that forgets to run cargo-cache first fails loudly here rather # than silently publishing a snapshot of the wrong directory. + # + # The cache root is taken from that same environment for the same reason, + # and this action's own cache-root/cache-lineage inputs are checked against + # it rather than used. The two have always had to agree; until a lineage + # existed they always did, because nobody overrode the default. - id: resolve shell: bash env: @@ -92,6 +106,18 @@ runs: : "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}" echo "active=yes" >> "$GITHUB_OUTPUT" + # Only in publish mode. The `release-lock` call is an `if: always()` + # step that consuming workflows invoke with `mode:` and nothing else — + # it releases $CARGO_CACHE_LOCK_ID on $CARGO_TARGET_DIR and never + # touches a cache root at all, so holding its default inputs to the + # consume step's would fail the cleanup step of every job that sets a + # lineage, for a value it does not use. + if [ "${{ inputs.mode }}" = "publish" ]; then + : "${CARGO_CACHE_ROOT:?cargo-cache-publish: CARGO_CACHE_ROOT not set by the cargo-cache action}" + bash "${CARGO_CACHE_SCRIPTS}/cache-root.sh" verify \ + "${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}" "$CARGO_CACHE_ROOT" + fi + OWN_REF="${{ inputs.own-ref }}" [ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}" @@ -131,7 +157,7 @@ runs: run: | set -euo pipefail bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \ - "$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \ + "$CARGO_CACHE_KEY" "$CARGO_CACHE_ROOT" \ "${{ github.job }}-${{ github.run_id }}-$$" # Released in both modes. In `publish` mode this is the normal end-of-job diff --git a/cargo-cache/action.yml b/cargo-cache/action.yml index f3eba09..84d3777 100644 --- a/cargo-cache/action.yml +++ b/cargo-cache/action.yml @@ -10,6 +10,18 @@ inputs: description: 'Mount point of the persistent cache volume inside the job container.' required: false default: '/cache' + cache-lineage: + description: >- + Distinguishes two jobs that build the SAME ref for different targets or + profiles (a host build and a wasm32 build, say) and would otherwise + resolve to one CARGO_TARGET_DIR and serialise on Cargo's exclusive + build-directory lock. Names one directory level under cache-root: + //target-. Must be a single path component; + several names are refused outright because a reader elsewhere would stop + seeing the caches under them (see validate_cache_lineage in + scripts/cache-lib.sh). Pass the same value to cargo-cache-publish. + required: false + default: '' protected-branches: description: >- Space-separated refs that publish snapshots and are never evicted. @@ -54,8 +66,9 @@ inputs: watermark-file: description: >- Name of this job's build-watermark file inside the target dir. MUST be - distinct per job when two jobs share one cache key. Defaults to - .ci-watermark--sha. + distinct per job when two jobs share one target directory — which two + jobs no longer need to do; cache-lineage gives them separate ones. + Defaults to .ci-watermark--sha, already distinct per job. required: false default: '' lock-id: @@ -74,6 +87,12 @@ outputs: cache-key: description: 'Sanitized cache key for this run''s own ref.' value: ${{ steps.resolve.outputs.cache-key }} + cache-root: + description: >- + Resolved cache root — cache-root, plus the lineage directory when one is + set. Every directory this action reads or writes is under it. Also + exported as CARGO_CACHE_ROOT. + value: ${{ steps.resolve.outputs.cache-root }} seeded-from: description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.' value: ${{ steps.seed.outputs.seeded-from }} @@ -95,6 +114,16 @@ runs: # `base_ref` is populated only for pull_request events. A push run has # nothing to layer over: its own ref IS the reference branch. It # publishes, it does not consume. + # + # The cache root is resolved first because every path below hangs off it. + # A lineage nests one directory level (`//target-`), + # which is what lets two jobs on ONE ref hold two build directories and so + # not serialise on Cargo's exclusive lock. It is resolved through + # cache-root.sh rather than interpolated here so the name is validated — + # several otherwise-reasonable lineage names put their whole subtree out of + # reach of a pass that has to see it. Everything downstream reads the + # resolved value, and it is exported as CARGO_CACHE_ROOT so the publish + # action can check it agrees with its own inputs. - id: resolve shell: bash run: | @@ -102,6 +131,8 @@ runs: SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts [ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; } echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV" + CACHE_ROOT=$(bash "${SCRIPTS}/cache-root.sh" resolve \ + "${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}") OWN_REF="${{ inputs.own-ref }}" [ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}" BASE_REF="${{ inputs.base-ref }}" @@ -116,7 +147,7 @@ runs: echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)" fi - TARGET_DIR="${{ inputs.cache-root }}/target-${OWN_KEY}" + TARGET_DIR="${CACHE_ROOT}/target-${OWN_KEY}" WATERMARK="${{ inputs.watermark-file }}" [ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha" LOCK_ID="${{ inputs.lock-id }}" @@ -128,10 +159,11 @@ runs: echo "base-key=${BASE_KEY}" echo "lock-id=${LOCK_ID}" echo "watermark-file=${WATERMARK}" + echo "cache-root=${CACHE_ROOT}" } >> "$GITHUB_OUTPUT" { echo "CARGO_TARGET_DIR=${TARGET_DIR}" - echo "CARGO_CACHE_ROOT=${{ inputs.cache-root }}" + echo "CARGO_CACHE_ROOT=${CACHE_ROOT}" echo "CARGO_CACHE_KEY=${OWN_KEY}" echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}" echo "CI_WATERMARK_FILE=${WATERMARK}" @@ -155,7 +187,7 @@ runs: bash "${SCRIPTS}/seed-target-dir.sh" \ "${{ steps.resolve.outputs.cache-key }}" \ "${{ steps.resolve.outputs.base-key }}" \ - "${{ inputs.cache-root }}" \ + "${{ steps.resolve.outputs.cache-root }}" \ "${{ github.job }}-${{ github.run_id }}-$$" \ "${{ inputs.seed-fallback-dir }}" \ "${{ steps.resolve.outputs.lock-id }}" @@ -206,7 +238,7 @@ runs: set -euo pipefail SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts bash "${SCRIPTS}/prune-cache.sh" \ - "${{ inputs.cache-root }}" \ + "${{ steps.resolve.outputs.cache-root }}" \ "${{ steps.resolve.outputs.target-dir }}" \ "${{ inputs.protected-branches }}" \ "${{ inputs.min-free-percent }}" diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 6eabffa..7662251 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -114,6 +114,106 @@ cache_key() { target_dir_for() { printf '%s/target-%s' "$1" "$2"; } snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; } +# --------------------------------------------------------------------------- +# Cache lineages +# --------------------------------------------------------------------------- +# +# A cache key names a REF. What a target directory holds is the product of a +# ref and a BUILD CONFIGURATION, and the two are not the same thing: emowheel +# builds the same ref twice on every push, once for the host and once for +# wasm32, in two jobs that run concurrently. Keyed on the ref alone both +# resolve to one CARGO_TARGET_DIR, and Cargo's build-directory lock is +# exclusive — so the second job sits on `Blocking waiting for file lock on +# build directory` for the length of the first, holding a runner capacity slot +# while doing nothing (daniel/gitdan#60). +# +# A lineage is that second dimension, and it is expressed as ONE DIRECTORY +# LEVEL above the per-ref directories rather than as a suffix on the key: +# +# /target- no lineage (the flat layout) +# //target- a lineage +# +# Nesting rather than suffixing is what keeps every existing reader correct +# without teaching any of them a new name shape. `prune-cache.sh` resolves +# liveness by recomputing `target-` for every branch on +# origin and evicting whatever does not match — a suffixed `target--wasm32` +# matches nothing, so it would be classified dead and unconditionally evicted +# on every single run. The host-level arbiter in daniel/gitdan reads the same +# shape (its BRANCH_DIR_RE), and a suffixed name falls out of it too: not +# evicted there, but never a candidate either, so a whole lineage becomes +# invisible to the global disk budget. Nesting leaves both matchers reading +# exactly the names they already read, one directory deeper. +# +# ONE LEVEL, AND NOT TWO. The arbiter walks a volume to CI_CACHE_MAX_DEPTH, +# which is 2 — `_data/target-` and `_data//target-`. It is +# kept tight there on purpose (a deeper walk starts meeting Cargo's own +# `incremental/-` directories, which match the same name shape and +# must never be evicted individually), so a lineage is a single path component +# and validate_cache_lineage refuses one containing a slash. + +# Directory names daniel/gitdan's ci-cache-reclaim.sh refuses to descend into +# (its CI_CACHE_NODESCEND_NAMES). A lineage named one of these puts its whole +# subtree outside the global arbiter's reach: the caches accumulate and the one +# script whose job is the shared disk budget cannot see them. +CACHE_LINEAGE_RESERVED_NAMES="debug release deps incremental build .fingerprint tmp examples doc" + +# The shape that same script reads as a per-branch cache directory (its +# BRANCH_DIR_RE). A lineage matching it is taken for a cache dir in its own +# right — never descended into, and an eviction candidate whole, which is the +# entire lineage rather than one ref's share of it. +CACHE_LINEAGE_BRANCH_DIR_RE='^.+-[0-9a-f]{7,40}$' + +# Every rejection below names the reader that imposes it, because that is the +# only way the constraint survives: none of these is a filesystem limit, and a +# name that trips one produces no error anywhere — it produces a lineage that +# silently stops being pruned, or silently stops being reclaimed. +validate_cache_lineage() { + local lineage="$1" reserved + [ -n "$lineage" ] || return 0 + + case "$lineage" in + */*) + echo "::error::cache lineage '${lineage}' must be a single path component: the host-level arbiter walks a cache volume to depth 2, so //target- is as deep as a cache directory may sit and still be reclaimable" >&2 + return 1 + ;; + .*) + echo "::error::cache lineage '${lineage}' must not start with a dot: every dot-prefixed entry under a cache root belongs to the leftover-naming contract (see the top of this file), and a lineage is not garbage to be reclaimed" >&2 + return 1 + ;; + target-* | snapshot-*) + echo "::error::cache lineage '${lineage}' must not start with 'target-' or 'snapshot-': prune-cache.sh globs both prefixes at the cache root, so the lineage directory itself would become an eviction candidate" >&2 + return 1 + ;; + *[!A-Za-z0-9._-]*) + echo "::error::cache lineage '${lineage}' may contain only [A-Za-z0-9._-] — the same charset cache_key() sanitises a ref down to" >&2 + return 1 + ;; + esac + + for reserved in $CACHE_LINEAGE_RESERVED_NAMES; do + if [ "$lineage" = "$reserved" ]; then + echo "::error::cache lineage '${lineage}' is one of the Cargo directory names daniel/gitdan's ci-cache-reclaim.sh never descends into (CI_CACHE_NODESCEND_NAMES) — every cache under it would be invisible to the host-level disk budget" >&2 + return 1 + fi + done + + if [[ $lineage =~ $CACHE_LINEAGE_BRANCH_DIR_RE ]]; then + echo "::error::cache lineage '${lineage}' ends in a hex suffix, which is the shape daniel/gitdan's ci-cache-reclaim.sh reads as a per-branch cache directory — it would treat the lineage directory as one cache and evict the whole thing" >&2 + return 1 + fi + + return 0 +} + +# The cache root a lineage's directories actually live under. An empty lineage +# resolves to the cache root unchanged, byte for byte: that is what makes this +# a no-op for every consumer that does not set one, rather than a migration. +cache_root_for() { + local root="$1" lineage="${2:-}" + validate_cache_lineage "$lineage" || return 1 + printf '%s%s' "$root" "${lineage:+/$lineage}" +} + # --------------------------------------------------------------------------- # Disk accounting # --------------------------------------------------------------------------- diff --git a/scripts/cache-root-selftest.sh b/scripts/cache-root-selftest.sh new file mode 100755 index 0000000..f33fee7 --- /dev/null +++ b/scripts/cache-root-selftest.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# Regression test for cache-root.sh and the lineage rules in cache-lib.sh — +# the fix for daniel/gitdan#60, where two jobs building the same ref for +# different targets resolved to one CARGO_TARGET_DIR and serialised on Cargo's +# exclusive build-directory lock. +# +# 1. NO LINEAGE CHANGES NOTHING — the effective root is the cache root byte +# for byte, so every consumer that does not set a lineage keeps the exact +# directories it already has on the volume. This is the whole of the +# migration story for lublub, zemyna and emowheel's `ci` job, so it is +# asserted rather than assumed. +# 2. TWO LINEAGES, ONE REF, TWO TARGET DIRS — the bug itself. The two jobs +# keep one cache key (they are the same ref) and still get directories +# that are neither equal nor nested one inside the other, which is what +# Cargo's per-directory lock needs in order not to serialise them. +# 3. THE WHOLE PIPELINE MOVES TOGETHER — seed, publish and prune all operate +# inside the lineage root. A pass run in one lineage must not evict, or +# even see, a sibling lineage's caches or the flat layout's. +# 4. BASE SEEDING IS PER LINEAGE — a PR branch layers over ITS OWN lineage's +# base snapshot, not over whatever the flat root happens to hold. This is +# the property that keeps a warm start for both jobs rather than one. +# 5. A NAME NO READER CAN HANDLE IS REFUSED AT RESOLVE TIME — one assertion +# per constraint, each named for the reader that imposes it. None of these +# is a filesystem limit: every one of them produces a working directory +# that some pass silently stops seeing, which is the failure mode this +# whole scheme exists to avoid rather than to relocate. +# 6. A PUBLISH THAT DISAGREES WITH ITS CONSUME STEP FAILS LOUDLY — the +# footgun the lineage input introduces. cargo-cache-publish derives both +# ends of the snapshot swap from its own `cache-root`, so a publish step +# left at the default while its consume step nested would republish the +# OTHER lineage's live target dir over that lineage's snapshot, silently. +set -euo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$script_dir/cache-lib.sh" + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +root="$scratch/cache"; mkdir -p "$root" +pass_count=0 +fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; } +ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; } + +resolve() { bash "$script_dir/cache-root.sh" resolve "$@"; } + +# A tree that looks enough like a Cargo target dir for the pipeline scripts, +# with a marker naming which lineage produced it — scenario 4 turns on reading +# that marker back out of a seeded directory. +make_tree() { + local d="$1" marker="$2" + mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x" + echo "$marker" > "$d/debug/deps/libx.rlib" + echo "$marker" > "$d/lineage-marker" + echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x" +} + +echo "=== 1. no lineage changes nothing ===" + +[ "$(resolve /cache)" = /cache ] || fail "an omitted lineage changed the root" +[ "$(resolve /cache '')" = /cache ] || fail "an empty lineage changed the root" +ok "no lineage resolves to the cache root unchanged" + +KEY=$(cache_key dev) +[ "$(target_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/target-$KEY" ] \ + || fail "the flat target-dir name moved" +[ "$(snapshot_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/snapshot-$KEY" ] \ + || fail "the flat snapshot name moved" +ok "the flat layout's directory names are untouched" + +echo +echo "=== 2. two lineages, one ref, two target dirs ===" + +HOST_ROOT=$(resolve /cache) +WASM_ROOT=$(resolve /cache wasm32) +[ "$WASM_ROOT" = /cache/wasm32 ] || fail "lineage root resolved to '$WASM_ROOT'" +HOST_DIR=$(target_dir_for "$HOST_ROOT" "$KEY") +WASM_DIR=$(target_dir_for "$WASM_ROOT" "$KEY") +[ "$HOST_DIR" != "$WASM_DIR" ] || fail "both lineages resolved to $HOST_DIR" +case "$WASM_DIR" in "$HOST_DIR"/*) fail "the wasm target dir sits inside the host one" ;; esac +case "$HOST_DIR" in "$WASM_DIR"/*) fail "the host target dir sits inside the wasm one" ;; esac +ok "one cache key ($KEY), two disjoint target dirs: $HOST_DIR and $WASM_DIR" + +echo +echo "=== 3. the whole pipeline moves together ===" + +DEAD=$(cache_key feat/dead) +lin="$root/wasm32" +mkdir -p "$lin" +make_tree "$root/target-$DEAD" flat +make_tree "$root/native/target-$DEAD" native +make_tree "$lin/target-$DEAD" wasm32 +bash "$script_dir/seed-target-dir.sh" "$KEY" "" "$lin" tag-3 > "$scratch/seed3.log" 2>&1 \ + || { cat "$scratch/seed3.log"; fail "seed inside a lineage root failed"; } +[ -d "$lin/target-$KEY" ] || fail "seed did not create $lin/target-$KEY" +[ -d "$root/target-$KEY" ] && fail "seed created a directory at the flat root as well" +ok "seed creates its directory under the lineage root and nowhere else" + +make_tree "$lin/target-$KEY" wasm32 +bash "$script_dir/publish-snapshot.sh" "$KEY" "$lin" tag-3 > "$scratch/pub3.log" 2>&1 \ + || { cat "$scratch/pub3.log"; fail "publish inside a lineage root failed"; } +[ -d "$lin/snapshot-$KEY" ] || fail "publish did not create $lin/snapshot-$KEY" +[ -d "$root/snapshot-$KEY" ] && fail "publish created a snapshot at the flat root as well" +ok "publish writes its snapshot under the lineage root and nowhere else" + +# Free space far below the threshold, so pass 2 evicts every eligible +# directory it can see. What it can see is the point of the scenario. +CACHE_LIVENESS=false CACHE_DF_OVERRIDE="1000000 1000" \ + bash "$script_dir/prune-cache.sh" "$lin" "$lin/target-$KEY" 'dev main' 10 \ + > "$scratch/prune3.log" 2>&1 || { cat "$scratch/prune3.log"; fail "prune inside a lineage root failed"; } +[ -d "$lin/target-$DEAD" ] && { cat "$scratch/prune3.log"; fail "prune left its own lineage's evictable cache in place"; } +[ -d "$root/target-$DEAD" ] || fail "prune reached out of its lineage and evicted the flat root's cache" +[ -d "$root/native/target-$DEAD" ] || fail "prune reached into a sibling lineage and evicted its cache" +ok "prune under disk pressure evicts inside its own lineage only" + +echo +echo "=== 4. base seeding is per lineage ===" + +BASE=$(cache_key dev) +PR=$(cache_key feat/pr) +rm -rf "$lin" "$root/snapshot-$BASE" +mkdir -p "$lin" +make_tree "$root/snapshot-$BASE" flat-base +make_tree "$lin/snapshot-$BASE" wasm32-base +bash "$script_dir/seed-target-dir.sh" "$PR" "$BASE" "$lin" tag-4 > "$scratch/seed4.log" 2>&1 \ + || { cat "$scratch/seed4.log"; fail "seeding a PR branch inside a lineage failed"; } +[ -d "$lin/target-$PR" ] || fail "the PR branch's lineage target dir was not created" +got=$(cat "$lin/target-$PR/lineage-marker") +[ "$got" = wasm32-base ] || fail "the PR branch layered over '$got', not its own lineage's base snapshot" +grep -q 'base snapshot' "$scratch/seed4.log" || { cat "$scratch/seed4.log"; fail "seed did not report a base-snapshot clone"; } +ok "a PR branch layers over its own lineage's base snapshot ($got)" + +echo +echo "=== 5. a name no reader can handle is refused ===" + +reject() { + local lineage="$1" want="$2" desc="$3" out + if out=$(resolve /cache "$lineage" 2>&1); then + fail "lineage '$lineage' was accepted (resolved to '$out') — $desc" + fi + case "$out" in + *"$want"*) ;; + *) fail "lineage '$lineage' was rejected without naming '$want': $out" ;; + esac + ok "rejected '$lineage' — $desc" +} + +reject 'a/b' 'single path component' "the arbiter walks a volume to depth 2" +reject '.hidden' 'dot' "every dot-prefixed name under a cache root is leftover-contract territory" +reject 'target-x' 'prune-cache.sh' "prune-cache.sh globs target-* at the cache root" +reject 'snapshot-x' 'prune-cache.sh' "prune-cache.sh globs snapshot-* at the cache root" +reject 'wasm 32' 'A-Za-z0-9._-' "a cache key is sanitised to that charset and a lineage sits beside one" +reject 'release' 'CI_CACHE_NODESCEND_NAMES' "the arbiter never descends into a Cargo profile name" +reject 'doc' 'CI_CACHE_NODESCEND_NAMES' "same, for the docs profile directory" +reject 'lineage-deadbeef' 'per-branch cache directory' "the arbiter reads a hex-suffixed name as one cache dir" + +for good in wasm32 web android host wasm32.release lineage_2; do + out=$(resolve /cache "$good") || fail "lineage '$good' was rejected: $out" + [ "$out" = "/cache/$good" ] || fail "lineage '$good' resolved to '$out'" +done +ok "ordinary lineage names still resolve" + +echo +echo "=== 6. a publish that disagrees with its consume step fails loudly ===" + +verify() { bash "$script_dir/cache-root.sh" verify "$@"; } + +verify /cache wasm32 /cache/wasm32 > /dev/null 2>&1 \ + || fail "verify rejected a publish step that agrees with its consume step" +verify /cache '' /cache > /dev/null 2>&1 \ + || fail "verify rejected an unmigrated consumer's matching default pair" +ok "verify accepts a publish step whose inputs match what the consume step exported" + +# The exact shape of the mistake: the consume step nested, the publish step +# kept the default. Left unchecked this republishes the host lineage's live +# target dir over the host lineage's snapshot, from the wasm job. +if out=$(verify /cache '' /cache/wasm32 2>&1); then + fail "verify accepted a publish step that resolved to /cache while the job exported /cache/wasm32" +fi +case "$out" in + *'/cache/wasm32'*) ;; + *) fail "the mismatch error does not quote what the consume step exported: $out" ;; +esac +case "$out" in + *'SAME cache-root and cache-lineage'*) ;; + *) fail "the mismatch error does not say what to do about it: $out" ;; +esac +ok "verify rejects a publish step that forgot the lineage, and says so" + +if verify /cache 'a/b' /cache/a/b > /dev/null 2>&1; then + fail "verify accepted an invalid lineage as long as both sides agreed on it" +fi +ok "verify validates the lineage as well as comparing it" + +echo +echo "cache-root-selftest: ${pass_count} assertions passed" diff --git a/scripts/cache-root.sh b/scripts/cache-root.sh new file mode 100755 index 0000000..eb218ec --- /dev/null +++ b/scripts/cache-root.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Resolves — and cross-checks — the cache root a job's directories live under. +# +# cache-root.sh resolve [lineage] +# cache-root.sh verify +# +# `resolve` prints the effective root: the cache root unchanged when no lineage +# is given, or `/` when one is. Invalid lineage names are +# rejected here rather than downstream — see validate_cache_lineage() in +# cache-lib.sh, where every rejection names the reader that imposes it. +# +# `verify` is the publish side's guard. cargo-cache-publish resolves the same +# two inputs the consume action was given and compares the result against the +# CARGO_CACHE_ROOT the consume step exported into the job environment. The two +# actions have always had to agree — `cache-root`'s description in the publish +# action says "must match the consume action" — and until a lineage existed +# they always did, because nobody overrode the default. A disagreement is not +# a harmless no-op: publish-snapshot.sh takes the root as an argument and +# derives BOTH ends of the swap from it, so a publish step that kept the +# default while its consume step nested would read `/target-` — the +# OTHER lineage's live target dir — and republish it over `/snapshot-`, +# which is that lineage's snapshot. Two jobs would then be publishing one +# snapshot from one tree on every push, and nothing in either action would say +# so. Hence: fail the job, loudly, rather than resolve the ambiguity in +# either direction. +# +# A thin CLI over cache-lib.sh, kept as its own entry point for the same +# reason branch-cache-key.sh is: an out-of-band job that needs to find a +# lineage's directories should resolve the path the way the action does +# instead of reimplementing the rule. +set -euo pipefail +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" + +MODE="${1:-}" +case "$MODE" in + resolve) + [ $# -ge 2 ] && [ $# -le 3 ] || { + echo "::error::cache-root.sh resolve: expected [lineage]" >&2 + exit 1 + } + [ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; } + cache_root_for "$2" "${3:-}" + ;; + verify) + [ $# -eq 4 ] || { + echo "::error::cache-root.sh verify: expected " >&2 + exit 1 + } + [ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; } + expected=$(cache_root_for "$2" "$3") + if [ "$expected" != "$4" ]; then + echo "::error::cache-root.sh: this step resolves its cache root to '${expected}' (cache-root '$2', cache-lineage '$3') but the cargo-cache step in this job exported '$4'. Pass the SAME cache-root and cache-lineage to both actions." >&2 + exit 1 + fi + echo "cache root: ${expected} (agrees with the cargo-cache step in this job)" + ;; + *) + echo "::error::cache-root.sh: unknown mode '${MODE}' (expected resolve or verify)" >&2 + exit 1 + ;; +esac diff --git a/scripts/selftest.sh b/scripts/selftest.sh index 9bf3f88..0c7168a 100755 --- a/scripts/selftest.sh +++ b/scripts/selftest.sh @@ -13,7 +13,7 @@ script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) FAST=0 [ "${1:-}" = "--fast" ] && FAST=1 -FIXTURE_TESTS=(seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh) +FIXTURE_TESTS=(cache-root-selftest.sh seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh) CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh) TESTS=("${FIXTURE_TESTS[@]}")