Files
gitdan-actions/cargo-cache/action.yml
T
claude fb7a788c90
CI / shellcheck + selftests (pull_request) Failing after 1m19s
feat(cache): give same-ref jobs separate build directories via cache-lineage
A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.

`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:

    <cache-root>/target-<key>              no lineage (unchanged)
    <cache-root>/<lineage>/target-<key>    a lineage

Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).

Every interacting part, checked rather than assumed:

- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
  a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
  The publish action now takes the root from the `CARGO_CACHE_ROOT` the
  consume step exported, and CHECKS its own inputs against it — a publish step
  left at the default while its consume step nested would otherwise republish
  a different lineage's live target dir over that lineage's snapshot, on every
  push, silently. `mode: release-lock` is exempt: it releases a lock on
  `$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
  lineage neither evicts nor sees a sibling's caches, or the flat layout's.
  Liveness keeps resolving real branch names, which is what a key suffix would
  have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
  all six nested and flat dirs collected as candidates, protection resolved
  correctly on the nested ones, and a `.stage-` stranded inside the lineage
  found by the leftover sweep.

Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.

Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.

New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
2026-08-26 12:32:21 -05:00

245 lines
11 KiB
YAML

name: 'Cargo cache (consume)'
description: >-
Per-ref Cargo target-dir cache for Gitea Actions runners: hardlink-clones
this ref's target directory from its base branch's published, immutable
snapshot, restores git-history file mtimes, and prunes the volume.
author: 'gitdan'
inputs:
cache-root:
description: 'Mount point of the persistent cache volume inside the job container.'
required: false
default: '/cache'
cache-lineage:
description: >-
Distinguishes two jobs that build the SAME ref for different targets or
profiles (a host build and a wasm32 build, say) and would otherwise
resolve to one CARGO_TARGET_DIR and serialise on Cargo's exclusive
build-directory lock. Names one directory level under cache-root:
<cache-root>/<lineage>/target-<key>. Must be a single path component;
several names are refused outright because a reader elsewhere would stop
seeing the caches under them (see validate_cache_lineage in
scripts/cache-lib.sh). Pass the same value to cargo-cache-publish.
required: false
default: ''
protected-branches:
description: >-
Space-separated refs that publish snapshots and are never evicted.
These are the branches PR caches layer over.
required: false
default: 'dev main'
min-free-percent:
description: 'Prune when free space on the cache volume drops below this percentage.'
required: false
default: '10'
restore-mtimes:
description: >-
Restore every tracked file's mtime from git history. Requires a
full-history checkout (fetch-depth: 0). Set to false only if the build
does not use Cargo's mtime-based freshness at all.
required: false
default: 'true'
prune:
description: 'Run the eviction pass (dead-branch liveness + disk pressure).'
required: false
default: 'true'
liveness-prune:
description: >-
Within the prune pass, remove caches for branches that no longer exist
on origin. Set to false on a runner that cannot reach origin.
required: false
default: 'true'
own-ref:
description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.'
required: false
default: ''
base-ref:
description: 'Override the ref to layer over. Defaults to github.base_ref (empty on push).'
required: false
default: ''
seed-fallback-dir:
description: >-
Absolute path to seed from when no snapshot exists yet — a pre-existing
flat cache directory during a migration. Optional.
required: false
default: ''
watermark-file:
description: >-
Name of this job's build-watermark file inside the target dir. MUST be
distinct per job when two jobs share one target directory — which two
jobs no longer need to do; cache-lineage gives them separate ones.
Defaults to .ci-watermark-<job>-sha, already distinct per job.
required: false
default: ''
lock-id:
description: 'Identifier for this job''s cache lock. Defaults to <job>-<run_id>.'
required: false
default: ''
stale-lock-seconds:
description: 'Age past which another job''s cache lock is treated as abandoned.'
required: false
default: '7200'
outputs:
target-dir:
description: 'Resolved CARGO_TARGET_DIR. Also exported to the job environment.'
value: ${{ steps.resolve.outputs.target-dir }}
cache-key:
description: 'Sanitized cache key for this run''s own ref.'
value: ${{ steps.resolve.outputs.cache-key }}
cache-root:
description: >-
Resolved cache root — cache-root, plus the lineage directory when one is
set. Every directory this action reads or writes is under it. Also
exported as CARGO_CACHE_ROOT.
value: ${{ steps.resolve.outputs.cache-root }}
seeded-from:
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
value: ${{ steps.seed.outputs.seeded-from }}
runs:
using: 'composite'
steps:
# Resolves both cache keys and exports the environment every later step
# (and the consuming job's own build steps) reads. Must run before
# anything that touches CARGO_TARGET_DIR, which is why it is first.
#
# `head_ref || ref_name` rather than `ref_name` alone: on a pull_request
# event `ref_name` is a synthetic merge-ref name that changes on every
# push to the PR, so keying on it would give the same PR a different cache
# directory every time — defeating the reuse this action exists to
# provide. On a push event `head_ref` is empty and `ref_name` is the real
# branch, which is what we want there.
#
# `base_ref` is populated only for pull_request events. A push run has
# nothing to layer over: its own ref IS the reference branch. It
# publishes, it does not consume.
#
# The cache root is resolved first because every path below hangs off it.
# A lineage nests one directory level (`<root>/<lineage>/target-<key>`),
# which is what lets two jobs on ONE ref hold two build directories and so
# not serialise on Cargo's exclusive lock. It is resolved through
# cache-root.sh rather than interpolated here so the name is validated —
# several otherwise-reasonable lineage names put their whole subtree out of
# reach of a pass that has to see it. Everything downstream reads the
# resolved value, and it is exported as CARGO_CACHE_ROOT so the publish
# action can check it agrees with its own inputs.
- id: resolve
shell: bash
run: |
set -euo pipefail
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
CACHE_ROOT=$(bash "${SCRIPTS}/cache-root.sh" resolve \
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}")
OWN_REF="${{ inputs.own-ref }}"
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
BASE_REF="${{ inputs.base-ref }}"
[ -n "$BASE_REF" ] || BASE_REF="${{ github.base_ref }}"
OWN_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$OWN_REF")
BASE_KEY=""
if [ -n "$BASE_REF" ]; then
BASE_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$BASE_REF")
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY}; layering over base ref '${BASE_REF}' -> ${BASE_KEY}"
else
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
fi
TARGET_DIR="${CACHE_ROOT}/target-${OWN_KEY}"
WATERMARK="${{ inputs.watermark-file }}"
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
LOCK_ID="${{ inputs.lock-id }}"
[ -n "$LOCK_ID" ] || LOCK_ID="${{ github.job }}-${{ github.run_id }}"
{
echo "target-dir=${TARGET_DIR}"
echo "cache-key=${OWN_KEY}"
echo "base-key=${BASE_KEY}"
echo "lock-id=${LOCK_ID}"
echo "watermark-file=${WATERMARK}"
echo "cache-root=${CACHE_ROOT}"
} >> "$GITHUB_OUTPUT"
{
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
echo "CARGO_CACHE_ROOT=${CACHE_ROOT}"
echo "CARGO_CACHE_KEY=${OWN_KEY}"
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
echo "CI_WATERMARK_FILE=${WATERMARK}"
} >> "$GITHUB_ENV"
# Seeds this ref's target dir from the base's published snapshot. See
# scripts/seed-target-dir.sh — the staging-then-atomic-rename is what
# makes concurrent jobs sharing one cache key safe by construction rather
# than by the runner happening to have a single execution slot, and the
# clone's own consistency check plus the publish side's reader interlock
# are what make it safe against the base republishing MID-CLONE.
#
# The lock id is passed here as well as acquired in the next step: the
# seed writes it into the staging tree, so the directory carries a lock
# the instant it appears under its final name rather than a step later.
- id: seed
shell: bash
run: |
set -euo pipefail
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
bash "${SCRIPTS}/seed-target-dir.sh" \
"${{ steps.resolve.outputs.cache-key }}" \
"${{ steps.resolve.outputs.base-key }}" \
"${{ steps.resolve.outputs.cache-root }}" \
"${{ github.job }}-${{ github.run_id }}-$$" \
"${{ inputs.seed-fallback-dir }}" \
"${{ steps.resolve.outputs.lock-id }}"
# Re-stamps the lock the seed step already wrote (acquiring is idempotent
# — it rewrites the timestamp) and stamps the LRU marker. The marker is
# touched unconditionally every run: a run that hits the cache for every
# crate may write nothing at all inside the tree, which would make a
# just-used directory look stale to the eviction pass.
#
# `.cache-last-used` IS A CROSS-REPO CONTRACT NAME with daniel/gitdan's
# `scripts/ci-cache-reclaim.sh`, which reads this exact marker for its own
# LRU ordering and to recognise a directory as a cache dir at all. Rename
# it here without a matching change there and that script silently falls
# back to directory mtime for both — see README.md's "Scratch names in a
# cache root" section and the `DEPENDED-UPON NAMES CONTRACT` block in
# gitdan's script.
- shell: bash
run: |
set -euo pipefail
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
bash "${SCRIPTS}/cache-lock.sh" acquire \
"${{ steps.resolve.outputs.target-dir }}" "${{ steps.resolve.outputs.lock-id }}"
touch "${{ steps.resolve.outputs.target-dir }}/.cache-last-used"
# Runs AFTER seeding, deliberately: restore-mtimes.sh reads the build
# watermark out of the target dir, so that directory has to be in its
# final form for this run (reused, seeded, or freshly created) before the
# watermark it may carry can be read.
# CARGO_TARGET_DIR and CI_WATERMARK_FILE are passed explicitly rather
# than read from the job environment the resolve step exported: the export
# is what the consuming workflow's own build steps rely on, but a step
# inside this action should not depend on cross-step propagation working
# when it can just be handed the value.
- if: ${{ inputs.restore-mtimes == 'true' }}
shell: bash
env:
CARGO_TARGET_DIR: ${{ steps.resolve.outputs.target-dir }}
CI_WATERMARK_FILE: ${{ steps.resolve.outputs.watermark-file }}
run: bash "$(cd "${{ github.action_path }}/.." && pwd)/scripts/restore-mtimes.sh"
- if: ${{ inputs.prune == 'true' }}
shell: bash
env:
STALE_LOCK_SECONDS: ${{ inputs.stale-lock-seconds }}
CACHE_LIVENESS: ${{ inputs.liveness-prune }}
run: |
set -euo pipefail
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
bash "${SCRIPTS}/prune-cache.sh" \
"${{ steps.resolve.outputs.cache-root }}" \
"${{ steps.resolve.outputs.target-dir }}" \
"${{ inputs.protected-branches }}" \
"${{ inputs.min-free-percent }}"