diff --git a/README.md b/README.md index 45d826b..f5bdd5f 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,10 @@ env: the only thing a consumer ever clones from. ``` +Transient dot-prefixed entries appear alongside those two — staging trees, +eviction asides, reader markers. Their names are a contract with the host that +owns the volume; see [Scratch names in a cache root](#scratch-names-in-a-cache-root-are-a-cross-repo-contract). + `` is the ref sanitised to a safe path component, capped at 48 characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise @@ -228,6 +232,54 @@ last build, where the historically-correct mtime is exactly the wrong answer. --- +## Scratch names in a cache root are a cross-repo contract + +Read this before adding a dot-prefixed directory under a cache root. + +The volume is not swept by these scripts alone. A host-level arbiter — +daniel/gitdan's `scripts/ci-cache-reclaim.sh`, which runs outside any job — +reclaims the scratch trees a killed job strands here, and reads the reader +markers to decide whether a tree is still live. Its `LEFTOVER NAMING CONTRACT` +block is the canonical description; this side owns the names. + +| name | produced by | if the job dies holding it | +|---|---|---| +| `.stage-` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it | +| `.publish-new-` | `publish-snapshot.sh` | stranded, and not in the arbiter's list | +| `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key | +| `.evicting--` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass | +| `.reading--` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below | + +A `.reading-*` marker is protective, not scratch: it is how both this repo's +prune pass and the arbiter tell an in-flight clone from an abandoned one, and +the arbiter never deletes one. Delete a live marker and the tree it covers +becomes eligible for an unlink underneath the walk that is reading it, which is +the silent truncation the whole interlock exists to prevent. + +**The rule: no new dot-prefixed entry under a cache root without a matching +prefix in `ci-cache-reclaim.sh`.** Adding a shape counts exactly as much as +renaming one. That script enumerates by explicit prefix rather than by dotglob +— deliberately, because a dotglob would pull reader markers into the candidate +stream alongside the trees they protect — so a name it has not been told about +is not handled conservatively, it is invisible, and an unreclaimed staging tree +is a full clone of a multi-GB target dir on the one volume whose entire problem +is disk. `.publish-new-` is the standing example: it strands exactly as +`.stage-` does and is in neither side's sweep. + +**The staleness constants are part of the same contract, and that half has a +direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and +`STALE_LOCK_SECONDS` (`prune-cache.sh`) are mirrored there, and the arbiter's +copies must be **greater than or equal to** these. Raising one here for longer +jobs, without raising its mirror first, makes the arbiter treat a marker whose +owner still considers it live as stale and delete a tree under an in-flight +clone — and its own minimum-age guard does not back-stop that, since a clone +holding a three-hour-old marker has a roughly three-hour-old staging tree. +Lowering either here needs no coordination: the arbiter then only defers a +reclamation this side would already have permitted, which costs disk rather +than correctness. + +--- + ## Inputs ### `cargo-cache` diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 55ceb9d..6927e96 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -8,6 +8,64 @@ # needs as an argument, so the selftests can drive them against scratch # directories without a CI context. +# --------------------------------------------------------------------------- +# CROSS-REPO CONTRACT: the dot-prefixed names left in a cache root +# --------------------------------------------------------------------------- +# +# The volume these scripts write into is also swept by a host-level arbiter +# that runs outside any job and outside this repo: daniel/gitdan's +# `scripts/ci-cache-reclaim.sh`. It reclaims the dot-prefixed trees a killed +# job strands here, and it reads the reader markers below to decide whether one +# of those trees is still live. Its `LEFTOVER NAMING CONTRACT` block is the +# canonical description of the arrangement; what follows is the producing +# side's half — which names this repo creates, and what changing one obliges. +# +# Created directly under a cache root: +# +# .stage- cache-lib.sh, hardlink_clone_into(): the tree a +# clone is built in before the atomic rename that +# gives it its real name. is unique per job +# per run, so a job killed before the rename +# strands a whole hardlink clone under a name no +# later run reuses. Nothing in this repo sweeps it. +# .publish-new- publish-snapshot.sh: the staged snapshot, between +# its clone and the swap. Strands the same way. +# .publish-old-- publish-snapshot.sh: the rotated-away generation, +# kept while a reader still holds it and swept by +# the next publish of the same key. +# .evicting-- prune-cache.sh, evict_dir(): a cache renamed +# aside so the decision to unlink it can be retaken +# after the rename. Swept at the start of every +# prune pass, so it only strands when this repo's +# workflow stops running at all. +# .reading-- cache-lib.sh, reader_lock_acquire(): NOT garbage. +# It is the live-reader signal the arbiter reads, +# and the one shape it must never delete — removing +# one clears the way to unlink a tree out from +# under an in-flight walk, which is the silent +# truncation this whole interlock exists to +# prevent. +# +# THE RULE, which the arbiter states as its own: no new dot-prefixed entry +# under a cache root without a matching prefix in that script. ADDING a shape +# counts exactly as much as renaming one, because that script enumerates by +# explicit prefix rather than by dotglob — deliberately, since a dotglob would +# pull reader markers into the candidate stream alongside the trees they +# protect. A shape it has not been told about is not handled conservatively, it +# is invisible: an unreclaimed staging tree is a full clone of a multi-GB +# target dir on the one volume whose entire problem is disk. +# +# Of the five above, the arbiter enumerates `.stage-`, `.evicting-` and +# `.reading-`. The two `.publish-*` names predate the contract and are not in +# it, which is why `.publish-new-` — tagged per job per run exactly as +# `.stage-` is — is a shape neither side reclaims today. Closing that is a +# change over there, not here; the rule above is what stops the list growing +# another one. +# +# The two staleness constants the arbiter mirrors are part of the same +# contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS +# below and STALE_LOCK_SECONDS in prune-cache.sh. + # --------------------------------------------------------------------------- # Cache keys # --------------------------------------------------------------------------- @@ -249,8 +307,24 @@ CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}" # A marker older than this belongs to a job the runner killed before it could # clean up. Honouring one forever would let a crashed job pin an entire # snapshot generation on disk permanently. +# +# RAISING THIS IS A CROSS-REPO CHANGE, and the drift is not symmetric. +# daniel/gitdan's ci-cache-reclaim.sh mirrors this value as +# CI_CACHE_READER_STALE_SECONDS (and CI_CACHE_LEFTOVER_MIN_AGE_SECONDS beside +# it), and its copies must be GREATER THAN OR EQUAL TO this one. Raise this for +# longer jobs while that one stays at 7200 and the arbiter reads a marker whose +# owner still considers it live as stale, then deletes the tree under an +# in-flight clone; its minimum-age guard does not back-stop that, because a +# clone holding a three-hour-old marker has a staging tree roughly three hours +# old too, so both of its guards pass. Raise theirs first. Lowering this one +# needs no coordination at all: the arbiter then defers a reclamation this side +# would already have permitted, which costs disk and not correctness. CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}" +# `.reading--` is a contract name, not a private one: the +# host-level arbiter reads these to tell a live clone from an abandoned one, +# and never deletes one. See the cross-repo contract at the top of this file +# before changing the spelling. reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; } reader_lock_acquire() { @@ -388,6 +462,10 @@ hardlink_clone_into() { parent=$(dirname "$dst") src_name=$(basename "$src") + # `.stage-` is a contract name (see the top of this file): a job killed + # between the copy below and the rename at the end strands this tree, and the + # only thing that ever reclaims one is the host-level arbiter, by this exact + # prefix. tmp="${parent}/.stage-${tag}" attempt=1 diff --git a/scripts/prune-cache.sh b/scripts/prune-cache.sh index 23f07a5..6efa4fc 100755 --- a/scripts/prune-cache.sh +++ b/scripts/prune-cache.sh @@ -85,6 +85,10 @@ ROOT="${1:?usage: prune-cache.sh = this +# one — raising this without raising theirs first lets that script treat a lock +# this side still honours as abandoned. Same direction and same reasoning as +# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh. STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}" # An aside directory is in flight for one rename plus one marker glob — # milliseconds. Anything older belongs to a pass that died between the two, so @@ -174,6 +178,12 @@ is_locked() { # leaves the source inode unchanged, which is precisely why the publish side # can rotate a snapshot out from under a live reader. A declined eviction # therefore costs a deferred eviction and nothing else. +# +# `.evicting--` is a contract name shared with daniel/gitdan's +# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh). +# The sweep at the top of this script reclaims these on every pass, so the only +# one that reaches the arbiter belongs to a repo whose workflow has stopped +# running — which is exactly the case no in-workflow pass can reach. evict_dir() { local dir="$1" name aside name=$(basename "$dir") diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index 3f33682..d040864 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -51,6 +51,16 @@ SRC=$(target_dir_for "$ROOT" "$OWN_KEY") DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") # Keyed by cache key as well as tag, so a deferred generation can be matched # back to the snapshot whose readers must drain before it is safe to reclaim. +# +# `.publish-old-` and `.publish-new-` are dot-prefixed names under a cache +# root, so the cross-repo contract at the top of cache-lib.sh covers them: +# renaming either, or adding a third, requires a matching prefix in +# daniel/gitdan's ci-cache-reclaim.sh. Neither is in its list today, and +# `.publish-new-` is the shape that most needs to be — it is tagged per job per +# run, so a publisher killed between staging its snapshot and the swap below +# strands a full hardlink clone that no later run of this script will ever +# match. `.publish-old-` is milder: the sweep further down reclaims it on the +# next publish of the same key. OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" SNAP_NAME=$(basename "$DST") GRACE="${CACHE_READ_GRACE_SECONDS}"