diff --git a/README.md b/README.md index 45d826b..9b27b21 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,10 @@ env: the only thing a consumer ever clones from. ``` +Transient dot-prefixed entries appear alongside those two — staging trees, +eviction asides, reader markers. Their names are a contract with the host that +owns the volume; see [Scratch names in a cache root](#scratch-names-in-a-cache-root-are-a-cross-repo-contract). + `` is the ref sanitised to a safe path component, capped at 48 characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise @@ -228,6 +232,63 @@ last build, where the historically-correct mtime is exactly the wrong answer. --- +## Scratch names in a cache root are a cross-repo contract + +Read this before adding a dot-prefixed directory under a cache root. + +The volume is not swept by these scripts alone. A host-level arbiter — +daniel/gitdan's `scripts/ci-cache-reclaim.sh`, which runs outside any job — +reclaims the scratch trees a killed job strands here, and reads the reader +markers to decide whether a tree is still live. Its `LEFTOVER NAMING CONTRACT` +block is the canonical description; this side owns the names. + +| name | produced by | if the job dies holding it | +|---|---|---| +| `.stage-` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it | +| `.publish-new-` | `publish-snapshot.sh` | stranded; see `daniel/gitdan#30` | +| `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key, if there is one | +| `.evicting--` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass | +| `.reading--` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below | + +A `.reading-*` marker is protective, not scratch: it is how both this repo's +prune pass and the arbiter tell an in-flight clone from an abandoned one, and +the arbiter never deletes one. Delete a live marker and the tree it covers +becomes eligible for an unlink underneath the walk that is reading it, which is +the silent truncation the whole interlock exists to prevent. + +**The rule: no new dot-prefixed entry under a cache root without a matching +prefix in `ci-cache-reclaim.sh`.** Adding a shape counts exactly as much as +renaming one. That script enumerates by explicit prefix rather than by dotglob +— deliberately, because a dotglob would pull reader markers into the candidate +stream alongside the trees they protect — so a name it has not been told about +is not handled conservatively, it is invisible, and an unreclaimed staging tree +is a full clone of a multi-GB target dir on the one volume whose entire problem +is disk. `.publish-new-` is the worked example: the two `.publish-*` names were +outside the contract when it was written, and `.publish-new-` strands exactly +as `.stage-` does, so it is the one that needed catching. Bringing both under +the arbiter's enumeration is tracked as `daniel/gitdan#30`. + +The two lists are meant to be the same length — the five names above, and the +prefix constants `ci-cache-reclaim.sh` *declares*. Not the subset it enumerates +as reclaim candidates: that one is smaller, because `.reading-` is read and +never swept. A mismatch means one side gained a shape without telling the +other, which is the drift the rule exists to catch and the cheapest thing to +check. + +**The staleness constants are part of the same contract, and that half has a +direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and +`STALE_LOCK_SECONDS` (`prune-cache.sh`) are mirrored there, and the arbiter's +copies must be **greater than or equal to** these. Raising one here for longer +jobs, without raising its mirror first, makes the arbiter treat a marker whose +owner still considers it live as stale and delete a tree under an in-flight +clone — and its own minimum-age guard does not back-stop that, since a clone +holding a three-hour-old marker has a roughly three-hour-old staging tree. +Lowering either here needs no coordination: the arbiter then only defers a +reclamation this side would already have permitted, which costs disk rather +than correctness. + +--- + ## Inputs ### `cargo-cache` diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 55ceb9d..2f94aea 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -8,6 +8,73 @@ # needs as an argument, so the selftests can drive them against scratch # directories without a CI context. +# --------------------------------------------------------------------------- +# CROSS-REPO CONTRACT: the dot-prefixed names left in a cache root +# --------------------------------------------------------------------------- +# +# The volume these scripts write into is also swept by a host-level arbiter +# that runs outside any job and outside this repo: daniel/gitdan's +# `scripts/ci-cache-reclaim.sh`. It reclaims the dot-prefixed trees a killed +# job strands here, and it reads the reader markers below to decide whether one +# of those trees is still live. Its `LEFTOVER NAMING CONTRACT` block is the +# canonical description of the arrangement; what follows is the producing +# side's half — which names this repo creates, and what changing one obliges. +# +# Created directly under a cache root: +# +# .stage- cache-lib.sh, hardlink_clone_into(): the tree a +# clone is built in before the atomic rename that +# gives it its real name. is unique per job +# per run, so a job killed before the rename +# strands a whole hardlink clone under a name no +# later run reuses. Nothing in this repo sweeps it. +# .publish-new- publish-snapshot.sh: the staged snapshot, between +# its clone and the swap. Strands the same way. +# .publish-old-- publish-snapshot.sh: the rotated-away generation, +# kept while a reader still holds it and swept by +# the next publish of the same key — if there ever +# is one. A merged or renamed branch never +# publishes again, and its last one stays. +# .evicting-- prune-cache.sh, evict_dir(): a cache renamed +# aside so the decision to unlink it can be retaken +# after the rename. Swept at the start of every +# prune pass, so it only strands when this repo's +# workflow stops running at all. +# .reading-- cache-lib.sh, reader_lock_acquire(): NOT garbage. +# It is the live-reader signal the arbiter reads, +# and the one shape it must never delete — removing +# one clears the way to unlink a tree out from +# under an in-flight walk, which is the silent +# truncation this whole interlock exists to +# prevent. +# +# THE RULE, which the arbiter states as its own: no new dot-prefixed entry +# under a cache root without a matching prefix in that script. ADDING a shape +# counts exactly as much as renaming one, because that script enumerates by +# explicit prefix rather than by dotglob — deliberately, since a dotglob would +# pull reader markers into the candidate stream alongside the trees they +# protect. A shape it has not been told about is not handled conservatively, it +# is invisible: an unreclaimed staging tree is a full clone of a multi-GB +# target dir on the one volume whose entire problem is disk. +# +# The two `.publish-*` names predate the contract and were outside it when this +# block was written — the rule catching an uncovered shape on its first +# application. `.publish-new-` is the one that mattered: it is tagged per job +# per run exactly as `.stage-` is, so a publisher killed before the swap +# strands a tree under a name no later run of that script matches, which is +# precisely the shape only the arbiter can reach. Bringing both in is tracked +# as daniel/gitdan#30. +# +# The two lists are meant to be the same length: the five names above, and the +# prefix constants that script DECLARES — not the subset it enumerates as +# reclaim candidates, which is smaller because `.reading-` is read and never +# swept. A shape here without a constant there is one side having changed +# without telling the other, and it is cheapest to notice by counting. +# +# The two staleness constants the arbiter mirrors are part of the same +# contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS +# below and STALE_LOCK_SECONDS in prune-cache.sh. + # --------------------------------------------------------------------------- # Cache keys # --------------------------------------------------------------------------- @@ -249,8 +316,24 @@ CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}" # A marker older than this belongs to a job the runner killed before it could # clean up. Honouring one forever would let a crashed job pin an entire # snapshot generation on disk permanently. +# +# RAISING THIS IS A CROSS-REPO CHANGE, and the drift is not symmetric. +# daniel/gitdan's ci-cache-reclaim.sh mirrors this value as +# CI_CACHE_READER_STALE_SECONDS (and CI_CACHE_LEFTOVER_MIN_AGE_SECONDS beside +# it), and its copies must be GREATER THAN OR EQUAL TO this one. Raise this for +# longer jobs while that one stays at 7200 and the arbiter reads a marker whose +# owner still considers it live as stale, then deletes the tree under an +# in-flight clone; its minimum-age guard does not back-stop that, because a +# clone holding a three-hour-old marker has a staging tree roughly three hours +# old too, so both of its guards pass. Raise theirs first. Lowering this one +# needs no coordination at all: the arbiter then defers a reclamation this side +# would already have permitted, which costs disk and not correctness. CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}" +# `.reading--` is a contract name, not a private one: the +# host-level arbiter reads these to tell a live clone from an abandoned one, +# and never deletes one. See the cross-repo contract at the top of this file +# before changing the spelling. reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; } reader_lock_acquire() { @@ -388,6 +471,10 @@ hardlink_clone_into() { parent=$(dirname "$dst") src_name=$(basename "$src") + # `.stage-` is a contract name (see the top of this file): a job killed + # between the copy below and the rename at the end strands this tree, and the + # only thing that ever reclaims one is the host-level arbiter, by this exact + # prefix. tmp="${parent}/.stage-${tag}" attempt=1 diff --git a/scripts/prune-cache.sh b/scripts/prune-cache.sh index 23f07a5..6efa4fc 100755 --- a/scripts/prune-cache.sh +++ b/scripts/prune-cache.sh @@ -85,6 +85,10 @@ ROOT="${1:?usage: prune-cache.sh = this +# one — raising this without raising theirs first lets that script treat a lock +# this side still honours as abandoned. Same direction and same reasoning as +# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh. STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}" # An aside directory is in flight for one rename plus one marker glob — # milliseconds. Anything older belongs to a pass that died between the two, so @@ -174,6 +178,12 @@ is_locked() { # leaves the source inode unchanged, which is precisely why the publish side # can rotate a snapshot out from under a live reader. A declined eviction # therefore costs a deferred eviction and nothing else. +# +# `.evicting--` is a contract name shared with daniel/gitdan's +# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh). +# The sweep at the top of this script reclaims these on every pass, so the only +# one that reaches the arbiter belongs to a repo whose workflow has stopped +# running — which is exactly the case no in-workflow pass can reach. evict_dir() { local dir="$1" name aside name=$(basename "$dir") diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index 4c26834..bb9c647 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -12,9 +12,11 @@ # The swap is two renames, not one, because POSIX rename() can only replace an # EMPTY directory and a snapshot from a prior publish is not one: # -# 1. stage the new snapshot at .stage- (copy time is off every -# consumer's hot path — nothing reads a staging path); -# 2. rename the current snapshot aside to .publish-old-, if present; +# 1. stage the new snapshot at .publish-new-, which hardlink_clone_into +# builds at its own .stage- and renames there (copy time is off +# every consumer's hot path — nothing reads either path); +# 2. rename the current snapshot aside to .publish-old--, if +# present; # 3. rename the staged snapshot into place. # # Step 3 is a single atomic rename onto a path now guaranteed absent, so it @@ -29,8 +31,9 @@ # consumer still WALKING the old generation has its entries unlinked out from # under it, and `cp -al` does not report a subtree that was removed before it # read the parent's listing. That is a silently truncated clone — the failure -# mode this script's own selftest (scenario 8) reproduces against the -# unguarded version. +# mode seed-target-dir-selftest.sh's scenario 8b reproduces from the consumer's +# side, against the unguarded version. (This script's own selftest also has a +# scenario 8; that one is the abandoned-reader-marker case, not this.) # # So the unlink is interlocked with the consume side rather than # unconditional: after the swap, this script waits for every in-flight reader @@ -50,6 +53,19 @@ SRC=$(target_dir_for "$ROOT" "$OWN_KEY") DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") # Keyed by cache key as well as tag, so a deferred generation can be matched # back to the snapshot whose readers must drain before it is safe to reclaim. +# +# `.publish-old-` and `.publish-new-` are dot-prefixed names under a cache +# root, so the cross-repo contract at the top of cache-lib.sh covers them: +# renaming either, or adding a third, requires a matching prefix in +# daniel/gitdan's ci-cache-reclaim.sh. Both were outside its list when that +# contract was written, and `.publish-new-` is why it mattered: it is tagged +# per job per run, so a publisher killed between staging its snapshot and the +# swap below strands a full hardlink clone that no later run of this script +# will ever match. `.publish-old-` is milder — the sweep further down reclaims +# it on the next publish of the same key, if there ever is one; a key whose +# branch is merged, deleted or renamed is never published again, and its +# rotated generation stays until something outside this repo takes it. +# Bringing both under the arbiter's enumeration is tracked as daniel/gitdan#30. OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" SNAP_NAME=$(basename "$DST") GRACE="${CACHE_READ_GRACE_SECONDS}" @@ -85,6 +101,8 @@ start=$(date +%s) # the snapshot it just published — the same aliasing hazard the consume side # closes, pointing the other way. The staging path is not the final name, so # `hardlink_clone_into`'s rename lands on DST only after OLD is out of the way. +# `.publish-new-` is a contract name — see the note at OLD above before +# changing it. TMP_DST="${ROOT}/.publish-new-${TAG}" rm -rf "$TMP_DST" hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {