docs(cache): write down the producer half of the leftover naming contract

daniel/gitdan's host-level arbiter (scripts/ci-cache-reclaim.sh) reclaims the
dot-prefixed trees this repo's scripts strand inside the cache volumes it
scans, and reads this repo's reader markers to decide whether one is still
live. That arrangement was documented only on the consuming side: a
contributor here could add or rename a dot-prefixed shape with no local signal
that anything outside the repo depended on the spelling, and the arbiter
enumerates by explicit prefix — deliberately, so it never sees a .reading-*
marker as a candidate — which makes an unannounced shape invisible to it
rather than conservatively handled.

Adds the producing side's half at the sites someone changing a name will
actually be looking at, pointing at gitdan's LEFTOVER NAMING CONTRACT block as
canonical rather than restating it:

- cache-lib.sh gains a header block naming every shape this repo creates under
  a cache root, its producing function, and how each strands; plus the rule
  that adding a shape obliges the same matching prefix over there as renaming
  one does.
- Site notes at .stage-'s and .reading-'s producing lines, and at
  .evicting-'s in prune-cache.sh.
- publish-snapshot.sh's .publish-old- / .publish-new- pair is documented as
  the shapes that are NOT in the arbiter's list today, with .publish-new-
  called out as the one that strands exactly as .stage- does and that neither
  side reclaims.
- The staleness direction: CACHE_READ_STALE_SECONDS and STALE_LOCK_SECONDS are
  mirrored there and the mirrors must be >= ours, because raising ours alone
  makes the arbiter delete a tree under an in-flight clone (its minimum-age
  guard does not back-stop that case). Lowering ours is safe in any order.
- README gains a short section a newcomer meets before adding a scratch
  directory under a cache root, cross-linked from the cache-layout block.

Comments and docs only; no behaviour change.

Closes #7
This commit is contained in:
2026-08-24 12:50:07 -05:00
parent 1f42064d20
commit 0cf6cc5973
4 changed files with 150 additions and 0 deletions
+78
View File
@@ -8,6 +8,64 @@
# needs as an argument, so the selftests can drive them against scratch
# directories without a CI context.
# ---------------------------------------------------------------------------
# CROSS-REPO CONTRACT: the dot-prefixed names left in a cache root
# ---------------------------------------------------------------------------
#
# The volume these scripts write into is also swept by a host-level arbiter
# that runs outside any job and outside this repo: daniel/gitdan's
# `scripts/ci-cache-reclaim.sh`. It reclaims the dot-prefixed trees a killed
# job strands here, and it reads the reader markers below to decide whether one
# of those trees is still live. Its `LEFTOVER NAMING CONTRACT` block is the
# canonical description of the arrangement; what follows is the producing
# side's half — which names this repo creates, and what changing one obliges.
#
# Created directly under a cache root:
#
# .stage-<tag> cache-lib.sh, hardlink_clone_into(): the tree a
# clone is built in before the atomic rename that
# gives it its real name. <tag> is unique per job
# per run, so a job killed before the rename
# strands a whole hardlink clone under a name no
# later run reuses. Nothing in this repo sweeps it.
# .publish-new-<tag> publish-snapshot.sh: the staged snapshot, between
# its clone and the swap. Strands the same way.
# .publish-old-<key>-<tag> publish-snapshot.sh: the rotated-away generation,
# kept while a reader still holds it and swept by
# the next publish of the same key.
# .evicting-<name>-<pid> prune-cache.sh, evict_dir(): a cache renamed
# aside so the decision to unlink it can be retaken
# after the rename. Swept at the start of every
# prune pass, so it only strands when this repo's
# workflow stops running at all.
# .reading-<source>-<tag> cache-lib.sh, reader_lock_acquire(): NOT garbage.
# It is the live-reader signal the arbiter reads,
# and the one shape it must never delete — removing
# one clears the way to unlink a tree out from
# under an in-flight walk, which is the silent
# truncation this whole interlock exists to
# prevent.
#
# THE RULE, which the arbiter states as its own: no new dot-prefixed entry
# under a cache root without a matching prefix in that script. ADDING a shape
# counts exactly as much as renaming one, because that script enumerates by
# explicit prefix rather than by dotglob — deliberately, since a dotglob would
# pull reader markers into the candidate stream alongside the trees they
# protect. A shape it has not been told about is not handled conservatively, it
# is invisible: an unreclaimed staging tree is a full clone of a multi-GB
# target dir on the one volume whose entire problem is disk.
#
# Of the five above, the arbiter enumerates `.stage-`, `.evicting-` and
# `.reading-`. The two `.publish-*` names predate the contract and are not in
# it, which is why `.publish-new-` — tagged per job per run exactly as
# `.stage-` is — is a shape neither side reclaims today. Closing that is a
# change over there, not here; the rule above is what stops the list growing
# another one.
#
# The two staleness constants the arbiter mirrors are part of the same
# contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS
# below and STALE_LOCK_SECONDS in prune-cache.sh.
# ---------------------------------------------------------------------------
# Cache keys
# ---------------------------------------------------------------------------
@@ -249,8 +307,24 @@ CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}"
# A marker older than this belongs to a job the runner killed before it could
# clean up. Honouring one forever would let a crashed job pin an entire
# snapshot generation on disk permanently.
#
# RAISING THIS IS A CROSS-REPO CHANGE, and the drift is not symmetric.
# daniel/gitdan's ci-cache-reclaim.sh mirrors this value as
# CI_CACHE_READER_STALE_SECONDS (and CI_CACHE_LEFTOVER_MIN_AGE_SECONDS beside
# it), and its copies must be GREATER THAN OR EQUAL TO this one. Raise this for
# longer jobs while that one stays at 7200 and the arbiter reads a marker whose
# owner still considers it live as stale, then deletes the tree under an
# in-flight clone; its minimum-age guard does not back-stop that, because a
# clone holding a three-hour-old marker has a staging tree roughly three hours
# old too, so both of its guards pass. Raise theirs first. Lowering this one
# needs no coordination at all: the arbiter then defers a reclamation this side
# would already have permitted, which costs disk and not correctness.
CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}"
# `.reading-<source>-<tag>` is a contract name, not a private one: the
# host-level arbiter reads these to tell a live clone from an abandoned one,
# and never deletes one. See the cross-repo contract at the top of this file
# before changing the spelling.
reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; }
reader_lock_acquire() {
@@ -388,6 +462,10 @@ hardlink_clone_into() {
parent=$(dirname "$dst")
src_name=$(basename "$src")
# `.stage-<tag>` is a contract name (see the top of this file): a job killed
# between the copy below and the rename at the end strands this tree, and the
# only thing that ever reclaims one is the host-level arbiter, by this exact
# prefix.
tmp="${parent}/.stage-${tag}"
attempt=1
+10
View File
@@ -85,6 +85,10 @@ ROOT="${1:?usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branch
OWN_DIR="${2:?}"
PROTECTED_REFS="${3:-}"
MIN_FREE_PCT="${4:-10}"
# Mirrored by daniel/gitdan's ci-cache-reclaim.sh, whose copy must be >= this
# one — raising this without raising theirs first lets that script treat a lock
# this side still honours as abandoned. Same direction and same reasoning as
# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh.
STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}"
# An aside directory is in flight for one rename plus one marker glob —
# milliseconds. Anything older belongs to a pass that died between the two, so
@@ -174,6 +178,12 @@ is_locked() {
# leaves the source inode unchanged, which is precisely why the publish side
# can rotate a snapshot out from under a live reader. A declined eviction
# therefore costs a deferred eviction and nothing else.
#
# `.evicting-<name>-<pid>` is a contract name shared with daniel/gitdan's
# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh).
# The sweep at the top of this script reclaims these on every pass, so the only
# one that reaches the arbiter belongs to a repo whose workflow has stopped
# running — which is exactly the case no in-workflow pass can reach.
evict_dir() {
local dir="$1" name aside
name=$(basename "$dir")
+10
View File
@@ -51,6 +51,16 @@ SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
# Keyed by cache key as well as tag, so a deferred generation can be matched
# back to the snapshot whose readers must drain before it is safe to reclaim.
#
# `.publish-old-` and `.publish-new-` are dot-prefixed names under a cache
# root, so the cross-repo contract at the top of cache-lib.sh covers them:
# renaming either, or adding a third, requires a matching prefix in
# daniel/gitdan's ci-cache-reclaim.sh. Neither is in its list today, and
# `.publish-new-` is the shape that most needs to be — it is tagged per job per
# run, so a publisher killed between staging its snapshot and the swap below
# strands a full hardlink clone that no later run of this script will ever
# match. `.publish-old-` is milder: the sweep further down reclaims it on the
# next publish of the same key.
OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}"
SNAP_NAME=$(basename "$DST")
GRACE="${CACHE_READ_GRACE_SECONDS}"