From 1f42064d20dd3d9e554e6e418fa66b8a1d1da295 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 12:49:53 -0500 Subject: [PATCH 1/4] docs(publish): point the torn-clone note at the suite that reproduces it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit publish-snapshot.sh's header described the silent-truncation failure mode as the one "this script's own selftest (scenario 8)" reproduces. Wrong twice: the scenario that reproduces a truncated clone is in seed-target-dir-selftest.sh, and since #4 split the old scenario 8 into 8a and 8b it is 8b. publish-snapshot-selftest.sh does have a scenario 8 — the abandoned-marker case — so the pointer landed on a real scenario with a plausible number that tests something else. Names the suite as well as the number, and says what the local scenario 8 actually is so the collision cannot re-form. Closes #6 --- scripts/publish-snapshot.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index 4c26834..3f33682 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -29,8 +29,9 @@ # consumer still WALKING the old generation has its entries unlinked out from # under it, and `cp -al` does not report a subtree that was removed before it # read the parent's listing. That is a silently truncated clone — the failure -# mode this script's own selftest (scenario 8) reproduces against the -# unguarded version. +# mode seed-target-dir-selftest.sh's scenario 8b reproduces from the consumer's +# side, against the unguarded version. (This script's own selftest also has a +# scenario 8; that one is the abandoned-reader-marker case, not this.) # # So the unlink is interlocked with the consume side rather than # unconditional: after the swap, this script waits for every in-flight reader From 0cf6cc597338292f94646cc3feb8292c653fd87a Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 12:50:07 -0500 Subject: [PATCH 2/4] docs(cache): write down the producer half of the leftover naming contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit daniel/gitdan's host-level arbiter (scripts/ci-cache-reclaim.sh) reclaims the dot-prefixed trees this repo's scripts strand inside the cache volumes it scans, and reads this repo's reader markers to decide whether one is still live. That arrangement was documented only on the consuming side: a contributor here could add or rename a dot-prefixed shape with no local signal that anything outside the repo depended on the spelling, and the arbiter enumerates by explicit prefix — deliberately, so it never sees a .reading-* marker as a candidate — which makes an unannounced shape invisible to it rather than conservatively handled. Adds the producing side's half at the sites someone changing a name will actually be looking at, pointing at gitdan's LEFTOVER NAMING CONTRACT block as canonical rather than restating it: - cache-lib.sh gains a header block naming every shape this repo creates under a cache root, its producing function, and how each strands; plus the rule that adding a shape obliges the same matching prefix over there as renaming one does. - Site notes at .stage-'s and .reading-'s producing lines, and at .evicting-'s in prune-cache.sh. - publish-snapshot.sh's .publish-old- / .publish-new- pair is documented as the shapes that are NOT in the arbiter's list today, with .publish-new- called out as the one that strands exactly as .stage- does and that neither side reclaims. - The staleness direction: CACHE_READ_STALE_SECONDS and STALE_LOCK_SECONDS are mirrored there and the mirrors must be >= ours, because raising ours alone makes the arbiter delete a tree under an in-flight clone (its minimum-age guard does not back-stop that case). Lowering ours is safe in any order. - README gains a short section a newcomer meets before adding a scratch directory under a cache root, cross-linked from the cache-layout block. Comments and docs only; no behaviour change. Closes #7 --- README.md | 52 +++++++++++++++++++++++++ scripts/cache-lib.sh | 78 +++++++++++++++++++++++++++++++++++++ scripts/prune-cache.sh | 10 +++++ scripts/publish-snapshot.sh | 10 +++++ 4 files changed, 150 insertions(+) diff --git a/README.md b/README.md index 45d826b..f5bdd5f 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,10 @@ env: the only thing a consumer ever clones from. ``` +Transient dot-prefixed entries appear alongside those two — staging trees, +eviction asides, reader markers. Their names are a contract with the host that +owns the volume; see [Scratch names in a cache root](#scratch-names-in-a-cache-root-are-a-cross-repo-contract). + `` is the ref sanitised to a safe path component, capped at 48 characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise @@ -228,6 +232,54 @@ last build, where the historically-correct mtime is exactly the wrong answer. --- +## Scratch names in a cache root are a cross-repo contract + +Read this before adding a dot-prefixed directory under a cache root. + +The volume is not swept by these scripts alone. A host-level arbiter — +daniel/gitdan's `scripts/ci-cache-reclaim.sh`, which runs outside any job — +reclaims the scratch trees a killed job strands here, and reads the reader +markers to decide whether a tree is still live. Its `LEFTOVER NAMING CONTRACT` +block is the canonical description; this side owns the names. + +| name | produced by | if the job dies holding it | +|---|---|---| +| `.stage-` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it | +| `.publish-new-` | `publish-snapshot.sh` | stranded, and not in the arbiter's list | +| `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key | +| `.evicting--` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass | +| `.reading--` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below | + +A `.reading-*` marker is protective, not scratch: it is how both this repo's +prune pass and the arbiter tell an in-flight clone from an abandoned one, and +the arbiter never deletes one. Delete a live marker and the tree it covers +becomes eligible for an unlink underneath the walk that is reading it, which is +the silent truncation the whole interlock exists to prevent. + +**The rule: no new dot-prefixed entry under a cache root without a matching +prefix in `ci-cache-reclaim.sh`.** Adding a shape counts exactly as much as +renaming one. That script enumerates by explicit prefix rather than by dotglob +— deliberately, because a dotglob would pull reader markers into the candidate +stream alongside the trees they protect — so a name it has not been told about +is not handled conservatively, it is invisible, and an unreclaimed staging tree +is a full clone of a multi-GB target dir on the one volume whose entire problem +is disk. `.publish-new-` is the standing example: it strands exactly as +`.stage-` does and is in neither side's sweep. + +**The staleness constants are part of the same contract, and that half has a +direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and +`STALE_LOCK_SECONDS` (`prune-cache.sh`) are mirrored there, and the arbiter's +copies must be **greater than or equal to** these. Raising one here for longer +jobs, without raising its mirror first, makes the arbiter treat a marker whose +owner still considers it live as stale and delete a tree under an in-flight +clone — and its own minimum-age guard does not back-stop that, since a clone +holding a three-hour-old marker has a roughly three-hour-old staging tree. +Lowering either here needs no coordination: the arbiter then only defers a +reclamation this side would already have permitted, which costs disk rather +than correctness. + +--- + ## Inputs ### `cargo-cache` diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 55ceb9d..6927e96 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -8,6 +8,64 @@ # needs as an argument, so the selftests can drive them against scratch # directories without a CI context. +# --------------------------------------------------------------------------- +# CROSS-REPO CONTRACT: the dot-prefixed names left in a cache root +# --------------------------------------------------------------------------- +# +# The volume these scripts write into is also swept by a host-level arbiter +# that runs outside any job and outside this repo: daniel/gitdan's +# `scripts/ci-cache-reclaim.sh`. It reclaims the dot-prefixed trees a killed +# job strands here, and it reads the reader markers below to decide whether one +# of those trees is still live. Its `LEFTOVER NAMING CONTRACT` block is the +# canonical description of the arrangement; what follows is the producing +# side's half — which names this repo creates, and what changing one obliges. +# +# Created directly under a cache root: +# +# .stage- cache-lib.sh, hardlink_clone_into(): the tree a +# clone is built in before the atomic rename that +# gives it its real name. is unique per job +# per run, so a job killed before the rename +# strands a whole hardlink clone under a name no +# later run reuses. Nothing in this repo sweeps it. +# .publish-new- publish-snapshot.sh: the staged snapshot, between +# its clone and the swap. Strands the same way. +# .publish-old-- publish-snapshot.sh: the rotated-away generation, +# kept while a reader still holds it and swept by +# the next publish of the same key. +# .evicting-- prune-cache.sh, evict_dir(): a cache renamed +# aside so the decision to unlink it can be retaken +# after the rename. Swept at the start of every +# prune pass, so it only strands when this repo's +# workflow stops running at all. +# .reading-- cache-lib.sh, reader_lock_acquire(): NOT garbage. +# It is the live-reader signal the arbiter reads, +# and the one shape it must never delete — removing +# one clears the way to unlink a tree out from +# under an in-flight walk, which is the silent +# truncation this whole interlock exists to +# prevent. +# +# THE RULE, which the arbiter states as its own: no new dot-prefixed entry +# under a cache root without a matching prefix in that script. ADDING a shape +# counts exactly as much as renaming one, because that script enumerates by +# explicit prefix rather than by dotglob — deliberately, since a dotglob would +# pull reader markers into the candidate stream alongside the trees they +# protect. A shape it has not been told about is not handled conservatively, it +# is invisible: an unreclaimed staging tree is a full clone of a multi-GB +# target dir on the one volume whose entire problem is disk. +# +# Of the five above, the arbiter enumerates `.stage-`, `.evicting-` and +# `.reading-`. The two `.publish-*` names predate the contract and are not in +# it, which is why `.publish-new-` — tagged per job per run exactly as +# `.stage-` is — is a shape neither side reclaims today. Closing that is a +# change over there, not here; the rule above is what stops the list growing +# another one. +# +# The two staleness constants the arbiter mirrors are part of the same +# contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS +# below and STALE_LOCK_SECONDS in prune-cache.sh. + # --------------------------------------------------------------------------- # Cache keys # --------------------------------------------------------------------------- @@ -249,8 +307,24 @@ CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}" # A marker older than this belongs to a job the runner killed before it could # clean up. Honouring one forever would let a crashed job pin an entire # snapshot generation on disk permanently. +# +# RAISING THIS IS A CROSS-REPO CHANGE, and the drift is not symmetric. +# daniel/gitdan's ci-cache-reclaim.sh mirrors this value as +# CI_CACHE_READER_STALE_SECONDS (and CI_CACHE_LEFTOVER_MIN_AGE_SECONDS beside +# it), and its copies must be GREATER THAN OR EQUAL TO this one. Raise this for +# longer jobs while that one stays at 7200 and the arbiter reads a marker whose +# owner still considers it live as stale, then deletes the tree under an +# in-flight clone; its minimum-age guard does not back-stop that, because a +# clone holding a three-hour-old marker has a staging tree roughly three hours +# old too, so both of its guards pass. Raise theirs first. Lowering this one +# needs no coordination at all: the arbiter then defers a reclamation this side +# would already have permitted, which costs disk and not correctness. CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}" +# `.reading--` is a contract name, not a private one: the +# host-level arbiter reads these to tell a live clone from an abandoned one, +# and never deletes one. See the cross-repo contract at the top of this file +# before changing the spelling. reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; } reader_lock_acquire() { @@ -388,6 +462,10 @@ hardlink_clone_into() { parent=$(dirname "$dst") src_name=$(basename "$src") + # `.stage-` is a contract name (see the top of this file): a job killed + # between the copy below and the rename at the end strands this tree, and the + # only thing that ever reclaims one is the host-level arbiter, by this exact + # prefix. tmp="${parent}/.stage-${tag}" attempt=1 diff --git a/scripts/prune-cache.sh b/scripts/prune-cache.sh index 23f07a5..6efa4fc 100755 --- a/scripts/prune-cache.sh +++ b/scripts/prune-cache.sh @@ -85,6 +85,10 @@ ROOT="${1:?usage: prune-cache.sh = this +# one — raising this without raising theirs first lets that script treat a lock +# this side still honours as abandoned. Same direction and same reasoning as +# CACHE_READ_STALE_SECONDS; the argument is written out in cache-lib.sh. STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}" # An aside directory is in flight for one rename plus one marker glob — # milliseconds. Anything older belongs to a pass that died between the two, so @@ -174,6 +178,12 @@ is_locked() { # leaves the source inode unchanged, which is precisely why the publish side # can rotate a snapshot out from under a live reader. A declined eviction # therefore costs a deferred eviction and nothing else. +# +# `.evicting--` is a contract name shared with daniel/gitdan's +# host-level arbiter (see the cross-repo contract at the top of cache-lib.sh). +# The sweep at the top of this script reclaims these on every pass, so the only +# one that reaches the arbiter belongs to a repo whose workflow has stopped +# running — which is exactly the case no in-workflow pass can reach. evict_dir() { local dir="$1" name aside name=$(basename "$dir") diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index 3f33682..d040864 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -51,6 +51,16 @@ SRC=$(target_dir_for "$ROOT" "$OWN_KEY") DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") # Keyed by cache key as well as tag, so a deferred generation can be matched # back to the snapshot whose readers must drain before it is safe to reclaim. +# +# `.publish-old-` and `.publish-new-` are dot-prefixed names under a cache +# root, so the cross-repo contract at the top of cache-lib.sh covers them: +# renaming either, or adding a third, requires a matching prefix in +# daniel/gitdan's ci-cache-reclaim.sh. Neither is in its list today, and +# `.publish-new-` is the shape that most needs to be — it is tagged per job per +# run, so a publisher killed between staging its snapshot and the swap below +# strands a full hardlink clone that no later run of this script will ever +# match. `.publish-old-` is milder: the sweep further down reclaims it on the +# next publish of the same key. OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" SNAP_NAME=$(basename "$DST") GRACE="${CACHE_READ_GRACE_SECONDS}" From 0118c28f01fafeb234f1b9d129a9820380e8a0f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 12:56:18 -0500 Subject: [PATCH 3/4] docs(cache): state the .publish-* gap as a ticket, not a present state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract block asserted that neither side reclaims .publish-new- 'today'. True when written and about to stop being true: gitdan#30 tracks adding both .publish-* prefixes to the arbiter's enumeration, and a sibling track is landing it this round. A comment that dates itself against a merge in flight is worse than no comment. Rewords all four sites (cache-lib.sh, publish-snapshot.sh, and README's table row and prose) to reference gitdan#30 and keep the mechanism that made the shape worth catching — .publish-new- is tagged per job per run exactly as .stage- is — rather than the arbiter's momentary contents. The rule itself is unchanged; it is the durable part, and it is what found this. Adds the counting check while there: the five names here and the prefixes ci-cache-reclaim.sh enumerates are meant to be the same length, so a mismatch is the cheapest signal that one side gained a shape without telling the other. --- README.md | 13 ++++++++++--- scripts/cache-lib.sh | 17 +++++++++++------ scripts/publish-snapshot.sh | 13 +++++++------ 3 files changed, 28 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index f5bdd5f..76c725e 100644 --- a/README.md +++ b/README.md @@ -245,7 +245,7 @@ block is the canonical description; this side owns the names. | name | produced by | if the job dies holding it | |---|---|---| | `.stage-` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it | -| `.publish-new-` | `publish-snapshot.sh` | stranded, and not in the arbiter's list | +| `.publish-new-` | `publish-snapshot.sh` | stranded; see `daniel/gitdan#30` | | `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key | | `.evicting--` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass | | `.reading--` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below | @@ -263,8 +263,15 @@ renaming one. That script enumerates by explicit prefix rather than by dotglob stream alongside the trees they protect — so a name it has not been told about is not handled conservatively, it is invisible, and an unreclaimed staging tree is a full clone of a multi-GB target dir on the one volume whose entire problem -is disk. `.publish-new-` is the standing example: it strands exactly as -`.stage-` does and is in neither side's sweep. +is disk. `.publish-new-` is the worked example: the two `.publish-*` names were +outside the contract when it was written, and `.publish-new-` strands exactly +as `.stage-` does, so it is the one that needed catching. Bringing both under +the arbiter's enumeration is tracked as `daniel/gitdan#30`. + +The two lists are meant to be the same length — the five names above, and the +prefixes `ci-cache-reclaim.sh` enumerates. A mismatch means one side gained a +shape without telling the other, which is the drift the rule exists to catch +and the cheapest thing to check. **The staleness constants are part of the same contract, and that half has a direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 6927e96..bf50169 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -55,12 +55,17 @@ # is invisible: an unreclaimed staging tree is a full clone of a multi-GB # target dir on the one volume whose entire problem is disk. # -# Of the five above, the arbiter enumerates `.stage-`, `.evicting-` and -# `.reading-`. The two `.publish-*` names predate the contract and are not in -# it, which is why `.publish-new-` — tagged per job per run exactly as -# `.stage-` is — is a shape neither side reclaims today. Closing that is a -# change over there, not here; the rule above is what stops the list growing -# another one. +# The two `.publish-*` names predate the contract and were outside it when this +# block was written — the rule catching an uncovered shape on its first +# application. `.publish-new-` is the one that mattered: it is tagged per job +# per run exactly as `.stage-` is, so a publisher killed before the swap +# strands a tree under a name no later run of that script matches, which is +# precisely the shape only the arbiter can reach. Bringing both in is tracked +# as daniel/gitdan#30. +# +# The two lists are meant to be the same length. A shape here without a prefix +# there is one side having changed without telling the other, and it is +# cheapest to notice by counting. # # The two staleness constants the arbiter mirrors are part of the same # contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index d040864..0d8d2d5 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -55,12 +55,13 @@ DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") # `.publish-old-` and `.publish-new-` are dot-prefixed names under a cache # root, so the cross-repo contract at the top of cache-lib.sh covers them: # renaming either, or adding a third, requires a matching prefix in -# daniel/gitdan's ci-cache-reclaim.sh. Neither is in its list today, and -# `.publish-new-` is the shape that most needs to be — it is tagged per job per -# run, so a publisher killed between staging its snapshot and the swap below -# strands a full hardlink clone that no later run of this script will ever -# match. `.publish-old-` is milder: the sweep further down reclaims it on the -# next publish of the same key. +# daniel/gitdan's ci-cache-reclaim.sh. Both were outside its list when that +# contract was written, and `.publish-new-` is why it mattered: it is tagged +# per job per run, so a publisher killed between staging its snapshot and the +# swap below strands a full hardlink clone that no later run of this script +# will ever match. `.publish-old-` is milder — the sweep further down reclaims +# it on the next publish of the same key. Bringing both under the arbiter's +# enumeration is tracked as daniel/gitdan#30. OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" SNAP_NAME=$(basename "$DST") GRACE="${CACHE_READ_GRACE_SECONDS}" From e3869c59201b5586637226c093a70437edbc1338 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 13:06:49 -0500 Subject: [PATCH 4/4] docs(publish): address review nits on the contract block MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four corrections from the review of #8, all in the files this PR already touches: - A local signal on the line that creates .publish-new-. The other four shapes each got a note at their producing line, which is the whole premise of #7 — someone renaming TMP_DST reads its own comment block and would never see the contract note 45 lines up at OLD. - '.publish-old- is milder' understated it. Milder is true; bounded is not. A key whose branch is merged, deleted or renamed is never published again, so its rotated generation stays until something outside this repo takes it — which is the case gitdan#30 itself makes, two lines away. - The drift check now says the prefix constants ci-cache-reclaim.sh DECLARES, not the ones it enumerates. Those are different numbers: collect_entries() globs .stage- and .evicting- only, because .reading- is read and never swept. Only the declared reading makes the five-against-five count work, and a countability check that needs a coin flip to count is not one. - publish-snapshot.sh's own header had two stale names ten lines above the stale pointer this PR fixes: step 1 staged at .stage- (that is hardlink_clone_into's inner path; the staged snapshot is .publish-new-) and step 2 named .publish-old- without the key. Pre-existing and outside both ACs, but #6's thesis is that a plausible-looking wrong name is the worst kind, and these are in the file the PR is about. Comments and docs only. With comments and blank lines stripped, all three scripts hash identically to origin/main. --- README.md | 10 ++++++---- scripts/cache-lib.sh | 12 ++++++++---- scripts/publish-snapshot.sh | 16 +++++++++++----- 3 files changed, 25 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 76c725e..9b27b21 100644 --- a/README.md +++ b/README.md @@ -246,7 +246,7 @@ block is the canonical description; this side owns the names. |---|---|---| | `.stage-` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it | | `.publish-new-` | `publish-snapshot.sh` | stranded; see `daniel/gitdan#30` | -| `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key | +| `.publish-old--` | `publish-snapshot.sh` | swept by the next publish of that key, if there is one | | `.evicting--` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass | | `.reading--` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below | @@ -269,9 +269,11 @@ as `.stage-` does, so it is the one that needed catching. Bringing both under the arbiter's enumeration is tracked as `daniel/gitdan#30`. The two lists are meant to be the same length — the five names above, and the -prefixes `ci-cache-reclaim.sh` enumerates. A mismatch means one side gained a -shape without telling the other, which is the drift the rule exists to catch -and the cheapest thing to check. +prefix constants `ci-cache-reclaim.sh` *declares*. Not the subset it enumerates +as reclaim candidates: that one is smaller, because `.reading-` is read and +never swept. A mismatch means one side gained a shape without telling the +other, which is the drift the rule exists to catch and the cheapest thing to +check. **The staleness constants are part of the same contract, and that half has a direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index bf50169..2f94aea 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -32,7 +32,9 @@ # its clone and the swap. Strands the same way. # .publish-old-- publish-snapshot.sh: the rotated-away generation, # kept while a reader still holds it and swept by -# the next publish of the same key. +# the next publish of the same key — if there ever +# is one. A merged or renamed branch never +# publishes again, and its last one stays. # .evicting-- prune-cache.sh, evict_dir(): a cache renamed # aside so the decision to unlink it can be retaken # after the rename. Swept at the start of every @@ -63,9 +65,11 @@ # precisely the shape only the arbiter can reach. Bringing both in is tracked # as daniel/gitdan#30. # -# The two lists are meant to be the same length. A shape here without a prefix -# there is one side having changed without telling the other, and it is -# cheapest to notice by counting. +# The two lists are meant to be the same length: the five names above, and the +# prefix constants that script DECLARES — not the subset it enumerates as +# reclaim candidates, which is smaller because `.reading-` is read and never +# swept. A shape here without a constant there is one side having changed +# without telling the other, and it is cheapest to notice by counting. # # The two staleness constants the arbiter mirrors are part of the same # contract, and that half has a direction to it — see CACHE_READ_STALE_SECONDS diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index 0d8d2d5..bb9c647 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -12,9 +12,11 @@ # The swap is two renames, not one, because POSIX rename() can only replace an # EMPTY directory and a snapshot from a prior publish is not one: # -# 1. stage the new snapshot at .stage- (copy time is off every -# consumer's hot path — nothing reads a staging path); -# 2. rename the current snapshot aside to .publish-old-, if present; +# 1. stage the new snapshot at .publish-new-, which hardlink_clone_into +# builds at its own .stage- and renames there (copy time is off +# every consumer's hot path — nothing reads either path); +# 2. rename the current snapshot aside to .publish-old--, if +# present; # 3. rename the staged snapshot into place. # # Step 3 is a single atomic rename onto a path now guaranteed absent, so it @@ -60,8 +62,10 @@ DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") # per job per run, so a publisher killed between staging its snapshot and the # swap below strands a full hardlink clone that no later run of this script # will ever match. `.publish-old-` is milder — the sweep further down reclaims -# it on the next publish of the same key. Bringing both under the arbiter's -# enumeration is tracked as daniel/gitdan#30. +# it on the next publish of the same key, if there ever is one; a key whose +# branch is merged, deleted or renamed is never published again, and its +# rotated generation stays until something outside this repo takes it. +# Bringing both under the arbiter's enumeration is tracked as daniel/gitdan#30. OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" SNAP_NAME=$(basename "$DST") GRACE="${CACHE_READ_GRACE_SECONDS}" @@ -97,6 +101,8 @@ start=$(date +%s) # the snapshot it just published — the same aliasing hazard the consume side # closes, pointing the other way. The staging path is not the final name, so # `hardlink_clone_into`'s rename lands on DST only after OLD is out of the way. +# `.publish-new-` is a contract name — see the note at OLD above before +# changing it. TMP_DST="${ROOT}/.publish-new-${TAG}" rm -rf "$TMP_DST" hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {