fix(cargo-cache): close the seed-vs-republish race the design claimed to close

The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
2026-08-23 16:40:21 -05:00
co-authored by Claude Opus 5
parent 248af3061e
commit f57e2a6013
9 changed files with 562 additions and 68 deletions
+24 -5
View File
@@ -1,8 +1,8 @@
name: 'Cargo cache (publish)'
description: >-
Records this run''s build watermark and, on a publisher branch, atomically
Records this run's build watermark and, on a publisher branch, atomically
republishes its target directory as the immutable snapshot that other
branches'' caches are hardlink-cloned from.
branches' caches are hardlink-cloned from.
author: 'gitdan'
inputs:
@@ -19,9 +19,9 @@ inputs:
mode:
description: >-
publish — record the watermark, publish a snapshot if eligible,
release this job''s cache lock (the normal call, after a
release this job's cache lock (the normal call, after a
green build).
release-lock — release this job''s cache lock and do nothing else. Use
release-lock — release this job's cache lock and do nothing else. Use
in a final `if: always()` step so a failed run does not
leave a lock behind for the staleness grace period.
required: false
@@ -37,9 +37,25 @@ inputs:
because its ref is not the reference branch even when it targets one.
required: false
default: 'push'
read-grace-seconds:
description: >-
How long the snapshot swap waits for in-flight consumers to finish
cloning the generation it is replacing before reclaiming it. On timeout
the old generation is LEFT ON DISK and swept by a later publish — the
unlink is never forced, because unlinking a tree a consumer is walking
is what silently truncates that consumer's clone.
required: false
default: '300'
reader-stale-seconds:
description: >-
Age past which a consumer's read marker is treated as abandoned by a
job the runner killed. Without it one crashed job would pin a snapshot
generation on disk permanently.
required: false
default: '7200'
record-watermark:
description: >-
Record this run''s HEAD as the build watermark for this target dir.
Record this run's HEAD as the build watermark for this target dir.
True for PR runs too, not just publishers: a feature branch accumulates
its own build history across several pushes and needs its own watermark.
required: false
@@ -109,6 +125,9 @@ runs:
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
shell: bash
env:
CACHE_READ_GRACE_SECONDS: ${{ inputs.read-grace-seconds }}
CACHE_READ_STALE_SECONDS: ${{ inputs.reader-stale-seconds }}
run: |
set -euo pipefail
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \