fix(cargo-cache): close the seed-vs-republish race the design claimed to close

The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
2026-08-23 16:40:21 -05:00
co-authored by Claude Opus 5
parent 248af3061e
commit f57e2a6013
9 changed files with 562 additions and 68 deletions
+49 -9
View File
@@ -130,14 +130,49 @@ the staging path into `target-<own>`.
**A push to a protected branch** has no base to layer over. It builds in its
own directory and, if the build goes green, republishes it as
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
one in. Consumers only ever observe a complete snapshot or none at all.
one in, then reclaim the old one once nothing is still reading it. Consumers
only ever observe a complete snapshot or none at all.
**Concurrency.** Two jobs sharing one cache key each stage under their own tag
and race on one atomic rename; the loser discards its staging copy. There is
no window in which a partially-populated directory is visible under the final
name — which means this does not depend on the runner having a single
execution slot. Two jobs then building in the same directory is Cargo's own
`.cargo-lock` territory, which is what that lock is for.
**Concurrency, on the destination.** Two jobs sharing one cache key each stage
under their own tag and race on one atomic rename; the loser discards its
staging copy. There is no window in which a partially-populated directory is
visible under the final name. Two jobs then building in the same directory is
Cargo's own `.cargo-lock` territory, which is what that lock is for.
**Concurrency, on the source.** The atomic rename is necessary and not
sufficient, because renaming a *truncated* tree publishes a truncated tree
atomically. A clone reads its source over many seconds, and a publisher
rotating that source unlinks the generation being read — at which point
`cp -al` can silently omit a subtree it never saw, and report success. Two
mechanisms, both required:
- **The publisher does not unlink under a reader.** A consumer publishes a
`.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
the publisher scans for markers *after* its first rename. A consumer holding
the old generation therefore published its marker before that scan and
cannot be missed, and one that arrives after the scan necessarily resolves
to the new generation. The publisher waits for readers to drain
(`read-grace-seconds`, default 300) and, if they do not, **defers** the
reclamation rather than forcing it — the old generation stays on disk and is
swept by a later publish.
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
exit status, the source directory's inode before and after (a wholesale
replacement mid-walk would otherwise splice two generations), and the entry
count (the only signal for a subtree unlinked before its parent was listed —
there is no error to read). A tree that fails any of the three is deleted
and the clone retried; one that fails the last attempt fails the job. A
partial tree never reaches the final name.
**What this does and does not guarantee.** *Correctness* is closed by
construction: no combination of publish and seed timing produces a target
directory holding part of one generation, and a source that cannot be read
consistently fails the job loudly instead of seeding a truncated cache.
*Disk reclamation* is bounded, not immediate: a consumer slower than the grace
period leaves one extra snapshot generation of directory entries on the volume
until the next publish of that snapshot sweeps it. That residual is capped at
one deferred generation per publisher ref, and its real cost is close to the
inode count rather than the byte count, since the artifacts are hardlinked to
whatever cloned them.
**Eviction** runs three passes: caches for branches that no longer exist on
origin are removed unconditionally; then, only if free space is under the
@@ -194,6 +229,8 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
@@ -251,14 +288,17 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
| suite | covers |
|---|---|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree |
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
Every suite runs the actual script, not a reimplementation of its logic, and
every fix scenario is paired with a control that reproduces the bug — a
scenario that passes either way proves nothing.
scenario that passes either way proves nothing. The concurrency scenarios race
real processes rather than mocking the interleaving, and gate the interfering
step on *observed* progress of the step it interferes with, so the window is
hit deterministically instead of on a fast machine's coin flip.
The action YAML holds no logic beyond wiring; everything testable lives in
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and