fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
@@ -130,14 +130,49 @@ the staging path into `target-<own>`.
|
||||
**A push to a protected branch** has no base to layer over. It builds in its
|
||||
own directory and, if the build goes green, republishes it as
|
||||
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
|
||||
one in. Consumers only ever observe a complete snapshot or none at all.
|
||||
one in, then reclaim the old one once nothing is still reading it. Consumers
|
||||
only ever observe a complete snapshot or none at all.
|
||||
|
||||
**Concurrency.** Two jobs sharing one cache key each stage under their own tag
|
||||
and race on one atomic rename; the loser discards its staging copy. There is
|
||||
no window in which a partially-populated directory is visible under the final
|
||||
name — which means this does not depend on the runner having a single
|
||||
execution slot. Two jobs then building in the same directory is Cargo's own
|
||||
`.cargo-lock` territory, which is what that lock is for.
|
||||
**Concurrency, on the destination.** Two jobs sharing one cache key each stage
|
||||
under their own tag and race on one atomic rename; the loser discards its
|
||||
staging copy. There is no window in which a partially-populated directory is
|
||||
visible under the final name. Two jobs then building in the same directory is
|
||||
Cargo's own `.cargo-lock` territory, which is what that lock is for.
|
||||
|
||||
**Concurrency, on the source.** The atomic rename is necessary and not
|
||||
sufficient, because renaming a *truncated* tree publishes a truncated tree
|
||||
atomically. A clone reads its source over many seconds, and a publisher
|
||||
rotating that source unlinks the generation being read — at which point
|
||||
`cp -al` can silently omit a subtree it never saw, and report success. Two
|
||||
mechanisms, both required:
|
||||
|
||||
- **The publisher does not unlink under a reader.** A consumer publishes a
|
||||
`.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
|
||||
the publisher scans for markers *after* its first rename. A consumer holding
|
||||
the old generation therefore published its marker before that scan and
|
||||
cannot be missed, and one that arrives after the scan necessarily resolves
|
||||
to the new generation. The publisher waits for readers to drain
|
||||
(`read-grace-seconds`, default 300) and, if they do not, **defers** the
|
||||
reclamation rather than forcing it — the old generation stays on disk and is
|
||||
swept by a later publish.
|
||||
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
|
||||
exit status, the source directory's inode before and after (a wholesale
|
||||
replacement mid-walk would otherwise splice two generations), and the entry
|
||||
count (the only signal for a subtree unlinked before its parent was listed —
|
||||
there is no error to read). A tree that fails any of the three is deleted
|
||||
and the clone retried; one that fails the last attempt fails the job. A
|
||||
partial tree never reaches the final name.
|
||||
|
||||
**What this does and does not guarantee.** *Correctness* is closed by
|
||||
construction: no combination of publish and seed timing produces a target
|
||||
directory holding part of one generation, and a source that cannot be read
|
||||
consistently fails the job loudly instead of seeding a truncated cache.
|
||||
*Disk reclamation* is bounded, not immediate: a consumer slower than the grace
|
||||
period leaves one extra snapshot generation of directory entries on the volume
|
||||
until the next publish of that snapshot sweeps it. That residual is capped at
|
||||
one deferred generation per publisher ref, and its real cost is close to the
|
||||
inode count rather than the byte count, since the artifacts are hardlinked to
|
||||
whatever cloned them.
|
||||
|
||||
**Eviction** runs three passes: caches for branches that no longer exist on
|
||||
origin are removed unconditionally; then, only if free space is under the
|
||||
@@ -194,6 +229,8 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
|
||||
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
|
||||
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
|
||||
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
|
||||
|
||||
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
|
||||
@@ -251,14 +288,17 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
||||
| suite | covers |
|
||||
|---|---|
|
||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree |
|
||||
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
|
||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
|
||||
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
||||
|
||||
Every suite runs the actual script, not a reimplementation of its logic, and
|
||||
every fix scenario is paired with a control that reproduces the bug — a
|
||||
scenario that passes either way proves nothing.
|
||||
scenario that passes either way proves nothing. The concurrency scenarios race
|
||||
real processes rather than mocking the interleaving, and gate the interfering
|
||||
step on *observed* progress of the step it interferes with, so the window is
|
||||
hit deterministically instead of on a fast machine's coin flip.
|
||||
|
||||
The action YAML holds no logic beyond wiring; everything testable lives in
|
||||
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
||||
|
||||
Reference in New Issue
Block a user