5 Commits

Include renames

Author	SHA1	Message	Date
claude	fb7a788c90	feat(cache): give same-ref jobs separate build directories via cache-lineage ... A cache key names a REF. What a target directory holds is the product of a ref and a build configuration, and emowheel builds the same ref twice on every push — once for the host, once for wasm32, in two jobs that start together. Keyed on the ref alone, `cargo-cache@v1` handed both the same CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second job sat on "Blocking waiting for file lock on build directory" for the length of the first while holding a runner capacity slot, so a third repo's queued job waited behind a job doing nothing. `cache-lineage` is that second dimension. It names ONE directory level under the cache root: <cache-root>/target-<key> no lineage (unchanged) <cache-root>/<lineage>/target-<key> a lineage Nesting, not a suffix on the key, and that is the whole design decision. `prune-cache.sh`'s liveness pass classifies a directory by recomputing `target-<cache_key(branch)>` for every branch on origin and evicting whatever does not match — a `target-<key>-wasm32` matches nothing, so it would be classified dead and evicted unconditionally on every run. daniel/gitdan's host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls out of that too, so those caches would never be reclaim candidates and a whole lineage would go missing from the shared disk budget. Nesting leaves both matchers reading exactly the names they already read, one level down — which is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own suite pins the depth-2 case). Every interacting part, checked rather than assumed: - SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in a lineage layers over THAT lineage's base snapshot. Asserted. - PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root. The publish action now takes the root from the `CARGO_CACHE_ROOT` the consume step exported, and CHECKS its own inputs against it — a publish step left at the default while its consume step nested would otherwise republish a different lineage's live target dir over that lineage's snapshot, on every push, silently. `mode: release-lock` is exempt: it releases a lock on `$CARGO_TARGET_DIR` and never touches a root. - WATERMARK: per target dir, so it follows the lineage. Unchanged. - PRUNE and LIVENESS: scoped to the root they are given, so a pass in one lineage neither evicts nor sees a sibling's caches, or the flat layout's. Liveness keeps resolving real branch names, which is what a key suffix would have broken. - ci_cache_reclaim: verified by dry-run against a fixture in this layout — all six nested and flat dirs collected as candidates, protection resolved correctly on the nested ones, and a `.stage-` stranded inside the lineage found by the leftover sweep. Refused lineage names are refused at resolve time, each rejection naming the reader that imposes it: a path separator (the arbiter's depth budget), a Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix (the leftover-naming contract). None of these fails visibly on its own — each produces a working directory that some pass silently stops seeing. Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna and emowheel's `ci` job keep the exact directories they have on the volume. New suite `cache-root-selftest.sh` (19 assertions), red-proven against three deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled validator, and a `verify` that never rejects a mismatch.	2026-08-26 12:32:21 -05:00
claude	df6f1b91fb	docs(cache): producer half of the depended-upon names contract (#11) ... Co-authored-by: Claude <claude@gitdan.com> Co-committed-by: Claude <claude@gitdan.com>	2026-08-24 23:03:29 +00:00
claudeandClaude Opus 5	f57e2a6013	fix(cargo-cache): close the seed-vs-republish race the design claimed to close ... The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L	2026-08-23 16:40:21 -05:00
claudeandClaude Opus 5	248af3061e	feat(cargo-cache): hardlink-clone a per-ref Cargo cache from a published snapshot ... Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L	2026-08-23 14:18:01 -05:00
claude	8503883138	feat: cargo-cache composite action skeleton (phase-0 resolution probe)	2026-08-23 11:52:31 -05:00