0553a6b956494ec667ee7d848da2a4e9ff662360
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude
|
0553a6b956
|
fix(hardlink): resolve an ambiguous out/ toward unsharing, and pin the linked-output scenario on a shape that exhibits it
CI / shellcheck + selftests (pull_request) Skipped
Two review findings on #16. The `out/` discriminator keyed on Cargo's record of a build-script execution, which Cargo writes only AFTER the script exits successfully. A build script that populates OUT_DIR and then fails leaves a unit with no record at all, so its OUT_DIR read as a compile unit's artifact directory and stayed shared — a regression against the old `-name build` selection, which real-copied that state by construction. Reproduced on cargo 1.93.1 stable. An `out` directory now stays shared only when two independent signals agree: it holds an `.rlib`/`.rmeta` of its own, and its unit carries no execution record. Either one missing real-copies it. The cost is unchanged to the byte — the newly-unshared directories hold only executables and `*.d`, both already privately owned by the file rules. The live linked-test-binary scenario was built on the lib+bin probe crate, whose test binaries relink to a fresh inode — a shape gitdan-actions#17 records as measured safe. Both halves passed green against the unfixed selection on dep-info mutations the previous scenario already covers. It now builds a bin-only crate with a unit test, reads only executables, and skips loudly with a warning rather than passing quietly if the toolchain does not exhibit the rewrite at all. Also: drop a clause asserting the linker writes in place "whenever the path has no other hard link", which this change's own evidence denies; move the load-bearing comment block back above `unshare_mutable_paths`; correct a superseded 99.998% figure; and record both cost rows in the README rather than only the flattering whole-tree one. |
||
|
claude
|
e5b26a9368
|
fix(hardlink): name the mutable set directly, under both build-dir layouts
CI / shellcheck + selftests (pull_request) Skipped
`unshare_mutable_paths` selected `.fingerprint` and `build` directories. Under Cargo's build-dir layout v2 the first clause matches nothing and the second matches the whole tree, because v2 regroups artifacts under `build/` alongside the metadata. Measured on one scratch crate: 39.3% of the tree real-copied under v1, 99.996% under v2. The selection now names the mutable set rather than the container it used to live in: fingerprint directories under either spelling, layout v2's `run/` directories, layout v1's loose build-script run metadata, and the `out` directories that are a build script's OUT_DIR rather than a compile unit's artifact directory. The two are told apart structurally, by Cargo's record of the build-script execution sitting beside the OUT_DIR and nowhere else. Verifying that turned up a second, layout-independent hazard: a linked executable is written through whatever inode is already at its path, so a `cargo test --no-run` inside a `cp -al` clone rewrites the source's own test binary. Reproduced on cargo 1.93.1 stable, 1.96.0-nightly, 1.98.0-nightly and 1.100.0-nightly, under both layouts. Every executable is now real-copied; `.rlib`, `.rmeta` and `incremental/` are what stay shared. hardlink-clone-selftest.sh gains two file-only layout fixtures that pin the partition in both directions without a compiler, and a live scenario that relinks a test binary. |
||
|
claude
|
554310186f
|
fix(hardlink): content freshness moved switches, it was not withdrawn
CI / shellcheck + selftests (pull_request) Successful in 1m17s
`unshare_mutable_paths`' comment recorded that upstream had stopped rewriting `dep-<target>` in place, on a measurement taken against 1.100.0-nightly. It had not. Two unrelated cargo changes landed within four days of each other and between them moved the switch that turns the behaviour on and the path it writes to: - cargo PR #17382 (2026-08-22) demoted `-Z checksum-freshness` to a gate and gave `build.fingerprint` the choice, defaulting to `mtime`. Setting only the gate is accepted and does nothing, which is exactly the result that was read as a withdrawal. - build-dir layout v2 (cargo PR #17354, stable 1.100.0 on 2026-11-12, nightly default since 1.99) moved the file from `.fingerprint/<unit>/dep-*` to `build/<pkg>/<hash>/fingerprint/dep-*`. Measured 2026-08-26 on 1.100.0-nightly (e8cb624d5): same toolchain, same clone procedure, one env var apart — with the gate alone a `cp -al` clone mutates only the build/ and *.d families; add `CARGO_BUILD_FINGERPRINT=content` and the source's dep-info file is mutated through the shared inode again. The hazard is intact. So the suite now exports both switches, and its strongest scenario runs again on current nightlies — verified passing against both layouts. Its control note learned the v2 path too: it looked for the v1 path only, and so printed "does NOT rewrite ... in place" three lines beneath a listing that showed the rewrite. Every claim these comments make is now dated and cited, because the defect being fixed is a comment that cited one measurement and silently stopped reproducing. Layout v2 also drags the artifacts under `build/`, which collapses this function's real-copy set from 21.3% of a target dir to 99.998%. That is a live cost, not a correctness problem, and it is filed as gitdan-actions#14 rather than fixed here. Part of daniel/gitdan#62. |
||
|
claude
|
5e8e773f78
|
test(hardlink): report the mutation families, don't assert one of them
CI / shellcheck + selftests (pull_request) Failing after 1m21s
CI's nightly is 1.100.0-nightly (2026-08-25); the machine this suite was written on had 1.96.0-nightly (2026-02-24). On the newer one the control's `cp -al` clone mutates only the build/ and *.d families — upstream appears to have stopped rewriting `.fingerprint/*/dep-*` in place under checksum freshness — so the suite went red on the *absence* of a hazard. That is the wrong shape for a gate. The control's job is to prove the hazard exists at all, which a non-empty mutated set already does; naming one family as mandatory makes the suite red whenever upstream stops doing something we never wanted it to do, and red in the CONTROL, where a failure reads as "the hazard is gone" rather than "upstream changed". It is now a note either way. Nothing is given up. The fix scenario asserts the source is byte-identical after a full rebuild in the clone, which covers every family the running Cargo has, named or not — and the checksum-freshness scenario after it tests the stale-reuse hazard directly. The dep-* line only ever documented which family was in play. `unshare_mutable_paths` keeps unsharing dep-* regardless, and its measurement block now records both observations with their versions: 22 MB of a 6.9 GB tree against a failure mode that is a wrong answer rather than a slow one. |
||
|
claude
|
fb7a788c90
|
feat(cache): give same-ref jobs separate build directories via cache-lineage
CI / shellcheck + selftests (pull_request) Failing after 1m19s
A cache key names a REF. What a target directory holds is the product of a ref
and a build configuration, and emowheel builds the same ref twice on every
push — once for the host, once for wasm32, in two jobs that start together.
Keyed on the ref alone, `cargo-cache@v1` handed both the same
CARGO_TARGET_DIR, and Cargo's target-directory lock is exclusive: the second
job sat on "Blocking waiting for file lock on build directory" for the length
of the first while holding a runner capacity slot, so a third repo's queued
job waited behind a job doing nothing.
`cache-lineage` is that second dimension. It names ONE directory level under
the cache root:
<cache-root>/target-<key> no lineage (unchanged)
<cache-root>/<lineage>/target-<key> a lineage
Nesting, not a suffix on the key, and that is the whole design decision.
`prune-cache.sh`'s liveness pass classifies a directory by recomputing
`target-<cache_key(branch)>` for every branch on origin and evicting whatever
does not match — a `target-<key>-wasm32` matches nothing, so it would be
classified dead and evicted unconditionally on every run. daniel/gitdan's
host-level arbiter reads the same shape (BRANCH_DIR_RE); a suffixed name falls
out of that too, so those caches would never be reclaim candidates and a whole
lineage would go missing from the shared disk budget. Nesting leaves both
matchers reading exactly the names they already read, one level down — which
is a layout that arbiter already walks (CI_CACHE_MAX_DEPTH is 2, and its own
suite pins the depth-2 case).
Every interacting part, checked rather than assumed:
- SEED: `seed-target-dir.sh` takes the root as an argument, so a PR branch in
a lineage layers over THAT lineage's base snapshot. Asserted.
- PUBLISH: `publish-snapshot.sh` derives both ends of the swap from the root.
The publish action now takes the root from the `CARGO_CACHE_ROOT` the
consume step exported, and CHECKS its own inputs against it — a publish step
left at the default while its consume step nested would otherwise republish
a different lineage's live target dir over that lineage's snapshot, on every
push, silently. `mode: release-lock` is exempt: it releases a lock on
`$CARGO_TARGET_DIR` and never touches a root.
- WATERMARK: per target dir, so it follows the lineage. Unchanged.
- PRUNE and LIVENESS: scoped to the root they are given, so a pass in one
lineage neither evicts nor sees a sibling's caches, or the flat layout's.
Liveness keeps resolving real branch names, which is what a key suffix would
have broken.
- ci_cache_reclaim: verified by dry-run against a fixture in this layout —
all six nested and flat dirs collected as candidates, protection resolved
correctly on the nested ones, and a `.stage-` stranded inside the lineage
found by the leftover sweep.
Refused lineage names are refused at resolve time, each rejection naming the
reader that imposes it: a path separator (the arbiter's depth budget), a
Cargo profile name (its no-descend list), a `target-`/`snapshot-` prefix (this
repo's own prune globs), a hex suffix (its per-branch-dir shape), a dot prefix
(the leftover-naming contract). None of these fails visibly on its own — each
produces a working directory that some pass silently stops seeing.
Setting no lineage resolves to the cache root byte for byte, so lublub, zemyna
and emowheel's `ci` job keep the exact directories they have on the volume.
New suite `cache-root-selftest.sh` (19 assertions), red-proven against three
deliberate breakages: a `cache_root_for` that ignores the lineage, a disabled
validator, and a `verify` that never rejects a mismatch.
|
||
|
claude
|
f76789358d
|
ci(gitea): gate scripts/ with shellcheck and the selftests
This repo had eight scripts, six selftest suites and nothing that ran any of
them. It is a composite action consumed at `@v1` — a moving tag — by three
repos' CI, so a bad edit here reaches zemyna, emowheel and lublub at once and
is discovered by whichever of them builds next.
One job: `shellcheck -x --source-path=scripts scripts/*.sh`, then
`bash scripts/selftest.sh`. `-x` follows the `. cache-lib.sh` every script
sources, which is where most of the logic being checked lives; without it
shellcheck reports SC1091 and analyses each file with a hole in it.
The full suite, not `--fast`: `hardlink-clone-selftest.sh` and
`restore-mtimes-selftest.sh` are the two that drive a real Cargo rather than a
fixture, and selftest.sh's own header says `--fast` is for iterating, not for
signing off a change. So the job installs a stable toolchain. The scratch
workspaces they build use path dependencies only, so nothing reaches
crates.io, and the job references no credentials at all.
Modelled on daniel/gitdan's workflow, including the two clauses of the
draft-skip guard (`github.event_name != 'pull_request' ||
!github.event.pull_request.draft` — the first is what stops the second from
also skipping pushes to `main`, where there is no `pull_request` context) and
the per-commit concurrency group for `push`.
No `container.volumes:` entry, deliberately: the suites want a cold target dir
every run, since "did this rebuild?" is exactly what restore-mtimes-selftest
asks. This job takes no share of the shared CI cache disk budget.
Turning the gate on surfaced three pre-existing findings, all fixed here
rather than suppressed or waived:
- `restore-mtimes-selftest.sh` SC2038: `find | xargs touch` -> `-print0 | xargs -0`.
- `hardlink-clone-selftest.sh` SC2295: `${f#$base_fix/}` -> `${f#"$base_fix"/}`.
- `cache-lib.sh` SC2016: a per-line disable with the rationale. The single
quotes are load-bearing — the program is for the inner shell, where `$f` is
its loop variable, `$rc` its accumulator and `$$` its pid — so this is the
documented-false-positive case, not a stopgap.
|
||
|
claude
|
e3869c5920
|
docs(publish): address review nits on the contract block
Four corrections from the review of #8, all in the files this PR already touches: - A local signal on the line that creates .publish-new-. The other four shapes each got a note at their producing line, which is the whole premise of #7 — someone renaming TMP_DST reads its own comment block and would never see the contract note 45 lines up at OLD. - '.publish-old- is milder' understated it. Milder is true; bounded is not. A key whose branch is merged, deleted or renamed is never published again, so its rotated generation stays until something outside this repo takes it — which is the case gitdan#30 itself makes, two lines away. - The drift check now says the prefix constants ci-cache-reclaim.sh DECLARES, not the ones it enumerates. Those are different numbers: collect_entries() globs .stage- and .evicting- only, because .reading- is read and never swept. Only the declared reading makes the five-against-five count work, and a countability check that needs a coin flip to count is not one. - publish-snapshot.sh's own header had two stale names ten lines above the stale pointer this PR fixes: step 1 staged at .stage-<tag> (that is hardlink_clone_into's inner path; the staged snapshot is .publish-new-<tag>) and step 2 named .publish-old-<tag> without the key. Pre-existing and outside both ACs, but #6's thesis is that a plausible-looking wrong name is the worst kind, and these are in the file the PR is about. Comments and docs only. With comments and blank lines stripped, all three scripts hash identically to origin/main. |
||
|
claude
|
0118c28f01
|
docs(cache): state the .publish-* gap as a ticket, not a present state
The contract block asserted that neither side reclaims .publish-new- 'today'. True when written and about to stop being true: gitdan#30 tracks adding both .publish-* prefixes to the arbiter's enumeration, and a sibling track is landing it this round. A comment that dates itself against a merge in flight is worse than no comment. Rewords all four sites (cache-lib.sh, publish-snapshot.sh, and README's table row and prose) to reference gitdan#30 and keep the mechanism that made the shape worth catching — .publish-new- is tagged per job per run exactly as .stage- is — rather than the arbiter's momentary contents. The rule itself is unchanged; it is the durable part, and it is what found this. Adds the counting check while there: the five names here and the prefixes ci-cache-reclaim.sh enumerates are meant to be the same length, so a mismatch is the cheapest signal that one side gained a shape without telling the other. |
||
|
claude
|
0cf6cc5973
|
docs(cache): write down the producer half of the leftover naming contract
daniel/gitdan's host-level arbiter (scripts/ci-cache-reclaim.sh) reclaims the dot-prefixed trees this repo's scripts strand inside the cache volumes it scans, and reads this repo's reader markers to decide whether one is still live. That arrangement was documented only on the consuming side: a contributor here could add or rename a dot-prefixed shape with no local signal that anything outside the repo depended on the spelling, and the arbiter enumerates by explicit prefix — deliberately, so it never sees a .reading-* marker as a candidate — which makes an unannounced shape invisible to it rather than conservatively handled. Adds the producing side's half at the sites someone changing a name will actually be looking at, pointing at gitdan's LEFTOVER NAMING CONTRACT block as canonical rather than restating it: - cache-lib.sh gains a header block naming every shape this repo creates under a cache root, its producing function, and how each strands; plus the rule that adding a shape obliges the same matching prefix over there as renaming one does. - Site notes at .stage-'s and .reading-'s producing lines, and at .evicting-'s in prune-cache.sh. - publish-snapshot.sh's .publish-old- / .publish-new- pair is documented as the shapes that are NOT in the arbiter's list today, with .publish-new- called out as the one that strands exactly as .stage- does and that neither side reclaims. - The staleness direction: CACHE_READ_STALE_SECONDS and STALE_LOCK_SECONDS are mirrored there and the mirrors must be >= ours, because raising ours alone makes the arbiter delete a tree under an in-flight clone (its minimum-age guard does not back-stop that case). Lowering ours is safe in any order. - README gains a short section a newcomer meets before adding a scratch directory under a cache root, cross-linked from the cache-layout block. Comments and docs only; no behaviour change. Closes #7 |
||
 claudeandClaude Opus 5
|
719475831b
|
fix(cargo-cache): scope the safety claim to what the code actually prevents
Follow-up to |
||
|
claudeandClaude Opus 5
|
f57e2a6013
|
fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L |
||
|
claudeandClaude Opus 5
|
248af3061e
|
feat(cargo-cache): hardlink-clone a per-ref Cargo cache from a published snapshot
Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L |