Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fa3cef53e0
|
||
|
|
554310186f
|
||
|
|
50e430f4de | ||
|
|
e94c46f2ec
|
||
|
|
5b46986cd7
|
||
|
|
5a90d5c829
|
||
|
|
eb3f0bd09b
|
||
|
|
6a80423bd0
|
||
|
|
5e8e773f78
|
||
|
|
3f97d3d1e7
|
||
|
|
fb7a788c90
|
||
|
|
f76789358d
|
||
|
|
1aca90b461 | ||
|
|
df6f1b91fb |
@@ -0,0 +1,103 @@
|
|||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
# Spelled out only to keep `ready_for_review` in the list — naming any type
|
||||||
|
# replaces the whole default set, so the other three have to be restated.
|
||||||
|
# It is inert on this instance (draft state here is the `WIP:` title
|
||||||
|
# prefix, so un-drafting is a title edit and raises no
|
||||||
|
# `ready_for_review` action) and costs nothing.
|
||||||
|
#
|
||||||
|
# The consequence, which is the part that bites: the `if:` guard below is
|
||||||
|
# evaluated when a run is CREATED, and un-drafting creates no run. A PR
|
||||||
|
# opened as a draft keeps its skip decision until something else produces
|
||||||
|
# one. Push an empty commit after un-WIP'ing.
|
||||||
|
types: [opened, synchronize, reopened, ready_for_review]
|
||||||
|
|
||||||
|
# gitdan-ci runs four repos' CI on two capacity slots, and the compiler-backed
|
||||||
|
# suites below are multi-minute. A superseded run costs a slot in front of
|
||||||
|
# somebody's build, so drop it.
|
||||||
|
#
|
||||||
|
# `push` groups on `github.sha` rather than `github.ref`: a constant per-branch
|
||||||
|
# group is what let this Gitea (1.26.0) cancel two of daniel/gitdan's merge
|
||||||
|
# runs outright while `cancel-in-progress` was gated away from `push` entirely
|
||||||
|
# — see the long note in that repo's ci.yaml for the evidence. Giving every
|
||||||
|
# commit its own group leaves that behaviour nothing to act on.
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
selftest:
|
||||||
|
name: shellcheck + selftests
|
||||||
|
# Two clauses, both load-bearing. The second skips draft PRs: Gitea sets
|
||||||
|
# draft:true when the title starts with `WIP:`, so work-in-progress pushes
|
||||||
|
# cost the shared runner nothing until the PR is un-WIP'd. The first is
|
||||||
|
# what keeps that from also skipping pushes to `main` — a `push` event has
|
||||||
|
# no `pull_request` context, so `github.event.pull_request.draft` is empty
|
||||||
|
# there and the negation alone would be unreliable. Never drop it.
|
||||||
|
if: ${{ github.event_name != 'pull_request' || !github.event.pull_request.draft }}
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 20
|
||||||
|
# Deliberately no `container.volumes:` entry, unlike every repo that
|
||||||
|
# CONSUMES this action. The suites below build throwaway workspaces under
|
||||||
|
# `mktemp -d` and want a cold target dir every time — a persistent cache
|
||||||
|
# would make "did this run rebuild?" unanswerable, which is the question
|
||||||
|
# restore-mtimes-selftest.sh exists to ask. So this job takes no share of
|
||||||
|
# the shared CI cache disk budget.
|
||||||
|
steps:
|
||||||
|
- name: Checkout sources
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Install shellcheck
|
||||||
|
uses: taiki-e/install-action@v2
|
||||||
|
with:
|
||||||
|
tool: shellcheck
|
||||||
|
|
||||||
|
# `hardlink-clone-selftest.sh` and `restore-mtimes-selftest.sh` drive a
|
||||||
|
# real Cargo against a real scratch workspace — they are the only things
|
||||||
|
# here that verify the hardlink-aliasing and mtime-freshness behaviour
|
||||||
|
# against the compiler rather than against a fixture, and selftest.sh's
|
||||||
|
# own header says `--fast` is for iterating, not for signing off a
|
||||||
|
# change. So CI installs a toolchain and runs the full set.
|
||||||
|
#
|
||||||
|
# The scratch workspaces use path dependencies only, so nothing here
|
||||||
|
# reaches crates.io.
|
||||||
|
# Nightly first, stable second, so stable ends up the default and
|
||||||
|
# nightly is only reachable through an explicit `+nightly`.
|
||||||
|
#
|
||||||
|
# `hardlink-clone-selftest.sh`'s last scenario needs a Cargo that
|
||||||
|
# resolves freshness by CONTENT — the mode where the dep-info file
|
||||||
|
# carries per-source checksums, which is the mutation that turns a
|
||||||
|
# hardlink clone into silent stale-artifact reuse rather than a slow
|
||||||
|
# build. This step is what supplies it, and as of 2026-08-26 it does:
|
||||||
|
# the scenario ran and passed against 1.100.0-nightly.
|
||||||
|
#
|
||||||
|
# It briefly did not. Cargo PR #17382 (2026-08-22) demoted
|
||||||
|
# `-Z checksum-freshness` to a gate and gave `build.fingerprint` the
|
||||||
|
# choice, defaulting to `mtime`, so the suite — which set only the gate —
|
||||||
|
# measured a genuine INACTIVE and skipped its strongest scenario. That
|
||||||
|
# read as "upstream withdrew content freshness" and was written up here
|
||||||
|
# as this step buying nothing. It was a moved switch, not a withdrawal;
|
||||||
|
# the suite now exports both and the coverage is back. See
|
||||||
|
# daniel/gitdan#62 for the investigation.
|
||||||
|
- name: Install Rust nightly
|
||||||
|
uses: dtolnay/rust-toolchain@nightly
|
||||||
|
- name: Install Rust toolchain
|
||||||
|
uses: dtolnay/rust-toolchain@stable
|
||||||
|
|
||||||
|
# Every script, including the suites themselves. `-x` follows the
|
||||||
|
# `. cache-lib.sh` each one sources, which is where most of the logic
|
||||||
|
# being checked actually lives; without it shellcheck reports SC1091 and
|
||||||
|
# analyses each file with a hole in it.
|
||||||
|
- name: shellcheck
|
||||||
|
run: shellcheck -x --source-path=scripts scripts/*.sh
|
||||||
|
|
||||||
|
# One command, not six: selftest.sh is the entry point a developer runs,
|
||||||
|
# so a suite added there is gated here without a matching edit in this
|
||||||
|
# file.
|
||||||
|
- name: Selftests
|
||||||
|
run: bash scripts/selftest.sh
|
||||||
@@ -31,14 +31,23 @@ One thing neither project had, and the reason the clone is not a plain
|
|||||||
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
||||||
cloned from.** Cargo writes real artifacts by replacing them, but writes its
|
cloned from.** Cargo writes real artifacts by replacing them, but writes its
|
||||||
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
||||||
write, straight through the shared inode. Under
|
write, straight through the shared inode. When Cargo resolves freshness by
|
||||||
`CARGO_UNSTABLE_CHECKSUM_FRESHNESS` the file that gets corrupted is
|
content the file that gets corrupted is its dep-info fingerprint —
|
||||||
`.fingerprint/<unit>/dep-<target>`, which holds the per-source checksums that
|
`.fingerprint/<unit>/dep-<target>`, or
|
||||||
decide freshness, and the failure is silent stale-artifact reuse rather than a
|
`build/<pkg>/<hash>/fingerprint/dep-<target>` under Cargo's build-dir layout v2
|
||||||
slow build. `scripts/hardlink-clone-selftest.sh` reproduces it as an explicit
|
— which holds the per-source checksums that decide freshness, and the failure
|
||||||
control and asserts the fix. The fix is to hardlink the artifacts (the GB) and
|
is silent stale-artifact reuse rather than a slow build.
|
||||||
real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory,
|
`scripts/hardlink-clone-selftest.sh` reproduces it as an explicit control and
|
||||||
against 100% for a full copy.
|
asserts the fix. The fix is to hardlink the artifacts (the GB) and real-copy
|
||||||
|
the metadata (the MB) — about 3.7% of a Bevy-sized target directory, against
|
||||||
|
100% for a full copy.
|
||||||
|
|
||||||
|
That ratio is a **layout-v1** figure and does not survive layout v2, which
|
||||||
|
regroups artifacts under `build/` alongside the metadata and so drags nearly
|
||||||
|
the whole tree into the real-copy set. Measured 2026-08-26 at 99.998% on a
|
||||||
|
scratch crate. v2 is the nightly default and stabilises in cargo 1.100.0 on
|
||||||
|
2026-11-12; the correctness guarantee above is unaffected, the saving is not.
|
||||||
|
Tracked as [#14](https://gitdan.com/daniel/gitdan-actions/issues/14).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -98,10 +107,14 @@ env:
|
|||||||
CARGO_INCREMENTAL: 0 # per-run bloat on a persistent volume
|
CARGO_INCREMENTAL: 0 # per-run bloat on a persistent volume
|
||||||
CARGO_PROFILE_DEV_DEBUG: line-tables-only
|
CARGO_PROFILE_DEV_DEBUG: line-tables-only
|
||||||
CARGO_PROFILE_TEST_DEBUG: line-tables-only
|
CARGO_PROFILE_TEST_DEBUG: line-tables-only
|
||||||
# Nightly only. Content-addressed freshness instead of mtime-based — a
|
# Nightly only, and BOTH are needed. Content-addressed freshness instead of
|
||||||
# strictly stronger guarantee, complementary to the mtime restore (which
|
# mtime-based — a strictly stronger guarantee, complementary to the mtime
|
||||||
# still covers directory-form `rerun-if-changed` build-script watches).
|
# restore (which still covers directory-form `rerun-if-changed` build-script
|
||||||
|
# watches). Since cargo PR #17382 (2026-08-22) the `-Z` gate below only
|
||||||
|
# unlocks the feature; `build.fingerprint` selects it and defaults to
|
||||||
|
# `mtime`, so the gate on its own is accepted and does nothing.
|
||||||
CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"
|
CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"
|
||||||
|
CARGO_BUILD_FINGERPRINT: "content"
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -287,6 +300,69 @@ Lowering either here needs no coordination: the arbiter then only defers a
|
|||||||
reclamation this side would already have permitted, which costs disk rather
|
reclamation this side would already have permitted, which costs disk rather
|
||||||
than correctness.
|
than correctness.
|
||||||
|
|
||||||
|
### Names this repo doesn't reclaim, but the arbiter depends on
|
||||||
|
|
||||||
|
The five names above are leftovers — dead trees gitdan's arbiter finds and
|
||||||
|
deletes. Four other dot-prefixed names change that arbiter's *behaviour*
|
||||||
|
without ever being a leftover: it reads them to make a correctness decision
|
||||||
|
and never reclaims them. They are a real cross-repo dependency too, on the
|
||||||
|
name's spelling rather than on the tree's lifetime — gitdan's
|
||||||
|
`DEPENDED-UPON NAMES CONTRACT` block (in `scripts/ci-cache-reclaim.sh`) is the
|
||||||
|
canonical description; this is the producer's half for the two we own.
|
||||||
|
|
||||||
|
| Name | Produced here by | Consequence of an unannounced rename |
|
||||||
|
|---|---|---|
|
||||||
|
| `.ci-lock-<id>` | `scripts/cache-lock.sh` (acquire/release) and `cache-lib.sh`'s `write_cache_lock()` | The arbiter's lock check silently stops matching — a live job's staging tree loses its liveness guard and becomes an ordinary reclaim candidate while still in use |
|
||||||
|
| `.cache-last-used` | `cargo-cache/action.yml`, stamped every run | The arbiter falls back to directory mtime — silently reordering its eviction order, and possibly failing to recognise the directory as a cache dir at all |
|
||||||
|
|
||||||
|
Two more names are in gitdan's list — `.gitea-last-used` and `.ci-keep` — and
|
||||||
|
neither is produced by anything in this repo, so there is no producer-side
|
||||||
|
half to write here. `.gitea-last-used` is a naming convention individual
|
||||||
|
repos used before adopting this shared action; nothing here writes it, and
|
||||||
|
gitdan's script reads it only for compatibility with directories created
|
||||||
|
under that older scheme. `.ci-keep` is a per-repo, hand-placed opt-out a
|
||||||
|
consuming repo's own workflow drops directly into a cache directory it wants
|
||||||
|
exempted from eviction — never something this action or its scripts write.
|
||||||
|
|
||||||
|
**Adding a fifth depended-upon name counts as much as renaming one of the two
|
||||||
|
above.** If a future change here makes gitdan's arbiter start depending on the
|
||||||
|
spelling of some new dot-prefixed name — a name it reads for a decision but
|
||||||
|
never reclaims — that is exactly this category, and it needs the matching
|
||||||
|
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
||||||
|
|
||||||
|
### The directory LAYOUT is part of that contract as well
|
||||||
|
|
||||||
|
Names are one half; where they sit is the other. gitdan's arbiter walks a
|
||||||
|
volume's `_data` tree to `CI_CACHE_MAX_DEPTH`, which is **2** — deliberately
|
||||||
|
tight, because a deeper walk starts meeting Cargo's own
|
||||||
|
`incremental/<crate>-<hash>` directories, which match the same name shape it
|
||||||
|
uses to recognise a cache dir and must never be evicted individually. So:
|
||||||
|
|
||||||
|
```
|
||||||
|
_data/target-<key> depth 1 — no lineage
|
||||||
|
_data/<lineage>/target-<key> depth 2 — a lineage
|
||||||
|
_data/<a>/<b>/target-<key> depth 3 — INVISIBLE to the arbiter
|
||||||
|
```
|
||||||
|
|
||||||
|
That budget is the whole reason `cache-lineage` is one path component and not
|
||||||
|
a path. Nesting deeper is not an error anywhere: the caches work, the
|
||||||
|
in-workflow prune pass keeps managing them, and the one script whose job is
|
||||||
|
the shared disk budget across every repo simply never sees them again.
|
||||||
|
|
||||||
|
It is also why the fix for daniel/gitdan#60 nests rather than suffixing the
|
||||||
|
cache key. A `target-<key>-<lineage>` name would be read as dead by
|
||||||
|
`prune-cache.sh`'s liveness pass — which classifies by recomputing
|
||||||
|
`target-<cache_key(branch)>` for every branch on origin — and evicted
|
||||||
|
unconditionally on every run; and it falls out of the arbiter's own
|
||||||
|
`BRANCH_DIR_RE` too, so the same directories would never be candidates there
|
||||||
|
either. Nesting leaves both matchers reading exactly the names they already
|
||||||
|
read, one level down.
|
||||||
|
|
||||||
|
One known rough edge, on gitdan's side and cosmetic: that script logs an
|
||||||
|
eviction as `<volume>/<basename>`, so a nested `target-<key>` and a flat one
|
||||||
|
of the same key are indistinguishable in its output. It evicts the right
|
||||||
|
directory; the line just doesn't say which.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
@@ -296,6 +372,7 @@ than correctness.
|
|||||||
| input | default | meaning |
|
| input | default | meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
||||||
|
| `cache-lineage` | *(empty)* | one directory level under `cache-root`, for a second job building the same ref for a different target or profile — see [Multiple jobs in one workflow](#multiple-jobs-in-one-workflow) |
|
||||||
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
||||||
| `min-free-percent` | `10` | prune when free space drops below this |
|
| `min-free-percent` | `10` | prune when free space drops below this |
|
||||||
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
||||||
@@ -304,7 +381,7 @@ than correctness.
|
|||||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||||
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
||||||
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
||||||
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
|
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one target directory; the default already does |
|
||||||
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
||||||
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
||||||
|
|
||||||
@@ -320,6 +397,7 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
|||||||
| input | default | meaning |
|
| input | default | meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cache-root` | `/cache` | must match the consume action |
|
| `cache-root` | `/cache` | must match the consume action |
|
||||||
|
| `cache-lineage` | *(empty)* | must match the consume action; a mismatch fails the step rather than publishing the wrong tree |
|
||||||
| `protected-branches` | `dev main` | refs that publish snapshots |
|
| `protected-branches` | `dev main` | refs that publish snapshots |
|
||||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||||
@@ -336,13 +414,70 @@ of a merge-preview build, which is not what `dev` is.
|
|||||||
|
|
||||||
## Multiple jobs in one workflow
|
## Multiple jobs in one workflow
|
||||||
|
|
||||||
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
|
Two jobs building the same ref — a `ci` job and a `wasm` job, say — are two
|
||||||
each need their **own** watermark file. A shared one breaks the moment two
|
consumers of one cache key, and the cache key alone is not enough to keep them
|
||||||
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
|
apart.
|
||||||
and job B then reads that just-advanced value, computes an empty diff, and
|
|
||||||
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
|
**Give each its own lineage.** A cache key names a *ref*; what a target
|
||||||
already gives each job its own; only override `watermark-file` if you also
|
directory holds is the product of a ref and a build configuration. Left to the
|
||||||
override `lock-id`, and then keep both distinct per job.
|
key alone, both jobs export the same `CARGO_TARGET_DIR`, and Cargo's
|
||||||
|
build-directory lock is exclusive — so on a runner with more than one slot the
|
||||||
|
second job sits on `Blocking waiting for file lock on build directory` for the
|
||||||
|
length of the first, occupying a capacity slot while doing nothing
|
||||||
|
(daniel/gitdan#60). `cache-lineage` is that second dimension:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: Restore the Cargo cache
|
||||||
|
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
|
||||||
|
with:
|
||||||
|
cache-lineage: wasm32 # the `ci` job sets none
|
||||||
|
|
||||||
|
# ... build steps ...
|
||||||
|
|
||||||
|
- name: Record watermark, publish cache snapshot
|
||||||
|
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
||||||
|
with:
|
||||||
|
cache-lineage: wasm32 # the SAME value, or the step fails
|
||||||
|
```
|
||||||
|
|
||||||
|
A lineage nests one directory level under the cache root
|
||||||
|
(`<cache-root>/<lineage>/target-<key>`), so each lineage gets its own target
|
||||||
|
dirs, its own snapshots, and its own prune pass. Everything else works as it
|
||||||
|
already did, one level down: a PR branch in a lineage layers over **that
|
||||||
|
lineage's** base snapshot, the publisher branch publishes into it, and a prune
|
||||||
|
pass run inside it never sees a sibling lineage's caches.
|
||||||
|
|
||||||
|
Setting no lineage resolves to the cache root unchanged, byte for byte, so a
|
||||||
|
workflow that does not use one keeps the exact directories it already has on
|
||||||
|
the volume.
|
||||||
|
|
||||||
|
**Both actions need the same value.** `cargo-cache-publish` derives both ends
|
||||||
|
of the snapshot swap from its own `cache-root`, so a publish step left at the
|
||||||
|
default while its consume step nested would republish a *different* lineage's
|
||||||
|
live target dir over that lineage's snapshot, on every push, with nothing in
|
||||||
|
the log to say so. The publish action therefore compares its own inputs
|
||||||
|
against the `CARGO_CACHE_ROOT` the consume step exported and fails the step on
|
||||||
|
a mismatch. (The `mode: release-lock` call is exempt: it releases a lock on
|
||||||
|
`$CARGO_TARGET_DIR` and never touches a cache root, so it takes no lineage.)
|
||||||
|
|
||||||
|
**Some lineage names are refused.** A lineage is one path component, drawn
|
||||||
|
from `[A-Za-z0-9._-]`, and several otherwise-reasonable names are rejected at
|
||||||
|
resolve time because a *reader elsewhere* would stop seeing the caches
|
||||||
|
underneath them: a Cargo profile name (`debug`, `release`, `doc`, …) is one
|
||||||
|
gitdan's arbiter never descends into, a `target-`/`snapshot-` prefix makes the
|
||||||
|
lineage directory itself an eviction candidate for this repo's own prune pass,
|
||||||
|
and a hex-suffixed name is read by that arbiter as a per-branch cache dir in
|
||||||
|
its own right. `validate_cache_lineage()` in `scripts/cache-lib.sh` states each
|
||||||
|
rejection with the reader that imposes it.
|
||||||
|
|
||||||
|
**Watermarks are still per job.** Two jobs in one lineage — or one job before
|
||||||
|
lineages were introduced — each need their **own** watermark file. A shared one
|
||||||
|
breaks the moment two jobs run in sequence within one trigger: job A advances
|
||||||
|
the watermark to HEAD, and job B then reads that just-advanced value, computes
|
||||||
|
an empty diff, and loses the merge protection entirely. The default
|
||||||
|
(`.ci-watermark-<job>-sha`) already gives each job its own; only override
|
||||||
|
`watermark-file` if you also override `lock-id`, and then keep both distinct
|
||||||
|
per job.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -415,13 +550,35 @@ not automatic.
|
|||||||
## Development
|
## Development
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash scripts/selftest.sh # everything (~1 min; needs cargo)
|
shellcheck -x --source-path=scripts scripts/*.sh
|
||||||
|
bash scripts/selftest.sh # everything (needs cargo)
|
||||||
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Both run in CI — `.gitea/workflows/ci.yaml`, one job, on pushes to `main` and
|
||||||
|
on PRs that were non-draft when the run was created. It installs shellcheck
|
||||||
|
and both a stable and a nightly Rust toolchain (nightly so
|
||||||
|
`hardlink-clone-selftest.sh` can run its content-freshness scenario, which as
|
||||||
|
of 2026-08-26 a nightly does enable — 1.100.0-nightly (787af2b8c 2026-08-25)
|
||||||
|
resolves freshness by content given both `CARGO_UNSTABLE_CHECKSUM_FRESHNESS`
|
||||||
|
and `CARGO_BUILD_FINGERPRINT: content`, per cargo PR #17382; the suite still
|
||||||
|
settles that by experiment on every run and skips the scenario loudly when it
|
||||||
|
cannot measure) and references no credentials; the scratch workspaces the
|
||||||
|
compiler-backed suites build use path dependencies only, so nothing reaches
|
||||||
|
crates.io. It runs the full suite rather than `--fast`,
|
||||||
|
because the two compiler-backed suites are the ones that check this scheme
|
||||||
|
against real Cargo instead of against a fixture. Draft (`WIP:`-titled) PRs
|
||||||
|
skip it, and un-drafting does **not** un-skip them — the guard is evaluated
|
||||||
|
when a run is created and un-drafting creates none, so push an empty commit
|
||||||
|
after un-WIP'ing.
|
||||||
|
|
||||||
|
This repo is consumed by three other repos' CI at `@v1`, a moving tag, so a
|
||||||
|
change here reaches all of them at once. That is what the gate is for.
|
||||||
|
|
||||||
| suite | covers |
|
| suite | covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
||||||
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler, **and a nightly that actually resolves freshness by content for its last scenario**: the source's-next-build check reasons about content rather than mtime, so under mtime freshness it would assert a bug. Whether the toolchain does is settled by experiment on a throwaway crate, not by asking it — accepting `-Z checksum-freshness` stopped implying it on 2026-08-22, when cargo PR #17382 demoted the flag to a gate and gave `build.fingerprint` (default `mtime`) the choice; the suite now exports both and 1.100.0-nightly measures ACTIVE again. The experiment reports **three** outcomes, not two: active, measured-inactive, and *not measured*. Its answer codes are `0` and `3`, deliberately clear of every status bash generates for its own errors — so nothing that goes wrong inside the probe, including an expansion failure no guard can catch, can be read as an answer. The scenario is skipped for the last two alike, but a failure to measure is never reported as a measurement. The control also reports which mutation families the running Cargo exhibits — a note, not an assertion, since that set moves upstream. |
|
||||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
||||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
||||||
|
|||||||
@@ -10,6 +10,15 @@ inputs:
|
|||||||
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
||||||
required: false
|
required: false
|
||||||
default: '/cache'
|
default: '/cache'
|
||||||
|
cache-lineage:
|
||||||
|
description: >-
|
||||||
|
Must match the cargo-cache step in this job. Checked rather than
|
||||||
|
assumed: this action derives both ends of the snapshot swap from its own
|
||||||
|
cache-root, so a publish step left at the default while its consume step
|
||||||
|
nested would republish a DIFFERENT lineage's live target dir over that
|
||||||
|
lineage's snapshot, on every push, silently. A mismatch fails the step.
|
||||||
|
required: false
|
||||||
|
default: ''
|
||||||
protected-branches:
|
protected-branches:
|
||||||
description: >-
|
description: >-
|
||||||
Space-separated refs that publish snapshots. A run whose own ref is not
|
Space-separated refs that publish snapshots. A run whose own ref is not
|
||||||
@@ -67,6 +76,11 @@ runs:
|
|||||||
# Everything here reads the environment the consume action exported, so a
|
# Everything here reads the environment the consume action exported, so a
|
||||||
# workflow that forgets to run cargo-cache first fails loudly here rather
|
# workflow that forgets to run cargo-cache first fails loudly here rather
|
||||||
# than silently publishing a snapshot of the wrong directory.
|
# than silently publishing a snapshot of the wrong directory.
|
||||||
|
#
|
||||||
|
# The cache root is taken from that same environment for the same reason,
|
||||||
|
# and this action's own cache-root/cache-lineage inputs are checked against
|
||||||
|
# it rather than used. The two have always had to agree; until a lineage
|
||||||
|
# existed they always did, because nobody overrode the default.
|
||||||
- id: resolve
|
- id: resolve
|
||||||
shell: bash
|
shell: bash
|
||||||
env:
|
env:
|
||||||
@@ -92,6 +106,18 @@ runs:
|
|||||||
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
||||||
echo "active=yes" >> "$GITHUB_OUTPUT"
|
echo "active=yes" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
# Only in publish mode. The `release-lock` call is an `if: always()`
|
||||||
|
# step that consuming workflows invoke with `mode:` and nothing else —
|
||||||
|
# it releases $CARGO_CACHE_LOCK_ID on $CARGO_TARGET_DIR and never
|
||||||
|
# touches a cache root at all, so holding its default inputs to the
|
||||||
|
# consume step's would fail the cleanup step of every job that sets a
|
||||||
|
# lineage, for a value it does not use.
|
||||||
|
if [ "${{ inputs.mode }}" = "publish" ]; then
|
||||||
|
: "${CARGO_CACHE_ROOT:?cargo-cache-publish: CARGO_CACHE_ROOT not set by the cargo-cache action}"
|
||||||
|
bash "${CARGO_CACHE_SCRIPTS}/cache-root.sh" verify \
|
||||||
|
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}" "$CARGO_CACHE_ROOT"
|
||||||
|
fi
|
||||||
|
|
||||||
OWN_REF="${{ inputs.own-ref }}"
|
OWN_REF="${{ inputs.own-ref }}"
|
||||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||||
|
|
||||||
@@ -131,7 +157,7 @@ runs:
|
|||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||||
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
"$CARGO_CACHE_KEY" "$CARGO_CACHE_ROOT" \
|
||||||
"${{ github.job }}-${{ github.run_id }}-$$"
|
"${{ github.job }}-${{ github.run_id }}-$$"
|
||||||
|
|
||||||
# Released in both modes. In `publish` mode this is the normal end-of-job
|
# Released in both modes. In `publish` mode this is the normal end-of-job
|
||||||
|
|||||||
+46
-6
@@ -10,6 +10,18 @@ inputs:
|
|||||||
description: 'Mount point of the persistent cache volume inside the job container.'
|
description: 'Mount point of the persistent cache volume inside the job container.'
|
||||||
required: false
|
required: false
|
||||||
default: '/cache'
|
default: '/cache'
|
||||||
|
cache-lineage:
|
||||||
|
description: >-
|
||||||
|
Distinguishes two jobs that build the SAME ref for different targets or
|
||||||
|
profiles (a host build and a wasm32 build, say) and would otherwise
|
||||||
|
resolve to one CARGO_TARGET_DIR and serialise on Cargo's exclusive
|
||||||
|
build-directory lock. Names one directory level under cache-root:
|
||||||
|
<cache-root>/<lineage>/target-<key>. Must be a single path component;
|
||||||
|
several names are refused outright because a reader elsewhere would stop
|
||||||
|
seeing the caches under them (see validate_cache_lineage in
|
||||||
|
scripts/cache-lib.sh). Pass the same value to cargo-cache-publish.
|
||||||
|
required: false
|
||||||
|
default: ''
|
||||||
protected-branches:
|
protected-branches:
|
||||||
description: >-
|
description: >-
|
||||||
Space-separated refs that publish snapshots and are never evicted.
|
Space-separated refs that publish snapshots and are never evicted.
|
||||||
@@ -54,8 +66,9 @@ inputs:
|
|||||||
watermark-file:
|
watermark-file:
|
||||||
description: >-
|
description: >-
|
||||||
Name of this job's build-watermark file inside the target dir. MUST be
|
Name of this job's build-watermark file inside the target dir. MUST be
|
||||||
distinct per job when two jobs share one cache key. Defaults to
|
distinct per job when two jobs share one target directory — which two
|
||||||
.ci-watermark-<job>-sha.
|
jobs no longer need to do; cache-lineage gives them separate ones.
|
||||||
|
Defaults to .ci-watermark-<job>-sha, already distinct per job.
|
||||||
required: false
|
required: false
|
||||||
default: ''
|
default: ''
|
||||||
lock-id:
|
lock-id:
|
||||||
@@ -74,6 +87,12 @@ outputs:
|
|||||||
cache-key:
|
cache-key:
|
||||||
description: 'Sanitized cache key for this run''s own ref.'
|
description: 'Sanitized cache key for this run''s own ref.'
|
||||||
value: ${{ steps.resolve.outputs.cache-key }}
|
value: ${{ steps.resolve.outputs.cache-key }}
|
||||||
|
cache-root:
|
||||||
|
description: >-
|
||||||
|
Resolved cache root — cache-root, plus the lineage directory when one is
|
||||||
|
set. Every directory this action reads or writes is under it. Also
|
||||||
|
exported as CARGO_CACHE_ROOT.
|
||||||
|
value: ${{ steps.resolve.outputs.cache-root }}
|
||||||
seeded-from:
|
seeded-from:
|
||||||
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
|
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
|
||||||
value: ${{ steps.seed.outputs.seeded-from }}
|
value: ${{ steps.seed.outputs.seeded-from }}
|
||||||
@@ -95,6 +114,16 @@ runs:
|
|||||||
# `base_ref` is populated only for pull_request events. A push run has
|
# `base_ref` is populated only for pull_request events. A push run has
|
||||||
# nothing to layer over: its own ref IS the reference branch. It
|
# nothing to layer over: its own ref IS the reference branch. It
|
||||||
# publishes, it does not consume.
|
# publishes, it does not consume.
|
||||||
|
#
|
||||||
|
# The cache root is resolved first because every path below hangs off it.
|
||||||
|
# A lineage nests one directory level (`<root>/<lineage>/target-<key>`),
|
||||||
|
# which is what lets two jobs on ONE ref hold two build directories and so
|
||||||
|
# not serialise on Cargo's exclusive lock. It is resolved through
|
||||||
|
# cache-root.sh rather than interpolated here so the name is validated —
|
||||||
|
# several otherwise-reasonable lineage names put their whole subtree out of
|
||||||
|
# reach of a pass that has to see it. Everything downstream reads the
|
||||||
|
# resolved value, and it is exported as CARGO_CACHE_ROOT so the publish
|
||||||
|
# action can check it agrees with its own inputs.
|
||||||
- id: resolve
|
- id: resolve
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
@@ -102,6 +131,8 @@ runs:
|
|||||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||||
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
|
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
|
||||||
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
|
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
|
||||||
|
CACHE_ROOT=$(bash "${SCRIPTS}/cache-root.sh" resolve \
|
||||||
|
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}")
|
||||||
OWN_REF="${{ inputs.own-ref }}"
|
OWN_REF="${{ inputs.own-ref }}"
|
||||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||||
BASE_REF="${{ inputs.base-ref }}"
|
BASE_REF="${{ inputs.base-ref }}"
|
||||||
@@ -116,7 +147,7 @@ runs:
|
|||||||
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
|
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
TARGET_DIR="${{ inputs.cache-root }}/target-${OWN_KEY}"
|
TARGET_DIR="${CACHE_ROOT}/target-${OWN_KEY}"
|
||||||
WATERMARK="${{ inputs.watermark-file }}"
|
WATERMARK="${{ inputs.watermark-file }}"
|
||||||
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
|
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
|
||||||
LOCK_ID="${{ inputs.lock-id }}"
|
LOCK_ID="${{ inputs.lock-id }}"
|
||||||
@@ -128,10 +159,11 @@ runs:
|
|||||||
echo "base-key=${BASE_KEY}"
|
echo "base-key=${BASE_KEY}"
|
||||||
echo "lock-id=${LOCK_ID}"
|
echo "lock-id=${LOCK_ID}"
|
||||||
echo "watermark-file=${WATERMARK}"
|
echo "watermark-file=${WATERMARK}"
|
||||||
|
echo "cache-root=${CACHE_ROOT}"
|
||||||
} >> "$GITHUB_OUTPUT"
|
} >> "$GITHUB_OUTPUT"
|
||||||
{
|
{
|
||||||
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
|
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
|
||||||
echo "CARGO_CACHE_ROOT=${{ inputs.cache-root }}"
|
echo "CARGO_CACHE_ROOT=${CACHE_ROOT}"
|
||||||
echo "CARGO_CACHE_KEY=${OWN_KEY}"
|
echo "CARGO_CACHE_KEY=${OWN_KEY}"
|
||||||
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
|
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
|
||||||
echo "CI_WATERMARK_FILE=${WATERMARK}"
|
echo "CI_WATERMARK_FILE=${WATERMARK}"
|
||||||
@@ -155,7 +187,7 @@ runs:
|
|||||||
bash "${SCRIPTS}/seed-target-dir.sh" \
|
bash "${SCRIPTS}/seed-target-dir.sh" \
|
||||||
"${{ steps.resolve.outputs.cache-key }}" \
|
"${{ steps.resolve.outputs.cache-key }}" \
|
||||||
"${{ steps.resolve.outputs.base-key }}" \
|
"${{ steps.resolve.outputs.base-key }}" \
|
||||||
"${{ inputs.cache-root }}" \
|
"${{ steps.resolve.outputs.cache-root }}" \
|
||||||
"${{ github.job }}-${{ github.run_id }}-$$" \
|
"${{ github.job }}-${{ github.run_id }}-$$" \
|
||||||
"${{ inputs.seed-fallback-dir }}" \
|
"${{ inputs.seed-fallback-dir }}" \
|
||||||
"${{ steps.resolve.outputs.lock-id }}"
|
"${{ steps.resolve.outputs.lock-id }}"
|
||||||
@@ -165,6 +197,14 @@ runs:
|
|||||||
# touched unconditionally every run: a run that hits the cache for every
|
# touched unconditionally every run: a run that hits the cache for every
|
||||||
# crate may write nothing at all inside the tree, which would make a
|
# crate may write nothing at all inside the tree, which would make a
|
||||||
# just-used directory look stale to the eviction pass.
|
# just-used directory look stale to the eviction pass.
|
||||||
|
#
|
||||||
|
# `.cache-last-used` IS A CROSS-REPO CONTRACT NAME with daniel/gitdan's
|
||||||
|
# `scripts/ci-cache-reclaim.sh`, which reads this exact marker for its own
|
||||||
|
# LRU ordering and to recognise a directory as a cache dir at all. Rename
|
||||||
|
# it here without a matching change there and that script silently falls
|
||||||
|
# back to directory mtime for both — see README.md's "Scratch names in a
|
||||||
|
# cache root" section and the `DEPENDED-UPON NAMES CONTRACT` block in
|
||||||
|
# gitdan's script.
|
||||||
- shell: bash
|
- shell: bash
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
@@ -198,7 +238,7 @@ runs:
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||||
bash "${SCRIPTS}/prune-cache.sh" \
|
bash "${SCRIPTS}/prune-cache.sh" \
|
||||||
"${{ inputs.cache-root }}" \
|
"${{ steps.resolve.outputs.cache-root }}" \
|
||||||
"${{ steps.resolve.outputs.target-dir }}" \
|
"${{ steps.resolve.outputs.target-dir }}" \
|
||||||
"${{ inputs.protected-branches }}" \
|
"${{ inputs.protected-branches }}" \
|
||||||
"${{ inputs.min-free-percent }}"
|
"${{ inputs.min-free-percent }}"
|
||||||
|
|||||||
+168
-9
@@ -114,6 +114,106 @@ cache_key() {
|
|||||||
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
|
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
|
||||||
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
|
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Cache lineages
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# A cache key names a REF. What a target directory holds is the product of a
|
||||||
|
# ref and a BUILD CONFIGURATION, and the two are not the same thing: emowheel
|
||||||
|
# builds the same ref twice on every push, once for the host and once for
|
||||||
|
# wasm32, in two jobs that run concurrently. Keyed on the ref alone both
|
||||||
|
# resolve to one CARGO_TARGET_DIR, and Cargo's build-directory lock is
|
||||||
|
# exclusive — so the second job sits on `Blocking waiting for file lock on
|
||||||
|
# build directory` for the length of the first, holding a runner capacity slot
|
||||||
|
# while doing nothing (daniel/gitdan#60).
|
||||||
|
#
|
||||||
|
# A lineage is that second dimension, and it is expressed as ONE DIRECTORY
|
||||||
|
# LEVEL above the per-ref directories rather than as a suffix on the key:
|
||||||
|
#
|
||||||
|
# <cache-root>/target-<key> no lineage (the flat layout)
|
||||||
|
# <cache-root>/<lineage>/target-<key> a lineage
|
||||||
|
#
|
||||||
|
# Nesting rather than suffixing is what keeps every existing reader correct
|
||||||
|
# without teaching any of them a new name shape. `prune-cache.sh` resolves
|
||||||
|
# liveness by recomputing `target-<cache_key(branch)>` for every branch on
|
||||||
|
# origin and evicting whatever does not match — a suffixed `target-<key>-wasm32`
|
||||||
|
# matches nothing, so it would be classified dead and unconditionally evicted
|
||||||
|
# on every single run. The host-level arbiter in daniel/gitdan reads the same
|
||||||
|
# shape (its BRANCH_DIR_RE), and a suffixed name falls out of it too: not
|
||||||
|
# evicted there, but never a candidate either, so a whole lineage becomes
|
||||||
|
# invisible to the global disk budget. Nesting leaves both matchers reading
|
||||||
|
# exactly the names they already read, one directory deeper.
|
||||||
|
#
|
||||||
|
# ONE LEVEL, AND NOT TWO. The arbiter walks a volume to CI_CACHE_MAX_DEPTH,
|
||||||
|
# which is 2 — `_data/target-<key>` and `_data/<lineage>/target-<key>`. It is
|
||||||
|
# kept tight there on purpose (a deeper walk starts meeting Cargo's own
|
||||||
|
# `incremental/<crate>-<hash>` directories, which match the same name shape and
|
||||||
|
# must never be evicted individually), so a lineage is a single path component
|
||||||
|
# and validate_cache_lineage refuses one containing a slash.
|
||||||
|
|
||||||
|
# Directory names daniel/gitdan's ci-cache-reclaim.sh refuses to descend into
|
||||||
|
# (its CI_CACHE_NODESCEND_NAMES). A lineage named one of these puts its whole
|
||||||
|
# subtree outside the global arbiter's reach: the caches accumulate and the one
|
||||||
|
# script whose job is the shared disk budget cannot see them.
|
||||||
|
CACHE_LINEAGE_RESERVED_NAMES="debug release deps incremental build .fingerprint tmp examples doc"
|
||||||
|
|
||||||
|
# The shape that same script reads as a per-branch cache directory (its
|
||||||
|
# BRANCH_DIR_RE). A lineage matching it is taken for a cache dir in its own
|
||||||
|
# right — never descended into, and an eviction candidate whole, which is the
|
||||||
|
# entire lineage rather than one ref's share of it.
|
||||||
|
CACHE_LINEAGE_BRANCH_DIR_RE='^.+-[0-9a-f]{7,40}$'
|
||||||
|
|
||||||
|
# Every rejection below names the reader that imposes it, because that is the
|
||||||
|
# only way the constraint survives: none of these is a filesystem limit, and a
|
||||||
|
# name that trips one produces no error anywhere — it produces a lineage that
|
||||||
|
# silently stops being pruned, or silently stops being reclaimed.
|
||||||
|
validate_cache_lineage() {
|
||||||
|
local lineage="$1" reserved
|
||||||
|
[ -n "$lineage" ] || return 0
|
||||||
|
|
||||||
|
case "$lineage" in
|
||||||
|
*/*)
|
||||||
|
echo "::error::cache lineage '${lineage}' must be a single path component: the host-level arbiter walks a cache volume to depth 2, so <cache-root>/<lineage>/target-<key> is as deep as a cache directory may sit and still be reclaimable" >&2
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
.*)
|
||||||
|
echo "::error::cache lineage '${lineage}' must not start with a dot: every dot-prefixed entry under a cache root belongs to the leftover-naming contract (see the top of this file), and a lineage is not garbage to be reclaimed" >&2
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
target-* | snapshot-*)
|
||||||
|
echo "::error::cache lineage '${lineage}' must not start with 'target-' or 'snapshot-': prune-cache.sh globs both prefixes at the cache root, so the lineage directory itself would become an eviction candidate" >&2
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
*[!A-Za-z0-9._-]*)
|
||||||
|
echo "::error::cache lineage '${lineage}' may contain only [A-Za-z0-9._-] — the same charset cache_key() sanitises a ref down to" >&2
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
for reserved in $CACHE_LINEAGE_RESERVED_NAMES; do
|
||||||
|
if [ "$lineage" = "$reserved" ]; then
|
||||||
|
echo "::error::cache lineage '${lineage}' is one of the Cargo directory names daniel/gitdan's ci-cache-reclaim.sh never descends into (CI_CACHE_NODESCEND_NAMES) — every cache under it would be invisible to the host-level disk budget" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ $lineage =~ $CACHE_LINEAGE_BRANCH_DIR_RE ]]; then
|
||||||
|
echo "::error::cache lineage '${lineage}' ends in a hex suffix, which is the shape daniel/gitdan's ci-cache-reclaim.sh reads as a per-branch cache directory — it would treat the lineage directory as one cache and evict the whole thing" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# The cache root a lineage's directories actually live under. An empty lineage
|
||||||
|
# resolves to the cache root unchanged, byte for byte: that is what makes this
|
||||||
|
# a no-op for every consumer that does not set one, rather than a migration.
|
||||||
|
cache_root_for() {
|
||||||
|
local root="$1" lineage="${2:-}"
|
||||||
|
validate_cache_lineage "$lineage" || return 1
|
||||||
|
printf '%s%s' "$root" "${lineage:+/$lineage}"
|
||||||
|
}
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Disk accounting
|
# Disk accounting
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -210,6 +310,9 @@ _unshare_files() {
|
|||||||
# The inner shell propagates a failure of any individual copy-and-rename out
|
# The inner shell propagates a failure of any individual copy-and-rename out
|
||||||
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
||||||
# partially-unshared tree is reported rather than silently accepted.
|
# partially-unshared tree is reported rather than silently accepted.
|
||||||
|
# shellcheck disable=SC2016 # the quoted program is for the INNER shell: $f
|
||||||
|
# is its loop variable, $rc its accumulator and $$ its pid. Expanding any of
|
||||||
|
# them here is what the single quotes exist to prevent.
|
||||||
find "$@" -links +1 -print0 2>/dev/null |
|
find "$@" -links +1 -print0 2>/dev/null |
|
||||||
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
||||||
}
|
}
|
||||||
@@ -226,11 +329,18 @@ _unshare_files() {
|
|||||||
# the source) the following files in the SOURCE were mutated through the
|
# the source) the following files in the SOURCE were mutated through the
|
||||||
# shared inode:
|
# shared inode:
|
||||||
#
|
#
|
||||||
# <profile>/.fingerprint/<unit>/dep-<target> (only under
|
# <profile>/.fingerprint/<unit>/dep-<target> (build-dir layout v1) — or,
|
||||||
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS,
|
# <profile>/build/<pkg>/<hash>/fingerprint/dep-<target>
|
||||||
# where this file carries the
|
# (build-dir layout v2; see the
|
||||||
# per-source blake3 checksums)
|
# dated note below for which
|
||||||
# <profile>/build/<pkg>/output, root-output (Cargo build-script metadata)
|
# Cargo writes which). Only
|
||||||
|
# when Cargo resolves freshness
|
||||||
|
# by CONTENT, where this file
|
||||||
|
# carries the per-source blake3
|
||||||
|
# checksums.
|
||||||
|
# <profile>/build/<pkg>/output, root-output (Cargo build-script metadata;
|
||||||
|
# `<pkg>/<hash>/run/root-output`
|
||||||
|
# under layout v2)
|
||||||
# <profile>/build/<pkg>/out/** (whatever the build script
|
# <profile>/build/<pkg>/out/** (whatever the build script
|
||||||
# writes into OUT_DIR — build
|
# writes into OUT_DIR — build
|
||||||
# scripts overwhelmingly use a
|
# scripts overwhelmingly use a
|
||||||
@@ -246,10 +356,59 @@ _unshare_files() {
|
|||||||
# sources, reports `Fresh`, and reuses a binary built from the PRE-merge code.
|
# sources, reports `Fresh`, and reuses a binary built from the PRE-merge code.
|
||||||
# That is silent stale-artifact reuse — a wrong answer, not a slow one.
|
# That is silent stale-artifact reuse — a wrong answer, not a slow one.
|
||||||
#
|
#
|
||||||
# So: hardlink the artifacts (the GB), real-copy the metadata (the MB).
|
# WHAT UPSTREAM CHANGED, AND WHAT IT DID NOT (measured 2026-08-26, daniel/gitdan#62).
|
||||||
# Measured on a 6.9 GB Bevy workspace target dir, the unshared set is
|
#
|
||||||
# .fingerprint 22 MB + build/ 237 MB + a handful of dep-info files — about
|
# An earlier revision of this comment recorded that the dep-info write was
|
||||||
# 3.7% of the tree, against 100% for a plain `cp -a`.
|
# "NOT reproduced on 1.100.0-nightly (2026-08-25) — upstream appears to have
|
||||||
|
# stopped writing it in place". That reading was wrong, and the way it was
|
||||||
|
# wrong is the reason this paragraph is dated. Two unrelated upstream changes
|
||||||
|
# landed within days of each other, and between them they moved both the
|
||||||
|
# switch that turns the behaviour on and the path it writes to:
|
||||||
|
#
|
||||||
|
# 1. The ON-SWITCH MOVED. cargo PR #17382 `feat(config): Add build.fingerprint`
|
||||||
|
# (merged 2026-08-22) demoted `-Z checksum-freshness` to a gate: it now
|
||||||
|
# only UNLOCKS the feature, and `build.fingerprint` SELECTS it, defaulting
|
||||||
|
# to `"mtime"`. So `CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true` on its own is
|
||||||
|
# accepted and does nothing, which is exactly the "flag accepted, mtime
|
||||||
|
# anyway" result that was mistaken for a withdrawal. Content freshness
|
||||||
|
# needs BOTH, and with both it is entirely intact:
|
||||||
|
#
|
||||||
|
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true CARGO_BUILD_FINGERPRINT=content
|
||||||
|
#
|
||||||
|
# Measured on cargo 1.100.0-nightly (e8cb624d5 2026-08-22): with the gate
|
||||||
|
# alone a `cp -al` clone mutates only the build/ and *.d families; add
|
||||||
|
# `CARGO_BUILD_FINGERPRINT=content` and the source's dep-info file is
|
||||||
|
# mutated through the shared inode again. Same toolchain, same clone, one
|
||||||
|
# env var apart. The hazard was never removed — it was switched off.
|
||||||
|
#
|
||||||
|
# 2. THE PATH MOVED. Build-dir layout v2 (`-Z build-dir-new-layout`, cargo
|
||||||
|
# 1.91) became the nightly default in cargo 1.99 (PR #17258) and was
|
||||||
|
# stabilised by PR #17354, merged 2026-08-18, shipping in cargo 1.100.0
|
||||||
|
# stable on 2026-11-12. Under v2 there is no `<profile>/.fingerprint` and
|
||||||
|
# no `<profile>/deps` at all: everything is regrouped per build unit under
|
||||||
|
# `<profile>/build/<pkg>/<hash>/{fingerprint,out,run}/`, artifacts
|
||||||
|
# included. Bracketed locally: cargo 1.97.1 and 1.98.0-nightly write v1,
|
||||||
|
# 1.100.0-nightly writes v2.
|
||||||
|
#
|
||||||
|
# The `-name .fingerprint` clause below therefore matches nothing under a v2
|
||||||
|
# Cargo, and the dep-info file is covered only because the `-name build` clause
|
||||||
|
# happens to swallow its new home. That is belt-and-braces by accident, not by
|
||||||
|
# design — and the same accident makes this function real-copy essentially the
|
||||||
|
# whole tree, because the artifacts moved under `build/` too. Measured on one
|
||||||
|
# scratch crate (serde + serde_json + regex), same sources both ways:
|
||||||
|
#
|
||||||
|
# cargo 1.97.1 (layout v1) 27.0 MB unshared of 126.7 MB — 21.3%
|
||||||
|
# 1.100.0-nightly (layout v2) 105.9 MB unshared of 105.9 MB — 99.998%
|
||||||
|
#
|
||||||
|
# So the guard still holds and the saving does not. Deliberately NOT fixed
|
||||||
|
# here: adjusting the selection is a change to what gets hardlinked on every
|
||||||
|
# consumer, which wants its own change and its own review, and the deadline is
|
||||||
|
# cargo 1.100.0 stable on 2026-11-12. Tracked as gitdan-actions#14.
|
||||||
|
#
|
||||||
|
# The historical v1 figure this block used to quote stands as measured: on a
|
||||||
|
# 6.9 GB Bevy workspace target dir the unshared set was .fingerprint 22 MB +
|
||||||
|
# build/ 237 MB + a handful of dep-info files — about 3.7% of the tree, against
|
||||||
|
# 100% for a plain `cp -a`. It describes layout v1 only.
|
||||||
#
|
#
|
||||||
# `incremental/` is deliberately left shared: rustc writes each incremental
|
# `incremental/` is deliberately left shared: rustc writes each incremental
|
||||||
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
||||||
|
|||||||
@@ -14,6 +14,19 @@
|
|||||||
# until STALE_LOCK_SECONDS, after which it is treated as abandoned by a job
|
# until STALE_LOCK_SECONDS, after which it is treated as abandoned by a job
|
||||||
# the runner killed before it reached its own release step. Honouring a lock
|
# the runner killed before it reached its own release step. Honouring a lock
|
||||||
# forever would let one crashed job pin a directory permanently.
|
# forever would let one crashed job pin a directory permanently.
|
||||||
|
#
|
||||||
|
# `.ci-lock-<id>` IS A CROSS-REPO CONTRACT NAME with daniel/gitdan's
|
||||||
|
# `scripts/ci-cache-reclaim.sh`: that host-level arbiter reads this same
|
||||||
|
# marker (its `leftover_is_locked()`) to decide whether a leftover it is about
|
||||||
|
# to reclaim is still held open by a live job. `cache-lib.sh`'s
|
||||||
|
# `write_cache_lock()` writes the identical shape independently of this
|
||||||
|
# script. Rename or reshape this marker here without a matching change on
|
||||||
|
# gitdan's side and the arbiter's lock check silently stops matching — a live
|
||||||
|
# job's staging tree loses its liveness guard and becomes an ordinary reclaim
|
||||||
|
# candidate while still in use. See the "Scratch names in a cache root" section
|
||||||
|
# in README.md, and daniel/gitdan's `DEPENDED-UPON NAMES CONTRACT` block in
|
||||||
|
# `scripts/ci-cache-reclaim.sh`, for the full arrangement — this repo's half
|
||||||
|
# is: don't rename `.ci-lock-` without telling that script.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
MODE="${1:?usage: cache-lock.sh acquire|release <dir> <lock-id>}"
|
MODE="${1:?usage: cache-lock.sh acquire|release <dir> <lock-id>}"
|
||||||
|
|||||||
Executable
+194
@@ -0,0 +1,194 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Regression test for cache-root.sh and the lineage rules in cache-lib.sh —
|
||||||
|
# the fix for daniel/gitdan#60, where two jobs building the same ref for
|
||||||
|
# different targets resolved to one CARGO_TARGET_DIR and serialised on Cargo's
|
||||||
|
# exclusive build-directory lock.
|
||||||
|
#
|
||||||
|
# 1. NO LINEAGE CHANGES NOTHING — the effective root is the cache root byte
|
||||||
|
# for byte, so every consumer that does not set a lineage keeps the exact
|
||||||
|
# directories it already has on the volume. This is the whole of the
|
||||||
|
# migration story for lublub, zemyna and emowheel's `ci` job, so it is
|
||||||
|
# asserted rather than assumed.
|
||||||
|
# 2. TWO LINEAGES, ONE REF, TWO TARGET DIRS — the bug itself. The two jobs
|
||||||
|
# keep one cache key (they are the same ref) and still get directories
|
||||||
|
# that are neither equal nor nested one inside the other, which is what
|
||||||
|
# Cargo's per-directory lock needs in order not to serialise them.
|
||||||
|
# 3. THE WHOLE PIPELINE MOVES TOGETHER — seed, publish and prune all operate
|
||||||
|
# inside the lineage root. A pass run in one lineage must not evict, or
|
||||||
|
# even see, a sibling lineage's caches or the flat layout's.
|
||||||
|
# 4. BASE SEEDING IS PER LINEAGE — a PR branch layers over ITS OWN lineage's
|
||||||
|
# base snapshot, not over whatever the flat root happens to hold. This is
|
||||||
|
# the property that keeps a warm start for both jobs rather than one.
|
||||||
|
# 5. A NAME NO READER CAN HANDLE IS REFUSED AT RESOLVE TIME — one assertion
|
||||||
|
# per constraint, each named for the reader that imposes it. None of these
|
||||||
|
# is a filesystem limit: every one of them produces a working directory
|
||||||
|
# that some pass silently stops seeing, which is the failure mode this
|
||||||
|
# whole scheme exists to avoid rather than to relocate.
|
||||||
|
# 6. A PUBLISH THAT DISAGREES WITH ITS CONSUME STEP FAILS LOUDLY — the
|
||||||
|
# footgun the lineage input introduces. cargo-cache-publish derives both
|
||||||
|
# ends of the snapshot swap from its own `cache-root`, so a publish step
|
||||||
|
# left at the default while its consume step nested would republish the
|
||||||
|
# OTHER lineage's live target dir over that lineage's snapshot, silently.
|
||||||
|
set -euo pipefail
|
||||||
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
|
. "$script_dir/cache-lib.sh"
|
||||||
|
|
||||||
|
scratch=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$scratch"' EXIT
|
||||||
|
root="$scratch/cache"; mkdir -p "$root"
|
||||||
|
pass_count=0
|
||||||
|
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
||||||
|
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||||
|
|
||||||
|
resolve() { bash "$script_dir/cache-root.sh" resolve "$@"; }
|
||||||
|
|
||||||
|
# A tree that looks enough like a Cargo target dir for the pipeline scripts,
|
||||||
|
# with a marker naming which lineage produced it — scenario 4 turns on reading
|
||||||
|
# that marker back out of a seeded directory.
|
||||||
|
make_tree() {
|
||||||
|
local d="$1" marker="$2"
|
||||||
|
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x"
|
||||||
|
echo "$marker" > "$d/debug/deps/libx.rlib"
|
||||||
|
echo "$marker" > "$d/lineage-marker"
|
||||||
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "=== 1. no lineage changes nothing ==="
|
||||||
|
|
||||||
|
[ "$(resolve /cache)" = /cache ] || fail "an omitted lineage changed the root"
|
||||||
|
[ "$(resolve /cache '')" = /cache ] || fail "an empty lineage changed the root"
|
||||||
|
ok "no lineage resolves to the cache root unchanged"
|
||||||
|
|
||||||
|
KEY=$(cache_key dev)
|
||||||
|
[ "$(target_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/target-$KEY" ] \
|
||||||
|
|| fail "the flat target-dir name moved"
|
||||||
|
[ "$(snapshot_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/snapshot-$KEY" ] \
|
||||||
|
|| fail "the flat snapshot name moved"
|
||||||
|
ok "the flat layout's directory names are untouched"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 2. two lineages, one ref, two target dirs ==="
|
||||||
|
|
||||||
|
HOST_ROOT=$(resolve /cache)
|
||||||
|
WASM_ROOT=$(resolve /cache wasm32)
|
||||||
|
[ "$WASM_ROOT" = /cache/wasm32 ] || fail "lineage root resolved to '$WASM_ROOT'"
|
||||||
|
HOST_DIR=$(target_dir_for "$HOST_ROOT" "$KEY")
|
||||||
|
WASM_DIR=$(target_dir_for "$WASM_ROOT" "$KEY")
|
||||||
|
[ "$HOST_DIR" != "$WASM_DIR" ] || fail "both lineages resolved to $HOST_DIR"
|
||||||
|
case "$WASM_DIR" in "$HOST_DIR"/*) fail "the wasm target dir sits inside the host one" ;; esac
|
||||||
|
case "$HOST_DIR" in "$WASM_DIR"/*) fail "the host target dir sits inside the wasm one" ;; esac
|
||||||
|
ok "one cache key ($KEY), two disjoint target dirs: $HOST_DIR and $WASM_DIR"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 3. the whole pipeline moves together ==="
|
||||||
|
|
||||||
|
DEAD=$(cache_key feat/dead)
|
||||||
|
lin="$root/wasm32"
|
||||||
|
mkdir -p "$lin"
|
||||||
|
make_tree "$root/target-$DEAD" flat
|
||||||
|
make_tree "$root/native/target-$DEAD" native
|
||||||
|
make_tree "$lin/target-$DEAD" wasm32
|
||||||
|
bash "$script_dir/seed-target-dir.sh" "$KEY" "" "$lin" tag-3 > "$scratch/seed3.log" 2>&1 \
|
||||||
|
|| { cat "$scratch/seed3.log"; fail "seed inside a lineage root failed"; }
|
||||||
|
[ -d "$lin/target-$KEY" ] || fail "seed did not create $lin/target-$KEY"
|
||||||
|
[ -d "$root/target-$KEY" ] && fail "seed created a directory at the flat root as well"
|
||||||
|
ok "seed creates its directory under the lineage root and nowhere else"
|
||||||
|
|
||||||
|
make_tree "$lin/target-$KEY" wasm32
|
||||||
|
bash "$script_dir/publish-snapshot.sh" "$KEY" "$lin" tag-3 > "$scratch/pub3.log" 2>&1 \
|
||||||
|
|| { cat "$scratch/pub3.log"; fail "publish inside a lineage root failed"; }
|
||||||
|
[ -d "$lin/snapshot-$KEY" ] || fail "publish did not create $lin/snapshot-$KEY"
|
||||||
|
[ -d "$root/snapshot-$KEY" ] && fail "publish created a snapshot at the flat root as well"
|
||||||
|
ok "publish writes its snapshot under the lineage root and nowhere else"
|
||||||
|
|
||||||
|
# Free space far below the threshold, so pass 2 evicts every eligible
|
||||||
|
# directory it can see. What it can see is the point of the scenario.
|
||||||
|
CACHE_LIVENESS=false CACHE_DF_OVERRIDE="1000000 1000" \
|
||||||
|
bash "$script_dir/prune-cache.sh" "$lin" "$lin/target-$KEY" 'dev main' 10 \
|
||||||
|
> "$scratch/prune3.log" 2>&1 || { cat "$scratch/prune3.log"; fail "prune inside a lineage root failed"; }
|
||||||
|
[ -d "$lin/target-$DEAD" ] && { cat "$scratch/prune3.log"; fail "prune left its own lineage's evictable cache in place"; }
|
||||||
|
[ -d "$root/target-$DEAD" ] || fail "prune reached out of its lineage and evicted the flat root's cache"
|
||||||
|
[ -d "$root/native/target-$DEAD" ] || fail "prune reached into a sibling lineage and evicted its cache"
|
||||||
|
ok "prune under disk pressure evicts inside its own lineage only"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 4. base seeding is per lineage ==="
|
||||||
|
|
||||||
|
BASE=$(cache_key dev)
|
||||||
|
PR=$(cache_key feat/pr)
|
||||||
|
rm -rf "$lin" "$root/snapshot-$BASE"
|
||||||
|
mkdir -p "$lin"
|
||||||
|
make_tree "$root/snapshot-$BASE" flat-base
|
||||||
|
make_tree "$lin/snapshot-$BASE" wasm32-base
|
||||||
|
bash "$script_dir/seed-target-dir.sh" "$PR" "$BASE" "$lin" tag-4 > "$scratch/seed4.log" 2>&1 \
|
||||||
|
|| { cat "$scratch/seed4.log"; fail "seeding a PR branch inside a lineage failed"; }
|
||||||
|
[ -d "$lin/target-$PR" ] || fail "the PR branch's lineage target dir was not created"
|
||||||
|
got=$(cat "$lin/target-$PR/lineage-marker")
|
||||||
|
[ "$got" = wasm32-base ] || fail "the PR branch layered over '$got', not its own lineage's base snapshot"
|
||||||
|
grep -q 'base snapshot' "$scratch/seed4.log" || { cat "$scratch/seed4.log"; fail "seed did not report a base-snapshot clone"; }
|
||||||
|
ok "a PR branch layers over its own lineage's base snapshot ($got)"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 5. a name no reader can handle is refused ==="
|
||||||
|
|
||||||
|
reject() {
|
||||||
|
local lineage="$1" want="$2" desc="$3" out
|
||||||
|
if out=$(resolve /cache "$lineage" 2>&1); then
|
||||||
|
fail "lineage '$lineage' was accepted (resolved to '$out') — $desc"
|
||||||
|
fi
|
||||||
|
case "$out" in
|
||||||
|
*"$want"*) ;;
|
||||||
|
*) fail "lineage '$lineage' was rejected without naming '$want': $out" ;;
|
||||||
|
esac
|
||||||
|
ok "rejected '$lineage' — $desc"
|
||||||
|
}
|
||||||
|
|
||||||
|
reject 'a/b' 'single path component' "the arbiter walks a volume to depth 2"
|
||||||
|
reject '.hidden' 'dot' "every dot-prefixed name under a cache root is leftover-contract territory"
|
||||||
|
reject 'target-x' 'prune-cache.sh' "prune-cache.sh globs target-* at the cache root"
|
||||||
|
reject 'snapshot-x' 'prune-cache.sh' "prune-cache.sh globs snapshot-* at the cache root"
|
||||||
|
reject 'wasm 32' 'A-Za-z0-9._-' "a cache key is sanitised to that charset and a lineage sits beside one"
|
||||||
|
reject 'release' 'CI_CACHE_NODESCEND_NAMES' "the arbiter never descends into a Cargo profile name"
|
||||||
|
reject 'doc' 'CI_CACHE_NODESCEND_NAMES' "same, for the docs profile directory"
|
||||||
|
reject 'lineage-deadbeef' 'per-branch cache directory' "the arbiter reads a hex-suffixed name as one cache dir"
|
||||||
|
|
||||||
|
for good in wasm32 web android host wasm32.release lineage_2; do
|
||||||
|
out=$(resolve /cache "$good") || fail "lineage '$good' was rejected: $out"
|
||||||
|
[ "$out" = "/cache/$good" ] || fail "lineage '$good' resolved to '$out'"
|
||||||
|
done
|
||||||
|
ok "ordinary lineage names still resolve"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 6. a publish that disagrees with its consume step fails loudly ==="
|
||||||
|
|
||||||
|
verify() { bash "$script_dir/cache-root.sh" verify "$@"; }
|
||||||
|
|
||||||
|
verify /cache wasm32 /cache/wasm32 > /dev/null 2>&1 \
|
||||||
|
|| fail "verify rejected a publish step that agrees with its consume step"
|
||||||
|
verify /cache '' /cache > /dev/null 2>&1 \
|
||||||
|
|| fail "verify rejected an unmigrated consumer's matching default pair"
|
||||||
|
ok "verify accepts a publish step whose inputs match what the consume step exported"
|
||||||
|
|
||||||
|
# The exact shape of the mistake: the consume step nested, the publish step
|
||||||
|
# kept the default. Left unchecked this republishes the host lineage's live
|
||||||
|
# target dir over the host lineage's snapshot, from the wasm job.
|
||||||
|
if out=$(verify /cache '' /cache/wasm32 2>&1); then
|
||||||
|
fail "verify accepted a publish step that resolved to /cache while the job exported /cache/wasm32"
|
||||||
|
fi
|
||||||
|
case "$out" in
|
||||||
|
*'/cache/wasm32'*) ;;
|
||||||
|
*) fail "the mismatch error does not quote what the consume step exported: $out" ;;
|
||||||
|
esac
|
||||||
|
case "$out" in
|
||||||
|
*'SAME cache-root and cache-lineage'*) ;;
|
||||||
|
*) fail "the mismatch error does not say what to do about it: $out" ;;
|
||||||
|
esac
|
||||||
|
ok "verify rejects a publish step that forgot the lineage, and says so"
|
||||||
|
|
||||||
|
if verify /cache 'a/b' /cache/a/b > /dev/null 2>&1; then
|
||||||
|
fail "verify accepted an invalid lineage as long as both sides agreed on it"
|
||||||
|
fi
|
||||||
|
ok "verify validates the lineage as well as comparing it"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "cache-root-selftest: ${pass_count} assertions passed"
|
||||||
Executable
+61
@@ -0,0 +1,61 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Resolves — and cross-checks — the cache root a job's directories live under.
|
||||||
|
#
|
||||||
|
# cache-root.sh resolve <cache-root> [lineage]
|
||||||
|
# cache-root.sh verify <cache-root> <lineage> <exported-root>
|
||||||
|
#
|
||||||
|
# `resolve` prints the effective root: the cache root unchanged when no lineage
|
||||||
|
# is given, or `<cache-root>/<lineage>` when one is. Invalid lineage names are
|
||||||
|
# rejected here rather than downstream — see validate_cache_lineage() in
|
||||||
|
# cache-lib.sh, where every rejection names the reader that imposes it.
|
||||||
|
#
|
||||||
|
# `verify` is the publish side's guard. cargo-cache-publish resolves the same
|
||||||
|
# two inputs the consume action was given and compares the result against the
|
||||||
|
# CARGO_CACHE_ROOT the consume step exported into the job environment. The two
|
||||||
|
# actions have always had to agree — `cache-root`'s description in the publish
|
||||||
|
# action says "must match the consume action" — and until a lineage existed
|
||||||
|
# they always did, because nobody overrode the default. A disagreement is not
|
||||||
|
# a harmless no-op: publish-snapshot.sh takes the root as an argument and
|
||||||
|
# derives BOTH ends of the swap from it, so a publish step that kept the
|
||||||
|
# default while its consume step nested would read `<root>/target-<key>` — the
|
||||||
|
# OTHER lineage's live target dir — and republish it over `<root>/snapshot-<key>`,
|
||||||
|
# which is that lineage's snapshot. Two jobs would then be publishing one
|
||||||
|
# snapshot from one tree on every push, and nothing in either action would say
|
||||||
|
# so. Hence: fail the job, loudly, rather than resolve the ambiguity in
|
||||||
|
# either direction.
|
||||||
|
#
|
||||||
|
# A thin CLI over cache-lib.sh, kept as its own entry point for the same
|
||||||
|
# reason branch-cache-key.sh is: an out-of-band job that needs to find a
|
||||||
|
# lineage's directories should resolve the path the way the action does
|
||||||
|
# instead of reimplementing the rule.
|
||||||
|
set -euo pipefail
|
||||||
|
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||||
|
|
||||||
|
MODE="${1:-}"
|
||||||
|
case "$MODE" in
|
||||||
|
resolve)
|
||||||
|
[ $# -ge 2 ] && [ $# -le 3 ] || {
|
||||||
|
echo "::error::cache-root.sh resolve: expected <cache-root> [lineage]" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
[ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; }
|
||||||
|
cache_root_for "$2" "${3:-}"
|
||||||
|
;;
|
||||||
|
verify)
|
||||||
|
[ $# -eq 4 ] || {
|
||||||
|
echo "::error::cache-root.sh verify: expected <cache-root> <lineage> <exported-root>" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
[ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; }
|
||||||
|
expected=$(cache_root_for "$2" "$3")
|
||||||
|
if [ "$expected" != "$4" ]; then
|
||||||
|
echo "::error::cache-root.sh: this step resolves its cache root to '${expected}' (cache-root '$2', cache-lineage '$3') but the cargo-cache step in this job exported '$4'. Pass the SAME cache-root and cache-lineage to both actions." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "cache root: ${expected} (agrees with the cargo-cache step in this job)"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "::error::cache-root.sh: unknown mode '${MODE}' (expected resolve or verify)" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
@@ -5,8 +5,10 @@
|
|||||||
#
|
#
|
||||||
# That assumption is FALSE for a plain `cp -al`. Measured, and asserted below
|
# That assumption is FALSE for a plain `cp -al`. Measured, and asserted below
|
||||||
# as an explicit control: build in a raw `cp -al` clone and the source's
|
# as an explicit control: build in a raw `cp -al` clone and the source's
|
||||||
# `.fingerprint/<unit>/dep-*` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS),
|
# dep-info file (`.fingerprint/<unit>/dep-*` under Cargo's build-dir layout
|
||||||
# `build/<pkg>/output`, `build/<pkg>/out/**` and `deps/*.d` all change,
|
# v1, `build/<pkg>/<hash>/fingerprint/dep-*` under v2 — and under content
|
||||||
|
# freshness only, see the probe below), `build/<pkg>/output`,
|
||||||
|
# `build/<pkg>/out/**` and `deps/*.d` all change,
|
||||||
# because Cargo and build scripts write those with a plain truncating write
|
# because Cargo and build scripts write those with a plain truncating write
|
||||||
# rather than the write-then-rename Cargo uses for real artifacts.
|
# rather than the write-then-rename Cargo uses for real artifacts.
|
||||||
#
|
#
|
||||||
@@ -73,22 +75,173 @@ mkcrate "$crate_dir"
|
|||||||
cd "$crate_dir"
|
cd "$crate_dir"
|
||||||
|
|
||||||
export CARGO_INCREMENTAL=0
|
export CARGO_INCREMENTAL=0
|
||||||
|
|
||||||
|
# Every assertion below reads cargo's own words out of a build log
|
||||||
|
# (`Compiling libdep`, `Fresh probe`). A CI image that forces colour splices an
|
||||||
|
# ANSI reset between the status word and the crate name, at which point every
|
||||||
|
# one of those greps silently stops matching and the suite reports the
|
||||||
|
# opposite of what happened — observed on gitdan-ci's runner image, where
|
||||||
|
# scenario 2 failed while the log it printed plainly showed `Compiling libdep`.
|
||||||
|
# Pin the format the assertions are written against.
|
||||||
|
export CARGO_TERM_COLOR=never
|
||||||
# Checksum freshness is where the worst failure lives (the dep-* file carries
|
# Checksum freshness is where the worst failure lives (the dep-* file carries
|
||||||
# per-source checksums and is rewritten in place). Only available on nightly;
|
# per-source checksums and is rewritten in place). Only available on nightly;
|
||||||
# without it the test still covers the build/ and *.d families.
|
# without it the test still covers the build/ and *.d families.
|
||||||
CHECKSUM_MODE="off"
|
|
||||||
if cargo +nightly -V >/dev/null 2>&1; then
|
|
||||||
export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true
|
|
||||||
CARGO_BIN=(cargo +nightly)
|
|
||||||
CHECKSUM_MODE="on"
|
|
||||||
else
|
|
||||||
CARGO_BIN=(cargo)
|
|
||||||
fi
|
|
||||||
echo "=== checksum-freshness mode: ${CHECKSUM_MODE} ==="
|
|
||||||
|
|
||||||
CONTENT_A='pub fn f() -> u32 { 1 }'
|
CONTENT_A='pub fn f() -> u32 { 1 }'
|
||||||
CONTENT_B='pub fn f() -> u32 { 22222 } pub fn g() -> u32 { 7 }'
|
CONTENT_B='pub fn f() -> u32 { 22222 } pub fn g() -> u32 { 7 }'
|
||||||
|
|
||||||
|
# Probe the BEHAVIOUR, not the channel and not the flag. Two weaker probes
|
||||||
|
# were tried against gitdan-ci's runner and each let the suite assert a
|
||||||
|
# property the toolchain did not have:
|
||||||
|
#
|
||||||
|
# `cargo +nightly -V` — answers "did a proxy called with
|
||||||
|
# +nightly exit 0". `-V`
|
||||||
|
# short-circuits before `-Z` is
|
||||||
|
# even parsed.
|
||||||
|
# `cargo +nightly -Z checksum-freshness — answers "is this flag still
|
||||||
|
# locate-project` accepted", which since cargo PR
|
||||||
|
# #17382 (2026-08-22) is a
|
||||||
|
# different question from "is
|
||||||
|
# content freshness on". That PR
|
||||||
|
# demoted the flag to a gate and
|
||||||
|
# gave `build.fingerprint` the
|
||||||
|
# choice, defaulting to `mtime` —
|
||||||
|
# so 1.100.0-nightly accepts the
|
||||||
|
# flag and resolves freshness by
|
||||||
|
# mtime unless
|
||||||
|
# CARGO_BUILD_FINGERPRINT=content
|
||||||
|
# is set too. Measured 2026-08-26;
|
||||||
|
# see daniel/gitdan#62.
|
||||||
|
#
|
||||||
|
# The scenario at the end of this file depends on one thing and it is neither
|
||||||
|
# of those: that changed content with an OLDER mtime rebuilds. Under mtime
|
||||||
|
# freshness the correct answer is Fresh, so under mtime freshness that
|
||||||
|
# scenario asserts a bug. So the probe simply performs that experiment, on its
|
||||||
|
# own crate and its own target dir, with no clone anywhere near it — which is
|
||||||
|
# also what makes it a control rather than a restatement of the scenario: the
|
||||||
|
# probe establishes that the toolchain rebuilds on content, the scenario
|
||||||
|
# establishes that a hardlink clone did not take that away.
|
||||||
|
# THREE OUTCOMES, NOT TWO. An experiment that cannot tell a negative result
|
||||||
|
# from a failed measurement is not settling the question, and the two are not
|
||||||
|
# interchangeable here: "this toolchain resolves freshness by mtime" is a
|
||||||
|
# statement about Cargo, while "a probe build failed" is a statement about this
|
||||||
|
# machine. Collapsing them — which an earlier cut of this did, by returning
|
||||||
|
# non-zero for both — makes a half-installed toolchain print a confident and
|
||||||
|
# wrong explanation and quietly drop a scenario. The scenario still has to be
|
||||||
|
# skipped in either case; what must not happen is the log claiming to know why.
|
||||||
|
#
|
||||||
|
# 0 content freshness measured ACTIVE — both builds ran, the backdated
|
||||||
|
# rebuild recompiled
|
||||||
|
# 3 measured INACTIVE — both builds ran, the backdated
|
||||||
|
# rebuild reported Fresh
|
||||||
|
# anything else NOT MEASURED — nothing was learned about the
|
||||||
|
# toolchain
|
||||||
|
#
|
||||||
|
# THE ANSWER CODES ARE 0 AND 3, AND THE GAP IS THE MECHANISM. Bash produces 1
|
||||||
|
# for an ordinary command failure, 2 for a usage error, 126/127 for a command
|
||||||
|
# it could not run, 128+n for a signal, and — this is the one that matters —
|
||||||
|
# 1 for an unbound-variable or other EXPANSION failure, which happens before
|
||||||
|
# the command runs and is therefore invisible to a `||` guard and to an ERR
|
||||||
|
# trap alike. It never produces 3. So "not an answer code" is decided by a
|
||||||
|
# property of the shell rather than by an enumeration of the ways a step can
|
||||||
|
# go wrong, and a step added later without a guard, or with a guard that
|
||||||
|
# cannot fire, lands on NOT MEASURED by construction.
|
||||||
|
#
|
||||||
|
# That is the whole reason INACTIVE is not 1. It was, and three review rounds
|
||||||
|
# on this function each found a narrower way for a shell-generated 1 to be read
|
||||||
|
# as a measurement — an unguarded command, then a typo'd variable name on a
|
||||||
|
# line that HAS its guard. Each was closed by narrowing the failure surface,
|
||||||
|
# which is a game with no last move. Moving the answer off the codes bash can
|
||||||
|
# generate ends it instead: there is no longer a mutation that turns an error
|
||||||
|
# into an answer, only mutations that turn an error into a different error.
|
||||||
|
#
|
||||||
|
# The guards below stay, and so does the trap, but their job is now reporting
|
||||||
|
# rather than correctness: they make a failed step land on 2 with its logs
|
||||||
|
# printed instead of on some incidental status, which is nicer to debug and
|
||||||
|
# lands in the same place either way.
|
||||||
|
#
|
||||||
|
# One piece of that reporting layer is load-bearing and not obvious. A command
|
||||||
|
# on the left of `||` — or in an `if` condition — runs with errexit suppressed,
|
||||||
|
# and that suppression propagates into a subshell and is NOT undone by a
|
||||||
|
# `set -e` inside it (measured on bash 5.3: an unguarded `false` there falls
|
||||||
|
# through to `exit 0`). Calling with errexit disarmed at the site is the only
|
||||||
|
# form that lets the subshell re-arm it; hence the `set +e` bracket. The ERR
|
||||||
|
# trap is then required on top, because a bare `set -e` abort exits with the
|
||||||
|
# FAILING COMMAND's status, and `false` gives 1.
|
||||||
|
#
|
||||||
|
# WHY NOT-MEASURED SKIPS RATHER THAN FAILS. The scenario it gates is the only thing in
|
||||||
|
# this suite that depends on freshness mode; everything else still runs and
|
||||||
|
# still catches real regressions. Failing instead would turn a statement about
|
||||||
|
# one machine's toolchain into a red gate reading "the hardlink scheme is
|
||||||
|
# broken" across the three repos consuming this action — the same category
|
||||||
|
# error the three-state split exists to prevent, one level up. What would
|
||||||
|
# change the answer is not-measured becoming the everyday CI outcome; it is not
|
||||||
|
# — gitdan-ci's outcome is a measurement either way. It reported a measured
|
||||||
|
# INACTIVE until 2026-08-26, for the reason recorded at the `export` below, and
|
||||||
|
# an ACTIVE once both switches were set.
|
||||||
|
CHECKSUM_MODE="off"
|
||||||
|
CHECKSUM_REASON="no nightly on PATH accepting -Z checksum-freshness"
|
||||||
|
CARGO_BIN=(cargo)
|
||||||
|
checksum_freshness_probe() {
|
||||||
|
local d="$scratch/freshness-probe" t="$scratch/freshness-probe-target"
|
||||||
|
mkcrate "$d" || return 2 # 2 is simply "not 0 and not 3"; see the header
|
||||||
|
(
|
||||||
|
set -e
|
||||||
|
trap 'exit 2' ERR
|
||||||
|
cd "$d" || exit 2
|
||||||
|
printf '%s\n' "$CONTENT_A" > src/lib.rs || exit 2
|
||||||
|
CARGO_TARGET_DIR="$t" cargo +nightly build -q > "$scratch/freshness-probe-warm.log" 2>&1 || exit 2
|
||||||
|
printf '%s\n' "$CONTENT_B" > src/lib.rs || exit 2
|
||||||
|
touch -d '@1000000000' src/lib.rs || exit 2
|
||||||
|
CARGO_TARGET_DIR="$t" cargo +nightly build -v > "$scratch/freshness-probe.log" 2>&1 || exit 2
|
||||||
|
# 3, not 1: see the header. This is the only statement in the subshell that
|
||||||
|
# may report a measurement, and it is the only one that may exit 3.
|
||||||
|
if grep -qE '^\s+Fresh probe' "$scratch/freshness-probe.log"; then exit 3; fi
|
||||||
|
exit 0
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if cargo +nightly -Z checksum-freshness locate-project > /dev/null 2>&1; then
|
||||||
|
export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true
|
||||||
|
# BOTH, since cargo PR #17382 (2026-08-22): the -Z flag only unlocks the
|
||||||
|
# feature and `build.fingerprint` selects it, defaulting to `mtime`. Setting
|
||||||
|
# the gate alone is what made this suite report a measured INACTIVE on
|
||||||
|
# 1.100.0-nightly and skip its strongest scenario (daniel/gitdan#62). Safe to
|
||||||
|
# export unconditionally — a Cargo that does not know the key ignores it
|
||||||
|
# silently, verified 2026-08-26 on 1.93.1 stable and 1.96.0-nightly, both of
|
||||||
|
# which still measure ACTIVE from the gate alone.
|
||||||
|
export CARGO_BUILD_FINGERPRINT=content
|
||||||
|
# Errexit off across the call, so the subshell can arm its own — see the
|
||||||
|
# header. `probe_rc` is read before it is restored.
|
||||||
|
probe_rc=0
|
||||||
|
set +e
|
||||||
|
checksum_freshness_probe
|
||||||
|
probe_rc=$?
|
||||||
|
set -e
|
||||||
|
case "$probe_rc" in
|
||||||
|
0)
|
||||||
|
CARGO_BIN=(cargo +nightly)
|
||||||
|
CHECKSUM_MODE="on"
|
||||||
|
CHECKSUM_REASON=""
|
||||||
|
;;
|
||||||
|
3)
|
||||||
|
unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS CARGO_BUILD_FINGERPRINT
|
||||||
|
CHECKSUM_REASON="this nightly accepts -Z checksum-freshness and build.fingerprint=content but still resolves freshness by mtime"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
unset CARGO_UNSTABLE_CHECKSUM_FRESHNESS CARGO_BUILD_FINGERPRINT
|
||||||
|
CHECKSUM_MODE="unmeasured"
|
||||||
|
CHECKSUM_REASON="the probe exited ${probe_rc}, which is not one of its answer codes, so this was NOT MEASURED — this toolchain may or may not resolve freshness by content"
|
||||||
|
# Loud, because the cost is silently lost coverage on a machine that
|
||||||
|
# might have had it. The suite continues: everything else it asserts is
|
||||||
|
# independent of freshness mode.
|
||||||
|
echo "::warning::hardlink-clone-selftest: could not measure whether this toolchain resolves freshness by content — the probe exited ${probe_rc}. This is a failure to measure, not a finding about Cargo."
|
||||||
|
tail -n 15 "$scratch/freshness-probe-warm.log" "$scratch/freshness-probe.log" 2>/dev/null | sed 's/^/ /' >&2 || true
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
cd "$crate_dir"
|
||||||
|
echo "=== checksum-freshness mode: ${CHECKSUM_MODE}${CHECKSUM_REASON:+ — ${CHECKSUM_REASON}} ==="
|
||||||
|
|
||||||
build_base() {
|
build_base() {
|
||||||
local dir="$1"
|
local dir="$1"
|
||||||
printf '%s\n' "$CONTENT_A" > src/lib.rs
|
printf '%s\n' "$CONTENT_A" > src/lib.rs
|
||||||
@@ -112,11 +265,35 @@ fi
|
|||||||
ok "raw cp -al clone mutates the source ($(printf '%s\n' "$ctl_mutated" | wc -l) paths)"
|
ok "raw cp -al clone mutates the source ($(printf '%s\n' "$ctl_mutated" | wc -l) paths)"
|
||||||
printf '%s\n' "$ctl_mutated" | sed 's/^/ /'
|
printf '%s\n' "$ctl_mutated" | sed 's/^/ /'
|
||||||
|
|
||||||
|
# Reported, not asserted, and the distinction is the point. The control's job
|
||||||
|
# is to prove the hazard exists at all, which the non-empty set above already
|
||||||
|
# does; this line records WHICH families a given Cargo exhibits.
|
||||||
|
#
|
||||||
|
# The dep-info file is the worst of them — it carries the per-source
|
||||||
|
# checksums, so mutating it through a shared inode turns a hardlink clone into
|
||||||
|
# silent stale-artifact reuse rather than a slow build. Failing on its absence
|
||||||
|
# would mean this suite goes red whenever upstream stops doing something we
|
||||||
|
# never wanted it to do — and it would go red in the CONTROL, where a failure
|
||||||
|
# reads as "the hazard is gone" rather than "upstream changed". Nothing is lost
|
||||||
|
# by reporting it: the fix scenario below asserts the source is byte-identical
|
||||||
|
# after a full rebuild in the clone, which covers every family this Cargo has,
|
||||||
|
# named or not.
|
||||||
|
#
|
||||||
|
# THE PATTERN MUST MATCH BOTH LAYOUTS, and that is not a detail. Cargo's
|
||||||
|
# build-dir layout v2 moved the file from `<profile>/.fingerprint/<unit>/dep-*`
|
||||||
|
# to `<profile>/build/<pkg>/<hash>/fingerprint/dep-*` (stabilised by cargo PR
|
||||||
|
# #17354, cargo 1.100.0, stable 2026-11-12; nightly default since 1.99). An
|
||||||
|
# earlier cut of this line looked for the v1 path only, so on 2026-08-26,
|
||||||
|
# against 1.100.0-nightly with content freshness genuinely on, it printed
|
||||||
|
# "does NOT rewrite ... in place" directly beneath a control listing that
|
||||||
|
# showed the rewrite. A reporting line that can contradict the data three
|
||||||
|
# lines above it is worse than no line at all. `fingerprint/.*dep-` matches
|
||||||
|
# either layout and neither `.d` family.
|
||||||
if [ "$CHECKSUM_MODE" = "on" ]; then
|
if [ "$CHECKSUM_MODE" = "on" ]; then
|
||||||
if printf '%s' "$ctl_mutated" | grep -q '\.fingerprint/.*/dep-'; then
|
if printf '%s' "$ctl_mutated" | grep -q 'fingerprint/.*dep-'; then
|
||||||
ok "control confirms the checksum-freshness dep-info file is among the mutated set"
|
echo " note: this cargo DOES rewrite its dep-info fingerprint file in place under content freshness"
|
||||||
else
|
else
|
||||||
fail "expected .fingerprint/*/dep-* in the control's mutated set under checksum freshness"
|
echo " note: this cargo does NOT rewrite its dep-info fingerprint file in place; only the build/ and *.d families appear above"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -133,7 +310,7 @@ hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_
|
|||||||
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
||||||
shared=0; unshared=0
|
shared=0; unshared=0
|
||||||
while IFS= read -r f; do
|
while IFS= read -r f; do
|
||||||
rel="${f#$base_fix/}"
|
rel="${f#"$base_fix"/}"
|
||||||
[ -e "$clone_fix/$rel" ] || continue
|
[ -e "$clone_fix/$rel" ] || continue
|
||||||
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
||||||
case "$rel" in
|
case "$rel" in
|
||||||
@@ -161,18 +338,32 @@ fi
|
|||||||
ok "no file in the source changed after a full rebuild in the clone"
|
ok "no file in the source changed after a full rebuild in the clone"
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "=== the whole point: the source's next build is still correct ==="
|
if [ "$CHECKSUM_MODE" = "on" ]; then
|
||||||
# The source's cache holds artifacts built from CONTENT_A. Advance the source
|
echo "=== the whole point: the source's next build is still correct ==="
|
||||||
# to CONTENT_B (as a merge would) and rebuild in it. If the clone had
|
# The source's cache holds artifacts built from CONTENT_A. Advance the
|
||||||
# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib.
|
# source to CONTENT_B (as a merge would) and rebuild in it. If the clone had
|
||||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib.
|
||||||
touch -d '@1000000000' src/lib.rs
|
#
|
||||||
log="$scratch/rebuild.log"
|
# CHECKSUM-FRESHNESS ONLY, and the backdated mtime is why. Under checksum
|
||||||
CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; }
|
# freshness the dep-info file's per-source checksums decide, so a 2001
|
||||||
if grep -qE '^\s+Fresh probe' "$log"; then
|
# timestamp on changed content must still rebuild — the assertion below.
|
||||||
fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse"
|
# Under Cargo's ordinary MTIME freshness the same timestamp means the source
|
||||||
|
# is older than the artifact, and reporting Fresh is the correct answer;
|
||||||
|
# asserting otherwise asserts a bug. This scenario was written against a
|
||||||
|
# machine with a nightly installed and, run without one, failed on that
|
||||||
|
# correct answer.
|
||||||
|
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||||
|
touch -d '@1000000000' src/lib.rs
|
||||||
|
log="$scratch/rebuild.log"
|
||||||
|
CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; }
|
||||||
|
if grep -qE '^\s+Fresh probe' "$log"; then
|
||||||
|
fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse"
|
||||||
|
fi
|
||||||
|
ok "source correctly rebuilt its crate after advancing to the clone's content"
|
||||||
|
else
|
||||||
|
echo "=== skipped: the source's-next-build scenario needs content-based freshness ==="
|
||||||
|
echo " reason: ${CHECKSUM_REASON}"
|
||||||
fi
|
fi
|
||||||
ok "source correctly rebuilt its crate after advancing to the clone's content"
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
||||||
|
|||||||
@@ -65,10 +65,10 @@ origin="$scratch/origin.git"; git init -q --bare "$origin"
|
|||||||
work="$scratch/work"; git init -q "$work"
|
work="$scratch/work"; git init -q "$work"
|
||||||
(
|
(
|
||||||
cd "$work"
|
cd "$work"
|
||||||
git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init
|
git -c user.email=t@t -c user.name=t -c commit.gpgsign=false commit -q --allow-empty -m init
|
||||||
git branch -M main
|
git branch -M main
|
||||||
git checkout -q -b dev; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m dev
|
git checkout -q -b dev; git -c user.email=t@t -c user.name=t -c commit.gpgsign=false commit -q --allow-empty -m dev
|
||||||
git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m live
|
git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t -c commit.gpgsign=false commit -q --allow-empty -m live
|
||||||
git remote add origin "$origin"
|
git remote add origin "$origin"
|
||||||
git push -q origin main dev feat/live
|
git push -q origin main dev feat/live
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -75,6 +75,15 @@
|
|||||||
# Asserts BOTH jobs correctly recompile the dependency and succeed.
|
# Asserts BOTH jobs correctly recompile the dependency and succeed.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Every assertion below reads cargo's own words out of a build log
|
||||||
|
# (`Compiling libdep`, `Fresh probe`). A CI image that forces colour splices an
|
||||||
|
# ANSI reset between the status word and the crate name, at which point every
|
||||||
|
# one of those greps silently stops matching and the suite reports the
|
||||||
|
# opposite of what happened — observed on gitdan-ci's runner image, where
|
||||||
|
# scenario 2 failed while the log it printed plainly showed `Compiling libdep`.
|
||||||
|
# Pin the format the assertions are written against.
|
||||||
|
export CARGO_TERM_COLOR=never
|
||||||
|
|
||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
restore_mtimes="$script_dir/restore-mtimes.sh"
|
restore_mtimes="$script_dir/restore-mtimes.sh"
|
||||||
|
|
||||||
@@ -122,7 +131,7 @@ assert_log_lacks() {
|
|||||||
# every run — simulate that before each restore-mtimes.sh pass, exactly as
|
# every run — simulate that before each restore-mtimes.sh pass, exactly as
|
||||||
# CI would see it, so this test exercises the script the same way CI does.
|
# CI would see it, so this test exercises the script the same way CI does.
|
||||||
stamp_checkout_now() {
|
stamp_checkout_now() {
|
||||||
find . -path ./.git -prune -o -type f -print | xargs touch
|
find . -path ./.git -prune -o -type f -print0 | xargs -0 touch
|
||||||
}
|
}
|
||||||
|
|
||||||
echo "=== building scratch workspace ==="
|
echo "=== building scratch workspace ==="
|
||||||
@@ -131,6 +140,11 @@ cd "$repo"
|
|||||||
git init -q
|
git init -q
|
||||||
git config user.email test@example.com
|
git config user.email test@example.com
|
||||||
git config user.name "restore-mtimes-selftest"
|
git config user.name "restore-mtimes-selftest"
|
||||||
|
# Local to this mktemp'd throwaway repo. Without it the eight commits below
|
||||||
|
# inherit the developer's GLOBAL commit.gpgsign, which makes whether this gate
|
||||||
|
# passes depend on their gpg agent — observed as a red run caused by a full
|
||||||
|
# disk breaking gpg, in a suite that has nothing to say about either.
|
||||||
|
git config commit.gpgsign false
|
||||||
|
|
||||||
cat > Cargo.toml <<'EOF'
|
cat > Cargo.toml <<'EOF'
|
||||||
[workspace]
|
[workspace]
|
||||||
|
|||||||
@@ -64,7 +64,8 @@
|
|||||||
# two refs' fingerprints ever share a directory and this script never has to
|
# two refs' fingerprints ever share a directory and this script never has to
|
||||||
# arbitrate freshness across refs — only within one ref's own history, which
|
# arbitrate freshness across refs — only within one ref's own history, which
|
||||||
# is exactly what it is built to do soundly. On a nightly toolchain,
|
# is exactly what it is built to do soundly. On a nightly toolchain,
|
||||||
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS is a complementary, stronger guarantee
|
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS plus CARGO_BUILD_FINGERPRINT=content (both,
|
||||||
|
# since cargo PR #17382 on 2026-08-22) is a complementary, stronger guarantee
|
||||||
# (content-addressed rather than mtime-based freshness); this script is not
|
# (content-addressed rather than mtime-based freshness); this script is not
|
||||||
# made redundant by it, because directory-form `rerun-if-changed` build-script
|
# made redundant by it, because directory-form `rerun-if-changed` build-script
|
||||||
# watches are not covered by it and stable historical mtimes stay cheap
|
# watches are not covered by it and stable historical mtimes stay cheap
|
||||||
|
|||||||
+1
-1
@@ -13,7 +13,7 @@ script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|||||||
FAST=0
|
FAST=0
|
||||||
[ "${1:-}" = "--fast" ] && FAST=1
|
[ "${1:-}" = "--fast" ] && FAST=1
|
||||||
|
|
||||||
FIXTURE_TESTS=(seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh)
|
FIXTURE_TESTS=(cache-root-selftest.sh seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh)
|
||||||
CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh)
|
CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh)
|
||||||
|
|
||||||
TESTS=("${FIXTURE_TESTS[@]}")
|
TESTS=("${FIXTURE_TESTS[@]}")
|
||||||
|
|||||||
Reference in New Issue
Block a user