This repo had eight scripts, six selftest suites and nothing that ran any of
them. It is a composite action consumed at `@v1` — a moving tag — by three
repos' CI, so a bad edit here reaches zemyna, emowheel and lublub at once and
is discovered by whichever of them builds next.
One job: `shellcheck -x --source-path=scripts scripts/*.sh`, then
`bash scripts/selftest.sh`. `-x` follows the `. cache-lib.sh` every script
sources, which is where most of the logic being checked lives; without it
shellcheck reports SC1091 and analyses each file with a hole in it.
The full suite, not `--fast`: `hardlink-clone-selftest.sh` and
`restore-mtimes-selftest.sh` are the two that drive a real Cargo rather than a
fixture, and selftest.sh's own header says `--fast` is for iterating, not for
signing off a change. So the job installs a stable toolchain. The scratch
workspaces they build use path dependencies only, so nothing reaches
crates.io, and the job references no credentials at all.
Modelled on daniel/gitdan's workflow, including the two clauses of the
draft-skip guard (`github.event_name != 'pull_request' ||
!github.event.pull_request.draft` — the first is what stops the second from
also skipping pushes to `main`, where there is no `pull_request` context) and
the per-commit concurrency group for `push`.
No `container.volumes:` entry, deliberately: the suites want a cold target dir
every run, since "did this rebuild?" is exactly what restore-mtimes-selftest
asks. This job takes no share of the shared CI cache disk budget.
Turning the gate on surfaced three pre-existing findings, all fixed here
rather than suppressed or waived:
- `restore-mtimes-selftest.sh` SC2038: `find | xargs touch` -> `-print0 | xargs -0`.
- `hardlink-clone-selftest.sh` SC2295: `${f#$base_fix/}` -> `${f#"$base_fix"/}`.
- `cache-lib.sh` SC2016: a per-line disable with the rationale. The single
quotes are load-bearing — the program is for the inner shell, where `$f` is
its loop variable, `$rc` its accumulator and `$$` its pid — so this is the
documented-false-positive case, not a stopgap.
554 lines
30 KiB
Markdown
554 lines
30 KiB
Markdown
# gitdan-actions
|
|
|
|
Shared Gitea Actions composite actions for the gitdan forge.
|
|
|
|
Currently one thing, done properly: **`cargo-cache`** — a persistent,
|
|
per-branch Cargo build cache for self-hosted Gitea runners, where a pull
|
|
request's cache is a near-free hardlink clone of an immutable snapshot its
|
|
base branch published.
|
|
|
|
> **Final home:** this repository will live at `daniel/gitdan-actions`. Pin
|
|
> that path in `uses:` once the transfer completes.
|
|
|
|
---
|
|
|
|
## Why this exists
|
|
|
|
Two of this forge's Rust projects independently built the same idea and each
|
|
got one half right.
|
|
|
|
| | seeding mechanism | seed source |
|
|
|---|---|---|
|
|
| project A | `cp -al` hardlink clone — near-free, cost scales with inode count, not bytes | the base branch's **live** target dir — races a build that is still writing |
|
|
| project B | `cp -a` full copy — sound, but ~35 GB duplicated per branch | a **published immutable snapshot** — nothing ever writes it while it is read |
|
|
|
|
This action is the diagonal: **hardlink-clone from a published snapshot.**
|
|
Cheap like A, sound like B. It also closes a latent race in A by construction
|
|
(the seed is staged and swapped in with one atomic rename) rather than relying
|
|
on the runner having a single execution slot.
|
|
|
|
One thing neither project had, and the reason the clone is not a plain
|
|
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
|
cloned from.** Cargo writes real artifacts by replacing them, but writes its
|
|
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
|
write, straight through the shared inode. Under
|
|
`CARGO_UNSTABLE_CHECKSUM_FRESHNESS` the file that gets corrupted is
|
|
`.fingerprint/<unit>/dep-<target>`, which holds the per-source checksums that
|
|
decide freshness, and the failure is silent stale-artifact reuse rather than a
|
|
slow build. `scripts/hardlink-clone-selftest.sh` reproduces it as an explicit
|
|
control and asserts the fix. The fix is to hardlink the artifacts (the GB) and
|
|
real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory,
|
|
against 100% for a full copy.
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
```yaml
|
|
name: CI
|
|
on:
|
|
push:
|
|
branches: [main, dev]
|
|
pull_request:
|
|
branches: [main, dev]
|
|
|
|
jobs:
|
|
ci:
|
|
runs-on: ubuntu-latest
|
|
# REQUIRED, and it cannot come from the action: `container.volumes` is a
|
|
# job-level property, so the persistent cache volume must be declared
|
|
# here. Use a volume name unique to this repository.
|
|
container:
|
|
volumes:
|
|
- myrepo-ci-target:/cache
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
# REQUIRED. The mtime restore walks every commit that ever touched a
|
|
# tracked file; a depth-1 checkout makes every file resolve to the
|
|
# tip commit and the cache stops working. The action fails loudly
|
|
# rather than silently degrading if this is missing.
|
|
fetch-depth: 0
|
|
|
|
- uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
|
|
with:
|
|
protected-branches: 'dev main'
|
|
|
|
# ... toolchain, system deps, and the build itself. CARGO_TARGET_DIR is
|
|
# already exported to the job environment by the step above.
|
|
- run: cargo clippy --workspace --all-targets -- -D warnings
|
|
- run: cargo test --workspace
|
|
|
|
# After the build succeeds: record the watermark, and publish a snapshot
|
|
# if this run is a push to a protected branch.
|
|
- uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
|
|
|
# Release this job's cache lock even when the build failed, so the
|
|
# eviction pass does not have to wait out the staleness grace period.
|
|
- if: always()
|
|
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
|
with:
|
|
mode: release-lock
|
|
```
|
|
|
|
Recommended alongside it, in the workflow's `env:` block:
|
|
|
|
```yaml
|
|
env:
|
|
CARGO_INCREMENTAL: 0 # per-run bloat on a persistent volume
|
|
CARGO_PROFILE_DEV_DEBUG: line-tables-only
|
|
CARGO_PROFILE_TEST_DEBUG: line-tables-only
|
|
# Nightly only. Content-addressed freshness instead of mtime-based — a
|
|
# strictly stronger guarantee, complementary to the mtime restore (which
|
|
# still covers directory-form `rerun-if-changed` build-script watches).
|
|
CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"
|
|
```
|
|
|
|
---
|
|
|
|
## How it works
|
|
|
|
```
|
|
/cache/
|
|
target-<key> one per ref. Where a build actually runs.
|
|
snapshot-<key> one per publisher ref. Immutable between publishes;
|
|
the only thing a consumer ever clones from.
|
|
```
|
|
|
|
Transient dot-prefixed entries appear alongside those two — staging trees,
|
|
eviction asides, reader markers. Their names are a contract with the host that
|
|
owns the volume; see [Scratch names in a cache root](#scratch-names-in-a-cache-root-are-a-cross-repo-contract).
|
|
|
|
`<key>` is the ref sanitised to a safe path component, capped at 48
|
|
characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not
|
|
decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise
|
|
share one directory.
|
|
|
|
**A pull request run** resolves its own key from `github.head_ref` (not
|
|
`ref_name`, which on a `pull_request` event is a synthetic merge ref that
|
|
changes on every push) and its base key from `github.base_ref`. If it has no
|
|
directory yet, it hardlink-clones `snapshot-<base>` into a staging path, strips
|
|
Cargo's lock files, real-copies everything Cargo writes in place, and renames
|
|
the staging path into `target-<own>`.
|
|
|
|
**A push to a protected branch** has no base to layer over. It builds in its
|
|
own directory and, if the build goes green, republishes it as
|
|
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
|
|
one in, then reclaim the old one once nothing is still reading it. Consumers
|
|
only ever observe a complete snapshot or none at all.
|
|
|
|
**Concurrency, on the destination.** Two jobs sharing one cache key each stage
|
|
under their own tag and race on one atomic rename; the loser discards its
|
|
staging copy. There is no window in which a partially-populated directory is
|
|
visible under the final name. Two jobs then building in the same directory is
|
|
Cargo's own `.cargo-lock` territory, which is what that lock is for.
|
|
|
|
**Concurrency, on the source.** The atomic rename is necessary and not
|
|
sufficient, because renaming a *truncated* tree publishes a truncated tree
|
|
atomically. A clone reads its source over many seconds, and a publisher
|
|
rotating that source unlinks the generation being read — at which point
|
|
`cp -al` can silently omit a subtree it never saw, and report success. Two
|
|
mechanisms, both required:
|
|
|
|
- **The publisher does not unlink under a reader.** A consumer publishes a
|
|
`.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
|
|
the publisher scans for markers *after* its first rename. A consumer holding
|
|
the old generation therefore published its marker before that scan and
|
|
cannot be missed, and one that arrives after the scan necessarily resolves
|
|
to the new generation. The publisher waits for readers to drain
|
|
(`read-grace-seconds`, default 300) and, if they do not, **defers** the
|
|
reclamation rather than forcing it — the old generation stays on disk and is
|
|
swept by a later publish.
|
|
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
|
|
exit status, the source directory's inode before and after (a wholesale
|
|
replacement mid-walk would otherwise splice two generations), and the entry
|
|
count (the only signal for a subtree unlinked before its parent was listed —
|
|
there is no error to read). A tree that fails any of the three is deleted
|
|
and the clone retried; one that fails the last attempt fails the job. A
|
|
partial tree never reaches the final name. The two counts cost one metadata
|
|
walk each: measured on ext4 with a warm cache over a 78,554-entry tree, 44 ms
|
|
per walk against 3,126 ms for the `cp -al` they guard — about 2.8%.
|
|
|
|
**What this does and does not guarantee.** Four separate claims, deliberately
|
|
not collapsed into one:
|
|
|
|
- **A publisher rotating a snapshot cannot tear a clone of it — by
|
|
construction.** This is the case zemyna #911 is about, and the marker
|
|
ordering above is what closes it: the publisher's scan cannot miss a
|
|
consumer that resolved the old generation, and on timeout it defers the
|
|
unlink rather than forcing it. On this path the consumer's own verification
|
|
is a redundant second check, not the thing holding the guarantee up.
|
|
- **Nor can the eviction pass — by the same construction.** A cache chosen for
|
|
eviction is renamed aside and only then re-examined for readers, so the scan
|
|
the unlink rests on happens strictly after that rename, exactly as the
|
|
publisher's does. A consumer that resolved the directory published its
|
|
marker before the scan and so cannot be missed; one arriving after the
|
|
rename cannot resolve the path at all and starts cold instead. A cache
|
|
claimed inside that window is put back under its own name, and one whose
|
|
name a concurrent seed has taken in the meantime is left aside and reclaimed
|
|
by a later pass once its readers drain. That later pass leaves an aside
|
|
directory alone until it has been set aside for a minute — not for the
|
|
unlink's sake, which the ordering proof above already covers, but so that a
|
|
pass still deciding about one is never mistaken for a pass that died holding
|
|
it. That settle window is a bound rather than a construction, and it is the
|
|
only part of this that is. Until the rest of it was structural it was merely
|
|
policy — snapshots belong to protected refs, protected refs are never
|
|
eviction candidates — a property held by vigilance rather than by
|
|
construction.
|
|
- **Every other way the source can change mid-clone is detected, not
|
|
prevented.** A `seed-fallback-dir` pointing at a directory something else
|
|
writes has no interlock at all. There, the per-attempt verification is what
|
|
stands between a torn read and a corrupt cache: the clone is retried
|
|
(`CACHE_CLONE_ATTEMPTS`, default 4) and then **fails the job loudly** —
|
|
never seeded partially, and never degraded to a silent cold build.
|
|
- **Disk reclamation is bounded, not immediate.** A consumer slower than the
|
|
grace period leaves one extra snapshot generation of directory entries on
|
|
the volume until a later publish sweeps it; a consumer whose job was killed
|
|
outright holds it until its marker passes `reader-stale-seconds`. The
|
|
residual is capped at one deferred generation per publisher ref, and its
|
|
real cost is close to inode count rather than byte count, since the
|
|
artifacts are hardlinked to whatever cloned them. A declined eviction is
|
|
never unlinked under the job that claimed it; what it costs meanwhile is
|
|
disk, normally as the cache restored under its own name and otherwise as one
|
|
set aside for a later pass to reclaim.
|
|
|
|
**Eviction** runs three passes: caches for branches that no longer exist on
|
|
origin are removed unconditionally; then, only if free space is under the
|
|
threshold, live caches are evicted oldest-first; then, as a last resort, this
|
|
run's own cache. Protected refs and any cache held open by a running job are
|
|
never candidates. Within the pressure pass, `target-*` directories are evicted
|
|
before `snapshot-*` ones — the reverse of the obvious order, because a
|
|
snapshot is hardlinked to everything cloned from it, so removing one frees
|
|
almost no real bytes while costing every future PR its warm start.
|
|
|
|
**File mtimes.** `actions/checkout` stamps every file with "now", which makes
|
|
every crate look changed to Cargo's mtime-based freshness check — a persistent
|
|
target directory buys nothing without fixing that. Each tracked file is
|
|
restored to the timestamp of the most recent commit that touched it, plus a
|
|
watermark override: for any file that changed since *this cache's own last
|
|
successful build*, "now" is stamped instead. That override is what makes a
|
|
merge safe, since a merge can introduce a commit authored before this cache's
|
|
last build, where the historically-correct mtime is exactly the wrong answer.
|
|
|
|
---
|
|
|
|
## Scratch names in a cache root are a cross-repo contract
|
|
|
|
Read this before adding a dot-prefixed directory under a cache root.
|
|
|
|
The volume is not swept by these scripts alone. A host-level arbiter —
|
|
daniel/gitdan's `scripts/ci-cache-reclaim.sh`, which runs outside any job —
|
|
reclaims the scratch trees a killed job strands here, and reads the reader
|
|
markers to decide whether a tree is still live. Its `LEFTOVER NAMING CONTRACT`
|
|
block is the canonical description; this side owns the names.
|
|
|
|
| name | produced by | if the job dies holding it |
|
|
|---|---|---|
|
|
| `.stage-<tag>` | `cache-lib.sh`, `hardlink_clone_into()` | stranded; only the arbiter reclaims it |
|
|
| `.publish-new-<tag>` | `publish-snapshot.sh` | stranded; see `daniel/gitdan#30` |
|
|
| `.publish-old-<key>-<tag>` | `publish-snapshot.sh` | swept by the next publish of that key, if there is one |
|
|
| `.evicting-<name>-<pid>` | `prune-cache.sh`, `evict_dir()` | swept at the start of the next prune pass |
|
|
| `.reading-<source>-<tag>` | `cache-lib.sh`, `reader_lock_acquire()` | not garbage — see below |
|
|
|
|
A `.reading-*` marker is protective, not scratch: it is how both this repo's
|
|
prune pass and the arbiter tell an in-flight clone from an abandoned one, and
|
|
the arbiter never deletes one. Delete a live marker and the tree it covers
|
|
becomes eligible for an unlink underneath the walk that is reading it, which is
|
|
the silent truncation the whole interlock exists to prevent.
|
|
|
|
**The rule: no new dot-prefixed entry under a cache root without a matching
|
|
prefix in `ci-cache-reclaim.sh`.** Adding a shape counts exactly as much as
|
|
renaming one. That script enumerates by explicit prefix rather than by dotglob
|
|
— deliberately, because a dotglob would pull reader markers into the candidate
|
|
stream alongside the trees they protect — so a name it has not been told about
|
|
is not handled conservatively, it is invisible, and an unreclaimed staging tree
|
|
is a full clone of a multi-GB target dir on the one volume whose entire problem
|
|
is disk. `.publish-new-` is the worked example: the two `.publish-*` names were
|
|
outside the contract when it was written, and `.publish-new-` strands exactly
|
|
as `.stage-` does, so it is the one that needed catching. Bringing both under
|
|
the arbiter's enumeration is tracked as `daniel/gitdan#30`.
|
|
|
|
The two lists are meant to be the same length — the five names above, and the
|
|
prefix constants `ci-cache-reclaim.sh` *declares*. Not the subset it enumerates
|
|
as reclaim candidates: that one is smaller, because `.reading-` is read and
|
|
never swept. A mismatch means one side gained a shape without telling the
|
|
other, which is the drift the rule exists to catch and the cheapest thing to
|
|
check.
|
|
|
|
**The staleness constants are part of the same contract, and that half has a
|
|
direction.** `CACHE_READ_STALE_SECONDS` (`cache-lib.sh`) and
|
|
`STALE_LOCK_SECONDS` (`prune-cache.sh`) are mirrored there, and the arbiter's
|
|
copies must be **greater than or equal to** these. Raising one here for longer
|
|
jobs, without raising its mirror first, makes the arbiter treat a marker whose
|
|
owner still considers it live as stale and delete a tree under an in-flight
|
|
clone — and its own minimum-age guard does not back-stop that, since a clone
|
|
holding a three-hour-old marker has a roughly three-hour-old staging tree.
|
|
Lowering either here needs no coordination: the arbiter then only defers a
|
|
reclamation this side would already have permitted, which costs disk rather
|
|
than correctness.
|
|
|
|
### Names this repo doesn't reclaim, but the arbiter depends on
|
|
|
|
The five names above are leftovers — dead trees gitdan's arbiter finds and
|
|
deletes. Four other dot-prefixed names change that arbiter's *behaviour*
|
|
without ever being a leftover: it reads them to make a correctness decision
|
|
and never reclaims them. They are a real cross-repo dependency too, on the
|
|
name's spelling rather than on the tree's lifetime — gitdan's
|
|
`DEPENDED-UPON NAMES CONTRACT` block (in `scripts/ci-cache-reclaim.sh`) is the
|
|
canonical description; this is the producer's half for the two we own.
|
|
|
|
| Name | Produced here by | Consequence of an unannounced rename |
|
|
|---|---|---|
|
|
| `.ci-lock-<id>` | `scripts/cache-lock.sh` (acquire/release) and `cache-lib.sh`'s `write_cache_lock()` | The arbiter's lock check silently stops matching — a live job's staging tree loses its liveness guard and becomes an ordinary reclaim candidate while still in use |
|
|
| `.cache-last-used` | `cargo-cache/action.yml`, stamped every run | The arbiter falls back to directory mtime — silently reordering its eviction order, and possibly failing to recognise the directory as a cache dir at all |
|
|
|
|
Two more names are in gitdan's list — `.gitea-last-used` and `.ci-keep` — and
|
|
neither is produced by anything in this repo, so there is no producer-side
|
|
half to write here. `.gitea-last-used` is a naming convention individual
|
|
repos used before adopting this shared action; nothing here writes it, and
|
|
gitdan's script reads it only for compatibility with directories created
|
|
under that older scheme. `.ci-keep` is a per-repo, hand-placed opt-out a
|
|
consuming repo's own workflow drops directly into a cache directory it wants
|
|
exempted from eviction — never something this action or its scripts write.
|
|
|
|
**Adding a fifth depended-upon name counts as much as renaming one of the two
|
|
above.** If a future change here makes gitdan's arbiter start depending on the
|
|
spelling of some new dot-prefixed name — a name it reads for a decision but
|
|
never reclaims — that is exactly this category, and it needs the matching
|
|
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
|
|
|
---
|
|
|
|
## Inputs
|
|
|
|
### `cargo-cache`
|
|
|
|
| input | default | meaning |
|
|
|---|---|---|
|
|
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
|
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
|
| `min-free-percent` | `10` | prune when free space drops below this |
|
|
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
|
| `prune` | `true` | run the eviction pass |
|
|
| `liveness-prune` | `true` | within eviction, remove caches for branches gone from origin |
|
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
|
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
|
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
|
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
|
|
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
|
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
|
|
|
Outputs: `target-dir`, `cache-key`, `seeded-from` (`own` \| `base-snapshot` \|
|
|
`own-snapshot` \| `fallback-dir` \| `concurrent-peer` \| `cold`).
|
|
|
|
Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
|
`CARGO_CACHE_KEY`, `CARGO_CACHE_LOCK_ID`, `CARGO_CACHE_SCRIPTS`,
|
|
`CI_WATERMARK_FILE`.
|
|
|
|
### `cargo-cache-publish`
|
|
|
|
| input | default | meaning |
|
|
|---|---|---|
|
|
| `cache-root` | `/cache` | must match the consume action |
|
|
| `protected-branches` | `dev main` | refs that publish snapshots |
|
|
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
|
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
|
|
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
|
|
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
|
|
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
|
|
|
|
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
|
|
`dev` into `main` has `own-ref` `dev` and would otherwise publish a snapshot
|
|
of a merge-preview build, which is not what `dev` is.
|
|
|
|
---
|
|
|
|
## Multiple jobs in one workflow
|
|
|
|
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
|
|
each need their **own** watermark file. A shared one breaks the moment two
|
|
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
|
|
and job B then reads that just-advanced value, computes an empty diff, and
|
|
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
|
|
already gives each job its own; only override `watermark-file` if you also
|
|
override `lock-id`, and then keep both distinct per job.
|
|
|
|
---
|
|
|
|
## Constraints of this runner
|
|
|
|
- **The repository must be public.** act_runner fetches actions by anonymous
|
|
git clone and has no credentialed-fetch option, so a private action
|
|
repository simply fails to resolve. Nothing secret goes in here.
|
|
- **`uses:` needs the absolute URL.** A bare `owner/repo` resolves against
|
|
github.com, because Gitea's `DEFAULT_ACTIONS_URL` is unset — and it has to
|
|
stay unset, or `actions/checkout`, `dtolnay/rust-toolchain` and
|
|
`taiki-e/install-action` stop resolving.
|
|
- **The runner pre-fetches every referenced action before running any step**,
|
|
so a bad action reference fails the job at step 0 rather than where it is
|
|
used.
|
|
- **`container.volumes` is job-level** and cannot be set from inside a
|
|
composite action. The consuming workflow declares it; see the quick start.
|
|
- **The cache volume is ext4** — no reflink support, which is precisely why
|
|
hardlinks are the mechanism that makes cloning cheap.
|
|
|
|
---
|
|
|
|
## Versioning
|
|
|
|
Pin `@v1`. It is a moving major tag: fixes and backward-compatible inputs move
|
|
it forward, and anything that would break an existing consumer gets `v2`
|
|
instead. Pin a commit SHA if you want a frozen version.
|
|
|
|
The alternative is immutable release tags — `v1.0.1`, `v1.0.2`, … — with every
|
|
consumer edited to point at the new one per fix. That is the safer model in
|
|
general and the wrong one here. What it buys is the ability to hold one
|
|
consumer back on a known-good version; what it costs is a PR in every consumer
|
|
repo per fix, and its real failure mode with two consumers and one operator is
|
|
that the second one is simply never updated and quietly runs a version nobody
|
|
is testing. The moving pointer makes a release one action with one blast
|
|
radius, which is the thing worth being deliberate about. Anyone who wants the
|
|
immutable behaviour already has it, by pinning a SHA.
|
|
|
|
**What `@v1` promises** is that whatever it points at works with the inputs
|
|
documented above, spelled as they are documented. A change that renames or
|
|
removes an input, changes a default in a way that changes behaviour, or
|
|
requires something new of the consuming workflow — another `container.volumes`
|
|
entry, another permission — is a `v2`, not a `v1` move. Everything else moves
|
|
`v1`: correctness fixes, new optional inputs, and anything internal to
|
|
`scripts/`.
|
|
|
|
**Moving the tag is a release step, and it is the operator's.** Merging to
|
|
`main` ships nothing to anybody. `v1` is a lightweight tag and does not follow
|
|
a branch, so until it is re-pointed every consumer keeps fetching the commit it
|
|
already named, whatever `main` now says. The gap is deliberate: re-pointing
|
|
`v1` changes what another repository's CI executes on its next run, so it is a
|
|
decision taken once, knowingly, after the merge — never something a merge does
|
|
by itself.
|
|
|
|
```bash
|
|
git fetch origin
|
|
git tag -f v1 origin/main
|
|
git push -f origin v1
|
|
git ls-remote --tags origin v1 # must equal git rev-parse origin/main
|
|
```
|
|
|
|
**Downstream** are emowheel, which pins `cargo-cache@v1` and
|
|
`cargo-cache-publish@v1` across its CI workflow, and zemyna, migrating to the
|
|
same pin. Both pick a move up on their next run with no change on their side,
|
|
which is the whole point of the moving pointer and also the reason the move is
|
|
not automatic.
|
|
|
|
---
|
|
|
|
## Development
|
|
|
|
```bash
|
|
shellcheck -x --source-path=scripts scripts/*.sh
|
|
bash scripts/selftest.sh # everything (needs cargo)
|
|
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
|
```
|
|
|
|
Both run in CI — `.gitea/workflows/ci.yaml`, one job, on pushes to `main` and
|
|
on PRs that were non-draft when the run was created. It installs shellcheck
|
|
and a stable Rust toolchain and references no credentials; the scratch
|
|
workspaces the compiler-backed suites build use path dependencies only, so
|
|
nothing reaches crates.io. It runs the full suite rather than `--fast`,
|
|
because the two compiler-backed suites are the ones that check this scheme
|
|
against real Cargo instead of against a fixture. Draft (`WIP:`-titled) PRs
|
|
skip it, and un-drafting does **not** un-skip them — the guard is evaluated
|
|
when a run is created and un-drafting creates none, so push an empty commit
|
|
after un-WIP'ing.
|
|
|
|
This repo is consumed by three other repos' CI at `@v1`, a moving tag, so a
|
|
change here reaches all of them at once. That is what the gate is for.
|
|
|
|
| suite | covers |
|
|
|---|---|
|
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
|
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
|
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
|
|
|
Every suite runs the actual script, not a reimplementation of its logic, and
|
|
every fix scenario is paired with a control that reproduces the bug — a
|
|
scenario that passes either way proves nothing.
|
|
|
|
The concurrency scenarios take one of three shapes, and none of them races
|
|
for the interleaving its assertion depends on.
|
|
|
|
**A genuine race whose asserted invariant holds under any interleaving.**
|
|
`seed-target-dir-selftest.sh` scenario 7 starts two real seeds on one cache
|
|
key and asserts only what must be true whichever of them wins the rename.
|
|
|
|
**A `PATH` stub on a command the code under test calls at a known point**,
|
|
which places the interference inside the window rather than hoping it lands
|
|
there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
|
|
measurement publishes a reader marker strictly between its check and its
|
|
unlink. `seed-target-dir-selftest.sh` uses the shape five times over, on two
|
|
different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
|
|
clone is what rotates the snapshot underneath it, loses a subtree of its own
|
|
source, or reports a failure over a tree that is in fact whole — each strictly
|
|
inside the window that clone's checks cover; scenario 10 stubs `cp` one level
|
|
down instead, refusing the per-file copies that unshare the mutable paths.
|
|
Scenario 8d stubs `stat`, because the check it pins fires on an identity that
|
|
could not be READ rather than on one that changed, and the only way to make
|
|
that the sole witness is to fail the identity reads while the copy between
|
|
them succeeds. 8a also starts a second real process — the actual
|
|
`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
|
|
concurrency is incidental to the determinism. Scenario 11 reuses 8a's exact
|
|
stub and the same forced rotation, but reads a different witness: not the
|
|
consumer's own checks, but a line in the *publisher's* log reporting that it
|
|
waited on the reader marker this consumer's clone wrote — `reader_lock_acquire`
|
|
is exercised by 8a already, but nothing asserts it actually fired until this
|
|
scenario reads that line back. Every stub asserts that it fired, because a
|
|
scenario whose interference silently did not happen passes for the wrong
|
|
reason.
|
|
|
|
**A synthetic stand-in for the other side, where that artefact *is* the
|
|
contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
|
|
`.reading-*` marker instead of running a slow consumer: the marker is the
|
|
whole agreement between reader and publisher, so holding one is being a
|
|
reader, and racing a real one would make the suite's runtime the thing under
|
|
test.
|
|
|
|
Gating the interfering step on *observed progress* of the step it interferes
|
|
with was an earlier answer here, and it is not one: seeing that a walk has
|
|
started says nothing about where it will be when the interference lands, so
|
|
the assertion downstream held only some of the time (issue #3). No scenario
|
|
does it any more.
|
|
|
|
Where more than one guard could catch a fault, a scenario should assert
|
|
*which* one did — otherwise deleting the guard under test leaves the suite
|
|
green because a sibling fires in its place. Scenarios 8a to 8d and 10 of the
|
|
seed suite do, and each is reddened by exactly one mutation of the clone's
|
|
checks. Scenario 9 does not, and a mutation still survives it: with its source
|
|
unreadable, `cp -al` leaves the staging directory at mode `000`, so deleting
|
|
the exit-status check does not change the outcome — the unshare pass aborts
|
|
the clone instead, and the assertion is satisfied down a path it was not
|
|
written for.
|
|
|
|
That is the standing hazard here, and it is not hypothetical. Two guards that
|
|
can each catch the same fault mask each other, so **neither** is individually
|
|
necessary and no fixture built around that fault can pin either one — which is
|
|
how both `[ "$cp_rc" -eq 0 ]` and `[ "$i_before" != missing ]` sat unpinned
|
|
(issue #5) while looking well covered. Isolating a check means constructing
|
|
the state only it can see, not the state that trips several at once.
|
|
|
|
Scenario 11 applies the same discipline to a witness outside the clone
|
|
entirely: not which of several checks inside `hardlink_clone_into` caught a
|
|
fault, but whether `reader_lock_acquire`'s marker was observed by anything
|
|
outside it at all. The consumer's own log and exit status are silent either
|
|
way — a run with the marker deleted still succeeds — so what is asserted is
|
|
one line in the *publisher's* log reporting that it waited. Deleting
|
|
`reader_lock_acquire` (issue #10) leaves that line unwritten without failing
|
|
anything else in the suite.
|
|
|
|
The action YAML holds no logic beyond wiring; everything testable lives in
|
|
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
|
the actions reach their shared scripts through
|
|
`${{ github.action_path }}/../scripts`, which works because the runner clones
|
|
the whole repository when it fetches an action.
|