Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4da6e8d30f
|
@@ -1,83 +0,0 @@
|
|||||||
name: CI
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [main]
|
|
||||||
pull_request:
|
|
||||||
branches: [main]
|
|
||||||
# Spelled out only to keep `ready_for_review` in the list — naming any type
|
|
||||||
# replaces the whole default set, so the other three have to be restated.
|
|
||||||
# It is inert on this instance (draft state here is the `WIP:` title
|
|
||||||
# prefix, so un-drafting is a title edit and raises no
|
|
||||||
# `ready_for_review` action) and costs nothing.
|
|
||||||
#
|
|
||||||
# The consequence, which is the part that bites: the `if:` guard below is
|
|
||||||
# evaluated when a run is CREATED, and un-drafting creates no run. A PR
|
|
||||||
# opened as a draft keeps its skip decision until something else produces
|
|
||||||
# one. Push an empty commit after un-WIP'ing.
|
|
||||||
types: [opened, synchronize, reopened, ready_for_review]
|
|
||||||
|
|
||||||
# gitdan-ci runs four repos' CI on two capacity slots, and the compiler-backed
|
|
||||||
# suites below are multi-minute. A superseded run costs a slot in front of
|
|
||||||
# somebody's build, so drop it.
|
|
||||||
#
|
|
||||||
# `push` groups on `github.sha` rather than `github.ref`: a constant per-branch
|
|
||||||
# group is what let this Gitea (1.26.0) cancel two of daniel/gitdan's merge
|
|
||||||
# runs outright while `cancel-in-progress` was gated away from `push` entirely
|
|
||||||
# — see the long note in that repo's ci.yaml for the evidence. Giving every
|
|
||||||
# commit its own group leaves that behaviour nothing to act on.
|
|
||||||
concurrency:
|
|
||||||
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
selftest:
|
|
||||||
name: shellcheck + selftests
|
|
||||||
# Two clauses, both load-bearing. The second skips draft PRs: Gitea sets
|
|
||||||
# draft:true when the title starts with `WIP:`, so work-in-progress pushes
|
|
||||||
# cost the shared runner nothing until the PR is un-WIP'd. The first is
|
|
||||||
# what keeps that from also skipping pushes to `main` — a `push` event has
|
|
||||||
# no `pull_request` context, so `github.event.pull_request.draft` is empty
|
|
||||||
# there and the negation alone would be unreliable. Never drop it.
|
|
||||||
if: ${{ github.event_name != 'pull_request' || !github.event.pull_request.draft }}
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 20
|
|
||||||
# Deliberately no `container.volumes:` entry, unlike every repo that
|
|
||||||
# CONSUMES this action. The suites below build throwaway workspaces under
|
|
||||||
# `mktemp -d` and want a cold target dir every time — a persistent cache
|
|
||||||
# would make "did this run rebuild?" unanswerable, which is the question
|
|
||||||
# restore-mtimes-selftest.sh exists to ask. So this job takes no share of
|
|
||||||
# the shared CI cache disk budget.
|
|
||||||
steps:
|
|
||||||
- name: Checkout sources
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Install shellcheck
|
|
||||||
uses: taiki-e/install-action@v2
|
|
||||||
with:
|
|
||||||
tool: shellcheck
|
|
||||||
|
|
||||||
# `hardlink-clone-selftest.sh` and `restore-mtimes-selftest.sh` drive a
|
|
||||||
# real Cargo against a real scratch workspace — they are the only things
|
|
||||||
# here that verify the hardlink-aliasing and mtime-freshness behaviour
|
|
||||||
# against the compiler rather than against a fixture, and selftest.sh's
|
|
||||||
# own header says `--fast` is for iterating, not for signing off a
|
|
||||||
# change. So CI installs a toolchain and runs the full set.
|
|
||||||
#
|
|
||||||
# The scratch workspaces use path dependencies only, so nothing here
|
|
||||||
# reaches crates.io.
|
|
||||||
- name: Install Rust toolchain
|
|
||||||
uses: dtolnay/rust-toolchain@stable
|
|
||||||
|
|
||||||
# Every script, including the suites themselves. `-x` follows the
|
|
||||||
# `. cache-lib.sh` each one sources, which is where most of the logic
|
|
||||||
# being checked actually lives; without it shellcheck reports SC1091 and
|
|
||||||
# analyses each file with a hole in it.
|
|
||||||
- name: shellcheck
|
|
||||||
run: shellcheck -x --source-path=scripts scripts/*.sh
|
|
||||||
|
|
||||||
# One command, not six: selftest.sh is the entry point a developer runs,
|
|
||||||
# so a suite added there is gated here without a matching edit in this
|
|
||||||
# file.
|
|
||||||
- name: Selftests
|
|
||||||
run: bash scripts/selftest.sh
|
|
||||||
@@ -317,39 +317,6 @@ spelling of some new dot-prefixed name — a name it reads for a decision but
|
|||||||
never reclaims — that is exactly this category, and it needs the matching
|
never reclaims — that is exactly this category, and it needs the matching
|
||||||
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
||||||
|
|
||||||
### The directory LAYOUT is part of that contract as well
|
|
||||||
|
|
||||||
Names are one half; where they sit is the other. gitdan's arbiter walks a
|
|
||||||
volume's `_data` tree to `CI_CACHE_MAX_DEPTH`, which is **2** — deliberately
|
|
||||||
tight, because a deeper walk starts meeting Cargo's own
|
|
||||||
`incremental/<crate>-<hash>` directories, which match the same name shape it
|
|
||||||
uses to recognise a cache dir and must never be evicted individually. So:
|
|
||||||
|
|
||||||
```
|
|
||||||
_data/target-<key> depth 1 — no lineage
|
|
||||||
_data/<lineage>/target-<key> depth 2 — a lineage
|
|
||||||
_data/<a>/<b>/target-<key> depth 3 — INVISIBLE to the arbiter
|
|
||||||
```
|
|
||||||
|
|
||||||
That budget is the whole reason `cache-lineage` is one path component and not
|
|
||||||
a path. Nesting deeper is not an error anywhere: the caches work, the
|
|
||||||
in-workflow prune pass keeps managing them, and the one script whose job is
|
|
||||||
the shared disk budget across every repo simply never sees them again.
|
|
||||||
|
|
||||||
It is also why the fix for daniel/gitdan#60 nests rather than suffixing the
|
|
||||||
cache key. A `target-<key>-<lineage>` name would be read as dead by
|
|
||||||
`prune-cache.sh`'s liveness pass — which classifies by recomputing
|
|
||||||
`target-<cache_key(branch)>` for every branch on origin — and evicted
|
|
||||||
unconditionally on every run; and it falls out of the arbiter's own
|
|
||||||
`BRANCH_DIR_RE` too, so the same directories would never be candidates there
|
|
||||||
either. Nesting leaves both matchers reading exactly the names they already
|
|
||||||
read, one level down.
|
|
||||||
|
|
||||||
One known rough edge, on gitdan's side and cosmetic: that script logs an
|
|
||||||
eviction as `<volume>/<basename>`, so a nested `target-<key>` and a flat one
|
|
||||||
of the same key are indistinguishable in its output. It evicts the right
|
|
||||||
directory; the line just doesn't say which.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
@@ -359,7 +326,6 @@ directory; the line just doesn't say which.
|
|||||||
| input | default | meaning |
|
| input | default | meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
||||||
| `cache-lineage` | *(empty)* | one directory level under `cache-root`, for a second job building the same ref for a different target or profile — see [Multiple jobs in one workflow](#multiple-jobs-in-one-workflow) |
|
|
||||||
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
||||||
| `min-free-percent` | `10` | prune when free space drops below this |
|
| `min-free-percent` | `10` | prune when free space drops below this |
|
||||||
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
||||||
@@ -368,7 +334,7 @@ directory; the line just doesn't say which.
|
|||||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||||
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
||||||
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
||||||
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one target directory; the default already does |
|
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
|
||||||
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
||||||
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
||||||
|
|
||||||
@@ -384,7 +350,6 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
|||||||
| input | default | meaning |
|
| input | default | meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cache-root` | `/cache` | must match the consume action |
|
| `cache-root` | `/cache` | must match the consume action |
|
||||||
| `cache-lineage` | *(empty)* | must match the consume action; a mismatch fails the step rather than publishing the wrong tree |
|
|
||||||
| `protected-branches` | `dev main` | refs that publish snapshots |
|
| `protected-branches` | `dev main` | refs that publish snapshots |
|
||||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||||
@@ -401,70 +366,13 @@ of a merge-preview build, which is not what `dev` is.
|
|||||||
|
|
||||||
## Multiple jobs in one workflow
|
## Multiple jobs in one workflow
|
||||||
|
|
||||||
Two jobs building the same ref — a `ci` job and a `wasm` job, say — are two
|
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
|
||||||
consumers of one cache key, and the cache key alone is not enough to keep them
|
each need their **own** watermark file. A shared one breaks the moment two
|
||||||
apart.
|
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
|
||||||
|
and job B then reads that just-advanced value, computes an empty diff, and
|
||||||
**Give each its own lineage.** A cache key names a *ref*; what a target
|
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
|
||||||
directory holds is the product of a ref and a build configuration. Left to the
|
already gives each job its own; only override `watermark-file` if you also
|
||||||
key alone, both jobs export the same `CARGO_TARGET_DIR`, and Cargo's
|
override `lock-id`, and then keep both distinct per job.
|
||||||
build-directory lock is exclusive — so on a runner with more than one slot the
|
|
||||||
second job sits on `Blocking waiting for file lock on build directory` for the
|
|
||||||
length of the first, occupying a capacity slot while doing nothing
|
|
||||||
(daniel/gitdan#60). `cache-lineage` is that second dimension:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
- name: Restore the Cargo cache
|
|
||||||
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
|
|
||||||
with:
|
|
||||||
cache-lineage: wasm32 # the `ci` job sets none
|
|
||||||
|
|
||||||
# ... build steps ...
|
|
||||||
|
|
||||||
- name: Record watermark, publish cache snapshot
|
|
||||||
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
|
||||||
with:
|
|
||||||
cache-lineage: wasm32 # the SAME value, or the step fails
|
|
||||||
```
|
|
||||||
|
|
||||||
A lineage nests one directory level under the cache root
|
|
||||||
(`<cache-root>/<lineage>/target-<key>`), so each lineage gets its own target
|
|
||||||
dirs, its own snapshots, and its own prune pass. Everything else works as it
|
|
||||||
already did, one level down: a PR branch in a lineage layers over **that
|
|
||||||
lineage's** base snapshot, the publisher branch publishes into it, and a prune
|
|
||||||
pass run inside it never sees a sibling lineage's caches.
|
|
||||||
|
|
||||||
Setting no lineage resolves to the cache root unchanged, byte for byte, so a
|
|
||||||
workflow that does not use one keeps the exact directories it already has on
|
|
||||||
the volume.
|
|
||||||
|
|
||||||
**Both actions need the same value.** `cargo-cache-publish` derives both ends
|
|
||||||
of the snapshot swap from its own `cache-root`, so a publish step left at the
|
|
||||||
default while its consume step nested would republish a *different* lineage's
|
|
||||||
live target dir over that lineage's snapshot, on every push, with nothing in
|
|
||||||
the log to say so. The publish action therefore compares its own inputs
|
|
||||||
against the `CARGO_CACHE_ROOT` the consume step exported and fails the step on
|
|
||||||
a mismatch. (The `mode: release-lock` call is exempt: it releases a lock on
|
|
||||||
`$CARGO_TARGET_DIR` and never touches a cache root, so it takes no lineage.)
|
|
||||||
|
|
||||||
**Some lineage names are refused.** A lineage is one path component, drawn
|
|
||||||
from `[A-Za-z0-9._-]`, and several otherwise-reasonable names are rejected at
|
|
||||||
resolve time because a *reader elsewhere* would stop seeing the caches
|
|
||||||
underneath them: a Cargo profile name (`debug`, `release`, `doc`, …) is one
|
|
||||||
gitdan's arbiter never descends into, a `target-`/`snapshot-` prefix makes the
|
|
||||||
lineage directory itself an eviction candidate for this repo's own prune pass,
|
|
||||||
and a hex-suffixed name is read by that arbiter as a per-branch cache dir in
|
|
||||||
its own right. `validate_cache_lineage()` in `scripts/cache-lib.sh` states each
|
|
||||||
rejection with the reader that imposes it.
|
|
||||||
|
|
||||||
**Watermarks are still per job.** Two jobs in one lineage — or one job before
|
|
||||||
lineages were introduced — each need their **own** watermark file. A shared one
|
|
||||||
breaks the moment two jobs run in sequence within one trigger: job A advances
|
|
||||||
the watermark to HEAD, and job B then reads that just-advanced value, computes
|
|
||||||
an empty diff, and loses the merge protection entirely. The default
|
|
||||||
(`.ci-watermark-<job>-sha`) already gives each job its own; only override
|
|
||||||
`watermark-file` if you also override `lock-id`, and then keep both distinct
|
|
||||||
per job.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -537,30 +445,14 @@ not automatic.
|
|||||||
## Development
|
## Development
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
shellcheck -x --source-path=scripts scripts/*.sh
|
bash scripts/selftest.sh # everything (~1 min; needs cargo)
|
||||||
bash scripts/selftest.sh # everything (needs cargo)
|
|
||||||
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
||||||
```
|
```
|
||||||
|
|
||||||
Both run in CI — `.gitea/workflows/ci.yaml`, one job, on pushes to `main` and
|
|
||||||
on PRs that were non-draft when the run was created. It installs shellcheck
|
|
||||||
and a stable Rust toolchain and references no credentials; the scratch
|
|
||||||
workspaces the compiler-backed suites build use path dependencies only, so
|
|
||||||
nothing reaches crates.io. It runs the full suite rather than `--fast`,
|
|
||||||
because the two compiler-backed suites are the ones that check this scheme
|
|
||||||
against real Cargo instead of against a fixture. Draft (`WIP:`-titled) PRs
|
|
||||||
skip it, and un-drafting does **not** un-skip them — the guard is evaluated
|
|
||||||
when a run is created and un-drafting creates none, so push an empty commit
|
|
||||||
after un-WIP'ing.
|
|
||||||
|
|
||||||
This repo is consumed by three other repos' CI at `@v1`, a moving tag, so a
|
|
||||||
change here reaches all of them at once. That is what the gate is for.
|
|
||||||
|
|
||||||
| suite | covers |
|
| suite | covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `cache-root-selftest.sh` | that a lineage nests one level and nothing else moves: no lineage resolves byte-for-byte to the cache root, two lineages on one cache key get disjoint target dirs, seed/publish/prune all stay inside their own lineage, a PR layers over its own lineage's base snapshot — **and one rejection per lineage name a reader elsewhere would stop seeing**, plus the publish-side mismatch guard |
|
|
||||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published |
|
||||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
||||||
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
||||||
@@ -580,7 +472,7 @@ key and asserts only what must be true whichever of them wins the rename.
|
|||||||
which places the interference inside the window rather than hoping it lands
|
which places the interference inside the window rather than hoping it lands
|
||||||
there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
|
there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
|
||||||
measurement publishes a reader marker strictly between its check and its
|
measurement publishes a reader marker strictly between its check and its
|
||||||
unlink. `seed-target-dir-selftest.sh` uses the shape five times over, on two
|
unlink. `seed-target-dir-selftest.sh` uses the shape four times over, on two
|
||||||
different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
|
different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
|
||||||
clone is what rotates the snapshot underneath it, loses a subtree of its own
|
clone is what rotates the snapshot underneath it, loses a subtree of its own
|
||||||
source, or reports a failure over a tree that is in fact whole — each strictly
|
source, or reports a failure over a tree that is in fact whole — each strictly
|
||||||
@@ -591,14 +483,9 @@ could not be READ rather than on one that changed, and the only way to make
|
|||||||
that the sole witness is to fail the identity reads while the copy between
|
that the sole witness is to fail the identity reads while the copy between
|
||||||
them succeeds. 8a also starts a second real process — the actual
|
them succeeds. 8a also starts a second real process — the actual
|
||||||
`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
|
`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
|
||||||
concurrency is incidental to the determinism. Scenario 11 reuses 8a's exact
|
concurrency is incidental to the determinism. Every stub asserts that it
|
||||||
stub and the same forced rotation, but reads a different witness: not the
|
fired, because a scenario whose interference silently did not happen passes
|
||||||
consumer's own checks, but a line in the *publisher's* log reporting that it
|
for the wrong reason.
|
||||||
waited on the reader marker this consumer's clone wrote — `reader_lock_acquire`
|
|
||||||
is exercised by 8a already, but nothing asserts it actually fired until this
|
|
||||||
scenario reads that line back. Every stub asserts that it fired, because a
|
|
||||||
scenario whose interference silently did not happen passes for the wrong
|
|
||||||
reason.
|
|
||||||
|
|
||||||
**A synthetic stand-in for the other side, where that artefact *is* the
|
**A synthetic stand-in for the other side, where that artefact *is* the
|
||||||
contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
|
contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
|
||||||
@@ -630,15 +517,6 @@ how both `[ "$cp_rc" -eq 0 ]` and `[ "$i_before" != missing ]` sat unpinned
|
|||||||
(issue #5) while looking well covered. Isolating a check means constructing
|
(issue #5) while looking well covered. Isolating a check means constructing
|
||||||
the state only it can see, not the state that trips several at once.
|
the state only it can see, not the state that trips several at once.
|
||||||
|
|
||||||
Scenario 11 applies the same discipline to a witness outside the clone
|
|
||||||
entirely: not which of several checks inside `hardlink_clone_into` caught a
|
|
||||||
fault, but whether `reader_lock_acquire`'s marker was observed by anything
|
|
||||||
outside it at all. The consumer's own log and exit status are silent either
|
|
||||||
way — a run with the marker deleted still succeeds — so what is asserted is
|
|
||||||
one line in the *publisher's* log reporting that it waited. Deleting
|
|
||||||
`reader_lock_acquire` (issue #10) leaves that line unwritten without failing
|
|
||||||
anything else in the suite.
|
|
||||||
|
|
||||||
The action YAML holds no logic beyond wiring; everything testable lives in
|
The action YAML holds no logic beyond wiring; everything testable lives in
|
||||||
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
||||||
the actions reach their shared scripts through
|
the actions reach their shared scripts through
|
||||||
|
|||||||
@@ -10,15 +10,6 @@ inputs:
|
|||||||
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
||||||
required: false
|
required: false
|
||||||
default: '/cache'
|
default: '/cache'
|
||||||
cache-lineage:
|
|
||||||
description: >-
|
|
||||||
Must match the cargo-cache step in this job. Checked rather than
|
|
||||||
assumed: this action derives both ends of the snapshot swap from its own
|
|
||||||
cache-root, so a publish step left at the default while its consume step
|
|
||||||
nested would republish a DIFFERENT lineage's live target dir over that
|
|
||||||
lineage's snapshot, on every push, silently. A mismatch fails the step.
|
|
||||||
required: false
|
|
||||||
default: ''
|
|
||||||
protected-branches:
|
protected-branches:
|
||||||
description: >-
|
description: >-
|
||||||
Space-separated refs that publish snapshots. A run whose own ref is not
|
Space-separated refs that publish snapshots. A run whose own ref is not
|
||||||
@@ -76,11 +67,6 @@ runs:
|
|||||||
# Everything here reads the environment the consume action exported, so a
|
# Everything here reads the environment the consume action exported, so a
|
||||||
# workflow that forgets to run cargo-cache first fails loudly here rather
|
# workflow that forgets to run cargo-cache first fails loudly here rather
|
||||||
# than silently publishing a snapshot of the wrong directory.
|
# than silently publishing a snapshot of the wrong directory.
|
||||||
#
|
|
||||||
# The cache root is taken from that same environment for the same reason,
|
|
||||||
# and this action's own cache-root/cache-lineage inputs are checked against
|
|
||||||
# it rather than used. The two have always had to agree; until a lineage
|
|
||||||
# existed they always did, because nobody overrode the default.
|
|
||||||
- id: resolve
|
- id: resolve
|
||||||
shell: bash
|
shell: bash
|
||||||
env:
|
env:
|
||||||
@@ -106,18 +92,6 @@ runs:
|
|||||||
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
||||||
echo "active=yes" >> "$GITHUB_OUTPUT"
|
echo "active=yes" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
# Only in publish mode. The `release-lock` call is an `if: always()`
|
|
||||||
# step that consuming workflows invoke with `mode:` and nothing else —
|
|
||||||
# it releases $CARGO_CACHE_LOCK_ID on $CARGO_TARGET_DIR and never
|
|
||||||
# touches a cache root at all, so holding its default inputs to the
|
|
||||||
# consume step's would fail the cleanup step of every job that sets a
|
|
||||||
# lineage, for a value it does not use.
|
|
||||||
if [ "${{ inputs.mode }}" = "publish" ]; then
|
|
||||||
: "${CARGO_CACHE_ROOT:?cargo-cache-publish: CARGO_CACHE_ROOT not set by the cargo-cache action}"
|
|
||||||
bash "${CARGO_CACHE_SCRIPTS}/cache-root.sh" verify \
|
|
||||||
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}" "$CARGO_CACHE_ROOT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
OWN_REF="${{ inputs.own-ref }}"
|
OWN_REF="${{ inputs.own-ref }}"
|
||||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||||
|
|
||||||
@@ -157,7 +131,7 @@ runs:
|
|||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||||
"$CARGO_CACHE_KEY" "$CARGO_CACHE_ROOT" \
|
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
||||||
"${{ github.job }}-${{ github.run_id }}-$$"
|
"${{ github.job }}-${{ github.run_id }}-$$"
|
||||||
|
|
||||||
# Released in both modes. In `publish` mode this is the normal end-of-job
|
# Released in both modes. In `publish` mode this is the normal end-of-job
|
||||||
|
|||||||
+6
-38
@@ -10,18 +10,6 @@ inputs:
|
|||||||
description: 'Mount point of the persistent cache volume inside the job container.'
|
description: 'Mount point of the persistent cache volume inside the job container.'
|
||||||
required: false
|
required: false
|
||||||
default: '/cache'
|
default: '/cache'
|
||||||
cache-lineage:
|
|
||||||
description: >-
|
|
||||||
Distinguishes two jobs that build the SAME ref for different targets or
|
|
||||||
profiles (a host build and a wasm32 build, say) and would otherwise
|
|
||||||
resolve to one CARGO_TARGET_DIR and serialise on Cargo's exclusive
|
|
||||||
build-directory lock. Names one directory level under cache-root:
|
|
||||||
<cache-root>/<lineage>/target-<key>. Must be a single path component;
|
|
||||||
several names are refused outright because a reader elsewhere would stop
|
|
||||||
seeing the caches under them (see validate_cache_lineage in
|
|
||||||
scripts/cache-lib.sh). Pass the same value to cargo-cache-publish.
|
|
||||||
required: false
|
|
||||||
default: ''
|
|
||||||
protected-branches:
|
protected-branches:
|
||||||
description: >-
|
description: >-
|
||||||
Space-separated refs that publish snapshots and are never evicted.
|
Space-separated refs that publish snapshots and are never evicted.
|
||||||
@@ -66,9 +54,8 @@ inputs:
|
|||||||
watermark-file:
|
watermark-file:
|
||||||
description: >-
|
description: >-
|
||||||
Name of this job's build-watermark file inside the target dir. MUST be
|
Name of this job's build-watermark file inside the target dir. MUST be
|
||||||
distinct per job when two jobs share one target directory — which two
|
distinct per job when two jobs share one cache key. Defaults to
|
||||||
jobs no longer need to do; cache-lineage gives them separate ones.
|
.ci-watermark-<job>-sha.
|
||||||
Defaults to .ci-watermark-<job>-sha, already distinct per job.
|
|
||||||
required: false
|
required: false
|
||||||
default: ''
|
default: ''
|
||||||
lock-id:
|
lock-id:
|
||||||
@@ -87,12 +74,6 @@ outputs:
|
|||||||
cache-key:
|
cache-key:
|
||||||
description: 'Sanitized cache key for this run''s own ref.'
|
description: 'Sanitized cache key for this run''s own ref.'
|
||||||
value: ${{ steps.resolve.outputs.cache-key }}
|
value: ${{ steps.resolve.outputs.cache-key }}
|
||||||
cache-root:
|
|
||||||
description: >-
|
|
||||||
Resolved cache root — cache-root, plus the lineage directory when one is
|
|
||||||
set. Every directory this action reads or writes is under it. Also
|
|
||||||
exported as CARGO_CACHE_ROOT.
|
|
||||||
value: ${{ steps.resolve.outputs.cache-root }}
|
|
||||||
seeded-from:
|
seeded-from:
|
||||||
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
|
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
|
||||||
value: ${{ steps.seed.outputs.seeded-from }}
|
value: ${{ steps.seed.outputs.seeded-from }}
|
||||||
@@ -114,16 +95,6 @@ runs:
|
|||||||
# `base_ref` is populated only for pull_request events. A push run has
|
# `base_ref` is populated only for pull_request events. A push run has
|
||||||
# nothing to layer over: its own ref IS the reference branch. It
|
# nothing to layer over: its own ref IS the reference branch. It
|
||||||
# publishes, it does not consume.
|
# publishes, it does not consume.
|
||||||
#
|
|
||||||
# The cache root is resolved first because every path below hangs off it.
|
|
||||||
# A lineage nests one directory level (`<root>/<lineage>/target-<key>`),
|
|
||||||
# which is what lets two jobs on ONE ref hold two build directories and so
|
|
||||||
# not serialise on Cargo's exclusive lock. It is resolved through
|
|
||||||
# cache-root.sh rather than interpolated here so the name is validated —
|
|
||||||
# several otherwise-reasonable lineage names put their whole subtree out of
|
|
||||||
# reach of a pass that has to see it. Everything downstream reads the
|
|
||||||
# resolved value, and it is exported as CARGO_CACHE_ROOT so the publish
|
|
||||||
# action can check it agrees with its own inputs.
|
|
||||||
- id: resolve
|
- id: resolve
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
@@ -131,8 +102,6 @@ runs:
|
|||||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||||
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
|
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
|
||||||
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
|
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
|
||||||
CACHE_ROOT=$(bash "${SCRIPTS}/cache-root.sh" resolve \
|
|
||||||
"${{ inputs.cache-root }}" "${{ inputs.cache-lineage }}")
|
|
||||||
OWN_REF="${{ inputs.own-ref }}"
|
OWN_REF="${{ inputs.own-ref }}"
|
||||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||||
BASE_REF="${{ inputs.base-ref }}"
|
BASE_REF="${{ inputs.base-ref }}"
|
||||||
@@ -147,7 +116,7 @@ runs:
|
|||||||
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
|
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
TARGET_DIR="${CACHE_ROOT}/target-${OWN_KEY}"
|
TARGET_DIR="${{ inputs.cache-root }}/target-${OWN_KEY}"
|
||||||
WATERMARK="${{ inputs.watermark-file }}"
|
WATERMARK="${{ inputs.watermark-file }}"
|
||||||
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
|
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
|
||||||
LOCK_ID="${{ inputs.lock-id }}"
|
LOCK_ID="${{ inputs.lock-id }}"
|
||||||
@@ -159,11 +128,10 @@ runs:
|
|||||||
echo "base-key=${BASE_KEY}"
|
echo "base-key=${BASE_KEY}"
|
||||||
echo "lock-id=${LOCK_ID}"
|
echo "lock-id=${LOCK_ID}"
|
||||||
echo "watermark-file=${WATERMARK}"
|
echo "watermark-file=${WATERMARK}"
|
||||||
echo "cache-root=${CACHE_ROOT}"
|
|
||||||
} >> "$GITHUB_OUTPUT"
|
} >> "$GITHUB_OUTPUT"
|
||||||
{
|
{
|
||||||
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
|
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
|
||||||
echo "CARGO_CACHE_ROOT=${CACHE_ROOT}"
|
echo "CARGO_CACHE_ROOT=${{ inputs.cache-root }}"
|
||||||
echo "CARGO_CACHE_KEY=${OWN_KEY}"
|
echo "CARGO_CACHE_KEY=${OWN_KEY}"
|
||||||
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
|
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
|
||||||
echo "CI_WATERMARK_FILE=${WATERMARK}"
|
echo "CI_WATERMARK_FILE=${WATERMARK}"
|
||||||
@@ -187,7 +155,7 @@ runs:
|
|||||||
bash "${SCRIPTS}/seed-target-dir.sh" \
|
bash "${SCRIPTS}/seed-target-dir.sh" \
|
||||||
"${{ steps.resolve.outputs.cache-key }}" \
|
"${{ steps.resolve.outputs.cache-key }}" \
|
||||||
"${{ steps.resolve.outputs.base-key }}" \
|
"${{ steps.resolve.outputs.base-key }}" \
|
||||||
"${{ steps.resolve.outputs.cache-root }}" \
|
"${{ inputs.cache-root }}" \
|
||||||
"${{ github.job }}-${{ github.run_id }}-$$" \
|
"${{ github.job }}-${{ github.run_id }}-$$" \
|
||||||
"${{ inputs.seed-fallback-dir }}" \
|
"${{ inputs.seed-fallback-dir }}" \
|
||||||
"${{ steps.resolve.outputs.lock-id }}"
|
"${{ steps.resolve.outputs.lock-id }}"
|
||||||
@@ -238,7 +206,7 @@ runs:
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||||
bash "${SCRIPTS}/prune-cache.sh" \
|
bash "${SCRIPTS}/prune-cache.sh" \
|
||||||
"${{ steps.resolve.outputs.cache-root }}" \
|
"${{ inputs.cache-root }}" \
|
||||||
"${{ steps.resolve.outputs.target-dir }}" \
|
"${{ steps.resolve.outputs.target-dir }}" \
|
||||||
"${{ inputs.protected-branches }}" \
|
"${{ inputs.protected-branches }}" \
|
||||||
"${{ inputs.min-free-percent }}"
|
"${{ inputs.min-free-percent }}"
|
||||||
|
|||||||
@@ -114,106 +114,6 @@ cache_key() {
|
|||||||
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
|
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
|
||||||
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
|
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Cache lineages
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
#
|
|
||||||
# A cache key names a REF. What a target directory holds is the product of a
|
|
||||||
# ref and a BUILD CONFIGURATION, and the two are not the same thing: emowheel
|
|
||||||
# builds the same ref twice on every push, once for the host and once for
|
|
||||||
# wasm32, in two jobs that run concurrently. Keyed on the ref alone both
|
|
||||||
# resolve to one CARGO_TARGET_DIR, and Cargo's build-directory lock is
|
|
||||||
# exclusive — so the second job sits on `Blocking waiting for file lock on
|
|
||||||
# build directory` for the length of the first, holding a runner capacity slot
|
|
||||||
# while doing nothing (daniel/gitdan#60).
|
|
||||||
#
|
|
||||||
# A lineage is that second dimension, and it is expressed as ONE DIRECTORY
|
|
||||||
# LEVEL above the per-ref directories rather than as a suffix on the key:
|
|
||||||
#
|
|
||||||
# <cache-root>/target-<key> no lineage (the flat layout)
|
|
||||||
# <cache-root>/<lineage>/target-<key> a lineage
|
|
||||||
#
|
|
||||||
# Nesting rather than suffixing is what keeps every existing reader correct
|
|
||||||
# without teaching any of them a new name shape. `prune-cache.sh` resolves
|
|
||||||
# liveness by recomputing `target-<cache_key(branch)>` for every branch on
|
|
||||||
# origin and evicting whatever does not match — a suffixed `target-<key>-wasm32`
|
|
||||||
# matches nothing, so it would be classified dead and unconditionally evicted
|
|
||||||
# on every single run. The host-level arbiter in daniel/gitdan reads the same
|
|
||||||
# shape (its BRANCH_DIR_RE), and a suffixed name falls out of it too: not
|
|
||||||
# evicted there, but never a candidate either, so a whole lineage becomes
|
|
||||||
# invisible to the global disk budget. Nesting leaves both matchers reading
|
|
||||||
# exactly the names they already read, one directory deeper.
|
|
||||||
#
|
|
||||||
# ONE LEVEL, AND NOT TWO. The arbiter walks a volume to CI_CACHE_MAX_DEPTH,
|
|
||||||
# which is 2 — `_data/target-<key>` and `_data/<lineage>/target-<key>`. It is
|
|
||||||
# kept tight there on purpose (a deeper walk starts meeting Cargo's own
|
|
||||||
# `incremental/<crate>-<hash>` directories, which match the same name shape and
|
|
||||||
# must never be evicted individually), so a lineage is a single path component
|
|
||||||
# and validate_cache_lineage refuses one containing a slash.
|
|
||||||
|
|
||||||
# Directory names daniel/gitdan's ci-cache-reclaim.sh refuses to descend into
|
|
||||||
# (its CI_CACHE_NODESCEND_NAMES). A lineage named one of these puts its whole
|
|
||||||
# subtree outside the global arbiter's reach: the caches accumulate and the one
|
|
||||||
# script whose job is the shared disk budget cannot see them.
|
|
||||||
CACHE_LINEAGE_RESERVED_NAMES="debug release deps incremental build .fingerprint tmp examples doc"
|
|
||||||
|
|
||||||
# The shape that same script reads as a per-branch cache directory (its
|
|
||||||
# BRANCH_DIR_RE). A lineage matching it is taken for a cache dir in its own
|
|
||||||
# right — never descended into, and an eviction candidate whole, which is the
|
|
||||||
# entire lineage rather than one ref's share of it.
|
|
||||||
CACHE_LINEAGE_BRANCH_DIR_RE='^.+-[0-9a-f]{7,40}$'
|
|
||||||
|
|
||||||
# Every rejection below names the reader that imposes it, because that is the
|
|
||||||
# only way the constraint survives: none of these is a filesystem limit, and a
|
|
||||||
# name that trips one produces no error anywhere — it produces a lineage that
|
|
||||||
# silently stops being pruned, or silently stops being reclaimed.
|
|
||||||
validate_cache_lineage() {
|
|
||||||
local lineage="$1" reserved
|
|
||||||
[ -n "$lineage" ] || return 0
|
|
||||||
|
|
||||||
case "$lineage" in
|
|
||||||
*/*)
|
|
||||||
echo "::error::cache lineage '${lineage}' must be a single path component: the host-level arbiter walks a cache volume to depth 2, so <cache-root>/<lineage>/target-<key> is as deep as a cache directory may sit and still be reclaimable" >&2
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
.*)
|
|
||||||
echo "::error::cache lineage '${lineage}' must not start with a dot: every dot-prefixed entry under a cache root belongs to the leftover-naming contract (see the top of this file), and a lineage is not garbage to be reclaimed" >&2
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
target-* | snapshot-*)
|
|
||||||
echo "::error::cache lineage '${lineage}' must not start with 'target-' or 'snapshot-': prune-cache.sh globs both prefixes at the cache root, so the lineage directory itself would become an eviction candidate" >&2
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
*[!A-Za-z0-9._-]*)
|
|
||||||
echo "::error::cache lineage '${lineage}' may contain only [A-Za-z0-9._-] — the same charset cache_key() sanitises a ref down to" >&2
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
for reserved in $CACHE_LINEAGE_RESERVED_NAMES; do
|
|
||||||
if [ "$lineage" = "$reserved" ]; then
|
|
||||||
echo "::error::cache lineage '${lineage}' is one of the Cargo directory names daniel/gitdan's ci-cache-reclaim.sh never descends into (CI_CACHE_NODESCEND_NAMES) — every cache under it would be invisible to the host-level disk budget" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
if [[ $lineage =~ $CACHE_LINEAGE_BRANCH_DIR_RE ]]; then
|
|
||||||
echo "::error::cache lineage '${lineage}' ends in a hex suffix, which is the shape daniel/gitdan's ci-cache-reclaim.sh reads as a per-branch cache directory — it would treat the lineage directory as one cache and evict the whole thing" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
# The cache root a lineage's directories actually live under. An empty lineage
|
|
||||||
# resolves to the cache root unchanged, byte for byte: that is what makes this
|
|
||||||
# a no-op for every consumer that does not set one, rather than a migration.
|
|
||||||
cache_root_for() {
|
|
||||||
local root="$1" lineage="${2:-}"
|
|
||||||
validate_cache_lineage "$lineage" || return 1
|
|
||||||
printf '%s%s' "$root" "${lineage:+/$lineage}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Disk accounting
|
# Disk accounting
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -310,9 +210,6 @@ _unshare_files() {
|
|||||||
# The inner shell propagates a failure of any individual copy-and-rename out
|
# The inner shell propagates a failure of any individual copy-and-rename out
|
||||||
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
||||||
# partially-unshared tree is reported rather than silently accepted.
|
# partially-unshared tree is reported rather than silently accepted.
|
||||||
# shellcheck disable=SC2016 # the quoted program is for the INNER shell: $f
|
|
||||||
# is its loop variable, $rc its accumulator and $$ its pid. Expanding any of
|
|
||||||
# them here is what the single quotes exist to prevent.
|
|
||||||
find "$@" -links +1 -print0 2>/dev/null |
|
find "$@" -links +1 -print0 2>/dev/null |
|
||||||
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,194 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Regression test for cache-root.sh and the lineage rules in cache-lib.sh —
|
|
||||||
# the fix for daniel/gitdan#60, where two jobs building the same ref for
|
|
||||||
# different targets resolved to one CARGO_TARGET_DIR and serialised on Cargo's
|
|
||||||
# exclusive build-directory lock.
|
|
||||||
#
|
|
||||||
# 1. NO LINEAGE CHANGES NOTHING — the effective root is the cache root byte
|
|
||||||
# for byte, so every consumer that does not set a lineage keeps the exact
|
|
||||||
# directories it already has on the volume. This is the whole of the
|
|
||||||
# migration story for lublub, zemyna and emowheel's `ci` job, so it is
|
|
||||||
# asserted rather than assumed.
|
|
||||||
# 2. TWO LINEAGES, ONE REF, TWO TARGET DIRS — the bug itself. The two jobs
|
|
||||||
# keep one cache key (they are the same ref) and still get directories
|
|
||||||
# that are neither equal nor nested one inside the other, which is what
|
|
||||||
# Cargo's per-directory lock needs in order not to serialise them.
|
|
||||||
# 3. THE WHOLE PIPELINE MOVES TOGETHER — seed, publish and prune all operate
|
|
||||||
# inside the lineage root. A pass run in one lineage must not evict, or
|
|
||||||
# even see, a sibling lineage's caches or the flat layout's.
|
|
||||||
# 4. BASE SEEDING IS PER LINEAGE — a PR branch layers over ITS OWN lineage's
|
|
||||||
# base snapshot, not over whatever the flat root happens to hold. This is
|
|
||||||
# the property that keeps a warm start for both jobs rather than one.
|
|
||||||
# 5. A NAME NO READER CAN HANDLE IS REFUSED AT RESOLVE TIME — one assertion
|
|
||||||
# per constraint, each named for the reader that imposes it. None of these
|
|
||||||
# is a filesystem limit: every one of them produces a working directory
|
|
||||||
# that some pass silently stops seeing, which is the failure mode this
|
|
||||||
# whole scheme exists to avoid rather than to relocate.
|
|
||||||
# 6. A PUBLISH THAT DISAGREES WITH ITS CONSUME STEP FAILS LOUDLY — the
|
|
||||||
# footgun the lineage input introduces. cargo-cache-publish derives both
|
|
||||||
# ends of the snapshot swap from its own `cache-root`, so a publish step
|
|
||||||
# left at the default while its consume step nested would republish the
|
|
||||||
# OTHER lineage's live target dir over that lineage's snapshot, silently.
|
|
||||||
set -euo pipefail
|
|
||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|
||||||
. "$script_dir/cache-lib.sh"
|
|
||||||
|
|
||||||
scratch=$(mktemp -d)
|
|
||||||
trap 'rm -rf "$scratch"' EXIT
|
|
||||||
root="$scratch/cache"; mkdir -p "$root"
|
|
||||||
pass_count=0
|
|
||||||
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
|
||||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
|
||||||
|
|
||||||
resolve() { bash "$script_dir/cache-root.sh" resolve "$@"; }
|
|
||||||
|
|
||||||
# A tree that looks enough like a Cargo target dir for the pipeline scripts,
|
|
||||||
# with a marker naming which lineage produced it — scenario 4 turns on reading
|
|
||||||
# that marker back out of a seeded directory.
|
|
||||||
make_tree() {
|
|
||||||
local d="$1" marker="$2"
|
|
||||||
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x"
|
|
||||||
echo "$marker" > "$d/debug/deps/libx.rlib"
|
|
||||||
echo "$marker" > "$d/lineage-marker"
|
|
||||||
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
|
||||||
}
|
|
||||||
|
|
||||||
echo "=== 1. no lineage changes nothing ==="
|
|
||||||
|
|
||||||
[ "$(resolve /cache)" = /cache ] || fail "an omitted lineage changed the root"
|
|
||||||
[ "$(resolve /cache '')" = /cache ] || fail "an empty lineage changed the root"
|
|
||||||
ok "no lineage resolves to the cache root unchanged"
|
|
||||||
|
|
||||||
KEY=$(cache_key dev)
|
|
||||||
[ "$(target_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/target-$KEY" ] \
|
|
||||||
|| fail "the flat target-dir name moved"
|
|
||||||
[ "$(snapshot_dir_for "$(resolve /cache '')" "$KEY")" = "/cache/snapshot-$KEY" ] \
|
|
||||||
|| fail "the flat snapshot name moved"
|
|
||||||
ok "the flat layout's directory names are untouched"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 2. two lineages, one ref, two target dirs ==="
|
|
||||||
|
|
||||||
HOST_ROOT=$(resolve /cache)
|
|
||||||
WASM_ROOT=$(resolve /cache wasm32)
|
|
||||||
[ "$WASM_ROOT" = /cache/wasm32 ] || fail "lineage root resolved to '$WASM_ROOT'"
|
|
||||||
HOST_DIR=$(target_dir_for "$HOST_ROOT" "$KEY")
|
|
||||||
WASM_DIR=$(target_dir_for "$WASM_ROOT" "$KEY")
|
|
||||||
[ "$HOST_DIR" != "$WASM_DIR" ] || fail "both lineages resolved to $HOST_DIR"
|
|
||||||
case "$WASM_DIR" in "$HOST_DIR"/*) fail "the wasm target dir sits inside the host one" ;; esac
|
|
||||||
case "$HOST_DIR" in "$WASM_DIR"/*) fail "the host target dir sits inside the wasm one" ;; esac
|
|
||||||
ok "one cache key ($KEY), two disjoint target dirs: $HOST_DIR and $WASM_DIR"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 3. the whole pipeline moves together ==="
|
|
||||||
|
|
||||||
DEAD=$(cache_key feat/dead)
|
|
||||||
lin="$root/wasm32"
|
|
||||||
mkdir -p "$lin"
|
|
||||||
make_tree "$root/target-$DEAD" flat
|
|
||||||
make_tree "$root/native/target-$DEAD" native
|
|
||||||
make_tree "$lin/target-$DEAD" wasm32
|
|
||||||
bash "$script_dir/seed-target-dir.sh" "$KEY" "" "$lin" tag-3 > "$scratch/seed3.log" 2>&1 \
|
|
||||||
|| { cat "$scratch/seed3.log"; fail "seed inside a lineage root failed"; }
|
|
||||||
[ -d "$lin/target-$KEY" ] || fail "seed did not create $lin/target-$KEY"
|
|
||||||
[ -d "$root/target-$KEY" ] && fail "seed created a directory at the flat root as well"
|
|
||||||
ok "seed creates its directory under the lineage root and nowhere else"
|
|
||||||
|
|
||||||
make_tree "$lin/target-$KEY" wasm32
|
|
||||||
bash "$script_dir/publish-snapshot.sh" "$KEY" "$lin" tag-3 > "$scratch/pub3.log" 2>&1 \
|
|
||||||
|| { cat "$scratch/pub3.log"; fail "publish inside a lineage root failed"; }
|
|
||||||
[ -d "$lin/snapshot-$KEY" ] || fail "publish did not create $lin/snapshot-$KEY"
|
|
||||||
[ -d "$root/snapshot-$KEY" ] && fail "publish created a snapshot at the flat root as well"
|
|
||||||
ok "publish writes its snapshot under the lineage root and nowhere else"
|
|
||||||
|
|
||||||
# Free space far below the threshold, so pass 2 evicts every eligible
|
|
||||||
# directory it can see. What it can see is the point of the scenario.
|
|
||||||
CACHE_LIVENESS=false CACHE_DF_OVERRIDE="1000000 1000" \
|
|
||||||
bash "$script_dir/prune-cache.sh" "$lin" "$lin/target-$KEY" 'dev main' 10 \
|
|
||||||
> "$scratch/prune3.log" 2>&1 || { cat "$scratch/prune3.log"; fail "prune inside a lineage root failed"; }
|
|
||||||
[ -d "$lin/target-$DEAD" ] && { cat "$scratch/prune3.log"; fail "prune left its own lineage's evictable cache in place"; }
|
|
||||||
[ -d "$root/target-$DEAD" ] || fail "prune reached out of its lineage and evicted the flat root's cache"
|
|
||||||
[ -d "$root/native/target-$DEAD" ] || fail "prune reached into a sibling lineage and evicted its cache"
|
|
||||||
ok "prune under disk pressure evicts inside its own lineage only"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 4. base seeding is per lineage ==="
|
|
||||||
|
|
||||||
BASE=$(cache_key dev)
|
|
||||||
PR=$(cache_key feat/pr)
|
|
||||||
rm -rf "$lin" "$root/snapshot-$BASE"
|
|
||||||
mkdir -p "$lin"
|
|
||||||
make_tree "$root/snapshot-$BASE" flat-base
|
|
||||||
make_tree "$lin/snapshot-$BASE" wasm32-base
|
|
||||||
bash "$script_dir/seed-target-dir.sh" "$PR" "$BASE" "$lin" tag-4 > "$scratch/seed4.log" 2>&1 \
|
|
||||||
|| { cat "$scratch/seed4.log"; fail "seeding a PR branch inside a lineage failed"; }
|
|
||||||
[ -d "$lin/target-$PR" ] || fail "the PR branch's lineage target dir was not created"
|
|
||||||
got=$(cat "$lin/target-$PR/lineage-marker")
|
|
||||||
[ "$got" = wasm32-base ] || fail "the PR branch layered over '$got', not its own lineage's base snapshot"
|
|
||||||
grep -q 'base snapshot' "$scratch/seed4.log" || { cat "$scratch/seed4.log"; fail "seed did not report a base-snapshot clone"; }
|
|
||||||
ok "a PR branch layers over its own lineage's base snapshot ($got)"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 5. a name no reader can handle is refused ==="
|
|
||||||
|
|
||||||
reject() {
|
|
||||||
local lineage="$1" want="$2" desc="$3" out
|
|
||||||
if out=$(resolve /cache "$lineage" 2>&1); then
|
|
||||||
fail "lineage '$lineage' was accepted (resolved to '$out') — $desc"
|
|
||||||
fi
|
|
||||||
case "$out" in
|
|
||||||
*"$want"*) ;;
|
|
||||||
*) fail "lineage '$lineage' was rejected without naming '$want': $out" ;;
|
|
||||||
esac
|
|
||||||
ok "rejected '$lineage' — $desc"
|
|
||||||
}
|
|
||||||
|
|
||||||
reject 'a/b' 'single path component' "the arbiter walks a volume to depth 2"
|
|
||||||
reject '.hidden' 'dot' "every dot-prefixed name under a cache root is leftover-contract territory"
|
|
||||||
reject 'target-x' 'prune-cache.sh' "prune-cache.sh globs target-* at the cache root"
|
|
||||||
reject 'snapshot-x' 'prune-cache.sh' "prune-cache.sh globs snapshot-* at the cache root"
|
|
||||||
reject 'wasm 32' 'A-Za-z0-9._-' "a cache key is sanitised to that charset and a lineage sits beside one"
|
|
||||||
reject 'release' 'CI_CACHE_NODESCEND_NAMES' "the arbiter never descends into a Cargo profile name"
|
|
||||||
reject 'doc' 'CI_CACHE_NODESCEND_NAMES' "same, for the docs profile directory"
|
|
||||||
reject 'lineage-deadbeef' 'per-branch cache directory' "the arbiter reads a hex-suffixed name as one cache dir"
|
|
||||||
|
|
||||||
for good in wasm32 web android host wasm32.release lineage_2; do
|
|
||||||
out=$(resolve /cache "$good") || fail "lineage '$good' was rejected: $out"
|
|
||||||
[ "$out" = "/cache/$good" ] || fail "lineage '$good' resolved to '$out'"
|
|
||||||
done
|
|
||||||
ok "ordinary lineage names still resolve"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 6. a publish that disagrees with its consume step fails loudly ==="
|
|
||||||
|
|
||||||
verify() { bash "$script_dir/cache-root.sh" verify "$@"; }
|
|
||||||
|
|
||||||
verify /cache wasm32 /cache/wasm32 > /dev/null 2>&1 \
|
|
||||||
|| fail "verify rejected a publish step that agrees with its consume step"
|
|
||||||
verify /cache '' /cache > /dev/null 2>&1 \
|
|
||||||
|| fail "verify rejected an unmigrated consumer's matching default pair"
|
|
||||||
ok "verify accepts a publish step whose inputs match what the consume step exported"
|
|
||||||
|
|
||||||
# The exact shape of the mistake: the consume step nested, the publish step
|
|
||||||
# kept the default. Left unchecked this republishes the host lineage's live
|
|
||||||
# target dir over the host lineage's snapshot, from the wasm job.
|
|
||||||
if out=$(verify /cache '' /cache/wasm32 2>&1); then
|
|
||||||
fail "verify accepted a publish step that resolved to /cache while the job exported /cache/wasm32"
|
|
||||||
fi
|
|
||||||
case "$out" in
|
|
||||||
*'/cache/wasm32'*) ;;
|
|
||||||
*) fail "the mismatch error does not quote what the consume step exported: $out" ;;
|
|
||||||
esac
|
|
||||||
case "$out" in
|
|
||||||
*'SAME cache-root and cache-lineage'*) ;;
|
|
||||||
*) fail "the mismatch error does not say what to do about it: $out" ;;
|
|
||||||
esac
|
|
||||||
ok "verify rejects a publish step that forgot the lineage, and says so"
|
|
||||||
|
|
||||||
if verify /cache 'a/b' /cache/a/b > /dev/null 2>&1; then
|
|
||||||
fail "verify accepted an invalid lineage as long as both sides agreed on it"
|
|
||||||
fi
|
|
||||||
ok "verify validates the lineage as well as comparing it"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "cache-root-selftest: ${pass_count} assertions passed"
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Resolves — and cross-checks — the cache root a job's directories live under.
|
|
||||||
#
|
|
||||||
# cache-root.sh resolve <cache-root> [lineage]
|
|
||||||
# cache-root.sh verify <cache-root> <lineage> <exported-root>
|
|
||||||
#
|
|
||||||
# `resolve` prints the effective root: the cache root unchanged when no lineage
|
|
||||||
# is given, or `<cache-root>/<lineage>` when one is. Invalid lineage names are
|
|
||||||
# rejected here rather than downstream — see validate_cache_lineage() in
|
|
||||||
# cache-lib.sh, where every rejection names the reader that imposes it.
|
|
||||||
#
|
|
||||||
# `verify` is the publish side's guard. cargo-cache-publish resolves the same
|
|
||||||
# two inputs the consume action was given and compares the result against the
|
|
||||||
# CARGO_CACHE_ROOT the consume step exported into the job environment. The two
|
|
||||||
# actions have always had to agree — `cache-root`'s description in the publish
|
|
||||||
# action says "must match the consume action" — and until a lineage existed
|
|
||||||
# they always did, because nobody overrode the default. A disagreement is not
|
|
||||||
# a harmless no-op: publish-snapshot.sh takes the root as an argument and
|
|
||||||
# derives BOTH ends of the swap from it, so a publish step that kept the
|
|
||||||
# default while its consume step nested would read `<root>/target-<key>` — the
|
|
||||||
# OTHER lineage's live target dir — and republish it over `<root>/snapshot-<key>`,
|
|
||||||
# which is that lineage's snapshot. Two jobs would then be publishing one
|
|
||||||
# snapshot from one tree on every push, and nothing in either action would say
|
|
||||||
# so. Hence: fail the job, loudly, rather than resolve the ambiguity in
|
|
||||||
# either direction.
|
|
||||||
#
|
|
||||||
# A thin CLI over cache-lib.sh, kept as its own entry point for the same
|
|
||||||
# reason branch-cache-key.sh is: an out-of-band job that needs to find a
|
|
||||||
# lineage's directories should resolve the path the way the action does
|
|
||||||
# instead of reimplementing the rule.
|
|
||||||
set -euo pipefail
|
|
||||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
|
||||||
|
|
||||||
MODE="${1:-}"
|
|
||||||
case "$MODE" in
|
|
||||||
resolve)
|
|
||||||
[ $# -ge 2 ] && [ $# -le 3 ] || {
|
|
||||||
echo "::error::cache-root.sh resolve: expected <cache-root> [lineage]" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
[ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; }
|
|
||||||
cache_root_for "$2" "${3:-}"
|
|
||||||
;;
|
|
||||||
verify)
|
|
||||||
[ $# -eq 4 ] || {
|
|
||||||
echo "::error::cache-root.sh verify: expected <cache-root> <lineage> <exported-root>" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
[ -n "$2" ] || { echo "::error::cache-root.sh: cache-root must not be empty" >&2; exit 1; }
|
|
||||||
expected=$(cache_root_for "$2" "$3")
|
|
||||||
if [ "$expected" != "$4" ]; then
|
|
||||||
echo "::error::cache-root.sh: this step resolves its cache root to '${expected}' (cache-root '$2', cache-lineage '$3') but the cargo-cache step in this job exported '$4'. Pass the SAME cache-root and cache-lineage to both actions." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "cache root: ${expected} (agrees with the cargo-cache step in this job)"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "::error::cache-root.sh: unknown mode '${MODE}' (expected resolve or verify)" >&2
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
@@ -133,7 +133,7 @@ hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_
|
|||||||
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
||||||
shared=0; unshared=0
|
shared=0; unshared=0
|
||||||
while IFS= read -r f; do
|
while IFS= read -r f; do
|
||||||
rel="${f#"$base_fix"/}"
|
rel="${f#$base_fix/}"
|
||||||
[ -e "$clone_fix/$rel" ] || continue
|
[ -e "$clone_fix/$rel" ] || continue
|
||||||
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
||||||
case "$rel" in
|
case "$rel" in
|
||||||
|
|||||||
@@ -122,7 +122,7 @@ assert_log_lacks() {
|
|||||||
# every run — simulate that before each restore-mtimes.sh pass, exactly as
|
# every run — simulate that before each restore-mtimes.sh pass, exactly as
|
||||||
# CI would see it, so this test exercises the script the same way CI does.
|
# CI would see it, so this test exercises the script the same way CI does.
|
||||||
stamp_checkout_now() {
|
stamp_checkout_now() {
|
||||||
find . -path ./.git -prune -o -type f -print0 | xargs -0 touch
|
find . -path ./.git -prune -o -type f -print | xargs touch
|
||||||
}
|
}
|
||||||
|
|
||||||
echo "=== building scratch workspace ==="
|
echo "=== building scratch workspace ==="
|
||||||
|
|||||||
@@ -74,17 +74,6 @@
|
|||||||
# onto one set of fingerprints. That is the silent stale-reuse bug the
|
# onto one set of fingerprints. That is the silent stale-reuse bug the
|
||||||
# whole scheme exists to prevent, so the failure has to abort the clone
|
# whole scheme exists to prevent, so the failure has to abort the clone
|
||||||
# rather than be swallowed.
|
# rather than be swallowed.
|
||||||
# 11. THE CONSUMER'S MARKER ACTUALLY STOPS THE PUBLISHER — reader_lock_acquire
|
|
||||||
# is exercised (not faked, unlike publish-snapshot-selftest.sh's scenario
|
|
||||||
# 6) by a real hardlink_clone_into racing a real, concurrent
|
|
||||||
# publish-snapshot.sh republish of the exact snapshot being cloned. Pins
|
|
||||||
# that the publisher OBSERVABLY WAITS on this consumer's marker — its own
|
|
||||||
# log reports entering the drain wait — rather than only that the run
|
|
||||||
# succeeds, which stayed green with the marker call deleted (issue #10).
|
|
||||||
# Kept as its own scenario, not folded into 8a, because 8a already pins
|
|
||||||
# exactly one property (the identity check) for exactly one mutant, and
|
|
||||||
# the suite's one-scenario-one-mutant diagonal across 8a to 8d and 10 is
|
|
||||||
# deliberate.
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
. "$script_dir/cache-lib.sh"
|
. "$script_dir/cache-lib.sh"
|
||||||
@@ -646,84 +635,5 @@ leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \
|
|||||||
ok "no staging or reader-marker scratch left behind"
|
ok "no staging or reader-marker scratch left behind"
|
||||||
rm -f "$scratch/bin/cp"
|
rm -f "$scratch/bin/cp"
|
||||||
|
|
||||||
echo
|
|
||||||
echo "=== 11: the consumer's marker actually stops the publisher ==="
|
|
||||||
# hardlink_clone_into's reader_lock_acquire call is exercised here, not
|
|
||||||
# faked. publish-snapshot-selftest.sh's scenario 6 stands a hand-written
|
|
||||||
# marker file in for "a consumer whose clone outlasts the grace period" — a
|
|
||||||
# deliberate simplification that does not need this consumer's clone code at
|
|
||||||
# all, so it cannot tell reader_lock_acquire apart from no marker existing.
|
|
||||||
# This scenario forces the two real scripts to race on the same snapshot: a
|
|
||||||
# consumer hardlink-cloning it, and a publisher republishing it out from
|
|
||||||
# under that clone, exactly as 8a does to force the identity check — but what
|
|
||||||
# is pinned here is not the consumer's response to the rotation (8a's
|
|
||||||
# property), it is that the PUBLISHER, reading the SAME marker this
|
|
||||||
# consumer's clone wrote, is observed to have entered its drain wait. Deleting
|
|
||||||
# reader_lock_acquire (issue #10) leaves the marker never written: the
|
|
||||||
# publisher's live_reader_count sees zero readers on its first check and
|
|
||||||
# proceeds straight to reclaiming the rotated generation, silently — the run
|
|
||||||
# still succeeds, and nothing about its own outcome says so, only the absence
|
|
||||||
# of a line in the publisher's log.
|
|
||||||
INTLK=$(cache_key feat/interlock-consumer)
|
|
||||||
INTLK_BASE=$(cache_key release/5)
|
|
||||||
TAG_INTLK=jobIntlk
|
|
||||||
rm -rf "$root/snapshot-$INTLK_BASE" "$root/target-$INTLK_BASE"
|
|
||||||
make_tree "$root/snapshot-$INTLK_BASE" intlk-gen1
|
|
||||||
make_tree "$root/target-$INTLK_BASE" intlk-gen2
|
|
||||||
intlk_gen1_inode=$(stat -c '%i' "$root/snapshot-$INTLK_BASE")
|
|
||||||
|
|
||||||
cat > "$scratch/bin/cp" <<EOF
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
# Fires once, only on the consumer's own top-level hardlink clone — identified
|
|
||||||
# by its destination, the seed's private staging path. Every other cp in the
|
|
||||||
# process tree (the unshare copies, and the publisher's own staging clone)
|
|
||||||
# falls through to the real one.
|
|
||||||
if [ "\${@: -1}" = "$root/.stage-$TAG_INTLK" ] && [ ! -e "$scratch/firedIntlk" ]; then
|
|
||||||
: > "$scratch/firedIntlk"
|
|
||||||
rc=0; "$real_cp" "\$@" || rc=\$?
|
|
||||||
# Concurrently: a real, second publish of the exact snapshot this consumer
|
|
||||||
# is cloning — the republish that, without the marker this consumer's clone
|
|
||||||
# holds, would reclaim the generation out from under it.
|
|
||||||
( CACHE_READ_GRACE_SECONDS=10 bash "$script_dir/publish-snapshot.sh" "$INTLK_BASE" "$root" pubIntlk \
|
|
||||||
> "$scratch/logPubIntlk" 2>&1
|
|
||||||
echo \$? > "$scratch/rcPubIntlk" ) &
|
|
||||||
# Hand control back only once the swap is on disk, so the identity read
|
|
||||||
# immediately after this cp is guaranteed to resolve to the new generation
|
|
||||||
# — the same technique 8a uses to force the interleaving rather than hope
|
|
||||||
# for it. reader_lock_release does not run until AFTER this script exits,
|
|
||||||
# so the marker stays live for the publisher's whole swap-and-scan.
|
|
||||||
deadline=\$(( \$(date +%s) + 60 ))
|
|
||||||
while [ "\$(stat -c '%i' "$root/snapshot-$INTLK_BASE" 2>/dev/null)" = "$intlk_gen1_inode" ]; do
|
|
||||||
[ "\$(date +%s)" -lt "\$deadline" ] || { echo "stub cp: the publisher never swapped the snapshot" >&2; exit 92; }
|
|
||||||
sleep 0.05
|
|
||||||
done
|
|
||||||
exit \$rc
|
|
||||||
fi
|
|
||||||
exec "$real_cp" "\$@"
|
|
||||||
EOF
|
|
||||||
chmod +x "$scratch/bin/cp"
|
|
||||||
|
|
||||||
rcIntlk=0
|
|
||||||
seed_with_stub "$INTLK" "$INTLK_BASE" "$root" "$TAG_INTLK" > "$scratch/logIntlk" 2>&1 || rcIntlk=$?
|
|
||||||
[ -e "$scratch/firedIntlk" ] \
|
|
||||||
|| fail "the stubbed cp never fired: the consumer never raced the publisher, so this scenario proves nothing"
|
|
||||||
ok "the consumer's clone raced a real, concurrent republish of its own source"
|
|
||||||
wait_for_file "$scratch/rcPubIntlk" "the publisher never finished"
|
|
||||||
[ "$(cat "$scratch/rcPubIntlk")" = "0" ] || { tail -40 "$scratch/logPubIntlk"; fail "publish-snapshot.sh exited non-zero"; }
|
|
||||||
[ "$rcIntlk" = "0" ] || { tail -40 "$scratch/logIntlk"; fail "the seed exited non-zero"; }
|
|
||||||
ok "both sides of the race completed"
|
|
||||||
|
|
||||||
# The property under test: not that the run succeeded, but that the publisher
|
|
||||||
# itself reports having found a live reader and waited on it. This is silent
|
|
||||||
# in the consumer's own log and in the run's exit status alike — only the
|
|
||||||
# publisher's log carries it.
|
|
||||||
grep -q "readers: waiting for 1 in-flight clone(s) of snapshot-${INTLK_BASE}" "$scratch/logPubIntlk" \
|
|
||||||
|| { tail -40 "$scratch/logPubIntlk"; fail "the publisher never reported waiting on the consumer's reader marker — the interlock did not observably engage"; }
|
|
||||||
ok "the publisher observably waited on the consumer's own reader marker before reclaiming the rotated snapshot generation"
|
|
||||||
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)
|
|
||||||
[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
|
|
||||||
ok "no staging, reader-marker or deferred-generation scratch left behind"
|
|
||||||
rm -f "$scratch/bin/cp"
|
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
||||||
|
|||||||
+1
-1
@@ -13,7 +13,7 @@ script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
|||||||
FAST=0
|
FAST=0
|
||||||
[ "${1:-}" = "--fast" ] && FAST=1
|
[ "${1:-}" = "--fast" ] && FAST=1
|
||||||
|
|
||||||
FIXTURE_TESTS=(cache-root-selftest.sh seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh)
|
FIXTURE_TESTS=(seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh)
|
||||||
CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh)
|
CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh)
|
||||||
|
|
||||||
TESTS=("${FIXTURE_TESTS[@]}")
|
TESTS=("${FIXTURE_TESTS[@]}")
|
||||||
|
|||||||
Reference in New Issue
Block a user