feat(cargo-cache): hardlink-clone a per-ref Cargo cache from a published snapshot
Replaces the phase-0 resolution probe with the real actions, merging the two independent per-branch Cargo cache implementations on this forge into the design neither of them had. ## The merge - zemyna seeds a PR branch by `cp -al` hardlink clone (near-free: cost scales with inode count, not bytes) from the base branch's LIVE target dir — a torn read waiting for a second job slot (its own #911). - emowheel seeds from a PUBLISHED IMMUTABLE SNAPSHOT (no race by construction) but with `cp -a`, duplicating ~35 GB per branch. This ships hardlink-clone FROM a published snapshot: zemyna's cost profile, emowheel's soundness, and #911 closed structurally rather than by the runner happening to have one execution slot. ## The bug both implementations have A build inside a `cp -al` clone DOES mutate the directory it was cloned from. Cargo replaces real artifacts, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Measured set: `.fingerprint/<unit>/dep-<target>` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), `build/<pkg>/{output,root-output,out/**}`, `deps/*.d` and `<profile>/*.d`. The checksum-freshness case is a wrong answer, not a slow build: a PR clone rewrites the base's dep-info to describe the PR's sources while the base's cache still holds the artifact built from the base's; once the PR merges, the base's next run finds the checksums match, reports `Fresh`, and links a binary built from the pre-merge code. Reproduced end to end. Fix: hardlink the artifacts (the GB), real-copy the metadata (the MB) — about 3.7% of a 6.9 GB Bevy target dir, against 100% for a full copy. ## Contents - `cargo-cache/action.yml` — consume: resolve keys, seed from the base's snapshot via staging + one atomic rename, strip Cargo lock files, unshare the mutable paths, restore mtimes from git history, lock, prune. - `cargo-cache-publish/action.yml` — publish: record the build watermark, atomically republish the snapshot on a protected branch, release the lock (`mode: release-lock` for the `if: always()` step). - `scripts/` — all logic, so it is testable standalone; the YAML is wiring. - `scripts/*selftest.sh` + `selftest.sh` — five suites, 63 assertions, every fix paired with a control that reproduces the bug. All green locally. Eviction merges emowheel's liveness pass (dead branches pruned unconditionally, not gated on disk pressure) with LRU-under-pressure, but inverts the order within the pressure pass: `target-*` before `snapshot-*`, because a snapshot is hardlinked to everything cloned from it, so evicting one frees almost no real bytes while costing every future PR its warm start. restore-mtimes.sh is ported from emowheel (the watermark variant, which closes the merge hazard zemyna's copy still has) with its provenance de-projectised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
Claude Opus 5
@@ -1,3 +1,267 @@
|
||||
# gitea-actions
|
||||
# gitdan-actions
|
||||
|
||||
Shared Gitea Actions composite actions for gitdan repos (cargo-cache et al)
|
||||
Shared Gitea Actions composite actions for the gitdan forge.
|
||||
|
||||
Currently one thing, done properly: **`cargo-cache`** — a persistent,
|
||||
per-branch Cargo build cache for self-hosted Gitea runners, where a pull
|
||||
request's cache is a near-free hardlink clone of an immutable snapshot its
|
||||
base branch published.
|
||||
|
||||
> **Final home:** this repository will live at `daniel/gitdan-actions`. Pin
|
||||
> that path in `uses:` once the transfer completes.
|
||||
|
||||
---
|
||||
|
||||
## Why this exists
|
||||
|
||||
Two of this forge's Rust projects independently built the same idea and each
|
||||
got one half right.
|
||||
|
||||
| | seeding mechanism | seed source |
|
||||
|---|---|---|
|
||||
| project A | `cp -al` hardlink clone — near-free, cost scales with inode count, not bytes | the base branch's **live** target dir — races a build that is still writing |
|
||||
| project B | `cp -a` full copy — sound, but ~35 GB duplicated per branch | a **published immutable snapshot** — nothing ever writes it while it is read |
|
||||
|
||||
This action is the diagonal: **hardlink-clone from a published snapshot.**
|
||||
Cheap like A, sound like B. It also closes a latent race in A by construction
|
||||
(the seed is staged and swapped in with one atomic rename) rather than relying
|
||||
on the runner having a single execution slot.
|
||||
|
||||
One thing neither project had, and the reason the clone is not a plain
|
||||
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
|
||||
cloned from.** Cargo writes real artifacts by replacing them, but writes its
|
||||
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
|
||||
write, straight through the shared inode. Under
|
||||
`CARGO_UNSTABLE_CHECKSUM_FRESHNESS` the file that gets corrupted is
|
||||
`.fingerprint/<unit>/dep-<target>`, which holds the per-source checksums that
|
||||
decide freshness, and the failure is silent stale-artifact reuse rather than a
|
||||
slow build. `scripts/hardlink-clone-selftest.sh` reproduces it as an explicit
|
||||
control and asserts the fix. The fix is to hardlink the artifacts (the GB) and
|
||||
real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory,
|
||||
against 100% for a full copy.
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
```yaml
|
||||
name: CI
|
||||
on:
|
||||
push:
|
||||
branches: [main, dev]
|
||||
pull_request:
|
||||
branches: [main, dev]
|
||||
|
||||
jobs:
|
||||
ci:
|
||||
runs-on: ubuntu-latest
|
||||
# REQUIRED, and it cannot come from the action: `container.volumes` is a
|
||||
# job-level property, so the persistent cache volume must be declared
|
||||
# here. Use a volume name unique to this repository.
|
||||
container:
|
||||
volumes:
|
||||
- myrepo-ci-target:/cache
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# REQUIRED. The mtime restore walks every commit that ever touched a
|
||||
# tracked file; a depth-1 checkout makes every file resolve to the
|
||||
# tip commit and the cache stops working. The action fails loudly
|
||||
# rather than silently degrading if this is missing.
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
|
||||
with:
|
||||
protected-branches: 'dev main'
|
||||
|
||||
# ... toolchain, system deps, and the build itself. CARGO_TARGET_DIR is
|
||||
# already exported to the job environment by the step above.
|
||||
- run: cargo clippy --workspace --all-targets -- -D warnings
|
||||
- run: cargo test --workspace
|
||||
|
||||
# After the build succeeds: record the watermark, and publish a snapshot
|
||||
# if this run is a push to a protected branch.
|
||||
- uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
||||
|
||||
# Release this job's cache lock even when the build failed, so the
|
||||
# eviction pass does not have to wait out the staleness grace period.
|
||||
- if: always()
|
||||
uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
|
||||
with:
|
||||
mode: release-lock
|
||||
```
|
||||
|
||||
Recommended alongside it, in the workflow's `env:` block:
|
||||
|
||||
```yaml
|
||||
env:
|
||||
CARGO_INCREMENTAL: 0 # per-run bloat on a persistent volume
|
||||
CARGO_PROFILE_DEV_DEBUG: line-tables-only
|
||||
CARGO_PROFILE_TEST_DEBUG: line-tables-only
|
||||
# Nightly only. Content-addressed freshness instead of mtime-based — a
|
||||
# strictly stronger guarantee, complementary to the mtime restore (which
|
||||
# still covers directory-form `rerun-if-changed` build-script watches).
|
||||
CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
/cache/
|
||||
target-<key> one per ref. Where a build actually runs.
|
||||
snapshot-<key> one per publisher ref. Immutable between publishes;
|
||||
the only thing a consumer ever clones from.
|
||||
```
|
||||
|
||||
`<key>` is the ref sanitised to a safe path component, capped at 48
|
||||
characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not
|
||||
decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise
|
||||
share one directory.
|
||||
|
||||
**A pull request run** resolves its own key from `github.head_ref` (not
|
||||
`ref_name`, which on a `pull_request` event is a synthetic merge ref that
|
||||
changes on every push) and its base key from `github.base_ref`. If it has no
|
||||
directory yet, it hardlink-clones `snapshot-<base>` into a staging path, strips
|
||||
Cargo's lock files, real-copies everything Cargo writes in place, and renames
|
||||
the staging path into `target-<own>`.
|
||||
|
||||
**A push to a protected branch** has no base to layer over. It builds in its
|
||||
own directory and, if the build goes green, republishes it as
|
||||
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
|
||||
one in. Consumers only ever observe a complete snapshot or none at all.
|
||||
|
||||
**Concurrency.** Two jobs sharing one cache key each stage under their own tag
|
||||
and race on one atomic rename; the loser discards its staging copy. There is
|
||||
no window in which a partially-populated directory is visible under the final
|
||||
name — which means this does not depend on the runner having a single
|
||||
execution slot. Two jobs then building in the same directory is Cargo's own
|
||||
`.cargo-lock` territory, which is what that lock is for.
|
||||
|
||||
**Eviction** runs three passes: caches for branches that no longer exist on
|
||||
origin are removed unconditionally; then, only if free space is under the
|
||||
threshold, live caches are evicted oldest-first; then, as a last resort, this
|
||||
run's own cache. Protected refs and any cache held open by a running job are
|
||||
never candidates. Within the pressure pass, `target-*` directories are evicted
|
||||
before `snapshot-*` ones — the reverse of the obvious order, because a
|
||||
snapshot is hardlinked to everything cloned from it, so removing one frees
|
||||
almost no real bytes while costing every future PR its warm start.
|
||||
|
||||
**File mtimes.** `actions/checkout` stamps every file with "now", which makes
|
||||
every crate look changed to Cargo's mtime-based freshness check — a persistent
|
||||
target directory buys nothing without fixing that. Each tracked file is
|
||||
restored to the timestamp of the most recent commit that touched it, plus a
|
||||
watermark override: for any file that changed since *this cache's own last
|
||||
successful build*, "now" is stamped instead. That override is what makes a
|
||||
merge safe, since a merge can introduce a commit authored before this cache's
|
||||
last build, where the historically-correct mtime is exactly the wrong answer.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
### `cargo-cache`
|
||||
|
||||
| input | default | meaning |
|
||||
|---|---|---|
|
||||
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
|
||||
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
|
||||
| `min-free-percent` | `10` | prune when free space drops below this |
|
||||
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
|
||||
| `prune` | `true` | run the eviction pass |
|
||||
| `liveness-prune` | `true` | within eviction, remove caches for branches gone from origin |
|
||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
|
||||
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
|
||||
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
|
||||
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
|
||||
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |
|
||||
|
||||
Outputs: `target-dir`, `cache-key`, `seeded-from` (`own` \| `base-snapshot` \|
|
||||
`own-snapshot` \| `fallback-dir` \| `concurrent-peer` \| `cold`).
|
||||
|
||||
Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
||||
`CARGO_CACHE_KEY`, `CARGO_CACHE_LOCK_ID`, `CARGO_CACHE_SCRIPTS`,
|
||||
`CI_WATERMARK_FILE`.
|
||||
|
||||
### `cargo-cache-publish`
|
||||
|
||||
| input | default | meaning |
|
||||
|---|---|---|
|
||||
| `cache-root` | `/cache` | must match the consume action |
|
||||
| `protected-branches` | `dev main` | refs that publish snapshots |
|
||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
|
||||
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
|
||||
|
||||
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
|
||||
`dev` into `main` has `own-ref` `dev` and would otherwise publish a snapshot
|
||||
of a merge-preview build, which is not what `dev` is.
|
||||
|
||||
---
|
||||
|
||||
## Multiple jobs in one workflow
|
||||
|
||||
Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
|
||||
each need their **own** watermark file. A shared one breaks the moment two
|
||||
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
|
||||
and job B then reads that just-advanced value, computes an empty diff, and
|
||||
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
|
||||
already gives each job its own; only override `watermark-file` if you also
|
||||
override `lock-id`, and then keep both distinct per job.
|
||||
|
||||
---
|
||||
|
||||
## Constraints of this runner
|
||||
|
||||
- **The repository must be public.** act_runner fetches actions by anonymous
|
||||
git clone and has no credentialed-fetch option, so a private action
|
||||
repository simply fails to resolve. Nothing secret goes in here.
|
||||
- **`uses:` needs the absolute URL.** A bare `owner/repo` resolves against
|
||||
github.com, because Gitea's `DEFAULT_ACTIONS_URL` is unset — and it has to
|
||||
stay unset, or `actions/checkout`, `dtolnay/rust-toolchain` and
|
||||
`taiki-e/install-action` stop resolving.
|
||||
- **The runner pre-fetches every referenced action before running any step**,
|
||||
so a bad action reference fails the job at step 0 rather than where it is
|
||||
used.
|
||||
- **`container.volumes` is job-level** and cannot be set from inside a
|
||||
composite action. The consuming workflow declares it; see the quick start.
|
||||
- **The cache volume is ext4** — no reflink support, which is precisely why
|
||||
hardlinks are the mechanism that makes cloning cheap.
|
||||
|
||||
---
|
||||
|
||||
## Versioning
|
||||
|
||||
Pin `@v1`. It is a moving major tag: fixes and backward-compatible inputs move
|
||||
it forward, and anything that would break an existing consumer gets `v2`
|
||||
instead. Pin a commit SHA if you want a frozen version.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
bash scripts/selftest.sh # everything (~1 min; needs cargo)
|
||||
bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
||||
```
|
||||
|
||||
| suite | covers |
|
||||
|---|---|
|
||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key |
|
||||
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
|
||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
|
||||
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
||||
|
||||
Every suite runs the actual script, not a reimplementation of its logic, and
|
||||
every fix scenario is paired with a control that reproduces the bug — a
|
||||
scenario that passes either way proves nothing.
|
||||
|
||||
The action YAML holds no logic beyond wiring; everything testable lives in
|
||||
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
||||
the actions reach their shared scripts through
|
||||
`${{ github.action_path }}/../scripts`, which works because the runner clones
|
||||
the whole repository when it fetches an action.
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
name: 'Cargo cache (publish)'
|
||||
description: >-
|
||||
Records this run''s build watermark and, on a publisher branch, atomically
|
||||
republishes its target directory as the immutable snapshot that other
|
||||
branches'' caches are hardlink-cloned from.
|
||||
author: 'gitdan'
|
||||
|
||||
inputs:
|
||||
cache-root:
|
||||
description: 'Mount point of the persistent cache volume. Must match the consume action.'
|
||||
required: false
|
||||
default: '/cache'
|
||||
protected-branches:
|
||||
description: >-
|
||||
Space-separated refs that publish snapshots. A run whose own ref is not
|
||||
in this list records its watermark and skips publishing.
|
||||
required: false
|
||||
default: 'dev main'
|
||||
mode:
|
||||
description: >-
|
||||
publish — record the watermark, publish a snapshot if eligible,
|
||||
release this job''s cache lock (the normal call, after a
|
||||
green build).
|
||||
release-lock — release this job''s cache lock and do nothing else. Use
|
||||
in a final `if: always()` step so a failed run does not
|
||||
leave a lock behind for the staleness grace period.
|
||||
required: false
|
||||
default: 'publish'
|
||||
own-ref:
|
||||
description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.'
|
||||
required: false
|
||||
default: ''
|
||||
publish-on-events:
|
||||
description: >-
|
||||
Space-separated event names on which a publisher branch actually
|
||||
publishes. Defaults to `push` — a pull_request run never publishes,
|
||||
because its ref is not the reference branch even when it targets one.
|
||||
required: false
|
||||
default: 'push'
|
||||
record-watermark:
|
||||
description: >-
|
||||
Record this run''s HEAD as the build watermark for this target dir.
|
||||
True for PR runs too, not just publishers: a feature branch accumulates
|
||||
its own build history across several pushes and needs its own watermark.
|
||||
required: false
|
||||
default: 'true'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
# Everything here reads the environment the consume action exported, so a
|
||||
# workflow that forgets to run cargo-cache first fails loudly here rather
|
||||
# than silently publishing a snapshot of the wrong directory.
|
||||
- id: resolve
|
||||
shell: bash
|
||||
env:
|
||||
PROTECTED_BRANCHES: ${{ inputs.protected-branches }}
|
||||
PUBLISH_ON_EVENTS: ${{ inputs.publish-on-events }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# In release-lock mode this action is called from an `if: always()`
|
||||
# step, which can run after a failure that happened before the
|
||||
# cargo-cache action ever executed. Missing environment there means
|
||||
# "there is no lock to release", not an error worth failing the job a
|
||||
# second time over.
|
||||
if [ -z "${CARGO_CACHE_SCRIPTS:-}" ] || [ -z "${CARGO_TARGET_DIR:-}" ]; then
|
||||
if [ "${{ inputs.mode }}" = "release-lock" ]; then
|
||||
echo "cargo-cache-publish: no cache environment in this job — nothing to release"
|
||||
echo "publish=no" >> "$GITHUB_OUTPUT"
|
||||
echo "active=no" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "::error::cargo-cache-publish: run the cargo-cache action earlier in this job"
|
||||
exit 1
|
||||
fi
|
||||
: "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}"
|
||||
echo "active=yes" >> "$GITHUB_OUTPUT"
|
||||
|
||||
OWN_REF="${{ inputs.own-ref }}"
|
||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||
|
||||
# A publisher is a branch other branches layer over. Two conditions,
|
||||
# both required: its ref is in protected-branches, AND the event is one
|
||||
# where this ref really is the reference branch. A pull_request run
|
||||
# from `dev` into `main` has own-ref `dev` and would otherwise publish
|
||||
# a snapshot of a merge-preview build — which is not what `dev` is.
|
||||
PUBLISH=no
|
||||
for ref in $PROTECTED_BRANCHES; do
|
||||
[ "$ref" = "$OWN_REF" ] || continue
|
||||
for ev in $PUBLISH_ON_EVENTS; do
|
||||
[ "$ev" = "${{ github.event_name }}" ] && PUBLISH=yes
|
||||
done
|
||||
done
|
||||
echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT"
|
||||
echo "own-ref=${OWN_REF}" >> "$GITHUB_OUTPUT"
|
||||
echo "publisher check: ref '${OWN_REF}', event '${{ github.event_name }}' -> publish=${PUBLISH}"
|
||||
|
||||
# Ordered before the snapshot publish so a snapshot always carries a
|
||||
# watermark at least as new as the build it holds. Both steps sit after
|
||||
# the consuming job's build steps, so a run that fails an earlier gate
|
||||
# never reaches either: the watermark stays at the last GREEN build and a
|
||||
# red build can never overwrite a known-good snapshot.
|
||||
- if: ${{ steps.resolve.outputs.active == 'yes' && inputs.mode == 'publish' && inputs.record-watermark == 'true' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
bash "${CARGO_CACHE_SCRIPTS}/record-watermark.sh" \
|
||||
"$CARGO_TARGET_DIR" "${CI_WATERMARK_FILE:-.ci-watermark-sha}"
|
||||
|
||||
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||
"$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \
|
||||
"${{ github.job }}-${{ github.run_id }}-$$"
|
||||
|
||||
# Released in both modes. In `publish` mode this is the normal end-of-job
|
||||
# release; the separate `release-lock` call exists for `if: always()`, so a
|
||||
# failed run does not leave its lock sitting until the staleness grace
|
||||
# period expires.
|
||||
- if: ${{ steps.resolve.outputs.active == 'yes' }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${CARGO_CACHE_LOCK_ID:-}" ]; then
|
||||
echo "cargo-cache-publish: no lock id in the environment — nothing to release"
|
||||
exit 0
|
||||
fi
|
||||
bash "${CARGO_CACHE_SCRIPTS}/cache-lock.sh" release \
|
||||
"$CARGO_TARGET_DIR" "${CARGO_CACHE_LOCK_ID}"
|
||||
+183
-12
@@ -1,26 +1,197 @@
|
||||
name: 'Cargo CI cache'
|
||||
description: 'Per-branch layered Cargo target-dir cache for Gitea Actions runners.'
|
||||
name: 'Cargo cache (consume)'
|
||||
description: >-
|
||||
Per-ref Cargo target-dir cache for Gitea Actions runners: hardlink-clones
|
||||
this ref's target directory from its base branch's published, immutable
|
||||
snapshot, restores git-history file mtimes, and prunes the volume.
|
||||
author: 'gitdan'
|
||||
|
||||
inputs:
|
||||
cache-root:
|
||||
description: 'Mount point of the persistent cache volume.'
|
||||
description: 'Mount point of the persistent cache volume inside the job container.'
|
||||
required: false
|
||||
default: '/cache'
|
||||
protected-branches:
|
||||
description: 'Space-separated branches whose cache dirs are never evicted.'
|
||||
description: >-
|
||||
Space-separated refs that publish snapshots and are never evicted.
|
||||
These are the branches PR caches layer over.
|
||||
required: false
|
||||
default: 'dev main'
|
||||
min-free-percent:
|
||||
description: 'Prune when free space on the cache volume drops below this percentage.'
|
||||
required: false
|
||||
default: '10'
|
||||
restore-mtimes:
|
||||
description: >-
|
||||
Restore every tracked file''s mtime from git history. Requires a
|
||||
full-history checkout (fetch-depth: 0). Set to false only if the build
|
||||
does not use Cargo''s mtime-based freshness at all.
|
||||
required: false
|
||||
default: 'true'
|
||||
prune:
|
||||
description: 'Run the eviction pass (dead-branch liveness + disk pressure).'
|
||||
required: false
|
||||
default: 'true'
|
||||
liveness-prune:
|
||||
description: >-
|
||||
Within the prune pass, remove caches for branches that no longer exist
|
||||
on origin. Set to false on a runner that cannot reach origin.
|
||||
required: false
|
||||
default: 'true'
|
||||
own-ref:
|
||||
description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.'
|
||||
required: false
|
||||
default: ''
|
||||
base-ref:
|
||||
description: 'Override the ref to layer over. Defaults to github.base_ref (empty on push).'
|
||||
required: false
|
||||
default: ''
|
||||
seed-fallback-dir:
|
||||
description: >-
|
||||
Absolute path to seed from when no snapshot exists yet — a pre-existing
|
||||
flat cache directory during a migration. Optional.
|
||||
required: false
|
||||
default: ''
|
||||
watermark-file:
|
||||
description: >-
|
||||
Name of this job''s build-watermark file inside the target dir. MUST be
|
||||
distinct per job when two jobs share one cache key. Defaults to
|
||||
.ci-watermark-<job>-sha.
|
||||
required: false
|
||||
default: ''
|
||||
lock-id:
|
||||
description: 'Identifier for this job''s cache lock. Defaults to <job>-<run_id>.'
|
||||
required: false
|
||||
default: ''
|
||||
stale-lock-seconds:
|
||||
description: 'Age past which another job''s cache lock is treated as abandoned.'
|
||||
required: false
|
||||
default: '7200'
|
||||
|
||||
outputs:
|
||||
target-dir:
|
||||
description: 'Resolved CARGO_TARGET_DIR for this run.'
|
||||
value: ${{ steps.probe.outputs.target-dir }}
|
||||
description: 'Resolved CARGO_TARGET_DIR. Also exported to the job environment.'
|
||||
value: ${{ steps.resolve.outputs.target-dir }}
|
||||
cache-key:
|
||||
description: 'Sanitized cache key for this run''s own ref.'
|
||||
value: ${{ steps.resolve.outputs.cache-key }}
|
||||
seeded-from:
|
||||
description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.'
|
||||
value: ${{ steps.seed.outputs.seeded-from }}
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- id: probe
|
||||
# Resolves both cache keys and exports the environment every later step
|
||||
# (and the consuming job's own build steps) reads. Must run before
|
||||
# anything that touches CARGO_TARGET_DIR, which is why it is first.
|
||||
#
|
||||
# `head_ref || ref_name` rather than `ref_name` alone: on a pull_request
|
||||
# event `ref_name` is a synthetic merge-ref name that changes on every
|
||||
# push to the PR, so keying on it would give the same PR a different cache
|
||||
# directory every time — defeating the reuse this action exists to
|
||||
# provide. On a push event `head_ref` is empty and `ref_name` is the real
|
||||
# branch, which is what we want there.
|
||||
#
|
||||
# `base_ref` is populated only for pull_request events. A push run has
|
||||
# nothing to layer over: its own ref IS the reference branch. It
|
||||
# publishes, it does not consume.
|
||||
- id: resolve
|
||||
shell: bash
|
||||
run: |
|
||||
echo "PROBE-OK: composite action resolved and executed"
|
||||
echo " cache-root = ${{ inputs.cache-root }}"
|
||||
echo " protected-branches= ${{ inputs.protected-branches }}"
|
||||
echo "target-dir=${{ inputs.cache-root }}/probe" >> "$GITHUB_OUTPUT"
|
||||
echo "PROBE_ENV_WRITE=ok" >> "$GITHUB_ENV"
|
||||
set -euo pipefail
|
||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||
[ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; }
|
||||
echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV"
|
||||
OWN_REF="${{ inputs.own-ref }}"
|
||||
[ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}"
|
||||
BASE_REF="${{ inputs.base-ref }}"
|
||||
[ -n "$BASE_REF" ] || BASE_REF="${{ github.base_ref }}"
|
||||
|
||||
OWN_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$OWN_REF")
|
||||
BASE_KEY=""
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
BASE_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$BASE_REF")
|
||||
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY}; layering over base ref '${BASE_REF}' -> ${BASE_KEY}"
|
||||
else
|
||||
echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)"
|
||||
fi
|
||||
|
||||
TARGET_DIR="${{ inputs.cache-root }}/target-${OWN_KEY}"
|
||||
WATERMARK="${{ inputs.watermark-file }}"
|
||||
[ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha"
|
||||
LOCK_ID="${{ inputs.lock-id }}"
|
||||
[ -n "$LOCK_ID" ] || LOCK_ID="${{ github.job }}-${{ github.run_id }}"
|
||||
|
||||
{
|
||||
echo "target-dir=${TARGET_DIR}"
|
||||
echo "cache-key=${OWN_KEY}"
|
||||
echo "base-key=${BASE_KEY}"
|
||||
echo "lock-id=${LOCK_ID}"
|
||||
echo "watermark-file=${WATERMARK}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo "CARGO_TARGET_DIR=${TARGET_DIR}"
|
||||
echo "CARGO_CACHE_ROOT=${{ inputs.cache-root }}"
|
||||
echo "CARGO_CACHE_KEY=${OWN_KEY}"
|
||||
echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}"
|
||||
echo "CI_WATERMARK_FILE=${WATERMARK}"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
# Seeds this ref's target dir from the base's published snapshot. See
|
||||
# scripts/seed-target-dir.sh — the staging-then-atomic-rename is what
|
||||
# makes concurrent jobs sharing one cache key safe by construction rather
|
||||
# than by the runner happening to have a single execution slot.
|
||||
- id: seed
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||
bash "${SCRIPTS}/seed-target-dir.sh" \
|
||||
"${{ steps.resolve.outputs.cache-key }}" \
|
||||
"${{ steps.resolve.outputs.base-key }}" \
|
||||
"${{ inputs.cache-root }}" \
|
||||
"${{ github.job }}-${{ github.run_id }}-$$" \
|
||||
"${{ inputs.seed-fallback-dir }}"
|
||||
|
||||
# Marks the directory as held open, so any job's prune pass (this one
|
||||
# included) skips it, and stamps the LRU marker. The marker is touched
|
||||
# unconditionally every run: a run that hits the cache for every crate may
|
||||
# write nothing at all inside the tree, which would make a just-used
|
||||
# directory look stale to the eviction pass.
|
||||
- shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||
bash "${SCRIPTS}/cache-lock.sh" acquire \
|
||||
"${{ steps.resolve.outputs.target-dir }}" "${{ steps.resolve.outputs.lock-id }}"
|
||||
touch "${{ steps.resolve.outputs.target-dir }}/.cache-last-used"
|
||||
|
||||
# Runs AFTER seeding, deliberately: restore-mtimes.sh reads the build
|
||||
# watermark out of the target dir, so that directory has to be in its
|
||||
# final form for this run (reused, seeded, or freshly created) before the
|
||||
# watermark it may carry can be read.
|
||||
# CARGO_TARGET_DIR and CI_WATERMARK_FILE are passed explicitly rather
|
||||
# than read from the job environment the resolve step exported: the export
|
||||
# is what the consuming workflow's own build steps rely on, but a step
|
||||
# inside this action should not depend on cross-step propagation working
|
||||
# when it can just be handed the value.
|
||||
- if: ${{ inputs.restore-mtimes == 'true' }}
|
||||
shell: bash
|
||||
env:
|
||||
CARGO_TARGET_DIR: ${{ steps.resolve.outputs.target-dir }}
|
||||
CI_WATERMARK_FILE: ${{ steps.resolve.outputs.watermark-file }}
|
||||
run: bash "$(cd "${{ github.action_path }}/.." && pwd)/scripts/restore-mtimes.sh"
|
||||
|
||||
- if: ${{ inputs.prune == 'true' }}
|
||||
shell: bash
|
||||
env:
|
||||
STALE_LOCK_SECONDS: ${{ inputs.stale-lock-seconds }}
|
||||
CACHE_LIVENESS: ${{ inputs.liveness-prune }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts
|
||||
bash "${SCRIPTS}/prune-cache.sh" \
|
||||
"${{ inputs.cache-root }}" \
|
||||
"${{ steps.resolve.outputs.target-dir }}" \
|
||||
"${{ inputs.protected-branches }}" \
|
||||
"${{ inputs.min-free-percent }}"
|
||||
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# Prints the sanitized, collision-resistant cache key for a raw git ref.
|
||||
# Thin CLI wrapper over cache_key() in cache-lib.sh, kept as its own entry
|
||||
# point so a consuming workflow (or an out-of-band cleanup job, e.g. one
|
||||
# reclaiming a deleted branch's directory) can resolve the same key this
|
||||
# action uses without reimplementing the algorithm. Reimplementing it is the
|
||||
# one thing that must never happen: two spellings of the key silently
|
||||
# misclassify every directory on the volume.
|
||||
#
|
||||
# Usage: branch-cache-key.sh "<raw-ref>"
|
||||
set -euo pipefail
|
||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||
|
||||
if [ $# -ne 1 ] || [ -z "$1" ]; then
|
||||
echo "::error::branch-cache-key.sh: expected exactly one non-empty argument (the raw ref)" >&2
|
||||
exit 1
|
||||
fi
|
||||
cache_key "$1"
|
||||
Executable
+219
@@ -0,0 +1,219 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the cargo-cache actions. Sourced, never executed
|
||||
# directly — every caller does:
|
||||
#
|
||||
# . "$(dirname "${BASH_SOURCE[0]}")/cache-lib.sh"
|
||||
#
|
||||
# Nothing here reads the environment implicitly; every function takes what it
|
||||
# needs as an argument, so the selftests can drive them against scratch
|
||||
# directories without a CI context.
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cache keys
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Maps a raw git ref to a filesystem-safe, collision-resistant directory
|
||||
# component.
|
||||
#
|
||||
# Two properties matter and neither is free:
|
||||
#
|
||||
# Determinism — the same raw ref must always produce the same key, or a
|
||||
# branch's second run lands in a different directory from its first and the
|
||||
# whole cache is pointless.
|
||||
#
|
||||
# Collision resistance — `tr -c 'A-Za-z0-9._-' '-'` maps every disallowed
|
||||
# byte to the same `-`, so `feat/foo` and `feat-foo` sanitise identically
|
||||
# and would share one directory: two unrelated branches interleaving
|
||||
# fingerprints in one tree, which is the exact cross-branch-contamination
|
||||
# hazard this whole scheme exists to close, reopened through the sanitiser.
|
||||
# An 8-hex-char prefix of the SHA-1 of the *raw* (pre-sanitisation) ref is
|
||||
# appended so distinct refs always get distinct keys regardless of what
|
||||
# sanitisation or truncation did to the readable part.
|
||||
#
|
||||
# The readable part is capped at 48 characters so a long branch name can't
|
||||
# approach filesystem path-length limits; the hash suffix is what keeps two
|
||||
# refs sharing a 48-char prefix apart.
|
||||
cache_key() {
|
||||
local raw="$1" slug hash
|
||||
if [ -z "$raw" ]; then
|
||||
echo "cache_key: refusing to key an empty ref" >&2
|
||||
return 1
|
||||
fi
|
||||
slug=$(printf '%s' "$raw" | tr -c 'A-Za-z0-9._-' '-' | sed 's/-\{2,\}/-/g; s/^-//; s/-$//')
|
||||
hash=$(printf '%s' "$raw" | sha1sum | cut -c1-8)
|
||||
printf '%s-%s' "${slug:0:48}" "$hash"
|
||||
}
|
||||
|
||||
target_dir_for() { printf '%s/target-%s' "$1" "$2"; }
|
||||
snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Disk accounting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
usage_kb() {
|
||||
if [ -d "$1" ]; then du -sk "$1" 2>/dev/null | awk '{print $1}'; else echo 0; fi
|
||||
}
|
||||
|
||||
usage_gb() {
|
||||
awk "BEGIN { printf \"%.1f\", $(usage_kb "$1") / 1024 / 1024 }"
|
||||
}
|
||||
|
||||
# df -kP: portable POSIX one-line-per-fs output; columns are 1k-blocks total,
|
||||
# used, available, capacity%, mounted-on. Prints "<total_kb> <free_kb>".
|
||||
#
|
||||
# CACHE_DF_OVERRIDE exists for the selftests: a scratch tmpdir on the test
|
||||
# host's real filesystem won't sit below an arbitrary threshold on demand, and
|
||||
# the eviction passes are precisely what needs testing under pressure.
|
||||
read_df() {
|
||||
if [ -n "${CACHE_DF_OVERRIDE:-}" ]; then printf '%s\n' "$CACHE_DF_OVERRIDE"; return; fi
|
||||
df -kP "$1" | awk 'NR==2 {print $2, $4}'
|
||||
}
|
||||
|
||||
report_df() {
|
||||
local label="$1" free_kb="$2" total_kb="$3"
|
||||
awk -v l="$label" -v f="$free_kb" -v t="$total_kb" \
|
||||
'BEGIN { printf "%s %.1f GB free / %.0f GB (%.1f%%)", l, f/1048576, t/1048576, (f*100)/t }'
|
||||
}
|
||||
|
||||
# Appends to the Actions job summary, which is read on green runs — unlike a
|
||||
# ::warning:: buried in a log nobody opens. No-op outside Actions so the
|
||||
# scripts still run standalone under the selftests.
|
||||
summary_line() {
|
||||
[ -n "${GITHUB_STEP_SUMMARY:-}" ] && printf '%s\n' "$1" >> "$GITHUB_STEP_SUMMARY"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hardlink cloning, and the part that makes it sound
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cargo's own target-dir lock files (.cargo-lock, .cargo-build-lock,
|
||||
# .cargo-artifact-lock) are zero-byte files it opens and flock(2)s IN PLACE
|
||||
# for the duration of a build — it never truncates-and-renames them the way it
|
||||
# does real artifacts. `cp -al` leaves the clone's copy hardlinked to the same
|
||||
# inode as the source's, and flock() contention is inode-based, not
|
||||
# path-based, so a build in the clone and a build in the source would
|
||||
# serialize on one mutex.
|
||||
#
|
||||
# The glob deliberately reaches past the three observed names so a lock file
|
||||
# added by a future Cargo version is swept too; nothing else in a target dir
|
||||
# is named `.cargo-*lock*`. Cargo recreates whichever it needs as a fresh,
|
||||
# unshared inode the next time it opens the directory, at no cost.
|
||||
strip_cargo_locks() {
|
||||
find "$1" -type f -name '.cargo-*lock*' -delete 2>/dev/null || true
|
||||
return 0
|
||||
}
|
||||
|
||||
# Replaces a subtree with a real (non-hardlinked) copy of itself, in place.
|
||||
#
|
||||
# Staged through a sibling temp path and swapped with `mv -T` rather than
|
||||
# copied over the original file-by-file: the copy is a fresh tree of fresh
|
||||
# inodes, so nothing in it can alias the source it was cloned from. Callers
|
||||
# only ever run this against a staging directory nothing else can see yet
|
||||
# (see hardlink_clone_into's contract), so the brief window where the path is
|
||||
# absent is not observable.
|
||||
unshare_subtree() {
|
||||
local d="$1" tmp
|
||||
[ -d "$d" ] || return 0
|
||||
tmp="${d}.unshare.$$"
|
||||
rm -rf "$tmp"
|
||||
cp -a "$d" "$tmp"
|
||||
rm -rf "$d"
|
||||
mv -T "$tmp" "$d"
|
||||
}
|
||||
|
||||
_unshare_files() {
|
||||
# `-links +1` restricts the work to files that are actually shared, which
|
||||
# makes this idempotent and near-free on an already-unshared tree.
|
||||
find "$@" -links +1 -print0 2>/dev/null |
|
||||
xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _
|
||||
return 0
|
||||
}
|
||||
|
||||
# THE load-bearing function of this whole design.
|
||||
#
|
||||
# A hardlink clone is only safe if every write the clone's build performs
|
||||
# lands on a NEW inode, leaving the source's data untouched. That is true for
|
||||
# compilation artifacts — rustc and the linker replace `deps/*.rlib`,
|
||||
# `*.rmeta`, and binaries rather than truncating them in place — and it is
|
||||
# NOT true for the metadata Cargo and build scripts write with a plain
|
||||
# truncating write. Measured directly (Linux, ext4, cargo 1.9x nightly:
|
||||
# `cp -al` a warm target dir, change a source file, build in the clone, diff
|
||||
# the source) the following files in the SOURCE were mutated through the
|
||||
# shared inode:
|
||||
#
|
||||
# <profile>/.fingerprint/<unit>/dep-<target> (only under
|
||||
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS,
|
||||
# where this file carries the
|
||||
# per-source blake3 checksums)
|
||||
# <profile>/build/<pkg>/output, root-output (Cargo build-script metadata)
|
||||
# <profile>/build/<pkg>/out/** (whatever the build script
|
||||
# writes into OUT_DIR — build
|
||||
# scripts overwhelmingly use a
|
||||
# plain fs::write)
|
||||
# <profile>/deps/*.d, <profile>/*.d (Cargo's post-processed
|
||||
# dep-info)
|
||||
#
|
||||
# The checksum-freshness case is not a cosmetic one. Reproduced end to end:
|
||||
# branch B clones base's cache, builds its own content, and thereby rewrites
|
||||
# base's `dep-<target>` to describe B's sources while base's cache still holds
|
||||
# the artifact built from base's sources. When B merges and base next builds,
|
||||
# Cargo reads that dep file, finds the checksums match the (now merged)
|
||||
# sources, reports `Fresh`, and reuses a binary built from the PRE-merge code.
|
||||
# That is silent stale-artifact reuse — a wrong answer, not a slow one.
|
||||
#
|
||||
# So: hardlink the artifacts (the GB), real-copy the metadata (the MB).
|
||||
# Measured on a 6.9 GB Bevy workspace target dir, the unshared set is
|
||||
# .fingerprint 22 MB + build/ 237 MB + a handful of dep-info files — about
|
||||
# 3.7% of the tree, against 100% for a plain `cp -a`.
|
||||
#
|
||||
# `incremental/` is deliberately left shared: rustc writes each incremental
|
||||
# session to a fresh `s-*-working` directory and finalises it with a rename,
|
||||
# and garbage-collects old sessions by unlinking directory entries — neither
|
||||
# of which mutates a shared inode. CI should still set CARGO_INCREMENTAL=0,
|
||||
# for size rather than correctness.
|
||||
unshare_mutable_paths() {
|
||||
local root="$1" d
|
||||
[ -d "$root" ] || return 0
|
||||
# The list is materialised in full before anything is replaced: each
|
||||
# replacement deletes and recreates a directory, and a live `find` walk over
|
||||
# a tree being mutated underneath it is a needless hazard. `-prune` keeps a
|
||||
# match's own contents out of the list.
|
||||
local -a dirs=()
|
||||
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
||||
for d in "${dirs[@]}"; do
|
||||
[ -n "$d" ] || continue
|
||||
unshare_subtree "$d"
|
||||
done
|
||||
_unshare_files "$root" -type f -name '*.d'
|
||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json'
|
||||
return 0
|
||||
}
|
||||
|
||||
# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
|
||||
# with a single atomic rename.
|
||||
#
|
||||
# The staging + rename is what closes the concurrent-seed race structurally
|
||||
# rather than by runner topology: a second job sharing this cache key either
|
||||
# sees DST absent (and stages its own clone, losing the rename harmlessly) or
|
||||
# sees it complete. There is no observable half-populated state, because a
|
||||
# directory rename is atomic and DST is never written through.
|
||||
#
|
||||
# Returns 0 if this caller's clone won the rename, 1 if another caller got
|
||||
# there first (the staging copy is discarded; DST is already valid).
|
||||
hardlink_clone_into() {
|
||||
local src="$1" dst="$2" tag="$3" tmp parent
|
||||
parent=$(dirname "$dst")
|
||||
tmp="${parent}/.stage-${tag}"
|
||||
rm -rf "$tmp"
|
||||
cp -al "$src" "$tmp"
|
||||
strip_cargo_locks "$tmp"
|
||||
rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
|
||||
unshare_mutable_paths "$tmp"
|
||||
if mv -T "$tmp" "$dst" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
rm -rf "$tmp"
|
||||
return 1
|
||||
}
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Marks a cache directory as held open by a running job, so ANY job's prune
|
||||
# pass (including this run's own) skips it.
|
||||
#
|
||||
# Usage: cache-lock.sh acquire|release <dir> <lock-id>
|
||||
#
|
||||
# Without this, a directory's only protection from a concurrently running
|
||||
# job's eviction pass is "it happens to also be that job's own target dir",
|
||||
# which is true for the job that owns it and false for everyone else. The
|
||||
# marker is per-job (not just per-run) so two jobs sharing one cache key each
|
||||
# hold an independent lock rather than one clobbering the other's.
|
||||
#
|
||||
# A lock is a timestamp file, not a real mutex: prune-cache.sh honours it only
|
||||
# until STALE_LOCK_SECONDS, after which it is treated as abandoned by a job
|
||||
# the runner killed before it reached its own release step. Honouring a lock
|
||||
# forever would let one crashed job pin a directory permanently.
|
||||
set -euo pipefail
|
||||
|
||||
MODE="${1:?usage: cache-lock.sh acquire|release <dir> <lock-id>}"
|
||||
DIR="${2:?}"; ID="${3:?}"
|
||||
|
||||
case "$MODE" in
|
||||
acquire)
|
||||
mkdir -p "$DIR"
|
||||
date +%s > "$DIR/.ci-lock-${ID}"
|
||||
echo "lock: acquired .ci-lock-${ID} on $(basename "$DIR")"
|
||||
;;
|
||||
release)
|
||||
rm -f "$DIR/.ci-lock-${ID}" 2>/dev/null || true
|
||||
echo "lock: released .ci-lock-${ID} on $(basename "$DIR")"
|
||||
;;
|
||||
*)
|
||||
echo "::error::cache-lock.sh: unknown mode '$MODE' (expected acquire or release)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
Executable
+178
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for the single assumption this whole caching scheme rests
|
||||
# on: that a build running inside a hardlink clone cannot mutate the directory
|
||||
# it was cloned from.
|
||||
#
|
||||
# That assumption is FALSE for a plain `cp -al`. Measured, and asserted below
|
||||
# as an explicit control: build in a raw `cp -al` clone and the source's
|
||||
# `.fingerprint/<unit>/dep-*` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS),
|
||||
# `build/<pkg>/output`, `build/<pkg>/out/**` and `deps/*.d` all change,
|
||||
# because Cargo and build scripts write those with a plain truncating write
|
||||
# rather than the write-then-rename Cargo uses for real artifacts.
|
||||
#
|
||||
# The consequence is not a slow build, it is a wrong one: a PR clone rewrites
|
||||
# the base's dep-info to describe the PR's sources while the base's cache
|
||||
# still holds the artifact built from the base's sources; once the PR merges,
|
||||
# the base's next run finds the checksums match its (now merged) sources,
|
||||
# reports `Fresh`, and links a binary built from the pre-merge code.
|
||||
#
|
||||
# cache-lib.sh's unshare_mutable_paths() is what closes that, and this test is
|
||||
# what proves it stays closed. The control matters as much as the fix: a
|
||||
# scenario that passes for both would prove nothing.
|
||||
#
|
||||
# Needs a working cargo on PATH. Everything happens under a mktemp -d scratch
|
||||
# tree. Run by hand: bash scripts/hardlink-clone-selftest.sh
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
|
||||
command -v cargo >/dev/null || { echo "SKIP: no cargo on PATH"; exit 0; }
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
pass_count=0
|
||||
|
||||
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||
|
||||
# Content hash of every file in a tree, keyed by relative path.
|
||||
snapshot_tree() { (cd "$1" && find . -type f -print0 | sort -z | xargs -0 -r sha1sum) 2>/dev/null; }
|
||||
|
||||
# `diff` exits 1 when the trees differ, which is the expected case here and
|
||||
# must not trip `pipefail` — the difference IS the result.
|
||||
mutated_paths() {
|
||||
{ diff <(printf '%s' "$1") <(printf '%s' "$2") || true; } 2>/dev/null \
|
||||
| awk '/^[<>]/ { print $3 }' | sort -u
|
||||
}
|
||||
|
||||
# A crate with a build script, because build-script OUT_DIR writes are one of
|
||||
# the two mutation families and are invisible without one.
|
||||
mkcrate() {
|
||||
local dir="$1"
|
||||
mkdir -p "$dir/src"
|
||||
cat > "$dir/Cargo.toml" <<'TOML'
|
||||
[package]
|
||||
name = "probe"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
[workspace]
|
||||
TOML
|
||||
cat > "$dir/build.rs" <<'RS'
|
||||
use std::{env, fs, path::PathBuf};
|
||||
fn main() {
|
||||
println!("cargo::rerun-if-changed=src/lib.rs");
|
||||
let out = PathBuf::from(env::var("OUT_DIR").unwrap());
|
||||
let src = fs::read_to_string("src/lib.rs").unwrap();
|
||||
fs::write(out.join("gen.txt"), format!("generated from {} bytes", src.len())).unwrap();
|
||||
}
|
||||
RS
|
||||
}
|
||||
|
||||
crate_dir="$scratch/probe"
|
||||
mkcrate "$crate_dir"
|
||||
cd "$crate_dir"
|
||||
|
||||
export CARGO_INCREMENTAL=0
|
||||
# Checksum freshness is where the worst failure lives (the dep-* file carries
|
||||
# per-source checksums and is rewritten in place). Only available on nightly;
|
||||
# without it the test still covers the build/ and *.d families.
|
||||
CHECKSUM_MODE="off"
|
||||
if cargo +nightly -V >/dev/null 2>&1; then
|
||||
export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true
|
||||
CARGO_BIN=(cargo +nightly)
|
||||
CHECKSUM_MODE="on"
|
||||
else
|
||||
CARGO_BIN=(cargo)
|
||||
fi
|
||||
echo "=== checksum-freshness mode: ${CHECKSUM_MODE} ==="
|
||||
|
||||
CONTENT_A='pub fn f() -> u32 { 1 }'
|
||||
CONTENT_B='pub fn f() -> u32 { 22222 } pub fn g() -> u32 { 7 }'
|
||||
|
||||
build_base() {
|
||||
local dir="$1"
|
||||
printf '%s\n' "$CONTENT_A" > src/lib.rs
|
||||
CARGO_TARGET_DIR="$dir" "${CARGO_BIN[@]}" build -q
|
||||
}
|
||||
|
||||
echo
|
||||
echo "=== control: a raw \`cp -al\` clone DOES mutate its source ==="
|
||||
base_ctl="$scratch/base-ctl"; clone_ctl="$scratch/clone-ctl"
|
||||
build_base "$base_ctl"
|
||||
before=$(snapshot_tree "$base_ctl")
|
||||
cp -al "$base_ctl" "$clone_ctl"
|
||||
strip_cargo_locks "$clone_ctl" # locks alone are not the hazard under test
|
||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||
CARGO_TARGET_DIR="$clone_ctl" "${CARGO_BIN[@]}" build -q
|
||||
after=$(snapshot_tree "$base_ctl")
|
||||
ctl_mutated=$(mutated_paths "$before" "$after")
|
||||
if [ -z "$ctl_mutated" ]; then
|
||||
fail "control produced no mutation — the test can no longer distinguish fixed from broken"
|
||||
fi
|
||||
ok "raw cp -al clone mutates the source ($(printf '%s\n' "$ctl_mutated" | wc -l) paths)"
|
||||
printf '%s\n' "$ctl_mutated" | sed 's/^/ /'
|
||||
|
||||
if [ "$CHECKSUM_MODE" = "on" ]; then
|
||||
if printf '%s' "$ctl_mutated" | grep -q '\.fingerprint/.*/dep-'; then
|
||||
ok "control confirms the checksum-freshness dep-info file is among the mutated set"
|
||||
else
|
||||
fail "expected .fingerprint/*/dep-* in the control's mutated set under checksum freshness"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== fix: hardlink_clone_into() leaves the source byte-identical ==="
|
||||
base_fix="$scratch/base-fix"; clone_fix="$scratch/clone-fix"
|
||||
build_base "$base_fix"
|
||||
before=$(snapshot_tree "$base_fix")
|
||||
hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_into reported the destination already existed"
|
||||
|
||||
# The clone's contract, asserted before anything builds in it: artifacts
|
||||
# share inodes (that is what makes the clone near-free), and every file Cargo
|
||||
# rewrites in place does not (that is what makes it sound). Checking after a
|
||||
# rebuild would prove nothing — the rebuild replaces those files anyway.
|
||||
shared=0; unshared=0
|
||||
while IFS= read -r f; do
|
||||
rel="${f#$base_fix/}"
|
||||
[ -e "$clone_fix/$rel" ] || continue
|
||||
if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then
|
||||
case "$rel" in
|
||||
*/.fingerprint/*|*/build/*|*.d|.rustc_info.json)
|
||||
fail "mutable path still shares an inode with the source: $rel" ;;
|
||||
esac
|
||||
shared=$((shared + 1))
|
||||
else
|
||||
unshared=$((unshared + 1))
|
||||
fi
|
||||
done < <(find "$base_fix" -type f)
|
||||
[ "$shared" -gt 0 ] || fail "nothing is shared — the clone degenerated into a full copy"
|
||||
[ "$unshared" -gt 0 ] || fail "nothing was unshared — unshare_mutable_paths did not run"
|
||||
ok "fresh clone shares ${shared} artifact files and privately owns ${unshared} mutable ones"
|
||||
|
||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||
CARGO_TARGET_DIR="$clone_fix" "${CARGO_BIN[@]}" build -q
|
||||
after=$(snapshot_tree "$base_fix")
|
||||
fix_mutated=$(mutated_paths "$before" "$after")
|
||||
if [ -n "$fix_mutated" ]; then
|
||||
echo " still mutated:" >&2
|
||||
printf '%s\n' "$fix_mutated" | sed 's/^/ /' >&2
|
||||
fail "a build in the clone mutated the source through a shared inode"
|
||||
fi
|
||||
ok "no file in the source changed after a full rebuild in the clone"
|
||||
|
||||
echo
|
||||
echo "=== the whole point: the source's next build is still correct ==="
|
||||
# The source's cache holds artifacts built from CONTENT_A. Advance the source
|
||||
# to CONTENT_B (as a merge would) and rebuild in it. If the clone had
|
||||
# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib.
|
||||
printf '%s\n' "$CONTENT_B" > src/lib.rs
|
||||
touch -d '@1000000000' src/lib.rs
|
||||
log="$scratch/rebuild.log"
|
||||
CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; }
|
||||
if grep -qE '^\s+Fresh probe' "$log"; then
|
||||
fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse"
|
||||
fi
|
||||
ok "source correctly rebuilt its crate after advancing to the clone's content"
|
||||
|
||||
echo
|
||||
echo "hardlink-clone-selftest: ${pass_count} assertions passed"
|
||||
Executable
+174
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for prune-cache.sh. Builds a real scratch git repo standing
|
||||
# in for `origin` and a real scratch directory standing in for the cache root,
|
||||
# then runs the ACTUAL script against both — not a simulation of its logic.
|
||||
#
|
||||
# What each scenario demonstrates, and why the controls matter as much as the
|
||||
# fixes (a scenario that always passes proves nothing):
|
||||
#
|
||||
# 1. DEAD BRANCH PRUNED, not gated on disk pressure — a cache whose branch
|
||||
# no longer exists on origin is removed even with plenty of free space.
|
||||
# Waiting for pressure to notice means paying for dead caches until then.
|
||||
# 2. LIVE BRANCH SURVIVES despite being OLDER than the dead one — liveness,
|
||||
# not age, is what decides pass 1.
|
||||
# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when
|
||||
# their caches are the oldest on disk and would rank first for LRU.
|
||||
# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure.
|
||||
# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than
|
||||
# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory
|
||||
# permanently.
|
||||
# 6. LIVENESS UNAVAILABLE FAILS SAFE — origin unreachable: a genuinely dead
|
||||
# cache is NOT pruned, the log says so plainly, and the pressure fallback
|
||||
# still works independently. "Unavailable" degrades to pressure-only, not
|
||||
# to no eviction at all.
|
||||
# 7. TARGET DIRS EVICTED BEFORE SNAPSHOTS — the ordering that differs from
|
||||
# the obvious one. A snapshot is hardlinked to the caches cloned from it,
|
||||
# so evicting it frees almost nothing while costing every future PR its
|
||||
# warm start.
|
||||
# 8. SELF-CLEAR REPORTS LOUDLY to the job summary, not just a log warning.
|
||||
# 9. OWN CACHE NEVER EVICTED by a sibling pass.
|
||||
# 10. SCOPED TO THE CACHE ROOT — a decoy outside it (standing in for another
|
||||
# project's volume) is never touched.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
prune="$script_dir/prune-cache.sh"
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
pass_count=0
|
||||
fail() { echo "ASSERTION FAILED: $*" >&2; [ -n "${1:-}" ] && [ -f "$scratch/log" ] && { echo "--- log ---" >&2; cat "$scratch/log" >&2; }; exit 1; }
|
||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||
assert_gone() { [ -e "$1" ] && fail "expected gone: $1 ($2)"; ok "$2"; }
|
||||
assert_kept() { [ -e "$1" ] || fail "expected kept: $1 ($2)"; ok "$2"; }
|
||||
assert_log() { grep -q -- "$1" "$scratch/log" || fail "expected in log: $1 ($2)"; ok "$2"; }
|
||||
|
||||
echo "=== building a scratch origin with real branches ==="
|
||||
origin="$scratch/origin.git"; git init -q --bare "$origin"
|
||||
work="$scratch/work"; git init -q "$work"
|
||||
(
|
||||
cd "$work"
|
||||
git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init
|
||||
git branch -M main
|
||||
git checkout -q -b dev; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m dev
|
||||
git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m live
|
||||
git remote add origin "$origin"
|
||||
git push -q origin main dev feat/live
|
||||
)
|
||||
cd "$work"
|
||||
|
||||
MAIN=$(cache_key main); DEV=$(cache_key dev); LIVE=$(cache_key feat/live)
|
||||
DEAD=$(cache_key feat/dead); OWN=$(cache_key feat/own)
|
||||
|
||||
root="$scratch/cache"
|
||||
mk() { mkdir -p "$root/$1"; head -c 4096 /dev/zero > "$root/$1/blob"; touch -d "$2" "$root/$1/.cache-last-used"; }
|
||||
reset_cache() {
|
||||
rm -rf "$root"; mkdir -p "$root"
|
||||
mk "target-$MAIN" '2020-01-01'
|
||||
mk "snapshot-$MAIN" '2020-01-01'
|
||||
mk "target-$DEV" '2020-01-01'
|
||||
mk "snapshot-$DEV" '2020-01-01'
|
||||
mk "target-$LIVE" '2020-01-02' # older than the dead one, deliberately
|
||||
mk "target-$DEAD" '2030-01-01' # newest on disk, but its branch is gone
|
||||
mk "snapshot-$DEAD" '2030-01-01'
|
||||
mk "target-$OWN" '2025-01-01'
|
||||
}
|
||||
run_prune() {
|
||||
local free="${1:-}"
|
||||
CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \
|
||||
bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \
|
||||
|| { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; }
|
||||
}
|
||||
|
||||
echo
|
||||
echo "=== 1/2: dead pruned unconditionally; older-but-live survives ==="
|
||||
reset_cache
|
||||
run_prune "1000000 900000" # 90% free: no pressure at all
|
||||
assert_gone "$root/target-$DEAD" "dead branch's target dir pruned with no disk pressure"
|
||||
assert_gone "$root/snapshot-$DEAD" "dead branch's snapshot pruned too"
|
||||
assert_kept "$root/target-$LIVE" "live branch survives despite an older marker than the dead one"
|
||||
assert_log "no matching branch on origin" "eviction reason reported"
|
||||
|
||||
echo
|
||||
echo "=== 3: protected refs never evicted under forced pressure ==="
|
||||
reset_cache
|
||||
run_prune "1000000 1000" # 0.1% free
|
||||
assert_kept "$root/target-$DEV" "dev's target dir survives disk pressure"
|
||||
assert_kept "$root/snapshot-$DEV" "dev's snapshot survives disk pressure"
|
||||
assert_kept "$root/target-$MAIN" "main's target dir survives disk pressure"
|
||||
assert_kept "$root/snapshot-$MAIN" "main's snapshot survives disk pressure"
|
||||
|
||||
echo
|
||||
echo "=== 9: own cache never evicted by a sibling pass ==="
|
||||
assert_kept "$root/target-$OWN" "this run's own cache survives"
|
||||
|
||||
echo
|
||||
echo "=== 7: target dirs evicted before snapshots ==="
|
||||
reset_cache
|
||||
# Only the live branch is evictable; give it both a target dir and a snapshot
|
||||
# with identical markers so ordering, not age, decides.
|
||||
mk "snapshot-$LIVE" '2020-01-02'
|
||||
# The df override is a fixed reading, so the pressure loop drains everything
|
||||
# evictable — which is what makes the ORDER the observable property here, not
|
||||
# what survives. Assert the eviction order directly from the log.
|
||||
run_prune "1000000 1000"
|
||||
order=$(grep -o "evicting \(target\|snapshot\)-$LIVE" "$scratch/log" | sed "s/evicting //")
|
||||
[ "$(printf '%s\n' "$order" | head -1)" = "target-$LIVE" ] \
|
||||
|| fail "expected target-$LIVE to be evicted before snapshot-$LIVE, got: $order"
|
||||
ok "target dirs are evicted before snapshots"
|
||||
|
||||
echo
|
||||
echo "=== 4: a fresh lock protects a dead, old, under-pressure cache ==="
|
||||
reset_cache
|
||||
date +%s > "$root/target-$DEAD/.ci-lock-ci-1"
|
||||
run_prune "1000000 1000"
|
||||
assert_kept "$root/target-$DEAD" "locked cache survives both passes"
|
||||
assert_log "held open by" "lock reported in the log"
|
||||
|
||||
echo
|
||||
echo "=== 5: a stale lock is not honoured forever ==="
|
||||
reset_cache
|
||||
echo 0 > "$root/target-$DEAD/.ci-lock-ci-1"
|
||||
touch -d '2000-01-01' "$root/target-$DEAD/.ci-lock-ci-1"
|
||||
run_prune "1000000 900000"
|
||||
assert_gone "$root/target-$DEAD" "cache with an abandoned lock is evicted"
|
||||
assert_log "treating as abandoned" "abandoned lock reported in the log"
|
||||
|
||||
echo
|
||||
echo "=== 6: liveness unavailable fails safe, pressure fallback still works ==="
|
||||
reset_cache
|
||||
(
|
||||
cd "$work" && git remote set-url origin "$scratch/nonexistent.git"
|
||||
)
|
||||
run_prune "1000000 900000" # no pressure
|
||||
assert_kept "$root/target-$DEAD" "dead cache NOT pruned when liveness is unavailable"
|
||||
assert_log "treating as UNAVAILABLE" "unavailability reported plainly, not folded into 'no branches'"
|
||||
run_prune "1000000 1000" # now with pressure
|
||||
if [ -e "$root/target-$DEAD" ] && [ -e "$root/target-$LIVE" ]; then
|
||||
fail "pressure fallback did nothing when liveness was unavailable"
|
||||
fi
|
||||
ok "pressure fallback still evicts when liveness is unavailable"
|
||||
(cd "$work" && git remote set-url origin "$origin")
|
||||
|
||||
echo
|
||||
echo "=== 8: self-clear reports to the job summary ==="
|
||||
reset_cache
|
||||
rm -rf "$root/target-$DEAD" "$root/snapshot-$DEAD" "$root/target-$LIVE"
|
||||
: > "$scratch/summary"
|
||||
run_prune "1000000 1000" # nothing evictable left but the run's own cache
|
||||
assert_log "clearing own" "self-clear reported in the log"
|
||||
grep -q 'self-clear' "$scratch/summary" || fail "self-clear missing from the job summary"
|
||||
ok "self-clear reported to the job summary, not only the log"
|
||||
[ -d "$root/target-$OWN" ] || fail "self-clear left the own directory missing"
|
||||
[ -z "$(ls -A "$root/target-$OWN")" ] || fail "self-clear did not actually empty the directory"
|
||||
ok "own cache wiped and recreated empty"
|
||||
|
||||
echo
|
||||
echo "=== 10: scoped to the cache root ==="
|
||||
reset_cache
|
||||
decoy="$scratch/other-project"; mkdir -p "$decoy/target-$DEAD"; touch "$decoy/target-$DEAD/blob"
|
||||
run_prune "1000000 1000"
|
||||
assert_kept "$decoy/target-$DEAD" "a cache outside the cache root is never touched"
|
||||
|
||||
echo
|
||||
echo "prune-cache-selftest: ${pass_count} assertions passed"
|
||||
Executable
+211
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env bash
|
||||
# Eviction for the per-ref cache directories on the persistent volume.
|
||||
#
|
||||
# Usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> <min-free-percent>
|
||||
# protected-branches space-separated raw refs (e.g. "dev main")
|
||||
#
|
||||
# Optional environment:
|
||||
# STALE_LOCK_SECONDS age past which a .ci-lock-* marker is treated as
|
||||
# abandoned (default 7200)
|
||||
# CACHE_LIVENESS "false"/"0" to skip the liveness pass entirely
|
||||
# CACHE_DF_OVERRIDE "<total_kb> <free_kb>", for the selftest
|
||||
#
|
||||
# Three passes, in order:
|
||||
#
|
||||
# 1. LIVENESS — every target-*/snapshot-* directory whose branch no longer
|
||||
# exists on origin is removed UNCONDITIONALLY, not gated on free space.
|
||||
# A directory for a branch deleted days ago is pure loss: nothing will
|
||||
# ever read it again, since a merged PR's branch cannot be reopened.
|
||||
# Waiting for disk pressure to notice means paying for it until then.
|
||||
# Skipped entirely, loudly, if the liveness signal itself is
|
||||
# unavailable — "couldn't determine" is never folded into "dead".
|
||||
# 2. PRESSURE — if free space is still under the threshold, evict remaining
|
||||
# (now necessarily live) directories oldest-first until it clears.
|
||||
# 3. SELF-CLEAR — if pass 2 still isn't enough, wipe this run's own target
|
||||
# dir and pay a cold rebuild, reported to the job summary as well as the
|
||||
# log, because a warning on a green run is what lets a silently 4x-slower
|
||||
# job go unnoticed.
|
||||
#
|
||||
# Reactive-only, with no hard cap on cache size: a workspace's natural working
|
||||
# set is what it is, and bounding the footprint preemptively means wiping
|
||||
# useful content before it is actually causing host pressure. The bound is
|
||||
# physics, not an arbitrary GB number.
|
||||
#
|
||||
# EVICTION ORDER, and why it is the reverse of the obvious one: within the
|
||||
# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones.
|
||||
# A snapshot is a hardlink clone of a live target dir and of every consumer
|
||||
# cloned from it, so removing it frees almost no real bytes — its inodes stay
|
||||
# alive through those other links — while costing every future PR its warm
|
||||
# start. Evicting snapshots first would be nearly pure loss. Target
|
||||
# directories are where a branch's own divergent artifacts actually live, so
|
||||
# they are what freeing space means.
|
||||
#
|
||||
# Two exclusions every pass respects:
|
||||
#
|
||||
# PROTECTED — the publisher branches' target and snapshot directories, and
|
||||
# this run's own target dir, are never candidates in any pass. Evicting a
|
||||
# publisher's snapshot doesn't free real disk (every open PR's clone keeps
|
||||
# the data alive) but does force every subsequent PR to start cold, which is
|
||||
# the entire benefit this scheme exists to deliver.
|
||||
#
|
||||
# LOCKED — a directory carrying a .ci-lock-* marker younger than
|
||||
# STALE_LOCK_SECONDS is held open by a running job and is skipped by every
|
||||
# pass, however dead and however tight the disk. This is what makes eviction
|
||||
# safe on a runner with more than one execution slot. An older marker is
|
||||
# treated as abandoned and logged as such, so an actually-still-running job
|
||||
# that somehow exceeds the threshold is visible in the log rather than
|
||||
# silently losing its cache mid-build.
|
||||
#
|
||||
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
|
||||
# timeout. A directory name cannot be inverted back to a branch name (the
|
||||
# sanitiser is lossy and the disambiguating suffix is a one-way hash), so this
|
||||
# goes the other direction: it recomputes the expected directory names for
|
||||
# every branch origin reports, using cache-lib.sh's OWN cache_key function —
|
||||
# the same one the seed step used to create them. Reusing that function rather
|
||||
# than reimplementing a lookalike is what makes the classification sound; any
|
||||
# drift between two spellings would silently misclassify every directory.
|
||||
#
|
||||
# Only ever globs inside <cache-root>. Another project's volume is a different
|
||||
# Docker named volume and is not mounted in this container at all, so "stays
|
||||
# scoped to this repo's cache" holds structurally, not by convention.
|
||||
set -euo pipefail
|
||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||
|
||||
ROOT="${1:?usage: prune-cache.sh <cache-root> <own-target-dir> <protected-branches> <min-free-percent>}"
|
||||
OWN_DIR="${2:?}"
|
||||
PROTECTED_REFS="${3:-}"
|
||||
MIN_FREE_PCT="${4:-10}"
|
||||
STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}"
|
||||
|
||||
declare -A protected_ns=()
|
||||
for ref in $PROTECTED_REFS; do
|
||||
suffix=$(cache_key "$ref")
|
||||
protected_ns["target-${suffix}"]=1
|
||||
protected_ns["snapshot-${suffix}"]=1
|
||||
done
|
||||
|
||||
is_protected() {
|
||||
local dir="$1" name
|
||||
name=$(basename "$dir")
|
||||
[ "$dir" = "$OWN_DIR" ] && return 0
|
||||
[ -n "${protected_ns[$name]:-}" ] && return 0
|
||||
return 1
|
||||
}
|
||||
|
||||
is_locked() {
|
||||
local dir="$1" now lock_file lock_age locked=1
|
||||
now=$(date +%s)
|
||||
for lock_file in "$dir"/.ci-lock-*; do
|
||||
[ -e "$lock_file" ] || continue
|
||||
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
|
||||
if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then
|
||||
echo " $(basename "$dir"): held open by $(basename "$lock_file") (${lock_age}s old)"
|
||||
locked=0
|
||||
else
|
||||
echo " $(basename "$dir"): ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned"
|
||||
fi
|
||||
done
|
||||
return "$locked"
|
||||
}
|
||||
|
||||
# Directories oldest-first, target-* before snapshot-* (see the header).
|
||||
# The sort key is a rank digit followed by a zero-padded mtime, so the two
|
||||
# groups sort as blocks rather than interleaving by age. `.cache-last-used`
|
||||
# is the marker every run touches; a run that hits the cache for every crate
|
||||
# may write nothing at all inside the tree, which would make a
|
||||
# just-used directory look stale without it. Fall back to the directory's
|
||||
# own mtime when the marker is missing (a partially-written directory from an
|
||||
# interrupted run is still a valid, if less precise, "last touched" signal).
|
||||
list_by_lru() {
|
||||
local rank d ts
|
||||
for rank in 0:target 1:snapshot; do
|
||||
for d in "$ROOT"/"${rank#*:}"-*; do
|
||||
[ -d "$d" ] || continue
|
||||
if [ -e "$d/.cache-last-used" ]; then ts=$(stat -c '%Y' "$d/.cache-last-used" 2>/dev/null || echo 0)
|
||||
else ts=$(stat -c '%Y' "$d" 2>/dev/null || echo 0); fi
|
||||
printf '%s%012d\t%s\n' "${rank%%:*}" "$ts" "$d"
|
||||
done
|
||||
done | sort | cut -f2-
|
||||
}
|
||||
|
||||
echo "=== pass 1: liveness (unconditional, not gated on free space) ==="
|
||||
declare -A live_ns=()
|
||||
LIVENESS_AVAILABLE=0
|
||||
LIVENESS_REASON=""
|
||||
if [ "${CACHE_LIVENESS:-true}" = "0" ] || [ "${CACHE_LIVENESS:-true}" = "false" ]; then
|
||||
LIVENESS_REASON="disabled via CACHE_LIVENESS=${CACHE_LIVENESS}"
|
||||
elif remote_heads=$(timeout 20 git ls-remote --heads origin 2>&1); then
|
||||
LIVENESS_AVAILABLE=1
|
||||
branch_count=0
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] || continue
|
||||
case "$line" in *refs/heads/*) ;; *) continue ;; esac
|
||||
branch="${line#*refs/heads/}"
|
||||
[ -n "$branch" ] || continue
|
||||
suffix=$(cache_key "$branch")
|
||||
live_ns["target-${suffix}"]=1
|
||||
live_ns["snapshot-${suffix}"]=1
|
||||
branch_count=$((branch_count + 1))
|
||||
done <<< "$remote_heads"
|
||||
echo "liveness: ${branch_count} live branches on origin"
|
||||
else
|
||||
LIVENESS_REASON="git ls-remote --heads origin failed or timed out"
|
||||
fi
|
||||
|
||||
if [ "$LIVENESS_AVAILABLE" = "1" ]; then
|
||||
pruned_any=0
|
||||
for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do
|
||||
[ -d "$dir" ] || continue
|
||||
name=$(basename "$dir")
|
||||
is_protected "$dir" && continue
|
||||
[ -n "${live_ns[$name]:-}" ] && continue
|
||||
is_locked "$dir" && continue
|
||||
dir_gb=$(usage_gb "$dir")
|
||||
echo "::warning::pruning dead-branch cache ${name} (${dir_gb} GB) — no matching branch on origin"
|
||||
summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — branch no longer exists on origin"
|
||||
rm -rf "$dir"
|
||||
pruned_any=1
|
||||
done
|
||||
[ "$pruned_any" = "1" ] || echo "no dead-branch caches found"
|
||||
else
|
||||
echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "=== pass 2/3: disk pressure (threshold: free < ${MIN_FREE_PCT}%) ==="
|
||||
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
||||
THRESHOLD_KB=$(( TOTAL_KB * MIN_FREE_PCT / 100 ))
|
||||
|
||||
if [ "$FREE_KB" -ge "$THRESHOLD_KB" ]; then
|
||||
echo "cache: $(basename "$OWN_DIR") $(usage_gb "$OWN_DIR") GB | $(report_df host "$FREE_KB" "$TOTAL_KB")"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "::warning::$(report_df disk "$FREE_KB" "$TOTAL_KB") < ${MIN_FREE_PCT}% threshold"
|
||||
|
||||
# A plain loop over a pre-materialised, pre-sorted list rather than a live
|
||||
# `find | while` pipeline, so `rm -rf` inside the loop cannot make a running
|
||||
# directory walk re-observe its own deletions.
|
||||
mapfile -t LRU < <(list_by_lru)
|
||||
for dir in "${LRU[@]}"; do
|
||||
[ -d "$dir" ] || continue
|
||||
is_protected "$dir" && continue
|
||||
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
||||
[ "$FREE_KB" -ge "$THRESHOLD_KB" ] && break
|
||||
is_locked "$dir" && continue
|
||||
dir_gb=$(usage_gb "$dir")
|
||||
echo "::warning::evicting $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)"
|
||||
summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)"
|
||||
rm -rf "$dir"
|
||||
done
|
||||
|
||||
read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")"
|
||||
if [ "$FREE_KB" -lt "$THRESHOLD_KB" ]; then
|
||||
OWN_GB=$(usage_gb "$OWN_DIR")
|
||||
echo "::warning::still under threshold after evicting every eligible sibling; clearing own $(basename "$OWN_DIR") (was ${OWN_GB} GB) — this run pays a cold rebuild"
|
||||
summary_line "- **self-clear**: \`$(basename "$OWN_DIR")\` (was ${OWN_GB} GB) wiped — this run pays a cold rebuild"
|
||||
rm -rf "$OWN_DIR"
|
||||
mkdir -p "$OWN_DIR"
|
||||
else
|
||||
echo "$(report_df post-eviction "$FREE_KB" "$TOTAL_KB") — sibling eviction recovered enough space"
|
||||
fi
|
||||
Executable
+113
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for publish-snapshot.sh: the atomic swap, and the two
|
||||
# properties the swap exists to guarantee.
|
||||
#
|
||||
# 1. FIRST PUBLISH — with no prior snapshot, the target dir is published and
|
||||
# is a hardlink clone of it (cheap), with the mutable metadata privately
|
||||
# owned (sound: the publisher's NEXT build must not be able to mutate the
|
||||
# snapshot it just published).
|
||||
# 2. REPUBLISH REPLACES — a second publish replaces the snapshot's content
|
||||
# rather than merging into it, and leaves no scratch directories behind.
|
||||
# 3. A LIVE CONSUMER SURVIVES A REPUBLISH — a branch that already cloned the
|
||||
# previous generation keeps reading its own consistent copy. Removing the
|
||||
# old snapshot unlinks directory entries; the inodes stay alive through
|
||||
# the consumer's own links. This is why a republish can never pull data
|
||||
# out from under a running job.
|
||||
# 4. NO TARGET DIR — publishing when there is nothing to publish is a no-op,
|
||||
# not a failure.
|
||||
# 5. LOCKS AND MARKERS DO NOT RIDE ALONG — the publishing job's own cache
|
||||
# lock is still held while this runs, and must not be baked into the
|
||||
# snapshot: a lock timestamped at this run's start would look fresh to
|
||||
# the prune pass on every branch later seeded from it.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
root="$scratch/cache"; mkdir -p "$root"
|
||||
pass_count=0
|
||||
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||
|
||||
# Cargo and rustc REPLACE an artifact (write elsewhere, rename over the path)
|
||||
# rather than truncating it in place, which is exactly why a snapshot may
|
||||
# share artifact inodes with the live target dir it was cloned from. The
|
||||
# fixtures here have to model that faithfully — a plain `>` redirect truncates
|
||||
# in place and would write straight through the shared inode into the
|
||||
# snapshot and every consumer, which is a property of the test fixture, not of
|
||||
# a real build. hardlink-clone-selftest.sh is what verifies the real thing
|
||||
# against a real compiler.
|
||||
replace_file() {
|
||||
printf '%s\n' "$2" > "$1.new"
|
||||
mv -f "$1.new" "$1"
|
||||
}
|
||||
|
||||
make_tree() {
|
||||
local d="$1" marker="$2"
|
||||
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x"
|
||||
echo "$marker" > "$d/debug/deps/libx.rlib"
|
||||
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
||||
: > "$d/debug/.cargo-lock"
|
||||
}
|
||||
|
||||
KEY=$(cache_key dev)
|
||||
TGT="$root/target-$KEY"
|
||||
SNAP="$root/snapshot-$KEY"
|
||||
publish() { bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" "$1" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }; }
|
||||
|
||||
echo "=== 4: nothing to publish is a no-op ==="
|
||||
publish job1
|
||||
[ -d "$SNAP" ] && fail "published a snapshot with no target dir present"
|
||||
grep -q 'nothing to snapshot' "$scratch/log" || fail "expected a 'nothing to snapshot' line"
|
||||
ok "no target dir: no-op, reported plainly"
|
||||
|
||||
echo
|
||||
echo "=== 1: first publish ==="
|
||||
make_tree "$TGT" gen1
|
||||
date +%s > "$TGT/.ci-lock-ci-42"
|
||||
touch "$TGT/.cache-last-used"
|
||||
publish job1
|
||||
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen1" ] || fail "snapshot content wrong"
|
||||
ok "snapshot published"
|
||||
[ "$(stat -c '%i' "$SNAP/debug/deps/libx.rlib")" = "$(stat -c '%i' "$TGT/debug/deps/libx.rlib")" ] \
|
||||
|| fail "snapshot artifact was copied, not hardlinked"
|
||||
ok "snapshot shares artifact inodes with the target dir (cheap)"
|
||||
[ "$(stat -c '%i' "$SNAP/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$TGT/debug/.fingerprint/x/dep-lib-x")" ] \
|
||||
|| fail "snapshot fingerprint still aliases the live target dir"
|
||||
ok "snapshot owns its mutable metadata (publisher's next build cannot corrupt it)"
|
||||
|
||||
echo
|
||||
echo "=== 5: locks and LRU markers do not ride along ==="
|
||||
[ -e "$SNAP/.ci-lock-ci-42" ] && fail "the publishing job's lock was baked into the snapshot"
|
||||
ok "cache lock not published"
|
||||
[ -e "$SNAP/.cache-last-used" ] && fail "the LRU marker was baked into the snapshot"
|
||||
ok "LRU marker not published"
|
||||
[ -e "$SNAP/debug/.cargo-lock" ] && fail "a Cargo lock file was published"
|
||||
ok "Cargo lock file not published"
|
||||
|
||||
echo
|
||||
echo "=== 3: a consumer that cloned generation 1 ==="
|
||||
CONSUMER="$root/target-$(cache_key feat/consumer)"
|
||||
hardlink_clone_into "$SNAP" "$CONSUMER" consumer-tag || fail "consumer clone failed"
|
||||
ok "consumer cloned generation 1"
|
||||
|
||||
echo
|
||||
echo "=== 2: republish replaces, leaves no scratch behind ==="
|
||||
replace_file "$TGT/debug/deps/libx.rlib" gen2
|
||||
# The fingerprint IS written in place by Cargo — and the snapshot owns its own
|
||||
# copy precisely so that write cannot reach it. Truncating in place here is
|
||||
# the faithful model.
|
||||
echo gen2 > "$TGT/debug/.fingerprint/x/dep-lib-x"
|
||||
publish job1
|
||||
[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen2" ] || fail "republish did not replace the snapshot"
|
||||
ok "republished snapshot carries generation 2"
|
||||
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.publish-*' \) -print)
|
||||
[ -z "$leftovers" ] || fail "scratch directories left behind: $leftovers"
|
||||
ok "no scratch directories left behind"
|
||||
[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \
|
||||
|| fail "the consumer's clone changed under it when the snapshot was replaced"
|
||||
ok "the live consumer still reads its own consistent generation-1 copy"
|
||||
|
||||
echo
|
||||
echo "publish-snapshot-selftest: ${pass_count} assertions passed"
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Publish side: atomically republish this reference branch's live target dir
|
||||
# as the immutable snapshot other branches seed from.
|
||||
#
|
||||
# Usage: publish-snapshot.sh <own-key> <cache-root> <tag>
|
||||
#
|
||||
# Run only after the build has already succeeded, and only on a branch that is
|
||||
# a publisher (see the cargo-cache-publish action, which makes that decision).
|
||||
# A red build must never overwrite a known-good snapshot: the caller's step
|
||||
# ordering is the whole gate here, so keep this step last.
|
||||
#
|
||||
# The swap is two renames, not one, because POSIX rename() can only replace an
|
||||
# EMPTY directory and a snapshot from a prior publish is not one:
|
||||
#
|
||||
# 1. stage the new snapshot at .stage-<tag> (copy time is off every
|
||||
# consumer's hot path — nothing reads a staging path);
|
||||
# 2. rename the current snapshot aside to .publish-old-<tag>, if present;
|
||||
# 3. rename the staged snapshot into place.
|
||||
#
|
||||
# Step 3 is a single atomic rename onto a path now guaranteed absent, so it
|
||||
# can never partially overwrite a live snapshot. A consumer landing in the
|
||||
# window between 2 and 3 sees the snapshot as absent and falls through to its
|
||||
# own cold-start path — a safe degrade that self-heals on its next run, not
|
||||
# corruption.
|
||||
#
|
||||
# `rm -rf` on the old snapshot removes directory entries only. Any consumer
|
||||
# that already hardlink-cloned from it keeps every inode alive through its own
|
||||
# links, so a republish never pulls data out from under a running job — it
|
||||
# just stops new consumers from seeing the old generation. Disk is reclaimed
|
||||
# when the last clone referencing those inodes is itself evicted.
|
||||
set -euo pipefail
|
||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||
|
||||
OWN_KEY="${1:?usage: publish-snapshot.sh <own-key> <cache-root> <tag>}"
|
||||
ROOT="${2:?}"; TAG="${3:?}"
|
||||
|
||||
SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
|
||||
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
|
||||
OLD="${ROOT}/.publish-old-${TAG}"
|
||||
|
||||
if [ ! -d "$SRC" ]; then
|
||||
echo "publish: no target dir at ${SRC} — nothing to snapshot"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Leftovers from a run cancelled mid-swap. Scoped to this job's own tag so a
|
||||
# concurrently running job's staging directory is never touched.
|
||||
rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
|
||||
|
||||
start=$(date +%s)
|
||||
# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
|
||||
# of every file Cargo rewrites in place (unshare_mutable_paths, called inside
|
||||
# hardlink_clone_into). Without that, this branch's NEXT build would mutate
|
||||
# the snapshot it just published — the same aliasing hazard the consume side
|
||||
# closes, pointing the other way. The staging path is not the final name, so
|
||||
# `hardlink_clone_into`'s rename lands on DST only after OLD is out of the way.
|
||||
TMP_DST="${ROOT}/.publish-new-${TAG}"
|
||||
rm -rf "$TMP_DST"
|
||||
hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {
|
||||
echo "::error::publish: failed to stage a snapshot of ${SRC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
|
||||
mv -T "$TMP_DST" "$DST"
|
||||
rm -rf "$OLD"
|
||||
|
||||
echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
|
||||
summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Records this run's HEAD as the watermark restore-mtimes.sh reads back on
|
||||
# this cache directory's NEXT run.
|
||||
#
|
||||
# Usage: record-watermark.sh <target-dir> <watermark-file-name>
|
||||
#
|
||||
# Run only after the build has succeeded: a failed run leaves the watermark at
|
||||
# the last GREEN build, which is the conservative direction (over-invalidates,
|
||||
# never under-invalidates). See restore-mtimes.sh's header for what the
|
||||
# watermark is for.
|
||||
#
|
||||
# Written with write-then-rename so this directory's next run can never
|
||||
# observe a truncated, half-written watermark.
|
||||
set -euo pipefail
|
||||
DIR="${1:?usage: record-watermark.sh <target-dir> <watermark-file-name>}"
|
||||
NAME="${2:?}"
|
||||
mkdir -p "$DIR"
|
||||
git rev-parse HEAD > "$DIR/${NAME}.tmp"
|
||||
mv -f "$DIR/${NAME}.tmp" "$DIR/${NAME}"
|
||||
echo "watermark: $(cat "$DIR/${NAME}") recorded in ${DIR}/${NAME}"
|
||||
Executable
+475
@@ -0,0 +1,475 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for restore-mtimes.sh's per-target-dir watermark (issue
|
||||
# The merge hazard: a branch that merges an older commit could silently reuse a stale
|
||||
# build artifact and fail with an "impossible" compile error). Builds a
|
||||
# minimal, throwaway two-crate cargo workspace in a scratch git repo whose
|
||||
# history is shaped exactly like the real incident, then runs the ACTUAL
|
||||
# `restore-mtimes.sh` sitting next to this file against it under real
|
||||
# `cargo build` — not a simulation of the logic, the real script and a real
|
||||
# compiler. Every scenario ends in a hard assertion; the first failure dumps
|
||||
# the relevant log and exits non-zero, so a future change to restore-mtimes.sh
|
||||
# or the actions' step order that reopens this hazard fails loudly here
|
||||
# instead of surfacing weeks later as a confusing CI compile error.
|
||||
#
|
||||
# Run by hand:
|
||||
# bash .gitea/scripts/restore-mtimes-selftest.sh
|
||||
#
|
||||
# Slow by design — this builds a throwaway cargo workspace
|
||||
# and runs several real compiler invocations (~1-2 minutes), which is too
|
||||
# slow to pay on every ordinary push. Re-run it by hand whenever
|
||||
# restore-mtimes.sh's timestamp logic or the actions' cache/seed/watermark step
|
||||
# order changes.
|
||||
#
|
||||
# Needs a working `cargo`/`rustc` on PATH. Does not touch this repo or its
|
||||
# own target dir — everything happens under a `mktemp -d` scratch tree,
|
||||
# removed on exit via the trap below regardless of outcome.
|
||||
#
|
||||
# What each scenario demonstrates, and why the "control" runs matter as much
|
||||
# as the "fixed" ones (a scenario that always passes proves nothing):
|
||||
#
|
||||
# 1. NO WATERMARK, same-branch merge (control) — a feature branch builds
|
||||
# once, then merges a commit that changed a dependency crate but was
|
||||
# authored earlier in real time than that build. With no watermark to
|
||||
# consult, restore-mtimes.sh falls back to its pre-watermark behaviour
|
||||
# (git-log-only timestamps): asserts this reproduces the exact bug —
|
||||
# the dependency crate is judged Fresh and reused stale, and the
|
||||
# dependent crate fails to compile against it.
|
||||
# 2. WATERMARK PRESENT, same state (fix) — identical setup, but this
|
||||
# target dir's watermark names the pre-merge build. Asserts the
|
||||
# dependency crate is correctly identified as changed and recompiles,
|
||||
# and the build succeeds.
|
||||
# 3. WATERMARK PRESENT, no further change (warm-path) — re-runs scenario
|
||||
# 2's state with an up-to-date watermark and no new commits. Asserts
|
||||
# NOTHING recompiles — the ordinary no-merge case restore-mtimes.sh
|
||||
# exists for is unaffected by the watermark machinery.
|
||||
# 4. SEEDED NAMESPACE, watermark stripped (control) — a brand-new branch
|
||||
# namespace, seeded from scenario
|
||||
# 2/3's now-warm target dir, then merges a THIRD lineage's older commit.
|
||||
# With the inherited watermark file deliberately removed, asserts the
|
||||
# same staleness bug reproduces on a namespace that has never itself
|
||||
# run a build — proving the seed-snapshot path shares the hazard, not
|
||||
# just long-lived branches.
|
||||
# 5. SEEDED NAMESPACE, watermark inherited (fix) — identical to 4, but the
|
||||
# watermark file rides along with the `cp -a` seed copy exactly as
|
||||
# the cargo-cache action's seed step really does it.
|
||||
# Asserts the dependency crate correctly recompiles on this namespace's
|
||||
# very first run.
|
||||
# 6. TWO CONSUMERS, ONE SHARED WATERMARK FILE, sequential jobs (control) —
|
||||
# two jobs in one workflow can share a single per-ref
|
||||
# $CARGO_TARGET_DIR, so one cache holds two independent cargo profile
|
||||
# directories (say `debug/` built by a test job and `release/` built by a
|
||||
# wasm build job). Modelling this as "both jobs read before either
|
||||
# writes" is the wrong ordering: on a serial runner with no `needs:`
|
||||
# between the two jobs, one job's watermark WRITE lands before the other
|
||||
# job's own READ, within the same trigger. This scenario now runs restore-mtimes.sh TWICE IN
|
||||
# SEQUENCE against ONE shared watermark file, with a real watermark
|
||||
# write in between (job A builds debug/, succeeds, advances the shared
|
||||
# watermark to HEAD; job B then reads that JUST-ADVANCED watermark).
|
||||
# Asserts job A succeeds and job B reproduces the original bug —
|
||||
# an empty HEAD..HEAD diff, full fallback to git-log timestamps, stale
|
||||
# reuse in release/.
|
||||
# 7. TWO CONSUMERS, TWO WATERMARK FILES, sequential jobs (fix) — identical
|
||||
# ordering, but each job reads and writes its OWN watermark file
|
||||
# (`CI_WATERMARK_FILE`), exactly as the cargo-cache action does. Job B's read is
|
||||
# unaffected by job A's write because they touch different files.
|
||||
# Asserts BOTH jobs correctly recompile the dependency and succeed.
|
||||
set -euo pipefail
|
||||
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
restore_mtimes="$script_dir/restore-mtimes.sh"
|
||||
|
||||
if [ ! -x "$restore_mtimes" ] && [ ! -f "$restore_mtimes" ]; then
|
||||
echo "restore-mtimes-selftest: expected to find restore-mtimes.sh at $restore_mtimes" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
repo="$scratch/wk"
|
||||
target_feature="$scratch/target-feature"
|
||||
target_newbranch="$scratch/target-newbranch"
|
||||
target_multiprofile="$scratch/target-multiprofile"
|
||||
|
||||
pass_count=0
|
||||
|
||||
assert_log_has() {
|
||||
local file="$1" pattern="$2" desc="$3"
|
||||
if ! grep -q -- "$pattern" "$file"; then
|
||||
echo "ASSERTION FAILED: $desc" >&2
|
||||
echo " expected to find: $pattern" >&2
|
||||
echo " --- full log ($file) ---" >&2
|
||||
cat "$file" >&2
|
||||
exit 1
|
||||
fi
|
||||
pass_count=$((pass_count + 1))
|
||||
echo "PASS: $desc"
|
||||
}
|
||||
|
||||
assert_log_lacks() {
|
||||
local file="$1" pattern="$2" desc="$3"
|
||||
if grep -q -- "$pattern" "$file"; then
|
||||
echo "ASSERTION FAILED: $desc" >&2
|
||||
echo " expected NOT to find: $pattern" >&2
|
||||
echo " --- full log ($file) ---" >&2
|
||||
cat "$file" >&2
|
||||
exit 1
|
||||
fi
|
||||
pass_count=$((pass_count + 1))
|
||||
echo "PASS: $desc"
|
||||
}
|
||||
|
||||
# `actions/checkout` stamps every tracked file with wall-clock "now" on
|
||||
# every run — simulate that before each restore-mtimes.sh pass, exactly as
|
||||
# CI would see it, so this test exercises the script the same way CI does.
|
||||
stamp_checkout_now() {
|
||||
find . -path ./.git -prune -o -type f -print | xargs touch
|
||||
}
|
||||
|
||||
echo "=== building scratch workspace ==="
|
||||
mkdir -p "$repo/crates/libdep/src" "$repo/crates/libuser/src"
|
||||
cd "$repo"
|
||||
git init -q
|
||||
git config user.email test@example.com
|
||||
git config user.name "restore-mtimes-selftest"
|
||||
|
||||
cat > Cargo.toml <<'EOF'
|
||||
[workspace]
|
||||
resolver = "2"
|
||||
members = ["crates/libdep", "crates/libuser"]
|
||||
EOF
|
||||
cat > crates/libdep/Cargo.toml <<'EOF'
|
||||
[package]
|
||||
name = "libdep"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
EOF
|
||||
cat > crates/libdep/src/lib.rs <<'EOF'
|
||||
pub struct CheckIn;
|
||||
|
||||
impl CheckIn {
|
||||
pub fn new() -> Self {
|
||||
CheckIn
|
||||
}
|
||||
}
|
||||
EOF
|
||||
cat > crates/libuser/Cargo.toml <<'EOF'
|
||||
[package]
|
||||
name = "libuser"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
libdep = { path = "../libdep" }
|
||||
EOF
|
||||
cat > crates/libuser/src/lib.rs <<'EOF'
|
||||
pub fn make() -> libdep::CheckIn {
|
||||
libdep::CheckIn::new()
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
GIT_AUTHOR_DATE="2023-01-01T00:00:00" GIT_COMMITTER_DATE="2023-01-01T00:00:00" \
|
||||
git commit -q -m "base: CheckIn::new only"
|
||||
base_sha=$(git rev-parse HEAD)
|
||||
|
||||
# dev's own future: adds with_intensity_opt, with an OLD committer date —
|
||||
# analogous to a PR merged into dev before this branch's own build.
|
||||
git switch -q -c dev
|
||||
cat > crates/libdep/src/lib.rs <<'EOF'
|
||||
pub struct CheckIn;
|
||||
|
||||
impl CheckIn {
|
||||
pub fn new() -> Self {
|
||||
CheckIn
|
||||
}
|
||||
|
||||
pub fn with_intensity_opt(&self) -> Self {
|
||||
CheckIn
|
||||
}
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
GIT_AUTHOR_DATE="2023-06-01T01:02:51" GIT_COMMITTER_DATE="2023-06-01T01:02:51" \
|
||||
git commit -q -m "journal: add CheckIn::with_intensity_opt"
|
||||
dev2_sha=$(git rev-parse HEAD)
|
||||
|
||||
# feature branch, forked BEFORE dev2 exists — does its own unrelated work,
|
||||
# and gets its own real build (real wall-clock dep-info mtimes) before dev2
|
||||
# is ever merged in.
|
||||
git switch -q -c feature "$base_sha"
|
||||
cat > crates/libuser/src/lib.rs <<'EOF'
|
||||
pub fn make() -> libdep::CheckIn {
|
||||
libdep::CheckIn::new()
|
||||
}
|
||||
|
||||
pub fn unrelated_feature_work() -> u32 {
|
||||
42
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
git commit -q -m "feature: unrelated libuser work"
|
||||
feat1_sha=$(git rev-parse HEAD)
|
||||
|
||||
echo "=== feature branch's own real build (real wall-clock artifact mtimes) ==="
|
||||
CARGO_TARGET_DIR="$target_feature" cargo build --workspace --quiet
|
||||
# Two pristine snapshots taken right after this one real build, before
|
||||
# scenario 1's failed build attempt below partially rewrites fingerprints
|
||||
# against it: one WITHOUT a watermark file (scenario 1's starting state) and
|
||||
# one WITH (scenario 2's) — both need to restart from this same pre-merge
|
||||
# baseline, not from whatever a failed build left behind, and scenario 1
|
||||
# must never see a watermark or it isn't testing the control it claims to.
|
||||
cp -a "$target_feature" "$target_feature.pristine-no-watermark"
|
||||
echo "$feat1_sha" > "$target_feature/.ci-watermark-sha"
|
||||
cp -a "$target_feature" "$target_feature.pristine-with-watermark"
|
||||
|
||||
# The commit below ("feature: use with_intensity_opt") gets a real "now"
|
||||
# date — scenario 1 needs it to be unambiguously newer than the dep-info
|
||||
# artifact just built above, in whole-second terms git commit timestamps
|
||||
# use. This harness runs fast enough that, without a gap, the two could
|
||||
# otherwise land in the same wall-clock second and make scenario 1 flaky.
|
||||
# Real CI never runs this close together; this is purely a test-harness
|
||||
# margin, matching the one before scenarios 4/5 below.
|
||||
sleep 2
|
||||
|
||||
git merge -q dev -m "merge dev into feature"
|
||||
cat > crates/libuser/src/lib.rs <<'EOF'
|
||||
pub fn make() -> libdep::CheckIn {
|
||||
libdep::CheckIn::new().with_intensity_opt()
|
||||
}
|
||||
|
||||
pub fn unrelated_feature_work() -> u32 {
|
||||
42
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
git commit -q -m "feature: use with_intensity_opt"
|
||||
feat2_sha=$(git rev-parse HEAD)
|
||||
|
||||
echo
|
||||
echo "=== 1: NO WATERMARK, same-branch merge (control — expect the bug) ==="
|
||||
rm -rf "$target_feature"
|
||||
cp -a "$target_feature.pristine-no-watermark" "$target_feature"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes"
|
||||
set +e
|
||||
CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run1.log" 2>&1
|
||||
set -e
|
||||
assert_log_has "$scratch/run1.log" "no method named .with_intensity_opt." \
|
||||
"scenario 1: absent watermark reproduces the stale-reuse compile error"
|
||||
|
||||
echo
|
||||
echo "=== 2: WATERMARK PRESENT, same state (fix — expect success) ==="
|
||||
rm -rf "$target_feature"
|
||||
cp -a "$target_feature.pristine-with-watermark" "$target_feature"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run2.log" 2>&1
|
||||
assert_log_lacks "$scratch/run2.log" "no method named" \
|
||||
"scenario 2: watermark present — build succeeds, no stale reuse"
|
||||
assert_log_has "$scratch/run2.log" "Compiling libdep" \
|
||||
"scenario 2: libdep actually recompiled (not just Fresh-passthrough)"
|
||||
|
||||
echo
|
||||
echo "=== 3: WATERMARK PRESENT, no further change (warm-path preserved) ==="
|
||||
echo "$feat2_sha" > "$target_feature/.ci-watermark-sha"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run3.log" 2>&1
|
||||
assert_log_lacks "$scratch/run3.log" "Compiling" \
|
||||
"scenario 3: unchanged re-run stays warm — nothing recompiles"
|
||||
|
||||
# Scenarios 4/5: a brand-new namespace seeded from feature's own (now warm)
|
||||
# target dir, then merging a THIRD lineage's older change. `sleep` keeps the
|
||||
# THIRD lineage's own "now"-dated commits a full second clear of the
|
||||
# artifact just built above — git commit timestamps are whole-second, dep-info
|
||||
# mtimes are nanosecond, and this harness runs fast enough that the two could
|
||||
# otherwise land in the same second and make scenario 4's control flaky. Real
|
||||
# CI never runs this close together; this is purely a test-harness margin.
|
||||
sleep 2
|
||||
|
||||
git switch -q -c parallel2 "$dev2_sha"
|
||||
cat > crates/libdep/src/lib.rs <<'EOF'
|
||||
pub struct CheckIn;
|
||||
|
||||
impl CheckIn {
|
||||
pub fn new() -> Self {
|
||||
CheckIn
|
||||
}
|
||||
|
||||
pub fn with_intensity_opt(&self) -> Self {
|
||||
CheckIn
|
||||
}
|
||||
|
||||
pub fn with_note(&self) -> Self {
|
||||
CheckIn
|
||||
}
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
GIT_AUTHOR_DATE="2023-07-15T09:00:00" GIT_COMMITTER_DATE="2023-07-15T09:00:00" \
|
||||
git commit -q -m "journal: add CheckIn::with_note"
|
||||
|
||||
# newbranch forks from feature at feat2_sha (scenario 2/3's fixed, warm
|
||||
# state) and merges parallel2. Clean merge: newbranch already has dev2's
|
||||
# with_intensity_opt via feature's earlier merge, parallel2 only adds
|
||||
# with_note on top of that same ancestor.
|
||||
git switch -q -c newbranch "$feat2_sha"
|
||||
git merge -q parallel2 -m "merge parallel2 into newbranch"
|
||||
sed -i 's/CheckIn::new()\.with_intensity_opt()/CheckIn::new().with_intensity_opt().with_note()/' \
|
||||
crates/libuser/src/lib.rs
|
||||
git add -A
|
||||
git commit -q -m "newbranch: use with_note too"
|
||||
|
||||
echo
|
||||
echo "=== 4: SEEDED NAMESPACE, watermark stripped (control — expect the bug) ==="
|
||||
rm -rf "$target_newbranch"
|
||||
cp -a "$target_feature" "$target_newbranch"
|
||||
rm -f "$target_newbranch/.ci-watermark-sha"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_newbranch" bash "$restore_mtimes"
|
||||
set +e
|
||||
CARGO_TARGET_DIR="$target_newbranch" cargo build --workspace -v > "$scratch/run4.log" 2>&1
|
||||
set -e
|
||||
assert_log_has "$scratch/run4.log" "no method named .with_note." \
|
||||
"scenario 4: a seeded namespace with no inherited watermark reproduces the same bug"
|
||||
|
||||
echo
|
||||
echo "=== 5: SEEDED NAMESPACE, watermark inherited via cp -a (fix — expect success) ==="
|
||||
rm -rf "$target_newbranch"
|
||||
cp -a "$target_feature" "$target_newbranch"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_newbranch" bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_newbranch" cargo build --workspace -v > "$scratch/run5.log" 2>&1
|
||||
assert_log_lacks "$scratch/run5.log" "no method named" \
|
||||
"scenario 5: inherited watermark — build succeeds on the namespace's first run"
|
||||
assert_log_has "$scratch/run5.log" "Compiling libdep" \
|
||||
"scenario 5: libdep actually recompiled on the seeded namespace"
|
||||
|
||||
# Scenarios 6/7: a second job in the same workflow can share the
|
||||
# `ci` job's per-branch $CARGO_TARGET_DIR namespace — `debug/` (the `ci`
|
||||
# job's fmt/clippy/test steps) and `release/` (the `web` job's
|
||||
# build-web-opt) sit side by side in one directory. The FIRST version of
|
||||
# these scenarios ran restore-mtimes.sh ONCE, then built both profiles
|
||||
# against its single output — modelling "both jobs read before either
|
||||
# writes". Review caught that this is the wrong ordering: on a serial
|
||||
# runner with no `needs:` between the two jobs, one job's watermark WRITE
|
||||
# can land before the other job's own READ, on the very same trigger. These
|
||||
# scenarios now run restore-mtimes.sh TWICE IN SEQUENCE, with a watermark
|
||||
# write in between — the real ordering — and scenario 6 is required to
|
||||
# reproduce the bug against a single shared watermark file before scenario 7
|
||||
# demonstrates the fix (one watermark file per consumer). `target_feature`
|
||||
# is already warm and consistent at feat2_sha from scenarios 2/3 (debug/
|
||||
# built, watermark=feat2_sha); build release/ against the same namespace too
|
||||
# so both profile dirs start warm, exactly as they would after both real
|
||||
# jobs have run once against a branch.
|
||||
echo
|
||||
echo "=== building the release profile too, so both profile dirs start warm ==="
|
||||
CARGO_TARGET_DIR="$target_feature" cargo build --workspace --release --quiet
|
||||
cp -a "$target_feature" "$scratch/mp-pristine-shared-watermark"
|
||||
cp -a "$target_feature" "$scratch/mp-pristine-split-watermarks"
|
||||
cp "$scratch/mp-pristine-split-watermarks/.ci-watermark-sha" \
|
||||
"$scratch/mp-pristine-split-watermarks/.ci-watermark-sha-wasm32"
|
||||
|
||||
# A third lineage, forked from dev2_sha (same shape as parallel2 above),
|
||||
# adding a further method with an OLD committer date — analogous to a second,
|
||||
# later dev commit that both real jobs would need to see as a real change.
|
||||
sleep 2
|
||||
git switch -q -c parallel3 "$dev2_sha"
|
||||
cat > crates/libdep/src/lib.rs <<'EOF'
|
||||
pub struct CheckIn;
|
||||
|
||||
impl CheckIn {
|
||||
pub fn new() -> Self {
|
||||
CheckIn
|
||||
}
|
||||
|
||||
pub fn with_intensity_opt(&self) -> Self {
|
||||
CheckIn
|
||||
}
|
||||
|
||||
pub fn with_flag(&self) -> Self {
|
||||
CheckIn
|
||||
}
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
GIT_AUTHOR_DATE="2023-08-01T09:00:00" GIT_COMMITTER_DATE="2023-08-01T09:00:00" \
|
||||
git commit -q -m "journal: add CheckIn::with_flag"
|
||||
|
||||
# multiprofile forks from feat2_sha (feature's own fixed, warm point) and
|
||||
# merges parallel3 — clean merge, same shape as newbranch/parallel2 above.
|
||||
git switch -q -c multiprofile "$feat2_sha"
|
||||
git merge -q parallel3 -m "merge parallel3 into multiprofile"
|
||||
cat > crates/libuser/src/lib.rs <<'EOF'
|
||||
pub fn make() -> libdep::CheckIn {
|
||||
libdep::CheckIn::new().with_intensity_opt().with_flag()
|
||||
}
|
||||
|
||||
pub fn unrelated_feature_work() -> u32 {
|
||||
42
|
||||
}
|
||||
EOF
|
||||
git add -A
|
||||
git commit -q -m "multiprofile: use with_flag too"
|
||||
mp_sha=$(git rev-parse HEAD)
|
||||
|
||||
echo
|
||||
echo "=== 6: TWO CONSUMERS, ONE SHARED WATERMARK FILE, sequential jobs (control — expect the bug) ==="
|
||||
# "Job A" (ci-equivalent, debug/ consumer) runs FIRST: its own checkout,
|
||||
# its own restore-mtimes call, its own build, then its own watermark write —
|
||||
# exactly the actions' real step order, just inlined here instead of split
|
||||
# across two jobs.
|
||||
rm -rf "$target_multiprofile"
|
||||
cp -a "$scratch/mp-pristine-shared-watermark" "$target_multiprofile"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace -v > "$scratch/run6-debug.log" 2>&1
|
||||
assert_log_lacks "$scratch/run6-debug.log" "no method named" \
|
||||
"scenario 6: job A (debug/, running first) builds cleanly off the true prior watermark"
|
||||
assert_log_has "$scratch/run6-debug.log" "Compiling libdep" \
|
||||
"scenario 6: job A actually recompiled libdep in debug/"
|
||||
echo "$mp_sha" > "$target_multiprofile/.ci-watermark-sha"
|
||||
|
||||
# "Job B" (web-equivalent, release/ consumer) runs SECOND, same trigger, same
|
||||
# HEAD, same $CARGO_TARGET_DIR, no `needs:` between them — the real ordering
|
||||
# review caught. Its own checkout (stamp_checkout_now again, faithfully
|
||||
# modelling a separate checkout), then its own restore-mtimes call, which
|
||||
# reads the SAME watermark file job A just advanced to mp_sha: HEAD..mp_sha
|
||||
# is an empty diff, so the override never fires and every path falls back to
|
||||
# its plain git-log timestamp — including the backdated libdep change.
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes"
|
||||
set +e
|
||||
CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace --release -v > "$scratch/run6-release.log" 2>&1
|
||||
set -e
|
||||
assert_log_has "$scratch/run6-release.log" "no method named .with_flag." \
|
||||
"scenario 6: job B (release/, running second) reads the watermark job A just advanced, gets an empty diff, and reproduces the original stale-reuse bug"
|
||||
|
||||
echo
|
||||
echo "=== 7: TWO CONSUMERS, TWO WATERMARK FILES, sequential jobs (fix — expect success) ==="
|
||||
# Same sequential ordering as scenario 6, but each job reads and writes its
|
||||
# OWN watermark file (CI_WATERMARK_FILE), exactly as the cargo-cache action does.
|
||||
rm -rf "$target_multiprofile"
|
||||
cp -a "$scratch/mp-pristine-split-watermarks" "$target_multiprofile"
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace -v > "$scratch/run7-debug.log" 2>&1
|
||||
assert_log_lacks "$scratch/run7-debug.log" "no method named" \
|
||||
"scenario 7: job A (debug/) builds cleanly off its own watermark"
|
||||
assert_log_has "$scratch/run7-debug.log" "Compiling libdep" \
|
||||
"scenario 7: job A actually recompiled libdep in debug/"
|
||||
echo "$mp_sha" > "$target_multiprofile/.ci-watermark-sha"
|
||||
|
||||
# Job B reads ITS OWN file (.ci-watermark-sha-wasm32), untouched by job A's
|
||||
# write above — still feat2_sha, so the diff against mp_sha is exactly the
|
||||
# real change set, not empty.
|
||||
stamp_checkout_now
|
||||
CARGO_TARGET_DIR="$target_multiprofile" CI_WATERMARK_FILE=.ci-watermark-sha-wasm32 \
|
||||
bash "$restore_mtimes"
|
||||
CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace --release -v > "$scratch/run7-release.log" 2>&1
|
||||
assert_log_lacks "$scratch/run7-release.log" "no method named" \
|
||||
"scenario 7: job B (release/) builds cleanly off its OWN watermark, unaffected by job A's write"
|
||||
assert_log_has "$scratch/run7-release.log" "Compiling libdep" \
|
||||
"scenario 7: job B actually recompiled libdep in release/"
|
||||
|
||||
echo
|
||||
echo "ALL ${pass_count} ASSERTIONS PASSED"
|
||||
Executable
+345
@@ -0,0 +1,345 @@
|
||||
#!/usr/bin/env bash
|
||||
# Restores every git-tracked file's mtime to the timestamp of the most recent
|
||||
# commit that touched it. `actions/checkout` stamps every file with the
|
||||
# checkout's wall-clock "now", and Cargo's fingerprint freshness check is
|
||||
# mtime-based by default. Result: even though the persistent volume genuinely
|
||||
# holds prior build artifacts, every checkout makes every tracked file look
|
||||
# "newer than what was recorded at last build", so every crate rebuilds from
|
||||
# scratch on every run — measured directly on a Bevy-sized workspace before
|
||||
# this fix, by grepping a warm-volume run's log for Cargo's `Fresh` cache-hit
|
||||
# marker: zero matches across ~28 crates. Do not remove this as "redundant
|
||||
# with the target-dir cache": it is the fix that makes that cache take effect.
|
||||
#
|
||||
# Algorithm: a single `git log --name-only --no-renames` traversal
|
||||
# (newest-commit-first, git log's default order) instead of one `git log`
|
||||
# call per file — for a workspace this size a per-file walk would mean
|
||||
# hundreds of separate history walks. For each path, the first timestamp
|
||||
# encountered under newest-first traversal is the timestamp of the most
|
||||
# recent commit that touched it. Only files currently tracked by git
|
||||
# (`git ls-files`) are touched; deleted/renamed-away historical paths and
|
||||
# anything gitignored are left alone.
|
||||
#
|
||||
# Each commit's timestamp line carries a `@@MTIME@@` sentinel prefix rather
|
||||
# than relying on blank-line paragraph boundaries between commits. Consecutive
|
||||
# merge commits with an empty diff (under `--no-renames`) print their
|
||||
# timestamp lines back-to-back with no blank line between them — a
|
||||
# blank-line-based parser misreads the second bare timestamp line as a
|
||||
# filename and desyncs every timestamp after it. The sentinel can't collide
|
||||
# with a real tracked path (it's a fixed literal no repo file is named), so
|
||||
# every commit's timestamp line is unambiguous regardless of what precedes or
|
||||
# follows it.
|
||||
#
|
||||
# Note: for a handful of paths this single raw pass picks a nominally
|
||||
# different (usually newer) commit than `git log -1 -- <path>` would, which
|
||||
# applies pathspec history-simplification (it can skip a commit whose change
|
||||
# to a path got superseded by how a later merge resolved that path). That
|
||||
# divergence is harmless here: correctness only requires (a) determinism —
|
||||
# same HEAD always yields the same mtime for a given path, so a warm cache
|
||||
# stays warm — and (b) change-sensitivity — a commit that actually changes a
|
||||
# path's content is always the newest entry touching that path in traversal
|
||||
# order, so it always wins the "first seen" race and invalidates that path's
|
||||
# fingerprint. Reproducing full history-simplification per path would mean
|
||||
# back to one `git log` call per file, the exact cost this single pass
|
||||
# avoids.
|
||||
#
|
||||
# Directory mtimes, not just file mtimes: a build script that declares
|
||||
# `cargo::rerun-if-changed=<some-dir>` (a directory rather than a file) has
|
||||
# its rerun decision keyed partly on that directory inode's own mtime, and
|
||||
# checkout recreates every directory with a fresh "now" mtime just like it
|
||||
# does files. Restoring only tracked FILES leaves such a directory always
|
||||
# looking newer than the recorded fingerprint, so the build script re-runs
|
||||
# every time, marking its crate dirty and cascading to every downstream crate
|
||||
# regardless of whether anything under the directory changed. Fixed by
|
||||
# restoring every ancestor directory of every tracked path (excluding `.git/`,
|
||||
# which no tracked path can be under) to the MAXIMUM restored timestamp among
|
||||
# the tracked files in its subtree — deterministic given the per-file
|
||||
# timestamps already computed above, and it still fires a rerun when a file
|
||||
# under that directory genuinely changed, since that file's newer timestamp
|
||||
# propagates to every ancestor's max. Costs about a second; worth paying even
|
||||
# on a workspace with no such build script yet, so the day one appears the
|
||||
# cache stability already covers it.
|
||||
#
|
||||
# Soundness against the cross-branch stale-binary hazard: the cargo-cache
|
||||
# action gives every ref its own target directory on the shared volume, so no
|
||||
# two refs' fingerprints ever share a directory and this script never has to
|
||||
# arbitrate freshness across refs — only within one ref's own history, which
|
||||
# is exactly what it is built to do soundly. On a nightly toolchain,
|
||||
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS is a complementary, stronger guarantee
|
||||
# (content-addressed rather than mtime-based freshness); this script is not
|
||||
# made redundant by it, because directory-form `rerun-if-changed` build-script
|
||||
# watches are not covered by it and stable historical mtimes stay cheap
|
||||
# belt-and-braces alongside it.
|
||||
#
|
||||
# The merge hazard (the merge hazard), and why it needed a watermark, not just a
|
||||
# namespace: namespacing keeps two branches' fingerprints apart, but it does
|
||||
# NOT make the comparison this script feeds sound on its own. The comparison
|
||||
# that actually matters is not "is this file's mtime the historically correct
|
||||
# one for its most recent commit" (which the git-log pass above answers
|
||||
# exactly right) — it's **"is this file's mtime newer than the dep-info
|
||||
# artifact Cargo already has on disk for it in THIS target dir"**, because
|
||||
# that dep-info's own mtime is a real wall-clock timestamp stamped by rustc
|
||||
# when it last wrote the artifact, not anything git-derived. Ordinary
|
||||
# same-branch iteration makes the git-log answer and the Cargo answer agree
|
||||
# for free, by causality: you cannot commit a change before the build it is
|
||||
# meant to invalidate has already finished, so every file changed since the
|
||||
# last build necessarily carries a commit timestamp after that build's real
|
||||
# completion time. A merge breaks exactly that causal link — it can pull in a
|
||||
# commit authored on a parallel lineage *before* this branch's own last
|
||||
# build, so the git-log answer ("this file's most recent commit is old") and
|
||||
# the Cargo answer ("this file changed since my last build") diverge, and
|
||||
# restore-mtimes stamping the historically-correct-but-too-old mtime is what
|
||||
# let Cargo reuse an rlib built before the method the merge introduced
|
||||
# existed (observed in production as an "impossible" compile error: a crate
|
||||
# reusing an rlib built before the method a merge introduced existed).
|
||||
#
|
||||
# The fix: for a file changed since *this target dir's own last successful
|
||||
# build* — not "recently" in absolute git-log time — always stamp "now"
|
||||
# (this run's own wall-clock start), never the historical commit timestamp,
|
||||
# because "now" is provably later than that build's real completion time no
|
||||
# matter what any commit's authored/committer date says. the cargo-cache-publish action's watermark step writes the built HEAD SHA into
|
||||
# "$CARGO_TARGET_DIR/${CI_WATERMARK_FILE:-.ci-watermark-sha}" after every
|
||||
# green run of THIS branch's own namespace; this script reads it back (if
|
||||
# present — see the read below for what "absent" means and why it's still
|
||||
# safe) and treats `git diff --name-only <watermark>..HEAD` as authoritative
|
||||
# over the git-log-only answer for exactly those paths. Everything NOT in
|
||||
# that diff keeps the git-log timestamp exactly as before, so the ordinary
|
||||
# no-merge warm-cache case (the whole reason this script exists) is
|
||||
# untouched.
|
||||
#
|
||||
# CI_WATERMARK_FILE: the watermark filename is a
|
||||
# per-job parameter, NOT a per-branch constant — two jobs sharing one
|
||||
# $CARGO_TARGET_DIR namespace must use two DIFFERENT watermark files, never
|
||||
# one. A watermark answers "what did THIS consumer last successfully build",
|
||||
# and two jobs building different target triples (e.g. one job's host debug/release vs another's
|
||||
# wasm32-unknown-unknown) are two separate
|
||||
# consumers with two separate answers. A single shared file breaks the
|
||||
# moment a second job joins the namespace: job A reads the true prior
|
||||
# watermark, rebuilds correctly, and advances the file to HEAD; job B then
|
||||
# runs SECOND (same trigger, same HEAD, no `needs:` ordering — a serial
|
||||
# runner still executes both jobs one after another within one trigger) and
|
||||
# reads that JUST-ADVANCED watermark, computing an empty HEAD..HEAD diff, and
|
||||
# falls back to git-log-only timestamps for everything — exactly this
|
||||
# script's pre-fix behaviour, reintroduced for whichever job happens to run
|
||||
# second on any trigger that includes a backdating merge. Caught via a two-crate repro, not by inspection alone.
|
||||
# restore-mtimes-selftest.sh's scenarios 6/7 exercise this exact sequential
|
||||
# ordering (one job's watermark write landing between two restore-mtimes
|
||||
# invocations against the same namespace), not just two simultaneous reads.
|
||||
# Defaults to ".ci-watermark-sha" — unset callers (a local/manual run, or a
|
||||
# namespace with only ever one consumer) get the original filename and
|
||||
# behaviour unchanged.
|
||||
#
|
||||
# restore-mtimes-selftest.sh, alongside this file, is a regression test for
|
||||
# the watermark mechanism: a real scratch cargo workspace, a real
|
||||
# restore-mtimes.sh run, a real `cargo build`, asserting both that the
|
||||
# staleness bug reproduces without a watermark and that it's fixed with one
|
||||
# (same-branch merge AND seeded-namespace cases). Run it by hand after touching this
|
||||
# script's timestamp logic or the actions' seed/watermark step order.
|
||||
#
|
||||
# The watermark file deliberately lives INSIDE $CARGO_TARGET_DIR rather than
|
||||
# a separate location, for two reasons that both matter: it rides along for
|
||||
# free with prune-cache.sh's eviction pass (an evicted
|
||||
# namespace loses its watermark along with everything else it would have
|
||||
# been wrong to trust anyway), and it rides along for free with the
|
||||
# seed-snapshot mechanism (the cargo-cache-publish action publishes
|
||||
# $CARGO_TARGET_DIR wholesale, including this file) — so a brand-new branch
|
||||
# namespace seeded from its base's published snapshot inherits the base's
|
||||
# watermark along with the base's artifacts, and this script correctly diffs
|
||||
# from THAT SHA rather than treating the seeded branch as having no watermark
|
||||
# at all. That is exactly the same hazard class one level up (a new branch's
|
||||
# first run reusing artifacts seeded from a base build that predates some of
|
||||
# the base's own history), and it needs the same fix — which it gets for
|
||||
# free rather than needing separate handling, as long as the cargo-cache action orders "seed
|
||||
# or reuse the target dir" before "restore mtimes" so the watermark is
|
||||
# actually in place, on disk, by the time this script runs. (It is — see the
|
||||
# step order in the cargo-cache action.)
|
||||
#
|
||||
# Absent watermark (no file, or the target dir doesn't exist yet at all) is
|
||||
# always safe, never a silent miscompare: this script simply skips the
|
||||
# override and falls back fully to the git-log timestamps, exactly as it did
|
||||
# before this fix existed. That is correct for a genuinely first run against
|
||||
# an empty, unseeded namespace (nothing to compare against — everything is
|
||||
# about to be a cold build regardless of what mtime it gets) and conservative
|
||||
# for every other case that could produce an absent watermark: a watermark SHA
|
||||
# that is no longer reachable in this checkout (a force-push or rebase
|
||||
# rewrote the branch the watermark was recorded against) is NOT treated the
|
||||
# same as "no watermark at all" — the reachability check below instead
|
||||
# treats every tracked file as changed, rather than silently trusting a diff
|
||||
# against a commit that may no longer describe any real ancestry
|
||||
# relationship to HEAD. A run that fails before reaching
|
||||
# the "Record build watermark" step simply never advances the watermark, so
|
||||
# the next run's diff base stays at the last GREEN build — over-inclusive
|
||||
# (it may re-stamp files that a failed run partially rebuilt anyway) but
|
||||
# never under-inclusive, which is the only direction that matters for
|
||||
# soundness.set -euo pipefail
|
||||
|
||||
repo_root=$(git rev-parse --show-toplevel)
|
||||
cd "$repo_root"
|
||||
|
||||
# A shallow checkout (the `actions/checkout` default) only has the tip
|
||||
# commit available, which diffs against the empty tree — every tracked file
|
||||
# would resolve to that single commit's timestamp, a uniform stamp that
|
||||
# changes with every new commit and still defeats the fingerprint cache
|
||||
# run-over-run (the exact bug this script fixes). That failure mode is
|
||||
# silent otherwise: every path still "resolves" to some timestamp (just the
|
||||
# wrong, unhelpfully-uniform one), so the missing-path fallback below never
|
||||
# fires and nothing else in this script would surface it. Fail loudly here
|
||||
# instead so a future `fetch-depth` regression on the checkout step shows up
|
||||
# in the CI log rather than silently reverting to full rebuilds every run.
|
||||
if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then
|
||||
echo "::error::restore-mtimes: shallow clone detected — per-file mtimes unavailable, cache reuse will be defeated; set fetch-depth: 0 on the checkout step" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
start=$(date +%s)
|
||||
|
||||
resolved=$(mktemp)
|
||||
tracked=$(mktemp)
|
||||
to_touch=$(mktemp)
|
||||
missing=$(mktemp)
|
||||
all_touched=$(mktemp)
|
||||
dir_touch=$(mktemp)
|
||||
changed_since_watermark=$(mktemp)
|
||||
to_touch_overridden=$(mktemp)
|
||||
trap 'rm -f "$resolved" "$tracked" "$to_touch" "$missing" "$all_touched" "$dir_touch" "$changed_since_watermark" "$to_touch_overridden"' EXIT
|
||||
|
||||
# Emit "<epoch>\t<path>" once per path, first-seen-under-newest-first wins.
|
||||
git log --format='@@MTIME@@%ct' --name-only --no-renames | awk '
|
||||
/^@@MTIME@@[0-9]+$/ { ts = substr($0, 10); next }
|
||||
NF == 0 { next }
|
||||
!($0 in seen) { seen[$0] = 1; printf "%s\t%s\n", ts, $0 }
|
||||
' > "$resolved"
|
||||
|
||||
git ls-files > "$tracked"
|
||||
|
||||
# Restrict to files git currently tracks — never .git/ internals, never
|
||||
# gitignored build output, never a path that's since been deleted/renamed.
|
||||
awk -F'\t' '
|
||||
NR == FNR { track[$0] = 1; next }
|
||||
$2 in track { print; seen[$2] = 1 }
|
||||
' "$tracked" "$resolved" > "$to_touch"
|
||||
|
||||
# Watermark override: see the header comment above for why this
|
||||
# comparison — against this target dir's own last successful build, not
|
||||
# against absolute git-log history — is the one Cargo actually needs.
|
||||
watermark_count=0
|
||||
watermark_status="no CARGO_TARGET_DIR set (not running under the cargo-cache action) — skipped"
|
||||
watermark_name="${CI_WATERMARK_FILE:-.ci-watermark-sha}"
|
||||
watermark_file="${CARGO_TARGET_DIR:-}/${watermark_name}"
|
||||
if [ -n "${CARGO_TARGET_DIR:-}" ] && [ -f "$watermark_file" ]; then
|
||||
watermark_sha=$(cat "$watermark_file")
|
||||
if git cat-file -e "${watermark_sha}^{commit}" 2>/dev/null; then
|
||||
# Every path that differs between the watermark commit and HEAD —
|
||||
# added, modified, or deleted — content-wise, regardless of what any
|
||||
# commit along the way claims its own timestamp is.
|
||||
git diff --name-only "$watermark_sha" HEAD > "$changed_since_watermark"
|
||||
watermark_status="diffing against watermark ${watermark_sha} in ${CARGO_TARGET_DIR}/${watermark_name}"
|
||||
else
|
||||
# The watermark names a commit this checkout can no longer see — a
|
||||
# force-push or rebase rewrote history out from under it. Trusting the
|
||||
# diff would silently compare HEAD against a commit with no real
|
||||
# ancestry relationship to it. Fall back to treating every tracked file
|
||||
# as changed since the (unknowable) last build: safe (over-invalidates,
|
||||
# never under-invalidates) and self-heals the moment the next green run
|
||||
# records a fresh, reachable watermark.
|
||||
cp "$tracked" "$changed_since_watermark"
|
||||
watermark_status="watermark ${watermark_sha} unreachable (force-push/rebase?) in ${CARGO_TARGET_DIR}/${watermark_name} — treating every tracked file as changed since last build"
|
||||
fi
|
||||
else
|
||||
: > "$changed_since_watermark"
|
||||
if [ -n "${CARGO_TARGET_DIR:-}" ]; then
|
||||
watermark_status="no watermark yet in ${CARGO_TARGET_DIR}/${watermark_name} (first run against this namespace) — skipped"
|
||||
fi
|
||||
fi
|
||||
|
||||
# For every path in the diff, override its git-log timestamp with "now"
|
||||
# ($start, this run's own wall-clock start) — provably later than the real
|
||||
# completion time of the build the watermark names, unlike any commit's
|
||||
# authored/committer date. Everything NOT in the diff keeps its git-log
|
||||
# timestamp exactly as before.
|
||||
if [ -s "$changed_since_watermark" ]; then
|
||||
# Re-derive the count from the intersection with $to_touch (rather than
|
||||
# `wc -l < "$changed_since_watermark"` directly) so the reported number
|
||||
# only counts paths this script is actually about to stamp differently,
|
||||
# not every path `git diff` names (which, unlike $to_touch, is not
|
||||
# restricted to currently-tracked files — e.g. a path deleted since the
|
||||
# watermark commit).
|
||||
watermark_count=$(awk -F'\t' 'NR == FNR { c[$0] = 1; next } $2 in c' "$changed_since_watermark" "$to_touch" | wc -l | tr -d ' ')
|
||||
|
||||
awk -F'\t' -v now="$start" '
|
||||
NR == FNR { changed[$0] = 1; next }
|
||||
{ if ($2 in changed) print now "\t" $2; else print }
|
||||
' "$changed_since_watermark" "$to_touch" > "$to_touch_overridden"
|
||||
mv "$to_touch_overridden" "$to_touch"
|
||||
fi
|
||||
|
||||
# Any tracked path whose history couldn't resolve (only possible under an
|
||||
# unexpectedly shallow checkout, since `fetch-depth: 0` on the checkout step
|
||||
# means full history is normally present) falls back to a fixed constant
|
||||
# epoch rather than being left alone at checkout-time "now". What Cargo's
|
||||
# fingerprint check needs is an mtime that's STABLE run-over-run for
|
||||
# unchanged content, not necessarily a historically accurate one — a
|
||||
# constant epoch gives that stability so the file still gets cache reuse
|
||||
# instead of permanently missing the cache on every single run.
|
||||
awk -F'\t' '
|
||||
NR == FNR { seen[$2] = 1; next }
|
||||
!($0 in seen)
|
||||
' "$to_touch" "$tracked" > "$missing"
|
||||
|
||||
touched=0
|
||||
while IFS=$'\t' read -r ts path; do
|
||||
[ -e "$path" ] || [ -L "$path" ] || continue
|
||||
# `-h`: set the mtime of a git-tracked symlink itself, never the target it
|
||||
# resolves to. No path in this repo is currently a tracked symlink, but a
|
||||
# future one (e.g. a shared-config symlink into a dotfiles
|
||||
# checkout) would otherwise get its mtime applied
|
||||
# to whatever it resolves to outside the repo — keeping `-h` costs nothing
|
||||
# today and avoids that surprise later.
|
||||
touch -h -d "@$ts" -- "$path"
|
||||
touched=$((touched + 1))
|
||||
done < "$to_touch"
|
||||
|
||||
fallback_count=0
|
||||
if [ -s "$missing" ]; then
|
||||
while IFS= read -r path; do
|
||||
[ -e "$path" ] || [ -L "$path" ] || continue
|
||||
touch -h -d "@0" -- "$path"
|
||||
fallback_count=$((fallback_count + 1))
|
||||
done < "$missing"
|
||||
fi
|
||||
|
||||
# Directory pass: for every tracked file's full ancestor chain (excluding
|
||||
# the file's own leaf name), track the maximum timestamp seen. A directory
|
||||
# with no tracked file directly in it but with tracked files somewhere in
|
||||
# its subtree (e.g. `crates/`) still accumulates a max from those deeper
|
||||
# files, since every ancestor level of every file's path is a separate
|
||||
# prefix entry here. Fixed-epoch fallback entries (ts=0) are included too —
|
||||
# harmless, since 0 can never win a max against any real commit timestamp.
|
||||
cat "$to_touch" > "$all_touched"
|
||||
if [ -s "$missing" ]; then
|
||||
awk '{ print "0\t" $0 }' "$missing" >> "$all_touched"
|
||||
fi
|
||||
awk -F'\t' '
|
||||
{
|
||||
ts = $1; path = $2
|
||||
n = split(path, parts, "/")
|
||||
prefix = ""
|
||||
for (i = 1; i < n; i++) {
|
||||
prefix = (prefix == "" ? parts[i] : prefix "/" parts[i])
|
||||
if (!(prefix in maxts) || ts + 0 > maxts[prefix] + 0) maxts[prefix] = ts
|
||||
}
|
||||
if (!("." in maxts) || ts + 0 > maxts["."] + 0) maxts["."] = ts
|
||||
}
|
||||
END {
|
||||
for (d in maxts) printf "%s\t%s\n", maxts[d], d
|
||||
}
|
||||
' "$all_touched" > "$dir_touch"
|
||||
|
||||
dirs_touched=0
|
||||
while IFS=$'\t' read -r ts dir; do
|
||||
[ -d "$dir" ] || continue
|
||||
touch -d "@$ts" -- "$dir"
|
||||
dirs_touched=$((dirs_touched + 1))
|
||||
done < "$dir_touch"
|
||||
|
||||
elapsed=$(( $(date +%s) - start ))
|
||||
echo "restore-mtimes: touched ${touched} files + ${dirs_touched} dirs of $(wc -l < "$tracked") tracked files from history in ${elapsed}s (${fallback_count} fell back to a fixed epoch, ${watermark_count} overridden to now: ${watermark_status})"
|
||||
Executable
+131
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for seed-target-dir.sh: which source a run seeds from, and
|
||||
# what happens when two jobs sharing one cache key seed at the same time.
|
||||
#
|
||||
# Runs the ACTUAL script against a real scratch cache directory with fake
|
||||
# target trees standing in for cargo output — no compiler needed, so this is
|
||||
# the fast half of the suite. hardlink-clone-selftest.sh covers the parts that
|
||||
# need a real build.
|
||||
#
|
||||
# What each scenario demonstrates:
|
||||
#
|
||||
# 1. BASE SNAPSHOT PREFERRED — a PR whose base has published a snapshot
|
||||
# seeds from it, and the seeded directory really is a hardlink clone
|
||||
# (shared inodes), not a copy.
|
||||
# 2. OWN DIR WINS — a second run of the same ref reuses what is already
|
||||
# there and does not re-seed over its own work.
|
||||
# 3. OWN SNAPSHOT AS SELF-RESTORE — a publisher whose live target dir was
|
||||
# evicted restores from the snapshot it last published, instead of
|
||||
# rebuilding cold.
|
||||
# 4. FALLBACK DIR — with no snapshot at all, an explicitly configured
|
||||
# fallback (a pre-existing flat cache, during a migration) is used.
|
||||
# 5. COLD — with nothing available, the directory is created empty rather
|
||||
# than the script failing.
|
||||
# 6. LOCK FILES STRIPPED — Cargo's in-place-flock'd lock files never
|
||||
# survive a clone, because a shared lock inode would make two branches
|
||||
# contend on one mutex.
|
||||
# 7. CONCURRENT SEED IS ATOMIC — two seeds racing on one cache key: the
|
||||
# loser discards its staging copy and uses the winner's directory, and
|
||||
# at no point is a partially-populated directory visible under the final
|
||||
# name. This is the property that replaces "the runner only has one job
|
||||
# slot" with an actual guarantee.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
|
||||
scratch=$(mktemp -d)
|
||||
trap 'rm -rf "$scratch"' EXIT
|
||||
root="$scratch/cache"
|
||||
mkdir -p "$root"
|
||||
pass_count=0
|
||||
|
||||
fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; }
|
||||
ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; }
|
||||
assert_file() { [ -e "$1" ] || fail "expected $1 to exist ($2)"; ok "$2"; }
|
||||
assert_absent() { [ -e "$1" ] && fail "expected $1 to be gone ($2)"; ok "$2"; }
|
||||
assert_content() { [ "$(cat "$1")" = "$2" ] || fail "expected '$2' in $1, got '$(cat "$1")' ($3)"; ok "$3"; }
|
||||
|
||||
# A plausible target tree: a big shared artifact, a mutable fingerprint, a
|
||||
# build-script output, and a lock file.
|
||||
make_tree() {
|
||||
local d="$1" marker="$2"
|
||||
mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x" "$d/debug/build/x/out"
|
||||
echo "$marker" > "$d/debug/deps/libx.rlib"
|
||||
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
||||
echo "$marker" > "$d/debug/build/x/out/gen.txt"
|
||||
: > "$d/debug/.cargo-lock"
|
||||
}
|
||||
|
||||
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
||||
|
||||
BASE_KEY=$(cache_key dev)
|
||||
OWN_KEY=$(cache_key feat/thing)
|
||||
|
||||
echo "=== 1: base snapshot preferred, and cloned by hardlink ==="
|
||||
make_tree "$root/snapshot-$BASE_KEY" base-content
|
||||
seed "$OWN_KEY" "$BASE_KEY" "$root" job1
|
||||
own="$root/target-$OWN_KEY"
|
||||
assert_content "$own/debug/deps/libx.rlib" base-content "seeded from the base snapshot"
|
||||
[ "$(stat -c '%i' "$own/debug/deps/libx.rlib")" = "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/deps/libx.rlib")" ] \
|
||||
|| fail "artifact was copied, not hardlinked"
|
||||
ok "artifact shares an inode with the snapshot (hardlink clone, not a copy)"
|
||||
[ "$(stat -c '%i' "$own/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/.fingerprint/x/dep-lib-x")" ] \
|
||||
|| fail "fingerprint still shares an inode with the snapshot"
|
||||
ok "fingerprint is privately owned (unshare_mutable_paths ran)"
|
||||
|
||||
echo
|
||||
echo "=== 6: Cargo lock files never survive a clone ==="
|
||||
assert_absent "$own/debug/.cargo-lock" "cloned .cargo-lock removed"
|
||||
|
||||
echo
|
||||
echo "=== 2: an existing own dir is reused, never re-seeded over ==="
|
||||
echo own-work > "$own/debug/deps/libx.rlib"
|
||||
seed "$OWN_KEY" "$BASE_KEY" "$root" job1
|
||||
assert_content "$own/debug/deps/libx.rlib" own-work "own directory reused as-is"
|
||||
grep -q 'reusing this ref' "$scratch/log" || fail "expected the reuse path in the log"
|
||||
ok "reuse is reported in the log"
|
||||
|
||||
echo
|
||||
echo "=== 3: a publisher restores from its own snapshot after eviction ==="
|
||||
make_tree "$root/snapshot-$BASE_KEY" published-dev
|
||||
seed "$BASE_KEY" "" "$root" job1
|
||||
assert_content "$root/target-$BASE_KEY/debug/deps/libx.rlib" published-dev "publisher self-restored from its own snapshot"
|
||||
|
||||
echo
|
||||
echo "=== 4: explicit fallback dir when no snapshot exists ==="
|
||||
OTHER=$(cache_key feat/other)
|
||||
make_tree "$scratch/legacy-flat" legacy
|
||||
seed "$OTHER" "$(cache_key nosuch)" "$root" job1 "$scratch/legacy-flat"
|
||||
assert_content "$root/target-$OTHER/debug/deps/libx.rlib" legacy "seeded from the fallback dir"
|
||||
|
||||
echo
|
||||
echo "=== 5: cold start when nothing is available ==="
|
||||
COLD=$(cache_key feat/cold)
|
||||
seed "$COLD" "$(cache_key nosuch)" "$root" job1
|
||||
[ -d "$root/target-$COLD" ] || fail "cold start did not create the directory"
|
||||
[ -z "$(ls -A "$root/target-$COLD")" ] || fail "cold start directory is not empty"
|
||||
ok "cold start creates an empty directory rather than failing"
|
||||
|
||||
echo
|
||||
echo "=== 7: two jobs racing on one cache key ==="
|
||||
RACE=$(cache_key feat/race)
|
||||
make_tree "$root/snapshot-$BASE_KEY" race-source
|
||||
# Both jobs seed concurrently from the same snapshot into the same key. Each
|
||||
# stages under its own tag, so the only interaction is the final rename.
|
||||
( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobA > "$scratch/logA" 2>&1 ) &
|
||||
( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobB > "$scratch/logB" 2>&1 ) &
|
||||
wait
|
||||
race_dir="$root/target-$RACE"
|
||||
assert_content "$race_dir/debug/deps/libx.rlib" race-source "the surviving directory is complete"
|
||||
[ -z "$(find "$root" -maxdepth 1 -name '.stage-*' -print -quit)" ] || fail "a staging directory was left behind"
|
||||
ok "no staging directory survived the race"
|
||||
# Exactly one job may claim it seeded; the other must report either the
|
||||
# concurrent-peer path or a plain reuse (if it started after the winner
|
||||
# finished). Neither may report a cold start.
|
||||
if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
|
||||
cat "$scratch/logA" "$scratch/logB"; fail "a racing job reported a cold start"
|
||||
fi
|
||||
ok "neither racing job fell through to a cold start"
|
||||
|
||||
echo
|
||||
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
||||
Executable
+81
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
# Consume side: make this run's own target dir exist, warm, and private.
|
||||
#
|
||||
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir]
|
||||
# own-key cache key for this run's own ref
|
||||
# base-key cache key for the ref to layer over ("" for a run whose own
|
||||
# ref IS a reference branch)
|
||||
# cache-root mount point of the persistent volume
|
||||
# tag a per-job-per-run unique string, used to name the staging
|
||||
# directory so two concurrent jobs can never collide on it
|
||||
# fallback-dir optional absolute path to seed from when no snapshot exists
|
||||
# (a legacy flat cache dir during a migration, typically)
|
||||
#
|
||||
# The design in one paragraph: a PR branch's first run hardlink-clones the
|
||||
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
|
||||
# time proportional to inode count rather than data volume — on ext4, with no
|
||||
# reflink support, that is the only way to make "layer over the base" cheap.
|
||||
# Snapshot rather than the base's live target dir, because a live directory is
|
||||
# being written by its own job while a consumer reads it, and a torn read
|
||||
# pairs one build's fingerprint with another build's artifact — which degrades
|
||||
# to a WRONG reuse, not to a safe miss. Neither half is novel; the combination
|
||||
# is what makes the scheme both cheap and sound, instead of one or the other.
|
||||
#
|
||||
# Everything about correctness under concurrency lives in
|
||||
# hardlink_clone_into() and unshare_mutable_paths() in cache-lib.sh — read
|
||||
# those before changing anything here.
|
||||
set -euo pipefail
|
||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||
|
||||
if [ $# -lt 4 ] || [ $# -gt 5 ]; then
|
||||
echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"
|
||||
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
|
||||
|
||||
mkdir -p "$ROOT"
|
||||
|
||||
if [ -d "$OWN_DIR" ]; then
|
||||
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
|
||||
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Source preference, most specific first:
|
||||
#
|
||||
# 1. the base ref's snapshot — the ordinary PR case, and the whole point.
|
||||
# 2. this ref's OWN snapshot — a reference branch whose live target dir was
|
||||
# evicted under disk pressure can restore itself from the last snapshot
|
||||
# it published, instead of paying a cold rebuild. Free, given the
|
||||
# snapshot already exists.
|
||||
# 3. an explicit fallback directory — migration off a pre-existing flat
|
||||
# cache, so the first run under this scheme isn't a needless cold build.
|
||||
CANDIDATES=()
|
||||
[ -n "$BASE_KEY" ] && CANDIDATES+=("$(snapshot_dir_for "$ROOT" "$BASE_KEY"):base snapshot")
|
||||
CANDIDATES+=("$(snapshot_dir_for "$ROOT" "$OWN_KEY"):own snapshot")
|
||||
[ -n "$FALLBACK" ] && CANDIDATES+=("${FALLBACK}:fallback dir")
|
||||
|
||||
for entry in "${CANDIDATES[@]}"; do
|
||||
src="${entry%%:*}"; label="${entry#*:}"
|
||||
[ -d "$src" ] || continue
|
||||
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
|
||||
start=$(date +%s)
|
||||
if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then
|
||||
echo "seed: cloned in $(( $(date +%s) - start ))s"
|
||||
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||
else
|
||||
# Another job sharing this cache key won the rename while we were
|
||||
# cloning. Its directory is complete (the rename is the publish step), so
|
||||
# there is nothing to do but use it — and nothing was ever observable in
|
||||
# a half-seeded state.
|
||||
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
|
||||
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||
fi
|
||||
exit 0
|
||||
done
|
||||
|
||||
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
|
||||
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||
mkdir -p "$OWN_DIR"
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs every selftest in this directory and reports a single verdict.
|
||||
#
|
||||
# bash scripts/selftest.sh # everything
|
||||
# bash scripts/selftest.sh --fast # skip the ones that invoke a compiler
|
||||
#
|
||||
# The compiler-backed tests (hardlink-clone, restore-mtimes) are the ones that
|
||||
# verify behaviour against real Cargo rather than against a fixture, so
|
||||
# --fast is for iterating, not for signing off a change.
|
||||
set -uo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
|
||||
FAST=0
|
||||
[ "${1:-}" = "--fast" ] && FAST=1
|
||||
|
||||
FIXTURE_TESTS=(seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh)
|
||||
CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh)
|
||||
|
||||
TESTS=("${FIXTURE_TESTS[@]}")
|
||||
if [ "$FAST" = "0" ]; then TESTS+=("${CARGO_TESTS[@]}"); fi
|
||||
|
||||
failed=()
|
||||
for t in "${TESTS[@]}"; do
|
||||
echo
|
||||
echo "############################################################"
|
||||
echo "# $t"
|
||||
echo "############################################################"
|
||||
if bash "$script_dir/$t"; then
|
||||
echo "--> $t OK"
|
||||
else
|
||||
echo "--> $t FAILED"
|
||||
failed+=("$t")
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
if [ "${#failed[@]}" -eq 0 ]; then
|
||||
echo "selftest: all ${#TESTS[@]} suites passed"
|
||||
[ "$FAST" = "1" ] && echo "NOTE: --fast skipped ${CARGO_TESTS[*]}"
|
||||
exit 0
|
||||
fi
|
||||
echo "selftest: ${#failed[@]} of ${#TESTS[@]} suites FAILED: ${failed[*]}"
|
||||
exit 1
|
||||
Reference in New Issue
Block a user