diff --git a/README.md b/README.md index 818a517..badbcf3 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,267 @@ -# gitea-actions +# gitdan-actions -Shared Gitea Actions composite actions for gitdan repos (cargo-cache et al) \ No newline at end of file +Shared Gitea Actions composite actions for the gitdan forge. + +Currently one thing, done properly: **`cargo-cache`** — a persistent, +per-branch Cargo build cache for self-hosted Gitea runners, where a pull +request's cache is a near-free hardlink clone of an immutable snapshot its +base branch published. + +> **Final home:** this repository will live at `daniel/gitdan-actions`. Pin +> that path in `uses:` once the transfer completes. + +--- + +## Why this exists + +Two of this forge's Rust projects independently built the same idea and each +got one half right. + +| | seeding mechanism | seed source | +|---|---|---| +| project A | `cp -al` hardlink clone — near-free, cost scales with inode count, not bytes | the base branch's **live** target dir — races a build that is still writing | +| project B | `cp -a` full copy — sound, but ~35 GB duplicated per branch | a **published immutable snapshot** — nothing ever writes it while it is read | + +This action is the diagonal: **hardlink-clone from a published snapshot.** +Cheap like A, sound like B. It also closes a latent race in A by construction +(the seed is staged and swapped in with one atomic rename) rather than relying +on the runner having a single execution slot. + +One thing neither project had, and the reason the clone is not a plain +`cp -al`: **a build inside a hardlink clone does mutate the directory it was +cloned from.** Cargo writes real artifacts by replacing them, but writes its +metadata — and build scripts write their `OUT_DIR` — with a plain truncating +write, straight through the shared inode. Under +`CARGO_UNSTABLE_CHECKSUM_FRESHNESS` the file that gets corrupted is +`.fingerprint//dep-`, which holds the per-source checksums that +decide freshness, and the failure is silent stale-artifact reuse rather than a +slow build. `scripts/hardlink-clone-selftest.sh` reproduces it as an explicit +control and asserts the fix. The fix is to hardlink the artifacts (the GB) and +real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory, +against 100% for a full copy. + +--- + +## Quick start + +```yaml +name: CI +on: + push: + branches: [main, dev] + pull_request: + branches: [main, dev] + +jobs: + ci: + runs-on: ubuntu-latest + # REQUIRED, and it cannot come from the action: `container.volumes` is a + # job-level property, so the persistent cache volume must be declared + # here. Use a volume name unique to this repository. + container: + volumes: + - myrepo-ci-target:/cache + steps: + - uses: actions/checkout@v4 + with: + # REQUIRED. The mtime restore walks every commit that ever touched a + # tracked file; a depth-1 checkout makes every file resolve to the + # tip commit and the cache stops working. The action fails loudly + # rather than silently degrading if this is missing. + fetch-depth: 0 + + - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1 + with: + protected-branches: 'dev main' + + # ... toolchain, system deps, and the build itself. CARGO_TARGET_DIR is + # already exported to the job environment by the step above. + - run: cargo clippy --workspace --all-targets -- -D warnings + - run: cargo test --workspace + + # After the build succeeds: record the watermark, and publish a snapshot + # if this run is a push to a protected branch. + - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1 + + # Release this job's cache lock even when the build failed, so the + # eviction pass does not have to wait out the staleness grace period. + - if: always() + uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1 + with: + mode: release-lock +``` + +Recommended alongside it, in the workflow's `env:` block: + +```yaml +env: + CARGO_INCREMENTAL: 0 # per-run bloat on a persistent volume + CARGO_PROFILE_DEV_DEBUG: line-tables-only + CARGO_PROFILE_TEST_DEBUG: line-tables-only + # Nightly only. Content-addressed freshness instead of mtime-based — a + # strictly stronger guarantee, complementary to the mtime restore (which + # still covers directory-form `rerun-if-changed` build-script watches). + CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true" +``` + +--- + +## How it works + +``` +/cache/ + target- one per ref. Where a build actually runs. + snapshot- one per publisher ref. Immutable between publishes; + the only thing a consumer ever clones from. +``` + +`` is the ref sanitised to a safe path component, capped at 48 +characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not +decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise +share one directory. + +**A pull request run** resolves its own key from `github.head_ref` (not +`ref_name`, which on a `pull_request` event is a synthetic merge ref that +changes on every push) and its base key from `github.base_ref`. If it has no +directory yet, it hardlink-clones `snapshot-` into a staging path, strips +Cargo's lock files, real-copies everything Cargo writes in place, and renames +the staging path into `target-`. + +**A push to a protected branch** has no base to layer over. It builds in its +own directory and, if the build goes green, republishes it as +`snapshot-`: stage a clone, rename the old snapshot aside, rename the new +one in. Consumers only ever observe a complete snapshot or none at all. + +**Concurrency.** Two jobs sharing one cache key each stage under their own tag +and race on one atomic rename; the loser discards its staging copy. There is +no window in which a partially-populated directory is visible under the final +name — which means this does not depend on the runner having a single +execution slot. Two jobs then building in the same directory is Cargo's own +`.cargo-lock` territory, which is what that lock is for. + +**Eviction** runs three passes: caches for branches that no longer exist on +origin are removed unconditionally; then, only if free space is under the +threshold, live caches are evicted oldest-first; then, as a last resort, this +run's own cache. Protected refs and any cache held open by a running job are +never candidates. Within the pressure pass, `target-*` directories are evicted +before `snapshot-*` ones — the reverse of the obvious order, because a +snapshot is hardlinked to everything cloned from it, so removing one frees +almost no real bytes while costing every future PR its warm start. + +**File mtimes.** `actions/checkout` stamps every file with "now", which makes +every crate look changed to Cargo's mtime-based freshness check — a persistent +target directory buys nothing without fixing that. Each tracked file is +restored to the timestamp of the most recent commit that touched it, plus a +watermark override: for any file that changed since *this cache's own last +successful build*, "now" is stamped instead. That override is what makes a +merge safe, since a merge can introduce a commit authored before this cache's +last build, where the historically-correct mtime is exactly the wrong answer. + +--- + +## Inputs + +### `cargo-cache` + +| input | default | meaning | +|---|---|---| +| `cache-root` | `/cache` | mount point of the persistent volume inside the job container | +| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted | +| `min-free-percent` | `10` | prune when free space drops below this | +| `restore-mtimes` | `true` | restore tracked-file mtimes from git history | +| `prune` | `true` | run the eviction pass | +| `liveness-prune` | `true` | within eviction, remove caches for branches gone from origin | +| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` | +| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) | +| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache | +| `watermark-file` | `.ci-watermark--sha` | must differ per job when two jobs share one cache key | +| `lock-id` | `-` | identifies this job's cache lock | +| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned | + +Outputs: `target-dir`, `cache-key`, `seeded-from` (`own` \| `base-snapshot` \| +`own-snapshot` \| `fallback-dir` \| `concurrent-peer` \| `cold`). + +Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`, +`CARGO_CACHE_KEY`, `CARGO_CACHE_LOCK_ID`, `CARGO_CACHE_SCRIPTS`, +`CI_WATERMARK_FILE`. + +### `cargo-cache-publish` + +| input | default | meaning | +|---|---|---| +| `cache-root` | `/cache` | must match the consume action | +| `protected-branches` | `dev main` | refs that publish snapshots | +| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step | +| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` | +| `publish-on-events` | `push` | events on which a protected ref actually publishes | +| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) | + +`publish-on-events` defaults to `push` on purpose: a `pull_request` run from +`dev` into `main` has `own-ref` `dev` and would otherwise publish a snapshot +of a merge-preview build, which is not what `dev` is. + +--- + +## Multiple jobs in one workflow + +Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say) +each need their **own** watermark file. A shared one breaks the moment two +jobs run in sequence within one trigger: job A advances the watermark to HEAD, +and job B then reads that just-advanced value, computes an empty diff, and +loses the merge protection entirely. The default (`.ci-watermark--sha`) +already gives each job its own; only override `watermark-file` if you also +override `lock-id`, and then keep both distinct per job. + +--- + +## Constraints of this runner + +- **The repository must be public.** act_runner fetches actions by anonymous + git clone and has no credentialed-fetch option, so a private action + repository simply fails to resolve. Nothing secret goes in here. +- **`uses:` needs the absolute URL.** A bare `owner/repo` resolves against + github.com, because Gitea's `DEFAULT_ACTIONS_URL` is unset — and it has to + stay unset, or `actions/checkout`, `dtolnay/rust-toolchain` and + `taiki-e/install-action` stop resolving. +- **The runner pre-fetches every referenced action before running any step**, + so a bad action reference fails the job at step 0 rather than where it is + used. +- **`container.volumes` is job-level** and cannot be set from inside a + composite action. The consuming workflow declares it; see the quick start. +- **The cache volume is ext4** — no reflink support, which is precisely why + hardlinks are the mechanism that makes cloning cheap. + +--- + +## Versioning + +Pin `@v1`. It is a moving major tag: fixes and backward-compatible inputs move +it forward, and anything that would break an existing consumer gets `v2` +instead. Pin a commit SHA if you want a frozen version. + +--- + +## Development + +```bash +bash scripts/selftest.sh # everything (~1 min; needs cargo) +bash scripts/selftest.sh --fast # fixture-only suites, no compiler +``` + +| suite | covers | +|---|---| +| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. | +| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key | +| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish | +| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` | +| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. | + +Every suite runs the actual script, not a reimplementation of its logic, and +every fix scenario is paired with a control that reproduces the bug — a +scenario that passes either way proves nothing. + +The action YAML holds no logic beyond wiring; everything testable lives in +`scripts/`. A composite action needs `shell: bash` on every `run:` step, and +the actions reach their shared scripts through +`${{ github.action_path }}/../scripts`, which works because the runner clones +the whole repository when it fetches an action. diff --git a/cargo-cache-publish/action.yml b/cargo-cache-publish/action.yml new file mode 100644 index 0000000..ed42f2c --- /dev/null +++ b/cargo-cache-publish/action.yml @@ -0,0 +1,131 @@ +name: 'Cargo cache (publish)' +description: >- + Records this run''s build watermark and, on a publisher branch, atomically + republishes its target directory as the immutable snapshot that other + branches'' caches are hardlink-cloned from. +author: 'gitdan' + +inputs: + cache-root: + description: 'Mount point of the persistent cache volume. Must match the consume action.' + required: false + default: '/cache' + protected-branches: + description: >- + Space-separated refs that publish snapshots. A run whose own ref is not + in this list records its watermark and skips publishing. + required: false + default: 'dev main' + mode: + description: >- + publish — record the watermark, publish a snapshot if eligible, + release this job''s cache lock (the normal call, after a + green build). + release-lock — release this job''s cache lock and do nothing else. Use + in a final `if: always()` step so a failed run does not + leave a lock behind for the staleness grace period. + required: false + default: 'publish' + own-ref: + description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.' + required: false + default: '' + publish-on-events: + description: >- + Space-separated event names on which a publisher branch actually + publishes. Defaults to `push` — a pull_request run never publishes, + because its ref is not the reference branch even when it targets one. + required: false + default: 'push' + record-watermark: + description: >- + Record this run''s HEAD as the build watermark for this target dir. + True for PR runs too, not just publishers: a feature branch accumulates + its own build history across several pushes and needs its own watermark. + required: false + default: 'true' + +runs: + using: 'composite' + steps: + # Everything here reads the environment the consume action exported, so a + # workflow that forgets to run cargo-cache first fails loudly here rather + # than silently publishing a snapshot of the wrong directory. + - id: resolve + shell: bash + env: + PROTECTED_BRANCHES: ${{ inputs.protected-branches }} + PUBLISH_ON_EVENTS: ${{ inputs.publish-on-events }} + run: | + set -euo pipefail + # In release-lock mode this action is called from an `if: always()` + # step, which can run after a failure that happened before the + # cargo-cache action ever executed. Missing environment there means + # "there is no lock to release", not an error worth failing the job a + # second time over. + if [ -z "${CARGO_CACHE_SCRIPTS:-}" ] || [ -z "${CARGO_TARGET_DIR:-}" ]; then + if [ "${{ inputs.mode }}" = "release-lock" ]; then + echo "cargo-cache-publish: no cache environment in this job — nothing to release" + echo "publish=no" >> "$GITHUB_OUTPUT" + echo "active=no" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "::error::cargo-cache-publish: run the cargo-cache action earlier in this job" + exit 1 + fi + : "${CARGO_CACHE_KEY:?cargo-cache-publish: CARGO_CACHE_KEY not set by the cargo-cache action}" + echo "active=yes" >> "$GITHUB_OUTPUT" + + OWN_REF="${{ inputs.own-ref }}" + [ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}" + + # A publisher is a branch other branches layer over. Two conditions, + # both required: its ref is in protected-branches, AND the event is one + # where this ref really is the reference branch. A pull_request run + # from `dev` into `main` has own-ref `dev` and would otherwise publish + # a snapshot of a merge-preview build — which is not what `dev` is. + PUBLISH=no + for ref in $PROTECTED_BRANCHES; do + [ "$ref" = "$OWN_REF" ] || continue + for ev in $PUBLISH_ON_EVENTS; do + [ "$ev" = "${{ github.event_name }}" ] && PUBLISH=yes + done + done + echo "publish=${PUBLISH}" >> "$GITHUB_OUTPUT" + echo "own-ref=${OWN_REF}" >> "$GITHUB_OUTPUT" + echo "publisher check: ref '${OWN_REF}', event '${{ github.event_name }}' -> publish=${PUBLISH}" + + # Ordered before the snapshot publish so a snapshot always carries a + # watermark at least as new as the build it holds. Both steps sit after + # the consuming job's build steps, so a run that fails an earlier gate + # never reaches either: the watermark stays at the last GREEN build and a + # red build can never overwrite a known-good snapshot. + - if: ${{ steps.resolve.outputs.active == 'yes' && inputs.mode == 'publish' && inputs.record-watermark == 'true' }} + shell: bash + run: | + set -euo pipefail + bash "${CARGO_CACHE_SCRIPTS}/record-watermark.sh" \ + "$CARGO_TARGET_DIR" "${CI_WATERMARK_FILE:-.ci-watermark-sha}" + + - if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }} + shell: bash + run: | + set -euo pipefail + bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \ + "$CARGO_CACHE_KEY" "${{ inputs.cache-root }}" \ + "${{ github.job }}-${{ github.run_id }}-$$" + + # Released in both modes. In `publish` mode this is the normal end-of-job + # release; the separate `release-lock` call exists for `if: always()`, so a + # failed run does not leave its lock sitting until the staleness grace + # period expires. + - if: ${{ steps.resolve.outputs.active == 'yes' }} + shell: bash + run: | + set -euo pipefail + if [ -z "${CARGO_CACHE_LOCK_ID:-}" ]; then + echo "cargo-cache-publish: no lock id in the environment — nothing to release" + exit 0 + fi + bash "${CARGO_CACHE_SCRIPTS}/cache-lock.sh" release \ + "$CARGO_TARGET_DIR" "${CARGO_CACHE_LOCK_ID}" diff --git a/cargo-cache/action.yml b/cargo-cache/action.yml index 3b2e822..6066693 100644 --- a/cargo-cache/action.yml +++ b/cargo-cache/action.yml @@ -1,26 +1,197 @@ -name: 'Cargo CI cache' -description: 'Per-branch layered Cargo target-dir cache for Gitea Actions runners.' +name: 'Cargo cache (consume)' +description: >- + Per-ref Cargo target-dir cache for Gitea Actions runners: hardlink-clones + this ref's target directory from its base branch's published, immutable + snapshot, restores git-history file mtimes, and prunes the volume. +author: 'gitdan' + inputs: cache-root: - description: 'Mount point of the persistent cache volume.' + description: 'Mount point of the persistent cache volume inside the job container.' required: false default: '/cache' protected-branches: - description: 'Space-separated branches whose cache dirs are never evicted.' + description: >- + Space-separated refs that publish snapshots and are never evicted. + These are the branches PR caches layer over. required: false default: 'dev main' + min-free-percent: + description: 'Prune when free space on the cache volume drops below this percentage.' + required: false + default: '10' + restore-mtimes: + description: >- + Restore every tracked file''s mtime from git history. Requires a + full-history checkout (fetch-depth: 0). Set to false only if the build + does not use Cargo''s mtime-based freshness at all. + required: false + default: 'true' + prune: + description: 'Run the eviction pass (dead-branch liveness + disk pressure).' + required: false + default: 'true' + liveness-prune: + description: >- + Within the prune pass, remove caches for branches that no longer exist + on origin. Set to false on a runner that cannot reach origin. + required: false + default: 'true' + own-ref: + description: 'Override this run''s ref. Defaults to github.head_ref, else github.ref_name.' + required: false + default: '' + base-ref: + description: 'Override the ref to layer over. Defaults to github.base_ref (empty on push).' + required: false + default: '' + seed-fallback-dir: + description: >- + Absolute path to seed from when no snapshot exists yet — a pre-existing + flat cache directory during a migration. Optional. + required: false + default: '' + watermark-file: + description: >- + Name of this job''s build-watermark file inside the target dir. MUST be + distinct per job when two jobs share one cache key. Defaults to + .ci-watermark--sha. + required: false + default: '' + lock-id: + description: 'Identifier for this job''s cache lock. Defaults to -.' + required: false + default: '' + stale-lock-seconds: + description: 'Age past which another job''s cache lock is treated as abandoned.' + required: false + default: '7200' + outputs: target-dir: - description: 'Resolved CARGO_TARGET_DIR for this run.' - value: ${{ steps.probe.outputs.target-dir }} + description: 'Resolved CARGO_TARGET_DIR. Also exported to the job environment.' + value: ${{ steps.resolve.outputs.target-dir }} + cache-key: + description: 'Sanitized cache key for this run''s own ref.' + value: ${{ steps.resolve.outputs.cache-key }} + seeded-from: + description: 'Where the target dir came from: own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold.' + value: ${{ steps.seed.outputs.seeded-from }} + runs: using: 'composite' steps: - - id: probe + # Resolves both cache keys and exports the environment every later step + # (and the consuming job's own build steps) reads. Must run before + # anything that touches CARGO_TARGET_DIR, which is why it is first. + # + # `head_ref || ref_name` rather than `ref_name` alone: on a pull_request + # event `ref_name` is a synthetic merge-ref name that changes on every + # push to the PR, so keying on it would give the same PR a different cache + # directory every time — defeating the reuse this action exists to + # provide. On a push event `head_ref` is empty and `ref_name` is the real + # branch, which is what we want there. + # + # `base_ref` is populated only for pull_request events. A push run has + # nothing to layer over: its own ref IS the reference branch. It + # publishes, it does not consume. + - id: resolve shell: bash run: | - echo "PROBE-OK: composite action resolved and executed" - echo " cache-root = ${{ inputs.cache-root }}" - echo " protected-branches= ${{ inputs.protected-branches }}" - echo "target-dir=${{ inputs.cache-root }}/probe" >> "$GITHUB_OUTPUT" - echo "PROBE_ENV_WRITE=ok" >> "$GITHUB_ENV" + set -euo pipefail + SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts + [ -d "$SCRIPTS" ] || { echo "::error::cargo-cache: scripts/ not found at $SCRIPTS"; exit 1; } + echo "CARGO_CACHE_SCRIPTS=${SCRIPTS}" >> "$GITHUB_ENV" + OWN_REF="${{ inputs.own-ref }}" + [ -n "$OWN_REF" ] || OWN_REF="${{ github.head_ref || github.ref_name }}" + BASE_REF="${{ inputs.base-ref }}" + [ -n "$BASE_REF" ] || BASE_REF="${{ github.base_ref }}" + + OWN_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$OWN_REF") + BASE_KEY="" + if [ -n "$BASE_REF" ]; then + BASE_KEY=$(bash "${SCRIPTS}/branch-cache-key.sh" "$BASE_REF") + echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY}; layering over base ref '${BASE_REF}' -> ${BASE_KEY}" + else + echo "cache: own ref '${OWN_REF}' -> ${OWN_KEY} (no base ref — this ref publishes, it does not consume)" + fi + + TARGET_DIR="${{ inputs.cache-root }}/target-${OWN_KEY}" + WATERMARK="${{ inputs.watermark-file }}" + [ -n "$WATERMARK" ] || WATERMARK=".ci-watermark-${{ github.job }}-sha" + LOCK_ID="${{ inputs.lock-id }}" + [ -n "$LOCK_ID" ] || LOCK_ID="${{ github.job }}-${{ github.run_id }}" + + { + echo "target-dir=${TARGET_DIR}" + echo "cache-key=${OWN_KEY}" + echo "base-key=${BASE_KEY}" + echo "lock-id=${LOCK_ID}" + echo "watermark-file=${WATERMARK}" + } >> "$GITHUB_OUTPUT" + { + echo "CARGO_TARGET_DIR=${TARGET_DIR}" + echo "CARGO_CACHE_ROOT=${{ inputs.cache-root }}" + echo "CARGO_CACHE_KEY=${OWN_KEY}" + echo "CARGO_CACHE_LOCK_ID=${LOCK_ID}" + echo "CI_WATERMARK_FILE=${WATERMARK}" + } >> "$GITHUB_ENV" + + # Seeds this ref's target dir from the base's published snapshot. See + # scripts/seed-target-dir.sh — the staging-then-atomic-rename is what + # makes concurrent jobs sharing one cache key safe by construction rather + # than by the runner happening to have a single execution slot. + - id: seed + shell: bash + run: | + set -euo pipefail + SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts + bash "${SCRIPTS}/seed-target-dir.sh" \ + "${{ steps.resolve.outputs.cache-key }}" \ + "${{ steps.resolve.outputs.base-key }}" \ + "${{ inputs.cache-root }}" \ + "${{ github.job }}-${{ github.run_id }}-$$" \ + "${{ inputs.seed-fallback-dir }}" + + # Marks the directory as held open, so any job's prune pass (this one + # included) skips it, and stamps the LRU marker. The marker is touched + # unconditionally every run: a run that hits the cache for every crate may + # write nothing at all inside the tree, which would make a just-used + # directory look stale to the eviction pass. + - shell: bash + run: | + set -euo pipefail + SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts + bash "${SCRIPTS}/cache-lock.sh" acquire \ + "${{ steps.resolve.outputs.target-dir }}" "${{ steps.resolve.outputs.lock-id }}" + touch "${{ steps.resolve.outputs.target-dir }}/.cache-last-used" + + # Runs AFTER seeding, deliberately: restore-mtimes.sh reads the build + # watermark out of the target dir, so that directory has to be in its + # final form for this run (reused, seeded, or freshly created) before the + # watermark it may carry can be read. + # CARGO_TARGET_DIR and CI_WATERMARK_FILE are passed explicitly rather + # than read from the job environment the resolve step exported: the export + # is what the consuming workflow's own build steps rely on, but a step + # inside this action should not depend on cross-step propagation working + # when it can just be handed the value. + - if: ${{ inputs.restore-mtimes == 'true' }} + shell: bash + env: + CARGO_TARGET_DIR: ${{ steps.resolve.outputs.target-dir }} + CI_WATERMARK_FILE: ${{ steps.resolve.outputs.watermark-file }} + run: bash "$(cd "${{ github.action_path }}/.." && pwd)/scripts/restore-mtimes.sh" + + - if: ${{ inputs.prune == 'true' }} + shell: bash + env: + STALE_LOCK_SECONDS: ${{ inputs.stale-lock-seconds }} + CACHE_LIVENESS: ${{ inputs.liveness-prune }} + run: | + set -euo pipefail + SCRIPTS=$(cd "${{ github.action_path }}/.." && pwd)/scripts + bash "${SCRIPTS}/prune-cache.sh" \ + "${{ inputs.cache-root }}" \ + "${{ steps.resolve.outputs.target-dir }}" \ + "${{ inputs.protected-branches }}" \ + "${{ inputs.min-free-percent }}" diff --git a/scripts/branch-cache-key.sh b/scripts/branch-cache-key.sh new file mode 100755 index 0000000..181d734 --- /dev/null +++ b/scripts/branch-cache-key.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Prints the sanitized, collision-resistant cache key for a raw git ref. +# Thin CLI wrapper over cache_key() in cache-lib.sh, kept as its own entry +# point so a consuming workflow (or an out-of-band cleanup job, e.g. one +# reclaiming a deleted branch's directory) can resolve the same key this +# action uses without reimplementing the algorithm. Reimplementing it is the +# one thing that must never happen: two spellings of the key silently +# misclassify every directory on the volume. +# +# Usage: branch-cache-key.sh "" +set -euo pipefail +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" + +if [ $# -ne 1 ] || [ -z "$1" ]; then + echo "::error::branch-cache-key.sh: expected exactly one non-empty argument (the raw ref)" >&2 + exit 1 +fi +cache_key "$1" diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh new file mode 100755 index 0000000..569e099 --- /dev/null +++ b/scripts/cache-lib.sh @@ -0,0 +1,219 @@ +#!/usr/bin/env bash +# Shared helpers for the cargo-cache actions. Sourced, never executed +# directly — every caller does: +# +# . "$(dirname "${BASH_SOURCE[0]}")/cache-lib.sh" +# +# Nothing here reads the environment implicitly; every function takes what it +# needs as an argument, so the selftests can drive them against scratch +# directories without a CI context. + +# --------------------------------------------------------------------------- +# Cache keys +# --------------------------------------------------------------------------- + +# Maps a raw git ref to a filesystem-safe, collision-resistant directory +# component. +# +# Two properties matter and neither is free: +# +# Determinism — the same raw ref must always produce the same key, or a +# branch's second run lands in a different directory from its first and the +# whole cache is pointless. +# +# Collision resistance — `tr -c 'A-Za-z0-9._-' '-'` maps every disallowed +# byte to the same `-`, so `feat/foo` and `feat-foo` sanitise identically +# and would share one directory: two unrelated branches interleaving +# fingerprints in one tree, which is the exact cross-branch-contamination +# hazard this whole scheme exists to close, reopened through the sanitiser. +# An 8-hex-char prefix of the SHA-1 of the *raw* (pre-sanitisation) ref is +# appended so distinct refs always get distinct keys regardless of what +# sanitisation or truncation did to the readable part. +# +# The readable part is capped at 48 characters so a long branch name can't +# approach filesystem path-length limits; the hash suffix is what keeps two +# refs sharing a 48-char prefix apart. +cache_key() { + local raw="$1" slug hash + if [ -z "$raw" ]; then + echo "cache_key: refusing to key an empty ref" >&2 + return 1 + fi + slug=$(printf '%s' "$raw" | tr -c 'A-Za-z0-9._-' '-' | sed 's/-\{2,\}/-/g; s/^-//; s/-$//') + hash=$(printf '%s' "$raw" | sha1sum | cut -c1-8) + printf '%s-%s' "${slug:0:48}" "$hash" +} + +target_dir_for() { printf '%s/target-%s' "$1" "$2"; } +snapshot_dir_for() { printf '%s/snapshot-%s' "$1" "$2"; } + +# --------------------------------------------------------------------------- +# Disk accounting +# --------------------------------------------------------------------------- + +usage_kb() { + if [ -d "$1" ]; then du -sk "$1" 2>/dev/null | awk '{print $1}'; else echo 0; fi +} + +usage_gb() { + awk "BEGIN { printf \"%.1f\", $(usage_kb "$1") / 1024 / 1024 }" +} + +# df -kP: portable POSIX one-line-per-fs output; columns are 1k-blocks total, +# used, available, capacity%, mounted-on. Prints " ". +# +# CACHE_DF_OVERRIDE exists for the selftests: a scratch tmpdir on the test +# host's real filesystem won't sit below an arbitrary threshold on demand, and +# the eviction passes are precisely what needs testing under pressure. +read_df() { + if [ -n "${CACHE_DF_OVERRIDE:-}" ]; then printf '%s\n' "$CACHE_DF_OVERRIDE"; return; fi + df -kP "$1" | awk 'NR==2 {print $2, $4}' +} + +report_df() { + local label="$1" free_kb="$2" total_kb="$3" + awk -v l="$label" -v f="$free_kb" -v t="$total_kb" \ + 'BEGIN { printf "%s %.1f GB free / %.0f GB (%.1f%%)", l, f/1048576, t/1048576, (f*100)/t }' +} + +# Appends to the Actions job summary, which is read on green runs — unlike a +# ::warning:: buried in a log nobody opens. No-op outside Actions so the +# scripts still run standalone under the selftests. +summary_line() { + [ -n "${GITHUB_STEP_SUMMARY:-}" ] && printf '%s\n' "$1" >> "$GITHUB_STEP_SUMMARY" + return 0 +} + +# --------------------------------------------------------------------------- +# Hardlink cloning, and the part that makes it sound +# --------------------------------------------------------------------------- + +# Cargo's own target-dir lock files (.cargo-lock, .cargo-build-lock, +# .cargo-artifact-lock) are zero-byte files it opens and flock(2)s IN PLACE +# for the duration of a build — it never truncates-and-renames them the way it +# does real artifacts. `cp -al` leaves the clone's copy hardlinked to the same +# inode as the source's, and flock() contention is inode-based, not +# path-based, so a build in the clone and a build in the source would +# serialize on one mutex. +# +# The glob deliberately reaches past the three observed names so a lock file +# added by a future Cargo version is swept too; nothing else in a target dir +# is named `.cargo-*lock*`. Cargo recreates whichever it needs as a fresh, +# unshared inode the next time it opens the directory, at no cost. +strip_cargo_locks() { + find "$1" -type f -name '.cargo-*lock*' -delete 2>/dev/null || true + return 0 +} + +# Replaces a subtree with a real (non-hardlinked) copy of itself, in place. +# +# Staged through a sibling temp path and swapped with `mv -T` rather than +# copied over the original file-by-file: the copy is a fresh tree of fresh +# inodes, so nothing in it can alias the source it was cloned from. Callers +# only ever run this against a staging directory nothing else can see yet +# (see hardlink_clone_into's contract), so the brief window where the path is +# absent is not observable. +unshare_subtree() { + local d="$1" tmp + [ -d "$d" ] || return 0 + tmp="${d}.unshare.$$" + rm -rf "$tmp" + cp -a "$d" "$tmp" + rm -rf "$d" + mv -T "$tmp" "$d" +} + +_unshare_files() { + # `-links +1` restricts the work to files that are actually shared, which + # makes this idempotent and near-free on an already-unshared tree. + find "$@" -links +1 -print0 2>/dev/null | + xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _ + return 0 +} + +# THE load-bearing function of this whole design. +# +# A hardlink clone is only safe if every write the clone's build performs +# lands on a NEW inode, leaving the source's data untouched. That is true for +# compilation artifacts — rustc and the linker replace `deps/*.rlib`, +# `*.rmeta`, and binaries rather than truncating them in place — and it is +# NOT true for the metadata Cargo and build scripts write with a plain +# truncating write. Measured directly (Linux, ext4, cargo 1.9x nightly: +# `cp -al` a warm target dir, change a source file, build in the clone, diff +# the source) the following files in the SOURCE were mutated through the +# shared inode: +# +# /.fingerprint//dep- (only under +# CARGO_UNSTABLE_CHECKSUM_FRESHNESS, +# where this file carries the +# per-source blake3 checksums) +# /build//output, root-output (Cargo build-script metadata) +# /build//out/** (whatever the build script +# writes into OUT_DIR — build +# scripts overwhelmingly use a +# plain fs::write) +# /deps/*.d, /*.d (Cargo's post-processed +# dep-info) +# +# The checksum-freshness case is not a cosmetic one. Reproduced end to end: +# branch B clones base's cache, builds its own content, and thereby rewrites +# base's `dep-` to describe B's sources while base's cache still holds +# the artifact built from base's sources. When B merges and base next builds, +# Cargo reads that dep file, finds the checksums match the (now merged) +# sources, reports `Fresh`, and reuses a binary built from the PRE-merge code. +# That is silent stale-artifact reuse — a wrong answer, not a slow one. +# +# So: hardlink the artifacts (the GB), real-copy the metadata (the MB). +# Measured on a 6.9 GB Bevy workspace target dir, the unshared set is +# .fingerprint 22 MB + build/ 237 MB + a handful of dep-info files — about +# 3.7% of the tree, against 100% for a plain `cp -a`. +# +# `incremental/` is deliberately left shared: rustc writes each incremental +# session to a fresh `s-*-working` directory and finalises it with a rename, +# and garbage-collects old sessions by unlinking directory entries — neither +# of which mutates a shared inode. CI should still set CARGO_INCREMENTAL=0, +# for size rather than correctness. +unshare_mutable_paths() { + local root="$1" d + [ -d "$root" ] || return 0 + # The list is materialised in full before anything is replaced: each + # replacement deletes and recreates a directory, and a live `find` walk over + # a tree being mutated underneath it is a needless hazard. `-prune` keeps a + # match's own contents out of the list. + local -a dirs=() + mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null) + for d in "${dirs[@]}"; do + [ -n "$d" ] || continue + unshare_subtree "$d" + done + _unshare_files "$root" -type f -name '*.d' + _unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' + return 0 +} + +# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST +# with a single atomic rename. +# +# The staging + rename is what closes the concurrent-seed race structurally +# rather than by runner topology: a second job sharing this cache key either +# sees DST absent (and stages its own clone, losing the rename harmlessly) or +# sees it complete. There is no observable half-populated state, because a +# directory rename is atomic and DST is never written through. +# +# Returns 0 if this caller's clone won the rename, 1 if another caller got +# there first (the staging copy is discarded; DST is already valid). +hardlink_clone_into() { + local src="$1" dst="$2" tag="$3" tmp parent + parent=$(dirname "$dst") + tmp="${parent}/.stage-${tag}" + rm -rf "$tmp" + cp -al "$src" "$tmp" + strip_cargo_locks "$tmp" + rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true + unshare_mutable_paths "$tmp" + if mv -T "$tmp" "$dst" 2>/dev/null; then + return 0 + fi + rm -rf "$tmp" + return 1 +} diff --git a/scripts/cache-lock.sh b/scripts/cache-lock.sh new file mode 100755 index 0000000..8665c45 --- /dev/null +++ b/scripts/cache-lock.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Marks a cache directory as held open by a running job, so ANY job's prune +# pass (including this run's own) skips it. +# +# Usage: cache-lock.sh acquire|release +# +# Without this, a directory's only protection from a concurrently running +# job's eviction pass is "it happens to also be that job's own target dir", +# which is true for the job that owns it and false for everyone else. The +# marker is per-job (not just per-run) so two jobs sharing one cache key each +# hold an independent lock rather than one clobbering the other's. +# +# A lock is a timestamp file, not a real mutex: prune-cache.sh honours it only +# until STALE_LOCK_SECONDS, after which it is treated as abandoned by a job +# the runner killed before it reached its own release step. Honouring a lock +# forever would let one crashed job pin a directory permanently. +set -euo pipefail + +MODE="${1:?usage: cache-lock.sh acquire|release }" +DIR="${2:?}"; ID="${3:?}" + +case "$MODE" in + acquire) + mkdir -p "$DIR" + date +%s > "$DIR/.ci-lock-${ID}" + echo "lock: acquired .ci-lock-${ID} on $(basename "$DIR")" + ;; + release) + rm -f "$DIR/.ci-lock-${ID}" 2>/dev/null || true + echo "lock: released .ci-lock-${ID} on $(basename "$DIR")" + ;; + *) + echo "::error::cache-lock.sh: unknown mode '$MODE' (expected acquire or release)" >&2 + exit 1 + ;; +esac diff --git a/scripts/hardlink-clone-selftest.sh b/scripts/hardlink-clone-selftest.sh new file mode 100755 index 0000000..a066c2b --- /dev/null +++ b/scripts/hardlink-clone-selftest.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Regression test for the single assumption this whole caching scheme rests +# on: that a build running inside a hardlink clone cannot mutate the directory +# it was cloned from. +# +# That assumption is FALSE for a plain `cp -al`. Measured, and asserted below +# as an explicit control: build in a raw `cp -al` clone and the source's +# `.fingerprint//dep-*` (under CARGO_UNSTABLE_CHECKSUM_FRESHNESS), +# `build//output`, `build//out/**` and `deps/*.d` all change, +# because Cargo and build scripts write those with a plain truncating write +# rather than the write-then-rename Cargo uses for real artifacts. +# +# The consequence is not a slow build, it is a wrong one: a PR clone rewrites +# the base's dep-info to describe the PR's sources while the base's cache +# still holds the artifact built from the base's sources; once the PR merges, +# the base's next run finds the checksums match its (now merged) sources, +# reports `Fresh`, and links a binary built from the pre-merge code. +# +# cache-lib.sh's unshare_mutable_paths() is what closes that, and this test is +# what proves it stays closed. The control matters as much as the fix: a +# scenario that passes for both would prove nothing. +# +# Needs a working cargo on PATH. Everything happens under a mktemp -d scratch +# tree. Run by hand: bash scripts/hardlink-clone-selftest.sh +set -euo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$script_dir/cache-lib.sh" + +command -v cargo >/dev/null || { echo "SKIP: no cargo on PATH"; exit 0; } + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +pass_count=0 + +fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; } +ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; } + +# Content hash of every file in a tree, keyed by relative path. +snapshot_tree() { (cd "$1" && find . -type f -print0 | sort -z | xargs -0 -r sha1sum) 2>/dev/null; } + +# `diff` exits 1 when the trees differ, which is the expected case here and +# must not trip `pipefail` — the difference IS the result. +mutated_paths() { + { diff <(printf '%s' "$1") <(printf '%s' "$2") || true; } 2>/dev/null \ + | awk '/^[<>]/ { print $3 }' | sort -u +} + +# A crate with a build script, because build-script OUT_DIR writes are one of +# the two mutation families and are invisible without one. +mkcrate() { + local dir="$1" + mkdir -p "$dir/src" + cat > "$dir/Cargo.toml" <<'TOML' +[package] +name = "probe" +version = "0.1.0" +edition = "2021" +[workspace] +TOML + cat > "$dir/build.rs" <<'RS' +use std::{env, fs, path::PathBuf}; +fn main() { + println!("cargo::rerun-if-changed=src/lib.rs"); + let out = PathBuf::from(env::var("OUT_DIR").unwrap()); + let src = fs::read_to_string("src/lib.rs").unwrap(); + fs::write(out.join("gen.txt"), format!("generated from {} bytes", src.len())).unwrap(); +} +RS +} + +crate_dir="$scratch/probe" +mkcrate "$crate_dir" +cd "$crate_dir" + +export CARGO_INCREMENTAL=0 +# Checksum freshness is where the worst failure lives (the dep-* file carries +# per-source checksums and is rewritten in place). Only available on nightly; +# without it the test still covers the build/ and *.d families. +CHECKSUM_MODE="off" +if cargo +nightly -V >/dev/null 2>&1; then + export CARGO_UNSTABLE_CHECKSUM_FRESHNESS=true + CARGO_BIN=(cargo +nightly) + CHECKSUM_MODE="on" +else + CARGO_BIN=(cargo) +fi +echo "=== checksum-freshness mode: ${CHECKSUM_MODE} ===" + +CONTENT_A='pub fn f() -> u32 { 1 }' +CONTENT_B='pub fn f() -> u32 { 22222 } pub fn g() -> u32 { 7 }' + +build_base() { + local dir="$1" + printf '%s\n' "$CONTENT_A" > src/lib.rs + CARGO_TARGET_DIR="$dir" "${CARGO_BIN[@]}" build -q +} + +echo +echo "=== control: a raw \`cp -al\` clone DOES mutate its source ===" +base_ctl="$scratch/base-ctl"; clone_ctl="$scratch/clone-ctl" +build_base "$base_ctl" +before=$(snapshot_tree "$base_ctl") +cp -al "$base_ctl" "$clone_ctl" +strip_cargo_locks "$clone_ctl" # locks alone are not the hazard under test +printf '%s\n' "$CONTENT_B" > src/lib.rs +CARGO_TARGET_DIR="$clone_ctl" "${CARGO_BIN[@]}" build -q +after=$(snapshot_tree "$base_ctl") +ctl_mutated=$(mutated_paths "$before" "$after") +if [ -z "$ctl_mutated" ]; then + fail "control produced no mutation — the test can no longer distinguish fixed from broken" +fi +ok "raw cp -al clone mutates the source ($(printf '%s\n' "$ctl_mutated" | wc -l) paths)" +printf '%s\n' "$ctl_mutated" | sed 's/^/ /' + +if [ "$CHECKSUM_MODE" = "on" ]; then + if printf '%s' "$ctl_mutated" | grep -q '\.fingerprint/.*/dep-'; then + ok "control confirms the checksum-freshness dep-info file is among the mutated set" + else + fail "expected .fingerprint/*/dep-* in the control's mutated set under checksum freshness" + fi +fi + +echo +echo "=== fix: hardlink_clone_into() leaves the source byte-identical ===" +base_fix="$scratch/base-fix"; clone_fix="$scratch/clone-fix" +build_base "$base_fix" +before=$(snapshot_tree "$base_fix") +hardlink_clone_into "$base_fix" "$clone_fix" "selftest" || fail "hardlink_clone_into reported the destination already existed" + +# The clone's contract, asserted before anything builds in it: artifacts +# share inodes (that is what makes the clone near-free), and every file Cargo +# rewrites in place does not (that is what makes it sound). Checking after a +# rebuild would prove nothing — the rebuild replaces those files anyway. +shared=0; unshared=0 +while IFS= read -r f; do + rel="${f#$base_fix/}" + [ -e "$clone_fix/$rel" ] || continue + if [ "$(stat -c '%i' "$f")" = "$(stat -c '%i' "$clone_fix/$rel")" ]; then + case "$rel" in + */.fingerprint/*|*/build/*|*.d|.rustc_info.json) + fail "mutable path still shares an inode with the source: $rel" ;; + esac + shared=$((shared + 1)) + else + unshared=$((unshared + 1)) + fi +done < <(find "$base_fix" -type f) +[ "$shared" -gt 0 ] || fail "nothing is shared — the clone degenerated into a full copy" +[ "$unshared" -gt 0 ] || fail "nothing was unshared — unshare_mutable_paths did not run" +ok "fresh clone shares ${shared} artifact files and privately owns ${unshared} mutable ones" + +printf '%s\n' "$CONTENT_B" > src/lib.rs +CARGO_TARGET_DIR="$clone_fix" "${CARGO_BIN[@]}" build -q +after=$(snapshot_tree "$base_fix") +fix_mutated=$(mutated_paths "$before" "$after") +if [ -n "$fix_mutated" ]; then + echo " still mutated:" >&2 + printf '%s\n' "$fix_mutated" | sed 's/^/ /' >&2 + fail "a build in the clone mutated the source through a shared inode" +fi +ok "no file in the source changed after a full rebuild in the clone" + +echo +echo "=== the whole point: the source's next build is still correct ===" +# The source's cache holds artifacts built from CONTENT_A. Advance the source +# to CONTENT_B (as a merge would) and rebuild in it. If the clone had +# corrupted its dep-info, Cargo would report Fresh and keep the stale rlib. +printf '%s\n' "$CONTENT_B" > src/lib.rs +touch -d '@1000000000' src/lib.rs +log="$scratch/rebuild.log" +CARGO_TARGET_DIR="$base_fix" "${CARGO_BIN[@]}" build -v > "$log" 2>&1 || { cat "$log"; fail "rebuild in the source failed"; } +if grep -qE '^\s+Fresh probe' "$log"; then + fail "source declared its own crate Fresh against sources it has never built — stale-artifact reuse" +fi +ok "source correctly rebuilt its crate after advancing to the clone's content" + +echo +echo "hardlink-clone-selftest: ${pass_count} assertions passed" diff --git a/scripts/prune-cache-selftest.sh b/scripts/prune-cache-selftest.sh new file mode 100755 index 0000000..d95fd10 --- /dev/null +++ b/scripts/prune-cache-selftest.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# Regression test for prune-cache.sh. Builds a real scratch git repo standing +# in for `origin` and a real scratch directory standing in for the cache root, +# then runs the ACTUAL script against both — not a simulation of its logic. +# +# What each scenario demonstrates, and why the controls matter as much as the +# fixes (a scenario that always passes proves nothing): +# +# 1. DEAD BRANCH PRUNED, not gated on disk pressure — a cache whose branch +# no longer exists on origin is removed even with plenty of free space. +# Waiting for pressure to notice means paying for dead caches until then. +# 2. LIVE BRANCH SURVIVES despite being OLDER than the dead one — liveness, +# not age, is what decides pass 1. +# 3. PROTECTED REFS NEVER EVICTED under forced disk pressure, even when +# their caches are the oldest on disk and would rank first for LRU. +# 4. LOCKED CACHE PROTECTED even when dead, old, and under pressure. +# 5. STALE LOCK NOT HONOURED FOREVER — the same cache with a lock older than +# STALE_LOCK_SECONDS is evicted, so a crashed job cannot pin a directory +# permanently. +# 6. LIVENESS UNAVAILABLE FAILS SAFE — origin unreachable: a genuinely dead +# cache is NOT pruned, the log says so plainly, and the pressure fallback +# still works independently. "Unavailable" degrades to pressure-only, not +# to no eviction at all. +# 7. TARGET DIRS EVICTED BEFORE SNAPSHOTS — the ordering that differs from +# the obvious one. A snapshot is hardlinked to the caches cloned from it, +# so evicting it frees almost nothing while costing every future PR its +# warm start. +# 8. SELF-CLEAR REPORTS LOUDLY to the job summary, not just a log warning. +# 9. OWN CACHE NEVER EVICTED by a sibling pass. +# 10. SCOPED TO THE CACHE ROOT — a decoy outside it (standing in for another +# project's volume) is never touched. +set -euo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$script_dir/cache-lib.sh" +prune="$script_dir/prune-cache.sh" + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +pass_count=0 +fail() { echo "ASSERTION FAILED: $*" >&2; [ -n "${1:-}" ] && [ -f "$scratch/log" ] && { echo "--- log ---" >&2; cat "$scratch/log" >&2; }; exit 1; } +ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; } +assert_gone() { [ -e "$1" ] && fail "expected gone: $1 ($2)"; ok "$2"; } +assert_kept() { [ -e "$1" ] || fail "expected kept: $1 ($2)"; ok "$2"; } +assert_log() { grep -q -- "$1" "$scratch/log" || fail "expected in log: $1 ($2)"; ok "$2"; } + +echo "=== building a scratch origin with real branches ===" +origin="$scratch/origin.git"; git init -q --bare "$origin" +work="$scratch/work"; git init -q "$work" +( + cd "$work" + git -c user.email=t@t -c user.name=t commit -q --allow-empty -m init + git branch -M main + git checkout -q -b dev; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m dev + git checkout -q -b feat/live; git -c user.email=t@t -c user.name=t commit -q --allow-empty -m live + git remote add origin "$origin" + git push -q origin main dev feat/live +) +cd "$work" + +MAIN=$(cache_key main); DEV=$(cache_key dev); LIVE=$(cache_key feat/live) +DEAD=$(cache_key feat/dead); OWN=$(cache_key feat/own) + +root="$scratch/cache" +mk() { mkdir -p "$root/$1"; head -c 4096 /dev/zero > "$root/$1/blob"; touch -d "$2" "$root/$1/.cache-last-used"; } +reset_cache() { + rm -rf "$root"; mkdir -p "$root" + mk "target-$MAIN" '2020-01-01' + mk "snapshot-$MAIN" '2020-01-01' + mk "target-$DEV" '2020-01-01' + mk "snapshot-$DEV" '2020-01-01' + mk "target-$LIVE" '2020-01-02' # older than the dead one, deliberately + mk "target-$DEAD" '2030-01-01' # newest on disk, but its branch is gone + mk "snapshot-$DEAD" '2030-01-01' + mk "target-$OWN" '2025-01-01' +} +run_prune() { + local free="${1:-}" + CACHE_DF_OVERRIDE="$free" GITHUB_STEP_SUMMARY="$scratch/summary" \ + bash "$prune" "$root" "$root/target-$OWN" "dev main" 10 > "$scratch/log" 2>&1 \ + || { cat "$scratch/log"; fail "prune-cache.sh exited non-zero"; } +} + +echo +echo "=== 1/2: dead pruned unconditionally; older-but-live survives ===" +reset_cache +run_prune "1000000 900000" # 90% free: no pressure at all +assert_gone "$root/target-$DEAD" "dead branch's target dir pruned with no disk pressure" +assert_gone "$root/snapshot-$DEAD" "dead branch's snapshot pruned too" +assert_kept "$root/target-$LIVE" "live branch survives despite an older marker than the dead one" +assert_log "no matching branch on origin" "eviction reason reported" + +echo +echo "=== 3: protected refs never evicted under forced pressure ===" +reset_cache +run_prune "1000000 1000" # 0.1% free +assert_kept "$root/target-$DEV" "dev's target dir survives disk pressure" +assert_kept "$root/snapshot-$DEV" "dev's snapshot survives disk pressure" +assert_kept "$root/target-$MAIN" "main's target dir survives disk pressure" +assert_kept "$root/snapshot-$MAIN" "main's snapshot survives disk pressure" + +echo +echo "=== 9: own cache never evicted by a sibling pass ===" +assert_kept "$root/target-$OWN" "this run's own cache survives" + +echo +echo "=== 7: target dirs evicted before snapshots ===" +reset_cache +# Only the live branch is evictable; give it both a target dir and a snapshot +# with identical markers so ordering, not age, decides. +mk "snapshot-$LIVE" '2020-01-02' +# The df override is a fixed reading, so the pressure loop drains everything +# evictable — which is what makes the ORDER the observable property here, not +# what survives. Assert the eviction order directly from the log. +run_prune "1000000 1000" +order=$(grep -o "evicting \(target\|snapshot\)-$LIVE" "$scratch/log" | sed "s/evicting //") +[ "$(printf '%s\n' "$order" | head -1)" = "target-$LIVE" ] \ + || fail "expected target-$LIVE to be evicted before snapshot-$LIVE, got: $order" +ok "target dirs are evicted before snapshots" + +echo +echo "=== 4: a fresh lock protects a dead, old, under-pressure cache ===" +reset_cache +date +%s > "$root/target-$DEAD/.ci-lock-ci-1" +run_prune "1000000 1000" +assert_kept "$root/target-$DEAD" "locked cache survives both passes" +assert_log "held open by" "lock reported in the log" + +echo +echo "=== 5: a stale lock is not honoured forever ===" +reset_cache +echo 0 > "$root/target-$DEAD/.ci-lock-ci-1" +touch -d '2000-01-01' "$root/target-$DEAD/.ci-lock-ci-1" +run_prune "1000000 900000" +assert_gone "$root/target-$DEAD" "cache with an abandoned lock is evicted" +assert_log "treating as abandoned" "abandoned lock reported in the log" + +echo +echo "=== 6: liveness unavailable fails safe, pressure fallback still works ===" +reset_cache +( + cd "$work" && git remote set-url origin "$scratch/nonexistent.git" +) +run_prune "1000000 900000" # no pressure +assert_kept "$root/target-$DEAD" "dead cache NOT pruned when liveness is unavailable" +assert_log "treating as UNAVAILABLE" "unavailability reported plainly, not folded into 'no branches'" +run_prune "1000000 1000" # now with pressure +if [ -e "$root/target-$DEAD" ] && [ -e "$root/target-$LIVE" ]; then + fail "pressure fallback did nothing when liveness was unavailable" +fi +ok "pressure fallback still evicts when liveness is unavailable" +(cd "$work" && git remote set-url origin "$origin") + +echo +echo "=== 8: self-clear reports to the job summary ===" +reset_cache +rm -rf "$root/target-$DEAD" "$root/snapshot-$DEAD" "$root/target-$LIVE" +: > "$scratch/summary" +run_prune "1000000 1000" # nothing evictable left but the run's own cache +assert_log "clearing own" "self-clear reported in the log" +grep -q 'self-clear' "$scratch/summary" || fail "self-clear missing from the job summary" +ok "self-clear reported to the job summary, not only the log" +[ -d "$root/target-$OWN" ] || fail "self-clear left the own directory missing" +[ -z "$(ls -A "$root/target-$OWN")" ] || fail "self-clear did not actually empty the directory" +ok "own cache wiped and recreated empty" + +echo +echo "=== 10: scoped to the cache root ===" +reset_cache +decoy="$scratch/other-project"; mkdir -p "$decoy/target-$DEAD"; touch "$decoy/target-$DEAD/blob" +run_prune "1000000 1000" +assert_kept "$decoy/target-$DEAD" "a cache outside the cache root is never touched" + +echo +echo "prune-cache-selftest: ${pass_count} assertions passed" diff --git a/scripts/prune-cache.sh b/scripts/prune-cache.sh new file mode 100755 index 0000000..5725448 --- /dev/null +++ b/scripts/prune-cache.sh @@ -0,0 +1,211 @@ +#!/usr/bin/env bash +# Eviction for the per-ref cache directories on the persistent volume. +# +# Usage: prune-cache.sh +# protected-branches space-separated raw refs (e.g. "dev main") +# +# Optional environment: +# STALE_LOCK_SECONDS age past which a .ci-lock-* marker is treated as +# abandoned (default 7200) +# CACHE_LIVENESS "false"/"0" to skip the liveness pass entirely +# CACHE_DF_OVERRIDE " ", for the selftest +# +# Three passes, in order: +# +# 1. LIVENESS — every target-*/snapshot-* directory whose branch no longer +# exists on origin is removed UNCONDITIONALLY, not gated on free space. +# A directory for a branch deleted days ago is pure loss: nothing will +# ever read it again, since a merged PR's branch cannot be reopened. +# Waiting for disk pressure to notice means paying for it until then. +# Skipped entirely, loudly, if the liveness signal itself is +# unavailable — "couldn't determine" is never folded into "dead". +# 2. PRESSURE — if free space is still under the threshold, evict remaining +# (now necessarily live) directories oldest-first until it clears. +# 3. SELF-CLEAR — if pass 2 still isn't enough, wipe this run's own target +# dir and pay a cold rebuild, reported to the job summary as well as the +# log, because a warning on a green run is what lets a silently 4x-slower +# job go unnoticed. +# +# Reactive-only, with no hard cap on cache size: a workspace's natural working +# set is what it is, and bounding the footprint preemptively means wiping +# useful content before it is actually causing host pressure. The bound is +# physics, not an arbitrary GB number. +# +# EVICTION ORDER, and why it is the reverse of the obvious one: within the +# pressure pass, `target-*` directories are evicted BEFORE `snapshot-*` ones. +# A snapshot is a hardlink clone of a live target dir and of every consumer +# cloned from it, so removing it frees almost no real bytes — its inodes stay +# alive through those other links — while costing every future PR its warm +# start. Evicting snapshots first would be nearly pure loss. Target +# directories are where a branch's own divergent artifacts actually live, so +# they are what freeing space means. +# +# Two exclusions every pass respects: +# +# PROTECTED — the publisher branches' target and snapshot directories, and +# this run's own target dir, are never candidates in any pass. Evicting a +# publisher's snapshot doesn't free real disk (every open PR's clone keeps +# the data alive) but does force every subsequent PR to start cold, which is +# the entire benefit this scheme exists to deliver. +# +# LOCKED — a directory carrying a .ci-lock-* marker younger than +# STALE_LOCK_SECONDS is held open by a running job and is skipped by every +# pass, however dead and however tight the disk. This is what makes eviction +# safe on a runner with more than one execution slot. An older marker is +# treated as abandoned and logged as such, so an actually-still-running job +# that somehow exceeds the threshold is visible in the log rather than +# silently losing its cache mid-build. +# +# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a +# timeout. A directory name cannot be inverted back to a branch name (the +# sanitiser is lossy and the disambiguating suffix is a one-way hash), so this +# goes the other direction: it recomputes the expected directory names for +# every branch origin reports, using cache-lib.sh's OWN cache_key function — +# the same one the seed step used to create them. Reusing that function rather +# than reimplementing a lookalike is what makes the classification sound; any +# drift between two spellings would silently misclassify every directory. +# +# Only ever globs inside . Another project's volume is a different +# Docker named volume and is not mounted in this container at all, so "stays +# scoped to this repo's cache" holds structurally, not by convention. +set -euo pipefail +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" + +ROOT="${1:?usage: prune-cache.sh }" +OWN_DIR="${2:?}" +PROTECTED_REFS="${3:-}" +MIN_FREE_PCT="${4:-10}" +STALE_LOCK_SECONDS="${STALE_LOCK_SECONDS:-7200}" + +declare -A protected_ns=() +for ref in $PROTECTED_REFS; do + suffix=$(cache_key "$ref") + protected_ns["target-${suffix}"]=1 + protected_ns["snapshot-${suffix}"]=1 +done + +is_protected() { + local dir="$1" name + name=$(basename "$dir") + [ "$dir" = "$OWN_DIR" ] && return 0 + [ -n "${protected_ns[$name]:-}" ] && return 0 + return 1 +} + +is_locked() { + local dir="$1" now lock_file lock_age locked=1 + now=$(date +%s) + for lock_file in "$dir"/.ci-lock-*; do + [ -e "$lock_file" ] || continue + lock_age=$(( now - $(stat -c '%Y' "$lock_file") )) + if [ "$lock_age" -lt "$STALE_LOCK_SECONDS" ]; then + echo " $(basename "$dir"): held open by $(basename "$lock_file") (${lock_age}s old)" + locked=0 + else + echo " $(basename "$dir"): ignoring stale lock $(basename "$lock_file") (${lock_age}s old > ${STALE_LOCK_SECONDS}s) — treating as abandoned" + fi + done + return "$locked" +} + +# Directories oldest-first, target-* before snapshot-* (see the header). +# The sort key is a rank digit followed by a zero-padded mtime, so the two +# groups sort as blocks rather than interleaving by age. `.cache-last-used` +# is the marker every run touches; a run that hits the cache for every crate +# may write nothing at all inside the tree, which would make a +# just-used directory look stale without it. Fall back to the directory's +# own mtime when the marker is missing (a partially-written directory from an +# interrupted run is still a valid, if less precise, "last touched" signal). +list_by_lru() { + local rank d ts + for rank in 0:target 1:snapshot; do + for d in "$ROOT"/"${rank#*:}"-*; do + [ -d "$d" ] || continue + if [ -e "$d/.cache-last-used" ]; then ts=$(stat -c '%Y' "$d/.cache-last-used" 2>/dev/null || echo 0) + else ts=$(stat -c '%Y' "$d" 2>/dev/null || echo 0); fi + printf '%s%012d\t%s\n' "${rank%%:*}" "$ts" "$d" + done + done | sort | cut -f2- +} + +echo "=== pass 1: liveness (unconditional, not gated on free space) ===" +declare -A live_ns=() +LIVENESS_AVAILABLE=0 +LIVENESS_REASON="" +if [ "${CACHE_LIVENESS:-true}" = "0" ] || [ "${CACHE_LIVENESS:-true}" = "false" ]; then + LIVENESS_REASON="disabled via CACHE_LIVENESS=${CACHE_LIVENESS}" +elif remote_heads=$(timeout 20 git ls-remote --heads origin 2>&1); then + LIVENESS_AVAILABLE=1 + branch_count=0 + while IFS= read -r line; do + [ -n "$line" ] || continue + case "$line" in *refs/heads/*) ;; *) continue ;; esac + branch="${line#*refs/heads/}" + [ -n "$branch" ] || continue + suffix=$(cache_key "$branch") + live_ns["target-${suffix}"]=1 + live_ns["snapshot-${suffix}"]=1 + branch_count=$((branch_count + 1)) + done <<< "$remote_heads" + echo "liveness: ${branch_count} live branches on origin" +else + LIVENESS_REASON="git ls-remote --heads origin failed or timed out" +fi + +if [ "$LIVENESS_AVAILABLE" = "1" ]; then + pruned_any=0 + for dir in "$ROOT"/target-* "$ROOT"/snapshot-*; do + [ -d "$dir" ] || continue + name=$(basename "$dir") + is_protected "$dir" && continue + [ -n "${live_ns[$name]:-}" ] && continue + is_locked "$dir" && continue + dir_gb=$(usage_gb "$dir") + echo "::warning::pruning dead-branch cache ${name} (${dir_gb} GB) — no matching branch on origin" + summary_line "- pruned dead-branch cache \`${name}\` (${dir_gb} GB) — branch no longer exists on origin" + rm -rf "$dir" + pruned_any=1 + done + [ "$pruned_any" = "1" ] || echo "no dead-branch caches found" +else + echo "liveness: ${LIVENESS_REASON} — treating as UNAVAILABLE (not as \"no branches\"); pass 1 skipped" +fi + +echo +echo "=== pass 2/3: disk pressure (threshold: free < ${MIN_FREE_PCT}%) ===" +read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" +THRESHOLD_KB=$(( TOTAL_KB * MIN_FREE_PCT / 100 )) + +if [ "$FREE_KB" -ge "$THRESHOLD_KB" ]; then + echo "cache: $(basename "$OWN_DIR") $(usage_gb "$OWN_DIR") GB | $(report_df host "$FREE_KB" "$TOTAL_KB")" + exit 0 +fi + +echo "::warning::$(report_df disk "$FREE_KB" "$TOTAL_KB") < ${MIN_FREE_PCT}% threshold" + +# A plain loop over a pre-materialised, pre-sorted list rather than a live +# `find | while` pipeline, so `rm -rf` inside the loop cannot make a running +# directory walk re-observe its own deletions. +mapfile -t LRU < <(list_by_lru) +for dir in "${LRU[@]}"; do + [ -d "$dir" ] || continue + is_protected "$dir" && continue + read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" + [ "$FREE_KB" -ge "$THRESHOLD_KB" ] && break + is_locked "$dir" && continue + dir_gb=$(usage_gb "$dir") + echo "::warning::evicting $(basename "$dir") (${dir_gb} GB, LRU under disk pressure)" + summary_line "- evicted \`$(basename "$dir")\` (${dir_gb} GB, LRU under disk pressure)" + rm -rf "$dir" +done + +read -r TOTAL_KB FREE_KB <<< "$(read_df "$ROOT")" +if [ "$FREE_KB" -lt "$THRESHOLD_KB" ]; then + OWN_GB=$(usage_gb "$OWN_DIR") + echo "::warning::still under threshold after evicting every eligible sibling; clearing own $(basename "$OWN_DIR") (was ${OWN_GB} GB) — this run pays a cold rebuild" + summary_line "- **self-clear**: \`$(basename "$OWN_DIR")\` (was ${OWN_GB} GB) wiped — this run pays a cold rebuild" + rm -rf "$OWN_DIR" + mkdir -p "$OWN_DIR" +else + echo "$(report_df post-eviction "$FREE_KB" "$TOTAL_KB") — sibling eviction recovered enough space" +fi diff --git a/scripts/publish-snapshot-selftest.sh b/scripts/publish-snapshot-selftest.sh new file mode 100755 index 0000000..4b70fc2 --- /dev/null +++ b/scripts/publish-snapshot-selftest.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Regression test for publish-snapshot.sh: the atomic swap, and the two +# properties the swap exists to guarantee. +# +# 1. FIRST PUBLISH — with no prior snapshot, the target dir is published and +# is a hardlink clone of it (cheap), with the mutable metadata privately +# owned (sound: the publisher's NEXT build must not be able to mutate the +# snapshot it just published). +# 2. REPUBLISH REPLACES — a second publish replaces the snapshot's content +# rather than merging into it, and leaves no scratch directories behind. +# 3. A LIVE CONSUMER SURVIVES A REPUBLISH — a branch that already cloned the +# previous generation keeps reading its own consistent copy. Removing the +# old snapshot unlinks directory entries; the inodes stay alive through +# the consumer's own links. This is why a republish can never pull data +# out from under a running job. +# 4. NO TARGET DIR — publishing when there is nothing to publish is a no-op, +# not a failure. +# 5. LOCKS AND MARKERS DO NOT RIDE ALONG — the publishing job's own cache +# lock is still held while this runs, and must not be baked into the +# snapshot: a lock timestamped at this run's start would look fresh to +# the prune pass on every branch later seeded from it. +set -euo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$script_dir/cache-lib.sh" + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +root="$scratch/cache"; mkdir -p "$root" +pass_count=0 +fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; } +ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; } + +# Cargo and rustc REPLACE an artifact (write elsewhere, rename over the path) +# rather than truncating it in place, which is exactly why a snapshot may +# share artifact inodes with the live target dir it was cloned from. The +# fixtures here have to model that faithfully — a plain `>` redirect truncates +# in place and would write straight through the shared inode into the +# snapshot and every consumer, which is a property of the test fixture, not of +# a real build. hardlink-clone-selftest.sh is what verifies the real thing +# against a real compiler. +replace_file() { + printf '%s\n' "$2" > "$1.new" + mv -f "$1.new" "$1" +} + +make_tree() { + local d="$1" marker="$2" + mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x" + echo "$marker" > "$d/debug/deps/libx.rlib" + echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x" + : > "$d/debug/.cargo-lock" +} + +KEY=$(cache_key dev) +TGT="$root/target-$KEY" +SNAP="$root/snapshot-$KEY" +publish() { bash "$script_dir/publish-snapshot.sh" "$KEY" "$root" "$1" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "publish-snapshot.sh exited non-zero"; }; } + +echo "=== 4: nothing to publish is a no-op ===" +publish job1 +[ -d "$SNAP" ] && fail "published a snapshot with no target dir present" +grep -q 'nothing to snapshot' "$scratch/log" || fail "expected a 'nothing to snapshot' line" +ok "no target dir: no-op, reported plainly" + +echo +echo "=== 1: first publish ===" +make_tree "$TGT" gen1 +date +%s > "$TGT/.ci-lock-ci-42" +touch "$TGT/.cache-last-used" +publish job1 +[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen1" ] || fail "snapshot content wrong" +ok "snapshot published" +[ "$(stat -c '%i' "$SNAP/debug/deps/libx.rlib")" = "$(stat -c '%i' "$TGT/debug/deps/libx.rlib")" ] \ + || fail "snapshot artifact was copied, not hardlinked" +ok "snapshot shares artifact inodes with the target dir (cheap)" +[ "$(stat -c '%i' "$SNAP/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$TGT/debug/.fingerprint/x/dep-lib-x")" ] \ + || fail "snapshot fingerprint still aliases the live target dir" +ok "snapshot owns its mutable metadata (publisher's next build cannot corrupt it)" + +echo +echo "=== 5: locks and LRU markers do not ride along ===" +[ -e "$SNAP/.ci-lock-ci-42" ] && fail "the publishing job's lock was baked into the snapshot" +ok "cache lock not published" +[ -e "$SNAP/.cache-last-used" ] && fail "the LRU marker was baked into the snapshot" +ok "LRU marker not published" +[ -e "$SNAP/debug/.cargo-lock" ] && fail "a Cargo lock file was published" +ok "Cargo lock file not published" + +echo +echo "=== 3: a consumer that cloned generation 1 ===" +CONSUMER="$root/target-$(cache_key feat/consumer)" +hardlink_clone_into "$SNAP" "$CONSUMER" consumer-tag || fail "consumer clone failed" +ok "consumer cloned generation 1" + +echo +echo "=== 2: republish replaces, leaves no scratch behind ===" +replace_file "$TGT/debug/deps/libx.rlib" gen2 +# The fingerprint IS written in place by Cargo — and the snapshot owns its own +# copy precisely so that write cannot reach it. Truncating in place here is +# the faithful model. +echo gen2 > "$TGT/debug/.fingerprint/x/dep-lib-x" +publish job1 +[ "$(cat "$SNAP/debug/deps/libx.rlib")" = "gen2" ] || fail "republish did not replace the snapshot" +ok "republished snapshot carries generation 2" +leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.publish-*' \) -print) +[ -z "$leftovers" ] || fail "scratch directories left behind: $leftovers" +ok "no scratch directories left behind" +[ "$(cat "$CONSUMER/debug/deps/libx.rlib")" = "gen1" ] \ + || fail "the consumer's clone changed under it when the snapshot was replaced" +ok "the live consumer still reads its own consistent generation-1 copy" + +echo +echo "publish-snapshot-selftest: ${pass_count} assertions passed" diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh new file mode 100755 index 0000000..ce7a025 --- /dev/null +++ b/scripts/publish-snapshot.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Publish side: atomically republish this reference branch's live target dir +# as the immutable snapshot other branches seed from. +# +# Usage: publish-snapshot.sh +# +# Run only after the build has already succeeded, and only on a branch that is +# a publisher (see the cargo-cache-publish action, which makes that decision). +# A red build must never overwrite a known-good snapshot: the caller's step +# ordering is the whole gate here, so keep this step last. +# +# The swap is two renames, not one, because POSIX rename() can only replace an +# EMPTY directory and a snapshot from a prior publish is not one: +# +# 1. stage the new snapshot at .stage- (copy time is off every +# consumer's hot path — nothing reads a staging path); +# 2. rename the current snapshot aside to .publish-old-, if present; +# 3. rename the staged snapshot into place. +# +# Step 3 is a single atomic rename onto a path now guaranteed absent, so it +# can never partially overwrite a live snapshot. A consumer landing in the +# window between 2 and 3 sees the snapshot as absent and falls through to its +# own cold-start path — a safe degrade that self-heals on its next run, not +# corruption. +# +# `rm -rf` on the old snapshot removes directory entries only. Any consumer +# that already hardlink-cloned from it keeps every inode alive through its own +# links, so a republish never pulls data out from under a running job — it +# just stops new consumers from seeing the old generation. Disk is reclaimed +# when the last clone referencing those inodes is itself evicted. +set -euo pipefail +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" + +OWN_KEY="${1:?usage: publish-snapshot.sh }" +ROOT="${2:?}"; TAG="${3:?}" + +SRC=$(target_dir_for "$ROOT" "$OWN_KEY") +DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") +OLD="${ROOT}/.publish-old-${TAG}" + +if [ ! -d "$SRC" ]; then + echo "publish: no target dir at ${SRC} — nothing to snapshot" + exit 0 +fi + +# Leftovers from a run cancelled mid-swap. Scoped to this job's own tag so a +# concurrently running job's staging directory is never touched. +rm -rf "${ROOT}/.stage-${TAG}" "$OLD" + +start=$(date +%s) +# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy +# of every file Cargo rewrites in place (unshare_mutable_paths, called inside +# hardlink_clone_into). Without that, this branch's NEXT build would mutate +# the snapshot it just published — the same aliasing hazard the consume side +# closes, pointing the other way. The staging path is not the final name, so +# `hardlink_clone_into`'s rename lands on DST only after OLD is out of the way. +TMP_DST="${ROOT}/.publish-new-${TAG}" +rm -rf "$TMP_DST" +hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || { + echo "::error::publish: failed to stage a snapshot of ${SRC}" + exit 1 +} + +if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi +mv -T "$TMP_DST" "$DST" +rm -rf "$OLD" + +echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s" +summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)" diff --git a/scripts/record-watermark.sh b/scripts/record-watermark.sh new file mode 100755 index 0000000..b87511b --- /dev/null +++ b/scripts/record-watermark.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Records this run's HEAD as the watermark restore-mtimes.sh reads back on +# this cache directory's NEXT run. +# +# Usage: record-watermark.sh +# +# Run only after the build has succeeded: a failed run leaves the watermark at +# the last GREEN build, which is the conservative direction (over-invalidates, +# never under-invalidates). See restore-mtimes.sh's header for what the +# watermark is for. +# +# Written with write-then-rename so this directory's next run can never +# observe a truncated, half-written watermark. +set -euo pipefail +DIR="${1:?usage: record-watermark.sh }" +NAME="${2:?}" +mkdir -p "$DIR" +git rev-parse HEAD > "$DIR/${NAME}.tmp" +mv -f "$DIR/${NAME}.tmp" "$DIR/${NAME}" +echo "watermark: $(cat "$DIR/${NAME}") recorded in ${DIR}/${NAME}" diff --git a/scripts/restore-mtimes-selftest.sh b/scripts/restore-mtimes-selftest.sh new file mode 100755 index 0000000..cc57414 --- /dev/null +++ b/scripts/restore-mtimes-selftest.sh @@ -0,0 +1,475 @@ +#!/usr/bin/env bash +# Regression test for restore-mtimes.sh's per-target-dir watermark (issue +# The merge hazard: a branch that merges an older commit could silently reuse a stale +# build artifact and fail with an "impossible" compile error). Builds a +# minimal, throwaway two-crate cargo workspace in a scratch git repo whose +# history is shaped exactly like the real incident, then runs the ACTUAL +# `restore-mtimes.sh` sitting next to this file against it under real +# `cargo build` — not a simulation of the logic, the real script and a real +# compiler. Every scenario ends in a hard assertion; the first failure dumps +# the relevant log and exits non-zero, so a future change to restore-mtimes.sh +# or the actions' step order that reopens this hazard fails loudly here +# instead of surfacing weeks later as a confusing CI compile error. +# +# Run by hand: +# bash .gitea/scripts/restore-mtimes-selftest.sh +# +# Slow by design — this builds a throwaway cargo workspace +# and runs several real compiler invocations (~1-2 minutes), which is too +# slow to pay on every ordinary push. Re-run it by hand whenever +# restore-mtimes.sh's timestamp logic or the actions' cache/seed/watermark step +# order changes. +# +# Needs a working `cargo`/`rustc` on PATH. Does not touch this repo or its +# own target dir — everything happens under a `mktemp -d` scratch tree, +# removed on exit via the trap below regardless of outcome. +# +# What each scenario demonstrates, and why the "control" runs matter as much +# as the "fixed" ones (a scenario that always passes proves nothing): +# +# 1. NO WATERMARK, same-branch merge (control) — a feature branch builds +# once, then merges a commit that changed a dependency crate but was +# authored earlier in real time than that build. With no watermark to +# consult, restore-mtimes.sh falls back to its pre-watermark behaviour +# (git-log-only timestamps): asserts this reproduces the exact bug — +# the dependency crate is judged Fresh and reused stale, and the +# dependent crate fails to compile against it. +# 2. WATERMARK PRESENT, same state (fix) — identical setup, but this +# target dir's watermark names the pre-merge build. Asserts the +# dependency crate is correctly identified as changed and recompiles, +# and the build succeeds. +# 3. WATERMARK PRESENT, no further change (warm-path) — re-runs scenario +# 2's state with an up-to-date watermark and no new commits. Asserts +# NOTHING recompiles — the ordinary no-merge case restore-mtimes.sh +# exists for is unaffected by the watermark machinery. +# 4. SEEDED NAMESPACE, watermark stripped (control) — a brand-new branch +# namespace, seeded from scenario +# 2/3's now-warm target dir, then merges a THIRD lineage's older commit. +# With the inherited watermark file deliberately removed, asserts the +# same staleness bug reproduces on a namespace that has never itself +# run a build — proving the seed-snapshot path shares the hazard, not +# just long-lived branches. +# 5. SEEDED NAMESPACE, watermark inherited (fix) — identical to 4, but the +# watermark file rides along with the `cp -a` seed copy exactly as +# the cargo-cache action's seed step really does it. +# Asserts the dependency crate correctly recompiles on this namespace's +# very first run. +# 6. TWO CONSUMERS, ONE SHARED WATERMARK FILE, sequential jobs (control) — +# two jobs in one workflow can share a single per-ref +# $CARGO_TARGET_DIR, so one cache holds two independent cargo profile +# directories (say `debug/` built by a test job and `release/` built by a +# wasm build job). Modelling this as "both jobs read before either +# writes" is the wrong ordering: on a serial runner with no `needs:` +# between the two jobs, one job's watermark WRITE lands before the other +# job's own READ, within the same trigger. This scenario now runs restore-mtimes.sh TWICE IN +# SEQUENCE against ONE shared watermark file, with a real watermark +# write in between (job A builds debug/, succeeds, advances the shared +# watermark to HEAD; job B then reads that JUST-ADVANCED watermark). +# Asserts job A succeeds and job B reproduces the original bug — +# an empty HEAD..HEAD diff, full fallback to git-log timestamps, stale +# reuse in release/. +# 7. TWO CONSUMERS, TWO WATERMARK FILES, sequential jobs (fix) — identical +# ordering, but each job reads and writes its OWN watermark file +# (`CI_WATERMARK_FILE`), exactly as the cargo-cache action does. Job B's read is +# unaffected by job A's write because they touch different files. +# Asserts BOTH jobs correctly recompile the dependency and succeed. +set -euo pipefail + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +restore_mtimes="$script_dir/restore-mtimes.sh" + +if [ ! -x "$restore_mtimes" ] && [ ! -f "$restore_mtimes" ]; then + echo "restore-mtimes-selftest: expected to find restore-mtimes.sh at $restore_mtimes" >&2 + exit 1 +fi + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +repo="$scratch/wk" +target_feature="$scratch/target-feature" +target_newbranch="$scratch/target-newbranch" +target_multiprofile="$scratch/target-multiprofile" + +pass_count=0 + +assert_log_has() { + local file="$1" pattern="$2" desc="$3" + if ! grep -q -- "$pattern" "$file"; then + echo "ASSERTION FAILED: $desc" >&2 + echo " expected to find: $pattern" >&2 + echo " --- full log ($file) ---" >&2 + cat "$file" >&2 + exit 1 + fi + pass_count=$((pass_count + 1)) + echo "PASS: $desc" +} + +assert_log_lacks() { + local file="$1" pattern="$2" desc="$3" + if grep -q -- "$pattern" "$file"; then + echo "ASSERTION FAILED: $desc" >&2 + echo " expected NOT to find: $pattern" >&2 + echo " --- full log ($file) ---" >&2 + cat "$file" >&2 + exit 1 + fi + pass_count=$((pass_count + 1)) + echo "PASS: $desc" +} + +# `actions/checkout` stamps every tracked file with wall-clock "now" on +# every run — simulate that before each restore-mtimes.sh pass, exactly as +# CI would see it, so this test exercises the script the same way CI does. +stamp_checkout_now() { + find . -path ./.git -prune -o -type f -print | xargs touch +} + +echo "=== building scratch workspace ===" +mkdir -p "$repo/crates/libdep/src" "$repo/crates/libuser/src" +cd "$repo" +git init -q +git config user.email test@example.com +git config user.name "restore-mtimes-selftest" + +cat > Cargo.toml <<'EOF' +[workspace] +resolver = "2" +members = ["crates/libdep", "crates/libuser"] +EOF +cat > crates/libdep/Cargo.toml <<'EOF' +[package] +name = "libdep" +version = "0.1.0" +edition = "2021" +EOF +cat > crates/libdep/src/lib.rs <<'EOF' +pub struct CheckIn; + +impl CheckIn { + pub fn new() -> Self { + CheckIn + } +} +EOF +cat > crates/libuser/Cargo.toml <<'EOF' +[package] +name = "libuser" +version = "0.1.0" +edition = "2021" + +[dependencies] +libdep = { path = "../libdep" } +EOF +cat > crates/libuser/src/lib.rs <<'EOF' +pub fn make() -> libdep::CheckIn { + libdep::CheckIn::new() +} +EOF +git add -A +GIT_AUTHOR_DATE="2023-01-01T00:00:00" GIT_COMMITTER_DATE="2023-01-01T00:00:00" \ + git commit -q -m "base: CheckIn::new only" +base_sha=$(git rev-parse HEAD) + +# dev's own future: adds with_intensity_opt, with an OLD committer date — +# analogous to a PR merged into dev before this branch's own build. +git switch -q -c dev +cat > crates/libdep/src/lib.rs <<'EOF' +pub struct CheckIn; + +impl CheckIn { + pub fn new() -> Self { + CheckIn + } + + pub fn with_intensity_opt(&self) -> Self { + CheckIn + } +} +EOF +git add -A +GIT_AUTHOR_DATE="2023-06-01T01:02:51" GIT_COMMITTER_DATE="2023-06-01T01:02:51" \ + git commit -q -m "journal: add CheckIn::with_intensity_opt" +dev2_sha=$(git rev-parse HEAD) + +# feature branch, forked BEFORE dev2 exists — does its own unrelated work, +# and gets its own real build (real wall-clock dep-info mtimes) before dev2 +# is ever merged in. +git switch -q -c feature "$base_sha" +cat > crates/libuser/src/lib.rs <<'EOF' +pub fn make() -> libdep::CheckIn { + libdep::CheckIn::new() +} + +pub fn unrelated_feature_work() -> u32 { + 42 +} +EOF +git add -A +git commit -q -m "feature: unrelated libuser work" +feat1_sha=$(git rev-parse HEAD) + +echo "=== feature branch's own real build (real wall-clock artifact mtimes) ===" +CARGO_TARGET_DIR="$target_feature" cargo build --workspace --quiet +# Two pristine snapshots taken right after this one real build, before +# scenario 1's failed build attempt below partially rewrites fingerprints +# against it: one WITHOUT a watermark file (scenario 1's starting state) and +# one WITH (scenario 2's) — both need to restart from this same pre-merge +# baseline, not from whatever a failed build left behind, and scenario 1 +# must never see a watermark or it isn't testing the control it claims to. +cp -a "$target_feature" "$target_feature.pristine-no-watermark" +echo "$feat1_sha" > "$target_feature/.ci-watermark-sha" +cp -a "$target_feature" "$target_feature.pristine-with-watermark" + +# The commit below ("feature: use with_intensity_opt") gets a real "now" +# date — scenario 1 needs it to be unambiguously newer than the dep-info +# artifact just built above, in whole-second terms git commit timestamps +# use. This harness runs fast enough that, without a gap, the two could +# otherwise land in the same wall-clock second and make scenario 1 flaky. +# Real CI never runs this close together; this is purely a test-harness +# margin, matching the one before scenarios 4/5 below. +sleep 2 + +git merge -q dev -m "merge dev into feature" +cat > crates/libuser/src/lib.rs <<'EOF' +pub fn make() -> libdep::CheckIn { + libdep::CheckIn::new().with_intensity_opt() +} + +pub fn unrelated_feature_work() -> u32 { + 42 +} +EOF +git add -A +git commit -q -m "feature: use with_intensity_opt" +feat2_sha=$(git rev-parse HEAD) + +echo +echo "=== 1: NO WATERMARK, same-branch merge (control — expect the bug) ===" +rm -rf "$target_feature" +cp -a "$target_feature.pristine-no-watermark" "$target_feature" +stamp_checkout_now +CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes" +set +e +CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run1.log" 2>&1 +set -e +assert_log_has "$scratch/run1.log" "no method named .with_intensity_opt." \ + "scenario 1: absent watermark reproduces the stale-reuse compile error" + +echo +echo "=== 2: WATERMARK PRESENT, same state (fix — expect success) ===" +rm -rf "$target_feature" +cp -a "$target_feature.pristine-with-watermark" "$target_feature" +stamp_checkout_now +CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run2.log" 2>&1 +assert_log_lacks "$scratch/run2.log" "no method named" \ + "scenario 2: watermark present — build succeeds, no stale reuse" +assert_log_has "$scratch/run2.log" "Compiling libdep" \ + "scenario 2: libdep actually recompiled (not just Fresh-passthrough)" + +echo +echo "=== 3: WATERMARK PRESENT, no further change (warm-path preserved) ===" +echo "$feat2_sha" > "$target_feature/.ci-watermark-sha" +stamp_checkout_now +CARGO_TARGET_DIR="$target_feature" bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_feature" cargo build --workspace -v > "$scratch/run3.log" 2>&1 +assert_log_lacks "$scratch/run3.log" "Compiling" \ + "scenario 3: unchanged re-run stays warm — nothing recompiles" + +# Scenarios 4/5: a brand-new namespace seeded from feature's own (now warm) +# target dir, then merging a THIRD lineage's older change. `sleep` keeps the +# THIRD lineage's own "now"-dated commits a full second clear of the +# artifact just built above — git commit timestamps are whole-second, dep-info +# mtimes are nanosecond, and this harness runs fast enough that the two could +# otherwise land in the same second and make scenario 4's control flaky. Real +# CI never runs this close together; this is purely a test-harness margin. +sleep 2 + +git switch -q -c parallel2 "$dev2_sha" +cat > crates/libdep/src/lib.rs <<'EOF' +pub struct CheckIn; + +impl CheckIn { + pub fn new() -> Self { + CheckIn + } + + pub fn with_intensity_opt(&self) -> Self { + CheckIn + } + + pub fn with_note(&self) -> Self { + CheckIn + } +} +EOF +git add -A +GIT_AUTHOR_DATE="2023-07-15T09:00:00" GIT_COMMITTER_DATE="2023-07-15T09:00:00" \ + git commit -q -m "journal: add CheckIn::with_note" + +# newbranch forks from feature at feat2_sha (scenario 2/3's fixed, warm +# state) and merges parallel2. Clean merge: newbranch already has dev2's +# with_intensity_opt via feature's earlier merge, parallel2 only adds +# with_note on top of that same ancestor. +git switch -q -c newbranch "$feat2_sha" +git merge -q parallel2 -m "merge parallel2 into newbranch" +sed -i 's/CheckIn::new()\.with_intensity_opt()/CheckIn::new().with_intensity_opt().with_note()/' \ + crates/libuser/src/lib.rs +git add -A +git commit -q -m "newbranch: use with_note too" + +echo +echo "=== 4: SEEDED NAMESPACE, watermark stripped (control — expect the bug) ===" +rm -rf "$target_newbranch" +cp -a "$target_feature" "$target_newbranch" +rm -f "$target_newbranch/.ci-watermark-sha" +stamp_checkout_now +CARGO_TARGET_DIR="$target_newbranch" bash "$restore_mtimes" +set +e +CARGO_TARGET_DIR="$target_newbranch" cargo build --workspace -v > "$scratch/run4.log" 2>&1 +set -e +assert_log_has "$scratch/run4.log" "no method named .with_note." \ + "scenario 4: a seeded namespace with no inherited watermark reproduces the same bug" + +echo +echo "=== 5: SEEDED NAMESPACE, watermark inherited via cp -a (fix — expect success) ===" +rm -rf "$target_newbranch" +cp -a "$target_feature" "$target_newbranch" +stamp_checkout_now +CARGO_TARGET_DIR="$target_newbranch" bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_newbranch" cargo build --workspace -v > "$scratch/run5.log" 2>&1 +assert_log_lacks "$scratch/run5.log" "no method named" \ + "scenario 5: inherited watermark — build succeeds on the namespace's first run" +assert_log_has "$scratch/run5.log" "Compiling libdep" \ + "scenario 5: libdep actually recompiled on the seeded namespace" + +# Scenarios 6/7: a second job in the same workflow can share the +# `ci` job's per-branch $CARGO_TARGET_DIR namespace — `debug/` (the `ci` +# job's fmt/clippy/test steps) and `release/` (the `web` job's +# build-web-opt) sit side by side in one directory. The FIRST version of +# these scenarios ran restore-mtimes.sh ONCE, then built both profiles +# against its single output — modelling "both jobs read before either +# writes". Review caught that this is the wrong ordering: on a serial +# runner with no `needs:` between the two jobs, one job's watermark WRITE +# can land before the other job's own READ, on the very same trigger. These +# scenarios now run restore-mtimes.sh TWICE IN SEQUENCE, with a watermark +# write in between — the real ordering — and scenario 6 is required to +# reproduce the bug against a single shared watermark file before scenario 7 +# demonstrates the fix (one watermark file per consumer). `target_feature` +# is already warm and consistent at feat2_sha from scenarios 2/3 (debug/ +# built, watermark=feat2_sha); build release/ against the same namespace too +# so both profile dirs start warm, exactly as they would after both real +# jobs have run once against a branch. +echo +echo "=== building the release profile too, so both profile dirs start warm ===" +CARGO_TARGET_DIR="$target_feature" cargo build --workspace --release --quiet +cp -a "$target_feature" "$scratch/mp-pristine-shared-watermark" +cp -a "$target_feature" "$scratch/mp-pristine-split-watermarks" +cp "$scratch/mp-pristine-split-watermarks/.ci-watermark-sha" \ + "$scratch/mp-pristine-split-watermarks/.ci-watermark-sha-wasm32" + +# A third lineage, forked from dev2_sha (same shape as parallel2 above), +# adding a further method with an OLD committer date — analogous to a second, +# later dev commit that both real jobs would need to see as a real change. +sleep 2 +git switch -q -c parallel3 "$dev2_sha" +cat > crates/libdep/src/lib.rs <<'EOF' +pub struct CheckIn; + +impl CheckIn { + pub fn new() -> Self { + CheckIn + } + + pub fn with_intensity_opt(&self) -> Self { + CheckIn + } + + pub fn with_flag(&self) -> Self { + CheckIn + } +} +EOF +git add -A +GIT_AUTHOR_DATE="2023-08-01T09:00:00" GIT_COMMITTER_DATE="2023-08-01T09:00:00" \ + git commit -q -m "journal: add CheckIn::with_flag" + +# multiprofile forks from feat2_sha (feature's own fixed, warm point) and +# merges parallel3 — clean merge, same shape as newbranch/parallel2 above. +git switch -q -c multiprofile "$feat2_sha" +git merge -q parallel3 -m "merge parallel3 into multiprofile" +cat > crates/libuser/src/lib.rs <<'EOF' +pub fn make() -> libdep::CheckIn { + libdep::CheckIn::new().with_intensity_opt().with_flag() +} + +pub fn unrelated_feature_work() -> u32 { + 42 +} +EOF +git add -A +git commit -q -m "multiprofile: use with_flag too" +mp_sha=$(git rev-parse HEAD) + +echo +echo "=== 6: TWO CONSUMERS, ONE SHARED WATERMARK FILE, sequential jobs (control — expect the bug) ===" +# "Job A" (ci-equivalent, debug/ consumer) runs FIRST: its own checkout, +# its own restore-mtimes call, its own build, then its own watermark write — +# exactly the actions' real step order, just inlined here instead of split +# across two jobs. +rm -rf "$target_multiprofile" +cp -a "$scratch/mp-pristine-shared-watermark" "$target_multiprofile" +stamp_checkout_now +CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace -v > "$scratch/run6-debug.log" 2>&1 +assert_log_lacks "$scratch/run6-debug.log" "no method named" \ + "scenario 6: job A (debug/, running first) builds cleanly off the true prior watermark" +assert_log_has "$scratch/run6-debug.log" "Compiling libdep" \ + "scenario 6: job A actually recompiled libdep in debug/" +echo "$mp_sha" > "$target_multiprofile/.ci-watermark-sha" + +# "Job B" (web-equivalent, release/ consumer) runs SECOND, same trigger, same +# HEAD, same $CARGO_TARGET_DIR, no `needs:` between them — the real ordering +# review caught. Its own checkout (stamp_checkout_now again, faithfully +# modelling a separate checkout), then its own restore-mtimes call, which +# reads the SAME watermark file job A just advanced to mp_sha: HEAD..mp_sha +# is an empty diff, so the override never fires and every path falls back to +# its plain git-log timestamp — including the backdated libdep change. +stamp_checkout_now +CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes" +set +e +CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace --release -v > "$scratch/run6-release.log" 2>&1 +set -e +assert_log_has "$scratch/run6-release.log" "no method named .with_flag." \ + "scenario 6: job B (release/, running second) reads the watermark job A just advanced, gets an empty diff, and reproduces the original stale-reuse bug" + +echo +echo "=== 7: TWO CONSUMERS, TWO WATERMARK FILES, sequential jobs (fix — expect success) ===" +# Same sequential ordering as scenario 6, but each job reads and writes its +# OWN watermark file (CI_WATERMARK_FILE), exactly as the cargo-cache action does. +rm -rf "$target_multiprofile" +cp -a "$scratch/mp-pristine-split-watermarks" "$target_multiprofile" +stamp_checkout_now +CARGO_TARGET_DIR="$target_multiprofile" bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace -v > "$scratch/run7-debug.log" 2>&1 +assert_log_lacks "$scratch/run7-debug.log" "no method named" \ + "scenario 7: job A (debug/) builds cleanly off its own watermark" +assert_log_has "$scratch/run7-debug.log" "Compiling libdep" \ + "scenario 7: job A actually recompiled libdep in debug/" +echo "$mp_sha" > "$target_multiprofile/.ci-watermark-sha" + +# Job B reads ITS OWN file (.ci-watermark-sha-wasm32), untouched by job A's +# write above — still feat2_sha, so the diff against mp_sha is exactly the +# real change set, not empty. +stamp_checkout_now +CARGO_TARGET_DIR="$target_multiprofile" CI_WATERMARK_FILE=.ci-watermark-sha-wasm32 \ + bash "$restore_mtimes" +CARGO_TARGET_DIR="$target_multiprofile" cargo build --workspace --release -v > "$scratch/run7-release.log" 2>&1 +assert_log_lacks "$scratch/run7-release.log" "no method named" \ + "scenario 7: job B (release/) builds cleanly off its OWN watermark, unaffected by job A's write" +assert_log_has "$scratch/run7-release.log" "Compiling libdep" \ + "scenario 7: job B actually recompiled libdep in release/" + +echo +echo "ALL ${pass_count} ASSERTIONS PASSED" diff --git a/scripts/restore-mtimes.sh b/scripts/restore-mtimes.sh new file mode 100755 index 0000000..18ac757 --- /dev/null +++ b/scripts/restore-mtimes.sh @@ -0,0 +1,345 @@ +#!/usr/bin/env bash +# Restores every git-tracked file's mtime to the timestamp of the most recent +# commit that touched it. `actions/checkout` stamps every file with the +# checkout's wall-clock "now", and Cargo's fingerprint freshness check is +# mtime-based by default. Result: even though the persistent volume genuinely +# holds prior build artifacts, every checkout makes every tracked file look +# "newer than what was recorded at last build", so every crate rebuilds from +# scratch on every run — measured directly on a Bevy-sized workspace before +# this fix, by grepping a warm-volume run's log for Cargo's `Fresh` cache-hit +# marker: zero matches across ~28 crates. Do not remove this as "redundant +# with the target-dir cache": it is the fix that makes that cache take effect. +# +# Algorithm: a single `git log --name-only --no-renames` traversal +# (newest-commit-first, git log's default order) instead of one `git log` +# call per file — for a workspace this size a per-file walk would mean +# hundreds of separate history walks. For each path, the first timestamp +# encountered under newest-first traversal is the timestamp of the most +# recent commit that touched it. Only files currently tracked by git +# (`git ls-files`) are touched; deleted/renamed-away historical paths and +# anything gitignored are left alone. +# +# Each commit's timestamp line carries a `@@MTIME@@` sentinel prefix rather +# than relying on blank-line paragraph boundaries between commits. Consecutive +# merge commits with an empty diff (under `--no-renames`) print their +# timestamp lines back-to-back with no blank line between them — a +# blank-line-based parser misreads the second bare timestamp line as a +# filename and desyncs every timestamp after it. The sentinel can't collide +# with a real tracked path (it's a fixed literal no repo file is named), so +# every commit's timestamp line is unambiguous regardless of what precedes or +# follows it. +# +# Note: for a handful of paths this single raw pass picks a nominally +# different (usually newer) commit than `git log -1 -- ` would, which +# applies pathspec history-simplification (it can skip a commit whose change +# to a path got superseded by how a later merge resolved that path). That +# divergence is harmless here: correctness only requires (a) determinism — +# same HEAD always yields the same mtime for a given path, so a warm cache +# stays warm — and (b) change-sensitivity — a commit that actually changes a +# path's content is always the newest entry touching that path in traversal +# order, so it always wins the "first seen" race and invalidates that path's +# fingerprint. Reproducing full history-simplification per path would mean +# back to one `git log` call per file, the exact cost this single pass +# avoids. +# +# Directory mtimes, not just file mtimes: a build script that declares +# `cargo::rerun-if-changed=` (a directory rather than a file) has +# its rerun decision keyed partly on that directory inode's own mtime, and +# checkout recreates every directory with a fresh "now" mtime just like it +# does files. Restoring only tracked FILES leaves such a directory always +# looking newer than the recorded fingerprint, so the build script re-runs +# every time, marking its crate dirty and cascading to every downstream crate +# regardless of whether anything under the directory changed. Fixed by +# restoring every ancestor directory of every tracked path (excluding `.git/`, +# which no tracked path can be under) to the MAXIMUM restored timestamp among +# the tracked files in its subtree — deterministic given the per-file +# timestamps already computed above, and it still fires a rerun when a file +# under that directory genuinely changed, since that file's newer timestamp +# propagates to every ancestor's max. Costs about a second; worth paying even +# on a workspace with no such build script yet, so the day one appears the +# cache stability already covers it. +# +# Soundness against the cross-branch stale-binary hazard: the cargo-cache +# action gives every ref its own target directory on the shared volume, so no +# two refs' fingerprints ever share a directory and this script never has to +# arbitrate freshness across refs — only within one ref's own history, which +# is exactly what it is built to do soundly. On a nightly toolchain, +# CARGO_UNSTABLE_CHECKSUM_FRESHNESS is a complementary, stronger guarantee +# (content-addressed rather than mtime-based freshness); this script is not +# made redundant by it, because directory-form `rerun-if-changed` build-script +# watches are not covered by it and stable historical mtimes stay cheap +# belt-and-braces alongside it. +# +# The merge hazard (the merge hazard), and why it needed a watermark, not just a +# namespace: namespacing keeps two branches' fingerprints apart, but it does +# NOT make the comparison this script feeds sound on its own. The comparison +# that actually matters is not "is this file's mtime the historically correct +# one for its most recent commit" (which the git-log pass above answers +# exactly right) — it's **"is this file's mtime newer than the dep-info +# artifact Cargo already has on disk for it in THIS target dir"**, because +# that dep-info's own mtime is a real wall-clock timestamp stamped by rustc +# when it last wrote the artifact, not anything git-derived. Ordinary +# same-branch iteration makes the git-log answer and the Cargo answer agree +# for free, by causality: you cannot commit a change before the build it is +# meant to invalidate has already finished, so every file changed since the +# last build necessarily carries a commit timestamp after that build's real +# completion time. A merge breaks exactly that causal link — it can pull in a +# commit authored on a parallel lineage *before* this branch's own last +# build, so the git-log answer ("this file's most recent commit is old") and +# the Cargo answer ("this file changed since my last build") diverge, and +# restore-mtimes stamping the historically-correct-but-too-old mtime is what +# let Cargo reuse an rlib built before the method the merge introduced +# existed (observed in production as an "impossible" compile error: a crate +# reusing an rlib built before the method a merge introduced existed). +# +# The fix: for a file changed since *this target dir's own last successful +# build* — not "recently" in absolute git-log time — always stamp "now" +# (this run's own wall-clock start), never the historical commit timestamp, +# because "now" is provably later than that build's real completion time no +# matter what any commit's authored/committer date says. the cargo-cache-publish action's watermark step writes the built HEAD SHA into +# "$CARGO_TARGET_DIR/${CI_WATERMARK_FILE:-.ci-watermark-sha}" after every +# green run of THIS branch's own namespace; this script reads it back (if +# present — see the read below for what "absent" means and why it's still +# safe) and treats `git diff --name-only ..HEAD` as authoritative +# over the git-log-only answer for exactly those paths. Everything NOT in +# that diff keeps the git-log timestamp exactly as before, so the ordinary +# no-merge warm-cache case (the whole reason this script exists) is +# untouched. +# +# CI_WATERMARK_FILE: the watermark filename is a +# per-job parameter, NOT a per-branch constant — two jobs sharing one +# $CARGO_TARGET_DIR namespace must use two DIFFERENT watermark files, never +# one. A watermark answers "what did THIS consumer last successfully build", +# and two jobs building different target triples (e.g. one job's host debug/release vs another's +# wasm32-unknown-unknown) are two separate +# consumers with two separate answers. A single shared file breaks the +# moment a second job joins the namespace: job A reads the true prior +# watermark, rebuilds correctly, and advances the file to HEAD; job B then +# runs SECOND (same trigger, same HEAD, no `needs:` ordering — a serial +# runner still executes both jobs one after another within one trigger) and +# reads that JUST-ADVANCED watermark, computing an empty HEAD..HEAD diff, and +# falls back to git-log-only timestamps for everything — exactly this +# script's pre-fix behaviour, reintroduced for whichever job happens to run +# second on any trigger that includes a backdating merge. Caught via a two-crate repro, not by inspection alone. +# restore-mtimes-selftest.sh's scenarios 6/7 exercise this exact sequential +# ordering (one job's watermark write landing between two restore-mtimes +# invocations against the same namespace), not just two simultaneous reads. +# Defaults to ".ci-watermark-sha" — unset callers (a local/manual run, or a +# namespace with only ever one consumer) get the original filename and +# behaviour unchanged. +# +# restore-mtimes-selftest.sh, alongside this file, is a regression test for +# the watermark mechanism: a real scratch cargo workspace, a real +# restore-mtimes.sh run, a real `cargo build`, asserting both that the +# staleness bug reproduces without a watermark and that it's fixed with one +# (same-branch merge AND seeded-namespace cases). Run it by hand after touching this +# script's timestamp logic or the actions' seed/watermark step order. +# +# The watermark file deliberately lives INSIDE $CARGO_TARGET_DIR rather than +# a separate location, for two reasons that both matter: it rides along for +# free with prune-cache.sh's eviction pass (an evicted +# namespace loses its watermark along with everything else it would have +# been wrong to trust anyway), and it rides along for free with the +# seed-snapshot mechanism (the cargo-cache-publish action publishes +# $CARGO_TARGET_DIR wholesale, including this file) — so a brand-new branch +# namespace seeded from its base's published snapshot inherits the base's +# watermark along with the base's artifacts, and this script correctly diffs +# from THAT SHA rather than treating the seeded branch as having no watermark +# at all. That is exactly the same hazard class one level up (a new branch's +# first run reusing artifacts seeded from a base build that predates some of +# the base's own history), and it needs the same fix — which it gets for +# free rather than needing separate handling, as long as the cargo-cache action orders "seed +# or reuse the target dir" before "restore mtimes" so the watermark is +# actually in place, on disk, by the time this script runs. (It is — see the +# step order in the cargo-cache action.) +# +# Absent watermark (no file, or the target dir doesn't exist yet at all) is +# always safe, never a silent miscompare: this script simply skips the +# override and falls back fully to the git-log timestamps, exactly as it did +# before this fix existed. That is correct for a genuinely first run against +# an empty, unseeded namespace (nothing to compare against — everything is +# about to be a cold build regardless of what mtime it gets) and conservative +# for every other case that could produce an absent watermark: a watermark SHA +# that is no longer reachable in this checkout (a force-push or rebase +# rewrote the branch the watermark was recorded against) is NOT treated the +# same as "no watermark at all" — the reachability check below instead +# treats every tracked file as changed, rather than silently trusting a diff +# against a commit that may no longer describe any real ancestry +# relationship to HEAD. A run that fails before reaching +# the "Record build watermark" step simply never advances the watermark, so +# the next run's diff base stays at the last GREEN build — over-inclusive +# (it may re-stamp files that a failed run partially rebuilt anyway) but +# never under-inclusive, which is the only direction that matters for +# soundness.set -euo pipefail + +repo_root=$(git rev-parse --show-toplevel) +cd "$repo_root" + +# A shallow checkout (the `actions/checkout` default) only has the tip +# commit available, which diffs against the empty tree — every tracked file +# would resolve to that single commit's timestamp, a uniform stamp that +# changes with every new commit and still defeats the fingerprint cache +# run-over-run (the exact bug this script fixes). That failure mode is +# silent otherwise: every path still "resolves" to some timestamp (just the +# wrong, unhelpfully-uniform one), so the missing-path fallback below never +# fires and nothing else in this script would surface it. Fail loudly here +# instead so a future `fetch-depth` regression on the checkout step shows up +# in the CI log rather than silently reverting to full rebuilds every run. +if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then + echo "::error::restore-mtimes: shallow clone detected — per-file mtimes unavailable, cache reuse will be defeated; set fetch-depth: 0 on the checkout step" >&2 + exit 1 +fi + +start=$(date +%s) + +resolved=$(mktemp) +tracked=$(mktemp) +to_touch=$(mktemp) +missing=$(mktemp) +all_touched=$(mktemp) +dir_touch=$(mktemp) +changed_since_watermark=$(mktemp) +to_touch_overridden=$(mktemp) +trap 'rm -f "$resolved" "$tracked" "$to_touch" "$missing" "$all_touched" "$dir_touch" "$changed_since_watermark" "$to_touch_overridden"' EXIT + +# Emit "\t" once per path, first-seen-under-newest-first wins. +git log --format='@@MTIME@@%ct' --name-only --no-renames | awk ' + /^@@MTIME@@[0-9]+$/ { ts = substr($0, 10); next } + NF == 0 { next } + !($0 in seen) { seen[$0] = 1; printf "%s\t%s\n", ts, $0 } +' > "$resolved" + +git ls-files > "$tracked" + +# Restrict to files git currently tracks — never .git/ internals, never +# gitignored build output, never a path that's since been deleted/renamed. +awk -F'\t' ' + NR == FNR { track[$0] = 1; next } + $2 in track { print; seen[$2] = 1 } +' "$tracked" "$resolved" > "$to_touch" + +# Watermark override: see the header comment above for why this +# comparison — against this target dir's own last successful build, not +# against absolute git-log history — is the one Cargo actually needs. +watermark_count=0 +watermark_status="no CARGO_TARGET_DIR set (not running under the cargo-cache action) — skipped" +watermark_name="${CI_WATERMARK_FILE:-.ci-watermark-sha}" +watermark_file="${CARGO_TARGET_DIR:-}/${watermark_name}" +if [ -n "${CARGO_TARGET_DIR:-}" ] && [ -f "$watermark_file" ]; then + watermark_sha=$(cat "$watermark_file") + if git cat-file -e "${watermark_sha}^{commit}" 2>/dev/null; then + # Every path that differs between the watermark commit and HEAD — + # added, modified, or deleted — content-wise, regardless of what any + # commit along the way claims its own timestamp is. + git diff --name-only "$watermark_sha" HEAD > "$changed_since_watermark" + watermark_status="diffing against watermark ${watermark_sha} in ${CARGO_TARGET_DIR}/${watermark_name}" + else + # The watermark names a commit this checkout can no longer see — a + # force-push or rebase rewrote history out from under it. Trusting the + # diff would silently compare HEAD against a commit with no real + # ancestry relationship to it. Fall back to treating every tracked file + # as changed since the (unknowable) last build: safe (over-invalidates, + # never under-invalidates) and self-heals the moment the next green run + # records a fresh, reachable watermark. + cp "$tracked" "$changed_since_watermark" + watermark_status="watermark ${watermark_sha} unreachable (force-push/rebase?) in ${CARGO_TARGET_DIR}/${watermark_name} — treating every tracked file as changed since last build" + fi +else + : > "$changed_since_watermark" + if [ -n "${CARGO_TARGET_DIR:-}" ]; then + watermark_status="no watermark yet in ${CARGO_TARGET_DIR}/${watermark_name} (first run against this namespace) — skipped" + fi +fi + +# For every path in the diff, override its git-log timestamp with "now" +# ($start, this run's own wall-clock start) — provably later than the real +# completion time of the build the watermark names, unlike any commit's +# authored/committer date. Everything NOT in the diff keeps its git-log +# timestamp exactly as before. +if [ -s "$changed_since_watermark" ]; then + # Re-derive the count from the intersection with $to_touch (rather than + # `wc -l < "$changed_since_watermark"` directly) so the reported number + # only counts paths this script is actually about to stamp differently, + # not every path `git diff` names (which, unlike $to_touch, is not + # restricted to currently-tracked files — e.g. a path deleted since the + # watermark commit). + watermark_count=$(awk -F'\t' 'NR == FNR { c[$0] = 1; next } $2 in c' "$changed_since_watermark" "$to_touch" | wc -l | tr -d ' ') + + awk -F'\t' -v now="$start" ' + NR == FNR { changed[$0] = 1; next } + { if ($2 in changed) print now "\t" $2; else print } + ' "$changed_since_watermark" "$to_touch" > "$to_touch_overridden" + mv "$to_touch_overridden" "$to_touch" +fi + +# Any tracked path whose history couldn't resolve (only possible under an +# unexpectedly shallow checkout, since `fetch-depth: 0` on the checkout step +# means full history is normally present) falls back to a fixed constant +# epoch rather than being left alone at checkout-time "now". What Cargo's +# fingerprint check needs is an mtime that's STABLE run-over-run for +# unchanged content, not necessarily a historically accurate one — a +# constant epoch gives that stability so the file still gets cache reuse +# instead of permanently missing the cache on every single run. +awk -F'\t' ' + NR == FNR { seen[$2] = 1; next } + !($0 in seen) +' "$to_touch" "$tracked" > "$missing" + +touched=0 +while IFS=$'\t' read -r ts path; do + [ -e "$path" ] || [ -L "$path" ] || continue + # `-h`: set the mtime of a git-tracked symlink itself, never the target it + # resolves to. No path in this repo is currently a tracked symlink, but a + # future one (e.g. a shared-config symlink into a dotfiles + # checkout) would otherwise get its mtime applied + # to whatever it resolves to outside the repo — keeping `-h` costs nothing + # today and avoids that surprise later. + touch -h -d "@$ts" -- "$path" + touched=$((touched + 1)) +done < "$to_touch" + +fallback_count=0 +if [ -s "$missing" ]; then + while IFS= read -r path; do + [ -e "$path" ] || [ -L "$path" ] || continue + touch -h -d "@0" -- "$path" + fallback_count=$((fallback_count + 1)) + done < "$missing" +fi + +# Directory pass: for every tracked file's full ancestor chain (excluding +# the file's own leaf name), track the maximum timestamp seen. A directory +# with no tracked file directly in it but with tracked files somewhere in +# its subtree (e.g. `crates/`) still accumulates a max from those deeper +# files, since every ancestor level of every file's path is a separate +# prefix entry here. Fixed-epoch fallback entries (ts=0) are included too — +# harmless, since 0 can never win a max against any real commit timestamp. +cat "$to_touch" > "$all_touched" +if [ -s "$missing" ]; then + awk '{ print "0\t" $0 }' "$missing" >> "$all_touched" +fi +awk -F'\t' ' + { + ts = $1; path = $2 + n = split(path, parts, "/") + prefix = "" + for (i = 1; i < n; i++) { + prefix = (prefix == "" ? parts[i] : prefix "/" parts[i]) + if (!(prefix in maxts) || ts + 0 > maxts[prefix] + 0) maxts[prefix] = ts + } + if (!("." in maxts) || ts + 0 > maxts["."] + 0) maxts["."] = ts + } + END { + for (d in maxts) printf "%s\t%s\n", maxts[d], d + } +' "$all_touched" > "$dir_touch" + +dirs_touched=0 +while IFS=$'\t' read -r ts dir; do + [ -d "$dir" ] || continue + touch -d "@$ts" -- "$dir" + dirs_touched=$((dirs_touched + 1)) +done < "$dir_touch" + +elapsed=$(( $(date +%s) - start )) +echo "restore-mtimes: touched ${touched} files + ${dirs_touched} dirs of $(wc -l < "$tracked") tracked files from history in ${elapsed}s (${fallback_count} fell back to a fixed epoch, ${watermark_count} overridden to now: ${watermark_status})" diff --git a/scripts/seed-target-dir-selftest.sh b/scripts/seed-target-dir-selftest.sh new file mode 100755 index 0000000..2079802 --- /dev/null +++ b/scripts/seed-target-dir-selftest.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Regression test for seed-target-dir.sh: which source a run seeds from, and +# what happens when two jobs sharing one cache key seed at the same time. +# +# Runs the ACTUAL script against a real scratch cache directory with fake +# target trees standing in for cargo output — no compiler needed, so this is +# the fast half of the suite. hardlink-clone-selftest.sh covers the parts that +# need a real build. +# +# What each scenario demonstrates: +# +# 1. BASE SNAPSHOT PREFERRED — a PR whose base has published a snapshot +# seeds from it, and the seeded directory really is a hardlink clone +# (shared inodes), not a copy. +# 2. OWN DIR WINS — a second run of the same ref reuses what is already +# there and does not re-seed over its own work. +# 3. OWN SNAPSHOT AS SELF-RESTORE — a publisher whose live target dir was +# evicted restores from the snapshot it last published, instead of +# rebuilding cold. +# 4. FALLBACK DIR — with no snapshot at all, an explicitly configured +# fallback (a pre-existing flat cache, during a migration) is used. +# 5. COLD — with nothing available, the directory is created empty rather +# than the script failing. +# 6. LOCK FILES STRIPPED — Cargo's in-place-flock'd lock files never +# survive a clone, because a shared lock inode would make two branches +# contend on one mutex. +# 7. CONCURRENT SEED IS ATOMIC — two seeds racing on one cache key: the +# loser discards its staging copy and uses the winner's directory, and +# at no point is a partially-populated directory visible under the final +# name. This is the property that replaces "the runner only has one job +# slot" with an actual guarantee. +set -euo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +. "$script_dir/cache-lib.sh" + +scratch=$(mktemp -d) +trap 'rm -rf "$scratch"' EXIT +root="$scratch/cache" +mkdir -p "$root" +pass_count=0 + +fail() { echo "ASSERTION FAILED: $*" >&2; exit 1; } +ok() { pass_count=$((pass_count + 1)); echo "PASS: $*"; } +assert_file() { [ -e "$1" ] || fail "expected $1 to exist ($2)"; ok "$2"; } +assert_absent() { [ -e "$1" ] && fail "expected $1 to be gone ($2)"; ok "$2"; } +assert_content() { [ "$(cat "$1")" = "$2" ] || fail "expected '$2' in $1, got '$(cat "$1")' ($3)"; ok "$3"; } + +# A plausible target tree: a big shared artifact, a mutable fingerprint, a +# build-script output, and a lock file. +make_tree() { + local d="$1" marker="$2" + mkdir -p "$d/debug/deps" "$d/debug/.fingerprint/x" "$d/debug/build/x/out" + echo "$marker" > "$d/debug/deps/libx.rlib" + echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x" + echo "$marker" > "$d/debug/build/x/out/gen.txt" + : > "$d/debug/.cargo-lock" +} + +seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; } + +BASE_KEY=$(cache_key dev) +OWN_KEY=$(cache_key feat/thing) + +echo "=== 1: base snapshot preferred, and cloned by hardlink ===" +make_tree "$root/snapshot-$BASE_KEY" base-content +seed "$OWN_KEY" "$BASE_KEY" "$root" job1 +own="$root/target-$OWN_KEY" +assert_content "$own/debug/deps/libx.rlib" base-content "seeded from the base snapshot" +[ "$(stat -c '%i' "$own/debug/deps/libx.rlib")" = "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/deps/libx.rlib")" ] \ + || fail "artifact was copied, not hardlinked" +ok "artifact shares an inode with the snapshot (hardlink clone, not a copy)" +[ "$(stat -c '%i' "$own/debug/.fingerprint/x/dep-lib-x")" != "$(stat -c '%i' "$root/snapshot-$BASE_KEY/debug/.fingerprint/x/dep-lib-x")" ] \ + || fail "fingerprint still shares an inode with the snapshot" +ok "fingerprint is privately owned (unshare_mutable_paths ran)" + +echo +echo "=== 6: Cargo lock files never survive a clone ===" +assert_absent "$own/debug/.cargo-lock" "cloned .cargo-lock removed" + +echo +echo "=== 2: an existing own dir is reused, never re-seeded over ===" +echo own-work > "$own/debug/deps/libx.rlib" +seed "$OWN_KEY" "$BASE_KEY" "$root" job1 +assert_content "$own/debug/deps/libx.rlib" own-work "own directory reused as-is" +grep -q 'reusing this ref' "$scratch/log" || fail "expected the reuse path in the log" +ok "reuse is reported in the log" + +echo +echo "=== 3: a publisher restores from its own snapshot after eviction ===" +make_tree "$root/snapshot-$BASE_KEY" published-dev +seed "$BASE_KEY" "" "$root" job1 +assert_content "$root/target-$BASE_KEY/debug/deps/libx.rlib" published-dev "publisher self-restored from its own snapshot" + +echo +echo "=== 4: explicit fallback dir when no snapshot exists ===" +OTHER=$(cache_key feat/other) +make_tree "$scratch/legacy-flat" legacy +seed "$OTHER" "$(cache_key nosuch)" "$root" job1 "$scratch/legacy-flat" +assert_content "$root/target-$OTHER/debug/deps/libx.rlib" legacy "seeded from the fallback dir" + +echo +echo "=== 5: cold start when nothing is available ===" +COLD=$(cache_key feat/cold) +seed "$COLD" "$(cache_key nosuch)" "$root" job1 +[ -d "$root/target-$COLD" ] || fail "cold start did not create the directory" +[ -z "$(ls -A "$root/target-$COLD")" ] || fail "cold start directory is not empty" +ok "cold start creates an empty directory rather than failing" + +echo +echo "=== 7: two jobs racing on one cache key ===" +RACE=$(cache_key feat/race) +make_tree "$root/snapshot-$BASE_KEY" race-source +# Both jobs seed concurrently from the same snapshot into the same key. Each +# stages under its own tag, so the only interaction is the final rename. +( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobA > "$scratch/logA" 2>&1 ) & +( bash "$script_dir/seed-target-dir.sh" "$RACE" "$BASE_KEY" "$root" jobB > "$scratch/logB" 2>&1 ) & +wait +race_dir="$root/target-$RACE" +assert_content "$race_dir/debug/deps/libx.rlib" race-source "the surviving directory is complete" +[ -z "$(find "$root" -maxdepth 1 -name '.stage-*' -print -quit)" ] || fail "a staging directory was left behind" +ok "no staging directory survived the race" +# Exactly one job may claim it seeded; the other must report either the +# concurrent-peer path or a plain reuse (if it started after the winner +# finished). Neither may report a cold start. +if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then + cat "$scratch/logA" "$scratch/logB"; fail "a racing job reported a cold start" +fi +ok "neither racing job fell through to a cold start" + +echo +echo "seed-target-dir-selftest: ${pass_count} assertions passed" diff --git a/scripts/seed-target-dir.sh b/scripts/seed-target-dir.sh new file mode 100755 index 0000000..0450b91 --- /dev/null +++ b/scripts/seed-target-dir.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Consume side: make this run's own target dir exist, warm, and private. +# +# Usage: seed-target-dir.sh [fallback-dir] +# own-key cache key for this run's own ref +# base-key cache key for the ref to layer over ("" for a run whose own +# ref IS a reference branch) +# cache-root mount point of the persistent volume +# tag a per-job-per-run unique string, used to name the staging +# directory so two concurrent jobs can never collide on it +# fallback-dir optional absolute path to seed from when no snapshot exists +# (a legacy flat cache dir during a migration, typically) +# +# The design in one paragraph: a PR branch's first run hardlink-clones the +# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs +# time proportional to inode count rather than data volume — on ext4, with no +# reflink support, that is the only way to make "layer over the base" cheap. +# Snapshot rather than the base's live target dir, because a live directory is +# being written by its own job while a consumer reads it, and a torn read +# pairs one build's fingerprint with another build's artifact — which degrades +# to a WRONG reuse, not to a safe miss. Neither half is novel; the combination +# is what makes the scheme both cheap and sound, instead of one or the other. +# +# Everything about correctness under concurrency lives in +# hardlink_clone_into() and unshare_mutable_paths() in cache-lib.sh — read +# those before changing anything here. +set -euo pipefail +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" + +if [ $# -lt 4 ] || [ $# -gt 5 ]; then + echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2 + exit 1 +fi + +OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}" +OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY") + +mkdir -p "$ROOT" + +if [ -d "$OWN_DIR" ]; then + echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)" + echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}" + exit 0 +fi + +# Source preference, most specific first: +# +# 1. the base ref's snapshot — the ordinary PR case, and the whole point. +# 2. this ref's OWN snapshot — a reference branch whose live target dir was +# evicted under disk pressure can restore itself from the last snapshot +# it published, instead of paying a cold rebuild. Free, given the +# snapshot already exists. +# 3. an explicit fallback directory — migration off a pre-existing flat +# cache, so the first run under this scheme isn't a needless cold build. +CANDIDATES=() +[ -n "$BASE_KEY" ] && CANDIDATES+=("$(snapshot_dir_for "$ROOT" "$BASE_KEY"):base snapshot") +CANDIDATES+=("$(snapshot_dir_for "$ROOT" "$OWN_KEY"):own snapshot") +[ -n "$FALLBACK" ] && CANDIDATES+=("${FALLBACK}:fallback dir") + +for entry in "${CANDIDATES[@]}"; do + src="${entry%%:*}"; label="${entry#*:}" + [ -d "$src" ] || continue + echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}" + start=$(date +%s) + if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then + echo "seed: cloned in $(( $(date +%s) - start ))s" + echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}" + else + # Another job sharing this cache key won the rename while we were + # cloning. Its directory is complete (the rename is the publish step), so + # there is nothing to do but use it — and nothing was ever observable in + # a half-seeded state. + echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs" + echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}" + fi + exit 0 +done + +echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold" +echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}" +mkdir -p "$OWN_DIR" diff --git a/scripts/selftest.sh b/scripts/selftest.sh new file mode 100755 index 0000000..9bf3f88 --- /dev/null +++ b/scripts/selftest.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Runs every selftest in this directory and reports a single verdict. +# +# bash scripts/selftest.sh # everything +# bash scripts/selftest.sh --fast # skip the ones that invoke a compiler +# +# The compiler-backed tests (hardlink-clone, restore-mtimes) are the ones that +# verify behaviour against real Cargo rather than against a fixture, so +# --fast is for iterating, not for signing off a change. +set -uo pipefail +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) + +FAST=0 +[ "${1:-}" = "--fast" ] && FAST=1 + +FIXTURE_TESTS=(seed-target-dir-selftest.sh publish-snapshot-selftest.sh prune-cache-selftest.sh) +CARGO_TESTS=(hardlink-clone-selftest.sh restore-mtimes-selftest.sh) + +TESTS=("${FIXTURE_TESTS[@]}") +if [ "$FAST" = "0" ]; then TESTS+=("${CARGO_TESTS[@]}"); fi + +failed=() +for t in "${TESTS[@]}"; do + echo + echo "############################################################" + echo "# $t" + echo "############################################################" + if bash "$script_dir/$t"; then + echo "--> $t OK" + else + echo "--> $t FAILED" + failed+=("$t") + fi +done + +echo +if [ "${#failed[@]}" -eq 0 ]; then + echo "selftest: all ${#TESTS[@]} suites passed" + [ "$FAST" = "1" ] && echo "NOTE: --fast skipped ${CARGO_TESTS[*]}" + exit 0 +fi +echo "selftest: ${#failed[@]} of ${#TESTS[@]} suites FAILED: ${failed[*]}" +exit 1