fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was that hardlink-cloning from a published snapshot closes gitdan #911 "by construction, not by the single job slot". Review disproved that. This makes the claim true, and corrects the README where it could only be bounded. Finding 1 (verdict-level) — silent partial clone ------------------------------------------------ `hardlink_clone_into` ran `cp -al` with no exit-status check, and both call sites invoked it as a condition, which suppresses `set -e` for the whole call. A publisher's `rm -rf` of the generation it rotated away therefore unlinked entries beneath an in-flight consumer walk, and the truncated tree was renamed into place and reported as success. Both layers are fixed: * The consumer verifies its own clone. Every attempt checks `cp -al`'s status explicitly, the source directory's inode before and after (a wholesale replacement mid-walk splices two generations), and the entry count — the only signal for a subtree unlinked before its parent was listed, since `cp -al` reports no error for one it never saw. Any failure discards the staging tree and retries; exhausting the attempts returns a distinct status 2 and fails the job rather than seeding a partial cache. `unshare_subtree` / `_unshare_files` now propagate failure too — a swallowed unshare leaves the clone aliasing its source, the exact corruption that step exists to prevent. * The publisher does not unlink under a reader. A consumer publishes a `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed; one arriving after the scan necessarily resolves to the new generation. The publisher waits for readers to drain and, on timeout, DEFERS reclamation rather than forcing it — the old generation is left as `.publish-old-<key>-<tag>` and swept by a later publish. So correctness is closed by construction; disk reclamation is bounded, not immediate. The residual is capped at one deferred generation per publisher ref, and the README now says exactly that instead of the disproved claim. Finding 2 — restore-mtimes.sh ran with no errexit ------------------------------------------------- `set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo pipefail`), so it was entirely commented out: a partial failure of the `git log | awk` pipeline would have produced wrong mtimes across the whole restore instead of failing loudly. Moved to its own line. Audited every other script for the same defect — this was the only instance. Independent confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"` disappear now that errexit is actually in effect. Finding 3 — lock-acquire window ------------------------------- A just-seeded directory was unlocked until a later action step, so a concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an optional lock-id and writes the lock marker on every path out of the script, including into the staging tree before its rename, so the directory carries a lock the instant it appears under its final name. The action's acquire step stays (it is idempotent and stamps the LRU marker). Also hardened `prune-cache.sh` to treat a directory with live reader markers as locked. Today no reachable configuration prunes a snapshot — only protected refs publish them and protected refs are excluded from every pass — so this is redundant by policy; it is here so that stops being the reason it is safe. Verification ------------ New selftest scenario 8 races a real seed against a real publish rotation, gating the rotation on the seed's *observed* clone progress so the window is hit deterministically rather than on a fast machine's coin flip. Red-proven against the unguarded scripts, three consecutive runs: ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the snapshot's 493 (was 48805 before the rotation) (15443 / 16986 / 16498) Green after the fix, six consecutive runs, catching the clone mid-walk at ~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and its later sweep; scenario 10 covers an unreadable source failing loudly. `bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63). shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved. Docs: README's republish-safety paragraph replaced with what the code now guarantees, including the bounded disk residual stated explicitly; new `read-grace-seconds` / `reader-stale-seconds` inputs documented in the `cargo-cache-publish` table; the selftest table names the new race. Refs: daniel/gitdan#11, zemyna#911 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
This commit is contained in:
@@ -130,14 +130,49 @@ the staging path into `target-<own>`.
|
|||||||
**A push to a protected branch** has no base to layer over. It builds in its
|
**A push to a protected branch** has no base to layer over. It builds in its
|
||||||
own directory and, if the build goes green, republishes it as
|
own directory and, if the build goes green, republishes it as
|
||||||
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
|
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
|
||||||
one in. Consumers only ever observe a complete snapshot or none at all.
|
one in, then reclaim the old one once nothing is still reading it. Consumers
|
||||||
|
only ever observe a complete snapshot or none at all.
|
||||||
|
|
||||||
**Concurrency.** Two jobs sharing one cache key each stage under their own tag
|
**Concurrency, on the destination.** Two jobs sharing one cache key each stage
|
||||||
and race on one atomic rename; the loser discards its staging copy. There is
|
under their own tag and race on one atomic rename; the loser discards its
|
||||||
no window in which a partially-populated directory is visible under the final
|
staging copy. There is no window in which a partially-populated directory is
|
||||||
name — which means this does not depend on the runner having a single
|
visible under the final name. Two jobs then building in the same directory is
|
||||||
execution slot. Two jobs then building in the same directory is Cargo's own
|
Cargo's own `.cargo-lock` territory, which is what that lock is for.
|
||||||
`.cargo-lock` territory, which is what that lock is for.
|
|
||||||
|
**Concurrency, on the source.** The atomic rename is necessary and not
|
||||||
|
sufficient, because renaming a *truncated* tree publishes a truncated tree
|
||||||
|
atomically. A clone reads its source over many seconds, and a publisher
|
||||||
|
rotating that source unlinks the generation being read — at which point
|
||||||
|
`cp -al` can silently omit a subtree it never saw, and report success. Two
|
||||||
|
mechanisms, both required:
|
||||||
|
|
||||||
|
- **The publisher does not unlink under a reader.** A consumer publishes a
|
||||||
|
`.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
|
||||||
|
the publisher scans for markers *after* its first rename. A consumer holding
|
||||||
|
the old generation therefore published its marker before that scan and
|
||||||
|
cannot be missed, and one that arrives after the scan necessarily resolves
|
||||||
|
to the new generation. The publisher waits for readers to drain
|
||||||
|
(`read-grace-seconds`, default 300) and, if they do not, **defers** the
|
||||||
|
reclamation rather than forcing it — the old generation stays on disk and is
|
||||||
|
swept by a later publish.
|
||||||
|
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
|
||||||
|
exit status, the source directory's inode before and after (a wholesale
|
||||||
|
replacement mid-walk would otherwise splice two generations), and the entry
|
||||||
|
count (the only signal for a subtree unlinked before its parent was listed —
|
||||||
|
there is no error to read). A tree that fails any of the three is deleted
|
||||||
|
and the clone retried; one that fails the last attempt fails the job. A
|
||||||
|
partial tree never reaches the final name.
|
||||||
|
|
||||||
|
**What this does and does not guarantee.** *Correctness* is closed by
|
||||||
|
construction: no combination of publish and seed timing produces a target
|
||||||
|
directory holding part of one generation, and a source that cannot be read
|
||||||
|
consistently fails the job loudly instead of seeding a truncated cache.
|
||||||
|
*Disk reclamation* is bounded, not immediate: a consumer slower than the grace
|
||||||
|
period leaves one extra snapshot generation of directory entries on the volume
|
||||||
|
until the next publish of that snapshot sweeps it. That residual is capped at
|
||||||
|
one deferred generation per publisher ref, and its real cost is close to the
|
||||||
|
inode count rather than the byte count, since the artifacts are hardlinked to
|
||||||
|
whatever cloned them.
|
||||||
|
|
||||||
**Eviction** runs three passes: caches for branches that no longer exist on
|
**Eviction** runs three passes: caches for branches that no longer exist on
|
||||||
origin are removed unconditionally; then, only if free space is under the
|
origin are removed unconditionally; then, only if free space is under the
|
||||||
@@ -194,6 +229,8 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
|
|||||||
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
|
||||||
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
|
||||||
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
|
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
|
||||||
|
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
|
||||||
|
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
|
||||||
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
|
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |
|
||||||
|
|
||||||
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
|
`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
|
||||||
@@ -251,14 +288,17 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
|||||||
| suite | covers |
|
| suite | covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key |
|
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree |
|
||||||
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
|
| `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish |
|
||||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
|
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` |
|
||||||
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
||||||
|
|
||||||
Every suite runs the actual script, not a reimplementation of its logic, and
|
Every suite runs the actual script, not a reimplementation of its logic, and
|
||||||
every fix scenario is paired with a control that reproduces the bug — a
|
every fix scenario is paired with a control that reproduces the bug — a
|
||||||
scenario that passes either way proves nothing.
|
scenario that passes either way proves nothing. The concurrency scenarios race
|
||||||
|
real processes rather than mocking the interleaving, and gate the interfering
|
||||||
|
step on *observed* progress of the step it interferes with, so the window is
|
||||||
|
hit deterministically instead of on a fast machine's coin flip.
|
||||||
|
|
||||||
The action YAML holds no logic beyond wiring; everything testable lives in
|
The action YAML holds no logic beyond wiring; everything testable lives in
|
||||||
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
name: 'Cargo cache (publish)'
|
name: 'Cargo cache (publish)'
|
||||||
description: >-
|
description: >-
|
||||||
Records this run''s build watermark and, on a publisher branch, atomically
|
Records this run's build watermark and, on a publisher branch, atomically
|
||||||
republishes its target directory as the immutable snapshot that other
|
republishes its target directory as the immutable snapshot that other
|
||||||
branches'' caches are hardlink-cloned from.
|
branches' caches are hardlink-cloned from.
|
||||||
author: 'gitdan'
|
author: 'gitdan'
|
||||||
|
|
||||||
inputs:
|
inputs:
|
||||||
@@ -19,9 +19,9 @@ inputs:
|
|||||||
mode:
|
mode:
|
||||||
description: >-
|
description: >-
|
||||||
publish — record the watermark, publish a snapshot if eligible,
|
publish — record the watermark, publish a snapshot if eligible,
|
||||||
release this job''s cache lock (the normal call, after a
|
release this job's cache lock (the normal call, after a
|
||||||
green build).
|
green build).
|
||||||
release-lock — release this job''s cache lock and do nothing else. Use
|
release-lock — release this job's cache lock and do nothing else. Use
|
||||||
in a final `if: always()` step so a failed run does not
|
in a final `if: always()` step so a failed run does not
|
||||||
leave a lock behind for the staleness grace period.
|
leave a lock behind for the staleness grace period.
|
||||||
required: false
|
required: false
|
||||||
@@ -37,9 +37,25 @@ inputs:
|
|||||||
because its ref is not the reference branch even when it targets one.
|
because its ref is not the reference branch even when it targets one.
|
||||||
required: false
|
required: false
|
||||||
default: 'push'
|
default: 'push'
|
||||||
|
read-grace-seconds:
|
||||||
|
description: >-
|
||||||
|
How long the snapshot swap waits for in-flight consumers to finish
|
||||||
|
cloning the generation it is replacing before reclaiming it. On timeout
|
||||||
|
the old generation is LEFT ON DISK and swept by a later publish — the
|
||||||
|
unlink is never forced, because unlinking a tree a consumer is walking
|
||||||
|
is what silently truncates that consumer's clone.
|
||||||
|
required: false
|
||||||
|
default: '300'
|
||||||
|
reader-stale-seconds:
|
||||||
|
description: >-
|
||||||
|
Age past which a consumer's read marker is treated as abandoned by a
|
||||||
|
job the runner killed. Without it one crashed job would pin a snapshot
|
||||||
|
generation on disk permanently.
|
||||||
|
required: false
|
||||||
|
default: '7200'
|
||||||
record-watermark:
|
record-watermark:
|
||||||
description: >-
|
description: >-
|
||||||
Record this run''s HEAD as the build watermark for this target dir.
|
Record this run's HEAD as the build watermark for this target dir.
|
||||||
True for PR runs too, not just publishers: a feature branch accumulates
|
True for PR runs too, not just publishers: a feature branch accumulates
|
||||||
its own build history across several pushes and needs its own watermark.
|
its own build history across several pushes and needs its own watermark.
|
||||||
required: false
|
required: false
|
||||||
@@ -109,6 +125,9 @@ runs:
|
|||||||
|
|
||||||
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
|
- if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }}
|
||||||
shell: bash
|
shell: bash
|
||||||
|
env:
|
||||||
|
CACHE_READ_GRACE_SECONDS: ${{ inputs.read-grace-seconds }}
|
||||||
|
CACHE_READ_STALE_SECONDS: ${{ inputs.reader-stale-seconds }}
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \
|
||||||
|
|||||||
+17
-10
@@ -22,9 +22,9 @@ inputs:
|
|||||||
default: '10'
|
default: '10'
|
||||||
restore-mtimes:
|
restore-mtimes:
|
||||||
description: >-
|
description: >-
|
||||||
Restore every tracked file''s mtime from git history. Requires a
|
Restore every tracked file's mtime from git history. Requires a
|
||||||
full-history checkout (fetch-depth: 0). Set to false only if the build
|
full-history checkout (fetch-depth: 0). Set to false only if the build
|
||||||
does not use Cargo''s mtime-based freshness at all.
|
does not use Cargo's mtime-based freshness at all.
|
||||||
required: false
|
required: false
|
||||||
default: 'true'
|
default: 'true'
|
||||||
prune:
|
prune:
|
||||||
@@ -53,7 +53,7 @@ inputs:
|
|||||||
default: ''
|
default: ''
|
||||||
watermark-file:
|
watermark-file:
|
||||||
description: >-
|
description: >-
|
||||||
Name of this job''s build-watermark file inside the target dir. MUST be
|
Name of this job's build-watermark file inside the target dir. MUST be
|
||||||
distinct per job when two jobs share one cache key. Defaults to
|
distinct per job when two jobs share one cache key. Defaults to
|
||||||
.ci-watermark-<job>-sha.
|
.ci-watermark-<job>-sha.
|
||||||
required: false
|
required: false
|
||||||
@@ -140,7 +140,13 @@ runs:
|
|||||||
# Seeds this ref's target dir from the base's published snapshot. See
|
# Seeds this ref's target dir from the base's published snapshot. See
|
||||||
# scripts/seed-target-dir.sh — the staging-then-atomic-rename is what
|
# scripts/seed-target-dir.sh — the staging-then-atomic-rename is what
|
||||||
# makes concurrent jobs sharing one cache key safe by construction rather
|
# makes concurrent jobs sharing one cache key safe by construction rather
|
||||||
# than by the runner happening to have a single execution slot.
|
# than by the runner happening to have a single execution slot, and the
|
||||||
|
# clone's own consistency check plus the publish side's reader interlock
|
||||||
|
# are what make it safe against the base republishing MID-CLONE.
|
||||||
|
#
|
||||||
|
# The lock id is passed here as well as acquired in the next step: the
|
||||||
|
# seed writes it into the staging tree, so the directory carries a lock
|
||||||
|
# the instant it appears under its final name rather than a step later.
|
||||||
- id: seed
|
- id: seed
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
@@ -151,13 +157,14 @@ runs:
|
|||||||
"${{ steps.resolve.outputs.base-key }}" \
|
"${{ steps.resolve.outputs.base-key }}" \
|
||||||
"${{ inputs.cache-root }}" \
|
"${{ inputs.cache-root }}" \
|
||||||
"${{ github.job }}-${{ github.run_id }}-$$" \
|
"${{ github.job }}-${{ github.run_id }}-$$" \
|
||||||
"${{ inputs.seed-fallback-dir }}"
|
"${{ inputs.seed-fallback-dir }}" \
|
||||||
|
"${{ steps.resolve.outputs.lock-id }}"
|
||||||
|
|
||||||
# Marks the directory as held open, so any job's prune pass (this one
|
# Re-stamps the lock the seed step already wrote (acquiring is idempotent
|
||||||
# included) skips it, and stamps the LRU marker. The marker is touched
|
# — it rewrites the timestamp) and stamps the LRU marker. The marker is
|
||||||
# unconditionally every run: a run that hits the cache for every crate may
|
# touched unconditionally every run: a run that hits the cache for every
|
||||||
# write nothing at all inside the tree, which would make a just-used
|
# crate may write nothing at all inside the tree, which would make a
|
||||||
# directory look stale to the eviction pass.
|
# just-used directory look stale to the eviction pass.
|
||||||
- shell: bash
|
- shell: bash
|
||||||
run: |
|
run: |
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|||||||
+223
-12
@@ -113,22 +113,31 @@ strip_cargo_locks() {
|
|||||||
# only ever run this against a staging directory nothing else can see yet
|
# only ever run this against a staging directory nothing else can see yet
|
||||||
# (see hardlink_clone_into's contract), so the brief window where the path is
|
# (see hardlink_clone_into's contract), so the brief window where the path is
|
||||||
# absent is not observable.
|
# absent is not observable.
|
||||||
|
#
|
||||||
|
# Returns non-zero if the copy or either rename failed. That status is
|
||||||
|
# load-bearing: a failed unshare leaves the staging tree still aliasing its
|
||||||
|
# source, which is the exact corruption `unshare_mutable_paths` exists to
|
||||||
|
# prevent, so it must abort the clone rather than be swallowed.
|
||||||
unshare_subtree() {
|
unshare_subtree() {
|
||||||
local d="$1" tmp
|
local d="$1" tmp
|
||||||
[ -d "$d" ] || return 0
|
[ -d "$d" ] || return 0
|
||||||
tmp="${d}.unshare.$$"
|
tmp="${d}.unshare.$$"
|
||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
cp -a "$d" "$tmp"
|
cp -a "$d" "$tmp" || { rm -rf "$tmp"; return 1; }
|
||||||
rm -rf "$d"
|
rm -rf "$d" || { rm -rf "$tmp"; return 1; }
|
||||||
mv -T "$tmp" "$d"
|
mv -T "$tmp" "$d" || return 1
|
||||||
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
_unshare_files() {
|
_unshare_files() {
|
||||||
# `-links +1` restricts the work to files that are actually shared, which
|
# `-links +1` restricts the work to files that are actually shared, which
|
||||||
# makes this idempotent and near-free on an already-unshared tree.
|
# makes this idempotent and near-free on an already-unshared tree.
|
||||||
|
#
|
||||||
|
# The inner shell propagates a failure of any individual copy-and-rename out
|
||||||
|
# through xargs (which exits 123 if any invocation exits 1-125), so a
|
||||||
|
# partially-unshared tree is reported rather than silently accepted.
|
||||||
find "$@" -links +1 -print0 2>/dev/null |
|
find "$@" -links +1 -print0 2>/dev/null |
|
||||||
xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _
|
xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _
|
||||||
return 0
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# THE load-bearing function of this whole design.
|
# THE load-bearing function of this whole design.
|
||||||
@@ -184,33 +193,235 @@ unshare_mutable_paths() {
|
|||||||
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null)
|
||||||
for d in "${dirs[@]}"; do
|
for d in "${dirs[@]}"; do
|
||||||
[ -n "$d" ] || continue
|
[ -n "$d" ] || continue
|
||||||
unshare_subtree "$d"
|
unshare_subtree "$d" || {
|
||||||
|
echo "::error::unshare_mutable_paths: failed to unshare ${d}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
done
|
done
|
||||||
_unshare_files "$root" -type f -name '*.d'
|
_unshare_files "$root" -type f -name '*.d' || {
|
||||||
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json'
|
echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
_unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || {
|
||||||
|
echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Reader markers: the consume side's half of the seed-vs-republish interlock
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# A hardlink clone reads its source over many seconds. The publish side
|
||||||
|
# rotates a snapshot with two renames and then unlinks the generation it
|
||||||
|
# rotated away — and unlinking entries out from under an in-flight directory
|
||||||
|
# walk is what produces a SILENTLY truncated clone: `cp -al` reports the
|
||||||
|
# entries it manages to stat, and simply never sees a subdirectory that was
|
||||||
|
# unlinked before it read the parent's listing. Exit status alone does not
|
||||||
|
# catch that case.
|
||||||
|
#
|
||||||
|
# So the two sides interlock through a marker file, and the ordering is what
|
||||||
|
# makes it sound rather than probabilistic:
|
||||||
|
#
|
||||||
|
# Consumer: create .reading-<snap>-<tag> -> stat <snap> -> cp -al
|
||||||
|
# Publisher: mv <snap> aside -> mv new into place -> scan for markers
|
||||||
|
# -> unlink the rotated-away generation
|
||||||
|
#
|
||||||
|
# If a consumer's `stat` resolved to the OLD generation, that stat happened
|
||||||
|
# before the publisher's first rename, so its marker — created strictly
|
||||||
|
# earlier still — was already on disk before the publisher's scan, which
|
||||||
|
# happens strictly after that rename. The publisher therefore cannot miss it.
|
||||||
|
# A consumer that creates its marker after the scan necessarily resolves the
|
||||||
|
# path to the NEW generation, which is not the one being unlinked.
|
||||||
|
#
|
||||||
|
# The wait is bounded (CACHE_READ_GRACE_SECONDS). Exceeding it does not force
|
||||||
|
# the unlink: reclamation of that generation is DEFERRED to a later publish
|
||||||
|
# instead. The residual is therefore disk, not correctness.
|
||||||
|
CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}"
|
||||||
|
# A marker older than this belongs to a job the runner killed before it could
|
||||||
|
# clean up. Honouring one forever would let a crashed job pin an entire
|
||||||
|
# snapshot generation on disk permanently.
|
||||||
|
CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}"
|
||||||
|
|
||||||
|
reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; }
|
||||||
|
|
||||||
|
reader_lock_acquire() {
|
||||||
|
date +%s > "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
reader_lock_release() {
|
||||||
|
rm -f "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Prints the number of live readers of <source-name> under <marker-root>, and
|
||||||
|
# sweeps markers past the staleness threshold as it goes.
|
||||||
|
live_reader_count() {
|
||||||
|
local root="$1" name="$2" now marker age n=0
|
||||||
|
now=$(date +%s)
|
||||||
|
for marker in "$root"/.reading-"$name"-*; do
|
||||||
|
[ -e "$marker" ] || continue
|
||||||
|
age=$(( now - $(stat -c '%Y' "$marker" 2>/dev/null || echo "$now") ))
|
||||||
|
if [ "$age" -lt "$CACHE_READ_STALE_SECONDS" ]; then
|
||||||
|
n=$((n + 1))
|
||||||
|
else
|
||||||
|
echo "readers: sweeping stale marker $(basename "$marker") (${age}s old > ${CACHE_READ_STALE_SECONDS}s)" >&2
|
||||||
|
rm -f "$marker" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
printf '%s' "$n"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Blocks until nothing is reading <source-name>, or until the grace period
|
||||||
|
# expires. Returns 0 when drained, 1 on timeout — the caller decides what to
|
||||||
|
# do with a timeout, and in this codebase that decision is always "defer the
|
||||||
|
# unlink", never "unlink anyway".
|
||||||
|
wait_for_readers() {
|
||||||
|
local root="$1" name="$2" grace="${3:-$CACHE_READ_GRACE_SECONDS}" deadline n waited=0
|
||||||
|
deadline=$(( $(date +%s) + grace ))
|
||||||
|
while :; do
|
||||||
|
n=$(live_reader_count "$root" "$name")
|
||||||
|
[ "$n" -eq 0 ] && {
|
||||||
|
[ "$waited" -gt 0 ] && echo "readers: ${name} drained after ${waited}s"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||||
|
echo "readers: ${n} job(s) still reading ${name} after ${grace}s" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
[ "$waited" = 0 ] && echo "readers: waiting for ${n} in-flight clone(s) of ${name} (grace ${grace}s)"
|
||||||
|
sleep 1
|
||||||
|
waited=$((waited + 1))
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The clone itself
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Number of times a torn clone is retried before the caller is failed. A tear
|
||||||
|
# means the source changed identity or lost entries mid-walk, which is a
|
||||||
|
# transient condition by definition — the publisher that caused it has already
|
||||||
|
# put a complete new generation at the same path — so one retry almost always
|
||||||
|
# suffices; the rest are headroom.
|
||||||
|
CACHE_CLONE_ATTEMPTS="${CACHE_CLONE_ATTEMPTS:-4}"
|
||||||
|
|
||||||
|
_tree_entries() {
|
||||||
|
local n
|
||||||
|
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
|
||||||
|
printf '%s' "$n"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
_dir_inode() {
|
||||||
|
stat -c '%i' "$1" 2>/dev/null || printf 'missing'
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
write_cache_lock() {
|
||||||
|
local dir="$1" id="$2"
|
||||||
|
[ -n "$id" ] || return 0
|
||||||
|
[ -d "$dir" ] || return 0
|
||||||
|
date +%s > "${dir}/.ci-lock-${id}" 2>/dev/null || true
|
||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
|
# Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST
|
||||||
# with a single atomic rename.
|
# with a single atomic rename.
|
||||||
#
|
#
|
||||||
|
# hardlink_clone_into <src> <dst> <tag> [lock-id]
|
||||||
|
#
|
||||||
# The staging + rename is what closes the concurrent-seed race structurally
|
# The staging + rename is what closes the concurrent-seed race structurally
|
||||||
# rather than by runner topology: a second job sharing this cache key either
|
# rather than by runner topology: a second job sharing this cache key either
|
||||||
# sees DST absent (and stages its own clone, losing the rename harmlessly) or
|
# sees DST absent (and stages its own clone, losing the rename harmlessly) or
|
||||||
# sees it complete. There is no observable half-populated state, because a
|
# sees it complete. There is no observable half-populated state, because a
|
||||||
# directory rename is atomic and DST is never written through.
|
# directory rename is atomic and DST is never written through.
|
||||||
#
|
#
|
||||||
|
# The rename is necessary but NOT sufficient, and that gap is what this
|
||||||
|
# function's retry loop closes. An atomic rename of a TRUNCATED tree publishes
|
||||||
|
# a truncated tree atomically. Three things can truncate one:
|
||||||
|
#
|
||||||
|
# * `cp -al` failing partway (a source entry vanished after readdir listed
|
||||||
|
# it) — caught by checking its exit status, which is why that status is
|
||||||
|
# read into a variable here rather than left to an ambient `set -e` the
|
||||||
|
# CALL SITES suppress anyway by invoking this function as a condition;
|
||||||
|
# * `cp -al` succeeding while having silently never seen a subtree that was
|
||||||
|
# unlinked before it read the parent's listing — caught only by the entry
|
||||||
|
# count, since there is no error to report;
|
||||||
|
# * the source being replaced wholesale mid-walk, so the clone splices two
|
||||||
|
# generations — caught by comparing the source directory's inode before
|
||||||
|
# and after.
|
||||||
|
#
|
||||||
|
# All three are verified on every attempt and a failing one restarts the
|
||||||
|
# clone; a tree that fails the last attempt is deleted and reported, never
|
||||||
|
# renamed into place. Combined with the reader marker (held across the copy,
|
||||||
|
# which is what stops the publish side unlinking underneath it in the first
|
||||||
|
# place), a partial tree cannot reach DST.
|
||||||
|
#
|
||||||
|
# `lock-id`, when given, writes this job's cache lock INTO the staging tree so
|
||||||
|
# the directory already carries it the instant it appears under its final
|
||||||
|
# name. Acquiring the lock after the rename would leave a freshly seeded
|
||||||
|
# directory momentarily unlocked and therefore evictable by a concurrent job's
|
||||||
|
# prune pass.
|
||||||
|
#
|
||||||
# Returns 0 if this caller's clone won the rename, 1 if another caller got
|
# Returns 0 if this caller's clone won the rename, 1 if another caller got
|
||||||
# there first (the staging copy is discarded; DST is already valid).
|
# there first (the staging copy is discarded; DST is already valid), and 2 if
|
||||||
|
# the source could not be cloned consistently at all.
|
||||||
hardlink_clone_into() {
|
hardlink_clone_into() {
|
||||||
local src="$1" dst="$2" tag="$3" tmp parent
|
local src="$1" dst="$2" tag="$3" lock_id="${4:-}"
|
||||||
|
local parent tmp src_name attempt cp_rc n_before n_after i_before i_after
|
||||||
|
|
||||||
|
if [ ! -d "$src" ]; then
|
||||||
|
echo "::error::clone: source ${src} does not exist" >&2
|
||||||
|
return 2
|
||||||
|
fi
|
||||||
|
|
||||||
parent=$(dirname "$dst")
|
parent=$(dirname "$dst")
|
||||||
|
src_name=$(basename "$src")
|
||||||
tmp="${parent}/.stage-${tag}"
|
tmp="${parent}/.stage-${tag}"
|
||||||
|
|
||||||
|
attempt=1
|
||||||
|
while : ; do
|
||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
cp -al "$src" "$tmp"
|
# Marker first, then the identity read, then the copy — see the ordering
|
||||||
|
# proof in the reader-marker section above; swapping the first two lines
|
||||||
|
# is what would reintroduce the race.
|
||||||
|
reader_lock_acquire "$parent" "$src_name" "$tag"
|
||||||
|
i_before=$(_dir_inode "$src")
|
||||||
|
n_before=$(_tree_entries "$src")
|
||||||
|
cp_rc=0
|
||||||
|
cp -al "$src" "$tmp" || cp_rc=$?
|
||||||
|
n_after=$(_tree_entries "$tmp")
|
||||||
|
i_after=$(_dir_inode "$src")
|
||||||
|
reader_lock_release "$parent" "$src_name" "$tag"
|
||||||
|
|
||||||
|
if [ "$cp_rc" -eq 0 ] && [ "$i_before" != missing ] && [ "$i_before" = "$i_after" ] \
|
||||||
|
&& [ "$n_after" -eq "$n_before" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "::warning::clone: attempt ${attempt}/${CACHE_CLONE_ATTEMPTS} of ${src_name} was torn (cp rc=${cp_rc}, ${n_after}/${n_before} entries, source inode ${i_before} -> ${i_after}) — discarding and retrying" >&2
|
||||||
|
rm -rf "$tmp"
|
||||||
|
if [ "$attempt" -ge "$CACHE_CLONE_ATTEMPTS" ]; then
|
||||||
|
echo "::error::clone: ${src} could not be read consistently in ${CACHE_CLONE_ATTEMPTS} attempts — refusing to publish a partial tree at ${dst}" >&2
|
||||||
|
return 2
|
||||||
|
fi
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
|
||||||
strip_cargo_locks "$tmp"
|
strip_cargo_locks "$tmp"
|
||||||
rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
|
rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true
|
||||||
unshare_mutable_paths "$tmp"
|
if ! unshare_mutable_paths "$tmp"; then
|
||||||
|
echo "::error::clone: could not privately own the mutable paths of ${dst} — discarding the staging tree rather than publishing one that aliases ${src}" >&2
|
||||||
|
rm -rf "$tmp"
|
||||||
|
return 2
|
||||||
|
fi
|
||||||
|
write_cache_lock "$tmp" "$lock_id"
|
||||||
|
|
||||||
if mv -T "$tmp" "$dst" 2>/dev/null; then
|
if mv -T "$tmp" "$dst" 2>/dev/null; then
|
||||||
return 0
|
return 0
|
||||||
fi
|
fi
|
||||||
|
|||||||
+20
-7
@@ -49,12 +49,13 @@
|
|||||||
# the entire benefit this scheme exists to deliver.
|
# the entire benefit this scheme exists to deliver.
|
||||||
#
|
#
|
||||||
# LOCKED — a directory carrying a .ci-lock-* marker younger than
|
# LOCKED — a directory carrying a .ci-lock-* marker younger than
|
||||||
# STALE_LOCK_SECONDS is held open by a running job and is skipped by every
|
# STALE_LOCK_SECONDS is held open by a running job, or named by a live
|
||||||
# pass, however dead and however tight the disk. This is what makes eviction
|
# .reading-<dir>-* marker (a job is hardlink-cloning it this instant), is
|
||||||
# safe on a runner with more than one execution slot. An older marker is
|
# skipped by every pass, however dead and however tight the disk. This is
|
||||||
# treated as abandoned and logged as such, so an actually-still-running job
|
# what makes eviction safe on a runner with more than one execution slot.
|
||||||
# that somehow exceeds the threshold is visible in the log rather than
|
# An older marker is treated as abandoned and logged as such, so an
|
||||||
# silently losing its cache mid-build.
|
# actually-still-running job that somehow exceeds the threshold is visible
|
||||||
|
# in the log rather than silently losing its cache mid-build.
|
||||||
#
|
#
|
||||||
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
|
# Liveness is resolved by `git ls-remote --heads origin`, wrapped in a
|
||||||
# timeout. A directory name cannot be inverted back to a branch name (the
|
# timeout. A directory name cannot be inverted back to a branch name (the
|
||||||
@@ -93,8 +94,20 @@ is_protected() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
is_locked() {
|
is_locked() {
|
||||||
local dir="$1" now lock_file lock_age locked=1
|
local dir="$1" now lock_file lock_age locked=1 readers
|
||||||
now=$(date +%s)
|
now=$(date +%s)
|
||||||
|
# A directory being hardlink-cloned right now carries no .ci-lock-* of its
|
||||||
|
# own — a snapshot has its locks stripped by construction — so the reader
|
||||||
|
# markers are the only signal that unlinking it would truncate somebody's
|
||||||
|
# in-flight clone. Today no reachable configuration prunes a snapshot (only
|
||||||
|
# protected refs publish them, and protected refs are excluded from every
|
||||||
|
# pass), which makes this guard redundant *by policy*. It is here so that
|
||||||
|
# stops being the reason it is safe.
|
||||||
|
readers=$(live_reader_count "$ROOT" "$(basename "$dir")")
|
||||||
|
if [ "$readers" -gt 0 ]; then
|
||||||
|
echo " $(basename "$dir"): ${readers} job(s) currently cloning it — not a candidate"
|
||||||
|
locked=0
|
||||||
|
fi
|
||||||
for lock_file in "$dir"/.ci-lock-*; do
|
for lock_file in "$dir"/.ci-lock-*; do
|
||||||
[ -e "$lock_file" ] || continue
|
[ -e "$lock_file" ] || continue
|
||||||
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
|
lock_age=$(( now - $(stat -c '%Y' "$lock_file") ))
|
||||||
|
|||||||
@@ -23,11 +23,23 @@
|
|||||||
# own cold-start path — a safe degrade that self-heals on its next run, not
|
# own cold-start path — a safe degrade that self-heals on its next run, not
|
||||||
# corruption.
|
# corruption.
|
||||||
#
|
#
|
||||||
# `rm -rf` on the old snapshot removes directory entries only. Any consumer
|
# `rm -rf` on the old snapshot removes directory entries only, so a consumer
|
||||||
# that already hardlink-cloned from it keeps every inode alive through its own
|
# that has ALREADY FINISHED cloning from it keeps every inode alive through
|
||||||
# links, so a republish never pulls data out from under a running job — it
|
# its own links. That is the easy half, and on its own it is not enough: a
|
||||||
# just stops new consumers from seeing the old generation. Disk is reclaimed
|
# consumer still WALKING the old generation has its entries unlinked out from
|
||||||
# when the last clone referencing those inodes is itself evicted.
|
# under it, and `cp -al` does not report a subtree that was removed before it
|
||||||
|
# read the parent's listing. That is a silently truncated clone — the failure
|
||||||
|
# mode this script's own selftest (scenario 8) reproduces against the
|
||||||
|
# unguarded version.
|
||||||
|
#
|
||||||
|
# So the unlink is interlocked with the consume side rather than
|
||||||
|
# unconditional: after the swap, this script waits for every in-flight reader
|
||||||
|
# of this snapshot to drain (see the reader-marker ordering proof in
|
||||||
|
# cache-lib.sh) and only then reclaims the rotated-away generation. If the
|
||||||
|
# grace period expires first, reclamation is DEFERRED — the directory is left
|
||||||
|
# under `.publish-old-<key>-<tag>` and swept by a later publish once no reader
|
||||||
|
# holds it. The residual of a very slow consumer is therefore one extra
|
||||||
|
# generation of directory entries on disk, never a torn clone.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||||
|
|
||||||
@@ -36,7 +48,11 @@ ROOT="${2:?}"; TAG="${3:?}"
|
|||||||
|
|
||||||
SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
|
SRC=$(target_dir_for "$ROOT" "$OWN_KEY")
|
||||||
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
|
DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY")
|
||||||
OLD="${ROOT}/.publish-old-${TAG}"
|
# Keyed by cache key as well as tag, so a deferred generation can be matched
|
||||||
|
# back to the snapshot whose readers must drain before it is safe to reclaim.
|
||||||
|
OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}"
|
||||||
|
SNAP_NAME=$(basename "$DST")
|
||||||
|
GRACE="${CACHE_READ_GRACE_SECONDS}"
|
||||||
|
|
||||||
if [ ! -d "$SRC" ]; then
|
if [ ! -d "$SRC" ]; then
|
||||||
echo "publish: no target dir at ${SRC} — nothing to snapshot"
|
echo "publish: no target dir at ${SRC} — nothing to snapshot"
|
||||||
@@ -47,6 +63,21 @@ fi
|
|||||||
# concurrently running job's staging directory is never touched.
|
# concurrently running job's staging directory is never touched.
|
||||||
rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
|
rm -rf "${ROOT}/.stage-${TAG}" "$OLD"
|
||||||
|
|
||||||
|
# Deferred reclamation from an earlier publish of THIS snapshot whose readers
|
||||||
|
# had not drained in time. Safe to sweep now only if nothing is reading the
|
||||||
|
# snapshot at all: a reader holds the snapshot path, not the deferred name, so
|
||||||
|
# "no readers of snapshot-<key>" is the condition that makes every deferred
|
||||||
|
# generation of it unreachable. Over-conservative by design — a reader of the
|
||||||
|
# CURRENT generation also defers the sweep to the next publish, which costs a
|
||||||
|
# directory listing, not correctness.
|
||||||
|
if [ "$(live_reader_count "$ROOT" "$SNAP_NAME")" -eq 0 ]; then
|
||||||
|
for stale_old in "${ROOT}/.publish-old-${OWN_KEY}-"*; do
|
||||||
|
[ -d "$stale_old" ] || continue
|
||||||
|
echo "publish: reclaiming deferred snapshot generation $(basename "$stale_old")"
|
||||||
|
rm -rf "$stale_old"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
start=$(date +%s)
|
start=$(date +%s)
|
||||||
# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
|
# The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy
|
||||||
# of every file Cargo rewrites in place (unshare_mutable_paths, called inside
|
# of every file Cargo rewrites in place (unshare_mutable_paths, called inside
|
||||||
@@ -63,7 +94,19 @@ hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || {
|
|||||||
|
|
||||||
if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
|
if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi
|
||||||
mv -T "$TMP_DST" "$DST"
|
mv -T "$TMP_DST" "$DST"
|
||||||
|
|
||||||
|
# The scan below happens strictly after the rename above, which is what makes
|
||||||
|
# it impossible for a consumer holding the OLD generation to be missed: such a
|
||||||
|
# consumer resolved the path before that rename, and published its marker
|
||||||
|
# before that. See cache-lib.sh's reader-marker section.
|
||||||
|
if [ -d "$OLD" ]; then
|
||||||
|
if wait_for_readers "$ROOT" "$SNAP_NAME" "$GRACE"; then
|
||||||
rm -rf "$OLD"
|
rm -rf "$OLD"
|
||||||
|
else
|
||||||
|
echo "::warning::publish: a consumer is still cloning the previous ${SNAP_NAME} after ${GRACE}s — deferring reclamation of $(basename "$OLD") rather than unlinking a tree being read"
|
||||||
|
summary_line "- deferred reclaiming the previous \`${SNAP_NAME}\` generation (a consumer is still cloning it); it will be swept by a later publish"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
|
echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s"
|
||||||
summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"
|
summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)"
|
||||||
|
|||||||
@@ -170,7 +170,8 @@
|
|||||||
# the next run's diff base stays at the last GREEN build — over-inclusive
|
# the next run's diff base stays at the last GREEN build — over-inclusive
|
||||||
# (it may re-stamp files that a failed run partially rebuilt anyway) but
|
# (it may re-stamp files that a failed run partially rebuilt anyway) but
|
||||||
# never under-inclusive, which is the only direction that matters for
|
# never under-inclusive, which is the only direction that matters for
|
||||||
# soundness.set -euo pipefail
|
# soundness.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
repo_root=$(git rev-parse --show-toplevel)
|
repo_root=$(git rev-parse --show-toplevel)
|
||||||
cd "$repo_root"
|
cd "$repo_root"
|
||||||
|
|||||||
@@ -29,6 +29,21 @@
|
|||||||
# at no point is a partially-populated directory visible under the final
|
# at no point is a partially-populated directory visible under the final
|
||||||
# name. This is the property that replaces "the runner only has one job
|
# name. This is the property that replaces "the runner only has one job
|
||||||
# slot" with an actual guarantee.
|
# slot" with an actual guarantee.
|
||||||
|
# 8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the
|
||||||
|
# one that actually mattered: a consumer hardlink-cloning a snapshot
|
||||||
|
# while the publisher of that snapshot rotates it and unlinks the
|
||||||
|
# generation being read. Two seeds racing on a DESTINATION is a different
|
||||||
|
# race from a seed racing a publisher on its SOURCE, and only the second
|
||||||
|
# one can truncate a tree. Against the unguarded version this scenario
|
||||||
|
# reproduces a silent partial clone reported as success — 20,328 of
|
||||||
|
# 48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot.
|
||||||
|
# 9. DEFERRED RECLAMATION — when a consumer is STILL reading after the grace
|
||||||
|
# period, the publisher leaves the rotated-away generation on disk rather
|
||||||
|
# than unlinking a tree under an in-flight walk, and a later publish
|
||||||
|
# sweeps it once the reader is gone. The residual is disk, not a torn
|
||||||
|
# clone.
|
||||||
|
# 10. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct
|
||||||
|
# status instead of renaming whatever it managed to produce into place.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
. "$script_dir/cache-lib.sh"
|
. "$script_dir/cache-lib.sh"
|
||||||
@@ -56,7 +71,32 @@ make_tree() {
|
|||||||
: > "$d/debug/.cargo-lock"
|
: > "$d/debug/.cargo-lock"
|
||||||
}
|
}
|
||||||
|
|
||||||
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; }
|
||||||
|
|
||||||
|
# Always succeeds and always prints a number: an absent directory is "0
|
||||||
|
# entries so far", which is the normal state at the top of the progress poll
|
||||||
|
# below, not an error worth aborting the suite over.
|
||||||
|
tree_entries() {
|
||||||
|
local n
|
||||||
|
n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0
|
||||||
|
printf '%s\n' "$n"
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# A tree wide enough that a hardlink clone of it takes long enough to be
|
||||||
|
# caught mid-walk. The race under test is a real interleaving, not a mocked
|
||||||
|
# one, so the fixture has to be big enough for the window to exist: a
|
||||||
|
# four-file tree clones in microseconds and no scheduling could ever land
|
||||||
|
# inside it. Built by cloning one small template directory N times, which is N
|
||||||
|
# forks rather than N*M file creations.
|
||||||
|
make_wide_tree() {
|
||||||
|
local d="$1" marker="$2" ndirs="$3" i
|
||||||
|
mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x"
|
||||||
|
for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done
|
||||||
|
for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done
|
||||||
|
rm -rf "$d/debug/deps/.tmpl"
|
||||||
|
echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x"
|
||||||
|
}
|
||||||
|
|
||||||
BASE_KEY=$(cache_key dev)
|
BASE_KEY=$(cache_key dev)
|
||||||
OWN_KEY=$(cache_key feat/thing)
|
OWN_KEY=$(cache_key feat/thing)
|
||||||
@@ -127,5 +167,96 @@ if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then
|
|||||||
fi
|
fi
|
||||||
ok "neither racing job fell through to a cold start"
|
ok "neither racing job fell through to a cold start"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 8: seeding while the base republishes the snapshot underneath it ==="
|
||||||
|
ROT=$(cache_key feat/rotate)
|
||||||
|
rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY"
|
||||||
|
# Generation 1 is wide (the consumer will still be walking it when the swap
|
||||||
|
# happens); the generation replacing it is small, so the publisher's own
|
||||||
|
# staging clone does not itself outlast the consumer's.
|
||||||
|
make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800
|
||||||
|
make_wide_tree "$root/target-$BASE_KEY" gen2 8
|
||||||
|
gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
||||||
|
|
||||||
|
( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) &
|
||||||
|
seed_pid=$!
|
||||||
|
|
||||||
|
# Rotate only once the clone is demonstrably mid-walk. Gating on observed
|
||||||
|
# progress rather than on a sleep is what makes the interleaving reproducible
|
||||||
|
# instead of a coin flip that passes on a fast machine for the wrong reason.
|
||||||
|
threshold=$(( gen1_entries / 5 ))
|
||||||
|
progress=0
|
||||||
|
deadline=$(( $(date +%s) + 60 ))
|
||||||
|
while :; do
|
||||||
|
progress=$(tree_entries "$root/.stage-jobRot")
|
||||||
|
if [ "$progress" -ge "$threshold" ]; then break; fi
|
||||||
|
if ! kill -0 "$seed_pid" 2>/dev/null; then
|
||||||
|
fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)"
|
||||||
|
fi
|
||||||
|
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||||
|
fail "the staging clone never reached ${threshold} of ${gen1_entries} entries"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries"
|
||||||
|
|
||||||
|
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \
|
||||||
|
|| { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; }
|
||||||
|
wait "$seed_pid"
|
||||||
|
|
||||||
|
rot_dir="$root/target-$ROT"
|
||||||
|
[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; }
|
||||||
|
ok "the seed completed"
|
||||||
|
# THE assertion. Before the guard, this is where it failed: the seed reported
|
||||||
|
# `cloned in 1s` and exit 0 while target-<rot> held less than half the entries
|
||||||
|
# of the snapshot it claimed to have cloned. Comparing against the snapshot as
|
||||||
|
# it stands NOW is the right bar either way — a clone that raced the rotation
|
||||||
|
# must end up holding one complete generation, and a consumer caught mid-walk
|
||||||
|
# re-reads, so that generation is the new one.
|
||||||
|
snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY")
|
||||||
|
rot_entries=$(tree_entries "$rot_dir")
|
||||||
|
[ "$rot_entries" -eq "$snap_entries" ] \
|
||||||
|
|| fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)"
|
||||||
|
ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)"
|
||||||
|
assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two"
|
||||||
|
grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read"
|
||||||
|
ok "the torn read was detected and reported, not swallowed"
|
||||||
|
[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \
|
||||||
|
|| fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)"
|
||||||
|
ok "no staging, reader-marker or deferred-generation scratch left behind"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 9: a reader that outlasts the grace period defers reclamation ==="
|
||||||
|
# A synthetic reader marker stands in for a consumer whose clone is slower
|
||||||
|
# than the grace period. Driving that with a real slow consumer would make the
|
||||||
|
# test's runtime the thing under test; the marker is the whole contract
|
||||||
|
# between the two sides, so holding one IS being a reader.
|
||||||
|
SNAP_NAME="snapshot-$BASE_KEY"
|
||||||
|
date +%s > "$root/.reading-${SNAP_NAME}-slowpoke"
|
||||||
|
make_wide_tree "$root/target-$BASE_KEY" gen3 4
|
||||||
|
CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubDefer > "$scratch/logDefer" 2>&1 \
|
||||||
|
|| { tail -40 "$scratch/logDefer"; fail "publish-snapshot.sh exited non-zero"; }
|
||||||
|
assert_content "$root/$SNAP_NAME/debug/.fingerprint/x/dep-lib-x" gen3 "the new generation was published regardless"
|
||||||
|
deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${BASE_KEY}-*" -print -quit)
|
||||||
|
[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it"
|
||||||
|
ok "the rotated-away generation was left on disk rather than unlinked under a reader"
|
||||||
|
grep -q 'deferring reclamation' "$scratch/logDefer" || fail "the deferral was not reported"
|
||||||
|
ok "the deferral is reported as a warning, not silent"
|
||||||
|
|
||||||
|
rm -f "$root/.reading-${SNAP_NAME}-slowpoke"
|
||||||
|
make_wide_tree "$root/target-$BASE_KEY" gen4 4
|
||||||
|
bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubSweep > "$scratch/logSweep" 2>&1 \
|
||||||
|
|| { tail -40 "$scratch/logSweep"; fail "publish-snapshot.sh exited non-zero"; }
|
||||||
|
[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \
|
||||||
|
|| fail "the deferred generation was never reclaimed"
|
||||||
|
ok "a later publish reclaims the deferred generation once no reader holds it"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 10: a source that cannot be read fails loudly ==="
|
||||||
|
rc=0
|
||||||
|
hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$?
|
||||||
|
[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}"
|
||||||
|
ok "an unreadable source returns the distinct hard-failure status"
|
||||||
|
assert_absent "$root/target-nosuch" "nothing was renamed into place"
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
||||||
|
|||||||
+39
-10
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Consume side: make this run's own target dir exist, warm, and private.
|
# Consume side: make this run's own target dir exist, warm, and private.
|
||||||
#
|
#
|
||||||
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir]
|
# Usage: seed-target-dir.sh <own-key> <base-key> <cache-root> <tag> [fallback-dir] [lock-id]
|
||||||
# own-key cache key for this run's own ref
|
# own-key cache key for this run's own ref
|
||||||
# base-key cache key for the ref to layer over ("" for a run whose own
|
# base-key cache key for the ref to layer over ("" for a run whose own
|
||||||
# ref IS a reference branch)
|
# ref IS a reference branch)
|
||||||
@@ -10,6 +10,14 @@
|
|||||||
# directory so two concurrent jobs can never collide on it
|
# directory so two concurrent jobs can never collide on it
|
||||||
# fallback-dir optional absolute path to seed from when no snapshot exists
|
# fallback-dir optional absolute path to seed from when no snapshot exists
|
||||||
# (a legacy flat cache dir during a migration, typically)
|
# (a legacy flat cache dir during a migration, typically)
|
||||||
|
# lock-id optional cache-lock id. When given, the lock marker is
|
||||||
|
# written on EVERY path out of this script — including into
|
||||||
|
# the staging tree before its rename — so the directory is
|
||||||
|
# never observable under its final name without a lock. The
|
||||||
|
# action acquires the lock in a later step too; that step is
|
||||||
|
# idempotent, and this closes the window before it runs, where
|
||||||
|
# a concurrent job's prune pass could evict a directory that
|
||||||
|
# exists but is not yet held.
|
||||||
#
|
#
|
||||||
# The design in one paragraph: a PR branch's first run hardlink-clones the
|
# The design in one paragraph: a PR branch's first run hardlink-clones the
|
||||||
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
|
# base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs
|
||||||
@@ -27,17 +35,18 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh"
|
||||||
|
|
||||||
if [ $# -lt 4 ] || [ $# -gt 5 ]; then
|
if [ $# -lt 4 ] || [ $# -gt 6 ]; then
|
||||||
echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2
|
echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"
|
OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}"
|
||||||
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
|
OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY")
|
||||||
|
|
||||||
mkdir -p "$ROOT"
|
mkdir -p "$ROOT"
|
||||||
|
|
||||||
if [ -d "$OWN_DIR" ]; then
|
if [ -d "$OWN_DIR" ]; then
|
||||||
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|
||||||
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
|
echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)"
|
||||||
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
|
echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||||
exit 0
|
exit 0
|
||||||
@@ -62,20 +71,40 @@ for entry in "${CANDIDATES[@]}"; do
|
|||||||
[ -d "$src" ] || continue
|
[ -d "$src" ] || continue
|
||||||
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
|
echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}"
|
||||||
start=$(date +%s)
|
start=$(date +%s)
|
||||||
if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then
|
# The status is captured and dispatched on explicitly. Calling this as a
|
||||||
|
# bare `if` condition — which is what this script used to do — suppresses
|
||||||
|
# `set -e` for the whole call, so a hard clone failure could not abort the
|
||||||
|
# seed even in principle; the three outcomes are genuinely distinct and each
|
||||||
|
# needs its own handling.
|
||||||
|
clone_rc=0
|
||||||
|
hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$?
|
||||||
|
case "$clone_rc" in
|
||||||
|
0)
|
||||||
echo "seed: cloned in $(( $(date +%s) - start ))s"
|
echo "seed: cloned in $(( $(date +%s) - start ))s"
|
||||||
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
|
echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||||
else
|
;;
|
||||||
|
1)
|
||||||
# Another job sharing this cache key won the rename while we were
|
# Another job sharing this cache key won the rename while we were
|
||||||
# cloning. Its directory is complete (the rename is the publish step), so
|
# cloning. Its directory is complete (the rename is the publish step),
|
||||||
# there is nothing to do but use it — and nothing was ever observable in
|
# so there is nothing to do but use it — and nothing was ever
|
||||||
# a half-seeded state.
|
# observable in a half-seeded state.
|
||||||
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|
||||||
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
|
echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs"
|
||||||
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
|
echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||||
fi
|
;;
|
||||||
|
*)
|
||||||
|
# A source that could not be read consistently. Failing the job is the
|
||||||
|
# only safe answer: the alternative that used to happen here was
|
||||||
|
# seeding a truncated tree and reporting success, which hands Cargo a
|
||||||
|
# directory whose fingerprints and artifacts disagree.
|
||||||
|
echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
exit 0
|
exit 0
|
||||||
done
|
done
|
||||||
|
|
||||||
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
|
echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold"
|
||||||
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
|
echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}"
|
||||||
mkdir -p "$OWN_DIR"
|
mkdir -p "$OWN_DIR"
|
||||||
|
write_cache_lock "$OWN_DIR" "$LOCK_ID"
|
||||||
|
|||||||
Reference in New Issue
Block a user