diff --git a/README.md b/README.md index badbcf3..58e1bc4 100644 --- a/README.md +++ b/README.md @@ -130,14 +130,49 @@ the staging path into `target-`. **A push to a protected branch** has no base to layer over. It builds in its own directory and, if the build goes green, republishes it as `snapshot-`: stage a clone, rename the old snapshot aside, rename the new -one in. Consumers only ever observe a complete snapshot or none at all. +one in, then reclaim the old one once nothing is still reading it. Consumers +only ever observe a complete snapshot or none at all. -**Concurrency.** Two jobs sharing one cache key each stage under their own tag -and race on one atomic rename; the loser discards its staging copy. There is -no window in which a partially-populated directory is visible under the final -name — which means this does not depend on the runner having a single -execution slot. Two jobs then building in the same directory is Cargo's own -`.cargo-lock` territory, which is what that lock is for. +**Concurrency, on the destination.** Two jobs sharing one cache key each stage +under their own tag and race on one atomic rename; the loser discards its +staging copy. There is no window in which a partially-populated directory is +visible under the final name. Two jobs then building in the same directory is +Cargo's own `.cargo-lock` territory, which is what that lock is for. + +**Concurrency, on the source.** The atomic rename is necessary and not +sufficient, because renaming a *truncated* tree publishes a truncated tree +atomically. A clone reads its source over many seconds, and a publisher +rotating that source unlinks the generation being read — at which point +`cp -al` can silently omit a subtree it never saw, and report success. Two +mechanisms, both required: + +- **The publisher does not unlink under a reader.** A consumer publishes a + `.reading--` marker *before* it resolves the snapshot path; + the publisher scans for markers *after* its first rename. A consumer holding + the old generation therefore published its marker before that scan and + cannot be missed, and one that arrives after the scan necessarily resolves + to the new generation. The publisher waits for readers to drain + (`read-grace-seconds`, default 300) and, if they do not, **defers** the + reclamation rather than forcing it — the old generation stays on disk and is + swept by a later publish. +- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s + exit status, the source directory's inode before and after (a wholesale + replacement mid-walk would otherwise splice two generations), and the entry + count (the only signal for a subtree unlinked before its parent was listed — + there is no error to read). A tree that fails any of the three is deleted + and the clone retried; one that fails the last attempt fails the job. A + partial tree never reaches the final name. + +**What this does and does not guarantee.** *Correctness* is closed by +construction: no combination of publish and seed timing produces a target +directory holding part of one generation, and a source that cannot be read +consistently fails the job loudly instead of seeding a truncated cache. +*Disk reclamation* is bounded, not immediate: a consumer slower than the grace +period leaves one extra snapshot generation of directory entries on the volume +until the next publish of that snapshot sweeps it. That residual is capped at +one deferred generation per publisher ref, and its real cost is close to the +inode count rather than the byte count, since the artifacts are hardlinked to +whatever cloned them. **Eviction** runs three passes: caches for branches that no longer exist on origin are removed unconditionally; then, only if free space is under the @@ -194,6 +229,8 @@ Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`, | `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step | | `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` | | `publish-on-events` | `push` | events on which a protected ref actually publishes | +| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced | +| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job | | `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) | `publish-on-events` defaults to `push` on purpose: a `pull_request` run from @@ -251,14 +288,17 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler | suite | covers | |---|---| | `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. | -| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, and two jobs racing on one cache key | +| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree | | `publish-snapshot-selftest.sh` | the atomic swap, and that a live consumer survives a republish | | `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear — against a real scratch `origin` | | `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. | Every suite runs the actual script, not a reimplementation of its logic, and every fix scenario is paired with a control that reproduces the bug — a -scenario that passes either way proves nothing. +scenario that passes either way proves nothing. The concurrency scenarios race +real processes rather than mocking the interleaving, and gate the interfering +step on *observed* progress of the step it interferes with, so the window is +hit deterministically instead of on a fast machine's coin flip. The action YAML holds no logic beyond wiring; everything testable lives in `scripts/`. A composite action needs `shell: bash` on every `run:` step, and diff --git a/cargo-cache-publish/action.yml b/cargo-cache-publish/action.yml index ed42f2c..86729ed 100644 --- a/cargo-cache-publish/action.yml +++ b/cargo-cache-publish/action.yml @@ -1,8 +1,8 @@ name: 'Cargo cache (publish)' description: >- - Records this run''s build watermark and, on a publisher branch, atomically + Records this run's build watermark and, on a publisher branch, atomically republishes its target directory as the immutable snapshot that other - branches'' caches are hardlink-cloned from. + branches' caches are hardlink-cloned from. author: 'gitdan' inputs: @@ -19,9 +19,9 @@ inputs: mode: description: >- publish — record the watermark, publish a snapshot if eligible, - release this job''s cache lock (the normal call, after a + release this job's cache lock (the normal call, after a green build). - release-lock — release this job''s cache lock and do nothing else. Use + release-lock — release this job's cache lock and do nothing else. Use in a final `if: always()` step so a failed run does not leave a lock behind for the staleness grace period. required: false @@ -37,9 +37,25 @@ inputs: because its ref is not the reference branch even when it targets one. required: false default: 'push' + read-grace-seconds: + description: >- + How long the snapshot swap waits for in-flight consumers to finish + cloning the generation it is replacing before reclaiming it. On timeout + the old generation is LEFT ON DISK and swept by a later publish — the + unlink is never forced, because unlinking a tree a consumer is walking + is what silently truncates that consumer's clone. + required: false + default: '300' + reader-stale-seconds: + description: >- + Age past which a consumer's read marker is treated as abandoned by a + job the runner killed. Without it one crashed job would pin a snapshot + generation on disk permanently. + required: false + default: '7200' record-watermark: description: >- - Record this run''s HEAD as the build watermark for this target dir. + Record this run's HEAD as the build watermark for this target dir. True for PR runs too, not just publishers: a feature branch accumulates its own build history across several pushes and needs its own watermark. required: false @@ -109,6 +125,9 @@ runs: - if: ${{ inputs.mode == 'publish' && steps.resolve.outputs.publish == 'yes' }} shell: bash + env: + CACHE_READ_GRACE_SECONDS: ${{ inputs.read-grace-seconds }} + CACHE_READ_STALE_SECONDS: ${{ inputs.reader-stale-seconds }} run: | set -euo pipefail bash "${CARGO_CACHE_SCRIPTS}/publish-snapshot.sh" \ diff --git a/cargo-cache/action.yml b/cargo-cache/action.yml index 6066693..a7e3d97 100644 --- a/cargo-cache/action.yml +++ b/cargo-cache/action.yml @@ -22,9 +22,9 @@ inputs: default: '10' restore-mtimes: description: >- - Restore every tracked file''s mtime from git history. Requires a + Restore every tracked file's mtime from git history. Requires a full-history checkout (fetch-depth: 0). Set to false only if the build - does not use Cargo''s mtime-based freshness at all. + does not use Cargo's mtime-based freshness at all. required: false default: 'true' prune: @@ -53,7 +53,7 @@ inputs: default: '' watermark-file: description: >- - Name of this job''s build-watermark file inside the target dir. MUST be + Name of this job's build-watermark file inside the target dir. MUST be distinct per job when two jobs share one cache key. Defaults to .ci-watermark--sha. required: false @@ -140,7 +140,13 @@ runs: # Seeds this ref's target dir from the base's published snapshot. See # scripts/seed-target-dir.sh — the staging-then-atomic-rename is what # makes concurrent jobs sharing one cache key safe by construction rather - # than by the runner happening to have a single execution slot. + # than by the runner happening to have a single execution slot, and the + # clone's own consistency check plus the publish side's reader interlock + # are what make it safe against the base republishing MID-CLONE. + # + # The lock id is passed here as well as acquired in the next step: the + # seed writes it into the staging tree, so the directory carries a lock + # the instant it appears under its final name rather than a step later. - id: seed shell: bash run: | @@ -151,13 +157,14 @@ runs: "${{ steps.resolve.outputs.base-key }}" \ "${{ inputs.cache-root }}" \ "${{ github.job }}-${{ github.run_id }}-$$" \ - "${{ inputs.seed-fallback-dir }}" + "${{ inputs.seed-fallback-dir }}" \ + "${{ steps.resolve.outputs.lock-id }}" - # Marks the directory as held open, so any job's prune pass (this one - # included) skips it, and stamps the LRU marker. The marker is touched - # unconditionally every run: a run that hits the cache for every crate may - # write nothing at all inside the tree, which would make a just-used - # directory look stale to the eviction pass. + # Re-stamps the lock the seed step already wrote (acquiring is idempotent + # — it rewrites the timestamp) and stamps the LRU marker. The marker is + # touched unconditionally every run: a run that hits the cache for every + # crate may write nothing at all inside the tree, which would make a + # just-used directory look stale to the eviction pass. - shell: bash run: | set -euo pipefail diff --git a/scripts/cache-lib.sh b/scripts/cache-lib.sh index 569e099..33a9288 100755 --- a/scripts/cache-lib.sh +++ b/scripts/cache-lib.sh @@ -113,22 +113,31 @@ strip_cargo_locks() { # only ever run this against a staging directory nothing else can see yet # (see hardlink_clone_into's contract), so the brief window where the path is # absent is not observable. +# +# Returns non-zero if the copy or either rename failed. That status is +# load-bearing: a failed unshare leaves the staging tree still aliasing its +# source, which is the exact corruption `unshare_mutable_paths` exists to +# prevent, so it must abort the clone rather than be swallowed. unshare_subtree() { local d="$1" tmp [ -d "$d" ] || return 0 tmp="${d}.unshare.$$" rm -rf "$tmp" - cp -a "$d" "$tmp" - rm -rf "$d" - mv -T "$tmp" "$d" + cp -a "$d" "$tmp" || { rm -rf "$tmp"; return 1; } + rm -rf "$d" || { rm -rf "$tmp"; return 1; } + mv -T "$tmp" "$d" || return 1 + return 0 } _unshare_files() { # `-links +1` restricts the work to files that are actually shared, which # makes this idempotent and near-free on an already-unshared tree. + # + # The inner shell propagates a failure of any individual copy-and-rename out + # through xargs (which exits 123 if any invocation exits 1-125), so a + # partially-unshared tree is reported rather than silently accepted. find "$@" -links +1 -print0 2>/dev/null | - xargs -0 -r -n 64 bash -c 'for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f"; done' _ - return 0 + xargs -0 -r -n 64 bash -c 'rc=0; for f; do cp -p -- "$f" "$f.unshare.$$" && mv -f -- "$f.unshare.$$" "$f" || rc=1; done; exit $rc' _ } # THE load-bearing function of this whole design. @@ -184,33 +193,235 @@ unshare_mutable_paths() { mapfile -t dirs < <(find "$root" -type d \( -name .fingerprint -o -name build \) -prune -print 2>/dev/null) for d in "${dirs[@]}"; do [ -n "$d" ] || continue - unshare_subtree "$d" + unshare_subtree "$d" || { + echo "::error::unshare_mutable_paths: failed to unshare ${d}" >&2 + return 1 + } done - _unshare_files "$root" -type f -name '*.d' - _unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' + _unshare_files "$root" -type f -name '*.d' || { + echo "::error::unshare_mutable_paths: failed to unshare dep-info files under ${root}" >&2 + return 1 + } + _unshare_files "$root" -maxdepth 3 -type f -name '.rustc_info.json' || { + echo "::error::unshare_mutable_paths: failed to unshare .rustc_info.json under ${root}" >&2 + return 1 + } + return 0 +} + +# --------------------------------------------------------------------------- +# Reader markers: the consume side's half of the seed-vs-republish interlock +# --------------------------------------------------------------------------- +# +# A hardlink clone reads its source over many seconds. The publish side +# rotates a snapshot with two renames and then unlinks the generation it +# rotated away — and unlinking entries out from under an in-flight directory +# walk is what produces a SILENTLY truncated clone: `cp -al` reports the +# entries it manages to stat, and simply never sees a subdirectory that was +# unlinked before it read the parent's listing. Exit status alone does not +# catch that case. +# +# So the two sides interlock through a marker file, and the ordering is what +# makes it sound rather than probabilistic: +# +# Consumer: create .reading-- -> stat -> cp -al +# Publisher: mv aside -> mv new into place -> scan for markers +# -> unlink the rotated-away generation +# +# If a consumer's `stat` resolved to the OLD generation, that stat happened +# before the publisher's first rename, so its marker — created strictly +# earlier still — was already on disk before the publisher's scan, which +# happens strictly after that rename. The publisher therefore cannot miss it. +# A consumer that creates its marker after the scan necessarily resolves the +# path to the NEW generation, which is not the one being unlinked. +# +# The wait is bounded (CACHE_READ_GRACE_SECONDS). Exceeding it does not force +# the unlink: reclamation of that generation is DEFERRED to a later publish +# instead. The residual is therefore disk, not correctness. +CACHE_READ_GRACE_SECONDS="${CACHE_READ_GRACE_SECONDS:-300}" +# A marker older than this belongs to a job the runner killed before it could +# clean up. Honouring one forever would let a crashed job pin an entire +# snapshot generation on disk permanently. +CACHE_READ_STALE_SECONDS="${CACHE_READ_STALE_SECONDS:-7200}" + +reader_marker_path() { printf '%s/.reading-%s-%s' "$1" "$2" "$3"; } + +reader_lock_acquire() { + date +%s > "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true + return 0 +} + +reader_lock_release() { + rm -f "$(reader_marker_path "$1" "$2" "$3")" 2>/dev/null || true + return 0 +} + +# Prints the number of live readers of under , and +# sweeps markers past the staleness threshold as it goes. +live_reader_count() { + local root="$1" name="$2" now marker age n=0 + now=$(date +%s) + for marker in "$root"/.reading-"$name"-*; do + [ -e "$marker" ] || continue + age=$(( now - $(stat -c '%Y' "$marker" 2>/dev/null || echo "$now") )) + if [ "$age" -lt "$CACHE_READ_STALE_SECONDS" ]; then + n=$((n + 1)) + else + echo "readers: sweeping stale marker $(basename "$marker") (${age}s old > ${CACHE_READ_STALE_SECONDS}s)" >&2 + rm -f "$marker" 2>/dev/null || true + fi + done + printf '%s' "$n" + return 0 +} + +# Blocks until nothing is reading , or until the grace period +# expires. Returns 0 when drained, 1 on timeout — the caller decides what to +# do with a timeout, and in this codebase that decision is always "defer the +# unlink", never "unlink anyway". +wait_for_readers() { + local root="$1" name="$2" grace="${3:-$CACHE_READ_GRACE_SECONDS}" deadline n waited=0 + deadline=$(( $(date +%s) + grace )) + while :; do + n=$(live_reader_count "$root" "$name") + [ "$n" -eq 0 ] && { + [ "$waited" -gt 0 ] && echo "readers: ${name} drained after ${waited}s" + return 0 + } + if [ "$(date +%s)" -ge "$deadline" ]; then + echo "readers: ${n} job(s) still reading ${name} after ${grace}s" >&2 + return 1 + fi + [ "$waited" = 0 ] && echo "readers: waiting for ${n} in-flight clone(s) of ${name} (grace ${grace}s)" + sleep 1 + waited=$((waited + 1)) + done +} + +# --------------------------------------------------------------------------- +# The clone itself +# --------------------------------------------------------------------------- + +# Number of times a torn clone is retried before the caller is failed. A tear +# means the source changed identity or lost entries mid-walk, which is a +# transient condition by definition — the publisher that caused it has already +# put a complete new generation at the same path — so one retry almost always +# suffices; the rest are headroom. +CACHE_CLONE_ATTEMPTS="${CACHE_CLONE_ATTEMPTS:-4}" + +_tree_entries() { + local n + n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0 + printf '%s' "$n" + return 0 +} + +_dir_inode() { + stat -c '%i' "$1" 2>/dev/null || printf 'missing' + return 0 +} + +write_cache_lock() { + local dir="$1" id="$2" + [ -n "$id" ] || return 0 + [ -d "$dir" ] || return 0 + date +%s > "${dir}/.ci-lock-${id}" 2>/dev/null || true return 0 } # Hardlink-clones SRC to a staging path, sanitises it, and publishes it to DST # with a single atomic rename. # +# hardlink_clone_into [lock-id] +# # The staging + rename is what closes the concurrent-seed race structurally # rather than by runner topology: a second job sharing this cache key either # sees DST absent (and stages its own clone, losing the rename harmlessly) or # sees it complete. There is no observable half-populated state, because a # directory rename is atomic and DST is never written through. # +# The rename is necessary but NOT sufficient, and that gap is what this +# function's retry loop closes. An atomic rename of a TRUNCATED tree publishes +# a truncated tree atomically. Three things can truncate one: +# +# * `cp -al` failing partway (a source entry vanished after readdir listed +# it) — caught by checking its exit status, which is why that status is +# read into a variable here rather than left to an ambient `set -e` the +# CALL SITES suppress anyway by invoking this function as a condition; +# * `cp -al` succeeding while having silently never seen a subtree that was +# unlinked before it read the parent's listing — caught only by the entry +# count, since there is no error to report; +# * the source being replaced wholesale mid-walk, so the clone splices two +# generations — caught by comparing the source directory's inode before +# and after. +# +# All three are verified on every attempt and a failing one restarts the +# clone; a tree that fails the last attempt is deleted and reported, never +# renamed into place. Combined with the reader marker (held across the copy, +# which is what stops the publish side unlinking underneath it in the first +# place), a partial tree cannot reach DST. +# +# `lock-id`, when given, writes this job's cache lock INTO the staging tree so +# the directory already carries it the instant it appears under its final +# name. Acquiring the lock after the rename would leave a freshly seeded +# directory momentarily unlocked and therefore evictable by a concurrent job's +# prune pass. +# # Returns 0 if this caller's clone won the rename, 1 if another caller got -# there first (the staging copy is discarded; DST is already valid). +# there first (the staging copy is discarded; DST is already valid), and 2 if +# the source could not be cloned consistently at all. hardlink_clone_into() { - local src="$1" dst="$2" tag="$3" tmp parent + local src="$1" dst="$2" tag="$3" lock_id="${4:-}" + local parent tmp src_name attempt cp_rc n_before n_after i_before i_after + + if [ ! -d "$src" ]; then + echo "::error::clone: source ${src} does not exist" >&2 + return 2 + fi + parent=$(dirname "$dst") + src_name=$(basename "$src") tmp="${parent}/.stage-${tag}" - rm -rf "$tmp" - cp -al "$src" "$tmp" + + attempt=1 + while : ; do + rm -rf "$tmp" + # Marker first, then the identity read, then the copy — see the ordering + # proof in the reader-marker section above; swapping the first two lines + # is what would reintroduce the race. + reader_lock_acquire "$parent" "$src_name" "$tag" + i_before=$(_dir_inode "$src") + n_before=$(_tree_entries "$src") + cp_rc=0 + cp -al "$src" "$tmp" || cp_rc=$? + n_after=$(_tree_entries "$tmp") + i_after=$(_dir_inode "$src") + reader_lock_release "$parent" "$src_name" "$tag" + + if [ "$cp_rc" -eq 0 ] && [ "$i_before" != missing ] && [ "$i_before" = "$i_after" ] \ + && [ "$n_after" -eq "$n_before" ]; then + break + fi + + echo "::warning::clone: attempt ${attempt}/${CACHE_CLONE_ATTEMPTS} of ${src_name} was torn (cp rc=${cp_rc}, ${n_after}/${n_before} entries, source inode ${i_before} -> ${i_after}) — discarding and retrying" >&2 + rm -rf "$tmp" + if [ "$attempt" -ge "$CACHE_CLONE_ATTEMPTS" ]; then + echo "::error::clone: ${src} could not be read consistently in ${CACHE_CLONE_ATTEMPTS} attempts — refusing to publish a partial tree at ${dst}" >&2 + return 2 + fi + attempt=$((attempt + 1)) + sleep 1 + done + strip_cargo_locks "$tmp" rm -f "$tmp"/.cache-last-used "$tmp"/.ci-lock-* 2>/dev/null || true - unshare_mutable_paths "$tmp" + if ! unshare_mutable_paths "$tmp"; then + echo "::error::clone: could not privately own the mutable paths of ${dst} — discarding the staging tree rather than publishing one that aliases ${src}" >&2 + rm -rf "$tmp" + return 2 + fi + write_cache_lock "$tmp" "$lock_id" + if mv -T "$tmp" "$dst" 2>/dev/null; then return 0 fi diff --git a/scripts/prune-cache.sh b/scripts/prune-cache.sh index 5725448..7e6efa6 100755 --- a/scripts/prune-cache.sh +++ b/scripts/prune-cache.sh @@ -49,12 +49,13 @@ # the entire benefit this scheme exists to deliver. # # LOCKED — a directory carrying a .ci-lock-* marker younger than -# STALE_LOCK_SECONDS is held open by a running job and is skipped by every -# pass, however dead and however tight the disk. This is what makes eviction -# safe on a runner with more than one execution slot. An older marker is -# treated as abandoned and logged as such, so an actually-still-running job -# that somehow exceeds the threshold is visible in the log rather than -# silently losing its cache mid-build. +# STALE_LOCK_SECONDS is held open by a running job, or named by a live +# .reading--* marker (a job is hardlink-cloning it this instant), is +# skipped by every pass, however dead and however tight the disk. This is +# what makes eviction safe on a runner with more than one execution slot. +# An older marker is treated as abandoned and logged as such, so an +# actually-still-running job that somehow exceeds the threshold is visible +# in the log rather than silently losing its cache mid-build. # # Liveness is resolved by `git ls-remote --heads origin`, wrapped in a # timeout. A directory name cannot be inverted back to a branch name (the @@ -93,8 +94,20 @@ is_protected() { } is_locked() { - local dir="$1" now lock_file lock_age locked=1 + local dir="$1" now lock_file lock_age locked=1 readers now=$(date +%s) + # A directory being hardlink-cloned right now carries no .ci-lock-* of its + # own — a snapshot has its locks stripped by construction — so the reader + # markers are the only signal that unlinking it would truncate somebody's + # in-flight clone. Today no reachable configuration prunes a snapshot (only + # protected refs publish them, and protected refs are excluded from every + # pass), which makes this guard redundant *by policy*. It is here so that + # stops being the reason it is safe. + readers=$(live_reader_count "$ROOT" "$(basename "$dir")") + if [ "$readers" -gt 0 ]; then + echo " $(basename "$dir"): ${readers} job(s) currently cloning it — not a candidate" + locked=0 + fi for lock_file in "$dir"/.ci-lock-*; do [ -e "$lock_file" ] || continue lock_age=$(( now - $(stat -c '%Y' "$lock_file") )) diff --git a/scripts/publish-snapshot.sh b/scripts/publish-snapshot.sh index ce7a025..4c26834 100755 --- a/scripts/publish-snapshot.sh +++ b/scripts/publish-snapshot.sh @@ -23,11 +23,23 @@ # own cold-start path — a safe degrade that self-heals on its next run, not # corruption. # -# `rm -rf` on the old snapshot removes directory entries only. Any consumer -# that already hardlink-cloned from it keeps every inode alive through its own -# links, so a republish never pulls data out from under a running job — it -# just stops new consumers from seeing the old generation. Disk is reclaimed -# when the last clone referencing those inodes is itself evicted. +# `rm -rf` on the old snapshot removes directory entries only, so a consumer +# that has ALREADY FINISHED cloning from it keeps every inode alive through +# its own links. That is the easy half, and on its own it is not enough: a +# consumer still WALKING the old generation has its entries unlinked out from +# under it, and `cp -al` does not report a subtree that was removed before it +# read the parent's listing. That is a silently truncated clone — the failure +# mode this script's own selftest (scenario 8) reproduces against the +# unguarded version. +# +# So the unlink is interlocked with the consume side rather than +# unconditional: after the swap, this script waits for every in-flight reader +# of this snapshot to drain (see the reader-marker ordering proof in +# cache-lib.sh) and only then reclaims the rotated-away generation. If the +# grace period expires first, reclamation is DEFERRED — the directory is left +# under `.publish-old--` and swept by a later publish once no reader +# holds it. The residual of a very slow consumer is therefore one extra +# generation of directory entries on disk, never a torn clone. set -euo pipefail . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" @@ -36,7 +48,11 @@ ROOT="${2:?}"; TAG="${3:?}" SRC=$(target_dir_for "$ROOT" "$OWN_KEY") DST=$(snapshot_dir_for "$ROOT" "$OWN_KEY") -OLD="${ROOT}/.publish-old-${TAG}" +# Keyed by cache key as well as tag, so a deferred generation can be matched +# back to the snapshot whose readers must drain before it is safe to reclaim. +OLD="${ROOT}/.publish-old-${OWN_KEY}-${TAG}" +SNAP_NAME=$(basename "$DST") +GRACE="${CACHE_READ_GRACE_SECONDS}" if [ ! -d "$SRC" ]; then echo "publish: no target dir at ${SRC} — nothing to snapshot" @@ -47,6 +63,21 @@ fi # concurrently running job's staging directory is never touched. rm -rf "${ROOT}/.stage-${TAG}" "$OLD" +# Deferred reclamation from an earlier publish of THIS snapshot whose readers +# had not drained in time. Safe to sweep now only if nothing is reading the +# snapshot at all: a reader holds the snapshot path, not the deferred name, so +# "no readers of snapshot-" is the condition that makes every deferred +# generation of it unreachable. Over-conservative by design — a reader of the +# CURRENT generation also defers the sweep to the next publish, which costs a +# directory listing, not correctness. +if [ "$(live_reader_count "$ROOT" "$SNAP_NAME")" -eq 0 ]; then + for stale_old in "${ROOT}/.publish-old-${OWN_KEY}-"*; do + [ -d "$stale_old" ] || continue + echo "publish: reclaiming deferred snapshot generation $(basename "$stale_old")" + rm -rf "$stale_old" + done +fi + start=$(date +%s) # The staged snapshot is hardlinked to SRC's artifacts and holds its OWN copy # of every file Cargo rewrites in place (unshare_mutable_paths, called inside @@ -63,7 +94,19 @@ hardlink_clone_into "$SRC" "$TMP_DST" "$TAG" || { if [ -d "$DST" ]; then mv -T "$DST" "$OLD"; fi mv -T "$TMP_DST" "$DST" -rm -rf "$OLD" + +# The scan below happens strictly after the rename above, which is what makes +# it impossible for a consumer holding the OLD generation to be missed: such a +# consumer resolved the path before that rename, and published its marker +# before that. See cache-lib.sh's reader-marker section. +if [ -d "$OLD" ]; then + if wait_for_readers "$ROOT" "$SNAP_NAME" "$GRACE"; then + rm -rf "$OLD" + else + echo "::warning::publish: a consumer is still cloning the previous ${SNAP_NAME} after ${GRACE}s — deferring reclamation of $(basename "$OLD") rather than unlinking a tree being read" + summary_line "- deferred reclaiming the previous \`${SNAP_NAME}\` generation (a consumer is still cloning it); it will be swept by a later publish" + fi +fi echo "publish: ${DST} ($(usage_gb "$DST") GB) published in $(( $(date +%s) - start ))s" summary_line "- published cache snapshot \`$(basename "$DST")\` ($(usage_gb "$DST") GB)" diff --git a/scripts/restore-mtimes.sh b/scripts/restore-mtimes.sh index 18ac757..063b54d 100755 --- a/scripts/restore-mtimes.sh +++ b/scripts/restore-mtimes.sh @@ -170,7 +170,8 @@ # the next run's diff base stays at the last GREEN build — over-inclusive # (it may re-stamp files that a failed run partially rebuilt anyway) but # never under-inclusive, which is the only direction that matters for -# soundness.set -euo pipefail +# soundness. +set -euo pipefail repo_root=$(git rev-parse --show-toplevel) cd "$repo_root" diff --git a/scripts/seed-target-dir-selftest.sh b/scripts/seed-target-dir-selftest.sh index 2079802..ab3ec4c 100755 --- a/scripts/seed-target-dir-selftest.sh +++ b/scripts/seed-target-dir-selftest.sh @@ -29,6 +29,21 @@ # at no point is a partially-populated directory visible under the final # name. This is the property that replaces "the runner only has one job # slot" with an actual guarantee. +# 8. SEED VS PUBLISH ROTATION — the race scenario 7 does NOT cover, and the +# one that actually mattered: a consumer hardlink-cloning a snapshot +# while the publisher of that snapshot rotates it and unlinks the +# generation being read. Two seeds racing on a DESTINATION is a different +# race from a seed racing a publisher on its SOURCE, and only the second +# one can truncate a tree. Against the unguarded version this scenario +# reproduces a silent partial clone reported as success — 20,328 of +# 48,805 entries, `seed: cloned in 1s`, exit 0, seeded-from=base-snapshot. +# 9. DEFERRED RECLAMATION — when a consumer is STILL reading after the grace +# period, the publisher leaves the rotated-away generation on disk rather +# than unlinking a tree under an in-flight walk, and a later publish +# sweeps it once the reader is gone. The residual is disk, not a torn +# clone. +# 10. AN UNREADABLE SOURCE FAILS LOUDLY — the clone reports a distinct +# status instead of renaming whatever it managed to produce into place. set -euo pipefail script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) . "$script_dir/cache-lib.sh" @@ -56,7 +71,32 @@ make_tree() { : > "$d/debug/.cargo-lock" } -seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { cat "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; } +seed() { bash "$script_dir/seed-target-dir.sh" "$@" > "$scratch/log" 2>&1 || { tail -40 "$scratch/log"; fail "seed-target-dir.sh exited non-zero"; }; } + +# Always succeeds and always prints a number: an absent directory is "0 +# entries so far", which is the normal state at the top of the progress poll +# below, not an error worth aborting the suite over. +tree_entries() { + local n + n=$(find "$1" -mindepth 1 2>/dev/null | wc -l) || n=0 + printf '%s\n' "$n" + return 0 +} + +# A tree wide enough that a hardlink clone of it takes long enough to be +# caught mid-walk. The race under test is a real interleaving, not a mocked +# one, so the fixture has to be big enough for the window to exist: a +# four-file tree clones in microseconds and no scheduling could ever land +# inside it. Built by cloning one small template directory N times, which is N +# forks rather than N*M file creations. +make_wide_tree() { + local d="$1" marker="$2" ndirs="$3" i + mkdir -p "$d/debug/deps/.tmpl" "$d/debug/.fingerprint/x" + for i in $(seq 0 59); do echo "$marker" > "$d/debug/deps/.tmpl/f$i"; done + for i in $(seq -w 1 "$ndirs"); do cp -al "$d/debug/deps/.tmpl" "$d/debug/deps/d$i"; done + rm -rf "$d/debug/deps/.tmpl" + echo "$marker" > "$d/debug/.fingerprint/x/dep-lib-x" +} BASE_KEY=$(cache_key dev) OWN_KEY=$(cache_key feat/thing) @@ -127,5 +167,96 @@ if grep -q 'starts cold' "$scratch/logA" "$scratch/logB"; then fi ok "neither racing job fell through to a cold start" +echo +echo "=== 8: seeding while the base republishes the snapshot underneath it ===" +ROT=$(cache_key feat/rotate) +rm -rf "$root/snapshot-$BASE_KEY" "$root/target-$BASE_KEY" +# Generation 1 is wide (the consumer will still be walking it when the swap +# happens); the generation replacing it is small, so the publisher's own +# staging clone does not itself outlast the consumer's. +make_wide_tree "$root/snapshot-$BASE_KEY" gen1 800 +make_wide_tree "$root/target-$BASE_KEY" gen2 8 +gen1_entries=$(tree_entries "$root/snapshot-$BASE_KEY") + +( bash "$script_dir/seed-target-dir.sh" "$ROT" "$BASE_KEY" "$root" jobRot > "$scratch/logRot" 2>&1; echo $? > "$scratch/rcRot" ) & +seed_pid=$! + +# Rotate only once the clone is demonstrably mid-walk. Gating on observed +# progress rather than on a sleep is what makes the interleaving reproducible +# instead of a coin flip that passes on a fast machine for the wrong reason. +threshold=$(( gen1_entries / 5 )) +progress=0 +deadline=$(( $(date +%s) + 60 )) +while :; do + progress=$(tree_entries "$root/.stage-jobRot") + if [ "$progress" -ge "$threshold" ]; then break; fi + if ! kill -0 "$seed_pid" 2>/dev/null; then + fail "the seed finished before its clone could be caught mid-walk (fixture too small for this machine?)" + fi + if [ "$(date +%s)" -ge "$deadline" ]; then + fail "the staging clone never reached ${threshold} of ${gen1_entries} entries" + fi +done +ok "caught the consumer's clone mid-walk at ${progress}/${gen1_entries} entries" + +bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubRot > "$scratch/logPub" 2>&1 \ + || { tail -40 "$scratch/logPub"; fail "publish-snapshot.sh exited non-zero"; } +wait "$seed_pid" + +rot_dir="$root/target-$ROT" +[ "$(cat "$scratch/rcRot")" = "0" ] || { tail -40 "$scratch/logRot"; fail "the seed exited non-zero"; } +ok "the seed completed" +# THE assertion. Before the guard, this is where it failed: the seed reported +# `cloned in 1s` and exit 0 while target- held less than half the entries +# of the snapshot it claimed to have cloned. Comparing against the snapshot as +# it stands NOW is the right bar either way — a clone that raced the rotation +# must end up holding one complete generation, and a consumer caught mid-walk +# re-reads, so that generation is the new one. +snap_entries=$(tree_entries "$root/snapshot-$BASE_KEY") +rot_entries=$(tree_entries "$rot_dir") +[ "$rot_entries" -eq "$snap_entries" ] \ + || fail "the seeded tree is truncated: ${rot_entries} entries against the snapshot's ${snap_entries} (was ${gen1_entries} before the rotation)" +ok "the seeded tree is complete (${rot_entries} entries, no silent truncation)" +assert_content "$rot_dir/debug/.fingerprint/x/dep-lib-x" gen2 "the seeded tree holds one whole generation, not a splice of two" +grep -q 'was torn' "$scratch/logRot" || fail "the rotation was not detected as a torn read" +ok "the torn read was detected and reported, not swallowed" +[ -z "$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print -quit)" ] \ + || fail "scratch left behind: $(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)" +ok "no staging, reader-marker or deferred-generation scratch left behind" + +echo +echo "=== 9: a reader that outlasts the grace period defers reclamation ===" +# A synthetic reader marker stands in for a consumer whose clone is slower +# than the grace period. Driving that with a real slow consumer would make the +# test's runtime the thing under test; the marker is the whole contract +# between the two sides, so holding one IS being a reader. +SNAP_NAME="snapshot-$BASE_KEY" +date +%s > "$root/.reading-${SNAP_NAME}-slowpoke" +make_wide_tree "$root/target-$BASE_KEY" gen3 4 +CACHE_READ_GRACE_SECONDS=1 bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubDefer > "$scratch/logDefer" 2>&1 \ + || { tail -40 "$scratch/logDefer"; fail "publish-snapshot.sh exited non-zero"; } +assert_content "$root/$SNAP_NAME/debug/.fingerprint/x/dep-lib-x" gen3 "the new generation was published regardless" +deferred=$(find "$root" -maxdepth 1 -name ".publish-old-${BASE_KEY}-*" -print -quit) +[ -n "$deferred" ] || fail "the previous generation was unlinked while a reader still held it" +ok "the rotated-away generation was left on disk rather than unlinked under a reader" +grep -q 'deferring reclamation' "$scratch/logDefer" || fail "the deferral was not reported" +ok "the deferral is reported as a warning, not silent" + +rm -f "$root/.reading-${SNAP_NAME}-slowpoke" +make_wide_tree "$root/target-$BASE_KEY" gen4 4 +bash "$script_dir/publish-snapshot.sh" "$BASE_KEY" "$root" pubSweep > "$scratch/logSweep" 2>&1 \ + || { tail -40 "$scratch/logSweep"; fail "publish-snapshot.sh exited non-zero"; } +[ -z "$(find "$root" -maxdepth 1 -name '.publish-old-*' -print -quit)" ] \ + || fail "the deferred generation was never reclaimed" +ok "a later publish reclaims the deferred generation once no reader holds it" + +echo +echo "=== 10: a source that cannot be read fails loudly ===" +rc=0 +hardlink_clone_into "$root/nosuch-source" "$root/target-nosuch" nosuch-tag > "$scratch/logMissing" 2>&1 || rc=$? +[ "$rc" -eq 2 ] || fail "expected status 2 for an unreadable source, got ${rc}" +ok "an unreadable source returns the distinct hard-failure status" +assert_absent "$root/target-nosuch" "nothing was renamed into place" + echo echo "seed-target-dir-selftest: ${pass_count} assertions passed" diff --git a/scripts/seed-target-dir.sh b/scripts/seed-target-dir.sh index 0450b91..8451d10 100755 --- a/scripts/seed-target-dir.sh +++ b/scripts/seed-target-dir.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Consume side: make this run's own target dir exist, warm, and private. # -# Usage: seed-target-dir.sh [fallback-dir] +# Usage: seed-target-dir.sh [fallback-dir] [lock-id] # own-key cache key for this run's own ref # base-key cache key for the ref to layer over ("" for a run whose own # ref IS a reference branch) @@ -10,6 +10,14 @@ # directory so two concurrent jobs can never collide on it # fallback-dir optional absolute path to seed from when no snapshot exists # (a legacy flat cache dir during a migration, typically) +# lock-id optional cache-lock id. When given, the lock marker is +# written on EVERY path out of this script — including into +# the staging tree before its rename — so the directory is +# never observable under its final name without a lock. The +# action acquires the lock in a later step too; that step is +# idempotent, and this closes the window before it runs, where +# a concurrent job's prune pass could evict a directory that +# exists but is not yet held. # # The design in one paragraph: a PR branch's first run hardlink-clones the # base branch's PUBLISHED SNAPSHOT. Hardlink, because the clone then costs @@ -27,17 +35,18 @@ set -euo pipefail . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/cache-lib.sh" -if [ $# -lt 4 ] || [ $# -gt 5 ]; then - echo "::error::seed-target-dir.sh: expected 4 or 5 arguments (own-key, base-key, cache-root, tag, [fallback-dir])" >&2 +if [ $# -lt 4 ] || [ $# -gt 6 ]; then + echo "::error::seed-target-dir.sh: expected 4 to 6 arguments (own-key, base-key, cache-root, tag, [fallback-dir], [lock-id])" >&2 exit 1 fi -OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}" +OWN_KEY="$1"; BASE_KEY="$2"; ROOT="$3"; TAG="$4"; FALLBACK="${5:-}"; LOCK_ID="${6:-}" OWN_DIR=$(target_dir_for "$ROOT" "$OWN_KEY") mkdir -p "$ROOT" if [ -d "$OWN_DIR" ]; then + write_cache_lock "$OWN_DIR" "$LOCK_ID" echo "seed: reusing this ref's own cache at ${OWN_DIR} ($(usage_gb "$OWN_DIR") GB)" echo "seeded-from=own" >> "${GITHUB_OUTPUT:-/dev/null}" exit 0 @@ -62,20 +71,40 @@ for entry in "${CANDIDATES[@]}"; do [ -d "$src" ] || continue echo "seed: hardlink-cloning ${label} ${src} ($(usage_gb "$src") GB) -> ${OWN_DIR}" start=$(date +%s) - if hardlink_clone_into "$src" "$OWN_DIR" "$TAG"; then - echo "seed: cloned in $(( $(date +%s) - start ))s" - echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}" - else - # Another job sharing this cache key won the rename while we were - # cloning. Its directory is complete (the rename is the publish step), so - # there is nothing to do but use it — and nothing was ever observable in - # a half-seeded state. - echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs" - echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}" - fi + # The status is captured and dispatched on explicitly. Calling this as a + # bare `if` condition — which is what this script used to do — suppresses + # `set -e` for the whole call, so a hard clone failure could not abort the + # seed even in principle; the three outcomes are genuinely distinct and each + # needs its own handling. + clone_rc=0 + hardlink_clone_into "$src" "$OWN_DIR" "$TAG" "$LOCK_ID" || clone_rc=$? + case "$clone_rc" in + 0) + echo "seed: cloned in $(( $(date +%s) - start ))s" + echo "seeded-from=${label// /-}" >> "${GITHUB_OUTPUT:-/dev/null}" + ;; + 1) + # Another job sharing this cache key won the rename while we were + # cloning. Its directory is complete (the rename is the publish step), + # so there is nothing to do but use it — and nothing was ever + # observable in a half-seeded state. + write_cache_lock "$OWN_DIR" "$LOCK_ID" + echo "seed: another job seeded ${OWN_DIR} concurrently; discarded our staging copy and using theirs" + echo "seeded-from=concurrent-peer" >> "${GITHUB_OUTPUT:-/dev/null}" + ;; + *) + # A source that could not be read consistently. Failing the job is the + # only safe answer: the alternative that used to happen here was + # seeding a truncated tree and reporting success, which hands Cargo a + # directory whose fingerprints and artifacts disagree. + echo "::error::seed: could not clone ${label} ${src} consistently — refusing to build against a partial cache" >&2 + exit 1 + ;; + esac exit 0 done echo "seed: no snapshot or fallback available — ${OWN_DIR} starts cold" echo "seeded-from=cold" >> "${GITHUB_OUTPUT:-/dev/null}" mkdir -p "$OWN_DIR" +write_cache_lock "$OWN_DIR" "$LOCK_ID"