Files
gitdan-actions/scripts/restore-mtimes.sh
claudeandClaude Opus 5 f57e2a6013 fix(cargo-cache): close the seed-vs-republish race the design claimed to close
The shared action's justification over zemyna's and emowheel's schemes was
that hardlink-cloning from a published snapshot closes gitdan #911 "by
construction, not by the single job slot". Review disproved that. This makes
the claim true, and corrects the README where it could only be bounded.

Finding 1 (verdict-level) — silent partial clone
------------------------------------------------
`hardlink_clone_into` ran `cp -al` with no exit-status check, and both call
sites invoked it as a condition, which suppresses `set -e` for the whole call.
A publisher's `rm -rf` of the generation it rotated away therefore unlinked
entries beneath an in-flight consumer walk, and the truncated tree was renamed
into place and reported as success.

Both layers are fixed:

* The consumer verifies its own clone. Every attempt checks `cp -al`'s status
  explicitly, the source directory's inode before and after (a wholesale
  replacement mid-walk splices two generations), and the entry count — the
  only signal for a subtree unlinked before its parent was listed, since
  `cp -al` reports no error for one it never saw. Any failure discards the
  staging tree and retries; exhausting the attempts returns a distinct status
  2 and fails the job rather than seeding a partial cache. `unshare_subtree` /
  `_unshare_files` now propagate failure too — a swallowed unshare leaves the
  clone aliasing its source, the exact corruption that step exists to prevent.
* The publisher does not unlink under a reader. A consumer publishes a
  `.reading-<snapshot>-<tag>` marker before it resolves the snapshot path; the
  publisher scans for markers after its first rename. A consumer holding the
  old generation therefore published its marker before that scan and cannot be
  missed; one arriving after the scan necessarily resolves to the new
  generation. The publisher waits for readers to drain and, on timeout,
  DEFERS reclamation rather than forcing it — the old generation is left as
  `.publish-old-<key>-<tag>` and swept by a later publish.

So correctness is closed by construction; disk reclamation is bounded, not
immediate. The residual is capped at one deferred generation per publisher
ref, and the README now says exactly that instead of the disproved claim.

Finding 2 — restore-mtimes.sh ran with no errexit
-------------------------------------------------
`set -euo pipefail` was glued to the end of a comment (`# soundness.set -euo
pipefail`), so it was entirely commented out: a partial failure of the
`git log | awk` pipeline would have produced wrong mtimes across the whole
restore instead of failing loudly. Moved to its own line. Audited every other
script for the same defect — this was the only instance. Independent
confirmation: shellcheck's two SC2164 warnings on this file's `cd "$repo_root"`
disappear now that errexit is actually in effect.

Finding 3 — lock-acquire window
-------------------------------
A just-seeded directory was unlocked until a later action step, so a
concurrent job's prune pass could evict it. `seed-target-dir.sh` now takes an
optional lock-id and writes the lock marker on every path out of the script,
including into the staging tree before its rename, so the directory carries a
lock the instant it appears under its final name. The action's acquire step
stays (it is idempotent and stamps the LRU marker).

Also hardened `prune-cache.sh` to treat a directory with live reader markers
as locked. Today no reachable configuration prunes a snapshot — only protected
refs publish them and protected refs are excluded from every pass — so this is
redundant by policy; it is here so that stops being the reason it is safe.

Verification
------------
New selftest scenario 8 races a real seed against a real publish rotation,
gating the rotation on the seed's *observed* clone progress so the window is
hit deterministically rather than on a fast machine's coin flip. Red-proven
against the unguarded scripts, three consecutive runs:

  ASSERTION FAILED: the seeded tree is truncated: 15443 entries against the
  snapshot's 493 (was 48805 before the rotation)      (15443 / 16986 / 16498)

Green after the fix, six consecutive runs, catching the clone mid-walk at
~10.5k of 48805 entries each time. Scenario 9 covers deferred reclamation and
its later sweep; scenario 10 covers an unreadable source failing loudly.

`bash scripts/selftest.sh`: 5 suites, exit 0, 75 assertions (was 63).
shellcheck over `scripts/`: no new findings, two SC2164 warnings resolved.

Docs: README's republish-safety paragraph replaced with what the code now
guarantees, including the bounded disk residual stated explicitly; new
`read-grace-seconds` / `reader-stale-seconds` inputs documented in the
`cargo-cache-publish` table; the selftest table names the new race.

Refs: daniel/gitdan#11, zemyna#911

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sqh2vscfzisk83VuPVQX9L
2026-08-23 16:40:21 -05:00

347 lines
19 KiB
Bash
Executable File

#!/usr/bin/env bash
# Restores every git-tracked file's mtime to the timestamp of the most recent
# commit that touched it. `actions/checkout` stamps every file with the
# checkout's wall-clock "now", and Cargo's fingerprint freshness check is
# mtime-based by default. Result: even though the persistent volume genuinely
# holds prior build artifacts, every checkout makes every tracked file look
# "newer than what was recorded at last build", so every crate rebuilds from
# scratch on every run — measured directly on a Bevy-sized workspace before
# this fix, by grepping a warm-volume run's log for Cargo's `Fresh` cache-hit
# marker: zero matches across ~28 crates. Do not remove this as "redundant
# with the target-dir cache": it is the fix that makes that cache take effect.
#
# Algorithm: a single `git log --name-only --no-renames` traversal
# (newest-commit-first, git log's default order) instead of one `git log`
# call per file — for a workspace this size a per-file walk would mean
# hundreds of separate history walks. For each path, the first timestamp
# encountered under newest-first traversal is the timestamp of the most
# recent commit that touched it. Only files currently tracked by git
# (`git ls-files`) are touched; deleted/renamed-away historical paths and
# anything gitignored are left alone.
#
# Each commit's timestamp line carries a `@@MTIME@@` sentinel prefix rather
# than relying on blank-line paragraph boundaries between commits. Consecutive
# merge commits with an empty diff (under `--no-renames`) print their
# timestamp lines back-to-back with no blank line between them — a
# blank-line-based parser misreads the second bare timestamp line as a
# filename and desyncs every timestamp after it. The sentinel can't collide
# with a real tracked path (it's a fixed literal no repo file is named), so
# every commit's timestamp line is unambiguous regardless of what precedes or
# follows it.
#
# Note: for a handful of paths this single raw pass picks a nominally
# different (usually newer) commit than `git log -1 -- <path>` would, which
# applies pathspec history-simplification (it can skip a commit whose change
# to a path got superseded by how a later merge resolved that path). That
# divergence is harmless here: correctness only requires (a) determinism —
# same HEAD always yields the same mtime for a given path, so a warm cache
# stays warm — and (b) change-sensitivity — a commit that actually changes a
# path's content is always the newest entry touching that path in traversal
# order, so it always wins the "first seen" race and invalidates that path's
# fingerprint. Reproducing full history-simplification per path would mean
# back to one `git log` call per file, the exact cost this single pass
# avoids.
#
# Directory mtimes, not just file mtimes: a build script that declares
# `cargo::rerun-if-changed=<some-dir>` (a directory rather than a file) has
# its rerun decision keyed partly on that directory inode's own mtime, and
# checkout recreates every directory with a fresh "now" mtime just like it
# does files. Restoring only tracked FILES leaves such a directory always
# looking newer than the recorded fingerprint, so the build script re-runs
# every time, marking its crate dirty and cascading to every downstream crate
# regardless of whether anything under the directory changed. Fixed by
# restoring every ancestor directory of every tracked path (excluding `.git/`,
# which no tracked path can be under) to the MAXIMUM restored timestamp among
# the tracked files in its subtree — deterministic given the per-file
# timestamps already computed above, and it still fires a rerun when a file
# under that directory genuinely changed, since that file's newer timestamp
# propagates to every ancestor's max. Costs about a second; worth paying even
# on a workspace with no such build script yet, so the day one appears the
# cache stability already covers it.
#
# Soundness against the cross-branch stale-binary hazard: the cargo-cache
# action gives every ref its own target directory on the shared volume, so no
# two refs' fingerprints ever share a directory and this script never has to
# arbitrate freshness across refs — only within one ref's own history, which
# is exactly what it is built to do soundly. On a nightly toolchain,
# CARGO_UNSTABLE_CHECKSUM_FRESHNESS is a complementary, stronger guarantee
# (content-addressed rather than mtime-based freshness); this script is not
# made redundant by it, because directory-form `rerun-if-changed` build-script
# watches are not covered by it and stable historical mtimes stay cheap
# belt-and-braces alongside it.
#
# The merge hazard (the merge hazard), and why it needed a watermark, not just a
# namespace: namespacing keeps two branches' fingerprints apart, but it does
# NOT make the comparison this script feeds sound on its own. The comparison
# that actually matters is not "is this file's mtime the historically correct
# one for its most recent commit" (which the git-log pass above answers
# exactly right) — it's **"is this file's mtime newer than the dep-info
# artifact Cargo already has on disk for it in THIS target dir"**, because
# that dep-info's own mtime is a real wall-clock timestamp stamped by rustc
# when it last wrote the artifact, not anything git-derived. Ordinary
# same-branch iteration makes the git-log answer and the Cargo answer agree
# for free, by causality: you cannot commit a change before the build it is
# meant to invalidate has already finished, so every file changed since the
# last build necessarily carries a commit timestamp after that build's real
# completion time. A merge breaks exactly that causal link — it can pull in a
# commit authored on a parallel lineage *before* this branch's own last
# build, so the git-log answer ("this file's most recent commit is old") and
# the Cargo answer ("this file changed since my last build") diverge, and
# restore-mtimes stamping the historically-correct-but-too-old mtime is what
# let Cargo reuse an rlib built before the method the merge introduced
# existed (observed in production as an "impossible" compile error: a crate
# reusing an rlib built before the method a merge introduced existed).
#
# The fix: for a file changed since *this target dir's own last successful
# build* — not "recently" in absolute git-log time — always stamp "now"
# (this run's own wall-clock start), never the historical commit timestamp,
# because "now" is provably later than that build's real completion time no
# matter what any commit's authored/committer date says. the cargo-cache-publish action's watermark step writes the built HEAD SHA into
# "$CARGO_TARGET_DIR/${CI_WATERMARK_FILE:-.ci-watermark-sha}" after every
# green run of THIS branch's own namespace; this script reads it back (if
# present — see the read below for what "absent" means and why it's still
# safe) and treats `git diff --name-only <watermark>..HEAD` as authoritative
# over the git-log-only answer for exactly those paths. Everything NOT in
# that diff keeps the git-log timestamp exactly as before, so the ordinary
# no-merge warm-cache case (the whole reason this script exists) is
# untouched.
#
# CI_WATERMARK_FILE: the watermark filename is a
# per-job parameter, NOT a per-branch constant — two jobs sharing one
# $CARGO_TARGET_DIR namespace must use two DIFFERENT watermark files, never
# one. A watermark answers "what did THIS consumer last successfully build",
# and two jobs building different target triples (e.g. one job's host debug/release vs another's
# wasm32-unknown-unknown) are two separate
# consumers with two separate answers. A single shared file breaks the
# moment a second job joins the namespace: job A reads the true prior
# watermark, rebuilds correctly, and advances the file to HEAD; job B then
# runs SECOND (same trigger, same HEAD, no `needs:` ordering — a serial
# runner still executes both jobs one after another within one trigger) and
# reads that JUST-ADVANCED watermark, computing an empty HEAD..HEAD diff, and
# falls back to git-log-only timestamps for everything — exactly this
# script's pre-fix behaviour, reintroduced for whichever job happens to run
# second on any trigger that includes a backdating merge. Caught via a two-crate repro, not by inspection alone.
# restore-mtimes-selftest.sh's scenarios 6/7 exercise this exact sequential
# ordering (one job's watermark write landing between two restore-mtimes
# invocations against the same namespace), not just two simultaneous reads.
# Defaults to ".ci-watermark-sha" — unset callers (a local/manual run, or a
# namespace with only ever one consumer) get the original filename and
# behaviour unchanged.
#
# restore-mtimes-selftest.sh, alongside this file, is a regression test for
# the watermark mechanism: a real scratch cargo workspace, a real
# restore-mtimes.sh run, a real `cargo build`, asserting both that the
# staleness bug reproduces without a watermark and that it's fixed with one
# (same-branch merge AND seeded-namespace cases). Run it by hand after touching this
# script's timestamp logic or the actions' seed/watermark step order.
#
# The watermark file deliberately lives INSIDE $CARGO_TARGET_DIR rather than
# a separate location, for two reasons that both matter: it rides along for
# free with prune-cache.sh's eviction pass (an evicted
# namespace loses its watermark along with everything else it would have
# been wrong to trust anyway), and it rides along for free with the
# seed-snapshot mechanism (the cargo-cache-publish action publishes
# $CARGO_TARGET_DIR wholesale, including this file) — so a brand-new branch
# namespace seeded from its base's published snapshot inherits the base's
# watermark along with the base's artifacts, and this script correctly diffs
# from THAT SHA rather than treating the seeded branch as having no watermark
# at all. That is exactly the same hazard class one level up (a new branch's
# first run reusing artifacts seeded from a base build that predates some of
# the base's own history), and it needs the same fix — which it gets for
# free rather than needing separate handling, as long as the cargo-cache action orders "seed
# or reuse the target dir" before "restore mtimes" so the watermark is
# actually in place, on disk, by the time this script runs. (It is — see the
# step order in the cargo-cache action.)
#
# Absent watermark (no file, or the target dir doesn't exist yet at all) is
# always safe, never a silent miscompare: this script simply skips the
# override and falls back fully to the git-log timestamps, exactly as it did
# before this fix existed. That is correct for a genuinely first run against
# an empty, unseeded namespace (nothing to compare against — everything is
# about to be a cold build regardless of what mtime it gets) and conservative
# for every other case that could produce an absent watermark: a watermark SHA
# that is no longer reachable in this checkout (a force-push or rebase
# rewrote the branch the watermark was recorded against) is NOT treated the
# same as "no watermark at all" — the reachability check below instead
# treats every tracked file as changed, rather than silently trusting a diff
# against a commit that may no longer describe any real ancestry
# relationship to HEAD. A run that fails before reaching
# the "Record build watermark" step simply never advances the watermark, so
# the next run's diff base stays at the last GREEN build — over-inclusive
# (it may re-stamp files that a failed run partially rebuilt anyway) but
# never under-inclusive, which is the only direction that matters for
# soundness.
set -euo pipefail
repo_root=$(git rev-parse --show-toplevel)
cd "$repo_root"
# A shallow checkout (the `actions/checkout` default) only has the tip
# commit available, which diffs against the empty tree — every tracked file
# would resolve to that single commit's timestamp, a uniform stamp that
# changes with every new commit and still defeats the fingerprint cache
# run-over-run (the exact bug this script fixes). That failure mode is
# silent otherwise: every path still "resolves" to some timestamp (just the
# wrong, unhelpfully-uniform one), so the missing-path fallback below never
# fires and nothing else in this script would surface it. Fail loudly here
# instead so a future `fetch-depth` regression on the checkout step shows up
# in the CI log rather than silently reverting to full rebuilds every run.
if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then
echo "::error::restore-mtimes: shallow clone detected — per-file mtimes unavailable, cache reuse will be defeated; set fetch-depth: 0 on the checkout step" >&2
exit 1
fi
start=$(date +%s)
resolved=$(mktemp)
tracked=$(mktemp)
to_touch=$(mktemp)
missing=$(mktemp)
all_touched=$(mktemp)
dir_touch=$(mktemp)
changed_since_watermark=$(mktemp)
to_touch_overridden=$(mktemp)
trap 'rm -f "$resolved" "$tracked" "$to_touch" "$missing" "$all_touched" "$dir_touch" "$changed_since_watermark" "$to_touch_overridden"' EXIT
# Emit "<epoch>\t<path>" once per path, first-seen-under-newest-first wins.
git log --format='@@MTIME@@%ct' --name-only --no-renames | awk '
/^@@MTIME@@[0-9]+$/ { ts = substr($0, 10); next }
NF == 0 { next }
!($0 in seen) { seen[$0] = 1; printf "%s\t%s\n", ts, $0 }
' > "$resolved"
git ls-files > "$tracked"
# Restrict to files git currently tracks — never .git/ internals, never
# gitignored build output, never a path that's since been deleted/renamed.
awk -F'\t' '
NR == FNR { track[$0] = 1; next }
$2 in track { print; seen[$2] = 1 }
' "$tracked" "$resolved" > "$to_touch"
# Watermark override: see the header comment above for why this
# comparison — against this target dir's own last successful build, not
# against absolute git-log history — is the one Cargo actually needs.
watermark_count=0
watermark_status="no CARGO_TARGET_DIR set (not running under the cargo-cache action) — skipped"
watermark_name="${CI_WATERMARK_FILE:-.ci-watermark-sha}"
watermark_file="${CARGO_TARGET_DIR:-}/${watermark_name}"
if [ -n "${CARGO_TARGET_DIR:-}" ] && [ -f "$watermark_file" ]; then
watermark_sha=$(cat "$watermark_file")
if git cat-file -e "${watermark_sha}^{commit}" 2>/dev/null; then
# Every path that differs between the watermark commit and HEAD —
# added, modified, or deleted — content-wise, regardless of what any
# commit along the way claims its own timestamp is.
git diff --name-only "$watermark_sha" HEAD > "$changed_since_watermark"
watermark_status="diffing against watermark ${watermark_sha} in ${CARGO_TARGET_DIR}/${watermark_name}"
else
# The watermark names a commit this checkout can no longer see — a
# force-push or rebase rewrote history out from under it. Trusting the
# diff would silently compare HEAD against a commit with no real
# ancestry relationship to it. Fall back to treating every tracked file
# as changed since the (unknowable) last build: safe (over-invalidates,
# never under-invalidates) and self-heals the moment the next green run
# records a fresh, reachable watermark.
cp "$tracked" "$changed_since_watermark"
watermark_status="watermark ${watermark_sha} unreachable (force-push/rebase?) in ${CARGO_TARGET_DIR}/${watermark_name} — treating every tracked file as changed since last build"
fi
else
: > "$changed_since_watermark"
if [ -n "${CARGO_TARGET_DIR:-}" ]; then
watermark_status="no watermark yet in ${CARGO_TARGET_DIR}/${watermark_name} (first run against this namespace) — skipped"
fi
fi
# For every path in the diff, override its git-log timestamp with "now"
# ($start, this run's own wall-clock start) — provably later than the real
# completion time of the build the watermark names, unlike any commit's
# authored/committer date. Everything NOT in the diff keeps its git-log
# timestamp exactly as before.
if [ -s "$changed_since_watermark" ]; then
# Re-derive the count from the intersection with $to_touch (rather than
# `wc -l < "$changed_since_watermark"` directly) so the reported number
# only counts paths this script is actually about to stamp differently,
# not every path `git diff` names (which, unlike $to_touch, is not
# restricted to currently-tracked files — e.g. a path deleted since the
# watermark commit).
watermark_count=$(awk -F'\t' 'NR == FNR { c[$0] = 1; next } $2 in c' "$changed_since_watermark" "$to_touch" | wc -l | tr -d ' ')
awk -F'\t' -v now="$start" '
NR == FNR { changed[$0] = 1; next }
{ if ($2 in changed) print now "\t" $2; else print }
' "$changed_since_watermark" "$to_touch" > "$to_touch_overridden"
mv "$to_touch_overridden" "$to_touch"
fi
# Any tracked path whose history couldn't resolve (only possible under an
# unexpectedly shallow checkout, since `fetch-depth: 0` on the checkout step
# means full history is normally present) falls back to a fixed constant
# epoch rather than being left alone at checkout-time "now". What Cargo's
# fingerprint check needs is an mtime that's STABLE run-over-run for
# unchanged content, not necessarily a historically accurate one — a
# constant epoch gives that stability so the file still gets cache reuse
# instead of permanently missing the cache on every single run.
awk -F'\t' '
NR == FNR { seen[$2] = 1; next }
!($0 in seen)
' "$to_touch" "$tracked" > "$missing"
touched=0
while IFS=$'\t' read -r ts path; do
[ -e "$path" ] || [ -L "$path" ] || continue
# `-h`: set the mtime of a git-tracked symlink itself, never the target it
# resolves to. No path in this repo is currently a tracked symlink, but a
# future one (e.g. a shared-config symlink into a dotfiles
# checkout) would otherwise get its mtime applied
# to whatever it resolves to outside the repo — keeping `-h` costs nothing
# today and avoids that surprise later.
touch -h -d "@$ts" -- "$path"
touched=$((touched + 1))
done < "$to_touch"
fallback_count=0
if [ -s "$missing" ]; then
while IFS= read -r path; do
[ -e "$path" ] || [ -L "$path" ] || continue
touch -h -d "@0" -- "$path"
fallback_count=$((fallback_count + 1))
done < "$missing"
fi
# Directory pass: for every tracked file's full ancestor chain (excluding
# the file's own leaf name), track the maximum timestamp seen. A directory
# with no tracked file directly in it but with tracked files somewhere in
# its subtree (e.g. `crates/`) still accumulates a max from those deeper
# files, since every ancestor level of every file's path is a separate
# prefix entry here. Fixed-epoch fallback entries (ts=0) are included too —
# harmless, since 0 can never win a max against any real commit timestamp.
cat "$to_touch" > "$all_touched"
if [ -s "$missing" ]; then
awk '{ print "0\t" $0 }' "$missing" >> "$all_touched"
fi
awk -F'\t' '
{
ts = $1; path = $2
n = split(path, parts, "/")
prefix = ""
for (i = 1; i < n; i++) {
prefix = (prefix == "" ? parts[i] : prefix "/" parts[i])
if (!(prefix in maxts) || ts + 0 > maxts[prefix] + 0) maxts[prefix] = ts
}
if (!("." in maxts) || ts + 0 > maxts["."] + 0) maxts["."] = ts
}
END {
for (d in maxts) printf "%s\t%s\n", maxts[d], d
}
' "$all_touched" > "$dir_touch"
dirs_touched=0
while IFS=$'\t' read -r ts dir; do
[ -d "$dir" ] || continue
touch -d "@$ts" -- "$dir"
dirs_touched=$((dirs_touched + 1))
done < "$dir_touch"
elapsed=$(( $(date +%s) - start ))
echo "restore-mtimes: touched ${touched} files + ${dirs_touched} dirs of $(wc -l < "$tracked") tracked files from history in ${elapsed}s (${fallback_count} fell back to a fixed epoch, ${watermark_count} overridden to now: ${watermark_status})"