#!/usr/bin/env bash # Restores every git-tracked file's mtime to the timestamp of the most recent # commit that touched it. `actions/checkout` stamps every file with the # checkout's wall-clock "now", and Cargo's fingerprint freshness check is # mtime-based by default. Result: even though the persistent volume genuinely # holds prior build artifacts, every checkout makes every tracked file look # "newer than what was recorded at last build", so every crate rebuilds from # scratch on every run — measured directly on a Bevy-sized workspace before # this fix, by grepping a warm-volume run's log for Cargo's `Fresh` cache-hit # marker: zero matches across ~28 crates. Do not remove this as "redundant # with the target-dir cache": it is the fix that makes that cache take effect. # # Algorithm: a single `git log --name-only --no-renames` traversal # (newest-commit-first, git log's default order) instead of one `git log` # call per file — for a workspace this size a per-file walk would mean # hundreds of separate history walks. For each path, the first timestamp # encountered under newest-first traversal is the timestamp of the most # recent commit that touched it. Only files currently tracked by git # (`git ls-files`) are touched; deleted/renamed-away historical paths and # anything gitignored are left alone. # # Each commit's timestamp line carries a `@@MTIME@@` sentinel prefix rather # than relying on blank-line paragraph boundaries between commits. Consecutive # merge commits with an empty diff (under `--no-renames`) print their # timestamp lines back-to-back with no blank line between them — a # blank-line-based parser misreads the second bare timestamp line as a # filename and desyncs every timestamp after it. The sentinel can't collide # with a real tracked path (it's a fixed literal no repo file is named), so # every commit's timestamp line is unambiguous regardless of what precedes or # follows it. # # Note: for a handful of paths this single raw pass picks a nominally # different (usually newer) commit than `git log -1 -- ` would, which # applies pathspec history-simplification (it can skip a commit whose change # to a path got superseded by how a later merge resolved that path). That # divergence is harmless here: correctness only requires (a) determinism — # same HEAD always yields the same mtime for a given path, so a warm cache # stays warm — and (b) change-sensitivity — a commit that actually changes a # path's content is always the newest entry touching that path in traversal # order, so it always wins the "first seen" race and invalidates that path's # fingerprint. Reproducing full history-simplification per path would mean # back to one `git log` call per file, the exact cost this single pass # avoids. # # Directory mtimes, not just file mtimes: a build script that declares # `cargo::rerun-if-changed=` (a directory rather than a file) has # its rerun decision keyed partly on that directory inode's own mtime, and # checkout recreates every directory with a fresh "now" mtime just like it # does files. Restoring only tracked FILES leaves such a directory always # looking newer than the recorded fingerprint, so the build script re-runs # every time, marking its crate dirty and cascading to every downstream crate # regardless of whether anything under the directory changed. Fixed by # restoring every ancestor directory of every tracked path (excluding `.git/`, # which no tracked path can be under) to the MAXIMUM restored timestamp among # the tracked files in its subtree — deterministic given the per-file # timestamps already computed above, and it still fires a rerun when a file # under that directory genuinely changed, since that file's newer timestamp # propagates to every ancestor's max. Costs about a second; worth paying even # on a workspace with no such build script yet, so the day one appears the # cache stability already covers it. # # Soundness against the cross-branch stale-binary hazard: the cargo-cache # action gives every ref its own target directory on the shared volume, so no # two refs' fingerprints ever share a directory and this script never has to # arbitrate freshness across refs — only within one ref's own history, which # is exactly what it is built to do soundly. On a nightly toolchain, # CARGO_UNSTABLE_CHECKSUM_FRESHNESS plus CARGO_BUILD_FINGERPRINT=content (both, # since cargo PR #17382 on 2026-08-22) is a complementary, stronger guarantee # (content-addressed rather than mtime-based freshness); this script is not # made redundant by it, because directory-form `rerun-if-changed` build-script # watches are not covered by it and stable historical mtimes stay cheap # belt-and-braces alongside it. # # The merge hazard (the merge hazard), and why it needed a watermark, not just a # namespace: namespacing keeps two branches' fingerprints apart, but it does # NOT make the comparison this script feeds sound on its own. The comparison # that actually matters is not "is this file's mtime the historically correct # one for its most recent commit" (which the git-log pass above answers # exactly right) — it's **"is this file's mtime newer than the dep-info # artifact Cargo already has on disk for it in THIS target dir"**, because # that dep-info's own mtime is a real wall-clock timestamp stamped by rustc # when it last wrote the artifact, not anything git-derived. Ordinary # same-branch iteration makes the git-log answer and the Cargo answer agree # for free, by causality: you cannot commit a change before the build it is # meant to invalidate has already finished, so every file changed since the # last build necessarily carries a commit timestamp after that build's real # completion time. A merge breaks exactly that causal link — it can pull in a # commit authored on a parallel lineage *before* this branch's own last # build, so the git-log answer ("this file's most recent commit is old") and # the Cargo answer ("this file changed since my last build") diverge, and # restore-mtimes stamping the historically-correct-but-too-old mtime is what # let Cargo reuse an rlib built before the method the merge introduced # existed (observed in production as an "impossible" compile error: a crate # reusing an rlib built before the method a merge introduced existed). # # The fix: for a file changed since *this target dir's own last successful # build* — not "recently" in absolute git-log time — always stamp "now" # (this run's own wall-clock start), never the historical commit timestamp, # because "now" is provably later than that build's real completion time no # matter what any commit's authored/committer date says. the cargo-cache-publish action's watermark step writes the built HEAD SHA into # "$CARGO_TARGET_DIR/${CI_WATERMARK_FILE:-.ci-watermark-sha}" after every # green run of THIS branch's own namespace; this script reads it back (if # present — see the read below for what "absent" means and why it's still # safe) and treats `git diff --name-only ..HEAD` as authoritative # over the git-log-only answer for exactly those paths. Everything NOT in # that diff keeps the git-log timestamp exactly as before, so the ordinary # no-merge warm-cache case (the whole reason this script exists) is # untouched. # # CI_WATERMARK_FILE: the watermark filename is a # per-job parameter, NOT a per-branch constant — two jobs sharing one # $CARGO_TARGET_DIR namespace must use two DIFFERENT watermark files, never # one. A watermark answers "what did THIS consumer last successfully build", # and two jobs building different target triples (e.g. one job's host debug/release vs another's # wasm32-unknown-unknown) are two separate # consumers with two separate answers. A single shared file breaks the # moment a second job joins the namespace: job A reads the true prior # watermark, rebuilds correctly, and advances the file to HEAD; job B then # runs SECOND (same trigger, same HEAD, no `needs:` ordering — a serial # runner still executes both jobs one after another within one trigger) and # reads that JUST-ADVANCED watermark, computing an empty HEAD..HEAD diff, and # falls back to git-log-only timestamps for everything — exactly this # script's pre-fix behaviour, reintroduced for whichever job happens to run # second on any trigger that includes a backdating merge. Caught via a two-crate repro, not by inspection alone. # restore-mtimes-selftest.sh's scenarios 6/7 exercise this exact sequential # ordering (one job's watermark write landing between two restore-mtimes # invocations against the same namespace), not just two simultaneous reads. # Defaults to ".ci-watermark-sha" — unset callers (a local/manual run, or a # namespace with only ever one consumer) get the original filename and # behaviour unchanged. # # restore-mtimes-selftest.sh, alongside this file, is a regression test for # the watermark mechanism: a real scratch cargo workspace, a real # restore-mtimes.sh run, a real `cargo build`, asserting both that the # staleness bug reproduces without a watermark and that it's fixed with one # (same-branch merge AND seeded-namespace cases). Run it by hand after touching this # script's timestamp logic or the actions' seed/watermark step order. # # The watermark file deliberately lives INSIDE $CARGO_TARGET_DIR rather than # a separate location, for two reasons that both matter: it rides along for # free with prune-cache.sh's eviction pass (an evicted # namespace loses its watermark along with everything else it would have # been wrong to trust anyway), and it rides along for free with the # seed-snapshot mechanism (the cargo-cache-publish action publishes # $CARGO_TARGET_DIR wholesale, including this file) — so a brand-new branch # namespace seeded from its base's published snapshot inherits the base's # watermark along with the base's artifacts, and this script correctly diffs # from THAT SHA rather than treating the seeded branch as having no watermark # at all. That is exactly the same hazard class one level up (a new branch's # first run reusing artifacts seeded from a base build that predates some of # the base's own history), and it needs the same fix — which it gets for # free rather than needing separate handling, as long as the cargo-cache action orders "seed # or reuse the target dir" before "restore mtimes" so the watermark is # actually in place, on disk, by the time this script runs. (It is — see the # step order in the cargo-cache action.) # # Absent watermark (no file, or the target dir doesn't exist yet at all) is # always safe, never a silent miscompare: this script simply skips the # override and falls back fully to the git-log timestamps, exactly as it did # before this fix existed. That is correct for a genuinely first run against # an empty, unseeded namespace (nothing to compare against — everything is # about to be a cold build regardless of what mtime it gets) and conservative # for every other case that could produce an absent watermark: a watermark SHA # that is no longer reachable in this checkout (a force-push or rebase # rewrote the branch the watermark was recorded against) is NOT treated the # same as "no watermark at all" — the reachability check below instead # treats every tracked file as changed, rather than silently trusting a diff # against a commit that may no longer describe any real ancestry # relationship to HEAD. A run that fails before reaching # the "Record build watermark" step simply never advances the watermark, so # the next run's diff base stays at the last GREEN build — over-inclusive # (it may re-stamp files that a failed run partially rebuilt anyway) but # never under-inclusive, which is the only direction that matters for # soundness. set -euo pipefail repo_root=$(git rev-parse --show-toplevel) cd "$repo_root" # A shallow checkout (the `actions/checkout` default) only has the tip # commit available, which diffs against the empty tree — every tracked file # would resolve to that single commit's timestamp, a uniform stamp that # changes with every new commit and still defeats the fingerprint cache # run-over-run (the exact bug this script fixes). That failure mode is # silent otherwise: every path still "resolves" to some timestamp (just the # wrong, unhelpfully-uniform one), so the missing-path fallback below never # fires and nothing else in this script would surface it. Fail loudly here # instead so a future `fetch-depth` regression on the checkout step shows up # in the CI log rather than silently reverting to full rebuilds every run. if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then echo "::error::restore-mtimes: shallow clone detected — per-file mtimes unavailable, cache reuse will be defeated; set fetch-depth: 0 on the checkout step" >&2 exit 1 fi start=$(date +%s) resolved=$(mktemp) tracked=$(mktemp) to_touch=$(mktemp) missing=$(mktemp) all_touched=$(mktemp) dir_touch=$(mktemp) changed_since_watermark=$(mktemp) to_touch_overridden=$(mktemp) trap 'rm -f "$resolved" "$tracked" "$to_touch" "$missing" "$all_touched" "$dir_touch" "$changed_since_watermark" "$to_touch_overridden"' EXIT # Emit "\t" once per path, first-seen-under-newest-first wins. git log --format='@@MTIME@@%ct' --name-only --no-renames | awk ' /^@@MTIME@@[0-9]+$/ { ts = substr($0, 10); next } NF == 0 { next } !($0 in seen) { seen[$0] = 1; printf "%s\t%s\n", ts, $0 } ' > "$resolved" git ls-files > "$tracked" # Restrict to files git currently tracks — never .git/ internals, never # gitignored build output, never a path that's since been deleted/renamed. awk -F'\t' ' NR == FNR { track[$0] = 1; next } $2 in track { print; seen[$2] = 1 } ' "$tracked" "$resolved" > "$to_touch" # Watermark override: see the header comment above for why this # comparison — against this target dir's own last successful build, not # against absolute git-log history — is the one Cargo actually needs. watermark_count=0 watermark_status="no CARGO_TARGET_DIR set (not running under the cargo-cache action) — skipped" watermark_name="${CI_WATERMARK_FILE:-.ci-watermark-sha}" watermark_file="${CARGO_TARGET_DIR:-}/${watermark_name}" if [ -n "${CARGO_TARGET_DIR:-}" ] && [ -f "$watermark_file" ]; then watermark_sha=$(cat "$watermark_file") if git cat-file -e "${watermark_sha}^{commit}" 2>/dev/null; then # Every path that differs between the watermark commit and HEAD — # added, modified, or deleted — content-wise, regardless of what any # commit along the way claims its own timestamp is. git diff --name-only "$watermark_sha" HEAD > "$changed_since_watermark" watermark_status="diffing against watermark ${watermark_sha} in ${CARGO_TARGET_DIR}/${watermark_name}" else # The watermark names a commit this checkout can no longer see — a # force-push or rebase rewrote history out from under it. Trusting the # diff would silently compare HEAD against a commit with no real # ancestry relationship to it. Fall back to treating every tracked file # as changed since the (unknowable) last build: safe (over-invalidates, # never under-invalidates) and self-heals the moment the next green run # records a fresh, reachable watermark. cp "$tracked" "$changed_since_watermark" watermark_status="watermark ${watermark_sha} unreachable (force-push/rebase?) in ${CARGO_TARGET_DIR}/${watermark_name} — treating every tracked file as changed since last build" fi else : > "$changed_since_watermark" if [ -n "${CARGO_TARGET_DIR:-}" ]; then watermark_status="no watermark yet in ${CARGO_TARGET_DIR}/${watermark_name} (first run against this namespace) — skipped" fi fi # For every path in the diff, override its git-log timestamp with "now" # ($start, this run's own wall-clock start) — provably later than the real # completion time of the build the watermark names, unlike any commit's # authored/committer date. Everything NOT in the diff keeps its git-log # timestamp exactly as before. if [ -s "$changed_since_watermark" ]; then # Re-derive the count from the intersection with $to_touch (rather than # `wc -l < "$changed_since_watermark"` directly) so the reported number # only counts paths this script is actually about to stamp differently, # not every path `git diff` names (which, unlike $to_touch, is not # restricted to currently-tracked files — e.g. a path deleted since the # watermark commit). watermark_count=$(awk -F'\t' 'NR == FNR { c[$0] = 1; next } $2 in c' "$changed_since_watermark" "$to_touch" | wc -l | tr -d ' ') awk -F'\t' -v now="$start" ' NR == FNR { changed[$0] = 1; next } { if ($2 in changed) print now "\t" $2; else print } ' "$changed_since_watermark" "$to_touch" > "$to_touch_overridden" mv "$to_touch_overridden" "$to_touch" fi # Any tracked path whose history couldn't resolve (only possible under an # unexpectedly shallow checkout, since `fetch-depth: 0` on the checkout step # means full history is normally present) falls back to a fixed constant # epoch rather than being left alone at checkout-time "now". What Cargo's # fingerprint check needs is an mtime that's STABLE run-over-run for # unchanged content, not necessarily a historically accurate one — a # constant epoch gives that stability so the file still gets cache reuse # instead of permanently missing the cache on every single run. awk -F'\t' ' NR == FNR { seen[$2] = 1; next } !($0 in seen) ' "$to_touch" "$tracked" > "$missing" touched=0 while IFS=$'\t' read -r ts path; do [ -e "$path" ] || [ -L "$path" ] || continue # `-h`: set the mtime of a git-tracked symlink itself, never the target it # resolves to. No path in this repo is currently a tracked symlink, but a # future one (e.g. a shared-config symlink into a dotfiles # checkout) would otherwise get its mtime applied # to whatever it resolves to outside the repo — keeping `-h` costs nothing # today and avoids that surprise later. touch -h -d "@$ts" -- "$path" touched=$((touched + 1)) done < "$to_touch" fallback_count=0 if [ -s "$missing" ]; then while IFS= read -r path; do [ -e "$path" ] || [ -L "$path" ] || continue touch -h -d "@0" -- "$path" fallback_count=$((fallback_count + 1)) done < "$missing" fi # Directory pass: for every tracked file's full ancestor chain (excluding # the file's own leaf name), track the maximum timestamp seen. A directory # with no tracked file directly in it but with tracked files somewhere in # its subtree (e.g. `crates/`) still accumulates a max from those deeper # files, since every ancestor level of every file's path is a separate # prefix entry here. Fixed-epoch fallback entries (ts=0) are included too — # harmless, since 0 can never win a max against any real commit timestamp. cat "$to_touch" > "$all_touched" if [ -s "$missing" ]; then awk '{ print "0\t" $0 }' "$missing" >> "$all_touched" fi awk -F'\t' ' { ts = $1; path = $2 n = split(path, parts, "/") prefix = "" for (i = 1; i < n; i++) { prefix = (prefix == "" ? parts[i] : prefix "/" parts[i]) if (!(prefix in maxts) || ts + 0 > maxts[prefix] + 0) maxts[prefix] = ts } if (!("." in maxts) || ts + 0 > maxts["."] + 0) maxts["."] = ts } END { for (d in maxts) printf "%s\t%s\n", maxts[d], d } ' "$all_touched" > "$dir_touch" dirs_touched=0 while IFS=$'\t' read -r ts dir; do [ -d "$dir" ] || continue touch -d "@$ts" -- "$dir" dirs_touched=$((dirs_touched + 1)) done < "$dir_touch" elapsed=$(( $(date +%s) - start )) echo "restore-mtimes: touched ${touched} files + ${dirs_touched} dirs of $(wc -l < "$tracked") tracked files from history in ${elapsed}s (${fallback_count} fell back to a fixed epoch, ${watermark_count} overridden to now: ${watermark_status})"