Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4da6e8d30f
|
@@ -287,6 +287,36 @@ Lowering either here needs no coordination: the arbiter then only defers a
|
||||
reclamation this side would already have permitted, which costs disk rather
|
||||
than correctness.
|
||||
|
||||
### Names this repo doesn't reclaim, but the arbiter depends on
|
||||
|
||||
The five names above are leftovers — dead trees gitdan's arbiter finds and
|
||||
deletes. Four other dot-prefixed names change that arbiter's *behaviour*
|
||||
without ever being a leftover: it reads them to make a correctness decision
|
||||
and never reclaims them. They are a real cross-repo dependency too, on the
|
||||
name's spelling rather than on the tree's lifetime — gitdan's
|
||||
`DEPENDED-UPON NAMES CONTRACT` block (in `scripts/ci-cache-reclaim.sh`) is the
|
||||
canonical description; this is the producer's half for the two we own.
|
||||
|
||||
| Name | Produced here by | Consequence of an unannounced rename |
|
||||
|---|---|---|
|
||||
| `.ci-lock-<id>` | `scripts/cache-lock.sh` (acquire/release) and `cache-lib.sh`'s `write_cache_lock()` | The arbiter's lock check silently stops matching — a live job's staging tree loses its liveness guard and becomes an ordinary reclaim candidate while still in use |
|
||||
| `.cache-last-used` | `cargo-cache/action.yml`, stamped every run | The arbiter falls back to directory mtime — silently reordering its eviction order, and possibly failing to recognise the directory as a cache dir at all |
|
||||
|
||||
Two more names are in gitdan's list — `.gitea-last-used` and `.ci-keep` — and
|
||||
neither is produced by anything in this repo, so there is no producer-side
|
||||
half to write here. `.gitea-last-used` is a naming convention individual
|
||||
repos used before adopting this shared action; nothing here writes it, and
|
||||
gitdan's script reads it only for compatibility with directories created
|
||||
under that older scheme. `.ci-keep` is a per-repo, hand-placed opt-out a
|
||||
consuming repo's own workflow drops directly into a cache directory it wants
|
||||
exempted from eviction — never something this action or its scripts write.
|
||||
|
||||
**Adding a fifth depended-upon name counts as much as renaming one of the two
|
||||
above.** If a future change here makes gitdan's arbiter start depending on the
|
||||
spelling of some new dot-prefixed name — a name it reads for a decision but
|
||||
never reclaims — that is exactly this category, and it needs the matching
|
||||
`DEPEND_*` entry on gitdan's side before it ships, not after.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
@@ -422,7 +452,7 @@ bash scripts/selftest.sh --fast # fixture-only suites, no compiler
|
||||
| suite | covers |
|
||||
|---|---|
|
||||
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published, and the publisher's log showing it waited on the consumer's own reader-lock marker before reclaiming a rotated snapshot |
|
||||
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and one scenario per check a hardlink clone is validated against**: a source rotated wholesale, a subtree silently lost from the walk, a copy that reports failure over a tree both other checks read as whole, and a source identity that resolved at neither end — plus a staging tree that could not be privately owned being discarded rather than published |
|
||||
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
|
||||
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
|
||||
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |
|
||||
@@ -442,7 +472,7 @@ key and asserts only what must be true whichever of them wins the rename.
|
||||
which places the interference inside the window rather than hoping it lands
|
||||
there. `prune-cache-selftest.sh` scenario 12 stubs `du`, so the pass's own
|
||||
measurement publishes a reader marker strictly between its check and its
|
||||
unlink. `seed-target-dir-selftest.sh` uses the shape five times over, on two
|
||||
unlink. `seed-target-dir-selftest.sh` uses the shape four times over, on two
|
||||
different commands. Scenarios 8a, 8b and 8c stub `cp`, so the consumer's own
|
||||
clone is what rotates the snapshot underneath it, loses a subtree of its own
|
||||
source, or reports a failure over a tree that is in fact whole — each strictly
|
||||
@@ -453,14 +483,9 @@ could not be READ rather than on one that changed, and the only way to make
|
||||
that the sole witness is to fail the identity reads while the copy between
|
||||
them succeeds. 8a also starts a second real process — the actual
|
||||
`publish-snapshot.sh` — but the stub is what fixes where its swap lands; the
|
||||
concurrency is incidental to the determinism. Scenario 11 reuses 8a's exact
|
||||
stub and the same forced rotation, but reads a different witness: not the
|
||||
consumer's own checks, but a line in the *publisher's* log reporting that it
|
||||
waited on the reader marker this consumer's clone wrote — `reader_lock_acquire`
|
||||
is exercised by 8a already, but nothing asserts it actually fired until this
|
||||
scenario reads that line back. Every stub asserts that it fired, because a
|
||||
scenario whose interference silently did not happen passes for the wrong
|
||||
reason.
|
||||
concurrency is incidental to the determinism. Every stub asserts that it
|
||||
fired, because a scenario whose interference silently did not happen passes
|
||||
for the wrong reason.
|
||||
|
||||
**A synthetic stand-in for the other side, where that artefact *is* the
|
||||
contract.** `publish-snapshot-selftest.sh` scenarios 6 to 8 hold a
|
||||
@@ -492,15 +517,6 @@ how both `[ "$cp_rc" -eq 0 ]` and `[ "$i_before" != missing ]` sat unpinned
|
||||
(issue #5) while looking well covered. Isolating a check means constructing
|
||||
the state only it can see, not the state that trips several at once.
|
||||
|
||||
Scenario 11 applies the same discipline to a witness outside the clone
|
||||
entirely: not which of several checks inside `hardlink_clone_into` caught a
|
||||
fault, but whether `reader_lock_acquire`'s marker was observed by anything
|
||||
outside it at all. The consumer's own log and exit status are silent either
|
||||
way — a run with the marker deleted still succeeds — so what is asserted is
|
||||
one line in the *publisher's* log reporting that it waited. Deleting
|
||||
`reader_lock_acquire` (issue #10) leaves that line unwritten without failing
|
||||
anything else in the suite.
|
||||
|
||||
The action YAML holds no logic beyond wiring; everything testable lives in
|
||||
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
|
||||
the actions reach their shared scripts through
|
||||
|
||||
@@ -165,6 +165,14 @@ runs:
|
||||
# touched unconditionally every run: a run that hits the cache for every
|
||||
# crate may write nothing at all inside the tree, which would make a
|
||||
# just-used directory look stale to the eviction pass.
|
||||
#
|
||||
# `.cache-last-used` IS A CROSS-REPO CONTRACT NAME with daniel/gitdan's
|
||||
# `scripts/ci-cache-reclaim.sh`, which reads this exact marker for its own
|
||||
# LRU ordering and to recognise a directory as a cache dir at all. Rename
|
||||
# it here without a matching change there and that script silently falls
|
||||
# back to directory mtime for both — see README.md's "Scratch names in a
|
||||
# cache root" section and the `DEPENDED-UPON NAMES CONTRACT` block in
|
||||
# gitdan's script.
|
||||
- shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
@@ -14,6 +14,19 @@
|
||||
# until STALE_LOCK_SECONDS, after which it is treated as abandoned by a job
|
||||
# the runner killed before it reached its own release step. Honouring a lock
|
||||
# forever would let one crashed job pin a directory permanently.
|
||||
#
|
||||
# `.ci-lock-<id>` IS A CROSS-REPO CONTRACT NAME with daniel/gitdan's
|
||||
# `scripts/ci-cache-reclaim.sh`: that host-level arbiter reads this same
|
||||
# marker (its `leftover_is_locked()`) to decide whether a leftover it is about
|
||||
# to reclaim is still held open by a live job. `cache-lib.sh`'s
|
||||
# `write_cache_lock()` writes the identical shape independently of this
|
||||
# script. Rename or reshape this marker here without a matching change on
|
||||
# gitdan's side and the arbiter's lock check silently stops matching — a live
|
||||
# job's staging tree loses its liveness guard and becomes an ordinary reclaim
|
||||
# candidate while still in use. See the "Scratch names in a cache root" section
|
||||
# in README.md, and daniel/gitdan's `DEPENDED-UPON NAMES CONTRACT` block in
|
||||
# `scripts/ci-cache-reclaim.sh`, for the full arrangement — this repo's half
|
||||
# is: don't rename `.ci-lock-` without telling that script.
|
||||
set -euo pipefail
|
||||
|
||||
MODE="${1:?usage: cache-lock.sh acquire|release <dir> <lock-id>}"
|
||||
|
||||
@@ -74,17 +74,6 @@
|
||||
# onto one set of fingerprints. That is the silent stale-reuse bug the
|
||||
# whole scheme exists to prevent, so the failure has to abort the clone
|
||||
# rather than be swallowed.
|
||||
# 11. THE CONSUMER'S MARKER ACTUALLY STOPS THE PUBLISHER — reader_lock_acquire
|
||||
# is exercised (not faked, unlike publish-snapshot-selftest.sh's scenario
|
||||
# 6) by a real hardlink_clone_into racing a real, concurrent
|
||||
# publish-snapshot.sh republish of the exact snapshot being cloned. Pins
|
||||
# that the publisher OBSERVABLY WAITS on this consumer's marker — its own
|
||||
# log reports entering the drain wait — rather than only that the run
|
||||
# succeeds, which stayed green with the marker call deleted (issue #10).
|
||||
# Kept as its own scenario, not folded into 8a, because 8a already pins
|
||||
# exactly one property (the identity check) for exactly one mutant, and
|
||||
# the suite's one-scenario-one-mutant diagonal across 8a to 8d and 10 is
|
||||
# deliberate.
|
||||
set -euo pipefail
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
. "$script_dir/cache-lib.sh"
|
||||
@@ -646,84 +635,5 @@ leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' \
|
||||
ok "no staging or reader-marker scratch left behind"
|
||||
rm -f "$scratch/bin/cp"
|
||||
|
||||
echo
|
||||
echo "=== 11: the consumer's marker actually stops the publisher ==="
|
||||
# hardlink_clone_into's reader_lock_acquire call is exercised here, not
|
||||
# faked. publish-snapshot-selftest.sh's scenario 6 stands a hand-written
|
||||
# marker file in for "a consumer whose clone outlasts the grace period" — a
|
||||
# deliberate simplification that does not need this consumer's clone code at
|
||||
# all, so it cannot tell reader_lock_acquire apart from no marker existing.
|
||||
# This scenario forces the two real scripts to race on the same snapshot: a
|
||||
# consumer hardlink-cloning it, and a publisher republishing it out from
|
||||
# under that clone, exactly as 8a does to force the identity check — but what
|
||||
# is pinned here is not the consumer's response to the rotation (8a's
|
||||
# property), it is that the PUBLISHER, reading the SAME marker this
|
||||
# consumer's clone wrote, is observed to have entered its drain wait. Deleting
|
||||
# reader_lock_acquire (issue #10) leaves the marker never written: the
|
||||
# publisher's live_reader_count sees zero readers on its first check and
|
||||
# proceeds straight to reclaiming the rotated generation, silently — the run
|
||||
# still succeeds, and nothing about its own outcome says so, only the absence
|
||||
# of a line in the publisher's log.
|
||||
INTLK=$(cache_key feat/interlock-consumer)
|
||||
INTLK_BASE=$(cache_key release/5)
|
||||
TAG_INTLK=jobIntlk
|
||||
rm -rf "$root/snapshot-$INTLK_BASE" "$root/target-$INTLK_BASE"
|
||||
make_tree "$root/snapshot-$INTLK_BASE" intlk-gen1
|
||||
make_tree "$root/target-$INTLK_BASE" intlk-gen2
|
||||
intlk_gen1_inode=$(stat -c '%i' "$root/snapshot-$INTLK_BASE")
|
||||
|
||||
cat > "$scratch/bin/cp" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
# Fires once, only on the consumer's own top-level hardlink clone — identified
|
||||
# by its destination, the seed's private staging path. Every other cp in the
|
||||
# process tree (the unshare copies, and the publisher's own staging clone)
|
||||
# falls through to the real one.
|
||||
if [ "\${@: -1}" = "$root/.stage-$TAG_INTLK" ] && [ ! -e "$scratch/firedIntlk" ]; then
|
||||
: > "$scratch/firedIntlk"
|
||||
rc=0; "$real_cp" "\$@" || rc=\$?
|
||||
# Concurrently: a real, second publish of the exact snapshot this consumer
|
||||
# is cloning — the republish that, without the marker this consumer's clone
|
||||
# holds, would reclaim the generation out from under it.
|
||||
( CACHE_READ_GRACE_SECONDS=10 bash "$script_dir/publish-snapshot.sh" "$INTLK_BASE" "$root" pubIntlk \
|
||||
> "$scratch/logPubIntlk" 2>&1
|
||||
echo \$? > "$scratch/rcPubIntlk" ) &
|
||||
# Hand control back only once the swap is on disk, so the identity read
|
||||
# immediately after this cp is guaranteed to resolve to the new generation
|
||||
# — the same technique 8a uses to force the interleaving rather than hope
|
||||
# for it. reader_lock_release does not run until AFTER this script exits,
|
||||
# so the marker stays live for the publisher's whole swap-and-scan.
|
||||
deadline=\$(( \$(date +%s) + 60 ))
|
||||
while [ "\$(stat -c '%i' "$root/snapshot-$INTLK_BASE" 2>/dev/null)" = "$intlk_gen1_inode" ]; do
|
||||
[ "\$(date +%s)" -lt "\$deadline" ] || { echo "stub cp: the publisher never swapped the snapshot" >&2; exit 92; }
|
||||
sleep 0.05
|
||||
done
|
||||
exit \$rc
|
||||
fi
|
||||
exec "$real_cp" "\$@"
|
||||
EOF
|
||||
chmod +x "$scratch/bin/cp"
|
||||
|
||||
rcIntlk=0
|
||||
seed_with_stub "$INTLK" "$INTLK_BASE" "$root" "$TAG_INTLK" > "$scratch/logIntlk" 2>&1 || rcIntlk=$?
|
||||
[ -e "$scratch/firedIntlk" ] \
|
||||
|| fail "the stubbed cp never fired: the consumer never raced the publisher, so this scenario proves nothing"
|
||||
ok "the consumer's clone raced a real, concurrent republish of its own source"
|
||||
wait_for_file "$scratch/rcPubIntlk" "the publisher never finished"
|
||||
[ "$(cat "$scratch/rcPubIntlk")" = "0" ] || { tail -40 "$scratch/logPubIntlk"; fail "publish-snapshot.sh exited non-zero"; }
|
||||
[ "$rcIntlk" = "0" ] || { tail -40 "$scratch/logIntlk"; fail "the seed exited non-zero"; }
|
||||
ok "both sides of the race completed"
|
||||
|
||||
# The property under test: not that the run succeeded, but that the publisher
|
||||
# itself reports having found a live reader and waited on it. This is silent
|
||||
# in the consumer's own log and in the run's exit status alike — only the
|
||||
# publisher's log carries it.
|
||||
grep -q "readers: waiting for 1 in-flight clone(s) of snapshot-${INTLK_BASE}" "$scratch/logPubIntlk" \
|
||||
|| { tail -40 "$scratch/logPubIntlk"; fail "the publisher never reported waiting on the consumer's reader marker — the interlock did not observably engage"; }
|
||||
ok "the publisher observably waited on the consumer's own reader marker before reclaiming the rotated snapshot generation"
|
||||
leftovers=$(find "$root" -maxdepth 1 \( -name '.stage-*' -o -name '.reading-*' -o -name '.publish-*' \) -print)
|
||||
[ -z "$leftovers" ] || fail "scratch left behind: ${leftovers}"
|
||||
ok "no staging, reader-marker or deferred-generation scratch left behind"
|
||||
rm -f "$scratch/bin/cp"
|
||||
|
||||
echo
|
||||
echo "seed-target-dir-selftest: ${pass_count} assertions passed"
|
||||
|
||||
Reference in New Issue
Block a user