gitdan-actions/README.md

gitdan-actions

Shared Gitea Actions composite actions for the gitdan forge.

Currently one thing, done properly: cargo-cache — a persistent, per-branch Cargo build cache for self-hosted Gitea runners, where a pull request's cache is a near-free hardlink clone of an immutable snapshot its base branch published.

Final home: this repository will live at daniel/gitdan-actions. Pin that path in uses: once the transfer completes.

---

Why this exists

Two of this forge's Rust projects independently built the same idea and each got one half right.

	seeding mechanism	seed source
project A	cp -al hardlink clone — near-free, cost scales with inode count, not bytes	the base branch's live target dir — races a build that is still writing
project B	cp -a full copy — sound, but ~35 GB duplicated per branch	a published immutable snapshot — nothing ever writes it while it is read

This action is the diagonal: hardlink-clone from a published snapshot. Cheap like A, sound like B. It also closes a latent race in A by construction (the seed is staged and swapped in with one atomic rename) rather than relying on the runner having a single execution slot.

One thing neither project had, and the reason the clone is not a plain cp -al: a build inside a hardlink clone does mutate the directory it was cloned from. Cargo writes real artifacts by replacing them, but writes its metadata — and build scripts write their OUT_DIR — with a plain truncating write, straight through the shared inode. Under CARGO_UNSTABLE_CHECKSUM_FRESHNESS the file that gets corrupted is .fingerprint/<unit>/dep-<target>, which holds the per-source checksums that decide freshness, and the failure is silent stale-artifact reuse rather than a slow build. scripts/hardlink-clone-selftest.sh reproduces it as an explicit control and asserts the fix. The fix is to hardlink the artifacts (the GB) and real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory, against 100% for a full copy.

---

Quick start

name: CI
on:
  push:
    branches: [main, dev]
  pull_request:
    branches: [main, dev]

jobs:
  ci:
    runs-on: ubuntu-latest
    # REQUIRED, and it cannot come from the action: `container.volumes` is a
    # job-level property, so the persistent cache volume must be declared
    # here. Use a volume name unique to this repository.
    container:
      volumes:
        - myrepo-ci-target:/cache
    steps:
      - uses: actions/checkout@v4
        with:
          # REQUIRED. The mtime restore walks every commit that ever touched a
          # tracked file; a depth-1 checkout makes every file resolve to the
          # tip commit and the cache stops working. The action fails loudly
          # rather than silently degrading if this is missing.
          fetch-depth: 0

      - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
        with:
          protected-branches: 'dev main'

      # ... toolchain, system deps, and the build itself. CARGO_TARGET_DIR is
      # already exported to the job environment by the step above.
      - run: cargo clippy --workspace --all-targets -- -D warnings
      - run: cargo test --workspace

      # After the build succeeds: record the watermark, and publish a snapshot
      # if this run is a push to a protected branch.
      - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1

      # Release this job's cache lock even when the build failed, so the
      # eviction pass does not have to wait out the staleness grace period.
      - if: always()
        uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
        with:
          mode: release-lock

Recommended alongside it, in the workflow's env: block:

env:
  CARGO_INCREMENTAL: 0                  # per-run bloat on a persistent volume
  CARGO_PROFILE_DEV_DEBUG: line-tables-only
  CARGO_PROFILE_TEST_DEBUG: line-tables-only
  # Nightly only. Content-addressed freshness instead of mtime-based — a
  # strictly stronger guarantee, complementary to the mtime restore (which
  # still covers directory-form `rerun-if-changed` build-script watches).
  CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"

---

How it works

/cache/
  target-<key>       one per ref. Where a build actually runs.
  snapshot-<key>     one per publisher ref. Immutable between publishes;
                     the only thing a consumer ever clones from.

<key> is the ref sanitised to a safe path component, capped at 48 characters, plus an 8-hex SHA-1 prefix of the raw ref. The hash is not decoration: feat/foo and feat-foo sanitise identically and would otherwise share one directory.

A pull request run resolves its own key from github.head_ref (not ref_name, which on a pull_request event is a synthetic merge ref that changes on every push) and its base key from github.base_ref. If it has no directory yet, it hardlink-clones snapshot-<base> into a staging path, strips Cargo's lock files, real-copies everything Cargo writes in place, and renames the staging path into target-<own>.

A push to a protected branch has no base to layer over. It builds in its own directory and, if the build goes green, republishes it as snapshot-<own>: stage a clone, rename the old snapshot aside, rename the new one in, then reclaim the old one once nothing is still reading it. Consumers only ever observe a complete snapshot or none at all.

Concurrency, on the destination. Two jobs sharing one cache key each stage under their own tag and race on one atomic rename; the loser discards its staging copy. There is no window in which a partially-populated directory is visible under the final name. Two jobs then building in the same directory is Cargo's own .cargo-lock territory, which is what that lock is for.

Concurrency, on the source. The atomic rename is necessary and not sufficient, because renaming a truncated tree publishes a truncated tree atomically. A clone reads its source over many seconds, and a publisher rotating that source unlinks the generation being read — at which point cp -al can silently omit a subtree it never saw, and report success. Two mechanisms, both required:

• The publisher does not unlink under a reader. A consumer publishes a .reading-<snapshot>-<tag> marker before it resolves the snapshot path; the publisher scans for markers after its first rename. A consumer holding the old generation therefore published its marker before that scan and cannot be missed, and one that arrives after the scan necessarily resolves to the new generation. The publisher waits for readers to drain (read-grace-seconds, default 300) and, if they do not, defers the reclamation rather than forcing it — the old generation stays on disk and is swept by a later publish.
• The consumer verifies its own clone. Every attempt checks cp -al's exit status, the source directory's inode before and after (a wholesale replacement mid-walk would otherwise splice two generations), and the entry count (the only signal for a subtree unlinked before its parent was listed — there is no error to read). A tree that fails any of the three is deleted and the clone retried; one that fails the last attempt fails the job. A partial tree never reaches the final name. The two counts cost one metadata walk each: measured on ext4 with a warm cache over a 78,554-entry tree, 44 ms per walk against 3,126 ms for the cp -al they guard — about 2.8%.

What this does and does not guarantee. Four separate claims, deliberately not collapsed into one:

• A publisher rotating a snapshot cannot tear a clone of it — by construction. This is the case zemyna #911 is about, and the marker ordering above is what closes it: the publisher's scan cannot miss a consumer that resolved the old generation, and on timeout it defers the unlink rather than forcing it. On this path the consumer's own verification is a redundant second check, not the thing holding the guarantee up.
• Nor can the eviction pass — by the same construction. A cache chosen for eviction is renamed aside and only then re-examined for readers, so the scan the unlink rests on happens strictly after that rename, exactly as the publisher's does. A consumer that resolved the directory published its marker before the scan and so cannot be missed; one arriving after the rename cannot resolve the path at all and starts cold instead. A cache claimed inside that window is put back under its own name, and one whose name a concurrent seed has taken in the meantime is left aside and reclaimed by a later pass once its readers drain. That later pass leaves an aside directory alone until it has been set aside for a minute — not for the unlink's sake, which the ordering proof above already covers, but so that a pass still deciding about one is never mistaken for a pass that died holding it. That settle window is a bound rather than a construction, and it is the only part of this that is. Until the rest of it was structural it was merely policy — snapshots belong to protected refs, protected refs are never eviction candidates — a property held by vigilance rather than by construction.
• Every other way the source can change mid-clone is detected, not prevented. A seed-fallback-dir pointing at a directory something else writes has no interlock at all. There, the per-attempt verification is what stands between a torn read and a corrupt cache: the clone is retried (CACHE_CLONE_ATTEMPTS, default 4) and then fails the job loudly — never seeded partially, and never degraded to a silent cold build.
• Disk reclamation is bounded, not immediate. A consumer slower than the grace period leaves one extra snapshot generation of directory entries on the volume until a later publish sweeps it; a consumer whose job was killed outright holds it until its marker passes reader-stale-seconds. The residual is capped at one deferred generation per publisher ref, and its real cost is close to inode count rather than byte count, since the artifacts are hardlinked to whatever cloned them. A declined eviction is never unlinked under the job that claimed it; what it costs meanwhile is disk, normally as the cache restored under its own name and otherwise as one set aside for a later pass to reclaim.

Eviction runs three passes: caches for branches that no longer exist on origin are removed unconditionally; then, only if free space is under the threshold, live caches are evicted oldest-first; then, as a last resort, this run's own cache. Protected refs and any cache held open by a running job are never candidates. Within the pressure pass, target-* directories are evicted before snapshot-* ones — the reverse of the obvious order, because a snapshot is hardlinked to everything cloned from it, so removing one frees almost no real bytes while costing every future PR its warm start.

File mtimes. actions/checkout stamps every file with "now", which makes every crate look changed to Cargo's mtime-based freshness check — a persistent target directory buys nothing without fixing that. Each tracked file is restored to the timestamp of the most recent commit that touched it, plus a watermark override: for any file that changed since this cache's own last successful build, "now" is stamped instead. That override is what makes a merge safe, since a merge can introduce a commit authored before this cache's last build, where the historically-correct mtime is exactly the wrong answer.

---

Inputs

cargo-cache

input	default	meaning
cache-root	/cache	mount point of the persistent volume inside the job container
protected-branches	dev main	refs that publish snapshots and are never evicted
min-free-percent	10	prune when free space drops below this
restore-mtimes	true	restore tracked-file mtimes from git history
prune	true	run the eviction pass
liveness-prune	true	within eviction, remove caches for branches gone from origin
own-ref	(auto)	override; defaults to github.head_ref, else github.ref_name
base-ref	(auto)	override; defaults to github.base_ref (empty on push)
seed-fallback-dir	(empty)	absolute path to seed from when no snapshot exists — for migrating off an existing flat cache
watermark-file	.ci-watermark-<job>-sha	must differ per job when two jobs share one cache key
lock-id	<job>-<run_id>	identifies this job's cache lock
stale-lock-seconds	7200	age past which another job's lock is treated as abandoned

Outputs: target-dir, cache-key, seeded-from (own | base-snapshot | own-snapshot | fallback-dir | concurrent-peer | cold).

Exports to the job environment: CARGO_TARGET_DIR, CARGO_CACHE_ROOT, CARGO_CACHE_KEY, CARGO_CACHE_LOCK_ID, CARGO_CACHE_SCRIPTS, CI_WATERMARK_FILE.

cargo-cache-publish

input	default	meaning
cache-root	/cache	must match the consume action
protected-branches	dev main	refs that publish snapshots
mode	publish	publish, or release-lock for the if: always() step
own-ref	(auto)	override; defaults to github.head_ref, else github.ref_name
publish-on-events	push	events on which a protected ref actually publishes
read-grace-seconds	300	how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced
reader-stale-seconds	7200	age past which a consumer's read marker is treated as abandoned by a killed job
record-watermark	true	record HEAD as this cache's watermark (PR runs too)

publish-on-events defaults to push on purpose: a pull_request run from dev into main has own-ref dev and would otherwise publish a snapshot of a merge-preview build, which is not what dev is.

---

Multiple jobs in one workflow

Jobs sharing a cache key (a ci job and a wasm job on the same branch, say) each need their own watermark file. A shared one breaks the moment two jobs run in sequence within one trigger: job A advances the watermark to HEAD, and job B then reads that just-advanced value, computes an empty diff, and loses the merge protection entirely. The default (.ci-watermark-<job>-sha) already gives each job its own; only override watermark-file if you also override lock-id, and then keep both distinct per job.

---

Constraints of this runner

• The repository must be public. act_runner fetches actions by anonymous git clone and has no credentialed-fetch option, so a private action repository simply fails to resolve. Nothing secret goes in here.
• uses: needs the absolute URL. A bare owner/repo resolves against github.com, because Gitea's DEFAULT_ACTIONS_URL is unset — and it has to stay unset, or actions/checkout, dtolnay/rust-toolchain and taiki-e/install-action stop resolving.
• The runner pre-fetches every referenced action before running any step, so a bad action reference fails the job at step 0 rather than where it is used.
• container.volumes is job-level and cannot be set from inside a composite action. The consuming workflow declares it; see the quick start.
• The cache volume is ext4 — no reflink support, which is precisely why hardlinks are the mechanism that makes cloning cheap.

---

Versioning

Pin @v1. It is a moving major tag: fixes and backward-compatible inputs move it forward, and anything that would break an existing consumer gets v2 instead. Pin a commit SHA if you want a frozen version.

The alternative is immutable release tags — v1.0.1, v1.0.2, … — with every consumer edited to point at the new one per fix. That is the safer model in general and the wrong one here. What it buys is the ability to hold one consumer back on a known-good version; what it costs is a PR in every consumer repo per fix, and its real failure mode with two consumers and one operator is that the second one is simply never updated and quietly runs a version nobody is testing. The moving pointer makes a release one action with one blast radius, which is the thing worth being deliberate about. Anyone who wants the immutable behaviour already has it, by pinning a SHA.

What @v1 promises is that whatever it points at works with the inputs documented above, spelled as they are documented. A change that renames or removes an input, changes a default in a way that changes behaviour, or requires something new of the consuming workflow — another container.volumes entry, another permission — is a v2, not a v1 move. Everything else moves v1: correctness fixes, new optional inputs, and anything internal to scripts/.

Moving the tag is a release step, and it is the operator's. Merging to main ships nothing to anybody. v1 is a lightweight tag and does not follow a branch, so until it is re-pointed every consumer keeps fetching the commit it already named, whatever main now says. The gap is deliberate: re-pointing v1 changes what another repository's CI executes on its next run, so it is a decision taken once, knowingly, after the merge — never something a merge does by itself.

git fetch origin
git tag -f v1 origin/main
git push -f origin v1
git ls-remote --tags origin v1     # must equal git rev-parse origin/main

Downstream are emowheel, which pins cargo-cache@v1 and cargo-cache-publish@v1 across its CI workflow, and zemyna, migrating to the same pin. Both pick a move up on their next run with no change on their side, which is the whole point of the moving pointer and also the reason the move is not automatic.

---

Development

bash scripts/selftest.sh          # everything (~1 min; needs cargo)
bash scripts/selftest.sh --fast   # fixture-only suites, no compiler

suite	covers
hardlink-clone-selftest.sh	that a build in a clone cannot mutate its source — with a control proving a raw cp -al does. Needs a real compiler.
seed-target-dir-selftest.sh	seed-source preference, lock-file stripping, two jobs racing on one cache key, and a seed whose source is rotated — or silently loses a subtree — underneath its clone: the two ways a hardlink clone tears
publish-snapshot-selftest.sh	the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone
prune-cache-selftest.sh	liveness, protection, locking, eviction order, self-clear, and that a cache a job claims inside the check-to-unlink window survives it — against a real scratch origin
restore-mtimes-selftest.sh	the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler.

Every suite runs the actual script, not a reimplementation of its logic, and every fix scenario is paired with a control that reproduces the bug — a scenario that passes either way proves nothing.

The concurrency scenarios take one of three shapes, and none of them races for the interleaving its assertion depends on.

A genuine race whose asserted invariant holds under any interleaving. seed-target-dir-selftest.sh scenario 7 starts two real seeds on one cache key and asserts only what must be true whichever of them wins the rename.

A PATH stub on a command the code under test calls at a known point, which places the interference inside the window rather than hoping it lands there. prune-cache-selftest.sh scenario 12 stubs du, so the pass's own measurement publishes a reader marker strictly between its check and its unlink; seed-target-dir-selftest.sh scenarios 8a and 8b stub cp, so the consumer's own clone is what rotates the snapshot underneath it, or what loses a subtree of its own source, strictly inside the identity window. 8a does start a second real process — the actual publish-snapshot.sh — but the stub is what fixes where its swap lands; the concurrency is incidental to the determinism. Every stub asserts that it fired, because a scenario whose interference silently did not happen passes for the wrong reason.

A synthetic stand-in for the other side, where that artefact is the contract. publish-snapshot-selftest.sh scenarios 6 to 8 hold a .reading-* marker instead of running a slow consumer: the marker is the whole agreement between reader and publisher, so holding one is being a reader, and racing a real one would make the suite's runtime the thing under test.

Gating the interfering step on observed progress of the step it interferes with was an earlier answer here, and it is not one: seeing that a walk has started says nothing about where it will be when the interference lands, so the assertion downstream held only some of the time (issue #3). No scenario does it any more.

Where more than one guard could catch a fault, a scenario should assert which one did — otherwise deleting the guard under test leaves the suite green because a sibling fires in its place. Scenarios 8a and 8b of the seed suite do; scenario 9 of the same suite does not yet, which is why a mutation survives it (issue #5).

The action YAML holds no logic beyond wiring; everything testable lives in scripts/. A composite action needs shell: bash on every run: step, and the actions reach their shared scripts through ${{ github.action_path }}/../scripts, which works because the runner clones the whole repository when it fetches an action.