gitdan-actions/README.md

# gitdan-actions

Shared Gitea Actions composite actions for the gitdan forge.

Currently one thing, done properly: **`cargo-cache`** — a persistent,
per-branch Cargo build cache for self-hosted Gitea runners, where a pull
request's cache is a near-free hardlink clone of an immutable snapshot its
base branch published.

> **Final home:** this repository will live at `daniel/gitdan-actions`. Pin
> that path in `uses:` once the transfer completes.

---

## Why this exists

Two of this forge's Rust projects independently built the same idea and each
got one half right.

| | seeding mechanism | seed source |
|---|---|---|
| project A | `cp -al` hardlink clone — near-free, cost scales with inode count, not bytes | the base branch's **live** target dir — races a build that is still writing |
| project B | `cp -a` full copy — sound, but ~35 GB duplicated per branch | a **published immutable snapshot** — nothing ever writes it while it is read |

This action is the diagonal: **hardlink-clone from a published snapshot.**
Cheap like A, sound like B. It also closes a latent race in A by construction
(the seed is staged and swapped in with one atomic rename) rather than relying
on the runner having a single execution slot.

One thing neither project had, and the reason the clone is not a plain
`cp -al`: **a build inside a hardlink clone does mutate the directory it was
cloned from.** Cargo writes real artifacts by replacing them, but writes its
metadata — and build scripts write their `OUT_DIR` — with a plain truncating
write, straight through the shared inode. Under
`CARGO_UNSTABLE_CHECKSUM_FRESHNESS` the file that gets corrupted is
`.fingerprint/<unit>/dep-<target>`, which holds the per-source checksums that
decide freshness, and the failure is silent stale-artifact reuse rather than a
slow build. `scripts/hardlink-clone-selftest.sh` reproduces it as an explicit
control and asserts the fix. The fix is to hardlink the artifacts (the GB) and
real-copy the metadata (the MB) — about 3.7% of a Bevy-sized target directory,
against 100% for a full copy.

---

## Quick start

```yaml
name: CI
on:
  push:
    branches: [main, dev]
  pull_request:
    branches: [main, dev]

jobs:
  ci:
    runs-on: ubuntu-latest
    # REQUIRED, and it cannot come from the action: `container.volumes` is a
    # job-level property, so the persistent cache volume must be declared
    # here. Use a volume name unique to this repository.
    container:
      volumes:
        - myrepo-ci-target:/cache
    steps:
      - uses: actions/checkout@v4
        with:
          # REQUIRED. The mtime restore walks every commit that ever touched a
          # tracked file; a depth-1 checkout makes every file resolve to the
          # tip commit and the cache stops working. The action fails loudly
          # rather than silently degrading if this is missing.
          fetch-depth: 0

      - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache@v1
        with:
          protected-branches: 'dev main'

      # ... toolchain, system deps, and the build itself. CARGO_TARGET_DIR is
      # already exported to the job environment by the step above.
      - run: cargo clippy --workspace --all-targets -- -D warnings
      - run: cargo test --workspace

      # After the build succeeds: record the watermark, and publish a snapshot
      # if this run is a push to a protected branch.
      - uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1

      # Release this job's cache lock even when the build failed, so the
      # eviction pass does not have to wait out the staleness grace period.
      - if: always()
        uses: https://gitdan.com/daniel/gitdan-actions/cargo-cache-publish@v1
        with:
          mode: release-lock
```

Recommended alongside it, in the workflow's `env:` block:

```yaml
env:
  CARGO_INCREMENTAL: 0                  # per-run bloat on a persistent volume
  CARGO_PROFILE_DEV_DEBUG: line-tables-only
  CARGO_PROFILE_TEST_DEBUG: line-tables-only
  # Nightly only. Content-addressed freshness instead of mtime-based — a
  # strictly stronger guarantee, complementary to the mtime restore (which
  # still covers directory-form `rerun-if-changed` build-script watches).
  CARGO_UNSTABLE_CHECKSUM_FRESHNESS: "true"
```

---

## How it works

```
/cache/
  target-<key>       one per ref. Where a build actually runs.
  snapshot-<key>     one per publisher ref. Immutable between publishes;
                     the only thing a consumer ever clones from.
```

`<key>` is the ref sanitised to a safe path component, capped at 48
characters, plus an 8-hex SHA-1 prefix of the *raw* ref. The hash is not
decoration: `feat/foo` and `feat-foo` sanitise identically and would otherwise
share one directory.

**A pull request run** resolves its own key from `github.head_ref` (not
`ref_name`, which on a `pull_request` event is a synthetic merge ref that
changes on every push) and its base key from `github.base_ref`. If it has no
directory yet, it hardlink-clones `snapshot-<base>` into a staging path, strips
Cargo's lock files, real-copies everything Cargo writes in place, and renames
the staging path into `target-<own>`.

**A push to a protected branch** has no base to layer over. It builds in its
own directory and, if the build goes green, republishes it as
`snapshot-<own>`: stage a clone, rename the old snapshot aside, rename the new
one in, then reclaim the old one once nothing is still reading it. Consumers
only ever observe a complete snapshot or none at all.

**Concurrency, on the destination.** Two jobs sharing one cache key each stage
under their own tag and race on one atomic rename; the loser discards its
staging copy. There is no window in which a partially-populated directory is
visible under the final name. Two jobs then building in the same directory is
Cargo's own `.cargo-lock` territory, which is what that lock is for.

**Concurrency, on the source.** The atomic rename is necessary and not
sufficient, because renaming a *truncated* tree publishes a truncated tree
atomically. A clone reads its source over many seconds, and a publisher
rotating that source unlinks the generation being read — at which point
`cp -al` can silently omit a subtree it never saw, and report success. Two
mechanisms, both required:

- **The publisher does not unlink under a reader.** A consumer publishes a
  `.reading-<snapshot>-<tag>` marker *before* it resolves the snapshot path;
  the publisher scans for markers *after* its first rename. A consumer holding
  the old generation therefore published its marker before that scan and
  cannot be missed, and one that arrives after the scan necessarily resolves
  to the new generation. The publisher waits for readers to drain
  (`read-grace-seconds`, default 300) and, if they do not, **defers** the
  reclamation rather than forcing it — the old generation stays on disk and is
  swept by a later publish.
- **The consumer verifies its own clone.** Every attempt checks `cp -al`'s
  exit status, the source directory's inode before and after (a wholesale
  replacement mid-walk would otherwise splice two generations), and the entry
  count (the only signal for a subtree unlinked before its parent was listed —
  there is no error to read). A tree that fails any of the three is deleted
  and the clone retried; one that fails the last attempt fails the job. A
  partial tree never reaches the final name. The two counts cost one metadata
  walk each: measured on ext4 with a warm cache over a 78,554-entry tree, 44 ms
  per walk against 3,126 ms for the `cp -al` they guard — about 2.8%.

**What this does and does not guarantee.** Four separate claims, deliberately
not collapsed into one:

- **A publisher rotating a snapshot cannot tear a clone of it — by
  construction.** This is the case zemyna #911 is about, and the marker
  ordering above is what closes it: the publisher's scan cannot miss a
  consumer that resolved the old generation, and on timeout it defers the
  unlink rather than forcing it. On this path the consumer's own verification
  is a redundant second check, not the thing holding the guarantee up.
- **Nor can the eviction pass — by the same construction.** A cache chosen for
  eviction is renamed aside and only then re-examined for readers, so the scan
  the unlink rests on happens strictly after that rename, exactly as the
  publisher's does. A consumer that resolved the directory published its
  marker before the scan and so cannot be missed; one arriving after the
  rename cannot resolve the path at all and starts cold instead. A cache
  claimed inside that window is put back under its own name, and one whose
  name a concurrent seed has taken in the meantime is left aside and reclaimed
  by a later pass once its readers drain. That later pass leaves an aside
  directory alone until it has been set aside for a minute — not for the
  unlink's sake, which the ordering proof above already covers, but so that a
  pass still deciding about one is never mistaken for a pass that died holding
  it. That settle window is a bound rather than a construction, and it is the
  only part of this that is. Until the rest of it was structural it was merely
  policy — snapshots belong to protected refs, protected refs are never
  eviction candidates — a property held by vigilance rather than by
  construction.
- **Every other way the source can change mid-clone is detected, not
  prevented.** A `seed-fallback-dir` pointing at a directory something else
  writes has no interlock at all. There, the per-attempt verification is what
  stands between a torn read and a corrupt cache: the clone is retried
  (`CACHE_CLONE_ATTEMPTS`, default 4) and then **fails the job loudly** —
  never seeded partially, and never degraded to a silent cold build.
- **Disk reclamation is bounded, not immediate.** A consumer slower than the
  grace period leaves one extra snapshot generation of directory entries on
  the volume until a later publish sweeps it; a consumer whose job was killed
  outright holds it until its marker passes `reader-stale-seconds`. The
  residual is capped at one deferred generation per publisher ref, and its
  real cost is close to inode count rather than byte count, since the
  artifacts are hardlinked to whatever cloned them. A declined eviction is
  never unlinked under the job that claimed it; what it costs meanwhile is
  disk, normally as the cache restored under its own name and otherwise as one
  set aside for a later pass to reclaim.

**Eviction** runs three passes: caches for branches that no longer exist on
origin are removed unconditionally; then, only if free space is under the
threshold, live caches are evicted oldest-first; then, as a last resort, this
run's own cache. Protected refs and any cache held open by a running job are
never candidates. Within the pressure pass, `target-*` directories are evicted
before `snapshot-*` ones — the reverse of the obvious order, because a
snapshot is hardlinked to everything cloned from it, so removing one frees
almost no real bytes while costing every future PR its warm start.

**File mtimes.** `actions/checkout` stamps every file with "now", which makes
every crate look changed to Cargo's mtime-based freshness check — a persistent
target directory buys nothing without fixing that. Each tracked file is
restored to the timestamp of the most recent commit that touched it, plus a
watermark override: for any file that changed since *this cache's own last
successful build*, "now" is stamped instead. That override is what makes a
merge safe, since a merge can introduce a commit authored before this cache's
last build, where the historically-correct mtime is exactly the wrong answer.

---

## Inputs

### `cargo-cache`

| input | default | meaning |
|---|---|---|
| `cache-root` | `/cache` | mount point of the persistent volume inside the job container |
| `protected-branches` | `dev main` | refs that publish snapshots and are never evicted |
| `min-free-percent` | `10` | prune when free space drops below this |
| `restore-mtimes` | `true` | restore tracked-file mtimes from git history |
| `prune` | `true` | run the eviction pass |
| `liveness-prune` | `true` | within eviction, remove caches for branches gone from origin |
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
| `base-ref` | *(auto)* | override; defaults to `github.base_ref` (empty on push) |
| `seed-fallback-dir` | *(empty)* | absolute path to seed from when no snapshot exists — for migrating off an existing flat cache |
| `watermark-file` | `.ci-watermark-<job>-sha` | must differ per job when two jobs share one cache key |
| `lock-id` | `<job>-<run_id>` | identifies this job's cache lock |
| `stale-lock-seconds` | `7200` | age past which another job's lock is treated as abandoned |

Outputs: `target-dir`, `cache-key`, `seeded-from` (`own` \| `base-snapshot` \|
`own-snapshot` \| `fallback-dir` \| `concurrent-peer` \| `cold`).

Exports to the job environment: `CARGO_TARGET_DIR`, `CARGO_CACHE_ROOT`,
`CARGO_CACHE_KEY`, `CARGO_CACHE_LOCK_ID`, `CARGO_CACHE_SCRIPTS`,
`CI_WATERMARK_FILE`.

### `cargo-cache-publish`

| input | default | meaning |
|---|---|---|
| `cache-root` | `/cache` | must match the consume action |
| `protected-branches` | `dev main` | refs that publish snapshots |
| `mode` | `publish` | `publish`, or `release-lock` for the `if: always()` step |
| `own-ref` | *(auto)* | override; defaults to `github.head_ref`, else `github.ref_name` |
| `publish-on-events` | `push` | events on which a protected ref actually publishes |
| `read-grace-seconds` | `300` | how long the swap waits for in-flight clones of the generation it replaces before reclaiming it; on timeout the reclamation is deferred, never forced |
| `reader-stale-seconds` | `7200` | age past which a consumer's read marker is treated as abandoned by a killed job |
| `record-watermark` | `true` | record HEAD as this cache's watermark (PR runs too) |

`publish-on-events` defaults to `push` on purpose: a `pull_request` run from
`dev` into `main` has `own-ref` `dev` and would otherwise publish a snapshot
of a merge-preview build, which is not what `dev` is.

---

## Multiple jobs in one workflow

Jobs sharing a cache key (a `ci` job and a `wasm` job on the same branch, say)
each need their **own** watermark file. A shared one breaks the moment two
jobs run in sequence within one trigger: job A advances the watermark to HEAD,
and job B then reads that just-advanced value, computes an empty diff, and
loses the merge protection entirely. The default (`.ci-watermark-<job>-sha`)
already gives each job its own; only override `watermark-file` if you also
override `lock-id`, and then keep both distinct per job.

---

## Constraints of this runner

- **The repository must be public.** act_runner fetches actions by anonymous
  git clone and has no credentialed-fetch option, so a private action
  repository simply fails to resolve. Nothing secret goes in here.
- **`uses:` needs the absolute URL.** A bare `owner/repo` resolves against
  github.com, because Gitea's `DEFAULT_ACTIONS_URL` is unset — and it has to
  stay unset, or `actions/checkout`, `dtolnay/rust-toolchain` and
  `taiki-e/install-action` stop resolving.
- **The runner pre-fetches every referenced action before running any step**,
  so a bad action reference fails the job at step 0 rather than where it is
  used.
- **`container.volumes` is job-level** and cannot be set from inside a
  composite action. The consuming workflow declares it; see the quick start.
- **The cache volume is ext4** — no reflink support, which is precisely why
  hardlinks are the mechanism that makes cloning cheap.

---

## Versioning

Pin `@v1`. It is a moving major tag: fixes and backward-compatible inputs move
it forward, and anything that would break an existing consumer gets `v2`
instead. Pin a commit SHA if you want a frozen version.

The alternative is immutable release tags — `v1.0.1`, `v1.0.2`, … — with every
consumer edited to point at the new one per fix. That is the safer model in
general and the wrong one here. What it buys is the ability to hold one
consumer back on a known-good version; what it costs is a PR in every consumer
repo per fix, and its real failure mode with two consumers and one operator is
that the second one is simply never updated and quietly runs a version nobody
is testing. The moving pointer makes a release one action with one blast
radius, which is the thing worth being deliberate about. Anyone who wants the
immutable behaviour already has it, by pinning a SHA.

**What `@v1` promises** is that whatever it points at works with the inputs
documented above, spelled as they are documented. A change that renames or
removes an input, changes a default in a way that changes behaviour, or
requires something new of the consuming workflow — another `container.volumes`
entry, another permission — is a `v2`, not a `v1` move. Everything else moves
`v1`: correctness fixes, new optional inputs, and anything internal to
`scripts/`.

**Moving the tag is a release step, and it is the operator's.** Merging to
`main` ships nothing to anybody. `v1` is a lightweight tag and does not follow
a branch, so until it is re-pointed every consumer keeps fetching the commit it
already named, whatever `main` now says. The gap is deliberate: re-pointing
`v1` changes what another repository's CI executes on its next run, so it is a
decision taken once, knowingly, after the merge — never something a merge does
by itself.

```bash
git fetch origin
git tag -f v1 origin/main
git push -f origin v1
git ls-remote --tags origin v1     # must equal git rev-parse origin/main
```

**Downstream** are emowheel, which pins `cargo-cache@v1` and
`cargo-cache-publish@v1` across its CI workflow, and zemyna, migrating to the
same pin. Both pick a move up on their next run with no change on their side,
which is the whole point of the moving pointer and also the reason the move is
not automatic.

---

## Development

```bash
bash scripts/selftest.sh          # everything (~1 min; needs cargo)
bash scripts/selftest.sh --fast   # fixture-only suites, no compiler
```

| suite | covers |
|---|---|
| `hardlink-clone-selftest.sh` | that a build in a clone cannot mutate its source — with a control proving a raw `cp -al` does. Needs a real compiler. |
| `seed-target-dir-selftest.sh` | seed-source preference, lock-file stripping, two jobs racing on one cache key, **and a seed racing a publisher's rotation of the source it is reading** — the race that actually truncates a tree |
| `publish-snapshot-selftest.sh` | the atomic swap, that a live consumer survives a republish, and the publisher's side of the rotation race: deferred reclamation under a live reader, and its sweep once the reader is gone |
| `prune-cache-selftest.sh` | liveness, protection, locking, eviction order, self-clear, **and that a cache a job claims *inside* the check-to-unlink window survives it** — against a real scratch `origin` |
| `restore-mtimes-selftest.sh` | the merge hazard and the watermark that closes it, including the two-jobs-one-namespace case. Needs a real compiler. |

Every suite runs the actual script, not a reimplementation of its logic, and
every fix scenario is paired with a control that reproduces the bug — a
scenario that passes either way proves nothing. The concurrency scenarios race
real processes rather than mocking the interleaving, and gate the interfering
step on *observed* progress of the step it interferes with, so the window is
hit deterministically instead of on a fast machine's coin flip.

The action YAML holds no logic beyond wiring; everything testable lives in
`scripts/`. A composite action needs `shell: bash` on every `run:` step, and
the actions reach their shared scripts through
`${{ github.action_path }}/../scripts`, which works because the runner clones
the whole repository when it fetches an action.