bug(prune): merged-but-undeleted branches keep 40 G target dirs alive and min-free-percent never fires before the seed refuses #20

New Issue

2026-09-07T04:58:03Z

claude commented

2026-09-07 04:58:03 +00:00

Symptom

On gitdan-ci (193 G root), zemyna's zemyna-ci-target volume reached 124 G and cache staging failed three times in a row on 2026-09-07 with

seed: hardlink-cloning base snapshot /cache/snapshot-dev-34c6fcec (44.5 GB) -> /cache/target-fix-second-game-stall-fed22776
::error::unshare_mutable_paths: failed to unshare linked outputs under /cache/.stage-ci-3407-177
::error::clone: could not privately own the mutable paths of /cache/target-fix-second-game-stall-fed22776 — discarding the staging tree …
::error::seed: could not clone base snapshot /cache/snapshot-dev-34c6fcec consistently — refusing to build against a partial cache

with ~34 G free. It passed after a manual rm -rf of one dead target-<branch>-<hash> dir (40 G → 74 G free), then failed again on the very next run (the dev push after the merge) because the PR's own run had published a fresh 40 G target-<branch>-*. Same on zemyna PR #1022 on 2026-09-05. Every failure was fixed by hand-evicting a 40 G per-branch dir. Measured layout at the time:

`zemyna-ci-target/_data/`	size
`snapshot-dev-34c6fcec`	45 G
`target-dev-34c6fcec`	40 G
`target-codex-harness-agnostic-agents-301033f6` (branch merged 2026-09-06)	40 G
later: `target-fix-second-game-stall-fed22776` (branch merged 2026-09-07)	40 G

Why `prune-cache.sh` did not reclaim them

Two design assumptions do not hold on this forge:

Pass 1 (liveness) keys on git ls-remote --heads origin. It evicts a target-*/snapshot-* dir only when its branch is gone from origin. daniel/zemyna has default_delete_branch_after_merge: false (checked via repos/daniel/zemyna), and 50 merged branches are still on origin — so a merged branch's cache is "live" forever. (The close-tracks skill's claim that this forge deletes at merge is wrong for zemyna; separate fix on the forge side.)
Passes 2–3 gate on min-free-percent (10 %) — 19 G on this disk — while seed-target-dir.sh needs headroom on the order of the snapshot's mutable set on top of the snapshot to unshare into a private clone. Staging failed at 34 G free and passed at 74 G, so the gate never fires before the seed refuses. The threshold is sized for a different failure than the one that actually occurs.

Acceptance criteria

A merged branch's target-<branch>-* is reclaimed without a human, on a repo that keeps its branches after merge: either liveness also treats "branch tip is an ancestor of a protected branch" as dead (git merge-base --is-ancestor against each protected ref, read from the same ls-remote), or the docs state that repos must enable delete-on-merge and the action fails loudly on the first pass when it finds N merged-live dirs above a size threshold — pick one and say why.
The free-space gate is derived from what the seed needs: before cloning, seed-target-dir.sh (or the prune ahead of it) computes the required headroom from the snapshot (its mutable set, which the unshare step already enumerates) and evicts oldest non-protected, non-locked target-* dirs until that headroom exists — rather than a fixed percentage. Red-proven in prune-cache-selftest.sh / seed-target-dir-selftest.sh with a stubbed df.
The README's eviction section and the workflow comment in consumers (zemyna/.gitea/workflows/ci.yaml:154-197) describe the new rule; the 10 % figure is deleted rather than corrected.
Selftests green; a dry-run of the prune against a copy of today's zemyna layout (three 40 G dirs, one merged-live) evicts exactly the merged-live dir.

Source

zemyna PR #1034 / run 3407 attempts 1–3 and run 3419 (dev push), 2026-09-07; zemyna PR #1022 run 3256, 2026-09-05. Filed by the zemyna orchestrator session after the user asked to fix the cause.

## Symptom On `gitdan-ci` (193 G root), zemyna's `zemyna-ci-target` volume reached 124 G and cache staging failed three times in a row on 2026-09-07 with ``` seed: hardlink-cloning base snapshot /cache/snapshot-dev-34c6fcec (44.5 GB) -> /cache/target-fix-second-game-stall-fed22776 ::error::unshare_mutable_paths: failed to unshare linked outputs under /cache/.stage-ci-3407-177 ::error::clone: could not privately own the mutable paths of /cache/target-fix-second-game-stall-fed22776 — discarding the staging tree … ::error::seed: could not clone base snapshot /cache/snapshot-dev-34c6fcec consistently — refusing to build against a partial cache ``` with ~34 G free. It passed after a manual `rm -rf` of one dead `target-<branch>-<hash>` dir (40 G → 74 G free), then failed again on the very next run (the `dev` push after the merge) because the PR's own run had published a fresh 40 G `target-<branch>-*`. Same on zemyna PR #1022 on 2026-09-05. Every failure was fixed by hand-evicting a 40 G per-branch dir. Measured layout at the time: | `zemyna-ci-target/_data/` | size | |---|---| | `snapshot-dev-34c6fcec` | 45 G | | `target-dev-34c6fcec` | 40 G | | `target-codex-harness-agnostic-agents-301033f6` (branch merged 2026-09-06) | 40 G | | later: `target-fix-second-game-stall-fed22776` (branch merged 2026-09-07) | 40 G | ## Why `prune-cache.sh` did not reclaim them Two design assumptions do not hold on this forge: 1. **Pass 1 (liveness) keys on `git ls-remote --heads origin`.** It evicts a `target-*`/`snapshot-*` dir only when its branch is gone from origin. `daniel/zemyna` has `default_delete_branch_after_merge: false` (checked via `repos/daniel/zemyna`), and 50 merged branches are still on origin — so a merged branch's cache is "live" forever. (The `close-tracks` skill's claim that this forge deletes at merge is wrong for zemyna; separate fix on the forge side.) 2. **Passes 2–3 gate on `min-free-percent` (10 %)** — 19 G on this disk — while `seed-target-dir.sh` needs headroom on the order of the snapshot's *mutable set* on top of the snapshot to unshare into a private clone. Staging failed at 34 G free and passed at 74 G, so the gate never fires before the seed refuses. The threshold is sized for a different failure than the one that actually occurs. ## Acceptance criteria 1. A merged branch's `target-<branch>-*` is reclaimed without a human, on a repo that keeps its branches after merge: either liveness also treats "branch tip is an ancestor of a protected branch" as dead (`git merge-base --is-ancestor` against each protected ref, read from the same `ls-remote`), or the docs state that repos must enable delete-on-merge and the action fails loudly on the first pass when it finds N merged-live dirs above a size threshold — pick one and say why. 2. The free-space gate is derived from what the seed needs: before cloning, `seed-target-dir.sh` (or the prune ahead of it) computes the required headroom from the snapshot (its mutable set, which the unshare step already enumerates) and evicts oldest non-protected, non-locked `target-*` dirs until that headroom exists — rather than a fixed percentage. Red-proven in `prune-cache-selftest.sh` / `seed-target-dir-selftest.sh` with a stubbed `df`. 3. The README's eviction section and the workflow comment in consumers (`zemyna/.gitea/workflows/ci.yaml:154-197`) describe the new rule; the 10 % figure is deleted rather than corrected. 4. Selftests green; a dry-run of the prune against a copy of today's zemyna layout (three 40 G dirs, one merged-live) evicts exactly the merged-live dir. ## Source zemyna PR #1034 / run 3407 attempts 1–3 and run 3419 (dev push), 2026-09-07; zemyna PR #1022 run 3256, 2026-09-05. Filed by the zemyna orchestrator session after the user asked to fix the cause.

claude added the bug label 2026-09-07 04:58:03 +00:00

daniel referenced this issue from a commit

2026-09-07 05:23:31 +00:00

refactor(cache): name the mutable set once, and measure it

daniel referenced this issue from a commit

2026-09-07 05:23:31 +00:00

fix(prune): reclaim merged branches, and size the volume for the clone

daniel referenced this issue from a commit

2026-09-07 05:23:31 +00:00

docs(cache): describe both liveness signals and the derived requirement

claude referenced a pull request that will close this issue

2026-09-07 05:24:28 +00:00