From f27d6a6954fdd03089e8a41cddfafd37ba831564 Mon Sep 17 00:00:00 2001 From: Dai Ha Date: Thu, 3 Sep 2026 12:00:26 +0700 Subject: [PATCH] Features: backend-error cool-off, workspace-trust seeding, worktree config visibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three merged capabilities that had no entry: - fleetd #201/#227 — the errorPattern knob, credential cool-off, and how cooling off differs from exhaustion quarantine. - fleetd #149 — seeding the workspace-trust flag before a claude-code spawn, why the rejected fixes were rejected, and the external-writer race the atomic write does NOT cover. - fleetd #134/#148 — the neutralized-config and parity-overlay log lines, the git-config channel a member can actually read, and the new [.env] default. --- 11-Features.md | 115 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/11-Features.md b/11-Features.md index f512326..8258dab 100644 --- a/11-Features.md +++ b/11-Features.md @@ -2924,6 +2924,121 @@ change for anyone who relied on the old silence. The refusal stays scoped to `ar and reviewer pools are placement candidates, not identity bindings, so an explicit profile outside them stays a supported override. +## A backend that starts erroring cools off, instead of swallowing the next spawn + +**What.** When two **different** members fail inside 60 seconds with output matching a profile's +backend-error pattern, fleetd treats it as one incident rather than two failures. The credential +those profiles share cools off for 60 seconds. Automatic placement skips it, an explicit +`fleet_spawn` naming it is refused before the backend adapter is ever called, and the lead gets +exactly one nudge in its own pane naming the credential and the affected members. `fleet_list` and +`fleet_profiles` report `coolingOffForSeconds`, and a cooling profile's `free` drops to `0`. + +**On.** The correlation and the cool-off are automatic. The pattern is the knob: + +```yaml +profiles: + terra: + errorPattern: "503 Service Unavailable" # optional +``` + +A profile that sets nothing still gets the built-in compatibility pattern, so this is never silently +off. A malformed regex is rejected at startup, by key name. The key is **deferred**: a running daemon +keeps the pattern it started with, so a change needs a restart. + +**Why it exists.** On 2026-09-01 one backend outage killed both running members mid-turn. fleetd +handled each one correctly and separately, and never noticed it was one event. Both tickets were +reported honestly, both members showed as `done` and `idle` — the same rows a clean finish produces — +and capacity still advertised a free slot. A third spawn onto the same credential would have died the +same way. The evidence that it was an outage only exists **across** members, so a per-turn view +cannot see it however correct each turn's own handling is. + +The pattern is per profile and configured for the same reason `exhaustedPattern` is: the previous +mechanism was one hard-coded `API Error:` string, which fails in the **silent** direction. When a +backend rewords its error the string stops matching, and a failed turn goes back to resolving as a +success. Wording belongs next to the backend definition that produces it. + +**The gotcha: cooling off is not quarantine, and the two can be true at once.** Quarantine (CB-578) +means the backend said it is out of capacity — a long, 1800s-default cooldown. Cooling off means a +credential is erroring right now — short, fixed at 60s, and deliberately not configurable per +profile. They are reported as independent fields precisely so an operator can tell "spent" from +"flaky at the moment", and either one alone already forces `free` to `0`. Read the refusal wording +too: "cooling off after repeated backend errors" is a different situation from an exhaustion refusal, +and they have different fixes. + +One error changes nothing, on purpose. One member failing repeatedly is a member problem; two +different members failing together is a backend problem. Correlation keys on the **credential**, not +the profile, because the credential is the thing that actually runs out — so cooling `terra` also +cools `sol` when they share one. + +## A claude-code member no longer blocks forever on the workspace-trust dialog + +**What.** Before starting a claude-code member in a worktree fleetd provisioned, fleetd marks that +directory as trusted in `~/.claude.json` (or the profile's `configDir`). The member reaches `idle` +instead of sitting on a prompt nobody can answer. + +**On.** Automatic, for `kind: claude-code` members, and **only** in a worktree fleetd itself +provisioned. Any failure is logged at debug and never blocks a spawn. + +**Why it exists.** Claude Code asks for confirmation the first time it opens an unfamiliar directory. +Every provisioned worktree is unfamiliar by construction — it is a fresh path with a nonce in it. The +member came up, printed a dialog, and waited. Nothing could answer it: the lead cannot see a member's +screen, and driving the pane directly is exactly what the bridge exists to prevent. The spawn then +failed as a readiness timeout, which points an investigation at the wrong thing entirely. + +The rejected fixes are worth naming, because each looks reasonable: `--dangerously-skip-permissions` +turns off a real control for a whole session; `permissions.additionalDirectories` widens what the +member may touch rather than answering the question asked; running non-interactively gives up the +pane the whole design depends on. Seeding the one flag the dialog sets is the narrow answer. + +**The gotcha: it writes a file fleetd does not own, and the guard is the worktree check.** While this +was being built, a mutation test aimed at the seeding logic overwrote the operator's real +`~/.claude.json`, shrinking it from 72581 bytes to 919 and taking the account entry with it. That is +why the write is gated on the directory actually being a fleetd-provisioned worktree rather than on +the path merely being set — a default-resolved cwd is the operator's own checkout. The write is +additive and atomic, and a process-wide lock serialises concurrent spawns. + +That lock cannot reach the operator's **own** running Claude Code, which writes the same file. A save +landing between fleetd's read and its write is still lost. The file is never torn, but a change can +vanish. Tracked separately; do not read the atomic write as covering that case. + +## A worktree tells you which of its config files are stubs + +**What.** A provisioned worktree neutralizes `.mcp.json`, `opencode.json` and `.autoenv`: the copy in +the worktree is a stub, not the repo's committed file. The daemon now logs which ones it actually +neutralized and which were absent, and — the part a member can reach — records the list in +worktree-scoped git config: + +```bash +git config --worktree --get-all fleet.neutralizedConfig +git config --worktree --get fleet.neutralizedConfigNote +``` + +The parity overlay reports itself the same way, naming what it copied and what it marked +`--skip-worktree`. + +**On.** Automatic, for every provisioned worktree. + +**Why it exists.** Both mechanisms were completely silent. A member opening `.mcp.json` saw a +plausible file and had no way to know it was a stub — so an edit to it was real work that could never +be committed, and nothing said so. The daemon side was no better: nobody could tell from the logs +whether a given member had been handed an `.envrc` at all, and a silent copy is what makes that kind +of problem hard to notice in the first place. + +Both log lines carry a **denominator** — `neutralized 1 of 3 configs: .mcp.json (opencode.json +absent, .autoenv absent)` — because `copied 1` on its own reads as success whether the candidate list +had one entry or ten. + +**The gotcha: worktree-scoped config is invisible to `git status`, and that is why it was chosen.** +It lives in `.git/worktrees//config.worktree`, never in the working tree, so it cannot show up +as a pending change or get swept into a commit. A marker file in the worktree would have needed a +gitignore entry, a name-collision check, and would still read as "this IS the file" to anything that +just opens it. + +The parity overlay's default changed with this work: it is `[.env]` now, not `[.env, .envrc]`. `.env` +is data, so copying it can only move values; `.envrc` is executable shell that `direnv` runs on every +`cd`, so copying it moves behaviour. An operator who wants it can still list it explicitly, and then +owns that choice. + ### A note for anyone briefing a worker to read this page **A worker cannot see the current version of this file.** `wiki/` is a submodule, and the parent