Compare commits

...

17 Commits

Author SHA1 Message Date
Dai Ha 0c10079755 #293: wrap the bare tab.close in HerdrPeerLauncher.stop()
CI / contract (pull_request) Successful in 1m41s
CI / build (pull_request) Successful in 1m59s
The pane is already closed by the time spaces.closeTab runs, so a failing
tab.close is cosmetic workspace tidying, not a real teardown failure. Left
bare, it propagated out of stop() and masked releaseZdotdir (ZDOTDIR leak)
and, worse, SessionManager.release()'s worktree removal (no self-heal,
no retry — the registry entry is already gone by then).

Wrap it in a try/catch that logs a WARN naming the tab id, matching the
"must not mask a real teardown failure above" comment already on
releaseZdotdir. isAlreadyGone is untouched — this continues past *any*
tab.close failure, not just *_not_found, since the failure is cosmetic
regardless of its cause.

Adds FakeHerdr#tabCloseFailsWith/tabCloseFailsForTab (the tab.close
counterpart to #290's paneCloseFailsForPane) plus two tests: one proving
releaseZdotdir still runs (the generated ZDOTDIR is deleted) and one
proving SessionManager.release() still removes the worktree, both with a
non-not_found tab.close failure.
2026-09-04 11:41:07 +07:00
Dai Ha d5128a1d35 #290: use the imports FakeHerdr already has
CI / build (push) Successful in 1m33s
CI / contract (push) Successful in 1m39s
2026-09-04 11:27:51 +07:00
Dai Ha 61097e5cf0 Merge #290: restore coverage for reapIdle's per-session guard 2026-09-04 11:25:45 +07:00
Dai Ha ef507bcd12 #290: restore reapIdle's per-session guard coverage via a new launcher.stop() trigger
CI / contract (pull_request) Successful in 1m29s
CI / build (pull_request) Successful in 1m56s
#283 fixed release() to catch and log a worktree-removal failure, which closed off
reapIdleCountsAllThreeSessionsWhenOnlyItsWorktreeRemovalFails as a trigger for
reapIdle's own per-session try/catch (CB-581) — that test now proves a different,
still-real thing (a swallowed removal failure doesn't shrink the reaped count),
but the try/catch itself lost its test.

Add FakeHerdr.paneCloseFailsForPane(paneId, code) so a test can make exactly one
session's launcher.stop() fail while its siblings still tear down normally
(paneCloseFailsWith already existed but fails every pane, which cannot isolate
one session in a three-session reap). Add
reapIdleSurvivesOneSessionWhoseLauncherStopFails beside the #283 test, using
launcher.stop() as the trigger the ticket names, and prove it catches removal of
reapIdle's try/catch: deleting the guard makes the test fail with the
HerdrException propagating out of reapIdle uncaught (quoted in the PR body).
2026-09-04 11:23:52 +07:00
Dai Ha 94f50e507a #284/#285: one reclaimable rule, one group-share helper
CI / contract (push) Successful in 1m2s
CI / build (push) Successful in 2m11s
Two corrections on top of the merged worker branches.

#284: I told the worker to report a BACKEND_ERROR/FAILED session as
reclaimable. That half of my own ticket was wrong. Once the live count
stops counting a terminal session, its seat is already in `free`;
counting it in `reclaimable` too reports the same seat twice, and
`free + reclaimable` reads as more capacity than maxLoad allows. Worse,
only the profile-level count was widened, so the same fleet_list
response said `reclaimable: 2` while every member row said
`reclaimable: false`.

Both views now call one shared predicate, FleetMcp.reclaimable, so they
cannot drift apart. A test runs it over every MemberSession.State value,
so a state added later cannot slip through unconsidered.

#285: the new per-file chgrp+chmod helper moved from ClaudeCodeLauncher
into EnvAllowListScrub as shareFileWithGroup, next to the directory-wide
shareWithGroup it was copied from. It now reuses that class's own
setGroupAndPermissions and also catches UnsupportedOperationException,
which the copy missed — on a filesystem without POSIX group ownership
the copy threw a raw runtime exception instead of the sibling's
UncheckedIOException.
2026-09-04 11:13:01 +07:00
Dai Ha 75b15086b0 Merge #285: the workspace-trust seed refuses rather than write fleetd's own home 2026-09-04 11:08:44 +07:00
Dai Ha dab9645906 Merge #284: a BACKEND_ERROR or FAILED member no longer holds a spawn seat 2026-09-04 11:04:09 +07:00
Dai Ha e93b5f6512 #282: record that answer()'s QUESTION guard is defence in depth, not load-bearing
CI / contract (push) Successful in 51s
CI / build (push) Successful in 1m44s
Measured after merging: removing the guard alone leaves the new test green,
because ask() calls markAsyncQuestion before resolveQuestion, so the task has
already moved to the new turnId. The PR claimed each half was necessary; only
the pair is. Keeping the guard, with the ordering written down so nobody
deletes it as dead code or trusts it as the only protection.
2026-09-04 10:59:10 +07:00
Dai Ha bfabe13e8f Merge #282 (PR #289): a chained second fleet_ask no longer kills its own async ticket 2026-09-04 10:56:42 +07:00
Dai Ha 7e49c6eca2 #285: seedTrustDialog refuses under memberHerdrSocket instead of seeding fleetd's own home
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 3m35s
seedTrustDialog gated only on isProvisionedWorktree(cwd) and, being static, could not
see memberHerdrSocketConfigured() — unlike its sibling writeCharterFile, which already
refuses the spawn when it cannot place a file where a different-uid member can read it.
Under memberHerdrSocket + configDir unset, seedTrustDialog wrote fleetd's OWN
~/.claude.json while believing it was seeding the member's, reintroducing the fleetd
#149 failure (interactive trust dialog, no fleet_reply, silent readiness timeout) for
this one config combination.

Makes seedTrustDialog an instance method so it can see memberHerdrSocketConfigured()
and memberGroup(), and applies writeCharterFile's "refuse, don't degrade" rule: under
memberHerdrSocket it now requires both configDir and worktreeGroup before touching any
file, naming exactly which is missing, and shares the written .claude.json group-
readable (rw-r-----) via a new shareTrustJsonWithGroup so the member's OS user can
actually open it. The memberHerdrSocket-absent path (today's only live mode) is
unchanged.
2026-09-04 10:55:35 +07:00
Dai Ha 11cbfa79b4 fleetd #284: free failed-session capacity
CI / contract (pull_request) Successful in 1m11s
CI / build (pull_request) Successful in 1m19s
2026-09-04 10:55:21 +07:00
Dai Ha c2c2746922 Merge #283 (PR #288): guard release()'s worktree removal, delete the orphaned branch on spawn failure
CI / contract (push) Successful in 52s
CI / build (push) Successful in 1m44s
2026-09-04 10:54:36 +07:00
Dai Ha 0087645da4 Merge #281 (PR #286): pin the Authz.Action every handler chooses
CI / contract (push) Successful in 1m1s
CI / build (push) Failing after 1m45s
2026-09-04 10:50:43 +07:00
Dai Ha 19cacf5b62 #283: guard release()'s bare worktree remove; delete orphaned branch on spawn failure
CI / contract (pull_request) Successful in 52s
CI / build (pull_request) Successful in 2m42s
Two teardown-cleanup leaks in SessionManager, same shape as #274.

Defect 1: release()'s last step (removing a released session's worktree)
was the one cleanup step in the method left unguarded, even though every
sibling step is wrapped because exec() can throw on a non-zero exit or its
own 30s timeout. By the time it ran, the registry entry, retained handle,
and pane were already gone, so a throw here escaped release() with no
retry path and made a fully-torn-down session look like a failed stop.
Now wrapped in try/catch with a WARN, matching the pattern already used
by every other step in this method.

Defect 2: acquireWithWorktree's catch (covering failures after add()
returns — overlayParity, shareWithGroup, launcher.spawn) removed the
worktree but left the branch it provisioned orphaned. #274 already fixed
the sibling failure inside add() itself (GitWorktrees.cleanupAfterAddFailure
deletes both). Extracted that branch-delete into a new Worktrees.deleteBranch
method, reused by both cleanupAfterAddFailure and this catch, so a routine
spawn failure (quarantined credential, backend refusal) no longer leaks a
worker/<slug>-<nonce> branch.

A normal release() still never deletes a branch — only the failed-provision
path does. releaseRemovesWorktreeButDoesNotDeleteBranch pins this, and
spawnFailureAfterAddDeletesTheOrphanedBranch / unchangedRegression* prove
the two paths stay apart.
2026-09-04 10:50:30 +07:00
Dai Ha 4dd12083ab fleetd #284: free backend-error capacity
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 2m42s
2026-09-04 10:47:09 +07:00
Dai Ha 30e21adec7 #281: cover registered authorization actions
CI / contract (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 1m33s
2026-09-04 10:46:26 +07:00
Dai Ha 719b79f892 wip: pin handler authorization actions
CI / build (pull_request) Successful in 1m30s
CI / contract (pull_request) Successful in 2m27s
2026-09-04 10:39:58 +07:00
20 changed files with 962 additions and 69 deletions
@@ -250,9 +250,7 @@ public final class Fleetd {
boolean clearAfterTurn = cfg.lifecycle() != null && cfg.lifecycle().clearAfterTurn();
SessionManager sessions = new SessionManager(workers, new GitWorktrees(cfg.worktreeRoot(), cfg.worktreeGroup()),
System::nanoTime, contextCap, clearAfterTurn);
liveCountRef.set(profileName -> (int) sessions.roster().stream()
.filter(s -> profileName.equals(s.profile()))
.count());
liveCountRef.set(profileName -> liveSessionCount(sessions.roster(), profileName));
// CB-303 part 1: idle-ttl reaper — only when configured, defaults to disabled.
final SessionReaper reaper;
@@ -859,6 +857,19 @@ public final class Fleetd {
.orElse(null);
}
/**
* Count sessions that occupy a profile's spawn capacity. A {@code BACKEND_ERROR} or
* {@code FAILED} session stays in the roster so {@code fleet_list} can show its failure, but a
* member that cannot accept another delivery does not use a seat.
*/
static int liveSessionCount(List<MemberSession> roster, String profileName) {
return (int) roster.stream()
.filter(session -> profileName.equals(session.profile()))
.filter(session -> session.state() != MemberSession.State.BACKEND_ERROR)
.filter(session -> session.state() != MemberSession.State.FAILED)
.count();
}
/**
* fleetd #248 / fleetd#201 Unit 5: factory for the production {@link BackendErrorSink} — the
* collaborator {@link CompletionResolver} notifies when a pane-scrape classification actually
@@ -322,8 +322,9 @@ public record FleetConfig(
* statement that does not stop being true just because the profile was
* named directly. A negative value has no sane meaning (there is no
* "excluded" to degrade to below zero) and is refused at config load
* instead, naming the profile and the key. Live means any session the
* registry still owns (acquired and not yet released), in any state.
* instead, naming the profile and the key. Live means a session that can
* receive another delivery. The roster keeps terminal {@code BACKEND_ERROR}
* and {@code FAILED} sessions for diagnostics, but they do not use capacity.
* @param kind which peer launcher spawns this profile: {@code "claude-code"} (default —
* the {@link dev.ltms.fleet.member.ClaudeCodeLauncher}) or {@code "opencode"}.
* The {@code CompositePeerLauncher} routes {@code spawn}/reap by this value, so
@@ -257,7 +257,7 @@ public final class FleetMcp {
// Each handler is built once and wired to its fleet_* tool below.
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> sendHandler =
(exchange, req) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.SEND,
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_send", req.arguments()),
str(req.arguments(), "sessionId"));
if (denied != null) return denied;
String caller = callerTerminal(exchange);
@@ -299,7 +299,7 @@ public final class FleetMcp {
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> replyHandler =
(exchange, req) -> {
String self = callerTerminal(exchange);
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.REPLY, self);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_reply", req.arguments()), self);
if (denied != null) return denied;
return reply(messages, self, str(req.arguments(), "content"));
};
@@ -307,13 +307,13 @@ public final class FleetMcp {
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> askHandler =
(exchange, req) -> {
String self = callerTerminal(exchange);
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.ASK, self);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_ask", req.arguments()), self);
if (denied != null) return denied;
return ask(messages, self, str(req.arguments(), "question"), timeoutMs(req.arguments()));
};
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> statusHandler =
(exchange, req) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.READ, null);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_status", req.arguments()), null);
if (denied != null) return denied;
return status(messages, str(req.arguments(), "sessionId"));
};
@@ -322,7 +322,7 @@ public final class FleetMcp {
Map<String, Object> a = req.arguments();
String target = str(a, "target");
// The action depends on the ARGUMENTS, not on the tool name -- see pollAction.
McpSchema.CallToolResult denied = deny(exchange, pollAction(target), target);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_poll", a), target);
if (denied != null) return denied;
return poll(messages, str(a, "ticket"), target);
};
@@ -331,14 +331,14 @@ public final class FleetMcp {
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> ackHandler =
(exchange, req) -> {
Map<String, Object> a = req.arguments();
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.DRAIN, str(a, "target"));
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_ack", a), str(a, "target"));
if (denied != null) return denied;
return ack(messages, str(a, "target"), str(a, "msgId"));
};
// Fleet management (CB-108): spawn/list/stop over ClaudeCodeLauncher.
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> spawnHandler =
(exchange, req) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.SPAWN, null);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_spawn", req.arguments()), null);
if (denied != null) return denied;
String caller = callerTerminal(exchange);
// SPAWN is already auth-gated to PRIMARY (architects can never call it), but
@@ -355,7 +355,7 @@ public final class FleetMcp {
};
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> listHandler =
(exchange, _) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.READ, null);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_list", Map.of()), null);
if (denied != null) return denied;
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, outage,
leadSeats, callers == null ? Map.of() : callers.leads(),
@@ -365,19 +365,19 @@ public final class FleetMcp {
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> stopHandler =
(exchange, req) -> {
String paneId = str(req.arguments(), "paneId");
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.STOP, paneId);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_stop", req.arguments()), paneId);
if (denied != null) return denied;
return stop(sessions, paneId);
};
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> profilesHandler =
(exchange, _) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.READ, null);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_profiles", Map.of()), null);
if (denied != null) return denied;
return profiles(workers, quarantine, outage);
};
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> whoamiHandler =
(exchange, _) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.READ, null);
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_whoami", Map.of()), null);
if (denied != null) return denied;
return whoami(principal(exchange), sessions);
};
@@ -754,6 +754,25 @@ public final class FleetMcp {
return isBlank(target) ? Authz.Action.READ : Authz.Action.DRAIN;
}
/**
* The action a registered tool handler actually hands to the authorization gate.
* Keeping this choice beside the registered-tool inventory makes a new tool fail the coverage
* test until its action is pinned.
*/
static Authz.Action toolAction(String toolName, Map<String, Object> arguments) {
return switch (toolName) {
case "fleet_send" -> Authz.Action.SEND;
case "fleet_reply" -> Authz.Action.REPLY;
case "fleet_ask" -> Authz.Action.ASK;
case "fleet_status", "fleet_list", "fleet_profiles", "fleet_whoami" -> Authz.Action.READ;
case "fleet_poll" -> pollAction(str(arguments, "target"));
case "fleet_ack" -> Authz.Action.DRAIN;
case "fleet_spawn" -> Authz.Action.SPAWN;
case "fleet_stop" -> Authz.Action.STOP;
default -> throw new IllegalArgumentException("unregistered tool: " + toolName);
};
}
/** {@code fleet_poll}: check an async delegation by ticket, or drain a worker's inbox by target. */
static McpSchema.CallToolResult poll(MessageService messages, String ticket, String target) {
if (!isBlank(target)) {
@@ -1148,15 +1167,35 @@ public final class FleetMcp {
private static Map<String, Object> memberCapacityView(MemberSession session, Agent live,
MessageService messages, long nowNanos) {
Map<String, Object> row = SessionManager.rosterView(session, live);
boolean open = messages != null && messages.hasAcceptedDelivery(session.terminalId());
boolean inbox = messages != null && messages.hasInboxMessage(session.terminalId());
boolean reclaimable = (session.state() == MemberSession.State.READY || session.state() == MemberSession.State.DONE)
&& !open && !inbox;
boolean reclaimable = reclaimable(session, messages);
row.put("reclaimable", reclaimable);
row.put("idleForSeconds", reclaimable ? Math.max(0, (nowNanos - session.lastActivityAtNanos()) / 1_000_000_000L) : null);
return row;
}
/**
* The one definition of {@code reclaimable}: this member holds a spawn seat, and has no open
* bridge work, so stopping it gives the seat back. Both views in a single {@code fleet_list}
* response call it — the per-member flag in {@link #memberCapacityView} and the per-profile
* count in {@link #capacityView} — because two copies of this rule in one response is how the
* two numbers come to disagree.
*
* <p>fleetd #284: {@code BACKEND_ERROR} and {@code FAILED} are deliberately NOT reclaimable.
* The ticket asked for them to be, and that half of the ticket was wrong. Once
* {@code Fleetd.liveSessionCount} stopped counting a terminal session as live, that seat is
* ALREADY in {@code free}; counting it here too reports the same seat twice, and
* {@code free + reclaimable} then reads as more capacity than {@code maxLoad} allows. The dead
* session stays visible either way: its roster row still carries {@code state:
* "backend_error"} or {@code "failed"}, which is what tells the lead to stop it.
*/
static boolean reclaimable(MemberSession session, MessageService messages) {
boolean holdsSeat = session.state() == MemberSession.State.READY
|| session.state() == MemberSession.State.DONE;
return holdsSeat && (messages == null
|| (!messages.hasAcceptedDelivery(session.terminalId())
&& !messages.hasInboxMessage(session.terminalId())));
}
/**
* CB-583: {@code free} alone cannot tell a lead "busy, will free up" from "refusing, and
* nothing changes for N seconds" — those need different decisions. So a quarantined profile
@@ -1195,8 +1234,7 @@ public final class FleetMcp {
int live = liveCount.apply(profile);
int leadSeatCount = leadSeats.seatsFor().apply(profile);
int reclaimable = (int) roster.stream().filter(s -> profile.equals(s.profile()))
.filter(s -> (s.state() == MemberSession.State.READY || s.state() == MemberSession.State.DONE))
.filter(s -> messages == null || (!messages.hasAcceptedDelivery(s.terminalId()) && !messages.hasInboxMessage(s.terminalId())))
.filter(s -> reclaimable(s, messages))
.count();
Map<String, Object> row = new LinkedHashMap<>();
row.put("profile", profile); row.put("maxLoad", cap); row.put("live", live);
@@ -540,16 +540,66 @@ public final class ClaudeCodeLauncher extends HerdrPeerLauncher {
* itself is still lost, because there is no OS-level compare-and-swap on a plain file, only this
* cooperative narrowing of the gap.
*
* <p><b>fleetd #285: refuses under {@code memberHerdrSocket} rather than writing somewhere the
* member cannot read.</b> Under {@code memberHerdrSocket:} the member pane runs as a
* <em>different OS user with its own {@code $HOME}</em> — the same reason {@link
* #writeCharterFile} routes the role/reply charter under {@code worktreeRoot} instead of
* {@code java.io.tmpdir} and refuses the spawn when it cannot. This method has no equivalent
* relocation available: unlike the charter (fleetd's own content, free to place anywhere and
* hand to the peer via an argv flag), {@code .claude.json} is a file Claude Code looks up for
* ITSELF at a fixed location — {@code CLAUDE_CONFIG_DIR/.claude.json}, or else the member OS
* user's own {@code ~/.claude.json}, a path fleetd has no channel to learn. So when {@code
* configDir} is unset, there is no member-readable target to seed at all — writing the
* unqualified default would land in <em>fleetd's own</em> {@code ~/.claude.json} instead, the
* exact defect this fix closes, not a workable fallback. And even with {@code configDir} set,
* the file this method itself just wrote is {@code 0600} (owner-only — see {@link
* #copyPosixPermissionsIfPresent}), unreadable by a different-uid member unless shared with
* {@code worktreeGroup}, the same group {@link EnvAllowListScrub#shareWithGroup} already uses
* for the ZDOTDIR scrub (fleetd #213) and the charter file (fleetd #219/#222). So under {@code
* memberHerdrSocket} this method requires BOTH {@code configDir} and {@code worktreeGroup}
* before it ever touches a file, and refuses the spawn — naming exactly which one is missing —
* rather than silently corrupt fleetd's own home or hand the member an unreadable path. This
* mirrors {@link #writeCharterFile}'s "refuse, don't degrade" decision: a member spawned without
* a readable trust seed is not degraded, it sits on the interactive dialog forever and never
* calls {@code fleet_reply} — exactly the failure fleetd #149 exists to prevent, so trading it
* for "spawn something" is not worth it. When {@code configDir} and {@code worktreeGroup} are
* both present, the write proceeds exactly as below and the resulting file is additionally
* chgrp'd/chmod'd group-readable ({@code rw-r-----}) via {@link
* EnvAllowListScrub#shareFileWithGroup(Path, String)} so the member's OS user can actually
* open it — the
* directory itself (unlike {@code worktreeRoot} or the charter's per-spawn directory) is not
* fleetd-managed, so its own traversal permissions remain the operator's setup, same as they
* already must be for the member to read anything else fleetd points {@code CLAUDE_CONFIG_DIR}
* at. With {@code memberHerdrSocket} ABSENT (today's only live mode) every branch below is
* byte-identical to before this fix.
*
* @param configDir the profile's {@code CLAUDE_CONFIG_DIR} ({@code cfg.configDir()}), or
* {@code null}/blank to target the default {@code ~/.claude.json}
* {@code null}/blank to target the default {@code ~/.claude.json} — refused
* outright when {@code memberHerdrSocket} is configured, see above
* @param cwd the spawn's resolved working directory — the exact key Claude Code will look
* up for itself once it starts there
* @throws IllegalStateException when {@code memberHerdrSocket} is configured but {@code
* configDir} and/or {@code worktreeGroup} is not — the same
* refusal shape as {@link #writeCharterFile}
*/
private static void seedTrustDialog(String configDir, String cwd) {
private void seedTrustDialog(String configDir, String cwd) {
if (!isProvisionedWorktree(cwd)) {
return;
}
boolean unsetConfigDir = configDir == null || configDir.isBlank();
boolean memberHerdrSocket = memberHerdrSocketConfigured();
String group = memberHerdrSocket ? memberGroup() : null;
if (memberHerdrSocket && (unsetConfigDir || group == null)) {
throw new IllegalStateException("memberHerdrSocket is configured, so the workspace-trust "
+ "seed (.claude.json, which gates Claude Code's interactive trust dialog) must be "
+ "placed where the member's OS user can read it — configDir, shared via "
+ "worktreeGroup — but " + (unsetConfigDir ? "configDir" : "worktreeGroup")
+ " is not configured. Refusing to spawn rather than write fleetd's own default "
+ "'~/.claude.json' or hand the member a config file it cannot read: that member "
+ "would sit on the interactive trust dialog forever and never reach an "
+ "injectable state. Configure configDir on this profile and worktreeGroup on the "
+ "fleet to enable claude-code member spawns under memberHerdrSocket.");
}
Path target = unsetConfigDir
? Path.of(System.getProperty("user.home"), ".claude.json")
: Path.of(configDir, ".claude.json");
@@ -559,18 +609,20 @@ public final class ClaudeCodeLauncher extends HerdrPeerLauncher {
// profile that simply forgot to set configDir gets no signal at all short of the
// operator noticing their own file changing. Say so loudly, every time it is about to
// happen, rather than only once ever: each occurrence is a live write to a real
// person's home config and deserves its own log line.
// person's home config and deserves its own log line. (Reached only when
// memberHerdrSocket is absent — the block above already refused otherwise.)
log.warn("seedTrustDialog: profile has no configDir set, so the workspace-trust seed "
+ "for cwd '{}' is about to write the operator's own default '{}' — set "
+ "configDir on this profile to target a per-member config file instead",
cwd, target);
}
synchronized (TRUST_JSON_LOCK) {
boolean written = false;
try {
if (target.getParent() != null) {
Files.createDirectories(target.getParent());
}
for (int attempt = 1; attempt <= MAX_TRUST_JSON_CAS_ATTEMPTS; attempt++) {
for (int attempt = 1; attempt <= MAX_TRUST_JSON_CAS_ATTEMPTS && !written; attempt++) {
byte[] before = Files.isRegularFile(target) ? Files.readAllBytes(target) : null;
ObjectNode root = parseTrustJsonOrEmpty(before);
JsonNode projectsNode = root.get("projects");
@@ -611,26 +663,39 @@ public final class ClaudeCodeLauncher extends HerdrPeerLauncher {
continue;
}
writeAtomically(target, newContent);
return;
written = true;
}
if (!written) {
// fleetd #247: deliberately do NOT write here. A member that starts without the
// seed still starts — it may hit the trust dialog fleetd #149 describes and fail
// to reach an injectable state, but that failure is visible (herdr reports it,
// the spawn-readiness gate times out) and recoverable (retry the spawn). Writing
// our stale copy over whatever the other writer left would be silent and, if that
// other writer is the operator's own live session, could destroy real
// configuration — fail toward the recoverable outcome, not the silent one.
log.warn("seedTrustDialog: gave up seeding workspace-trust for cwd '{}' into '{}' "
+ "after {} attempts — another writer (most plausibly the operator's own "
+ "live Claude Code sharing this file) kept changing it faster than we "
+ "could re-read it, so nothing was written; the member may show the "
+ "trust dialog instead", cwd, target, MAX_TRUST_JSON_CAS_ATTEMPTS);
}
// fleetd #247: deliberately do NOT write here. A member that starts without the
// seed still starts — it may hit the trust dialog fleetd #149 describes and fail to
// reach an injectable state, but that failure is visible (herdr reports it, the
// spawn-readiness gate times out) and recoverable (retry the spawn). Writing our
// stale copy over whatever the other writer left would be silent and, if that other
// writer is the operator's own live session, could destroy real configuration —
// fail toward the recoverable outcome, not the silent one.
log.warn("seedTrustDialog: gave up seeding workspace-trust for cwd '{}' into '{}' "
+ "after {} attempts — another writer (most plausibly the operator's own "
+ "live Claude Code sharing this file) kept changing it faster than we could "
+ "re-read it, so nothing was written; the member may show the trust dialog "
+ "instead", cwd, target, MAX_TRUST_JSON_CAS_ATTEMPTS);
} catch (Exception e) {
log.debug("cannot seed workspace-trust entry for cwd '{}' into '{}'", cwd, target, e);
return;
}
// fleetd #285: the write above lands as fleetd's own OS user; under memberHerdrSocket
// that is NOT the member's OS user, so without this the member still cannot read the
// file it exists to seed — a silent readiness timeout with the write looking "done".
// Deliberately OUTSIDE the swallow-all catch above: a group that fails to resolve here
// means the seed is unreadable despite a successful write, which must fail as loudly as
// writeCharterFile's own EnvAllowListScrub.shareWithGroup call already does.
if (written && memberHerdrSocket) {
EnvAllowListScrub.shareFileWithGroup(target, group);
}
}
}
/** Bound on {@link #seedTrustDialog}'s fleetd #247 compare-and-swap retry loop. */
private static final int MAX_TRUST_JSON_CAS_ATTEMPTS = 5;
@@ -181,6 +181,33 @@ public final class EnvAllowListScrub {
}
}
/**
* fleetd #285: share ONE file with {@code group}, read-only ({@code rw-r-----}) — the same
* per-file mode {@link #shareWithGroup} applies to a directory's entries, and the same error
* shapes, but without touching a parent directory. Used for a file fleetd writes into a
* directory it does NOT own — {@code configDir}'s own traversal permissions stay the
* operator's setup — where the directory-wide {@link #shareWithGroup} would be wrong.
*
* @throws UncheckedIOException when {@code group} does not resolve on this host, the
* filesystem has no POSIX group ownership, or a
* group-ownership/permission call is refused
*/
static void shareFileWithGroup(Path file, String group) {
try {
GroupPrincipal principal = file.getFileSystem().getUserPrincipalLookupService()
.lookupPrincipalByGroupName(group);
setGroupAndPermissions(file, principal, "rw-r-----");
} catch (IOException e) {
throw new UncheckedIOException("cannot share generated file " + file + " with group '"
+ group + "' — the group must exist, and the fleetd operator ("
+ System.getProperty("user.name") + ") must be a member of it", e);
} catch (UnsupportedOperationException e) {
throw new UncheckedIOException("cannot share generated file " + file + " with group '"
+ group + "' — this filesystem does not support POSIX group ownership",
new IOException(e));
}
}
private static void setGroupAndPermissions(Path path, GroupPrincipal group, String perms) throws IOException {
PosixFileAttributeView view = Files.getFileAttributeView(path, PosixFileAttributeView.class);
if (view == null) {
@@ -913,9 +913,14 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
* {@link #reapOrphanWorkers() orphan-reap} and spawn-gate-timeout paths, plus any caller that
* passes a pane directly, keep working without an owning id.
*
* <p>Resolves the tab from the pane <em>before</em> closing it. An already-gone pane/tab
* (repeated DELETE, crashed peer) is treated as success; any other failure propagates so a
* genuinely failed teardown is not reported as done.
* <p>Resolves the tab from the pane <em>before</em> closing it. {@code agents.close} (the pane)
* is the one step whose failure means the teardown itself may not have happened: an already-gone
* pane (repeated DELETE, crashed peer) is treated as success, but any other failure propagates so
* a genuinely failed teardown is not reported as done. {@code spaces.closeTab} (fleetd #293) is
* different — by the time it runs the pane is already closed, so it is cosmetic workspace tidying
* rather than a real teardown failure, and a failure there is logged and never propagates, so it
* cannot mask the two cleanups below it ({@link #releaseZdotdir}, and the caller's worktree
* removal in {@code SessionManager.release}).
*/
@Override
public void stop(String idOrPane) {
@@ -934,7 +939,18 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
log.debug("pane.close({}) ignored — already gone: {}", paneId, e.getMessage());
}
if (loc != null && loc.tabPaneCount() == 1) {
spaces.closeTab(loc.tabId());
// fleetd #293: the pane above is already closed by this point, so a failing tab.close is
// cosmetic workspace tidying, not a real teardown failure — it must not mask the two
// cleanups below it (releaseZdotdir, and the caller's worktree removal). Unlike
// agents.close above, this is not narrowed to "already gone": any failure here, whatever
// its cause, is one we continue past, so we log it at WARN (not debug) with the tab id a
// person can go close by hand.
try {
spaces.closeTab(loc.tabId());
} catch (HerdrException e) {
log.warn("tab.close({}) failed — the pane is already torn down, so continuing; the "
+ "tab may need manual cleanup: {}", loc.tabId(), e.getMessage());
}
} else if (loc != null) {
log.debug("not closing tab {} — it holds {} panes (not a dedicated peer tab)",
loc.tabId(), loc.tabPaneCount());
@@ -948,6 +948,14 @@ public final class MessageService {
// the worker chained a second fleet_ask before replying. Mirror sendAsync's own guard
// (:1000) and leave the ticket open (markAsyncQuestion above already re-armed it under
// the new turnId) instead of completing it here with a QUESTION "reply".
// Measured when #282 was merged: this guard is DEFENCE IN DEPTH, not the thing
// that makes the chained ask work. ask() calls markAsyncQuestion (:860) before
// resolveQuestion (:861), so by the time this thread wakes, the task has already
// moved to the new turnId and finishAsyncTask(oldTurnId, ...) finds nothing. Removing
// this guard alone leaves the test green. Keep it anyway: it mirrors sendAsync's
// sibling guard, and that sibling's own comment (:1017) warns the two orderings are
// not something to rely on. Do NOT delete it as dead code without re-checking that
// ordering, and do not treat it as the sole protection either.
if (result.outcome() != Outcome.QUESTION) {
finishAsyncTask(turnId, result);
}
@@ -47,6 +47,22 @@ import java.util.stream.Collectors;
*/
public final class FleetApp {
/** The authorization action the matching route handler hands to {@link #allow}. */
static Authz.Action routeAction(String route) {
return switch (route) {
case "GET /metrics" -> Authz.Action.METRICS;
case "POST /members" -> Authz.Action.SPAWN;
case "DELETE /members/{paneId}" -> Authz.Action.STOP;
case "POST /sessions/{id}/message" -> Authz.Action.SEND;
case "POST /sessions/{id}/reply" -> Authz.Action.REPLY;
case "GET /sessions/{id}/replies" -> Authz.Action.DRAIN;
case "POST /sessions/{id}/ask" -> Authz.Action.ASK;
case "GET /sessions", "GET /agents", "GET /members", "GET /profiles",
"GET /member-credentials", "GET /sessions/{id}/status", "GET /tasks/{ticket}" -> Authz.Action.READ;
default -> throw new IllegalArgumentException("route has no authorization gate: " + route);
};
}
/** Default blocking window for a message; kept under typical HTTP idle timeouts. */
private static final long DEFAULT_MESSAGE_TIMEOUT_MS = 25_000;
private static final long MAX_MESSAGE_TIMEOUT_MS = 120_000;
@@ -222,7 +238,7 @@ public final class FleetApp {
/** Prometheus scrape endpoint (CB-502). */
private void metrics(Context ctx) {
if (!allow(ctx, Authz.Action.METRICS, null)) {
if (!allow(ctx, routeAction("GET /metrics"), null)) {
return;
}
ctx.status(200).contentType("text/plain; version=0.0.4; charset=utf-8").result(metrics.render());
@@ -294,7 +310,7 @@ public final class FleetApp {
* member workspace (they live on the member daemon only).
*/
private void sessions(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /sessions"), null)) {
return;
}
List<Map<String, Object>> out = new ArrayList<>();
@@ -319,7 +335,7 @@ public final class FleetApp {
/** Discovery: every agent herdr tracks, keyed by its Claude session UUID. */
private void agents(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /agents"), null)) {
return;
}
ctx.status(200).json(Map.of("agents",
@@ -328,7 +344,7 @@ public final class FleetApp {
/** CB-304: bridge-owned roster merged with live herdr status by paneId. */
private void listMembers(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /members"), null)) {
return;
}
// CB-519: the registry key is a host-unique id, not the pane coordinate — join on terminal.
@@ -359,7 +375,7 @@ public final class FleetApp {
/** The configured worker profiles and which one a no-argument spawn uses. */
private void profiles(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /profiles"), null)) {
return;
}
ctx.status(200).json(Map.of(
@@ -376,7 +392,7 @@ public final class FleetApp {
* name list, which is exactly what let the list drift silently behind the real policy.
*/
private void memberCredentials(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /member-credentials"), null)) {
return;
}
MemberCredentialPolicyView view = memberCredentials.get();
@@ -396,7 +412,7 @@ public final class FleetApp {
* the subscription boundary, 400 for an unknown profile.
*/
private void spawnMember(Context ctx) {
if (!allow(ctx, Authz.Action.SPAWN, null)) {
if (!allow(ctx, routeAction("POST /members"), null)) {
return;
}
String role = ctx.queryParam("role");
@@ -467,7 +483,7 @@ public final class FleetApp {
/** Tear a worker down by pane id. */
private void stopMember(Context ctx) {
String paneId = ctx.pathParam("paneId");
if (!allow(ctx, Authz.Action.STOP, paneId)) {
if (!allow(ctx, routeAction("DELETE /members/{paneId}"), paneId)) {
return;
}
sessions.release(paneId);
@@ -482,7 +498,7 @@ public final class FleetApp {
*/
private void sendMessage(Context ctx) {
String id = ctx.pathParam("id");
if (!allow(ctx, Authz.Action.SEND, id)) {
if (!allow(ctx, routeAction("POST /sessions/{id}/message"), id)) {
return;
}
String content;
@@ -569,7 +585,7 @@ public final class FleetApp {
*/
private void askMessage(Context ctx) {
String id = ctx.pathParam("id");
if (!allow(ctx, Authz.Action.ASK, id)) {
if (!allow(ctx, routeAction("POST /sessions/{id}/ask"), id)) {
return;
}
String question;
@@ -608,7 +624,7 @@ public final class FleetApp {
// The rule that matters: a worker may reply only as itself. Over MCP this was already true
// structurally (identity comes from the connection, never an argument); over REST the path
// id was simply trusted, so this is where the invariant actually gets enforced.
if (!allow(ctx, Authz.Action.REPLY, id)) {
if (!allow(ctx, routeAction("POST /sessions/{id}/reply"), id)) {
return;
}
String content;
@@ -629,7 +645,7 @@ public final class FleetApp {
*/
private void drainReplies(Context ctx) {
String id = ctx.pathParam("id");
if (!allow(ctx, Authz.Action.DRAIN, id)) {
if (!allow(ctx, routeAction("GET /sessions/{id}/replies"), id)) {
return;
}
var replies = messages.drainReplies(id);
@@ -647,7 +663,7 @@ public final class FleetApp {
*/
private void sessionStatus(Context ctx) {
String id = ctx.pathParam("id");
if (!allow(ctx, Authz.Action.READ, id)) {
if (!allow(ctx, routeAction("GET /sessions/{id}/status"), id)) {
return;
}
try {
@@ -672,7 +688,7 @@ public final class FleetApp {
/** Poll an async (wait:false) delegation by ticket. 404 for an unknown/expired ticket. */
private void taskStatus(Context ctx) {
if (!allow(ctx, Authz.Action.READ, null)) {
if (!allow(ctx, routeAction("GET /tasks/{ticket}"), null)) {
return;
}
MessageService.TaskView v = messages.poll(ctx.pathParam("ticket"));
@@ -213,13 +213,18 @@ public final class GitWorktrees implements Worktrees {
worktreePath, cleanup.getMessage());
}
try {
exec("git", "-C", repoRoot, "branch", "-D", branch);
deleteBranch(repoRoot, branch);
} catch (RuntimeException cleanup) {
log.warn("failed to remove leaked branch {} after provisioning error: {}",
branch, cleanup.getMessage());
}
}
@Override
public void deleteBranch(String repoRoot, String branch) {
exec("git", "-C", repoRoot, "branch", "-D", branch);
}
/**
* A linked worktree shares its primary checkout's git config. Remove HTTPS user info before
* adding one, so a credential accidentally embedded in that config cannot reach the member.
@@ -334,7 +334,20 @@ public final class SessionManager implements TurnListener {
// slot that no longer appears in the roster and can never be reclaimed.
launcher.stop(paneId);
if (removed != null && !preserveWorktree && removed.worktree() != null) {
worktrees.remove(worktrees.repoRoot(removed.cwd()), removed.worktree());
// fleetd #283: this is the one cleanup step in this method that used to be bare. By the
// time it runs, the registry entry, the retained handle, and the pane are all already
// gone — so a throw here (a stale index lock, a slow filesystem, `remove`'s own 30s exec
// timeout) must not escape release(): there is no retry path (a second stop on this
// paneId is a no-op), and the caller would otherwise see a "failed stop" for a session
// that is in fact fully torn down. Log and swallow, matching every sibling step above.
try {
worktrees.remove(worktrees.repoRoot(removed.cwd()), removed.worktree());
} catch (RuntimeException e) {
log.warn("failed to remove worktree {} for pane={} terminal={} after release: the "
+ "pane is already stopped and the session already deregistered, so this is "
+ "not retryable — the directory must be reclaimed manually: {}",
removed.worktree(), paneId, removed.terminalId(), e.toString());
}
}
}
@@ -504,11 +517,26 @@ public final class SessionManager implements TurnListener {
log.warn("spawn failed for profile={} role={} branch={} path={}: {}",
preResolvedProfile, memberRole, branch, path, e.getMessage());
if (path != null) {
// fleetd #283: this catch covers every failure AFTER worktrees.add() returned —
// overlayParity, shareWithGroup, launcher.spawn itself — so by this point `branch`
// was actually created in git. #274 fixed the sibling failure INSIDE add() by having
// GitWorktrees.cleanupAfterAddFailure delete both the worktree and the branch it
// provisioned; this path removed only the worktree and left the branch orphaned. A
// spawn failure here is routine (a quarantined credential, a backend refusal), so
// every occurrence leaked a `worker/<slug>-<nonce>` branch nothing ever pointed at
// again. Reuse the same Worktrees.deleteBranch GitWorktrees already has, rather than
// a second copy of the git command. Best-effort and log-only, like the worktree
// removal right above it — neither cleanup step may mask the original exception.
try {
worktrees.remove(repoRoot, path);
} catch (RuntimeException cleanup) {
log.warn("failed to clean up worktree {} after spawn error: {}", path, cleanup.getMessage());
}
try {
worktrees.deleteBranch(repoRoot, branch);
} catch (RuntimeException cleanup) {
log.warn("failed to clean up branch {} after spawn error: {}", branch, cleanup.getMessage());
}
}
throw e;
}
@@ -11,6 +11,15 @@ public interface Worktrees {
/** git -C <repoRoot> worktree remove --force <path>. Idempotent (already-gone tolerated). */
void remove(String repoRoot, String worktreePath);
/**
* git -C {@code repoRoot} branch -D {@code branch}. Force-deletes a branch that has no other
* owner — used only on the failed-provisioning path (fleetd #274, #283), never on a normal
* release: {@link SessionManager#release} deliberately leaves a released session's branch
* behind so a lead can still recover the work, and this method must never be called from
* that path.
*/
void deleteBranch(String repoRoot, String branch);
/**
* True when the worktree holds uncommitted changes the bridge cannot see: tracked
* modifications, staged files, or untracked files. {@code git status --porcelain} is the
@@ -226,4 +226,44 @@ class FleetdBackendErrorSinkTest {
assertTrue(remaining.isPresent(), "two distinct targets must start a cool-off");
assertEquals(1, leadClient.sendCount());
}
@Test
@DisplayName("a backend-error session no longer blocks the real maxLoad spawn gate")
void backendErrorSessionDoesNotBlockFreshSpawnAtMaxLoad() {
SessionManager sessions = capacityLimitedSessions();
MemberSession failed = sessions.acquire("terra", null, null, null);
assertTrue(sessions.onBackendError(failed.terminalId(), "backend exited"));
MemberSession fresh = sessions.acquire("terra", null, null, null);
assertEquals("terra", fresh.profile(), "the real maxLoad gate grants a fresh spawn after a backend error");
}
@Test
@DisplayName("a failed session no longer blocks the real maxLoad spawn gate")
void failedSessionDoesNotBlockFreshSpawnAtMaxLoad() {
SessionManager sessions = capacityLimitedSessions();
MemberSession failed = sessions.acquire("terra", null, null, null);
sessions.onTurnFailed(failed.terminalId());
MemberSession fresh = sessions.acquire("terra", null, null, null);
assertEquals("terra", fresh.profile(), "the real maxLoad gate grants a fresh spawn after a failed turn");
}
private static SessionManager capacityLimitedSessions() {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile profile = new FleetConfig.Profile("terra", "http://gx00.gw:8000", "coder",
null, "FLEETD_WORKER_TOKEN", List.of("claude"), "tab", "fleetd-workers",
"w #{n}", null, null, null, null, null, null, null, 1.0f, 1);
Map<String, FleetConfig.Profile> profiles = Map.of("terra", profile);
ClaudeCodeLauncher adapter = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), profiles, "terra", _ -> "tok");
AtomicReference<SessionManager> sessionsRef = new AtomicReference<>();
CompositePeerLauncher workers = new CompositePeerLauncher(List.of(adapter), "terra", profiles,
PlacementPolicies.fixed(), name -> Fleetd.liveSessionCount(sessionsRef.get().roster(), name));
SessionManager sessions = new SessionManager(workers);
sessionsRef.set(sessions);
return sessions;
}
}
@@ -7,6 +7,7 @@ import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
/**
@@ -41,6 +42,9 @@ public final class FakeHerdr implements HerdrClient {
private int agentPaneBusyFor = 0;
private int workerTabPaneCount = 1;
private String paneCloseErrorCode = null;
private final Map<String, String> paneCloseErrorCodeFor = new ConcurrentHashMap<>();
private String tabCloseErrorCode = null;
private final Map<String, String> tabCloseErrorCodeFor = new ConcurrentHashMap<>();
private String agentSendErrorCode = null;
private boolean noPanes = false;
private volatile String agentStatus = "idle"; // steady-state agent.get status
@@ -86,12 +90,44 @@ public final class FakeHerdr implements HerdrClient {
return this;
}
/** Make {@code pane.close} fail with this herdr error code. */
/** Make {@code pane.close} fail with this herdr error code, for every pane. */
public FakeHerdr paneCloseFailsWith(String code) {
this.paneCloseErrorCode = code;
return this;
}
/**
* Make {@code pane.close} fail with this herdr error code, but only for the given {@code
* pane_id} — every other pane's {@code pane.close} still succeeds. Unlike {@link
* #paneCloseFailsWith}, which fails every call regardless of which pane it targets, this lets a
* test reap/release several sessions at once and make exactly one of them fail to stop, so the
* others' teardown can be asserted to proceed normally (fleetd #290).
*/
public FakeHerdr paneCloseFailsForPane(String paneId, String code) {
this.paneCloseErrorCodeFor.put(paneId, code);
return this;
}
/** Make {@code tab.close} fail with this herdr error code, for every tab. */
public FakeHerdr tabCloseFailsWith(String code) {
this.tabCloseErrorCode = code;
return this;
}
/**
* Make {@code tab.close} fail with this herdr error code, but only for the given {@code
* tab_id} — every other tab's {@code tab.close} still succeeds. The {@code tab.close}
* counterpart to {@link #paneCloseFailsForPane} (fleetd #290): lets a test make exactly one
* session's tab teardown fail while proving the rest of {@code stop()} — {@code
* releaseZdotdir}, and the caller's worktree removal — still runs (fleetd #293). Named "ForTab"
* rather than "ForPane" (unlike its sibling) because {@code tab.close} keys on {@code tab_id},
* not a pane id.
*/
public FakeHerdr tabCloseFailsForTab(String tabId, String code) {
this.tabCloseErrorCodeFor.put(tabId, code);
return this;
}
/**
* Make {@code pane.list} report no panes at all — models a second herdr daemon (CB-185) that
* simply does not host the pane a {@link PaneLocator} is searching for.
@@ -338,7 +374,17 @@ public final class FakeHerdr implements HerdrClient {
.formatted(workerTabPaneCount,
seeded.isEmpty() ? "" : "," + String.join(",", seeded)));
}
case "tab.close" -> mapper.readTree("{\"type\":\"ok\"}");
case "tab.close" -> {
Object tabIdParam = params instanceof Map<?, ?> m ? m.get("tab_id") : null;
String perTabCode = tabIdParam == null ? null
: tabCloseErrorCodeFor.get(String.valueOf(tabIdParam));
String code = perTabCode != null ? perTabCode : tabCloseErrorCode;
if (code != null) {
throw new HerdrException("herdr error [" + code + "]: tab.close failed",
code, null);
}
yield mapper.readTree("{\"type\":\"ok\"}");
}
case "pane.get" -> mapper.readTree("""
{"type":"pane_info","pane":{"pane_id":"w9:pW","workspace_id":"w9",
"tab_id":"w9:t2","agent_status":"idle"}}""");
@@ -360,9 +406,13 @@ public final class FakeHerdr implements HerdrClient {
"foreground_processes":[]}}""");
}
case "pane.close" -> {
if (paneCloseErrorCode != null) {
throw new HerdrException("herdr error [" + paneCloseErrorCode + "]: pane.close failed",
paneCloseErrorCode, null);
Object paneIdParam = params instanceof Map<?, ?> m ? m.get("pane_id") : null;
String perPaneCode = paneIdParam == null ? null
: paneCloseErrorCodeFor.get(String.valueOf(paneIdParam));
String code = perPaneCode != null ? perPaneCode : paneCloseErrorCode;
if (code != null) {
throw new HerdrException("herdr error [" + code + "]: pane.close failed",
code, null);
}
yield mapper.readTree("{\"type\":\"ok\"}");
}
@@ -24,8 +24,13 @@ import io.modelcontextprotocol.spec.McpSchema;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import static org.junit.jupiter.api.Assertions.*;
@@ -44,6 +49,9 @@ import static org.junit.jupiter.api.Assertions.*;
*/
class FleetMcpAuthzTest {
private static final Path MCP_SOURCE = Path.of("src/main/java/dev/ltms/fleet/mcp/FleetMcp.java");
private static final Pattern TOOL_REGISTRATION = Pattern.compile("tool\\(\\\"(fleet_[a-z_]+)\\\"");
private final FakeHerdr herdr = new FakeHerdr();
private final AgentControl agents = new AgentControl(herdr);
private Metrics metrics;
@@ -201,6 +209,42 @@ class FleetMcpAuthzTest {
"a blank target is an absent target");
}
@Test
void everyRegisteredToolHasItsHandlerActionPinned() {
Set<String> registered = toolsTheServerRegisters();
assertTrue(registered.size() >= 10,
"scraped only " + registered.size() + " tool registrations from FleetMcp (" + registered
+ "); the server registers eleven, so the tool(\"…\") scrape has stopped matching");
registered.forEach(tool -> assertDoesNotThrow(() -> FleetMcp.toolAction(tool, Map.of()),
() -> tool + " is registered but has no pinned authorization action"));
assertEquals(Authz.Action.SEND, FleetMcp.toolAction("fleet_send", Map.of()));
assertEquals(Authz.Action.REPLY, FleetMcp.toolAction("fleet_reply", Map.of()));
assertEquals(Authz.Action.ASK, FleetMcp.toolAction("fleet_ask", Map.of()));
assertEquals(Authz.Action.READ, FleetMcp.toolAction("fleet_status", Map.of()));
assertEquals(Authz.Action.DRAIN, FleetMcp.toolAction("fleet_ack", Map.of()));
assertEquals(Authz.Action.SPAWN, FleetMcp.toolAction("fleet_spawn", Map.of()));
assertEquals(Authz.Action.READ, FleetMcp.toolAction("fleet_list", Map.of()));
assertEquals(Authz.Action.STOP, FleetMcp.toolAction("fleet_stop", Map.of()));
assertEquals(Authz.Action.READ, FleetMcp.toolAction("fleet_profiles", Map.of()));
assertEquals(Authz.Action.READ, FleetMcp.toolAction("fleet_whoami", Map.of()));
assertEquals(Authz.Action.READ, FleetMcp.toolAction("fleet_poll", Map.of("ticket", "task")));
assertEquals(Authz.Action.DRAIN, FleetMcp.toolAction("fleet_poll", Map.of("target", "term_b")));
}
private static Set<String> toolsTheServerRegisters() {
try {
Matcher matcher = TOOL_REGISTRATION.matcher(Files.readString(MCP_SOURCE));
Set<String> tools = new LinkedHashSet<>();
while (matcher.find()) {
tools.add(matcher.group(1));
}
return tools;
} catch (Exception e) {
throw new AssertionError("could not scrape FleetMcp tool registrations", e);
}
}
@Test
void aWorkerMayNotDrainAnotherSessionsInboxByPolling() {
FleetMcp m = mcp(true);
@@ -31,6 +31,7 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.EnumSet;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
@@ -604,6 +605,60 @@ class FleetMcpTest {
assertTrue(out.contains("\"reclaimable\":0"), out);
}
/**
* fleetd #284: {@code reclaimable} has exactly one definition, and this pins it over EVERY
* {@link MemberSession.State} — so a state added later cannot slip through unconsidered. Both
* views in a {@code fleet_list} response call {@link FleetMcp#reclaimable}, so they cannot
* drift apart.
*
* <p>{@code BACKEND_ERROR} and {@code FAILED} are NOT reclaimable on purpose. The ticket asked
* for them to be; that half of the ticket was wrong. Their seat is already out of
* {@code live}, so it is already in {@code free} — counting it here too would report the same
* seat twice.
*/
@Test
void onlyReadyAndDoneSessionsAreReclaimable() {
Set<MemberSession.State> expected = EnumSet.of(MemberSession.State.READY, MemberSession.State.DONE);
for (MemberSession.State state : MemberSession.State.values()) {
MemberSession session = new MemberSession("p1", "term1", "ltms-local", null,
"/tmp", null, 0L, 0L, 0, state, null, null);
assertEquals(expected.contains(state), FleetMcp.reclaimable(session, null),
"state " + state + " must " + (expected.contains(state) ? "" : "not ")
+ "count as reclaimable");
}
}
/**
* fleetd #284, the operator-visible half. {@code liveCount} here is the value the real counter
* ({@code Fleetd.liveSessionCount}, proven against the actual spawn gate in
* {@code FleetdBackendErrorSinkTest}) produces for this roster: 0, because both sessions are
* terminal. What this test pins is what {@code fleet_list} says around it — the two seats show
* up once, in {@code free}, and are NOT counted a second time as {@code reclaimable}; neither
* is any member row; and both dead sessions are still listed so the lead can see why.
*/
@Test
void terminalFailureSessionsFreeTheirSeatWithoutBeingCountedReclaimable() {
FakeHerdr h = new FakeHerdr();
SessionManager sessions = new SessionManager(workerService(h, "http://gx00.gw:8000", Set.of("gx00.gw")));
MemberSession backendError = sessions.acquire("ltms-local", null, null, null);
MemberSession failed = sessions.acquire("ltms-local", null, null, null);
assertTrue(sessions.onBackendError(backendError.terminalId(), "backend exited"));
sessions.onTurnFailed(failed.terminalId());
String out = textOf(FleetMcp.listFleet(workerService(h, "http://gx00.gw:8000", Set.of("gx00.gw")),
sessions, null, new FleetMcp.CapacitySource(profile -> 0, profile -> 2,
() -> Set.of("ltms-local"), () -> 0), new FleetMcp.HealthCoverageSource(() -> "off"),
FleetMcp.QuarantineSource.none(), Map.of(), ""));
assertTrue(out.contains("\"free\":2"), "both seats are back: " + out);
assertTrue(out.contains("\"reclaimable\":0"),
"the freed seats must not be counted a second time as reclaimable: " + out);
assertFalse(out.contains("\"reclaimable\":true"),
"no member row may claim a seat the profile count says is not held: " + out);
assertTrue(out.contains("\"state\":\"backend_error\""), "the dead session stays visible: " + out);
assertTrue(out.contains("\"state\":\"failed\""), "the failed session stays visible: " + out);
}
@Test
void inertCapacitySourceOmitsCapacityBlock() {
FakeHerdr h = new FakeHerdr();
@@ -925,6 +925,71 @@ class ClaudeCodeLauncherTest {
assertTrue(herdr.called("pane.close"), "stop via handle.id() must close the pane");
}
/** A tab-placement launcher with {@code memberCredentials policy=allow-list} under a zsh shell — the
* combination that makes {@link HerdrPeerLauncher#spawn} generate a real ZDOTDIR, so {@code
* releaseZdotdir}'s effect (the directory's deletion) is observable from a test. */
private ClaudeCodeLauncher serviceWithAllowList(FakeHerdr herdr) {
FleetConfig.Profile cfg = new FleetConfig.Profile(
"ltms-local", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN",
List.of("ccs", "ltms-local"), "tab", "fleetd-workers",
"worker: {profile} #{n}", null, null, null);
Supplier<FleetConfig.MemberCredentials> creds = () -> new FleetConfig.MemberCredentials(
FleetConfig.MemberCredentials.POLICY_ALLOW_LIST, List.of(), List.of(), null);
Function<String, String> env = name -> "SHELL".equals(name) ? "/bin/zsh" : null;
return new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(),
env, 0, System::currentTimeMillis, () -> { }, null, creds);
}
/**
* fleetd #293: {@code stop()} used to run {@code spaces.closeTab} bare — any non-{@code
* *_not_found} herdr error propagated straight out of {@code stop()}, skipping {@code
* releaseZdotdir} entirely (the pane was already closed by that point, so the tab-close failure
* is cosmetic, not a real teardown failure). Proves both halves of the fix: {@code stop()} no
* longer throws for this failure, and {@code releaseZdotdir} still runs — observed here by the
* generated ZDOTDIR actually being deleted, since {@code releaseZdotdir}'s last line is {@code
* EnvAllowListScrub.deleteRecursively(dir)}.
*/
@Test
@SuppressWarnings("unchecked")
void stopStillReleasesZdotdirWhenCloseTabFailsWithANonNotFoundCode() {
FakeHerdr herdr = new FakeHerdr();
ClaudeCodeLauncher svc = serviceWithAllowList(herdr);
PeerHandle handle = svc.spawn(new SpawnRequest(null, null, null));
Map<String, Object> tabCreateParams = (Map<String, Object>) herdr.lastCall("tab.create").params();
Map<String, String> tabEnv = (Map<String, String>) tabCreateParams.get("env");
String zdotdir = tabEnv.get("ZDOTDIR");
assertNotNull(zdotdir, "policy=allow-list under a zsh shell must have generated a ZDOTDIR: " + tabEnv);
Path dir = Path.of(zdotdir);
assertTrue(Files.isDirectory(dir), "the generated ZDOTDIR must exist before stop(): " + dir);
herdr.tabCloseFailsForTab("w9:t2", "internal_error");
Logger logger = (Logger) LoggerFactory.getLogger(HerdrPeerLauncher.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
try {
assertDoesNotThrow(() -> svc.stop(handle.id()),
"fleetd #293: a failing tab.close is cosmetic — it must not propagate out of stop()");
} finally {
logger.detachAppender(appender);
}
assertTrue(herdr.called("tab.close"), "tab.close was still attempted");
assertFalse(Files.exists(dir),
"releaseZdotdir must still run and delete the generated ZDOTDIR despite the tab.close "
+ "failure: " + dir);
String warn = appender.list.stream()
.filter(e -> e.getLevel().equals(Level.WARN))
.map(ILoggingEvent::getFormattedMessage)
.filter(m -> m.contains("tab.close") && m.contains("w9:t2"))
.findFirst()
.orElse(null);
assertNotNull(warn, "the failing tab.close must be logged at WARN naming the tab id — a "
+ "silently swallowed failure with no message is not an improvement. Log lines: "
+ appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList());
}
// --- CB-519: host-unique id, decoupled from the pane coordinate ------------------------------
@Test
@@ -2742,4 +2807,126 @@ class ClaudeCodeLauncherTest {
"a WARN naming the cwd must fire when the profile sets no configDir: "
+ appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList());
}
// --- fleetd #285: seedTrustDialog under memberHerdrSocket --------------------------------------
//
// seedTrustDialog gated only on isProvisionedWorktree(cwd) and, being static, could not see
// memberHerdrSocketConfigured() at all — unlike its sibling writeCharterFile one method below,
// which already refuses the spawn when it cannot place the charter where a different-uid member
// can read it. Under memberHerdrSocket + configDir unset, seedTrustDialog wrote fleetd's OWN
// ~/.claude.json (the operator's real file) while believing it was seeding the member's. The fix
// makes the method an instance method so it can see memberHerdrSocketConfigured(), and applies
// the same "refuse, don't silently write somewhere wrong" rule writeCharterFile already uses.
@Test
void seedTrustDialogUnderMemberHerdrSocketSharesTheFileWithTheConfiguredGroup(
@TempDir Path configDir, @TempDir Path worktree, @TempDir Path worktreeRoot) throws Exception {
markAsProvisionedWorktree(worktree);
String group = currentUserGroup();
FakeHerdr herdr = new FakeHerdr();
// trustProfile always sets a bridge mcpUrl, so a reply charter is generated too, which
// means writeCharterFile ALSO runs under memberHerdrSocket and needs its own worktreeRoot —
// pass one so this test isolates the trust-seed behaviour instead of tripping that refusal.
FleetConfig.Profile cfg = trustProfile(configDir.toString(), worktree.toString());
serviceWithConfig(herdr, cfg, () -> configWithMemberHerdrSocket(worktreeRoot.toString(), group)).spawn();
Path claudeJson = configDir.resolve(".claude.json");
assertTrue(Files.exists(claudeJson), "still seeded into <configDir>/.claude.json under memberHerdrSocket");
JsonNode project = new ObjectMapper().readTree(claudeJson.toFile())
.path("projects").path(worktree.toString());
assertTrue(project.path("hasTrustDialogAccepted").asBoolean(false));
assertEquals("rw-r-----", PosixFilePermissions.toString(Files.getPosixFilePermissions(claudeJson)),
"under memberHerdrSocket the file must be shared group-readable, mirroring the "
+ "charter file's own per-file mode (fleetd #222) — a 0600 file (Claude "
+ "Code's own default) is unreadable by the member's different OS user");
String actualGroup = Files.getFileAttributeView(claudeJson, PosixFileAttributeView.class)
.readAttributes().group().getName();
assertEquals(group, actualGroup, "the file must be chgrp'd to the configured worktreeGroup");
}
/**
* The bug itself: {@code memberHerdrSocket} configured, {@code configDir} unset. Before the fix
* this wrote fleetd's own default {@code ~/.claude.json} (here redirected to {@code fakeHome} so
* a reintroduced bug still cannot touch the real operator file); after the fix it must refuse the
* spawn instead, naming {@code configDir} as the missing key, before the member is ever started.
*/
@Test
void seedTrustDialogUnderMemberHerdrSocketRefusesWhenConfigDirUnset(
@TempDir Path fakeHome, @TempDir Path worktree) throws Exception {
markAsProvisionedWorktree(worktree);
String originalHome = System.getProperty("user.home");
System.setProperty("user.home", fakeHome.toString());
try {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = trustProfile(null, worktree.toString());
ClaudeCodeLauncher launcher = serviceWithConfig(herdr, cfg,
() -> configWithMemberHerdrSocket(null, "some-group"));
IllegalStateException ex = assertThrows(IllegalStateException.class, launcher::spawn,
"memberHerdrSocket + no configDir must refuse the spawn, not write fleetd's own "
+ "default ~/.claude.json");
assertTrue(ex.getMessage().contains("configDir"),
"the refusal must name the missing config key — got: " + ex.getMessage());
assertFalse(Files.exists(fakeHome.resolve(".claude.json")),
"nothing may be written to fleetd's own default home — this is the exact fleetd "
+ "#285 defect: writing the operator's own home instead of the member's");
assertFalse(herdr.called("agent.start"),
"the spawn must be refused BEFORE the member is ever started — got calls: " + herdr.calls);
} finally {
System.setProperty("user.home", originalHome);
}
}
/**
* {@code configDir} alone is not enough — without {@code worktreeGroup} the file fleetd writes
* stays {@code 0600} and the member's different OS user still cannot read it, so this must also
* refuse, naming {@code worktreeGroup} this time.
*/
@Test
void seedTrustDialogUnderMemberHerdrSocketRefusesWhenWorktreeGroupUnset(
@TempDir Path configDir, @TempDir Path worktree) throws Exception {
markAsProvisionedWorktree(worktree);
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = trustProfile(configDir.toString(), worktree.toString());
ClaudeCodeLauncher launcher = serviceWithConfig(herdr, cfg,
() -> configWithMemberHerdrSocket(null, null));
IllegalStateException ex = assertThrows(IllegalStateException.class, launcher::spawn,
"memberHerdrSocket + no worktreeGroup must refuse the spawn, not write an unreadable file");
assertTrue(ex.getMessage().contains("worktreeGroup"),
"configDir alone is not enough — got: " + ex.getMessage());
assertFalse(Files.exists(configDir.resolve(".claude.json")),
"nothing may be written when the file cannot be shared with the member's group");
assertFalse(herdr.called("agent.start"),
"the spawn must be refused BEFORE the member is ever started");
}
/**
* Regression proof: with {@code memberHerdrSocket} ABSENT — even given a LIVE, non-null {@code
* config} supplier (not merely {@code config == null}, which every other seedTrustDialog test in
* this file already exercises) — the write must stay byte-identical to before this fix: existing
* {@code 0600} permissions preserved, no chgrp/chmod attempted. This is the proof the ticket asks
* for: the path the whole live fleet uses today is unchanged by this fix.
*/
@Test
void seedTrustDialogPreservesExisting0600PermissionsWhenMemberHerdrSocketAbsentEvenWithALiveConfigSupplier(
@TempDir Path configDir, @TempDir Path worktree) throws Exception {
markAsProvisionedWorktree(worktree);
Path claudeJson = configDir.resolve(".claude.json");
Files.writeString(claudeJson, "{}");
assumeTrue(Files.getFileAttributeView(claudeJson, PosixFileAttributeView.class) != null,
"no POSIX permissions on this filesystem — skipping rather than failing");
Files.setPosixFilePermissions(claudeJson, PosixFilePermissions.fromString("rw-------"));
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = trustProfile(configDir.toString(), worktree.toString());
FleetConfig config = new FleetConfig(null, null, null, Map.of(), null, null, null, null, null,
null, null, null, null, null, null, null, null, null, null, null, null, null).withDefaults();
serviceWithConfig(herdr, cfg, () -> config).spawn();
assertEquals("rw-------", PosixFilePermissions.toString(Files.getPosixFilePermissions(claudeJson)),
"with memberHerdrSocket absent — even given a live config supplier — the seed must "
+ "stay byte-identical to before this fix: no chgrp/chmod attempted");
}
}
@@ -1,6 +1,7 @@
package dev.ltms.fleet.rest;
import dev.ltms.fleet.auth.CallerResolver;
import dev.ltms.fleet.auth.Authz;
import dev.ltms.fleet.auth.MemberRegistry;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
@@ -25,8 +26,15 @@ import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.LinkedHashSet;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import static org.junit.jupiter.api.Assertions.*;
@@ -36,6 +44,10 @@ import static org.junit.jupiter.api.Assertions.*;
*/
class FleetAppAuthTest {
private static final Path REST_SOURCE = Path.of("src/main/java/dev/ltms/fleet/rest/FleetApp.java");
private static final Pattern ROUTE_REGISTRATION =
Pattern.compile("app\\.(get|post|delete|put|patch)\\(\\s*\"([^\"]+)\"");
private final HttpClient http = HttpClient.newHttpClient();
private Javalin app;
private Metrics metrics;
@@ -92,6 +104,49 @@ class FleetAppAuthTest {
return http.send(b.build(), HttpResponse.BodyHandlers.ofString());
}
@Test
void everyRegisteredRouteHasItsHandlerActionPinned() {
Set<String> registered = routesTheServerRegisters();
assertTrue(registered.size() >= 15,
"scraped only " + registered.size() + " route registrations from FleetApp (" + registered
+ "); the app.<verb>(\"…\") scrape has stopped matching");
// Liveness must work before credentials can be checked, so this route is deliberately open.
assertTrue(registered.remove("GET /healthz"), "GET /healthz must stay an explicit ungated exception");
registered.forEach(route -> assertDoesNotThrow(() -> FleetApp.routeAction(route),
() -> route + " is registered but has no pinned authorization action"));
assertEquals(Authz.Action.METRICS, FleetApp.routeAction("GET /metrics"));
assertEquals(Authz.Action.SPAWN, FleetApp.routeAction("POST /members"));
assertEquals(Authz.Action.STOP, FleetApp.routeAction("DELETE /members/{paneId}"));
assertEquals(Authz.Action.SEND, FleetApp.routeAction("POST /sessions/{id}/message"));
assertEquals(Authz.Action.REPLY, FleetApp.routeAction("POST /sessions/{id}/reply"));
assertEquals(Authz.Action.DRAIN, FleetApp.routeAction("GET /sessions/{id}/replies"));
assertEquals(Authz.Action.ASK, FleetApp.routeAction("POST /sessions/{id}/ask"));
for (String route : Set.of("GET /sessions", "GET /agents", "GET /members", "GET /profiles",
"GET /member-credentials", "GET /sessions/{id}/status", "GET /tasks/{ticket}")) {
assertEquals(Authz.Action.READ, FleetApp.routeAction(route), route);
}
assertThrows(IllegalArgumentException.class, () -> FleetApp.routeAction("GET /healthz"));
}
private static Set<String> routesTheServerRegisters() {
try {
String source = Files.readString(REST_SOURCE).lines()
.filter(line -> {
String stripped = line.stripLeading();
return !(stripped.startsWith("//") || stripped.startsWith("*") || stripped.startsWith("/*"));
})
.collect(Collectors.joining("\n"));
Matcher matcher = ROUTE_REGISTRATION.matcher(source);
Set<String> routes = new LinkedHashSet<>();
while (matcher.find()) {
routes.add(matcher.group(1).toUpperCase(Locale.ROOT) + " " + matcher.group(2));
}
return routes;
} catch (Exception e) {
throw new AssertionError("could not scrape FleetApp route registrations", e);
}
}
// --- loopback-trust: the caller is the primary -------------------------------------------
@Test
@@ -17,6 +17,9 @@ public final class FakeWorktrees implements Worktrees {
public record RemoveCall(String repoRoot, String worktreePath) {
}
public record DeleteBranchCall(String repoRoot, String branch) {
}
public record OverlayCall(String repoRoot, String worktreePath,
List<String> requested, List<String> copied, List<String> skipWorktree) {
}
@@ -35,6 +38,7 @@ public final class FakeWorktrees implements Worktrees {
private final List<AddCall> addCalls = new CopyOnWriteArrayList<>();
private final List<RemoveCall> removeCalls = new CopyOnWriteArrayList<>();
private final List<DeleteBranchCall> deleteBranchCalls = new CopyOnWriteArrayList<>();
private final List<OverlayCall> overlayCalls = new CopyOnWriteArrayList<>();
private final List<RepoRootCall> repoRootCalls = new CopyOnWriteArrayList<>();
private final List<SnapshotCall> snapshotCalls = new CopyOnWriteArrayList<>();
@@ -51,6 +55,8 @@ public final class FakeWorktrees implements Worktrees {
private final AtomicLong snapshotSeq = new AtomicLong();
private volatile RuntimeException addFailure;
private volatile RuntimeException snapshotFailure;
private volatile RuntimeException removeFailure;
private volatile RuntimeException overlayFailure;
private volatile boolean dirty = false;
private volatile String repoRoot = "/repo";
private volatile String prefix = "/worktrees";
@@ -98,6 +104,21 @@ public final class FakeWorktrees implements Worktrees {
return this;
}
/** Make subsequent {@link #remove} calls throw (fleetd #283: a stale index lock, a slow
* filesystem, or {@code remove}'s own 30s exec timeout escaping the last, previously bare,
* step of {@link SessionManager#release}). */
public FakeWorktrees failRemove(String message) {
this.removeFailure = new WorktreeException(message);
return this;
}
/** Make subsequent {@link #overlayParity} calls throw (simulates a post-{@code add()} spawn
* failure — fleetd #283 defect 2 — so the {@code acquireWithWorktree} catch runs). */
public FakeWorktrees failOverlay(String message) {
this.overlayFailure = new WorktreeException(message);
return this;
}
/** Configure the value returned by {@link #wipRefs}. */
public FakeWorktrees withWipRefs(WipRefStats stats) {
this.wipRefs = stats;
@@ -132,6 +153,14 @@ public final class FakeWorktrees implements Worktrees {
@Override
public void remove(String repoRoot, String worktreePath) {
removeCalls.add(new RemoveCall(repoRoot, worktreePath));
if (removeFailure != null) {
throw removeFailure;
}
}
@Override
public void deleteBranch(String repoRoot, String branch) {
deleteBranchCalls.add(new DeleteBranchCall(repoRoot, branch));
}
@Override
@@ -146,6 +175,9 @@ public final class FakeWorktrees implements Worktrees {
@Override
public void overlayParity(String repoRoot, String worktreePath, List<String> overlay) {
if (overlayFailure != null) {
throw overlayFailure;
}
List<String> copied = new java.util.ArrayList<>();
List<String> skipped = new java.util.ArrayList<>();
for (String rel : overlay) {
@@ -218,6 +250,14 @@ public final class FakeWorktrees implements Worktrees {
return removeCalls.isEmpty() ? null : removeCalls.getLast();
}
public List<DeleteBranchCall> deleteBranchCalls() {
return List.copyOf(deleteBranchCalls);
}
public DeleteBranchCall lastDeleteBranch() {
return deleteBranchCalls.isEmpty() ? null : deleteBranchCalls.getLast();
}
public OverlayCall lastOverlay() {
return overlayCalls.isEmpty() ? null : overlayCalls.getLast();
}
@@ -74,6 +74,7 @@ class SessionManagerTest {
*/
private static final class RecordingWorktrees implements Worktrees {
private final List<String> removeCalls = new java.util.ArrayList<>();
private final List<String> deleteBranchCalls = new java.util.ArrayList<>();
private final List<String> snapshotCalls = new java.util.ArrayList<>();
private final java.util.Set<String> failRemoveFor = new java.util.HashSet<>();
private volatile boolean dirty = false;
@@ -114,6 +115,11 @@ class SessionManagerTest {
removeCalls.add(worktreePath);
}
@Override
public void deleteBranch(String repoRoot, String branch) {
deleteBranchCalls.add(branch);
}
@Override
public boolean hasUncommitted(String worktreePath) {
if (hasUncommittedFailure != null) {
@@ -158,6 +164,10 @@ class SessionManagerTest {
return List.copyOf(removeCalls);
}
List<String> deleteBranchCalls() {
return List.copyOf(deleteBranchCalls);
}
List<String> snapshotCalls() {
return List.copyOf(snapshotCalls);
}
@@ -989,8 +999,21 @@ class SessionManagerTest {
+ "dirty check threw");
}
/**
* fleetd #283 defect 1 changed this test's own premise, so its assertions are updated along
* with the production fix. Before #283, the middle session's worktree-removal failure escaped
* {@code release()} uncaught, and this test proved {@code reapIdle}'s own per-session try/catch
* (CB-581) kept the rest of the pass going regardless. Now that {@code release()} itself catches
* a worktree-removal failure (matching every sibling cleanup step in that method) and only logs
* a WARN, {@code release()} no longer throws for this reason — so all three idle sessions are
* released and counted, and the middle one's removal failure is now visible only as the WARN
* {@code release()} itself logs, not as a reap-loop catch. {@code reapIdle}'s own guard (for a
* failure {@code release()} still cannot swallow, e.g. from {@code launcher.stop}) is untouched
* by this ticket. This test no longer exercises that guard — reaching it now needs a failure
* that #283 does not catch inside {@code release()} itself.
*/
@Test
void reapIdleSurvivesOneSessionThatFailsToRelease() {
void reapIdleCountsAllThreeSessionsWhenOnlyItsWorktreeRemovalFails() {
long[] clock = {0};
FakeHerdr herdr = new FakeHerdr();
RecordingWorktrees worktrees = new RecordingWorktrees();
@@ -1004,8 +1027,8 @@ class SessionManagerTest {
sessions.asPresence().markPresent(a.terminalId());
sessions.asPresence().markPresent(b.terminalId());
sessions.asPresence().markPresent(c.terminalId());
// The middle session's worktree removal fails — release() propagates that, so this is the
// one call reapIdle's per-session guard must survive without skipping the rest of the pass.
// The middle session's worktree removal fails — fleetd #283 makes release() catch and log
// this itself, so it no longer propagates out of release() at all.
worktrees.failRemoveFor(b.worktree());
LoggerContext ctx = (LoggerContext) LoggerFactory.getILoggerFactory();
@@ -1026,14 +1049,16 @@ class SessionManagerTest {
.map(ILoggingEvent::getFormattedMessage)
.filter(m -> m.contains(b.paneId()))
.findFirst()
.orElse("no reap-failure WARN logged");
.orElse("no worktree-removal-failure WARN logged");
assertTrue(warn.contains(b.terminalId()), "the WARN names the failed session's terminal: " + warn);
assertTrue(warn.contains(b.worktree()), "the WARN names the failed session's worktree: " + warn);
} finally {
sessionLog.detachAppender(appender);
}
assertEquals(2, reaped, "the middle session's failure is logged, not counted as reaped");
assertEquals(3, reaped,
"fleetd #283: release() no longer throws for a worktree-removal failure, so reapIdle "
+ "counts all three idle sessions as reaped");
assertTrue(sessions.get(a.paneId()).isEmpty(), "the first session is still released");
assertTrue(sessions.get(c.paneId()).isEmpty(), "the third session is still released");
assertTrue(sessions.get(b.paneId()).isEmpty(),
@@ -1044,6 +1069,79 @@ class SessionManagerTest {
assertEquals(1, paneCloseCallsFor(herdr, "w9:pRoot_3"), "the third pane is stopped");
}
/**
* fleetd #290: the #283 fix above closed the one trigger this suite used for {@code
* reapIdle}'s own per-session try/catch (CB-581) — a worktree-removal failure is now caught
* and logged inside {@code release()} itself, so it never reaches {@code reapIdle}'s guard at
* all. This test restores coverage of that guard using the trigger the ticket names: {@code
* release()} calls {@code launcher.stop(paneId)} with no try/catch around it, so a failing
* {@code pane.close} propagates straight out of {@code release()} uncaught. {@link
* FakeHerdr#paneCloseFailsForPane} (added for this ticket) makes exactly the middle session's
* stop fail, while the other two still succeed, so this proves {@code reapIdle} keeps reaping
* the rest of the roster rather than aborting the whole pass.
*/
@Test
void reapIdleSurvivesOneSessionWhoseLauncherStopFails() {
long[] clock = {0};
FakeHerdr herdr = new FakeHerdr();
RecordingWorktrees worktrees = new RecordingWorktrees();
SessionManager sessions = sessionManager(herdr, worktrees, () -> clock[0]);
MemberSession a = sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-290a", null));
MemberSession b = sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-290b", null));
MemberSession c = sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-290c", null));
sessions.asPresence().markPresent(a.terminalId());
sessions.asPresence().markPresent(b.terminalId());
sessions.asPresence().markPresent(c.terminalId());
// Only the middle session's herdr pane fails to close — a and c stop normally. This is the
// trigger reapIdle's own guard is for, now that #283 closed the worktree-removal trigger.
herdr.paneCloseFailsForPane("w9:pRoot_2", "internal_error");
LoggerContext ctx = (LoggerContext) LoggerFactory.getILoggerFactory();
ch.qos.logback.classic.Logger sessionLog =
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger(SessionManager.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.setContext(ctx);
appender.start();
sessionLog.addAppender(appender);
sessionLog.setLevel(Level.WARN);
int reaped;
try {
clock[0] = 100;
reaped = sessions.reapIdle(10);
String warn = appender.list.stream()
.filter(e -> e.getLevel().equals(Level.WARN))
.map(ILoggingEvent::getFormattedMessage)
.filter(m -> m.contains("reap failed") && m.contains(b.paneId()))
.findFirst()
.orElse("no reap-failed WARN logged for the failing session");
assertTrue(warn.contains(b.terminalId()), "the WARN names the failed session's terminal: " + warn);
} finally {
sessionLog.detachAppender(appender);
}
assertEquals(2, reaped,
"the middle session's launcher.stop failure is not counted as reaped, but must not "
+ "abort reaping the other two");
assertTrue(sessions.get(a.paneId()).isEmpty(), "the first session is still released");
assertTrue(sessions.get(c.paneId()).isEmpty(),
"the third session is still reached and released — proves the pass did not abort "
+ "when the middle session's release() threw");
assertTrue(sessions.get(b.paneId()).isEmpty(),
"the middle session is still deregistered — release() removes it from the registry "
+ "before launcher.stop() runs, regardless of whether stop() then throws");
assertEquals(1, paneCloseCallsFor(herdr, "w9:pRoot_1"), "the first pane is stopped");
assertEquals(1, paneCloseCallsFor(herdr, "w9:pRoot_2"),
"the middle pane's stop was attempted, even though it failed");
assertEquals(1, paneCloseCallsFor(herdr, "w9:pRoot_3"), "the third pane is stopped");
assertEquals(List.of(a.worktree(), c.worktree()), worktrees.removeCalls().stream().sorted().toList(),
"the middle session's worktree removal never runs — release() throws before reaching "
+ "it — while the other two, unaffected, still have theirs removed");
}
@Test
void unchangedRegressionCleanCompletedReleaseStillRemovesTheWorktree() {
FakeHerdr herdr = new FakeHerdr();
@@ -1058,6 +1156,41 @@ class SessionManagerTest {
"COMPLETED release of a clean worktree still removes it");
}
/**
* fleetd #293: {@code HerdrPeerLauncher.stop()} used to run {@code spaces.closeTab} bare — a
* failing {@code tab.close} (any code other than {@code *_not_found}) propagated straight out
* of {@code stop()}. {@code SessionManager.release} calls {@code launcher.stop(paneId)} with
* no try/catch (fleetd #283 wrapped the WORKTREE-removal step further down, not this one), so
* the throw happened <em>before</em> that worktree-removal step ever ran — and by then {@code
* registry.remove(paneId)} had already run, so a second {@code stop} is a no-op: the worktree
* leaked with no retry path. The pane itself is already closed by the time {@code tab.close}
* runs, so its failure is cosmetic workspace tidying, not a real teardown failure. The fix
* wraps {@code closeTab} inside {@code stop()} so it no longer throws for this reason; this
* test proves both halves at once: {@code release()} does not throw, and it still removes the
* worktree.
*/
@Test
void releaseStillRemovesTheWorktreeWhenCloseTabFails() {
FakeHerdr herdr = new FakeHerdr();
RecordingWorktrees worktrees = new RecordingWorktrees();
SessionManager sessions = sessionManager(herdr, worktrees);
MemberSession s = sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-293a", null));
// FakeHerdr's pane.get always answers with tab_id "w9:t2" for a tab-placement spawn.
herdr.tabCloseFailsForTab("w9:t2", "internal_error");
assertDoesNotThrow(() -> sessions.release(s.paneId()),
"a failing tab.close is cosmetic (the pane is already closed by then) — it must not "
+ "propagate out of release()");
assertTrue(herdr.called("tab.close"), "tab.close was still attempted");
assertEquals(List.of(s.worktree()), worktrees.removeCalls(),
"release() must still remove the worktree even though tab.close failed — this is "
+ "the leak fleetd #293 reports: before the fix, release() never reached this "
+ "step at all");
assertTrue(sessions.get(s.paneId()).isEmpty(), "the session is still deregistered");
}
@Test
void unchangedRegressionDirtyCompletedReleaseStillPreservesTheWorktree() {
FakeHerdr herdr = new FakeHerdr();
@@ -212,9 +212,41 @@ class WorktreeSessionManagerTest {
assertEquals("/repo", remove.repoRoot());
assertEquals(s.worktree(), remove.worktreePath());
// The fake records no branch-delete calls because Worktrees.remove only removes the checkout.
assertTrue(worktrees.deleteBranchCalls().isEmpty(),
"a normal release must NEVER delete the branch — it is the worker's only recoverable "
+ "copy of committed work, and only the failed-provisioning path may remove it");
assertTrue(sessions.get(paneId).isEmpty(), "released session is no longer retrievable");
}
/**
* fleetd #283 defect 1. Every other cleanup step in {@code release()} is wrapped in try/catch,
* because {@code exec()} can throw on a non-zero exit or its own 30s timeout — this was the one
* step left bare. By the time it runs, the registry entry, the retained handle, and the pane are
* all already gone, so a throw here used to escape {@code release()} after the session was
* already fully torn down: a second stop on the same paneId is a no-op (nothing left to find),
* so there was no retry path, and the caller saw a failed stop for a session that was in fact
* gone. This test makes the worktree removal throw and asserts release() still completes with
* the pane stopped and the registry clean.
*/
@Test
void releaseCompletesAndStopsPaneEvenWhenWorktreeRemovalThrows() {
FakeHerdr herdr = new FakeHerdr();
FakeWorktrees worktrees = new FakeWorktrees().withRepoRoot("/repo").withPrefix("/wt")
.failRemove("stale index lock");
SessionManager sessions = new SessionManager(workerService(herdr), worktrees);
MemberSession s = sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-283-1", null));
String paneId = s.paneId();
sessions.release(paneId); // must not throw
assertTrue(herdr.called("pane.close"), "the pane is still stopped despite the removal failure");
assertEquals(1, worktrees.removeCalls().size(), "worktree removal was still attempted");
assertTrue(sessions.get(paneId).isEmpty(),
"the session is deregistered regardless of the removal failure");
assertEquals(0, sessions.size(), "the registry is left clean");
}
/**
* CB-576. A normal {@code COMPLETED} release whose worktree holds uncommitted work must NOT
* remove it — {@code --force} would destroy the worker's only copy. The bridge cannot see
@@ -352,6 +384,39 @@ class WorktreeSessionManagerTest {
assertEquals(0, sessions.size(), "failed acquire leaves no registry entry");
assertFalse(herdr.called("agent.start"), "spawn is never reached when add fails");
assertTrue(worktrees.removeCalls().isEmpty(), "no worktree was added, so none is removed");
assertTrue(worktrees.deleteBranchCalls().isEmpty(),
"add() itself never created the branch in git, so there is nothing to delete");
}
/**
* fleetd #283 defect 2. {@code acquireWithWorktree}'s catch covers every failure AFTER
* {@code worktrees.add()} returns — {@code overlayParity}, {@code shareWithGroup},
* {@code launcher.spawn} itself — so by the time it runs, {@code branch} was actually created in
* git. It removed only the worktree and forgot the branch, leaking a {@code worker/<slug>-<nonce>}
* branch on every routine spawn failure (a quarantined credential, a backend refusal). This test
* makes {@code overlayParity} (a post-add() step) throw and asserts the branch is deleted, the
* same way #274 already does for the sibling failure inside {@code add()} itself.
*/
@Test
void spawnFailureAfterAddDeletesTheOrphanedBranch() {
FakeHerdr herdr = new FakeHerdr();
FakeWorktrees worktrees = new FakeWorktrees().withRepoRoot("/repo").withPrefix("/wt")
.failOverlay("overlayParity failed");
SessionManager sessions = new SessionManager(workerService(herdr), worktrees);
assertThrows(WorktreeException.class, () ->
sessions.acquire("ltms-local", null, "/caller/proj", null,
new WorktreeRequest("cb-283-2", null)));
assertEquals(0, sessions.size(), "failed acquire leaves no registry entry");
assertFalse(herdr.called("agent.start"), "spawn is never reached when overlayParity fails");
assertEquals(1, worktrees.removeCalls().size(), "the worktree checkout is still removed");
assertEquals(1, worktrees.deleteBranchCalls().size(),
"the orphaned branch that add() actually created must also be deleted");
FakeWorktrees.DeleteBranchCall del = worktrees.lastDeleteBranch();
assertEquals("/repo", del.repoRoot());
FakeWorktrees.AddCall add = worktrees.lastAdd();
assertEquals(add.branch(), del.branch(), "the branch deleted is the exact one add() created");
}
@Test