Compare commits

...

232 Commits

Author SHA1 Message Date
Dai Ha 002329adb5 #309: clean partial worktrees after add failure
CI / contract (pull_request) Successful in 1m23s
CI / build (pull_request) Successful in 2m2s
2026-09-04 13:18:57 +07:00
Dai Ha 76672ff016 #306: the post-turn phase gets the same unknown-stall escape as a turn
CI / build (push) Successful in 1m43s
CI / contract (push) Successful in 1m50s
Four latches gate delivery in Injector, and only awaitingCompletion had a way
out of a sustained unknown streak. CB-109 added that escape because a worker
stuck in a state herdr cannot classify never produces a working->idle
boundary. The same is true during post-turn housekeeping, but the escape was
never extended there.

awaitingPostTurnPickup and postTurnObserved are both released only on an
injectable sample, so a worker that goes unknown and stays there wedges: the
target is polled forever, every later message to it is blocked by the delivery
gate, and no onTurnFailed fires, so the session sits at DONE and looks healthy.
The counter did not even increment, since ++unknownSinceTurn sits inside the
awaitingCompletion short-circuit.

postTurnPending needs no escape; it is cleared on the line after the listener
call that sets it.

The escape does not set turnFailed. The delegated turn already completed and
its waiter already resolved — what is outstanding is the /clear. Failing the
turn would drive SessionManager.onFailed on a session that genuinely finished.

Not reachable in the live configuration: the path needs lifecycle.clearAfterTurn,
which fleetd.yaml does not set. It becomes reachable as soon as anyone turns
that supported knob on.

Fixes #306
2026-09-04 13:06:29 +07:00
Dai Ha 9379f92c23 #305: one definition of loopback, so a worker cannot become the primary
CI / contract (push) Successful in 52s
CI / build (push) Successful in 1m40s
ConnectionIdentity and CallerResolver each kept their own isLoopback. They
drifted: the identity resolver accepted only 127.0.0.1, the authorization
check accepted all of 127.0.0.0/8.

A caller from 127.0.0.2 therefore had its identity resolution skipped, so it
carried no terminal, and CallerResolver reads a missing terminal as "not a
worker" — which under loopback-trust, the default mode, is the primary. A
worker got spawn, stop, send and drain. The skip also happens before the PID
ancestry walk, so that defence is bypassed too.

Being strict in ConnectionIdentity was not the safe direction. That predicate
decides whether identity is resolved at all, and resolution is what demotes a
worker, so every address it excluded was one where a worker became the lead.

Measured, not assumed: on Linux the whole 127.0.0.0/8 is bound to lo, and
binding a source of 127.0.0.2 on the fleet host succeeds (curl rc=7, the
connect refused rather than the bind). On macOS the source bind fails (rc=45),
so this workstation was never exposed.

The shared predicate also accepts the IPv4-mapped IPv6 form, which neither
copy handled. That one failed in the safe direction: a primary on
::ffff:127.0.0.1 was refused as anonymous.

No transport-level test binds a real 127.0.0.2 source — it cannot run on
macOS. The reasoning is recorded on the issue.

Fixes #305
2026-09-04 12:58:21 +07:00
Dai Ha 21ff63b11d #304: the member routes report a herdr failure instead of a bare 500
CI / build (push) Successful in 1m34s
CI / contract (push) Successful in 1m53s
POST /members and DELETE /members/{paneId} were the two routes in FleetApp
with no catch (HerdrException). FleetApp has no Javalin exception mapper, so
the exception escaped as the default 500 with the body "Server Error" — no
herdr code, no herdr message. fleet_spawn and fleet_stop catch the same
exception and report a named error, so this was the same one-door-guarded
shape as #297.

Both now go through the existing herdrError mapper: 404 when herdr says the
target is gone, 502 otherwise. That is an answer the caller can act on.

stopMember matters more than spawnMember. SessionManager.release deregisters
the session, notifies the release listener and preserves a dirty worktree
before it calls launcher.stop, so a throw from that stop arrives after the
teardown the caller asked for has already happened. A bare 500 told the caller
to retry and carried nothing to explain what went wrong.

The two existing tests that asserted 500 now assert 502 and check the error
body. Neither was about the status code: one guards that a failed teardown is
not reported as a successful 204, the other that a failed spawn still closes
its tab. Both properties are unchanged.

Fixes #304
2026-09-04 12:48:21 +07:00
Dai Ha 21844b54d7 #302: fleet_reply refuses blank content too, matching fleet_send
CI / contract (push) Successful in 1m15s
CI / build (push) Successful in 3m34s
MessageService.reply now throws on blank content. FleetMcp.reply guarded only
against null, and its handler is a bare BiFunction with no try/catch, so a
whitespace-only fleet_reply left the handler as an uncaught
IllegalArgumentException instead of the clean tool error null already got.
fleet_send has always used isBlank here; reply now matches it.

The worker found that null/isBlank difference and reported it as a correction
to my ticket, which had quoted the guard wrongly. It was right: I grepped the
error string and assumed the condition matched its sibling.
2026-09-04 12:36:23 +07:00
Dai Ha 4769481515 Merge #302: a reply with no content is refused, not silently delivered
REST read content with .path("content").asText(""), so a body missing the key
became an empty string that resolved the lead's waiter. The turn completed and
the lead saw a member that finished and reported nothing, indistinguishable
from one that genuinely said nothing. The guard went into MessageService.reply,
which both doors call, rather than being written a second time in FleetApp.
2026-09-04 12:33:47 +07:00
Dai Ha bbbb4c1eb3 fleetd #302: require content in MessageService.reply so a REST reply with no content cannot silently resolve a waiter
CI / build (pull_request) Successful in 1m44s
CI / contract (pull_request) Successful in 3m17s
2026-09-04 12:31:09 +07:00
Dai Ha 38c248e617 Merge #298: a released reply is requeued, not dropped
CI / contract (push) Successful in 1m27s
CI / build (push) Successful in 1m46s
release() cancelled the consumer and dropped its local record of deliveries the
broker still held as outstanding. Cancelling a consumer does not requeue them:
they stay unacked on the still-open channel until it or the connection closes.
So a held-but-undrained reply became permanently unreachable — a worker's
report lost with no error and no log line. It now nacks with requeue, after
cancelling, so a later own() can still receive it.
2026-09-04 12:21:30 +07:00
Dai Ha 9debc0de27 #297: render both profiles doors from one body builder, not two copies
CI / contract (push) Successful in 1m23s
CI / build (push) Successful in 1m31s
The merged change shared the QuarantineSource and OutageSource instances
between fleet_profiles and GET /profiles, so the two doors read identical
facts. It then rendered those facts through a character-for-character copy of
the loop, in a different file. Shared inputs do not make duplicated computation
safe: a later edit to the row shape lands on one door and not the other, and
the two disagree about a live outage. That is what #284 was.

The ticket caused this. It said 'read from the same shared instances' and 'do
not change FleetMcp', and together those made copying the loop the only legal
move. Extracting FleetMcp.profilesView and calling it from both is what the
ticket should have asked for.
2026-09-04 12:19:02 +07:00
Dai Ha f34361b263 Merge #297: REST reports what MCP reports
GET /agents and GET /members now map a herdr transport failure into the same
{error, detail} envelope every other handler in FleetApp uses, instead of
letting it escape to Javalin's default handling. GET /profiles now reports the
quarantined and coolingOff states, read from the same shared sources FleetMcp
reads. REST is the door a lead falls back to when its MCP mount drops, so it
was weakest exactly when it was load-bearing.
2026-09-04 12:16:00 +07:00
Dai Ha be5ba22c75 #298: AmqpReplyInbox.release() requeues held deliveries instead of dropping them
CI / contract (pull_request) Successful in 1m12s
CI / build (pull_request) Successful in 2m3s
release(target) used to cancel the target's consumer and clear held's local
record for it. Cancelling a consumer does not requeue the broker's in-flight
deliveries — they stay unacked on the still-open channel until a real
connection drop. So a held-but-undrained reply became permanently
unreachable: never acked, never nacked, never requeued, invisible to peek.

Fix: cancel the consumer first (so it can no longer receive redeliveries),
then nack-with-requeue every held delivery for that target before dropping
the local record. Nacking before the cancel was tried first but a real
broker demonstrated a race: the still-active consumer immediately received
the requeued message back, racing held.remove and leaving peek non-empty.
Cancelling first avoids that. A failed requeue is logged at WARN and does
not abort release(), matching the best-effort teardown style #293 settled
for HerdrPeerLauncher.stop().

Extends AmqpReplyInboxContractTest.releaseCancelsConsumer... to assert the
held delivery is recoverable via a later own(), not just absent from peek.
2026-09-04 12:14:00 +07:00
Dai Ha 85c90d440a #297: map HerdrException on GET /agents and /members; GET /profiles reports quarantine + cool-off
CI / contract (pull_request) Successful in 45s
CI / build (pull_request) Successful in 1m33s
Two REST-only visibility gaps, both against the same shared instances FleetMcp
reads (BackendQuarantine/BackendOutagePolicy), never recomputed:

- GET /agents and GET /members let a HerdrException escape uncaught, outside
  the {error, detail} envelope every other failure path in FleetApp uses.
  Both now route through the existing herdrError() helper, matching healthz/
  sessionStatus. GET /members is the endpoint's own comment names as the
  out-of-band path a lead falls back to when its MCP mount drops.
- GET /profiles omitted the two outage states fleet_profiles already reports:
  quarantined (CB-578 stage B) and coolingOff (fleetd #201 Unit 5). FleetApp
  now takes the SAME FleetMcp.QuarantineSource/OutageSource instances Fleetd
  wires into FleetMcp (extracted to local vars in Fleetd.java so both doors
  share one object, not two independently-built copies of the same rule).

FleetMcp itself is unchanged. Item 3 of the ticket (a capacity block on
GET /members) is explicitly out of scope and was not added.
2026-09-04 12:12:35 +07:00
Dai Ha e97502d550 drainAll: record that the timeout is a whole-drain budget, not a per-session grace
CI / build (push) Successful in 1m27s
CI / contract (push) Successful in 1m48s
The javadoc said 'for each session that is BUSY, poll up to timeoutNanos',
which reads as a per-session grace period. The deadline is taken once, before
the loop, so the first BUSY session can spend all of it. That is deliberate and
is the safer of the two designs: the drain is one phase of a shutdown sequence
that must finish inside launchd's exit window, and a per-session grace would
overrun it and get the daemon SIGKILLed part-way through, leaving the sessions
not yet reached with no clean release, no preserved-worktree log and no
snapshot. Found by a read-only hunt that read the code correctly and drew the
opposite conclusion from the wording.
2026-09-04 12:09:44 +07:00
Dai Ha aef14ff46e Merge #296: a failed spawn no longer leaks the pane it opened
Two exits created a pane and left it running. The readiness gate propagated an
unrelated herdr error without teardown, and spawnAsPane never closed the pane it
split when the peer failed to start. Neither could be cleaned up by the caller:
SessionManager.acquire never learns the pane id, because spawn throws before it
returns one. It removed the worktree anyway, so the leak was a live backend with
a deleted cwd, invisible to fleet_list and holding a seat nothing decremented.
2026-09-04 12:08:23 +07:00
Dai Ha cba516bda4 fleetd #296: close panes on failed spawn
CI / contract (pull_request) Successful in 1m38s
CI / build (pull_request) Successful in 2m7s
2026-09-04 12:05:26 +07:00
Dai Ha ba51e0c6cc #293: catch RuntimeException on closeTab, matching its sibling guard
CI / contract (push) Successful in 50s
CI / build (push) Successful in 2m15s
No behaviour change today. HerdrCodec wraps every encode/decode failure
and UnixSocketHerdrClient wraps every IOException, so HerdrException is
all closeTab can currently throw.

But releaseZdotdir five lines below catches RuntimeException, and the
whole point of this fix is that nothing here may mask the cleanups
below. Guarding against the expected exception type and staying bare
against any other is the same asymmetry the ticket exists to remove,
one level down. This stops a later change inside
WorkspaceControl.closeTab reopening it.
2026-09-04 11:47:42 +07:00
Dai Ha 086c59848e Merge #293: a failing tab.close no longer masks the cleanups below it 2026-09-04 11:46:00 +07:00
Dai Ha 0c10079755 #293: wrap the bare tab.close in HerdrPeerLauncher.stop()
CI / contract (pull_request) Successful in 1m41s
CI / build (pull_request) Successful in 1m59s
The pane is already closed by the time spaces.closeTab runs, so a failing
tab.close is cosmetic workspace tidying, not a real teardown failure. Left
bare, it propagated out of stop() and masked releaseZdotdir (ZDOTDIR leak)
and, worse, SessionManager.release()'s worktree removal (no self-heal,
no retry — the registry entry is already gone by then).

Wrap it in a try/catch that logs a WARN naming the tab id, matching the
"must not mask a real teardown failure above" comment already on
releaseZdotdir. isAlreadyGone is untouched — this continues past *any*
tab.close failure, not just *_not_found, since the failure is cosmetic
regardless of its cause.

Adds FakeHerdr#tabCloseFailsWith/tabCloseFailsForTab (the tab.close
counterpart to #290's paneCloseFailsForPane) plus two tests: one proving
releaseZdotdir still runs (the generated ZDOTDIR is deleted) and one
proving SessionManager.release() still removes the worktree, both with a
non-not_found tab.close failure.
2026-09-04 11:41:07 +07:00
Dai Ha ece2091b53 #280: states is now touched by two scheduler tasks, so make it concurrent
CI / contract (push) Successful in 48s
CI / build (push) Successful in 1m56s
The delayed re-check reads `states` from its own scheduled task, while
`tick` writes and prunes it. Both run on the single-threaded scheduler
Fleetd passes in today, so they are serialised — but nothing in the
class enforces that, and an unsynchronised HashMap read racing a resize
can spin a CPU forever rather than fail visibly.

`priors` and `orphanStreaks` stay plain maps: `tick` is still their only
toucher. The comment says which is which, so the next person does not
have to re-derive it.
2026-09-04 11:32:56 +07:00
Dai Ha b3f917e6f5 Merge #280: one bounded delayed re-check sweeps a ticket whose ask lapsed after GONE 2026-09-04 11:31:00 +07:00
Dai Ha fa39a5f55e #280: sweep a GONE/NEVER_READY target's lapsed fleet_ask once, not just on transition
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 1m58s
FleetHealthMonitor's fire-once-per-transition rule (CB-580) means a target
that is genuinely mid-fleet_ask when health first classifies it GONE is
correctly skipped (sweepAsking=false). But nothing re-fires abandon() once
that ask lapses on its own 55-115s later: FleetHealth.decide keeps reporting
GONE every tick, and reportTransition's previous==next guard never lets the
sweep run again. The ticket then sat PENDING forever, the same destination
#275 fixed for an explicit teardown, reached here by a health guess instead.

SessionManager.reapIdle only reaps READY/DONE sessions (SessionManager.java:844),
and a session mid-turn (including mid-ask) stays BUSY the whole time
(onDelivered sets BUSY, nothing clears it until the turn completes) — so
SessionReaper never releases such a session and onRelease's sweepAsking=true
path is never reached.

Fix: schedule one bounded, delayed re-check per terminal transition (not a
per-tick retry — that shape was rejected by CB-580). It fires failTerminalTarget
again after a delay that exceeds the worst-case ask-lapse window, and only if
the target is still classified in the same terminal state at that time, so a
recovered or since-released target is never reached into. sweepAsking stays
false throughout, so a ticket whose ask has not yet lapsed is still never
touched — same invariant abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer pins.

Proven with a mutation: neutering recheckTerminalTarget's body made
delayedRecheckSweepsATicketWhoseAskLapsedAfterGoneWasFirstObserved fail with
"expected: <FAILED> but was: <PENDING>", all 20 other FleetHealthMonitorTest
cases still green; restored and reran clean (1300 tests, 0 failures).
2026-09-04 11:28:12 +07:00
Dai Ha d5128a1d35 #290: use the imports FakeHerdr already has
CI / build (push) Successful in 1m33s
CI / contract (push) Successful in 1m39s
2026-09-04 11:27:51 +07:00
Dai Ha 61097e5cf0 Merge #290: restore coverage for reapIdle's per-session guard 2026-09-04 11:25:45 +07:00
Dai Ha ef507bcd12 #290: restore reapIdle's per-session guard coverage via a new launcher.stop() trigger
CI / contract (pull_request) Successful in 1m29s
CI / build (pull_request) Successful in 1m56s
#283 fixed release() to catch and log a worktree-removal failure, which closed off
reapIdleCountsAllThreeSessionsWhenOnlyItsWorktreeRemovalFails as a trigger for
reapIdle's own per-session try/catch (CB-581) — that test now proves a different,
still-real thing (a swallowed removal failure doesn't shrink the reaped count),
but the try/catch itself lost its test.

Add FakeHerdr.paneCloseFailsForPane(paneId, code) so a test can make exactly one
session's launcher.stop() fail while its siblings still tear down normally
(paneCloseFailsWith already existed but fails every pane, which cannot isolate
one session in a three-session reap). Add
reapIdleSurvivesOneSessionWhoseLauncherStopFails beside the #283 test, using
launcher.stop() as the trigger the ticket names, and prove it catches removal of
reapIdle's try/catch: deleting the guard makes the test fail with the
HerdrException propagating out of reapIdle uncaught (quoted in the PR body).
2026-09-04 11:23:52 +07:00
Dai Ha 94f50e507a #284/#285: one reclaimable rule, one group-share helper
CI / contract (push) Successful in 1m2s
CI / build (push) Successful in 2m11s
Two corrections on top of the merged worker branches.

#284: I told the worker to report a BACKEND_ERROR/FAILED session as
reclaimable. That half of my own ticket was wrong. Once the live count
stops counting a terminal session, its seat is already in `free`;
counting it in `reclaimable` too reports the same seat twice, and
`free + reclaimable` reads as more capacity than maxLoad allows. Worse,
only the profile-level count was widened, so the same fleet_list
response said `reclaimable: 2` while every member row said
`reclaimable: false`.

Both views now call one shared predicate, FleetMcp.reclaimable, so they
cannot drift apart. A test runs it over every MemberSession.State value,
so a state added later cannot slip through unconsidered.

#285: the new per-file chgrp+chmod helper moved from ClaudeCodeLauncher
into EnvAllowListScrub as shareFileWithGroup, next to the directory-wide
shareWithGroup it was copied from. It now reuses that class's own
setGroupAndPermissions and also catches UnsupportedOperationException,
which the copy missed — on a filesystem without POSIX group ownership
the copy threw a raw runtime exception instead of the sibling's
UncheckedIOException.
2026-09-04 11:13:01 +07:00
Dai Ha 75b15086b0 Merge #285: the workspace-trust seed refuses rather than write fleetd's own home 2026-09-04 11:08:44 +07:00
Dai Ha dab9645906 Merge #284: a BACKEND_ERROR or FAILED member no longer holds a spawn seat 2026-09-04 11:04:09 +07:00
Dai Ha e93b5f6512 #282: record that answer()'s QUESTION guard is defence in depth, not load-bearing
CI / contract (push) Successful in 51s
CI / build (push) Successful in 1m44s
Measured after merging: removing the guard alone leaves the new test green,
because ask() calls markAsyncQuestion before resolveQuestion, so the task has
already moved to the new turnId. The PR claimed each half was necessary; only
the pair is. Keeping the guard, with the ordering written down so nobody
deletes it as dead code or trusts it as the only protection.
2026-09-04 10:59:10 +07:00
Dai Ha bfabe13e8f Merge #282 (PR #289): a chained second fleet_ask no longer kills its own async ticket 2026-09-04 10:56:42 +07:00
Dai Ha 7e49c6eca2 #285: seedTrustDialog refuses under memberHerdrSocket instead of seeding fleetd's own home
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 3m35s
seedTrustDialog gated only on isProvisionedWorktree(cwd) and, being static, could not
see memberHerdrSocketConfigured() — unlike its sibling writeCharterFile, which already
refuses the spawn when it cannot place a file where a different-uid member can read it.
Under memberHerdrSocket + configDir unset, seedTrustDialog wrote fleetd's OWN
~/.claude.json while believing it was seeding the member's, reintroducing the fleetd
#149 failure (interactive trust dialog, no fleet_reply, silent readiness timeout) for
this one config combination.

Makes seedTrustDialog an instance method so it can see memberHerdrSocketConfigured()
and memberGroup(), and applies writeCharterFile's "refuse, don't degrade" rule: under
memberHerdrSocket it now requires both configDir and worktreeGroup before touching any
file, naming exactly which is missing, and shares the written .claude.json group-
readable (rw-r-----) via a new shareTrustJsonWithGroup so the member's OS user can
actually open it. The memberHerdrSocket-absent path (today's only live mode) is
unchanged.
2026-09-04 10:55:35 +07:00
Dai Ha 11cbfa79b4 fleetd #284: free failed-session capacity
CI / contract (pull_request) Successful in 1m11s
CI / build (pull_request) Successful in 1m19s
2026-09-04 10:55:21 +07:00
Dai Ha c2c2746922 Merge #283 (PR #288): guard release()'s worktree removal, delete the orphaned branch on spawn failure
CI / contract (push) Successful in 52s
CI / build (push) Successful in 1m44s
2026-09-04 10:54:36 +07:00
Dai Ha 6ed70700a0 #282: don't let a chained fleet_ask kill its own async ticket
CI / contract (pull_request) Successful in 1m6s
CI / build (pull_request) Successful in 1m38s
answer() opened a fresh forward waiter but, unlike send(), never
registered it in asyncTasksByWaiter. So when a worker chained a
second fleet_ask inside the same resumed turn (before calling
fleet_reply), markAsyncQuestion had no Task to re-associate, and
answer() then completed the async ticket's future with the second
QUESTION as if it were a terminal reply — fleet_poll reported FAILED
while the worker was still alive and mid-conversation.

Fix: register answer()'s waiter in asyncTasksByWaiter (mirroring
send()) so a chained ask can re-arm the ticket under its new turnId,
and guard answer()'s finishAsyncTask call the same way sendAsync's
own lambda already does (skip on Outcome.QUESTION). Also drop the
stale asyncTasksByTurn entry left behind when markAsyncQuestion
re-arms a task under a new turnId, a leak the fix makes reachable
for the first time.

Reachability confirmed by driving the exact sequence through the
public API (sendAsync -> ask -> answer -> ask again) in a new test;
reverting the production change makes it fail with
"expected: <ASKING> but was: <FAILED>", confirming it catches the
regression.
2026-09-04 10:52:54 +07:00
Dai Ha 0087645da4 Merge #281 (PR #286): pin the Authz.Action every handler chooses
CI / contract (push) Successful in 1m1s
CI / build (push) Failing after 1m45s
2026-09-04 10:50:43 +07:00
Dai Ha 19cacf5b62 #283: guard release()'s bare worktree remove; delete orphaned branch on spawn failure
CI / contract (pull_request) Successful in 52s
CI / build (pull_request) Successful in 2m42s
Two teardown-cleanup leaks in SessionManager, same shape as #274.

Defect 1: release()'s last step (removing a released session's worktree)
was the one cleanup step in the method left unguarded, even though every
sibling step is wrapped because exec() can throw on a non-zero exit or its
own 30s timeout. By the time it ran, the registry entry, retained handle,
and pane were already gone, so a throw here escaped release() with no
retry path and made a fully-torn-down session look like a failed stop.
Now wrapped in try/catch with a WARN, matching the pattern already used
by every other step in this method.

Defect 2: acquireWithWorktree's catch (covering failures after add()
returns — overlayParity, shareWithGroup, launcher.spawn) removed the
worktree but left the branch it provisioned orphaned. #274 already fixed
the sibling failure inside add() itself (GitWorktrees.cleanupAfterAddFailure
deletes both). Extracted that branch-delete into a new Worktrees.deleteBranch
method, reused by both cleanupAfterAddFailure and this catch, so a routine
spawn failure (quarantined credential, backend refusal) no longer leaks a
worker/<slug>-<nonce> branch.

A normal release() still never deletes a branch — only the failed-provision
path does. releaseRemovesWorktreeButDoesNotDeleteBranch pins this, and
spawnFailureAfterAddDeletesTheOrphanedBranch / unchangedRegression* prove
the two paths stay apart.
2026-09-04 10:50:30 +07:00
Dai Ha 4dd12083ab fleetd #284: free backend-error capacity
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 2m42s
2026-09-04 10:47:09 +07:00
Dai Ha 30e21adec7 #281: cover registered authorization actions
CI / contract (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 1m33s
2026-09-04 10:46:26 +07:00
Dai Ha 719b79f892 wip: pin handler authorization actions
CI / build (pull_request) Successful in 1m30s
CI / contract (pull_request) Successful in 2m27s
2026-09-04 10:39:58 +07:00
Dai Ha 66e5247b6d Merge #275 (PR #279): sweep an ASKING ticket on a definite teardown
CI / contract (push) Successful in 49s
CI / build (push) Successful in 1m39s
A member torn down while parked in fleet_ask left its async ticket pending
for good. resolveQuestion had already closed the forward waiter, so
abandon()'s waiter branch found nothing; the 'question == null' guard then
excluded the task from the matching loop. By the time the worker's own ask
lapsed (~55-115s), the released session was gone from the roster, so
nothing was left to call abandon() on that target again. fleet_poll{ticket}
reported PENDING forever.

The ticket told the worker to drop the 'question == null' guard. That was
wrong, and the worker said so with evidence: an existing test
(abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer) deliberately pins that
an ASKING ticket must SURVIVE abandon(), because the primary may be mid
answer() for that same turn. Widening the shared method would have traded
this bug for a worse one — a health guess killing a live conversation.

So the fix splits the two callers by what they actually know:

  - sessions.onRelease (fleet_stop / idle reaper) knows the pane is being
    stopped right now, so it sweeps: sweepAsking=true.
  - FleetHealthMonitor keeps sweepAsking=false. GONE/NEVER_READY is a
    classification from the live agent list, not a teardown it performed.

I verified the reachability chain myself rather than taking it on trust.
FleetHealth.decide returns GONE before it can ever return
DELEGATION_ORPHANED; FleetHealthMonitor.reportTransition returns early when
previous == next; and terminal() is GONE/NEVER_READY only. So after the one
GONE transition fires and no-ops, nothing re-fires. Every link holds.

Verified: the real merge into current main builds green (1283 tests), the
protective test still passes untouched, and my own mutation — reverting the
sweepAsking widening — fails the new test with 'a released target's open ask
can never resume, so it must fail right here'.
2026-09-04 10:18:59 +07:00
Dai Ha 1e41bd63b4 Merge #274 (PR #277): clean up the worktree when add() fails after creating it
CI / contract (push) Successful in 59s
CI / build (push) Successful in 1m34s
GitWorktrees.add() created the worktree and branch, then ran more steps that
can throw — requireCredentialFreeHttpsOrigin among them, which is an
intended security refusal, not an IO accident. Any throw meant add() never
returned, so SessionManager.acquireWithWorktree never learned the path, its
'if (path != null)' cleanup could not fire, and the worktree and branch
leaked with nothing tracking them. Every OTHER exit from that method was
cleaned up correctly; only the exits inside add() were uncounted.

add() now cleans up what it created before rethrowing, reusing remove() and
additionally deleting the branch — a branch that never finished provisioning
has no session and no PR behind it. Worktree first, since a checked-out
branch cannot be deleted. Cleanup failure is logged and never masks the
original exception.

Verified by me: the real merge into current main builds green (1281 tests),
and I reran the mutation myself without git stash — dropping the cleanup
call fails the new test with 'the worktree directory leaked after a
post-creation step threw'.

The test drives add() itself through the existing afterWorktreeAdded seam,
so the failure happens after the worktree exists rather than downstream in
another caller.
2026-09-04 10:14:05 +07:00
Dai Ha 4887d03d88 #275: abandon() sweeps an ASKING ticket only on a definite teardown
CI / contract (pull_request) Successful in 1m21s
CI / build (pull_request) Successful in 2m5s
Confirmed reachable: a target torn down for good (fleet_stop / the idle
reaper) while its async ticket sits in fleet_ask (Phase.ASKING) got
permanently stuck. resolveQuestion already closes the forward waiter, the
question == null guard excluded the task from abandon()'s sweep, and by the
time the worker's own fleet_ask lapses (~55-115s) the released session no
longer appears in FleetHealthMonitor's roster, so nothing ever calls
abandon() again. fleet_poll{ticket} then reports PENDING forever.

Add abandon(target, reason, sweepAsking) — sessions.onRelease (a definite
teardown: the pane is being stopped right now) passes true and now fails the
ASKING ticket and closes its reverse-rendezvous ask. FleetHealthMonitor's
health-classification call keeps the 2-arg overload (sweepAsking=false):
a GONE/NEVER_READY reading is a guess from the live agent list, not a
teardown it performed, and abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer
already covers why an active ask must survive that guess (the primary may
be mid-answer for the same turn). hasOrphanedDelegation is left unchanged
for the same reason — it must not flag a live, active ask as orphaned.

Proven with a test driving the real public sequence (sendAsync -> ask ->
abandon(..., true)), not a hand-built task map; reverted the widening to
confirm it goes red, then restored it.
2026-09-04 10:13:56 +07:00
Dai Ha 7d5434455d implementer: never git stash — the stash stack is shared across worktrees
CI / build (push) Successful in 1m41s
CI / contract (push) Successful in 1m48s
A worker's worktree is isolated; refs/stash is not. It is one stack shared
by the primary's checkout and every worker worktree of this repo.

This bit a real worker today. Two ran in parallel; one called git stash
while the other was mid-edit, and the second worker's in-progress change
was silently overwritten by the first's stashed content. It recovered by
retyping the edit and diffing to confirm, and pushed the other worker's
change back onto the stack untouched — but nothing warned either of them,
and nothing would have.

Measured before writing this: 'git stash list' from a worker worktree and
from the primary's checkout return byte-identical output, and refs/stash
is a single common ref, not a per-worktree one.

The branch already IS the isolation, so the skill now points at committing
a wip commit or writing a patch file instead.
2026-09-04 10:10:08 +07:00
Dai Ha f71ee4926e Merge #273 (PR #278): validate exhaustedPattern at config load
CI / contract (push) Successful in 44s
CI / build (push) Successful in 1m44s
A malformed exhaustedPattern passed FleetConfig.load and then crashed the
daemon at startup, in Fleetd.main's unguarded Pattern.compile, with a
message naming neither the profile nor the key. Its sibling errorPattern
had a load-time validator whose own javadoc explains exactly why that is
bad. The validator was correct; its coverage was not.

rejectMalformedErrorPattern becomes rejectMalformedProfilePatterns and now
compiles both keys, reporting failures from either in one message.

Verified by me, not taken on the worker's word: the actual merge of this
branch into main builds green (1280 tests), and I reran the mutation myself
— narrowing the loop back to errorPattern turns exactly the two new tests
red, one of them with 'Expected IllegalStateException to be thrown, but
nothing was thrown', which is the defect stated out loud.
2026-09-04 10:09:15 +07:00
Dai Ha 282a2fc2b8 fleetd #274: clean up the worktree and branch when add() fails after creating them
CI / contract (pull_request) Successful in 56s
CI / build (pull_request) Successful in 1m22s
GitWorktrees.add() created the worktree and branch, then ran several more
steps that can throw (requireCredentialFreeHttpsOrigin — an intended
security refusal, not only an IO accident — plus the credential-helper and
tool-surface isolation steps). Any exception there meant add() never
returned, so its caller (SessionManager#acquireWithWorktree) never learned
the path: its local `path` stayed null, the `if (path != null)` cleanup
guard never ran, and the worktree directory and branch leaked on disk
forever with nothing tracking them.

Wrap those steps in try/catch; on failure, clean up via the same
`git worktree remove --force` path remove() already uses, additionally
force-delete the new branch (remove() alone deliberately leaves a
released session's branch behind, but a branch that never finished
provisioning has nothing else pointing at it), log the cleanup outcome,
and rethrow the original exception so it is never masked.

Test drives add() itself via the existing afterWorktreeAdded seam with a
mutation that trips requireCredentialFreeHttpsOrigin after the worktree
exists, then asserts both the worktree directory and the branch are gone.
Reverting the fix (git stash on GitWorktrees.java, test unchanged) turns
it red: "the worktree directory leaked after a post-creation step threw
==> expected: <false> but was: <true>". Restored afterward.

mvn clean install: BUILD SUCCESS, Tests run: 1275, Failures: 0, Errors: 0
2026-09-04 10:05:59 +07:00
Dai Ha f04e934b94 fleetd #273: validate exhaustedPattern regex at load, like errorPattern
CI / build (pull_request) Successful in 1m36s
CI / contract (pull_request) Successful in 2m11s
FleetConfig.rejectMalformedErrorPattern only compiled errorPattern eagerly
at config load. exhaustedPattern was compiled unguarded in Fleetd.main,
so profiles.<name>.exhaustedPattern: "[" passed load() and then crashed
the whole daemon at boot with a raw PatternSyntaxException naming neither
the profile nor the key.

Rename the validator to rejectMalformedProfilePatterns and extend it to
also compile every non-blank exhaustedPattern, reporting
profiles.<name>.exhaustedPattern ("<value>"): <message> in the same style
as errorPattern. Both keys are collected and reported together from a
single load. Fleetd.java's compile site is left as-is per scope — it is
now safe because load already rejects a bad value.

Added tests covering: a bad exhaustedPattern is refused; a bad pattern in
each key is reported together in one message; valid patterns still load;
a blank/absent exhaustedPattern is ignored.
2026-09-04 10:05:36 +07:00
Dai Ha 3fae35c357 #276: say whose environment the "allowed N of M" line counted
CI / contract (push) Successful in 55s
CI / build (push) Successful in 1m42s
A gap in my own #269 fix. That ticket stopped four sites claiming things
about the member's environment that fleetd cannot see when memberHerdrSocket
is configured, and gave the WARN in logCredentialGap a guard. The INFO line
called two lines earlier never got one:

    logAllowListCoverage(allowed);      // no guard
    logCredentialGap(creds, allowed);   // guarded since #269

Read plainly, "member credentials: allowed 7 of 39" is a statement about the
member's credentials. Under memberHerdrSocket the pane is routed to a second
herdr whose environment fleetd has no channel to inspect, so those counts
come from fleetd's own process instead. Same overclaim #269 existed to
remove, in the line next door.

The method's javadoc does carry the caveat, by cross-reference to another
field's javadoc. That does not help the operator reading fleetd.out.

The counts stay useful, so this is not a WARN and not a refusal — only the
claim is narrowed. The unguarded path keeps its exact original wording, so
the existing assertion on "member credentials: allowed 1 of 3" still holds.

The new test pins the pair together so a later edit cannot fix one line and
leave the other. Mutation-proved: with the guard removed it fails printing
the old line verbatim.

Also worth recording: no test covered #269's own guard — that WARN wording
shipped unverified, and still has no coverage.
2026-09-04 10:01:23 +07:00
Dai Ha 18aecbfe67 #272: fleet_poll{target} is a drain, so gate it as one
CI / build (push) Successful in 1m46s
CI / contract (push) Successful in 2m29s
fleet_poll is two operations behind one tool name. With `ticket` it observes
an async delegation and changes nothing. With `target` it calls
MessageService.drainReplies, which REMOVES the replies — a second call
returns nothing.

The handler gated both branches with a constant Authz.Action.READ, and did
not pass the target at all. READ is open to every authenticated role, so any
worker could read a peer's sessionId out of fleet_list and destroy the
replies that peer had queued for the primary. The gate failed open, and a
drained reply is not recoverable.

Three things already said the tight gate was intended:

  - fleet_ack, four lines below, gates the same drain as DRAIN, with a
    comment giving the exact reasoning missed here ("Acking removes a reply
    from the inbox, so it is a drain, not a read").
  - the REST path checks DRAIN in FleetApp.drainReplies.
  - wiki/2-Message-Server.md lists fleet_poll as lead-only, and the tool
    schema says "drain that worker's inbox".

Nothing that works today breaks: the documented flow is fleet_poll{target}
then fleet_ack{target,msgId}, and fleet_ack is already primary-only. A
worker could never complete that flow — only destroy its first half.

The required action is a function of the arguments, but the handler chose it
before looking at them. pollAction(target) makes that choice explicit. The
ticket branch stays READ on purpose: an architect may fleet_send, so it owns
tickets and must be able to poll them.

Why the suite missed it: FleetMcpAuthzTest checks every Action against every
Role, including "a worker may not DRAIN", and passed the whole time. The
policy table was right; the action fed to it was wrong, and nothing tested
that mapping. The new tests assert against pollAction itself, so the handler
keeps no private copy of the rule.

Mutation-proved: reverting pollAction to a constant READ turns exactly the
two new defect tests red and leaves the ticket-branch test green.

Introduced in 9daf1ec, where Authz.READ's own javadoc ("...task polling")
describes only the ticket half.
2026-09-04 09:54:52 +07:00
Dai Ha 5d75f72473 Merge #271: warn when the model check cannot run for a no-worktree opencode spawn (#267)
CI / contract (push) Successful in 1m18s
CI / build (push) Successful in 1m40s
2026-09-04 08:38:05 +07:00
Dai Ha e028a0ae54 fleetd #267: warn once per profile when the model check can't run
CI / contract (pull_request) Successful in 1m6s
CI / build (pull_request) Successful in 1m50s
OpenCodeLauncher.SessionAwareHandle.agentSessionId() is the only caller of
checkModelMatch (fleetd #175), and it sits behind the fleetd #249 worktree
gate. A spawn with no worktree:true — the ordinary shape of most opencode
spawns — never reached the check at all, and the gap was totally silent.

The check cannot be decoupled from agentSessionId()'s resolved id: doing so
would re-derive 'whatever is newest in the shared directory' and reintroduce
the false-positive risk fleetd #234 fixed (a sibling's differently-configured
model looking like a mismatch for a profile that never actually ran it). The
#249 gate is correct and stays as-is.

Instead, log once per profile at WARN, naming the profile, the same
treatment discoveryUnavailable already gets a few lines above — a logged
UNKNOWN beats a check that silently never runs.
2026-09-04 08:36:12 +07:00
Dai Ha 2fa673d4c0 Merge #270: ArchUnit package-cycle test with explicit accepted exceptions (#131)
CI / contract (push) Successful in 1m0s
CI / build (push) Successful in 1m57s
2026-09-04 08:22:35 +07:00
Dai Ha 9020d01b40 Merge #268: rename sshAuthSock values to omit/inherit with a read-both shim (#266) 2026-09-03 20:27:55 +07:00
Dai Ha 1006805027 fleetd #131: enforce package boundaries with an ArchUnit cycle test
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 2m1s
Adds PackageCyclesTest, which fails the build on any new cycle between
the top-level dev.ltms.fleet.* packages. Today's five real cycles are
recorded as narrow, explicit exceptions (ignoreDependency per named
pair, both directions), each commented with the ticket step (or a note
that it needs its own) that removes it. No package moves in this PR.

archunit-junit5 1.5.0 (current stable, newer than an earlier 1.4.1
draft). Main code only (DO_NOT_INCLUDE_TESTS) and importPackages(...)
instead of a working-directory-relative target/classes path.
2026-09-03 20:22:46 +07:00
Dai Ha 27aefbf9a0 Merge #269: stop claiming memberHerdrSocket proves a different OS user (#184 item 5)
CI / contract (push) Successful in 1m26s
CI / build (push) Successful in 1m41s
2026-09-03 20:14:44 +07:00
Dai Ha d42c2bc204 fleetd #266: rename SSH agent environment setting
CI / contract (pull_request) Successful in 1m29s
CI / build (pull_request) Successful in 2m1s
2026-09-03 20:12:46 +07:00
Dai Ha 3916adc372 fleetd #184: stop claiming memberHerdrSocket proves a different OS user
CI / contract (pull_request) Successful in 1m18s
CI / build (pull_request) Successful in 1m19s
HerdrPeerLauncher asserted, as established fact, that member panes run under
a different OS user whenever memberHerdrSocket is configured. fleetd has no
channel to see the uid at the other end of a herdr unix socket — an operator
may point memberHerdrSocket at a second herdr under the SAME user for pane
isolation, in which case members do inherit fleetd's environment and the
count this WARN told them to disregard is the real gap.

Reworded the class javadoc on hostEnvNames, the WARN in
warnUnknownMemberEnvironment, the javadoc on memberHerdrSocketConfigured(),
and warnCannotShareScrubDirectory's "unreadable by another uid" claim to say
what is actually true: fleetd cannot confirm what OS user the second herdr
runs as, so the member credential gap is UNKNOWN, not known-clean or
known-dirty. No behaviour change — the fallback paths and the honest
UNKNOWN conclusion stay the same, only the stated reason changes.

Matches the framing already used by Fleetd.reportMemberTrustModel on main.

Added unknownEnvironmentWarnStatesUncertaintyNotAnAssertedDifferentUser to
HerdrPeerLauncherAllowListWiringTest asserting the new WARN wording and that
it no longer claims a different OS user as fact.
2026-09-03 20:10:22 +07:00
Dai Ha fa97f598dd Merge #265: state the member trust model at startup (#184)
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m26s
2026-09-03 16:48:44 +07:00
Dai Ha ea9aa4fd77 fleetd #184: report member trust model
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m59s
2026-09-03 16:48:13 +07:00
Dai Ha a2b8caf6b5 #184: keep the reason SSH_AUTH_SOCK matters, and the measurement
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m23s
The correction removed a false claim (blocking the socket breaks git over
SSH) but took a true one with it: the socket is a live handle to the agent,
so a member holding it can sign with every key the agent holds. Without that,
the entry reads as if the setting does not matter, and an operator has no
reason left not to set it to allow. Fixing an overclaim must not leave an
underclaim.

Also record the measurement and the mistake behind the old claim, so the next
person does not re-argue it from scratch.
2026-09-03 16:42:11 +07:00
Dai Ha b5ddbe5757 Merge #264: sshAuthSock is not a control, and the config now says so (#184) 2026-09-03 16:41:52 +07:00
Dai Ha d223a93039 fleetd #184: correct sshAuthSock guidance
CI / contract (pull_request) Successful in 1m5s
CI / build (pull_request) Failing after 1m37s
2026-09-03 16:38:56 +07:00
Dai Ha 96d8191149 Merge #263: free reports what the spawn gate grants (#257)
CI / contract (push) Successful in 48s
CI / build (push) Successful in 2m13s
2026-09-03 16:33:20 +07:00
Dai Ha f0e7ac73d6 #103: give the operator the string that is actually in the log
The merged fix told the operator to look in journalctl for
"Fleetd.reportRequiredSecrets". That string never appears there — it is a
method name. The logger is d.l.f.Fleetd and the lines read
"startup secret NAME: set|MISSING", so the advice sent the operator looking
for text that does not exist.

Give the grep instead, and say what the report does not cover: it lists only
names a configured profile references, so a secret nothing references is never
reported.
2026-09-03 16:32:23 +07:00
Dai Ha e2fe861b4d Merge #262: the systemd unit names all three secrets it needs (#103) 2026-09-03 16:32:07 +07:00
Dai Ha 279d6f5fbd fleetd #257: free must stop subtracting leadSeatCount
CI / build (pull_request) Successful in 1m43s
CI / contract (pull_request) Successful in 1m51s
fleet_list's free row subtracted leadSeatCount, but the real spawn gate
(CompositePeerLauncher#enforceMaxLoad) only ever compares live against
maxLoad and never reads leadSeatCount. So free could report 0 while a
fleet_spawn on that exact profile still succeeded, and a lead trusting
free:0 gave up on capacity the gate would still grant.

free now always equals max(0, maxLoad - live); leadSeats stays in the
row as an informational fact, never subtracted. Documented in the
fleet_list tool description and fleetd.example.yaml.
2026-09-03 16:28:32 +07:00
Dai Ha 34480cebef Merge #261: compare-and-swap the workspace-trust seed against an external writer (#247)
CI / contract (push) Successful in 54s
CI / build (push) Successful in 1m40s
2026-09-03 16:26:37 +07:00
Dai Ha 9d0bf14c46 fleetd #103: document systemd worker secrets
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 1m31s
2026-09-03 16:23:29 +07:00
Dai Ha 5a3ab5764c fleetd #247: CAS seedTrustDialog against the operator's own live Claude Code
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 1m49s
TRUST_JSON_LOCK only serialises seedTrustDialog calls this launcher makes
inside its own JVM. It cannot reach the one writer that actually shares the
target file on a real host: the operator's own live Claude Code, whose
CLAUDE_CONFIG_DIR is routinely the very configDir a profile is given, so the
file fleetd writes on every claude-code spawn is that session's own config.
A plain read-modify-write there is a routine lost update: fleetd reads v1,
the operator's session writes v2, fleetd's ATOMIC_MOVE lands v3 built from
v1 and v2 is gone, atomically.

Add a bounded compare-and-swap: before the move, re-read the target's exact
bytes and compare with what the update was built from; on a mismatch,
rebuild from the fresh bytes and retry (up to 5 attempts). Exhausting the
retries writes nothing and logs a WARN naming the file — the member shows
the trust dialog and fails to reach an injectable state instead, which is
visible and recoverable, unlike silently overwriting the operator's live
config. Also warn every time the seed is about to target the default
~/.claude.json (configDir unset), since that is the unsafe default.

This narrows the lost-update window, it does not close it: a write landing
between the final re-read and the ATOMIC_MOVE itself is still lost.
2026-09-03 16:20:20 +07:00
Dai Ha 3bad9f5785 Merge #260: restore the already-gone-worktree teardown test (#116)
CI / contract (push) Successful in 58s
CI / build (push) Successful in 2m9s
2026-09-03 16:13:15 +07:00
Dai Ha 8308c0b68f fleetd #116: recover the already-gone-worktree teardown regression test
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 1m23s
Ports the intent of the lost CB-576 commit c393600 (worker/cb576-01a04b-17,
never merged, package dev.ltms.bridged.*) onto main's dev.ltms.fleet.*
tree. Adds FakeWorktrees.markGone (tracks which add()'d worktree paths
still "exist", mirroring GitWorktrees.hasUncommitted's Files.exists guard
for the already-gone case) and a regression test,
releaseStillStopsPaneAndNotifiesWhenWorktreeIsGone, asserting that
SessionManager.release still fires notifyReleased, stops the pane, and
falls through to remove() when the worktree is already gone.
2026-09-03 16:11:27 +07:00
Dai Ha 3b3063eb2b #252: the route scrape counted commented-out registrations as live
CI / build (push) Successful in 1m44s
CI / contract (push) Successful in 2m21s
Follow-up to #259, found by verifying the guard rather than trusting it.

The scrape read FleetApp's raw source, so a registration disabled with `//`
still matched. Measured: commenting out `app.get("/tasks/{ticket}", ...)` left
the test GREEN, while deleting the same line was caught. Only the commented-out
shape was blind, and it is the silent direction — the inventory would keep
claiming a route the server no longer serves.

Drop whole-line comments before scraping. Only lines starting with //, * or /*
are dropped, deliberately not every // on a line: that would also cut a string
literal containing // (a URL) and could silently delete a real registration
sharing the line. The remaining gap is a trailing comment beside real code; no
registration here has that shape, and the vacuity test catches a scrape that
loses registrations wholesale.

Proof: with the fix, the comment-out mutation fails naming
"Removed ... [GET /tasks/{ticket}]"; reverted, FleetApp.java confirmed clean.
mvn clean install: 0 compile errors, 1257 tests, BUILD SUCCESS.
2026-09-03 15:59:35 +07:00
Dai Ha df9086263d Merge #259: guard the REST route inventory against drift (#252) 2026-09-03 15:56:15 +07:00
Dai Ha eb568ff451 fleetd #252: guard test for the REST route inventory
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 1m27s
FleetApp's route list has never been checked against anything and has
already drifted once (GET /member-credentials shipped hours before the
#252 ticket and was missing from its list). Add
RestRouteInventoryTest, modelled on McpContractDocTest, which scrapes
FleetApp.java's app.<verb>("path") calls with a regex and compares
them against an explicit expected inventory, failing loudly with the
added/removed routes when they diverge.
2026-09-03 15:54:45 +07:00
Dai Ha ac790e4cce #258: stop two test fixtures writing the operator's real ~/.claude.json
CI / contract (push) Successful in 48s
CI / build (push) Successful in 1m29s
seedTrustDialog targets ~/.claude.json when a profile sets no configDir.
Two IDE-overlay fixtures built a worktree-shaped @TempDir, which opens the
#149 isProvisionedWorktree gate, and left configDir null — so every test run
added two project entries to the operator's real file. 116 had accumulated,
none of them still existing on disk, and 32 of those came from current code.

The #149 gate only closed the opposite case: a fixture with cwd unset falling
back to user.dir. A fixture that builds a worktree on purpose walks straight
through it.

- ideProfile/ideProfileModule now take configDir first and mandatory, so each
  fixture states where the trust seed goes.
- noFixtureSeededTheDefaultClaudeJson snapshots the temp-dir project keys in
  @BeforeAll and fails in @AfterAll on any key this class added. Differential,
  not absolute: an absolute check would fail on every host still carrying the
  historical entries, and such a check gets deleted rather than fixed.

Not done: a blanket -Duser.home redirect in surefire. EnvAllowListScrubTest
tests the credential scrub against the operator's real login chain and guards
with assumeTrue($HOME/.zshrc exists), so the redirect would silently skip two
security tests.

Proof: with the bug put back on one fixture the guard fails and names the path;
reverted and confirmed identical with diff -q. A full suite run under a fake
home now creates no .claude.json at all. mvn clean install: 0 compile errors,
1255 tests, BUILD SUCCESS.
2026-09-03 14:01:45 +07:00
Dai Ha 6b5f3f472f Merge #111: the credential probe reads the policy instead of copying it
CI / contract (push) Successful in 1m19s
CI / build (push) Successful in 1m30s
scripts/probe-member-credentials.sh carried its own NAMES array of 31
names. The live policy has 34. The probe reported 26 blocked against a
policy that blocks 29, exited 0, and printed a table that looked
complete. A verification tool that under-reports is worse than none,
because its clean output stops anyone looking.

Same defect as #114, fixed the same way: DELETE the second copy rather
than correct it. The NAMES array is gone, not updated.

The daemon now serves GET /member-credentials — names and counts, never
a value; MemberCredentialPolicyView reads no environment at all, so
there is nothing to redact by construction. The probe fetches it and
refuses with a non-zero exit when the daemon is unreachable, the policy
is absent or empty, or knownCount disagrees with the length of known[].
No local fallback: a verification tool must not quietly degrade into a
weaker check.

The startup log line and the endpoint now share that one class, so the
counting exists once. That also protects a subtlety I measured before
briefing this: blocked is NOT known - allowed. Live, known=34 and
allow=7, but only 5 of those 7 appear in known, so blocked=29 and the
naive subtraction gives 27. The view reuses creds.blockedSet(), the
existing derivation, so it keeps 29.

Worker's mutation: MemberCredentialPolicyView.of(...) forced to return
ABSENT turned 4 tests red with 0 compile errors — including
MemberCredentialsGapReportTest, which proves the startup log really
does run through this path. Reverted and confirmed with diff -q.

It also caught a bug in its own first draft: jq's // operator treats
false and 0 as missing, so `.present // empty` turned a genuine
"present": false into "unknown". Fixed by reading the fields directly.

NOT yet verified: acceptance criterion 5, the live 34/29/5 run. The
route does not exist until the daemon is redeployed onto this jar, so
that check comes next and is mine, not the worker's.

Merged clean, then built on the merged tree: 1255 tests, 0 failures,
0 compile errors.
2026-09-03 13:32:16 +07:00
Dai Ha 38dec72152 Merge #155: refuse the spawn when allow-list policy cannot be enforced
CI / contract (push) Successful in 46s
CI / build (push) Successful in 2m15s
policy=allow-list is enforced by a ZDOTDIR scrub, and a non-zsh login
shell ignores ZDOTDIR entirely, so no scrub runs. The launcher already
DETECTED this and logged a WARN — then degraded to the weaker overlay
and spawned anyway. The operator asked for the blocking control and
silently got the weaker one, which is the defect the ticket is about.

Detection existed; refusal did not. Under policy=allow-list a non-zsh
shell now throws IllegalArgumentException before any ZDOTDIR or env
work, naming the actual shell and giving three ways out. Under
policy=deny-by-default nothing changes: that overlay is applied to the
pane before any shell runs, so it does not depend on the shell.

Checked against the live config myself, because this refuses spawns and
no worker can see fleetd.yaml:

  memberHerdrSocket : NOT set  -> the shell comes from fleetd's own
                                  $SHELL, not the unset memberLoginShell
  policy            : allow-list
  fleetd's $SHELL   : zsh, proven by behaviour rather than by reading
                      the process env — the daemon log shows the ZDOTDIR
                      scrub generating a directory 147 times, most
                      recently minutes ago, and that only happens when
                      isZshShell() returned true

So the new refusal cannot fire on this host. Had memberHerdrSocket been
set, the unset memberLoginShell would have read as "<unset>", non-zsh,
and refused every spawn — worth knowing before anyone sets that key.

Worker's mutation evidence, re-stated: `if (!zsh)` -> `if (false)` turned
the refusal test RED with 0 compile errors, then reverted clean.

NOT verified: a live non-zsh member spawn. Forcing it means changing the
daemon's own environment, and the value of the test does not justify
that. The unit tests drive the real launcher.spawn entry point.

Merged clean, then built on the merged tree: 1250 tests, 0 failures,
0 compile errors.
2026-09-03 13:29:53 +07:00
Dai Ha 0e8bfb74fc Merge #176 stage 2: group subscription profiles by account, not by name
CI / contract (push) Successful in 1m10s
CI / build (push) Successful in 1m34s
Stage 1 shipped INERT on this host and every test was green. The matcher
compared effectiveCredentialId(), which fell back to the profile's own
NAME when credentialId was unset. This host runs the lead on `opus` and
members on `sonnet`; both are subscription:true with no credentialId, so
it compared "opus" against "sonnet", never matched, and charged 0 seats.

Every stage-1 test put the lead on the SAME profile name as the target,
so the fixture encoded the one shape the live config does not have.

Stage 2 returns a "<subscription>" sentinel when credentialId is unset
and subscription is true. An explicit credentialId still wins, so an
operator with two genuinely separate Claude logins can keep them apart.

Verified by me on the live config shape, not by reasoning:

  opus.effectiveCredentialId()   = <subscription>
  sonnet.effectiveCredentialId() = <subscription>
  seats charged to sonnet = 1     (was 0 before this change)

Only opus and sonnet join the sentinel group on this host; local,
local-direct, gx, xf, sol and terra are unaffected. free is clamped
with Math.max(0, ...), so the subtraction cannot report a negative.

Second, wider consequence, flagged by the worker and confirmed here:
CompositePeerLauncher.credentialIdFor feeds enforceNotQuarantined and
enforceNotCoolingOff, so quarantining one subscription profile now also
refuses spawns on the other. That is correct — one Claude subscription
hitting a usage limit really does take out every profile on it — but it
is a behavioural change beyond fleet_list's numbers.

Checked all 5 logical callers of effectiveCredentialId(); every one
wants "this account", none wants "this exact profile".

Merged clean, then built: 1250 tests, 0 failures, 0 compile errors.
An auto-merge with no conflicts is not a compiling merge, so the build
was run on the merged tree before this landed.
2026-09-03 13:24:19 +07:00
Dai Ha 51f7b0a3ca fleetd #111: probe reads the live memberCredentials policy, no hardcoded name list
CI / contract (pull_request) Successful in 1m24s
CI / build (pull_request) Successful in 2m22s
scripts/probe-member-credentials.sh carried its own hand-maintained NAMES array
(31 names, recorded 2026-08-16), so a name added later to fleetd.yaml's
memberCredentials.known was never checked and the probe still exited 0 with a
clean-looking table. Same drift shape as #114's tool catalogue.

- New dev.ltms.fleet.member.MemberCredentialPolicyView: the single place that
  turns a MemberCredentials policy into names + counts (never a value). Reused
  by Fleetd.reportMemberCredentialsGap (startup log line) and by the new
  GET /member-credentials REST endpoint (FleetApp), so the two can no longer
  drift apart the way the probe and the policy did.
- FleetApp gains one route + handler + a Supplier<MemberCredentialPolicyView>
  constructor param (legacy constructors default to ::absent, so existing call
  sites are unaffected).
- probe-member-credentials.sh now fetches its name list from
  GET /member-credentials instead of carrying one. No local fallback: an
  unreachable daemon, an empty/absent policy, or a knownCount/known[] length
  mismatch all refuse with a non-zero exit rather than silently checking zero
  names. Prints "policy contains N; this run checked N" so the two numbers are
  visibly equal.
2026-09-03 13:21:35 +07:00
Dai Ha 21c539f22e #113: derive the config guard from the record tree, both directions
CI / contract (push) Successful in 53s
CI / build (push) Successful in 1m48s
The existing guard walks KNOWN_TOP_LEVEL_KEYS and anchors its regex at
column 0, so it sees only top-level keys. Every nested key was outside
its scope and nothing said so, which is the shape #113 collects: a
checker narrower than it looks, whose green run stops anyone looking.

Two derived guards replace the assumption:

  everyNestedConfigKeyIsDocumentedInTheExample
      walks FleetConfig's record components (17 records, 83 distinct
      key names) and requires each to be documented in the example.

  everyLiveKeyInTheExampleBindsToARecordComponent
      resolves every live key path in the example against the record
      tree, so a documented key that binds to nothing fails here
      instead of being silently ignored in production.

Neither carries a list, so a key added to any nested record is covered
the moment it compiles (criterion 2).

Both mutations run through the real caller, not the helper (criterion 1,
which asks for exactly that):

  removed every mention of paneProbeIntervalSeconds from the example
      -> FAILS, naming health.paneProbeIntervalSeconds
  added a live bind.totallyMadeUpKnob to the example
      -> FAILS, naming bind.totallyMadeUpKnob

0 compile errors in both; both reverted and confirmed with diff -q.
The first attempt at mutation 1 removed only the `key:` line and the
run stayed green — correctly, because the key was still documented in
prose. An incomplete mutation proves nothing, so it was redone.

Denominators (criterion 3): both guards print how many keys they
checked, and the floor for "did the walk descend?" is derived from
KNOWN_TOP_LEVEL_KEYS.size() rather than being a literal.

Scope is stated in the javadoc rather than implied: the guards do not
check a key sits at the right path, do not parse commented prose for
the reverse direction, and do not prove a parsed key is read by
anything. paneProbeIntervalSeconds is parsed and read by nothing, and
these guards pass it -- the example already says so in its own text.

everyOptionalKnobDocumentedInTheExampleBinds keeps its hand-written
list but is re-documented as a value-binding spot check, explicitly
not a coverage guard; coverage now comes from the two derived tests.

broker.uri is documented only in the example's prose convention
(`#  uri  -> ...`), never as a copy-pasteable `uri:` key, because
writing it out invites pasting a password into a file -- the thing
uriEnv exists to avoid. The matcher accepts that convention rather
than pushing the file toward doing it.

Full build: 1236 tests, 0 failures, 0 compile errors.
2026-09-03 13:18:50 +07:00
Dai Ha bd2774b5f1 fleetd #155: refuse a member spawn under memberCredentials.policy=allow-list on a non-zsh shell
CI / contract (pull_request) Successful in 1m1s
CI / build (pull_request) Successful in 1m56s
The ZDOTDIR scrub that enforces policy=allow-list only runs on zsh. The daemon
already detected a non-zsh login shell (isZshShell/warnNonZsh, from #213), but
degraded to the weaker CB-596 overlay and spawned anyway — the exact "control
silently does nothing" defect this ticket is about. Now a non-zsh shell under
policy=allow-list refuses the spawn (IllegalArgumentException, naming the
shell), surfaced by FleetMcp.spawn's existing catch(IllegalArgumentException).
policy=deny-by-default is unaffected in substance (its overlay never depended
on the shell) but now also logs a one-time WARN naming the shell, since the
stronger allow-list control is unavailable there.

The worktreeRoot/worktreeGroup-missing degrade path under memberHerdrSocket
is untouched — that gap is fleetd #213's scope, not this one.
2026-09-03 13:13:23 +07:00
Dai Ha c50f5b2d61 fleetd #176 stage 2: make effectiveCredentialId() subscription-aware
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m44s
Stage 1's lead-seat matcher (leadSeatLookup) was correct but inert on
the live host: the lead runs on profile 'opus', members on 'sonnet',
both subscription:true with no explicit credentialId. Because
effectiveCredentialId() fell back to the profile's own name, opus and
sonnet never matched even though they share one Claude login, so the
matcher charged zero seats.

FleetConfig.Profile.effectiveCredentialId() now falls back to a shared
sentinel (SUBSCRIPTION_CREDENTIAL_ID = "<subscription>") instead of the
profile name when subscription:true and credentialId is unset. An
explicit credentialId still wins, so two separate Claude logins on one
host can still be kept apart.

This is also BackendQuarantine's and BackendOutagePolicy's grouping
key and CompositePeerLauncher's spawn-time enforcement key, so the fix
also links quarantine/cool-off across subscription profiles sharing an
account -- intentional: one usage limit really does take out every
profile on that login, mirroring credentialId: openai-shared already
doing this for off-subscription profiles. Every caller was reviewed;
none wants "this exact profile" over "this account".

Tests added:
- FleetdLeadSeatLookupTest: the live shape itself (lead on a
  DIFFERENT subscription profile than the target, same account,
  neither sets credentialId) -- the case stage 1's suite never covered
- FleetMcpTest: quarantining one subscription profile's shared
  account zeroes free on another sharing it, via the same
  effectiveCredentialId()-driven wiring Fleetd.main uses

Mutation-tested: reverting the subscription branch to the old
fall-back-to-profile-name behavior sends both new tests RED with 0
compile errors; reverting the mutation restores byte-identical
(diff -q) source and green tests.

fleetd.example.yaml's fleetd #176 notes are rewritten for the sentinel
semantics and when to override it with an explicit credentialId.
2026-09-03 13:10:10 +07:00
Dai Ha 01a840cc14 #248 follow-up: drive the real backendErrorSink, not a copy of it
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m29s
BackendOutageFlowTest held a ~30-line hand-copy of the lambda in
Fleetd.main, under a comment promising it mirrored production "EXACTLY".
That promise was the defect. The test proved the copy, so any change to
the real sink left the flow test green.

#248 made Fleetd.backendErrorSink(...) public for exactly this reason.
The test now calls it.

Measured, same mutation in the real sink (an early return after
sessions.onBackendError, dropping the cool-off and the lead nudge):

  old test (hand-copy):  Tests run: 5, Failures: 0  -- blind
  new test (real sink):  Tests run: 5, Failures: 4  -- catches it

0 compile errors in both runs, so both are real results. Production
reverted and confirmed with diff -q.

Full build: 1234 tests, 0 failures, 0 compile errors.
2026-09-03 13:06:11 +07:00
Dai Ha c4deef08be fleetd #249: withhold agentSessionId when the cwd is not a provisioned worktree
CI / contract (push) Successful in 1m22s
CI / build (push) Successful in 1m22s
A member spawned without a worktree inherits the lead's cwd, which holds
many old opencode session rows. sessionIdForDirectory picks the most
recently updated row for that directory, so a brand-new member - which
has not written its own row yet - resolves to somebody else's session.
Measured: a row three days old, from a different profile.

The damage was at the tool surface. fleet_list told the lead that
agentSessionId is the id to pass as resumeSessionId, so acting on it
would resume a stranger's conversation, with foreign context, and
nothing to distinguish that from a correct resume.

Fixed by refusing to answer rather than by making the heuristic smarter.
#234 already established the heuristic cannot be made reliable at that
layer, and its javadoc records why, so the SQL is untouched.

agentSessionId() now returns null for a non-provisioned cwd, and
spawn() refuses a resumeSessionId request for one outright, before
anything starts. isProvisionedWorktree moved to HerdrPeerLauncher so
both adapters share it. fleet_list and fleet_spawn descriptions no
longer describe the id as always safe to resume.

Verified rather than taken on trust:
- the refusal reaches the lead as a readable message, not a stack trace
  - FleetMcp.spawn already catches IllegalArgumentException and returns
  error(e.getMessage()).
- the message tells the lead to pass fleet_spawn{worktree:<slug>}, which
  is valid: worktree is typed string, 'true' or a ticket slug.
- 13 existing tests moved off a placeholder "/work/dir" onto a real
  provisioned-worktree fixture. They cover #175/#234 model-mismatch
  machinery and would otherwise have tripped the new gate incidentally.

Worker's mutation evidence, both reverted and diff-confirmed:
- gate at OpenCodeLauncher:812 -> if(false): RED at
  OpenCodeLauncherTest:448, expected <null> but was <ses_someone_elses>.
- resume refusal at OpenCodeLauncher:683 -> 'false &&': RED at
  OpenCodeLauncherTest:379, expected IllegalArgumentException.
Both 0 compile errors.

Closes #249. PR #253.
2026-09-03 12:55:29 +07:00
Dai Ha c796eac09c fleetd #176: subtract the lead's own subscription seat from free
CI / contract (pull_request) Successful in 1m11s
CI / build (pull_request) Successful in 1m16s
maxLoad counted panes, never subscription seats: a subscription:true
profile's lead is itself a live claude session on that same account,
so free overstated capacity by the lead's own seat (measured free:1
with a real ceiling of 0, and free:3 on an idle fleet with a real
ceiling of 2).

Add FleetMcp.LeadSeatSource (same shape as QuarantineSource/
OutageSource) and Fleetd.leadSeatLookup, which derives the seat count
from fleet.leaders.<name>.profile matched against the target profile
by effectiveCredentialId() - no hardcoded "-1", and no new config key:
profile: already exists for this exact "which account does this lead
share" question. maxLoad itself is left untouched; only free (and a
new, additive-only leadSeats field) changes.

Exhaustion quarantine (cause 2 in the ticket) already forced free to 0
via the same BackendQuarantine capacityView already reads - confirmed
by reading the exhaustionSink wiring, no code change needed there.
2026-09-03 12:55:08 +07:00
Dai Ha e897e5257b fleetd #114: delete the drifted tool catalogue, keep the flows, guard the names
CI / contract (push) Successful in 51s
CI / build (push) Successful in 1m42s
docs/MCP-Contract.md was written 2026-07-14, before any MCP code existed,
and never caught up. CLAUDE.md points every session in the fleet at it.

Audited against the code today. The drift was not confined to the tool
table the ticket reported:

  section 3  still described the OLD identity rule - "any connection that
             does not map to a known worker is treated as a primary".
             That was a real privilege bug, fixed since by the ancestry
             walk in #161. The page still taught it.
  section 4  names port 8080 (the mount is 8765) and says the pom does
             not yet carry an MCP dependency.
  section 5  named fleet_read and fleet_cancel, which do not exist, and
             omitted fleet_poll, fleet_ack, fleet_profiles, fleet_whoami
             and fleet_list, which do.
  section 8  says turn_id where the code says turnId, and has no row for
             the exhausted outcome CB-578 added.
  sections
  9, 10, 11  pre-build planning: "new work" columns, open decisions long
             since decided, CB-1xx placeholders.

Every one of those is the same defect: a hand-maintained second copy of
something the code already states. So the copy is deleted rather than
corrected - correcting it just restarts the clock.

What survives is the flows and the status gating, because a flow is a
shape rather than a name, and shapes are what this page was ever good
for. They are rewritten with the names checked against the code, and
extended with what has been learned since: the ~60s cap on a blocking
send, the ~55s ask window, and the three ways the turn-done fallback
loses a report (clipped, echoed brief, slow member).

389 lines -> 188.

The names that remain are guarded. McpContractDocTest fails if the page
names a fleet_* tool FleetMcp does not register, and - because an empty
set is a subset of everything - a second test pins that both sides
actually found names, so the check cannot pass by checking nothing. A
third pins the "this is not the tool reference" sentence, which is the
fix itself: without it someone helpfully re-adds a tool table.

Mutation-tested both ways, 0 compile errors each: adding `fleet_read` to
the doc fails theDocNamesNoToolThatDoesNotExist ("names [fleet_read] ...
Checked 6 name(s)"); removing the disclaimer fails
theDocStillDisclaimsBeingTheToolReference.

All 5 mermaid diagrams render under mermaid-cli.

CLAUDE.md's pointer said "section 6 only" and now names the guard
instead. It is in the project addendum, so the canonical block is
untouched - verified still byte-identical with the wiki template.

REST is split out to #252: 14 routes, documented nowhere, and it IS a
supported operator surface - one of them drains on read.

1232 tests, 0 failures.
2026-09-03 12:52:12 +07:00
Dai Ha 2afa3652bb fleetd #249: withhold agentSessionId for a non-provisioned opencode cwd
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 1m52s
OpenCodeSessionDiscovery.sessionIdForDirectory keys on the worker's cwd, which
is reliable only when fleetd provisioned a unique git worktree for that
member. Without one (the default no-worktree spawn), the cwd is shared with
other sessions, and "most recently updated row for this directory" can pick a
stranger's session — fleet_list would then hand a lead an agentSessionId that
resumes someone else's conversation.

Move isProvisionedWorktree from ClaudeCodeLauncher to the shared
HerdrPeerLauncher base (both adapters need it now). OpenCodeLauncher.spawn now
refuses a resumeSessionId spawn outright when the target cwd is not a
provisioned worktree (fleetd can never verify or re-report that identity), and
SessionAwareHandle.agentSessionId() withholds the id — returns null rather
than guessing — for any member spawned without one, resumed or not. Corrected
fleet_list/fleet_spawn's tool descriptions, which previously implied
agentSessionId is always a safe resume handle.
2026-09-03 12:51:56 +07:00
Dai Ha 80092ff359 fleetd #247: stop writing a trust key Claude Code strips on every save
CI / contract (push) Successful in 1m26s
CI / build (push) Successful in 1m28s
seedTrustDialog wrote two keys into the shared .claude.json:
hasTrustDialogAccepted and hasCompletedProjectOnboarding. Only the first
one survives.

Measured live on 2026-09-03, minutes after a spawn seeded the file:

  hasTrustDialogAccepted:       28 of 28 project entries
  hasCompletedProjectOnboarding: 0 of 28 project entries

Our entry was written by the running jar and the key was already gone,
so it was written and then removed. It is absent from the 27 entries
Claude Code wrote for itself too, which says Claude Code normalises the
whole file when it saves and drops that key every time.

That reframes #247. I filed it as a race - a save landing between our
read and our ATOMIC_MOVE. It is not a race. The other writer removes
this key as its steady-state behaviour, with no window involved. So the
compare-and-swap retry proposed there would not have helped: it would
re-add a key that gets stripped again on the next save.

The seed's whole job is to stop the workspace-trust dialog blocking a
member (#149). The live probe reached idle with hasTrustDialogAccepted
alone, so the second key was never doing that job. Writing it only added
a contested key to a file two processes share, and made the next reader
think it mattered.

The atomic write and the lock stay. Both are still correct, both are
cheap, and hasTrustDialogAccepted is genuinely shared state.

The new assertion is assertFalse, not a deletion. Removing the old
assertion would leave nothing to stop someone re-adding the key later as
a plausible-looking completeness fix. Mutation-tested: restoring the
production line fails
seedTrustDialogWritesOnlyTheTrustFlagAndNotTheOnboardingKey:2198 with 0
compile errors.

1229 tests, 0 failures.
2026-09-03 12:41:20 +07:00
Dai Ha 9d37f3aa29 fleetd #201: the coverage line must name the key its caller actually means
CI / contract (push) Successful in 43s
CI / build (push) Successful in 1m56s
Found by reading a real boot log after the redeploy, not by a test.

coverage() is shared by two call sites — CB-578's exhaustedPattern line
and Unit 5's errorPattern line — but its 'off' branch hard-coded the
word exhaustedPattern. So this daemon printed:

  backend-exhausted classification (CB-578 stage A): partial
      (configured: [sol, terra]; not configured: [...])
  backend-error classification (fleetd #201 Unit 5): off
      (no profile has an exhaustedPattern configured; profiles: [...])

Two lines, one directly under the other, disagreeing about whether any
profile has an exhaustedPattern. Both were individually defensible and
together they were nonsense. Worse, the message sends an operator to
set the wrong key: the thing that is missing is errorPattern.

coverage now takes the key name. I changed the signature rather than
adding an overload, so the compiler found all three existing callers
instead of leaving them silently on the old path.

Every earlier coverage test passed the exhaustion case only, which is
why none of them could see this. The new test pins the errorPattern
case. Reverting the fix turns it red with 0 compile errors.

1229 tests, 0 failures, BUILD SUCCESS.

This is the second defect in two hours found only by reading the live
startup log — see #115, where the noise of a false warning had been
hiding a correct line saying a whole feature was off.
2026-09-03 12:29:27 +07:00
Dai Ha eaf89abaf6 fleetd #248: make Fleetd's CompletionResolver wiring provable
CI / contract (push) Successful in 42s
CI / build (push) Successful in 1m40s
Before this, dropping either #241's worktree lookup or Unit 5's
backend-error pair at Fleetd.main's new CompletionResolver(...) call
left all 1216 tests green with 0 compile errors. Every existing test
built its own CompletionResolver, so they proved the class and never
the wiring. BackendOutageFlowTest was the sharpest case: it copies
main's sink lambda line-for-line, so it proves the copy and cannot
notice the original being deleted.

The three inline arguments are now package-private static factories on
Fleetd, following the deliverableTo pattern the file already had, each
with its own behaviour test. backendErrorSink is public so a
cross-package test can drive the real production object rather than a
hand-mirrored copy.

The test that was actually missing is a source-text assertion. That is
the honest fallback for a composition root with no seam, and it is
labelled [SOURCE TEXT] in every test name and message so it cannot be
misread as a behaviour check. It is not vacuous: two tests pin that the
variables are assigned from the factories, and two pin that those
variables reach the call site, so renaming a variable while assigning
an inert value does not slip through.

Known cost, accepted: the assertions match exact source substrings, so
reformatting that statement will break them. That is the price of
covering a main method, and a spurious failure here is loud and
obvious, which is the right direction to fail.

Verified by the lead, both mutations re-run against the merged code —
see the merge check.

PR #251
2026-09-03 12:22:32 +07:00
Dai Ha d895f02bc1 fleetd #248: prove main() wires CompletionResolver's arguments, not just the class
CI / contract (pull_request) Successful in 48s
CI / build (pull_request) Failing after 1m40s
Fleetd.main built three of CompletionResolver's 8 constructor arguments inline
(a worktree/branch lookup lambda, and the backend-error pattern lookup + sink
locals). Dropping any of them at the call site compiled clean and left every
existing test green, because every existing test constructs its own
CompletionResolver and only ever proves the class, never main's wiring.

Extract each into a static factory on Fleetd (worktreeBranchLookup,
backendErrorPatternLookup, backendErrorSink — the same static-factory pattern
Fleetd.deliverableTo already uses), test each factory's own behaviour, and add
a source-text assertion (FleetdCompletionResolverWiringTest) proving main's
CompletionResolver call still passes all three. backendErrorSink is public so
BackendOutageFlowTest can exercise the real production sink directly instead
of the hand-mirrored copy its own class doc used to describe.

No production behaviour changes — mechanical extraction only.
2026-09-03 12:20:28 +07:00
Dai Ha 43206cac2f fleetd #148 point 2: drop .envrc from the default parity overlay
CI / contract (push) Successful in 1m9s
CI / build (push) Successful in 1m18s
The default is now [.env], not [.env, .envrc]. .env is data, so copying
it into a worker worktree can only move values. .envrc is executable
shell that direnv runs on every cd, so copying it moves behaviour. Those
are different risks and should not share a default.

The knob is unchanged. An operator who wants .envrc copied writes
parityOverlay: ['.env', '.envrc'] and owns that choice; a new test pins
that escape hatch, because without it this would be a removal rather
than a re-default.

Decision recorded on the ticket, with the evidence it asked for first:
this checkout has no .env and no .envrc, and direnv is not on PATH, so
there was no live exposure. Point 1 (extend the credential scrub to
direnv) is declined and the reason is on the ticket — the scrub is a
one-shot .zlogin and a direnv hook runs on every cd, so no amount of
work on the scrub can cover it. Not copying the executable file is the
smaller change and removes the need.

The worker also fixed WorktreeSessionManagerTest, which hardcoded the
same default at another layer and broke the build. Outside its named
scope, correctly flagged rather than done silently.

Verified by the lead: 1216 tests, 0 failures, 0 compile errors.

PR #250
2026-09-03 12:03:21 +07:00
Dai Ha 8bba3a8184 fleetd #148 (point 2): drop .envrc from the default parityOverlay
CI / contract (pull_request) Successful in 1m20s
CI / build (pull_request) Successful in 1m32s
.env is data; .envrc is executable shell that direnv runs on every cd, so
copying it into a worker moves behaviour, not just values. The default
parityOverlay is now [.env] only. The knob is unchanged: an operator who
wants .envrc copied can still write parityOverlay: [.env, .envrc]
explicitly.

Updates FleetConfig's default and javadoc, fleetd.example.yaml's two
mentions of the default, and the FleetConfigTest coverage: renamed the
default test, added parityOverlayExplicitEnvrcOptInStillWorks to prove
the .envrc opt-in escape hatch still works, and fixed
WorktreeSessionManagerTest#worktreeAcquireRunsParityOverlayWithProfileDefaults
which also hardcoded the old default.
2026-09-03 12:00:13 +07:00
Dai Ha 5cf3ca9a89 fleetd #241: never hand the lead back its own brief as the member's report
CI / build (push) Successful in 1m15s
CI / contract (push) Successful in 1m27s
The completion fallback scrapes a member's pane when a turn ends with no
fleet_reply. If the pane still shows the brief the lead injected, the
scrape returned that brief, and the lead read its own words as the
member's answer. A silent member looked like a member that had reported.

echoesInjectedBrief now recognises that case and refuses it. Round 1 used
plain containment in both directions, which destroyed real reports: a
genuine report that quotes the brief contains it. Round 2 keeps the safe
direction unbounded (the brief contains the scrape) and bounds the other
one at MAX_ECHO_EXCESS_CHARS, so a scrape only counts as an echo when it
adds almost nothing to the brief.

Merge note — the Fleetd.java conflict:

This call site was changed by both #201/#227 Unit 5 (backendErrorPatterns
+ backendErrorSink) and by this ticket (the worktree/branch lookup). I
resolved it onto the full 8-argument constructor so neither feature is
dropped; nowNanos has to be passed explicitly to reach that overload.

Verified by the lead: 1215 tests, 0 failures, 0 compile errors.

I also measured whether the resolution itself is protected, and it is
NOT. Both mutations at this call site stay green:
  - drop the worktree lookup (pass _ -> null): 1215 tests, 0 failures
  - drop Unit 5's patterns/sink (legacy()/none()): 1215 tests, 0 failures
Nothing in the suite covers Fleetd's composition root, so either feature
could be silently unwired here and the build would still be clean. The
tests prove the seams, not the caller. Filed separately rather than
fixed in a merge commit.

PR #245, branch worker/cb241-fallback-echo-1175e9-11
2026-09-03 11:54:06 +07:00
Dai Ha ac474981e4 fleetd #201/#227 Unit 5: wire the backend-error cool-off into config, placement and the MCP surface
A profile's credential that throws two distinct backend errors inside 60
seconds now cools off for 60 seconds. Automatic placement skips it,
an explicit fleet_spawn naming it is refused before the adapter is
called, and fleet_list/fleet_profiles report it as coolingOffForSeconds
next to the separate CB-578 quarantinedForSeconds.

Verified by the lead: see the merge check below. The worker ran 7
mutations, all killed with 0 compile errors; M7 was NOT killed on the
first pass (the assertion only checked .contains("quarantined"), which
is true of both the correct message and the mutated fallback), and the
worker strengthened it to assertEquals on the exact literal and kept
that change. That is the right call and it is reported honestly.

Two deviations, both justified in the PR:
- FixedPlacementPolicy needed the same coolingOff filter because it
  filters candidates inline instead of using PlacementPolicyUtil.
- BackendOutageFlowTest sits in dev.ltms.fleet.inject because
  CompletionResolver.InFlight is package-private there.

The startup coverage log line is a code-reading claim, not a captured
line from a live daemon. The worker said so rather than overclaiming.

PR #246, branch worker/cb201-unit5-wiring-6c12e6-8
2026-09-03 11:48:42 +07:00
Dai Ha ba04b2359b fleetd #201/#227 unit 5: wire errorPattern, cool-off spawn gate, and fleet views
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 1m52s
Wires the already-merged units into production:
- Per-profile errorPattern config (beside exhaustedPattern), compiled once at
  startup; falls back to the legacy (?i)\bAPI Error\s*: pattern when unset.
  Startup logs configured-vs-legacy coverage, same as exhaustedPattern.
- One production BackendErrorSink in Fleetd.java: mark backend_error on the
  session, resolve the profile's credential fail-loud (never
  Optional.ifPresent), record it in BackendOutagePolicy, and push a lead
  nudge on a new incident.
- CompositePeerLauncher's explicit and automatic spawn paths both refuse a
  cooling-off credential; exhaustion quarantine wins when both are active.
  PlacementContext gets a separate coolingOff set so refusal text says
  "cooling off", never "exhausted".
- fleet_list/fleet_profiles report coolingOffForSeconds as an independent
  fact from quarantinedForSeconds; both can appear together.
- fleetd.example.yaml documents errorPattern and the 2/60/60 cool-off policy;
  CLAUDE.md tells leads how to read the two independent outage states.

Also: FixedPlacementPolicy.java, not in the original file list, needed the
same coolingOff filtering as PlacementPolicyUtil (it does its own inline
candidate filtering rather than delegating).

1190 tests, 0 failures (up from the 1163 baseline); BUILD SUCCESS.
2026-09-03 11:45:11 +07:00
Dai Ha 2e5b63f6f6 fleetd #149: seed the workspace-trust entry before a claude-code spawn
CI / contract (push) Successful in 1m13s
CI / build (push) Successful in 1m29s
Claude Code asks 'is this a project you trust?' the first time it starts in a directory it
has not seen. It is interactive with no timeout, and every member spawned with worktree:true
lands in a brand-new directory. The member never reaches its first turn and never replies,
while herdr reports blocked/interactive_ready — which reads as healthy.

ClaudeCodeLauncher now seeds projects.<cwd>.hasTrustDialogAccepted in the profile's
.claude.json before the process starts. This is not a new grant: the operator already
trusted the repo by configuring the profile against it, and a worktree is a checkout of it.

The write is gated on isProvisionedWorktree(cwd) — a .git that is a regular gitdir-pointer
file, never a real checkout. That gate exists because an earlier revision of this change,
run under mutation testing, wrote to the operator's real ~/.claude.json and truncated it
from 72KB to 919 bytes. Tests using a null configDir fall back to the real user.home, so
an ungated seed reaches real files.

The write is atomic (sibling temp file + ATOMIC_MOVE, never truncate-in-place) and the
whole read-modify-write is under a lock, because .claude.json is large, live, and rewritten
by Claude Code itself while fleetd runs. Two parallel spawns are normal here.

Verified by the lead: 1173 tests, 0 failures. A truncating write turns the torn-read test
red; removing the lock turns concurrentSeedsForDifferentCwdsBothSurvive red. Both with 0
compile errors. copyPosixPermissionsIfPresent is NOT covered by a test — its mutation stays
green — but createTempFile is 0600 on POSIX by default, so the not-world-readable property
holds without it; the line only preserves a non-default mode.
2026-09-03 11:44:03 +07:00
Dai Ha 3437d6313d fleetd #241: bound the echo match so a real report is never swallowed
CI / contract (pull_request) Successful in 1m9s
CI / build (pull_request) Successful in 1m25s
Round 1 used plain bidirectional containment. The direction that catches the real bug --
the pane holds the brief plus a status bar, so the scrape contains the brief -- also fires
when a member restates the whole brief and then writes a genuine report under it. That
threw the report away and told the lead nothing was produced, which is worse than the bug
being fixed: it destroys a delivery instead of merely obscuring one.

The safe direction (the scrape is a fragment of the brief) stays unbounded, because a
fragment of the brief is by definition not a report. The dangerous direction now requires
the scrape to add at most MAX_ECHO_EXCESS_CHARS beyond the brief, which is the amount of
TUI chrome a real echo carries.

Work by the cb241 worker, committed by the lead: its backend stopped answering after the
fix was written, so two turns ended with no commit and no reply. Verified by the lead:
1169 tests, 0 failures; removing the bound turns pinsTheMaximumTuiChromeExcess and
completionFallbackKeepsARealReportThatRestatesTheWholeBrief red with 0 compile errors.
2026-09-03 11:39:11 +07:00
Dai Ha 743377d6cd fleetd #149 review round 2: make the trust-dialog seed atomic and lock-protected
CI / build (pull_request) Successful in 1m21s
CI / contract (pull_request) Successful in 1m25s
Files.writeString truncates the target in place before writing, so there
was a window where .claude.json could be observed empty or half-written
- exactly the shape of the incident this ticket already hit once, but
reachable in production too: a crash/kill mid-write, or two concurrent
claude-code spawns (normal here - several run in parallel routinely)
racing a naive read-modify-write and silently discarding one spawn's
entry.

Two independent fixes, each with its own dedicated test proving it (not
the other):

- ClaudeCodeLauncher.writeAtomically: serialise to a sibling temp file in
  the same directory, then Files.move with ATOMIC_MOVE + REPLACE_EXISTING,
  preserving the target's existing POSIX permissions (.claude.json ships
  0600). A reader now only ever observes the fully-old or fully-new file,
  never a torn one. Package-visible so a test can drive it directly.
- TRUST_JSON_LOCK: a process-wide lock around seedTrustDialog's whole
  read-modify-write, so two concurrent spawns for different cwds both
  keep their entry instead of the second write discarding the first.
  Sufficient because every spawn on this daemon runs in one JVM; it does
  NOT protect against a second daemon process or the operator's own live
  Claude Code writing at the same instant - writeAtomically covers that
  case instead.

Both fail soft, same as before: any I/O failure here must never block a
spawn.

Four new tests: a large (30-project) existing file survives without
collapsing (asserted on the restored key set, not just that the result
parses); two concurrent spawns for different cwds both keep their entry
(CountDownLatch-synchronised, not a sleep); existing 0600 permissions
survive the write; and a direct test of writeAtomically with a busy-poll
reader thread proving a concurrent reader never observes a torn file.

See PR body for the full mutation-testing table, including an
honest note on which of these tests the atomicity mutation actually
caught (not the one implied by the numbering in review) and why.
2026-09-03 11:38:10 +07:00
Dai Ha 5952d559c7 fleetd #134 point 3: tell the lead which files a worktree neutralizes
CI / contract (push) Successful in 57s
CI / build (push) Failing after 1m26s
The daemon log and the worktree git config are both new in 205ad82, but neither helps a
lead who is writing a brief. This is the line that does: never brief a worker to edit
.mcp.json, opencode.json or .autoenv in its worktree, because the edit cannot be
committed and nothing will say so.

Goes in the project addendum, not the canonical block, so the byte-sync with
wiki/7-Use-Cases.md is unaffected — re-checked and still true.
2026-09-03 11:32:03 +07:00
Dai Ha 205ad823b0 fleetd #134: make tool-surface neutralization visible to the daemon and the worker
CI / contract (push) Successful in 1m18s
CI / build (push) Successful in 1m34s
isolateToolSurface replaces .mcp.json, opencode.json and .autoenv with stubs in every
provisioned worktree and marks them --skip-worktree. That neutralisation is correct and
is unchanged here — the committed files would mount the primary's credentials.

The problem was that it was invisible. A worker told to edit opencode.json read a 3-byte
stub and reported, truthfully and wrongly, that the mount key did not exist. A missing
file would have prompted a question; a plausible stub did not.

Two changes, both visibility only. The daemon now logs one info summary per provisioning
with the denominator, the files neutralised, and the consequence. And the list is recorded
in worktree-scoped git config (fleet.neutralizedConfig / fleet.neutralizedConfigNote) so a
worker can discover it from inside its own worktree with
'git config --worktree --get-all fleet.neutralizedConfig'.

Worktree-scoped config was chosen over a file in the working tree because it lives in
.git/worktrees/<nonce>/config.worktree and so can never appear in git status, and because
configureEnvironmentCredentialHelper already uses the same mechanism in the same add() call.

Verified by the lead: baseline 1172 tests, 0 failures. Reverting the summary to log.debug
goes red (2 tests), and recording into --local rather than --worktree — which would leak
the record into the shared repo config — goes red too. Both with 0 compile errors.
2026-09-03 11:31:24 +07:00
Dai Ha d654ccb818 fleetd #134: make tool-surface neutralization visible to the daemon and the worker
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m35s
isolateToolSurface replaced .mcp.json/opencode.json/.autoenv with neutral stubs and
marked them --skip-worktree, but said nothing anywhere. A real worker read a 3-byte
{} stub for opencode.json, where the repo's real file is 30+ lines, and truthfully
(but wrongly) reported a mount key did not exist.

Two readers, two fixes:
- the daemon operator gets one info log per provisioning, naming the denominator,
  what was neutralized, and why anything was not (same shape as overlayParity's
  fix in #148 point 3).
- the worker gets the same fact recorded in worktree-scoped git config
  (fleet.neutralizedConfig / fleet.neutralizedConfigNote), discoverable with
  `git config --worktree --get-all fleet.neutralizedConfig` from inside its own
  worktree, without asking the lead. Not a working-tree file: this repo already
  uses worktree-scoped config for the credential helper and the SSH->HTTPS
  rewrite, and it lives under .git/worktrees/<nonce>/ so it can never appear in
  `git status` for the worker to trip on or commit.

The neutralization itself (stub content, --skip-worktree marking) is unchanged.
2026-09-03 11:24:41 +07:00
Dai Ha a89dcc9b7e fleetd #149: seed the workspace-trust entry before a claude-code spawn
CI / contract (pull_request) Successful in 51s
CI / build (pull_request) Successful in 1m43s
A claude-code member spawned into a fresh worktree hits an interactive,
un-timed workspace-trust prompt on its first start in a directory it has
never seen. It never reaches its first turn and never mounts the bridge.

Fix: ClaudeCodeLauncher.seedTrustDialog writes
projects.<cwd>.hasTrustDialogAccepted / hasCompletedProjectOnboarding into
the profile's configDir/.claude.json (or ~/.claude.json when configDir is
unset) BEFORE the herdr spawn call, additively (existing keys/projects are
preserved). Gated to isProvisionedWorktree(cwd) - a .git that is a regular
gitdir-pointer file, never a real checkout's .git directory - the same
signal writeIdeOverlay already used, now shared between both.

That gate is a fix for a real incident hit while building this: an
earlier ungated version ran against this file's own pre-existing tests
(configDir=null, no cwd -> falls back to the real user.dir and
~/.claude.json) and corrupted the operator's actual ~/.claude.json down
to a single entry during a mutation-testing run. See PR body for the
full incident report.

FakeHerdr gained onAgentStart(Runnable) so a test can assert the seed
is on disk at the exact instant herdr's agent.start call is reached -
i.e. strictly before the peer process itself would start.
2026-09-03 11:23:06 +07:00
Dai Ha 0d7b4fb026 fleetd #134 + #148 point 3: make the parity overlay say what it did
CI / contract (push) Successful in 51s
CI / build (push) Successful in 1m35s
Both defects lived in one method. overlayParity logged every step at debug, so at the
default level the copy was silent and nobody could tell which overlay files a member
actually got. It also marked a copied tracked file --skip-worktree and said nothing, so a
worker editing that file later found git ignoring the change with no error anywhere.

The summary now reports the denominator, not a bare count: 'copied 1 of 2 candidates:
.env (.envrc absent)'. A bare 'copied 1' is the same under-reporting shape as #113.
Neutralised files are named with the consequence in the message itself.

No marker file is written into the worktree: acceptance criterion 1 requires the worktree
to hold exactly the configured overlay set, so a marker would violate the fix it documents.

The copy and mark logic is unchanged — only logging is new.

Verified by the lead: baseline 1168 tests, 0 failures. Reverting either log.info to
log.debug goes red (2 reds and 1 red, 0 compile errors each), which is the regression that
matters since the whole fix is the log level.
2026-09-03 11:14:21 +07:00
Dai Ha 321d8dcbb5 fleetd #241: suppress echoed fallback briefs
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 1m20s
2026-09-03 11:10:48 +07:00
Dai Ha ef8c97871e fleetd #134/#148 point 3: make overlayParity's copy and skip-worktree visible
CI / contract (pull_request) Successful in 1m15s
CI / build (pull_request) Successful in 1m53s
overlayParity logged everything at debug, so at the default level nobody could
tell which overlay files a spawn actually received (#148 pt 3), and a tracked
file marked --skip-worktree gave no warning that it can no longer be edited
from that worktree (#134).

Report the outcome at info: a per-spawn summary naming the denominator (every
configured candidate), what was copied, and why anything was not — plus a
separate line naming every file marked --skip-worktree, stating plainly that
it cannot be committed from this worktree. No worktree-local marker file: the
worktree must hold exactly the configured overlay set and nothing else, so an
extra file would violate that invariant.
2026-09-03 11:09:50 +07:00
Dai Ha 26bafe824b fleetd #201/#227 unit 1: classify a backend error at the scrape
CI / contract (push) Successful in 43s
CI / build (push) Successful in 1m54s
CompletionResolver already reads the pane on every turn, so the classifier lives there
rather than in a new watcher. A matched pattern resolves the waiter as a failure and
fires BackendErrorSink, always inside the resolveFailure win-gate so exactly one thread
reports one incident.

The too-fast path now takes a fresh scrape instead of reusing the pre-turn text, so a
backend that dies immediately is still classified. BackendErrorSink is a single-method
functional interface by design — see #234 for what a default overload does to a lambda.
2026-09-03 10:53:51 +07:00
Dai Ha 838a701109 fleetd #234: carry the profile hint to the exhaustion sink
An exhausted opencode member could not always be mapped back to a credential, because
the launcher knew the profile and the sink did not. The sink now takes a profile hint.

The interface is inverted on purpose: the three-argument method is the single abstract
method and the two-argument one is the default. A lambda can only implement the abstract
method, so every lambda is now forced to carry the profile. The first round of this fix
added the third argument as a default overload, and the production forwarder in Fleetd
was a two-argument lambda — so the fix compiled, passed its tests, and never ran.

Verified by the lead: deleting the forwardingTo factory's override, and rewriting the
Fleetd call site as a plain lambda, both fail to compile now rather than passing silently.
2026-09-03 10:52:01 +07:00
Dai Ha e5eb3534c7 fleetd #201/#227 unit 3: lead outage nudge
Backend incidents become a fourth source inside ReplyPushLoop, not a new scheduler — a
second injector would race the one control that already owns lead-pane delivery. One
notice per incident per affected lead (not per member), one-shot, waiting while the lead
pane is not injectable, and combined into the same nudge as any pending failed ticket.

A classified target that cannot be mapped to a credential gets its own truthful notice
and its own pending/delivered records. It previously reused the incident message, which
told the lead a credential was 'cooling for 0 remaining seconds' when nothing was cooling,
and smuggled the free-text reason into the profiles field.

Verified by the lead: 1129 tests green; routing the unmapped path back through
onBackendIncident turns unmappedBackendTargetUsesTheKnownLeadSchedule red with 0 compile
errors, and that test now asserts the whole rendered message rather than two substrings.
2026-09-03 10:49:25 +07:00
Dai Ha 959c83534f fleetd #201/#227 unit 2: credential outage policy
Adds BackendOutagePolicy — a credential-keyed state machine on an injected monotonic
clock. Two classified backend errors from two DISTINCT targets on one credential inside
60 seconds mint one incident and start a 60-second cool-off. Errors during cool-off
neither extend it nor mint another; expiry clears evidence, so two fresh errors rearm.

Correlated on credentialId, never on profile name or error text. Deliberately not
BackendQuarantine: that restarts a 1800-second cooldown per exhaustion, and its name
would make every refusal say 'backend exhausted', which is a different condition.

Threshold counts distinct targets rather than raw events (lead decision): the classifier
is a heuristic and a valid member report can quote an 'API Error:' line, so one member
repeating that line must not remove a healthy credential's capacity. A real outage hits
every member on the credential, so true detection is unaffected.

Verified by the lead: 1135 tests green; reverting evidenceCount() to reasons.size()
turns two BackendOutagePolicyTest cases red with 0 compile errors.
2026-09-03 10:49:10 +07:00
Dai Ha 31b028e860 fleetd #234 round 4: invert ExhaustionSink's abstract method so the bug class is unrepresentable
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 1m23s
Round 3's factory fixed the two known call sites but the underlying shape
was still there: a lambda written against ExhaustionSink binds to whichever
overload is abstract, and the 2-arg form held that position, so ANY lambda
-- a call-site forwarder, a hand-built test double, a future caller who has
never heard of fleetd #234 -- could still silently take the hint-dropping
default. Two rounds shipped exactly that mistake in two different places.

Fix: made the 3-arg onExhausted(target, reason, profile) the interface's
single abstract method; the 2-arg form is now a default that delegates with
a null profile. A lambda declared against ExhaustionSink today is forced by
the compiler to take three parameters -- there is no overload left for it to
bind to that can drop the hint. This is enforced by the type system, not by
a test that has to remember to check for it.

Knock-on changes:
- ExhaustionSink.none() -- a 3-arg lambda, still a genuine no-op, now safe
  by construction rather than by care.
- ExhaustionSink.forwardingTo(...) -- collapses to a one-line 3-arg lambda;
  kept as a named factory (round 3's lesson: a test must call the real
  object, not rebuild its shape).
- Fleetd.java's real sink and the two OpenCodeLauncherTest sinks that used
  to be anonymous classes overriding both overloads are now plain lambdas
  too -- the 2-arg override each carried was pure boilerplate once the
  interface provides it as a default.
- CompletionResolver.java itself: UNCHANGED, zero diff (confirmed via
  `git diff --stat` before staging) -- its two call sites still call the
  2-arg onExhausted(target, reason), which is now the default and behaves
  identically. CompletionResolverTest (41 tests, 0 failures) proves this;
  its five ExhaustionSink lambdas needed a mechanical third parameter added
  to keep compiling against the new abstract method, no assertion changed.

Mutation proof, re-run against the new shape: forwardingTo's body edited to
call the 2-arg default instead of passing the hint through (the equivalent
of round 3's "delete the 3-arg override" now that there is only one method
to break) -- both new tests go red with the same assertions as round 3:

  ExhaustionSinkForwardingHazardTest...: expected: <gx> but was: <null>
  OpenCodeLauncherTest...ForwardingHop:  expected: <true> but was: <false>
  Tests run: 68, Failures: 2

Restored, re-ran: green (Tests run: 109, Failures: 0, including
CompletionResolverTest).

Compiler proof (not committed -- a scratch file outside the worktree,
compiled with the real ExhaustionSink.java on the classpath, then deleted):

  ExhaustionSink forwarder = (target, reason) -> System.out.println(target + reason);

  error: incompatible types: incompatible parameter types in lambda expression

A 2-arg lambda against this interface no longer compiles at all.

mvn clean install: Tests run: 1129, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-09-03 10:46:27 +07:00
Dai Ha 7840e9adf6 CB-201: make unmapped target notice truthful
CI / contract (pull_request) Successful in 1m13s
CI / build (pull_request) Successful in 1m15s
2026-09-03 10:44:18 +07:00
Dai Ha c3672f5472 fleetd #201/#227 unit 4: durable BACKEND_ERROR member outcome
CI / contract (push) Successful in 42s
CI / build (push) Successful in 1m47s
Adds MemberSession.State.BACKEND_ERROR with a nullable failureReason, surfaced in
rosterView, and SessionManager.onBackendError(target, reason). The CAS loop accepts
both sides of the completion race (BUSY and DONE); BACKEND_ERROR is terminal.

completeTurn now returns early when its CAS loses, so a stale DONE copy can no longer
release the pane or reset its context behind a member that just went BACKEND_ERROR.

Verified by the lead: 1130 tests green; mutating the completeTurn early return back to
the old fall-through turns losingCompletionDoesNotReleaseOrClearABackendErrorMember red
with 0 compile errors.
2026-09-03 10:43:58 +07:00
Dai Ha cf54aed451 CB-201 unit 2 review fix: threshold counts distinct targets, not raw events
CI / contract (pull_request) Successful in 58s
CI / build (pull_request) Successful in 1m46s
Two errors from the same target inside the window must never trip the
outage threshold on their own (a valid member report can legitimately
quote an "API Error:" line twice) — only two DIFFERENT targets on the
same credential do. Change evidenceCount() to targets.size() instead
of reasons.size(); reasons() still keeps every event, including
same-target repeats, so it can be longer than evidenceCount(). A real
outage still hits every target on the credential, so this loses no
true-positive coverage while cutting a real false-positive path.
2026-09-03 10:42:37 +07:00
Dai Ha bbf68f3e3c CB-201: cover losing completion CAS
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m37s
2026-09-03 10:35:36 +07:00
Dai Ha 776743cbe2 fleetd#201 Unit 1: typed backend-error classification in CompletionResolver
CI / build (pull_request) Successful in 1m19s
CI / contract (pull_request) Successful in 1m45s
Replace the hardcoded API-Error check with a target-keyed BackendErrorPatternLookup
plus a BackendErrorSink, mirroring the existing ExhaustedPatternLookup/ExhaustionSink
pair. Classifies in all three paths (normal block, #211 raw-scrape fallback, and the
fleetd#164 MIN_TURN_NANOS floor). The sink fires only after Rendezvous.resolveFailure
wins for the exact waiter. A target with no configured pattern still falls back to the
narrow (?i)\bAPI Error\s*: compatibility pattern. Existing constructors keep compiling
via BackendErrorPatternLookup.legacy() / BackendErrorSink.none() defaults.

Public send result is unchanged (still a failed send) — the typed sink event is the
internal seam Unit 5 will consume.
2026-09-03 10:33:59 +07:00
Dai Ha 826e0aeb2a CB-201 unit 2: credential-keyed backend outage policy
CI / build (pull_request) Successful in 1m11s
CI / contract (pull_request) Successful in 1m24s
Add BackendOutagePolicy: two classified backend errors on the same
credentialId within a 60s window mint one Incident and start a 60s
cool-off for that credential, one atomic ConcurrentHashMap.compute()
per credentialId so a concurrent second and third event can never
both cross the threshold. Errors during cool-off are ignored outright
(no extension, no incident); once cool-off elapses the next error
clears old evidence, requiring two fresh errors to rearm. This is a
new class, deliberately not BackendQuarantine (wrong store, wrong
1800s duration, misleading "exhausted" semantics for a 60s transient
fault). Knows nothing about panes, profiles, sessions, launchers, or
leads — takes events in, returns incidents out.
2026-09-03 10:30:40 +07:00
Dai Ha c935b181dd fleetd #234 round 3: make ExhaustionSink's forwarder a shared factory, not a rebuilt-per-caller shape
CI / contract (pull_request) Successful in 42s
CI / build (pull_request) Successful in 1m36s
Round 2's tests never reached Fleetd.java at all: both new tests declared
their OWN local copy of the forwarding shape instead of calling production's.
Mutating Fleetd.java's real forwarder back into the broken lambda left those
copies untouched, so the whole suite stayed green while production had
regressed to exactly the bug being fixed -- proven live by the reviewer.

Fix: extracted the forwarding shape into one named factory,
ExhaustionSink.forwardingTo(Supplier<ExhaustionSink> target), with the
"why a lambda here is wrong" explanation moved onto it (the one place the
shape is now written). Fleetd.java's forwarder collapses to one line:

    ExhaustionSink forwardingExhaustionSink = ExhaustionSink.forwardingTo(exhaustionSinkRef::get);

Both new tests now call this same factory instead of rebuilding an anonymous
class inline, so they exercise the identical object production builds:
- ExhaustionSinkForwardingHazardTest: calls ExhaustionSink.forwardingTo
  directly and asserts the hint reaches the real sink through it.
- OpenCodeLauncherTest#theSpawnTimeQuarantineSurvivesTheFleetdStyleForwardingHop:
  same factory call, inside the full Fleetd-shaped construction order
  (forwarder built first, real sink pointed at via the AtomicReference
  afterward), driven through the real SessionManager.acquire() path.

Mutation proof, this time on production code only: deleted the factory's
3-arg override (falls back to the interface default, dropping the hint) --
both new tests go red with no test file touched:

  ExhaustionSinkForwardingHazardTest...: expected: <gx> but was: <null>
  OpenCodeLauncherTest...ForwardingHop:  expected: <true> but was: <false>
  Tests run: 68, Failures: 2

Restored, re-ran: green (Tests run: 68, Failures: 0). Confirmed Fleetd.java
carries no lambda ExhaustionSink anywhere (grep). ExhaustionSink.none() stays
a lambda on purpose -- both its overloads are true no-ops regardless of
arity, so there is no hint to drop.

mvn clean install: Tests run: 1129, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-09-03 10:30:37 +07:00
Dai Ha ee932fd85b CB-201: nudge leads about backend outages
CI / contract (pull_request) Successful in 1m7s
CI / build (pull_request) Successful in 1m55s
2026-09-03 10:28:51 +07:00
Dai Ha fe2e5ede34 CB-201: retain backend failure outcome
CI / build (pull_request) Successful in 1m11s
CI / contract (pull_request) Successful in 1m23s
2026-09-03 10:26:54 +07:00
Dai Ha c325054242 fleetd #234 round 2: fix the ExhaustionSink forwarding hop Fleetd.java actually uses
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 1m18s
The round-1 fix was dead on the real production path. Fleetd.java:177 builds
a forwarding sink (needed because the adapters are constructed before
`sessions` exists, breaking a genuine cycle) as a LAMBDA:

    ExhaustionSink forwardingExhaustionSink =
            (target, reason) -> exhaustionSinkRef.get().onExhausted(target, reason);

A lambda can only implement the interface's one abstract method (the 2-arg
overload), so it silently inherited the 3-arg overload's default body, which
drops the profile hint and calls back into the 2-arg method. OpenCodeLauncher
is constructed with this forwarder, so the hint it supplies (its own
already-known profile name) was thrown away before it ever reached the real
sink built later in Fleetd.main -- reproducing the exact silent no-op round 1
was sent to fix. The 1127 tests from round 1 all injected a sink directly
into OpenCodeLauncher and never went through this forwarding hop, so none of
them could see it.

Fix: forwardingExhaustionSink is now an anonymous class overriding both
overloads, each delegating to whatever exhaustionSinkRef currently holds.

Audited every other ExhaustionSink value in main/: the only other one is
ExhaustionSink.none() (a lambda), which is safe regardless of arity since
both its 2-arg body and the inherited 3-arg default are true no-ops.

New tests:
- ExhaustionSinkForwardingHazardTest: isolates the hazard at the interface
  level (a lambda forwarder drops the hint; an anonymous-class forwarder
  does not), independent of Fleetd.java's specific wiring.
- OpenCodeLauncherTest#theSpawnTimeQuarantineSurvivesTheFleetdStyleForwardingHop:
  replicates Fleetd.java's actual construction order (forwarder built and
  handed to the launcher first, real sink built and pointed at via the
  AtomicReference afterward) and drives the quarantine through it via the
  real SessionManager.acquire() path.

Both proven by mutation: temporarily rewriting each fixed forwarder back
into the pre-fix lambda makes its test fail with a real assertion message
(both matched exactly: "expected: <gx> but was: <null>" for the interface
proof, "expected: <true> but was: <false>" for the composed-wiring test);
restoring makes it pass again. No reverts were committed.

mvn clean install: Tests run: 1130, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-09-03 10:22:31 +07:00
Dai Ha 7662e2d0c8 fleetd #201/#227: refine backend outage work
CI / contract (push) Successful in 53s
CI / build (push) Successful in 1m49s
2026-09-03 10:21:44 +07:00
Dai Ha 4877992a70 fleetd #234: key the opencode model check on the resolved session id, and make the spawn-time quarantine actually happen
CI / contract (pull_request) Successful in 58s
CI / build (pull_request) Successful in 1m29s
Defect 1: OpenCodeSessionDiscovery.actualModelForDirectory queried
WHERE directory = ?, the same heuristic sessionIdForDirectory uses. Since a
default fleet_spawn (no worktree:) shares the lead's cwd with every other
worker and every past session ever run there, the model read-back could
silently compare against a DIFFERENT session's row. Renamed to
actualModelForSessionId(sessionId), keyed on the primary key id instead, and
made OpenCodeLauncher's SessionAwareHandle cache the resolved id once
non-null (AtomicReference) so a later sibling row in the same directory can
never flip which session's evidence is read. sessionIdForDirectory (#209) is
left directory-based on purpose, with a comment explaining why the heuristic
is unavoidable at that layer.

Defect 2: the ERROR log claimed "quarantining this profile's credential" but
Fleetd's ExhaustionSink lambda resolved target -> roster -> profile ->
credential, while OpenCodeLauncher's model-mismatch check fires from
agentSessionId() during SessionManager.acquire(), before the session is
registered in the roster -- the lookup found nothing and silently no-opped.
Added a default 3-arg ExhaustionSink.onExhausted(target, reason, profile)
overload (defaults to the 2-arg method, so CompletionResolver's two call
sites are unchanged); OpenCodeLauncher now passes its own already-known
profile name; Fleetd's sink became an anonymous class that tries the roster
first, falls back to the hint, and logs loudly at ERROR naming target/reason
when neither resolves, instead of silently no-oping.

Both fixes proven by mutation: reverting each independently makes its new
test fail with a real assertion message, restoring makes it pass again.

mvn clean install: Tests run: 1127, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-09-03 10:09:26 +07:00
ltms 1c051c4e47 fleetd #115: a profile that needs no token no longer warns, and no longer crashes
CI / contract (push) Successful in 1m2s
CI / build (push) Successful in 1m50s
tokenEnv no longer defaults to FLEETD_WORKER_TOKEN. A profile that names no
tokenEnv is stating it needs none, which is different from one naming a variable
that turns out to be unset. requiredSecretEnvVars now warns only for an explicitly
declared tokenEnv, so the permanent false alarm about a token no profile needs is
gone and the real warnings beside it stay trustworthy.

Decision recorded in a code comment: an explicit tokenEnv is checked for every
kind, opencode included. opencode can use its own provider credentials, but an
explicit tokenEnv declares a required host secret for its configured provider.

Second commit fixes a crash the first one introduced. Making tokenEnv nullable
changed what every reader of that value can receive, and ClaudeCodeLauncher:261
passed it straight to env.apply — System::getenv in production, which throws on a
null name. The live profile local-direct (kind: claude-code, baseUrl set, no
tokenEnv) would have crashed on spawn. It is weight: 0 today, so the failure would
have surfaced whenever someone re-enabled it. Now uses the superclass helper
resolveEnv, which already tolerates a null name and which OpenCodeLauncher was
already using.

Reader audit, all four: ClaudeCodeLauncher fixed; OpenCodeLauncher already safe via
resolveEnv; ConfigRef uses Objects.equals; MemberEnvAllowList drops null and blank
names in addIfPresent.

Verified by the lead before merge: reverting the resolveEnv fix makes the new test
fail with a NullPointerException from the null variable name, and restoring it
passes. Independent build: 1125 tests, 0 failures, 0 errors, 0 skipped.

Not verified: the ticket's acceptance criterion 4, a real boot on this host showing
no FLEETD_WORKER_TOKEN line while the WORKER_GITEA_TOKEN and AI_GATEWAY_TOKEN lines
are unchanged. A worker cannot restart the daemon it talks through. The lead checks
that at the next redeploy.
2026-09-03 05:02:27 +02:00
Dai Ha adb7a67880 CB-612: support tokenless Claude Code profiles
CI / contract (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 1m24s
2026-09-03 09:59:21 +07:00
Dai Ha 723fe494e9 CB-612: suppress unneeded token warnings
CI / contract (pull_request) Successful in 1m7s
CI / build (pull_request) Successful in 1m17s
2026-09-03 09:50:49 +07:00
ltms 0f08b93659 fleetd #176: spawn gate fails fast on a dead backend, and refines UNKNOWN safely
CI / contract (push) Successful in 52s
CI / build (push) Successful in 1m47s
Two fixes to HerdrPeerLauncher.waitUntilInjectableOrThrow.

Fix 1: the status call is now guarded. A herdr *_not_found answer — what happens
when the backend process exited rather than being slow — used to escape as a raw
HerdrException, skipping stop() and leaking the pane and tab, with a message that
blamed a slow pane. It now fails immediately, runs the same teardown, and says the
process exited.

Fix 2: the gate can now resolve UNKNOWN with the same StatusRefiner the status
poller uses, behind two guards. It only runs for the claude adapter, because
StatusRefiner.classify reads a Claude Code TUI. And a refined result is accepted
only when the same agents.get sample still reports a non-null agentType. That
second guard closes a trap: a dead pane sits at a shell prompt containing the same
❯ glyph the classifier reads as idle, so refining without corroboration would turn
"the backend died" into "ready to inject".

Verified by the lead before merge: NAME_PREFIX really is "claude"/"opencode" so the
adapter split holds; AgentControl.status(t) was already get(t).status(), so moving
the loop to get() adds no herdr call; and a *_not_found already failed the spawn
before this change, so fix 1 improves an existing failure rather than creating one.
The adapter guard was mutation-tested independently (set it to `if (false)`, the new
opencode test fails with "Expected PeerUnreachableException to be thrown, but nothing
was thrown"; restored, it passes). Independent build: 1123 tests, 0 failures,
0 errors, 0 skipped.

Not fixed, and not claimed: that herdr reports a null agentType for a bare shell
pane is unverified against a live daemon. To be proved by a live spawn on both
backends after deploy. The seat-accounting suggestion in #176 was deliberately not
built — its cause was tested in that issue and not reproduced.
2026-09-03 04:20:23 +02:00
Dai Ha 432c1d92d1 fleetd #176 review: cover the namePrefix guard, the adapter-refinement corroboration that had no test
CI / contract (pull_request) Successful in 1m11s
CI / build (pull_request) Successful in 1m14s
The agentType corroboration (guard b) had positive and negative tests, but the
per-adapter namePrefix guard (guard a) had none — nothing proved that an
opencode pane can never reach StatusRefiner.classify, only that a claude pane
with a null agentType is rejected. Since a live opencode pane always reports a
non-null agentType ("opencode"), guard (b) alone cannot catch a broken guard
(a).

Added opencodePaneIsNeverRefinedEvenWhenItsContentLooksLikeAnIdleClaudePrompt
to OpenCodeLauncherTest: agentType("opencode") (non-null, satisfies guard b on
its own) + pane content containing "❯" (would classify as IDLE) + raw status
UNKNOWN throughout. Asserts the gate still times out, and that no agent.read
call used source=detection (StatusRefiner.PROBE_SOURCE) — proving the refiner
was never even reached, not just that its answer was discarded. A blanket
"agent.read is never called" does not hold here: HerdrPeerLauncher.readPaneQuietly
reads the pane tail (source=recent) for the timeout log on every timeout,
regardless of adapter, so the assertion is scoped to the refiner's own probe
source instead.

Mutation check performed and reverted before this commit: temporarily changed
`if (!"claude".equals(namePrefix))` to `if (false)` in
HerdrPeerLauncher.refinedInjectable — the new test failed
("Expected PeerUnreachableException to be thrown, but nothing was thrown"),
confirming it actually exercises the guard. Restored the guard and reran —
test passes again (1/1). No production code changed in this commit.

mvn clean install: Tests run: 1123, Failures: 0, Errors: 0, Skipped: 0 -- BUILD SUCCESS
2026-09-03 09:17:10 +07:00
Dai Ha 60b7e67b42 fleetd #176: fail fast when the spawn gate's backend exits mid-wait, and safely refine its UNKNOWN status
CI / build (pull_request) Successful in 1m13s
CI / contract (pull_request) Successful in 1m26s
Fix 1: agents.get(paneId) inside waitUntilInjectableOrThrow was unguarded, so a
herdr *_not_found answer (the backend process exited) propagated as a raw
HerdrException instead of PeerUnreachableException, and skipped teardown
entirely, leaking the pane/tab. Now caught via isAlreadyGone(); fails
immediately (does not burn the rest of the timeout), runs the same teardown
the timeout path runs, and the exception message says the process exited
rather than that the pane was slow. Any other HerdrException still
propagates unchanged.

Fix 2: the gate now resolves a raw UNKNOWN into StatusRefiner's pane-content
classification, like StatusPoller already does mid-life. Guarded against the
trap noted in the ticket: a pane whose backend exited settles at a bare shell
prompt that can also contain the "❯" glyph classify() reads as idle. So a
refined result is accepted only when the corroborating agentType from the
SAME agent.get sample is non-null, and only for the "claude" adapter (namePrefix)
since StatusRefiner.classify is written for the Claude Code TUI only. Refine
only runs when the raw status is UNKNOWN, so a healthy spawn adds zero extra
herdr calls.

Tests added to ClaudeCodeLauncherTest (FakeHerdr gained agentType()/
agentGetFailsWithAfter() fixtures):
 - spawnFailsFastWhenBackendProcessExitsMidWaitInsteadOfBurningTheTimeout
 - spawnLetsAnUnrelatedHerdrErrorPropagateUnchanged
 - refinedIdleIsNotAcceptedWhenAgentTypeIsNull
 - refinedIdleIsAcceptedWhenAgentTypeCorroboratesLiveness
 - refinementNeverFiresWhenRawStatusIsAlreadyInjectable

mvn clean install: Tests run: 1122, Failures: 0, Errors: 0, Skipped: 0 -- BUILD SUCCESS
2026-09-03 09:10:49 +07:00
ltms 3fbd43fe3f fleetd #175: read back the model opencode actually resolved, and quarantine a silent substitution
CI / contract (push) Successful in 44s
CI / build (push) Successful in 1m29s
Verified by the lead before merge. Read the full production diff, checkModelMatch, parseModel and actualModelForDirectory. Confirmed the check runs on the real #209 late-resolve path (not at spawn, which is why #203 was closed), that unknown/incomplete evidence never quarantines, and that claude-code is structurally excluded because SessionAwareHandle is only built by OpenCodeLauncher.spawn(). Round 2 closed the one gap I found: model JSON with an id but no providerID used to read as a mismatch for a provider-prefixed profile. Independent build: BUILD SUCCESS, 1117 tests, 0 failures, 0 skipped.
2026-09-02 13:01:33 +02:00
Dai Ha 32ebf065ac fleetd #175 review: a missing providerID in opencode's model JSON is UNKNOWN, not a mismatch
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 1m13s
parseModel already tolerates a model JSON with an id but no providerID (a real shape
opencode can write). checkModelMatch's providerMatches check did not: a provider-
prefixed profile whose id matched but whose evidence had no providerID was reported
as a mismatch and quarantined on incomplete data, which acceptance rule 4 forbids.

Compare the provider only when BOTH the profile requested one AND the evidence has
one. A genuine id mismatch is still caught either way — narrows the check, does not
disable it.
2026-09-02 17:59:04 +07:00
Dai Ha 1178b3f684 fleetd #175: check opencode's actual model against the profile, quarantine on a real mismatch
CI / contract (pull_request) Successful in 42s
CI / build (pull_request) Successful in 1m49s
opencode does not fail on an unknown -m <model> flag — it silently falls back to a
default model, which can be a paid credential. Extends the existing late-resolve
path (#209's SessionManager -> handle.agentSessionId() re-poll) so that once the
opencode session row exists, OpenCodeSessionDiscovery also reads its `model` JSON
column and OpenCodeLauncher's SessionAwareHandle compares it against the profile's
configured model.

Comparison rule: split the profile's model on the first '/' into provider+id. Compare
id always; compare provider only when the profile specified one. A bare model name
with no '/' matches on id alone. Absent/unparseable evidence is UNKNOWN, never a
mismatch, so a working profile is never quarantined on missing data. A real mismatch
logs an ERROR naming both models and the profile, then quarantines through the
existing ExhaustionSink path (wired via an AtomicReference forwarding sink in
Fleetd.java to break the sessions/workers/adapters construction cycle).
2026-09-02 17:53:59 +07:00
ltms 2a434ced2f fleetd #222: keep the member's charter file out of fleetd's own java.io.tmpdir
CI / contract (push) Successful in 58s
CI / build (push) Successful in 1m28s
Verified by the lead before merge: read the full production diff, confirmed the memberHerdrSocket-absent branch is the literal unmodified Files.createTempFile call in its own branch, and that the refusal names the missing key. Measured behaviour recorded: claude 2.1.258 exits 1 immediately on an unreadable --append-system-prompt-file, so the pre-fix bug was the loud readiness-gate failure, not a silent charter-less member. The /tmp full-suite failure the worker reported was the two other #225 copies, fixed by #230 which is already on main; main is built and checked after this merge.
2026-09-02 02:56:36 +02:00
ltms cc919aa2b6 fleetd #226: reserve an architect slot before launch, so no member runs on a charter it will not hold
CI / contract (push) Successful in 41s
CI / build (push) Successful in 1m31s
Verified by the lead before merge. Read the full production diff: locking is consistent (every MemberRegistry method uses synchronized(terminalToSlot)), and the refusal happens before the launcher starts a process. Proved the restored fallback test is real by removing the DEV fallback from SessionManager and re-running it: it failed with "expected: <DEV> but was: <ARCHITECT>", then reverted. Independent build: BUILD SUCCESS, 1092 tests, 0 failures, 0 skipped.
2026-09-02 02:54:52 +02:00
Dai Ha 6417b0edd9 fleetd#222: fix currentUserGroup() to resolve the real primary group (fleetd#225)
CI / contract (pull_request) Successful in 52s
CI / build (pull_request) Successful in 1m43s
The helper I copied from OpenCodeLauncherTest read the CWD's owning
group instead of the process's real primary group, so it silently
picked up whatever group owns the directory Maven was started from
(staff in a home checkout, wheel under /private/tmp on macOS) rather
than a group the operator is actually in. Replaced with the id -gn
based resolution that landed on #230 for the other two copies of this
helper, same shape and skip wording.
2026-09-02 07:54:11 +07:00
ltms 39c7ce76f3 fleetd #224 + #225: make worktreeRoot group-traversable, and stop two tests depending on the checkout location
CI / contract (push) Successful in 53s
CI / build (push) Successful in 1m41s
Verified by the lead before merge: read the full production diff, confirmed `group` is normalised to null at GitWorktrees:148 so the `group == null` guard is complete, and confirmed the refusal runs before `git worktree add` so a failure leaves no half-made worktree. Independent build in the worker's worktree: BUILD SUCCESS, 1093 tests, 0 failures, 0 skipped.
2026-09-02 02:52:12 +02:00
Dai Ha b40f477210 #226 retain architect fallback coverage
CI / contract (pull_request) Successful in 1m17s
CI / build (pull_request) Successful in 1m33s
2026-09-02 07:50:34 +07:00
Dai Ha 5c56cb347f fleetd #224 / #225: share worktreeRoot with the group, and fix the group-detection test helper
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m42s
#224: GitWorktrees#add created worktreeRoot with the daemon's umask and never shared it with
worktreeGroup, even though shareWithGroup shares every child underneath it (each worktree, and
the repo's common git dir). Under memberHerdrSocket: the member pane runs as a different OS
user, which needs execute on every ancestor directory to reach anything underneath, no matter
how carefully each child is shared — so a member could not read the opencode.json #219 places
under this root, could not reach its own worktree, and could not read #213's ZDOTDIR scrub when
placed here either.

Fix: add() now calls a new shareRootWithGroup(root) right after creating the root, chgrp+chmod
g+x on the root itself (non-recursive — each child is still shared individually by its own call
site). No-op when worktreeGroup is unset, so behaviour is byte-identical in today's only live
mode. On failure (group missing, or operator not a member of it) the spawn is refused with a
WorktreeException naming the root, its current mode, and the group — mirroring shareWithGroup's
existing refusal shape — before `git worktree add` ever runs, so no partial worktree is left
behind.

Also adds the assertion the #221 reviewer flagged as missing: a test driving
EnvAllowListScrub#shareWithGroup directly against a directory holding several flat files
(opencode.json, member-charter.md, ide-rules.md, plus an unrelated one) and asserting every one
of them gets group-readable/never-group-writable permissions, not just the two files someone
happened to think of.

#225: OpenCodeLauncherTest/HerdrPeerLauncherAllowListWiringTest's currentUserGroup() read the
group that owns the current working directory, not the process's own primary group, despite its
comment claiming the latter. Those coincide only by accident: a home checkout is typically owned
by a group the operator belongs to (staff), while a checkout under /private/tmp on macOS is
group wheel, which the operator is usually not a member of — so the same test fails for real
depending on where the repo happens to be checked out, and the existing assumeTrue only guarded
against "no POSIX groups at all", never "a resolvable but wrong group". Fixed by resolving the
process's REAL primary group via `id -gn` instead, with assumeTrue (skip, not fail) only when
that itself cannot be resolved on the host. The permission assertions these tests exist for are
unchanged.

Verified `mvn clean install` green from both a home checkout and a /private/tmp copy (mirroring
the exact repro in #225): 1093 tests, 0 failures, 0 errors in both locations.
2026-09-02 07:47:13 +07:00
Dai Ha e694deace3 #226 reserve architect slots before launch
CI / contract (pull_request) Successful in 1m19s
CI / build (pull_request) Failing after 1m33s
2026-09-02 07:44:41 +07:00
Dai Ha 748367b7d6 fleetd#222: put the claude-code charter file where the member can read it
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m16s
ClaudeCodeLauncher#writeCharterFile used Files.createTempFile with no
directory argument, which resolves against fleetd's own java.io.tmpdir
(macOS: the per-user $TMPDIR, mode 0700). Under memberHerdrSocket: the
member pane runs as a different OS user and cannot read that directory,
and since #220 the charter file is the ONLY delivery path for
--append-system-prompt-file. Following #219's refusal decision (a
charter is the member's turn contract, not a degradable control): with
memberHerdrSocket configured, the charter now goes into a fresh
per-spawn directory under worktreeRoot, shared read-only via
EnvAllowListScrub.shareWithGroup (reusing #213/#219's mechanism); a
missing worktreeRoot/worktreeGroup refuses the spawn by name instead of
writing an unreadable file. With memberHerdrSocket absent the path is
unchanged.
2026-09-02 07:44:34 +07:00
ltms dcf5fb3be3 Merge pull request 'CB-619 / fleetd #123: refuse an architect spawn with no matching slot' (#223) from worker/cb-123-role-demotion-c600f7-2 into main
CI / contract (push) Successful in 1m11s
CI / build (push) Successful in 1m55s
2026-09-01 10:41:47 +02:00
ltms f9d2ee2a2b Merge pull request 'fleetd#219: fix OpenCodeLauncher config/discovery roots under memberHerdrSocket' (#221) from worker/cb-219-opencode-roots-1f677e-1 into main
CI / contract (push) Successful in 45s
CI / build (push) Successful in 2m16s
2026-09-01 10:33:04 +02:00
Dai Ha 866c7f2e9a CB-619 / fleetd #123: refuse an architect spawn with no matching slot
CI / build (pull_request) Successful in 1m16s
CI / contract (pull_request) Successful in 1m18s
An explicit-profile spawn bypasses role-pool placement (CompositePeerLauncher
only constrains an UNQUALIFIED spawn to fleet.<role>), so it was the one path
that could ask for role=architect on a profile no architect slot carries.
MemberRegistry silently held the session as a plain worker while GET /members
still reported the requested "architect" and only fleet_whoami (which reads
live bindings, not the request) told the truth.

- MemberLifecycle.requireSlotFor(role, profile): refuses the acquire before
  anything spawns when no configured architect slot carries the profile,
  naming the role, the profile, and the pools that do carry it. No-op for
  dev/reviewer, which are placement candidates only, never a live identity
  binding — refusing a profile mismatch there would break the documented
  fleet_spawn{profile:"opus"} (role defaults to dev) flow.
- MemberLifecycle.acquired(...) now returns the role the session actually
  holds, so a residual race (a slot exists but every instance is already
  bound to a different terminal) still falls back to dev honestly instead of
  lying — this case logs at WARN (was INFO), naming profile and terminal.
- SessionManager now records the role acquired() returns on MemberSession,
  never the requested role, so GET /members and fleet_list can no longer
  report a role the member does not hold; no changes needed to memberView/
  rosterView, which just read session.role().

An architect's identity IS the slot it is bound to — binding a role with no
slot to bind means inventing an identity out of nothing, which is the quiet
failure the whole role system exists to prevent.

Tests: SessionManagerTest and FleetMcpTest each drive a real spawn through
FleetMcp.spawn -> SessionManager.acquire -> the real ClaudeCodeLauncher (via
FakeHerdr), then assert on GET /members and fleet_whoami for that same
session — not on MemberRegistry.bind directly (fleetd issue #113's mistake).
2026-09-01 15:29:39 +07:00
Dai Ha d9168de43e fleetd#219: OpenCodeLauncher config/discovery roots must not assume fleetd's own filesystem
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m48s
Site 1 (config root): under memberHerdrSocket, writeConfig() now places the
ephemeral opencode.json directory under worktreeRoot and shares it read-only
with worktreeGroup, reusing EnvAllowListScrub#shareWithGroup (widened to
package-private and generalized) — the same mechanism #213 built for the
ZDOTDIR scrub, rather than a second copy. Unlike the ZDOTDIR scrub's
degrade-to-overlay fallback, a missing worktreeRoot/worktreeGroup here
REFUSES the spawn (IllegalStateException from buildLaunch): this file is the
member's only way to learn where the bridge MCP is, so writing it somewhere
unreadable would just produce an undeliverable member with no signal
pointing at the cause. memberHerdrSocket absent stays byte-identical.

Site 2 (discovery root): under memberHerdrSocket, agentSessionId() now
declares session discovery unavailable and logs one WARN per launcher
instance instead of silently scanning fleetd's own $HOME (opencode.db lives
under the MEMBER's home under this config key). Decision + reasoning for why
this is a declare-unavailable rather than a new config key is in
defaultDiscoveryRoot()'s javadoc.

Widened HerdrPeerLauncher#memberHerdrSocketConfigured/memberScrubParentDir/
memberGroup to package-private so OpenCodeLauncher reuses the exact same
config resolution rather than re-deriving it.

Same-shape finding (not fixed, out of scope): ClaudeCodeLauncher#writeCharterFile
(line ~465) writes the role-charter temp file via Files.createTempFile with no
directory argument, i.e. under java.io.tmpdir — the same site-1 shape, unfixed
for the Claude Code adapter.
2026-09-01 14:38:48 +07:00
Dai Ha c3fa1136d4 #220: keep the launch command inside the pane's 1024-byte line
CI / build (push) Successful in 1m37s
CI / contract (push) Successful in 1m37s
herdr does not exec a member's launch command — it TYPES it into the pane,
and a pty line buffer holds 1024 bytes (BSD/macOS MAX_CANON). Past that the
tail is dropped and NOTHING reports it: herdr answers "agent started", the
backend exits on the mangled argument it was handed, the pane closes, and the
only symptom is the readiness gate timing out 20 seconds later with no reason.

That is what broke every claude-code spawn after #214. The reply charter rode
inline on --append-system-prompt, so the command was already 978 bytes; adding
--session-id <uuid> made it 1028, and the 4 bytes cut off the end turned
--autocompact 250000 into --autocompact 25, which claude rejects. Measured on
the live pane, the cut is at byte 1024 exactly.

- ClaudeCodeLauncher: the charter ALWAYS travels as --append-system-prompt-file.
  The file path already existed for the two-charter case; the inline form only
  ever saved a temp file, and it cost ~800 bytes of the line budget. This takes
  the prose off the command line for good.
- HerdrPeerLauncher.checkPaneCommandFits: refuse a command that cannot fit,
  naming the byte count and the longest argument, instead of spawning something
  that cannot work. The estimate is deliberately conservative — fleetd cannot
  see herdr's quoting, and an under-estimate would let the silent truncation
  back in.
- HerdrPeerLauncher.waitUntilInjectableOrThrow: log the pane tail and the last
  herdr status BEFORE stop() closes the pane. Without it the gate reports only
  that it timed out, which is true of every cause. This is what found the bug,
  and it stays.

The guard also catches a case that was already over the limit: a profile with
ideMcpUrl set assembles 1084 bytes. It is now impossible to ship that silently.

3 tests, all watched failing first: with the inline charter restored the guard
fires in the new fit test, in the pre-existing autocompact test and in the IDE
mount test. Full suite 1081 tests green. Proven live: sonnet spawns again, the
member obeys the file-delivered charter and ends its turn with fleet_reply, and
fleet_list reports the #214 agentSessionId.
2026-09-01 14:11:28 +07:00
Dai Ha cabcd87b66 #211 follow-up: the raw-scrape fallback must carry the pane, not just the matched line
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m50s
The normal backend-error path appends the pane tail to the failure reason on
purpose (fleetd#164): the BACKEND_ERROR pattern is a heuristic, and a member
that reported *about* an error while forgetting fleet_reply matches it too, so
dropping the rest of the pane destroys the report.

The new raw-scrape fallback did not do that. It matters more there, not less:
the fallback only runs when the trimmed assistant block was empty, so the raw
scrape is the ONLY copy of whatever the member managed to say. A lead read the
matched line and nothing else.

Clipped to the same cap the normal path uses, since a raw screen has no
boundary trimming to bound its size.

The assertion was watched failing without the fix:
  AssertionFailedError: fleetd#164: the failure must carry the pane, not only
  the matched line ... expected: <true> but was: <false>

mvn clean install: Tests run: 1079, Failures: 0, Errors: 0, Skipped: 0
2026-09-01 13:26:05 +07:00
Dai Ha bf0e09b1a2 #214: mint a session id on every claude-code spawn, so every member is resumable 2026-09-01 13:23:32 +07:00
Dai Ha e1eb50ce65 #213: fix ZDOTDIR credential scrub gate + directory under memberHerdrSocket 2026-09-01 13:23:32 +07:00
Dai Ha 049e7d9d54 fleetd#211: classify BACKEND_EXHAUSTED/BACKEND_ERROR from the raw scrape as a fallback 2026-09-01 13:23:32 +07:00
Dai Ha ff3b49cd1e #214: mint a session id on every claude-code spawn, so every member is resumable
CI / contract (pull_request) Successful in 1m15s
CI / build (pull_request) Successful in 1m52s
2026-09-01 11:27:10 +07:00
Dai Ha 0373b6c41b #213: fix ZDOTDIR credential scrub gate + directory under memberHerdrSocket
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 2m17s
The memberCredentials.policy: allow-list ZDOTDIR scrub decided zsh-vs-not
using fleetd's own process $SHELL and wrote the generated scrub dir into
fleetd's own java.io.tmpdir. Under memberHerdrSocket: (member panes run as
a different OS user than fleetd's own process) this silently protects
nothing: the wrong shell decides the gate, and the directory can be
unreachable to the member.

- New FleetConfig.memberLoginShell: the member OS user's login shell,
  only ever read when memberHerdrSocket: is configured; fleetd's own
  $SHELL keeps deciding everything when memberHerdrSocket: is absent
  (byte-identical to before).
- HerdrPeerLauncher.applyEnvironmentAllowListPolicy: memberHerdrSocket +
  memberLoginShell not configured/non-zsh falls back to the CB-596
  sentinel overlay (warn loudly, never refuse to spawn). memberHerdrSocket
  + zsh memberLoginShell generates the ZDOTDIR under the configured
  worktreeRoot instead of java.io.tmpdir, shared with the existing
  worktreeGroup (reused, not a new key).
- EnvAllowListScrub: new generate(parentDir, allowedNames, group) overload
  shares the generated directory via pure-Java POSIX group ownership
  (rwxr-x--- dir, rw-r----- files) — no external process spawn.
- fleetd.example.yaml documents memberLoginShell: and worktreeGroup:'s
  reuse for the scrub directory (the live fleetd.yaml is gitignored).

4 new tests in HerdrPeerLauncherAllowListWiringTest cover the acceptance
criteria; 3 of the 4 were watched failing against the pre-fix code.
2026-09-01 11:07:11 +07:00
Dai Ha 445a45f6e1 fleetd#211: classify BACKEND_EXHAUSTED/BACKEND_ERROR from the raw scrape as a fallback
CI / contract (pull_request) Successful in 50s
CI / build (pull_request) Successful in 1m17s
CompletionResolver.resolve() returned an empty-scrape failure before the
BACKEND_EXHAUSTED / BACKEND_ERROR classification ever ran, whenever
lastAssistantBlock() found no usable text — most commonly a pane with no ⏺
marker at all, whose boundary scan starts at the top of the raw screen and
breaks immediately on the first line of TUI chrome. Since BACKEND_EXHAUSTED
is the only caller of exhaustionSink, this meant an exhausted backend was
recorded as "produced nothing" instead of being quarantined.

Fix: run the same two classifications against the raw (untrimmed) scrape as
a fallback, only inside the empty-scrape failure branch. A pane that already
yields a usable assistant block never reaches this branch, so the existing
narrow match is unchanged. lastAssistantBlock stays the sole source of the
reply text; only classification ever consults the raw scrape.
2026-09-01 10:50:10 +07:00
Dai Ha 966c58a3b8 #137: don't hand one stranded reply to every open async ticket (#215)
CI / contract (push) Successful in 1m31s
CI / build (push) Successful in 1m41s
abandon() drained the target's stranded reply once and then reused that same
Reply for every open task it walked past. Two open tickets on one target
therefore both came back REPLIED with the same text — one of them a reply the
worker never gave for that delegation.

One worker answer can settle at most one delegation. It now goes to the oldest
open task (lowest createdNanos) and every other open task keeps the ordinary
WORKER_FAILED path. If the chosen task turns out to be already resolved by
another path, the drained reply is published back to the inbox instead of being
dropped.

reply()'s matching side gets the same rule: more than one candidate means the
reply goes to the inbox rather than to a guess.

Reachability, checked rather than assumed:
- matching.size() >= 2 alone is reachable and was a real bug before this change.
- More than one candidate in reply() is not reachable today — hasAsyncQuestion
  matches any task with a stamped turnId, and answer()'s
  clearAsyncQuestion(turnId, false) leaves that stamp until the resumed turn
  resolves. Kept as defensive code, documented, no test seam added.
- matching.size() >= 2 together with a live strand is not reachable either:
  send() and answer() are the only two lock holders and both open a Rendezvous
  waiter inside the lock, so "lock held" and "waiter open" are one fact, and an
  acceptance always clears the strand first.

I checked the last point by building it in a scratch worktree: it can be forced
by closing an accepted send's waiter directly through Rendezvous, and it does go
red against the pre-fix loop — but that breaks the lock-and-waiter invariant
from outside the class, so no such test is added. A note in abandon()'s javadoc
says so, to save the next reader the same round trip.

Worker's REPORT-cb137.md left out of main.

mvn clean install: Tests run: 1071, Failures: 0, Errors: 0, Skipped: 0
2026-08-31 23:23:07 +07:00
Dai Ha de026b8f8a #137 follow-up: document defect-1 and defect-2 reachability, drop unconstructible test
CI / contract (pull_request) Successful in 48s
CI / build (pull_request) Successful in 1m28s
Defect 1 (reply()'s askAnsweredAsyncTasks returning >1 candidate): confirmed
unreachable today. Documented why in three places — hasAsyncQuestion matches
any task with a stamped turnId (not just an open question), and answer()'s
clearAsyncQuestion(turnId, false) leaves that stamp in place until the
resumed turn's own future resolves — so a second task can never reach the
same eligible state while a first one holds it. Kept the defensive
inbox-fallback branch as defence in depth against that guarantee weakening,
per review instruction; no test seam added.

Defect 2 (abandon()'s broader `matching` filter applying a stranded reply to
more than one task): matching.size() >= 2 alone IS reachable (already
covered by abandonFailsEveryPendingAsyncTicketForTheReleasedTarget) and was
a real pre-fix bug (97f6c33's parent reused one drained reply for every
matching task). But hadStrandedReply == true together with matching.size()
>= 2, at the instant abandon() runs, is not constructible through the public
API: send() and answer() are the only two sites that ever hold a target's
session lock, and both open a Rendezvous waiter for that target as the first
thing they do while holding it — so "lock held" and "waiter open" are the
same fact throughout this class, and reply()'s fast path always resolves an
open waiter directly instead of stranding. A strand can only be created
while no task is accepted, and the moment the lock is next taken, that
acceptance clears the strand again before abandon() can observe both facts
together. Documented this in abandon()'s javadoc and removed the earlier
attempt at a deterministic test for the conjunction, whose apparent failure
was an invalid premise (the "accepted" task's own acceptance silently
cleared the strand it was meant to race against), not the fix being absent.
2026-08-31 23:17:50 +07:00
Dai Ha 97f6c33a45 #137: don't guess when a target has more than one open async task
CI / contract (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 1m17s
The CB-205-recovery fix in #205 assumed a target has at most one open
async task, with no guard. Fix three consequences:

- reply(): askAnsweredAsyncTask -> askAnsweredAsyncTasks (List). Exactly
  one candidate completes it (unchanged). Zero falls to the inbox
  (unchanged). More than one now ALSO falls to the inbox instead of
  picking an arbitrary ConcurrentHashMap iteration order, and logs a
  WARN naming the target and every candidate ticket.

- abandon(): a stranded reply now settles at most one matching task —
  the oldest by Task#createdNanos (a new field, the tiebreaker). Every
  other matching task keeps WORKER_FAILED, same as today.

- abandon(): if the chosen recovery task's complete() loses a race
  (another path resolved it first), the drained reply is republished
  to the inbox instead of being silently dropped.

Single-task behavior is unchanged; only the ambiguous case changes.
2026-08-31 22:46:23 +07:00
Dai Ha 735c837604 #209: resolve agentSessionId lazily via the retained PeerHandle
CI / contract (push) Successful in 48s
CI / build (push) Successful in 2m18s
SessionManager called handle.agentSessionId() once at spawn and froze it in the
immutable MemberSession. For opencode that value is always null - the session
row does not exist yet when the pane is created - so fleet_list never reported
an agentSessionId and fleet_spawn{resumeSessionId} was unusable for that
backend. The handle's javadoc said 'the caller re-calls later'; no caller did,
and SessionManager did not even retain the handle.

Retain the PeerHandle per pane and re-resolve while the stored id is still
null, sticky once found, CAS-swapped into the registry.

roster() stays non-resolving - it is the roster supplier for LeadHeartbeatLoop
and FleetHealthMonitor, and resolving there would open opencode's 841MB SQLite
database on every tick, for every unresolved member, forever. rosterResolved()
carries the resolve and is used only by fleet_list and the REST roster, the two
surfaces that report the id. get(paneId) and release() resolve too, both
caller-driven.

A test pins the split: the plain roster() must never call agentSessionId()
again.
2026-08-31 22:21:16 +07:00
Dai Ha 457dc0330d #185 stage 2: the credential gap detector must not report on the wrong environment
CI / contract (push) Successful in 55s
CI / build (push) Successful in 1m59s
Under memberHerdrSocket: member panes run as a different OS user, so
hostEnvNames (fleetd's own environment) no longer describes what a member
inherits. logCredentialGap now reports "unknown, not clean" in that mode
instead of its usual UNBLOCKED/blanked conclusions, once per launcher, naming
the config key and scoping its count to fleetd's own process.

Byte-identical when memberHerdrSocket: is absent, pinned by a test.
2026-08-31 22:17:40 +07:00
Dai Ha a1a9015217 #199: /members returns rows under "members" 2026-08-31 22:17:35 +07:00
Dai Ha 5a811a3695 #199: GET /members returns its rows under "members", with "workers" kept as an alias
The endpoint became /members in the CB-634 rename but the body key stayed
"workers", so a caller that read "members" saw an empty fleet and reported
no members at all.

Emit the canonical "members" key. Keep "workers" as a deprecated alias so an
existing REST consumer keeps working - the out-of-band path a lead falls back
to when its MCP mount drops reads this endpoint.

The test now pins both keys and asserts they carry the same rows, so the alias
cannot silently drift. Watched failing without the fix:
"GET /members must return its rows under \"members\" ==> expected: not <null>"
2026-08-31 22:17:22 +07:00
Dai Ha 1fdaa74eb3 fleetd #209 follow-up: keep roster() off the resolve path
CI / build (pull_request) Successful in 1m20s
CI / contract (pull_request) Successful in 1m21s
roster() is the supplier for LeadHeartbeatLoop and FleetHealthMonitor
(both timer-driven) and for placement/exhaustion checks and the
metrics scrape — none of which read agentSessionId. Resolving there
meant every tick could open a lazy-resolving adapter's (opencode's)
on-disk session database once per member whose id was still unknown,
with no bound: a member whose id never appears would pay that cost for
the life of the process.

roster() goes back to its pre-#209 behavior (no resolve, no I/O). A
new rosterResolved() carries the resolve logic, and is used only by
the two surfaces that actually report agentSessionId to a caller:
fleet_list (FleetMcp.listFleet) and the REST roster
(FleetApp.listMembers). fleet_whoami's roster().stream() at
FleetMcp.java:768 does not surface the field, so it stays on the plain
roster(). get(paneId) (fleet_status) and the release() resolve are
caller-driven, not timers, and are unchanged.

Retargeted the roster-facing tests from #209 at rosterResolved(), and
added plainRosterDoesNotResolveAgentSessionId, which pins the split by
asserting the handle's agentSessionId() is not called again by
roster().
2026-08-31 22:15:16 +07:00
Dai Ha 7d4a4339c2 Merge branch 'worker/cb-137-ask-ticket-e7760c-2' of https://git.ltms.dev/fleet/fleetd into worker/cb137-ambiguous-task-4df3d8-4 2026-08-31 22:13:37 +07:00
Dai Ha 847e8bd3fa #185 stage 2: stop the credential-gap detector reporting on the wrong environment
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 1m39s
When memberHerdrSocket: is configured, member panes run under a different OS
user than fleetd's own process, so hostEnvNames (fleetd's own environment)
no longer describes what a member pane inherits. logCredentialGap now checks
for that config key and, when set, logs a single WARN saying the gap is
UNKNOWN (not clean) and names the key, instead of printing the "inherits
them UNBLOCKED" / "the scrub blanks them" conclusions as fact. Behaviour is
byte-identical when memberHerdrSocket is absent (the default and only mode
this host runs).
2026-08-31 22:12:56 +07:00
Dai Ha f9fb387427 fleetd #209: re-poll agentSessionId against the retained PeerHandle
CI / contract (pull_request) Successful in 45s
CI / build (pull_request) Successful in 1m40s
SessionManager used to call PeerHandle.agentSessionId() exactly once at
spawn and freeze the answer into the immutable MemberSession. For
opencode that call always came back null, because opencode has not
written its on-disk session row yet when the pane is created, and no
caller ever re-asked the handle — it went out of scope at the end of
the spawn method. fleet_list therefore never reported agentSessionId
for an opencode member, and resumeSessionId was unusable for it.

Retain each spawn's PeerHandle in SessionManager, keyed by paneId, and
re-resolve a still-null agentSessionId against it from roster(), get(),
and release() (so a released member's detail also carries a
late-resolved id). Resolution is bounded: only sessions with a still-
null id do any work, a resolved id is never looked up again, and a
throwing handle degrades to "unresolved" rather than breaking the
caller. MemberSession gains a withAgentSessionId wither in the same
style as withState/withActivity.
2026-08-31 22:04:20 +07:00
Dai Ha a55079afbd #185: opt-in worktreeGroup, so a member running as another OS user can write its worktree
CI / contract (push) Successful in 50s
CI / build (push) Successful in 1m39s
Stage 3 of #185. A provisioned worktree and the repo's git store are made
group-writable when worktreeGroup names an OS group; absent, nothing changes.
The share pass runs after overlayParity, not inside add(), because
overlayParity copies more files in after add() returns.

This isolates credentials, not the repository: a member in the group can
still write the operator's git objects and refs.
2026-08-31 21:47:03 +07:00
Dai Ha e18ad4723b Merge remote-tracking branch 'refs/remotes/origin/cb206' 2026-08-31 21:47:03 +07:00
Dai Ha 6de8ac8972 #206: read opencode session ids from opencode.db, and pin the read-only open
opencode migrated its session store from a JSON file tree to SQLite in
January. OpenCodeSessionDiscovery still scanned the frozen tree, so it
returned null for every member: agentSessionId was never known and
resumeSessionId silently did nothing for every opencode profile, through
57 member spawns, with nothing logging that the search found nothing.
2026-08-31 21:46:32 +07:00
Dai Ha a052975420 #206: pin the read-only open with a test that actually fails without it
CI / build (pull_request) Successful in 1m18s
CI / contract (pull_request) Successful in 1m44s
setReadOnly(true) is the whole thing keeping fleetd out of the operator's
live 841MB opencode.db, and no test failed when it was removed.

The obvious test does not work. Making the database file unwritable and
checking the read still succeeds passes either way, because SQLite silently
downgrades a read-write open of an unwritable file to read-only. I wrote that
test, watched it pass with the flag removed, and threw it away.

What works: extract a package-private openReadOnly(), then ask that connection
to INSERT and require the refusal. Watched red with the flag removed, green
with it restored.

Also switches the test's INSERT helper to a PreparedStatement -- hand-escaped
SQL in a test is a pattern that gets copied into main code.
2026-08-31 21:46:07 +07:00
Dai Ha 6d82ca95a4 #185: skip absent git paths, and resolve the git dir instead of assuming .git
CI / contract (pull_request) Successful in 56s
CI / build (pull_request) Successful in 1m38s
Two defects in the stage-3 share pass, both of which would have failed EVERY
provisioning spawn once worktreeGroup was set, not only the two-user case.

.git/logs was handed to chgrp unguarded while packed-refs was guarded. It does
not exist with core.logAllRefUpdates=false, or before the first ref update, and
chgrp on a missing path exits non-zero -- surfacing as a WorktreeException that
blames a group which is in fact fine. Every path is now skipped when absent.

repoRoot + "/.git" was hardcoded. That is a FILE, not a directory, when the
checkout is itself a linked worktree -- the very thing this class creates for
every member. It now asks git: rev-parse --git-common-dir, resolved against
repoRoot because git answers relatively for an ordinary checkout.

Both new tests were watched failing with the fix removed before being kept.
2026-08-31 21:34:49 +07:00
Dai Ha 8067ee4ec4 fleetd #185: opt-in worktreeGroup config for group-shared worktrees
CI / contract (pull_request) Successful in 1m7s
CI / build (pull_request) Successful in 1m45s
Adds worktreeGroup (top-level FleetConfig key), Worktrees.shareWithGroup
(GitWorktrees impl: git config core.sharedRepository group + one-time
chgrp/chmod g+rwX/setgid fix-up over the worktree, .git/objects, refs,
logs, worktrees, and packed-refs when present), and wires SessionManager
to call it AFTER overlayParity so overlay files are covered too. Off by
default (byte-identical behaviour when unset). Documents the
credentials-not-repository caveat in the javadoc and example config.
2026-08-31 15:53:01 +07:00
Dai Ha 3743789e8d fleetd #206: read opencode session ids from opencode.db (SQLite), not the frozen JSON tree
CI / contract (pull_request) Successful in 1m5s
CI / build (pull_request) Successful in 1m44s
opencode migrated its session store to SQLite in January 2026; the JSON tree under
storage/session/<projectID>/ses_*.json stopped being written, so OpenCodeSessionDiscovery
returned null for every member forever, and fleet_spawn{resumeSessionId} was unreachable.

- Add org.xerial:sqlite-jdbc 3.53.4.0, opened read-only (SQLiteConfig.setReadOnly), so it
  never disturbs a live opencode process writing the WAL-mode database.
- Rewrite sessionIdForDirectory to run a parameterized SELECT ... WHERE directory = ?
  ORDER BY time_updated DESC LIMIT 1 against the session table. Still never throws: a
  missing database, a locked/corrupt one, or no matching row all return null.
- Log the silence that let this go unnoticed: WARN once per instance when opencode.db
  itself is missing (the layout moved again), DEBUG when it exists but no row matches
  yet (the normal interim answer right after a spawn).
- Replace the JSON-fixture tests with a synthetic-SQLite-db fixture; delete the tests
  that only proved the old JSON scan worked.
2026-08-31 15:52:06 +07:00
Dai Ha 388aba7632 #137: an answered turn's reply completes its own ticket, instead of a false failure
CI / contract (push) Successful in 1m12s
CI / build (push) Successful in 1m17s
A wait:false delegation whose worker used fleet_ask ended with fleet_poll{ticket}
reporting 'the worker session was released before it replied' -- naming a
worktree, a branch and a snapshot commit, so it read as lost work. The worker had
in fact replied in full.

The ticket guessed the ask rendezvous detached the turn. The real cause is
narrower: answer() (behind fleet_send{turnId}) waits only for the lead's own
bounded MCP call window. A resumed turn doing real work -- edits, a build, a
push, a PR -- routinely outlives it. On timeout answer()'s finally closed the
waiter, so the worker's later fleet_reply found none and fell to the session
inbox, leaving the ticket's future unresolved until fleet_stop forced it FAILED.

reply() now looks for the async task parked on this exact answered turn and
completes it with the real reply. That is safe against completing the wrong
ticket: answer() calls clearAsyncQuestion(turnId, false), so the task keeps its
turnId and stays in asyncTasksByTurn, and hasAsyncQuestion therefore still makes
send() return BUSY for a second async send to that target. At most one candidate
task can exist per target.

abandon() keeps an independent check: if a reply was stranded, a released session
reports REPLIED with that text rather than a failure -- so the recovery hint that
implies lost work never prints once a reply exists.

Verified before merging: build green unpiped, and both new tests drive the full
delegation path (send async, ask, answer, reply, poll the ticket) rather than
handing a Reply to a sink, which is the trap this ticket called out.

Co-authored-by: fleetd worker <worker@ltms.dev>
2026-08-31 14:24:54 +07:00
Dai Ha ad587eafa3 #172: keep the broker URI, password and all, out of every member pane
CI / contract (push) Successful in 1m10s
CI / build (push) Successful in 1m15s
broker.uriEnv names an environment variable holding amqp://user:password@host,
and it was reaching every member. Its name is not credential-shaped -- no TOKEN,
KEY or SECRET in it -- so every name-pattern heuristic missed it, and it sat on
neither credential list.

fleetd already knows the name: the operator wrote it in broker.uriEnv. So derive
the exclusion from the config rather than hoping an operator also remembers to
deny it. coordinator.uriEnv has the same shape and is excluded too; on this host
both resolve to the same variable.

Excluded even when the operator lists the name under memberCredentials.allow:,
following the SSH_AUTH_SOCK precedent. There is no override, because a member has
no legitimate use for the broker password.

Reviewer finding, recorded rather than overstated: this is only a hard guarantee
under policy: allow-list, where the ZDOTDIR scrub runs after the pane's shell has
sourced the operator's chain. Under deny-list the name is removed from the
pre-shell env only, and a login shell re-exports it. That is deny-list's existing
weakness rather than a regression here, but the javadoc now says so plainly
instead of implying a guarantee that path cannot give.

Co-authored-by: fleetd worker <worker@ltms.dev>
2026-08-31 14:21:48 +07:00
Dai Ha 08ce9aef11 #161: resolve a pane by process ancestry, closing a worker->primary escalation
CI / contract (push) Successful in 50s
CI / build (push) Successful in 1m52s
PaneLocator's javadoc always claimed it found 'the agent pane whose process
tree contains' a pid. It did not: paneOwnsPid matched only the pane's shell_pid
and its foreground_processes. A process a member spawned -- python3, curl, any
helper opening its own connection to 127.0.0.1:8765 -- matched no pane, so
CallerResolver fell through to loopback-trust and resolved it as the PRIMARY.
A member escalated to lead by shelling out.

terminalForPid now builds the caller's ancestor set once (bounded at 32
generations, with a cycle guard) and matches any ancestor against a pane's pids.
The set is reused across both herdr clients on the CB-185 two-daemon path.

This only ever ADDS matches, which is the safe direction: the failure mode of
the fix is a member correctly restricted, while the failure mode of the bug is a
member acting as the lead. The no-match case still returns null, so the lead --
which maps to a pane named by leaders: -- still resolves as primary.

Ancestry is walked through a new ParentResolver seam so the tests drive it from
a fake pid->parent map rather than spawning real processes.

Co-authored-by: fleetd worker <worker@ltms.dev>
2026-08-31 14:15:39 +07:00
Dai Ha ee5f8b932b #137: complete an async ticket's own reply after answer() times out
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 1m44s
fleet_send{turnId} (MessageService.answer) blocks the primary only for its
own bounded MCP-call window (25s default, 120s max) — far shorter than a
worker's resumed turn can genuinely take. When that window expires, answer()
closes its rendezvous waiter, so the worker's eventual fleet_reply has no
live waiter to resolve and falls back to the session inbox. The async
ticket's future was never completed by that path, so fleet_poll{ticket}
stayed PENDING until fleet_stop's abandon() forced it FAILED with a
misleading "the worker session was released before it replied" reason,
even though the reply had genuinely arrived.

- MessageService.reply(): before falling to the inbox, look for the async
  task this exact turn belongs to (already answered — question cleared,
  turnId still stamped — but not yet resolved) and complete it directly
  with the real reply, so fleet_poll{ticket} returns it.
- MessageService.abandon(): defense in depth, independent of the above —
  never write a false WORKER_FAILED once a reply reached the inbox for
  this target; recover and use its real content instead.
- Two new tests drive the full delegation path (async send -> ask ->
  answer with a short timeout -> reply -> poll/abandon), not a reply sink
  directly; both fail with the fix disabled and pass with it restored.
2026-08-31 14:12:23 +07:00
Dai Ha 4ac688b6d9 #164: classify a backend-error scrape as WORKER_FAILED, carrying the whole pane
CI / contract (push) Successful in 48s
CI / build (push) Successful in 1m22s
main already shipped the core of #164 in 3bfa828: the MIN_TURN_NANOS floor and
the hard fail on an empty or unreadable scrape. This adds the one case that was
still resolving as a success -- a scrape that reads cleanly but whose content is
the backend's own rejection (e.g. "API Error: 400 invalid request body").

The BACKEND_ERROR pattern is deliberately narrow. A growing list of ad-hoc error
strings rots as backends change their wording, and broader backend-error
surfacing is #164 point 3.

Because the pattern is a heuristic, it also matches a member that forgot
fleet_reply while reporting *about* a backend error. So the failure reason
carries the whole pane tail, not just the matched line: a genuine backend error
reads as before, and a false positive keeps its report instead of losing it.

Checked before merging: 3bfa828 is an ancestor of main; the branch was current
with main; mvn clean install green unpiped (1037 tests, 0 [ERROR] lines); and
each of the 5 new tests fails with the fix commented out.

Co-authored-by: fleetd worker <worker@ltms.dev>
2026-08-31 10:56:40 +07:00
ltms a814d1ef00 #197: measure the async ticket TTL from completion, not from creation
CI / build (push) Successful in 1m15s
CI / contract (push) Successful in 1m17s
Lead-authored and lead-verified: full mvn clean install green at 1032 tests, and the new test proved by restoring the old comparison and watching it fail.
2026-08-31 05:10:46 +02:00
Dai Ha ea98856130 #197: measure the async ticket TTL from completion, not from creation
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 1m32s
pruneTerminalTickets compared the cutoff against createdNanos, so the real
window to collect a reply was "TTL minus however long the task ran". A
delegation that ran longer than the 10-minute TTL was already past the cutoff
the moment it finished, so the next prune destroyed its reply.

That is the normal case here, not an edge case. Real delegated work runs well
past ten minutes. Three workers in one session did, and two of their complete
reports were lost. The reply lives only in Task.future, so pruning it discards
the worker's whole report, and fleet_poll{target} returns [] rather than
holding it — there is no fallback.

Task now stamps completedNanos from a whenComplete hook registered in its
constructor, so every completion path stamps it (a reply, the completion
fallback, a timeout, a failure, an abandon on teardown) without each one having
to remember to. The stamp is a boxed Long, not a long with a sentinel:
System.nanoTime may return any value, so no number can mean "not stamped yet".
A task that is done but not yet stamped is left for the next sweep.

createdNanos had no other reader and is removed.

The TTL still bounds tasks — an uncollected finished ticket is still evicted
once the TTL passes since it finished. Both halves are pinned by a test, and
the first one was proved by restoring the old comparison and watching it fail.

Tests run: 1032, Failures: 0, Errors: 0, Skipped: 0
2026-08-31 10:10:16 +07:00
ltms a49e96835a CB-189: cover every remote, both URLs, and any non-SSH scheme in the credential check
CI / contract (push) Successful in 44s
CI / build (push) Successful in 1m11s
Lead-verified: merged onto current main (which already carries #194 and #196), full `mvn clean install` green at 1030 tests. Confirmed no plain `exec` call that reads a remote URL remains.

Review round 2 closed the half-fix: execRedacted was added but applied only to the new calls, leaving the three pre-existing URL readers (lines 143, 171, 334) still copying stdout into exception messages — the exact hole CB-189 named.

Kept the check before the origin strip, on the worker's reasoning: the credential really is in the config at that moment, so WARN-found followed by INFO-fixed is the full audit trail, whereas moving it after would silence the origin case entirely.
2026-08-31 04:32:14 +02:00
ltms 63c19dcba7 CB-185: fix two blockers to switching on memberHerdrSocket
CI / build (push) Successful in 1m31s
CI / contract (push) Successful in 1m19s
Lead-verified: merged with #194 onto an integration branch off main, full `mvn clean install` green at 1025 tests.

Review round 2 fixed the stale ambiguous-pane message and, more importantly, a real abort: probeOwner called list() unguarded, so one unreachable daemon made panes on a different healthy daemon un-stoppable too — the same bug blocker 1 exists to fix, through a new door. Worker proved it by removing the guard and quoting the failure.
2026-08-31 04:30:57 +02:00
ltms 2823349c8e CB-192: fix false credential-gap WARN under allow-list+zsh, split its log guard
CI / contract (push) Successful in 44s
CI / build (push) Successful in 1m11s
Lead-verified: merged with #196 onto an integration branch off main, full `mvn clean install` green at 1025 tests. Deny-by-default WARN confirmed byte-identical to main.

Review round 2 fixed a defect I found in round 1: the new INFO asserted that gap names were not on the derived allow-list without ever checking, so a name derived from a profile's tokenEnv/gitTokenEnv/env: would be reported as safe while the member actually inherited it. Now split with MemberEnvAllowList.keeps — the same predicate the generated scrub evaluates.
2026-08-31 04:30:48 +02:00
Dai Ha 6fc301d62c CB-189 review fix: redact the three pre-existing URL-reading exec calls
CI / contract (pull_request) Successful in 1m11s
CI / build (pull_request) Successful in 1m42s
Review found that execRedacted was applied only to the new remote-enumeration code
and left three pre-existing calls reading remote.origin.url through the plain,
unredacted exec: removeUserInfoFromHttpsOrigin, requireCredentialFreeHttpsOrigin, and
configureHttpsUrlRewriteForSshOrigin. A non-zero exit or timeout on any of those could
still have copied the credentialed URL into a WorktreeException message. Switches all
three to execRedacted; the set-url write in removeUserInfoFromHttpsOrigin is left on
plain exec with a comment explaining why (it writes the already-stripped URL, not a
read).

Widens the shared exec(Map, boolean, String...) overload to package-private, the same
test-seam pattern already used by the afterWorktreeAdded constructor parameter, and
adds a test that drives it directly with a synthetic failing command whose stdout
carries a marker (passed via env, not argv, so the always-printed command line can't
carry it) and asserts the marker never reaches the exception message.
2026-08-31 09:29:45 +07:00
Dai Ha bc99d64786 CB-185: fix ambiguous-pane message and unreachable-daemon abort in probeOwner (review)
CI / build (pull_request) Successful in 1m6s
CI / contract (pull_request) Successful in 1m5s
Lead review of PR #196 found two issues in CompositePeerLauncher.probeOwner:

1. The "more than one daemon claims this pane" throw kept the old
   pre-fix message ("no owning herdr daemon was recorded"), which was
   only true of the code it replaced. Reworded to say what actually
   happened: N configured herdr daemons report this pane, so it is
   genuinely ambiguous. Updated the one test pinning the old string.

2. probeOwner let list() propagate straight out of the probe loop, so
   one unreachable daemon aborted the whole probe and made a pane on a
   DIFFERENT, healthy daemon un-stoppable too — resurrecting the exact
   bug blocker 1 fixes. Now catches HerdrException per daemon, logs the
   exception class only, and treats that daemon as not knowing the pane
   so probing continues. New test proves this: verified it fails with
   the try/catch removed (HerdrException propagates and the stop that
   should succeed via the healthy daemon throws instead), then restored.

Full mvn clean install: 1019 tests, 0 failures, 0 errors.
2026-08-31 09:28:29 +07:00
Dai Ha 615af4ed0a CB-192 review fix: split the allow-list gap by what the scrub actually keeps
CI / build (pull_request) Successful in 1m5s
CI / contract (pull_request) Successful in 1m20s
Lead review on PR #194 found that the allow-list INFO wording claimed the
whole gap ("credential-shaped names on neither known: nor allow:") is blanked
by the scrub, without checking that against effectiveAllowed. effectiveAllowed
is a SUPERSET of known+allow — MemberEnvAllowList.derive also unions in every
profile's gitTokenEnv/gitHostEnv/tokenEnv/env: keys, and derivedAllowedNames
further unions in the spawn's own env keys — so a gap name can still be kept
by the derived list (e.g. a profile's tokenEnv names it) and reach the member
unblocked while the INFO said "no member pane keeps them". That inversion is
exactly what #192 exists to remove.

logCredentialGap now splits the gap with MemberEnvAllowList.keeps (the same
predicate the generated scrub itself evaluates, so this cannot drift from
what the scrub does): names it keeps get a WARN, guarded by the same
unprotectedGapLogged flag as the deny-by-default case (same severity — a name
reaching a member unprotected is equally serious either way); names it
blanks keep the existing INFO, guarded by allowListGapLogged. The
deny-by-default WARN text and the non-zsh fallback are untouched.

Added allowListWarnsWhenTheDerivedAllowListKeepsAnUncoveredName and
allowListSplitsAMixedGapBetweenTheWarnAndTheInfo to ClaudeCodeLauncherTest.
2026-08-31 09:28:10 +07:00
Dai Ha 045d229728 CB-185: fix two blockers to switching on memberHerdrSocket (#185)
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 1m39s
1. CompositePeerLauncher.stop() was permanently un-stoppable for any
   member that survived a daemon restart, because spawnedBy is in-memory
   only. On a cache miss with more than one configured herdr daemon, probe
   each distinct daemon's agent.list() for the pane instead of refusing
   outright: exactly one owner routes and caches; zero owners is treated
   as already-stopped (a no-op, matching the tolerance HerdrPeerLauncher
   already gives an already-gone pane); more than one owner is the
   genuine per-daemon-pane-id ambiguity and still throws.

2. FleetApp#healthz always reported the LEAD daemon's herdr version/
   protocol even when a second (member) daemon was configured, so a
   member-daemon protocol mismatch was invisible behind a green
   /healthz while every spawn silently failed. Added a separate "member"
   key alongside the unchanged "herdr" key, and a "protocolMismatch"
   flag when the two differ. Verified scripts/redeploy-fleetd.sh and
   scripts/rename-checkout.sh only check the HTTP status code and print
   the body verbatim — neither parses a specific field — so adding a key
   is safe.

Both fixes are covered by tests written to fail without the fix
(verified by reverting each fix and watching the new tests fail, then
restoring). Full `mvn clean install`: 1018 tests, 0 failures, 0 errors.
2026-08-31 09:20:14 +07:00
Dai Ha d1fd5700f5 CB-189: cover every remote, both URLs, and any non-SSH scheme in the credential check
CI / build (pull_request) Successful in 1m7s
CI / contract (pull_request) Successful in 1m7s
GitWorktrees only ever inspected origin's HTTPS fetch URL for embedded credentials. A
credential on any other remote, on a pushurl, or on a plain http:// URL passed through
unreported. Adds an additive, reporting-only check that enumerates every remote and both
its fetch and push URLs, flagging non-empty user-info on any non-SSH-family scheme.

The existing origin/https strip-and-refuse behaviour is untouched. The new check is
wrapped so it can never abort a provision, and on failure logs only the exception's
class, never its message, since the enumerating `git remote` call is not redacted.
Also adds execRedacted, an exec variant that never copies captured stdout into a
WorktreeException message, for commands whose stdout may itself be a credentialed URL.
2026-08-31 09:17:54 +07:00
Dai Ha 2d55b0b9a5 #168: correct the audit's MISSING claim — the feature is documented, its index row was not
CI / contract (push) Successful in 41s
CI / build (push) Successful in 1m42s
The memberHerdrSocket section exists at 11-Features.md:2174; what was absent was
its row in the index table. My omission when I added the section. Wiki fixed at
b24965c.
2026-08-31 09:17:41 +07:00
Dai Ha d89ae94a2e CB-192: fix false credential-gap WARN under allow-list+zsh, split its log guard
CI / contract (pull_request) Successful in 1m5s
CI / build (pull_request) Successful in 1m7s
logCredentialGap(creds) always emitted the WARN wording ("every member pane
inherits them UNBLOCKED"), even under memberCredentials.policy: allow-list on
a zsh login shell, where the generated ZDOTDIR scrub genuinely blanks the
name. The line reported the control working as though it were a hole.

Pass an effectiveAllowed set instead: null keeps the WARN (deny-by-default,
and the allow-list non-zsh fallback, where nothing is ever scrubbed); the
derived allow-list set (only reachable after applyEnvironmentAllowListPolicy's
own zsh gate) selects a new INFO wording that says the scrub will blank the
name instead of claiming it is inherited unblocked.

Also split the single credentialGapLogged AtomicBoolean into two guards
(unprotectedGapLogged / allowListGapLogged) — one per report kind. Since
memberCredentials is a live, re-read-per-spawn supplier, a shared flag let a
harmless allow-list INFO on one spawn permanently suppress a later spawn's
real deny-by-default WARN after a policy reload.

Fixes gitea #192.
2026-08-31 09:17:20 +07:00
ltms e6193c4098 Merge pull request '#168: audit current wiki snapshot' (#193) from worker/cb-168-wiki-audit-3ef3d1-3 into main
CI / build (push) Successful in 1m6s
CI / contract (push) Successful in 1m5s
2026-08-31 04:17:07 +02:00
Dai Ha b66f0677ed #168: audit current wiki snapshot
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m37s
2026-08-31 09:14:22 +07:00
ltms 23ada1981e CB-185: route members to a separate herdr daemon (#186)
CI / build (push) Successful in 1m6s
CI / contract (push) Successful in 10m30s
2026-08-29 01:24:31 +02:00
ltms a22480c117 CB-185: route PaneLocator, StatusRefiner and FleetApp to the right herdr daemon (#188)
CI / build (pull_request) Successful in 1m3s
CI / contract (pull_request) Successful in 1m7s
2026-08-29 01:24:25 +02:00
Dai Ha 24f404f989 CB-185: fix three connection-identity/status/health gaps a second herdr daemon exposes
memberHerdrSocket splits lead operations from member operations onto two herdr
daemons. Three seams still assumed one shared daemon and broke silently when the
two clients differ (all three collapse to today's behaviour when they are the
same object):

1. ConnectionIdentity's PaneLocator was pinned to the member daemon only, so a
   lead's own MCP connection (which lives on the LEAD daemon) resolved to
   terminal == null, breaking fleet_reply/fleet_ask/fleet_whoami for a lead.
   PaneLocator now searches the lead client first, then the member client.

2. StatusPoller's StatusRefiner was pinned to the member daemon, so refining an
   UNKNOWN status for a lead target read the wrong daemon's pane content and
   never left UNKNOWN, wedging status-gated delivery to that lead forever.
   StatusRefiner gained a refine(target, raw, control) overload and the poller
   now refines through the same AgentControl the raw status was sampled from.

3. FleetApp was constructed with the raw lead-only herdr client, so /healthz
   stayed green while the member daemon was down (every spawn then fails
   invisibly) and GET /sessions silently dropped every member workspace.
   FleetApp now takes both clients: healthz requires both to answer, sessions
   merges workspaces from both.

Each fix has a test proven to fail without it (verified by reverting the
production change and re-running): FleetdConnectionIdentityConstructionTest /
FleetdFleetAppConstructionTest assert the actual Fleetd.java wiring (the same
technique as FleetdHerdrControlConstructionTest); StatusPollerRoutingTest and
the new PaneLocatorTest/FleetAppTwoDaemonTest cases exercise the real
production classes end to end rather than a hand-built object graph.
2026-08-29 06:20:32 +07:00
ltms a237fbff9d #185: refuse an unowned paneId when more than one herdr daemon could own it (#187)
CI / build (push) Successful in 1m4s
CI / contract (push) Successful in 1m17s
2026-08-29 01:10:21 +02:00
Dai Ha 31d5516991 #185 review: keep the stop owner on failure, key list() by daemon
CI / contract (pull_request) Successful in 43s
CI / build (pull_request) Successful in 1m30s
Three fixes on top of the pane-id PR, from my own read and the reviewer's:

- stop() removed the spawnedBy record BEFORE the delegate accepted the stop. A
  delegate that threw left the pane alive with its owner forgotten, so the retry
  fell into the ambiguous branch and refused the id for good. Remove after.
- list() deduplicated on the raw pane id. Pane ids are per-daemon counters, so
  two daemons can each hold w1:p1 on different panes, and one of the two real
  agents was silently dropped from fleet_list and every view built on it. The
  key is now (owning daemon, pane id). Delegates sharing one daemon still
  collapse, which is what the dedupe was for.
- The class javadoc still stated the single-herdr-connection premise as fact,
  next to the bullet this PR had just corrected for stop(). Fixed there too.

Also drops a redundantly qualified java.util.Collections.

Tests: 993 run, 0 failures, BUILD SUCCESS.
2026-08-29 06:09:43 +07:00
Ha Trong Dai 17af61e8dd CB-185: route message status by target
CI / contract (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 1m40s
2026-08-28 09:45:39 +07:00
Ha Trong Dai 6af87b6ad6 CB-185: share routed herdr controls
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 1m40s
2026-08-28 09:42:45 +07:00
Ha Trong Dai 5ba05d0bdb #185: count herdr owners for stop fallback
CI / contract (pull_request) Successful in 43s
CI / build (pull_request) Successful in 1m8s
2026-08-28 09:42:24 +07:00
Ha Trong Dai fc655e78c2 CB-185: route members to separate herdr
CI / contract (pull_request) Successful in 37s
CI / build (pull_request) Successful in 1m28s
2026-08-28 09:37:44 +07:00
Ha Trong Dai 25726a5ae7 #185: reject ambiguous unowned pane ids 2026-08-28 09:36:39 +07:00
Dai Ha 11c3ff67b6 fleets-status: fleet01 IS ssh-reachable; correct the 'denied' claim
CI / contract (push) Successful in 46s
CI / build (push) Successful in 1m12s
The skill said SSH to fleet01 is denied, so every report wrote 'not
reachable' for that fleet's daemon facts. That is true only for the user
dai.ha. The host alias fleet01 maps to user ltms and key auth works.

Checked 2026-08-28 while measuring #185: ssh fleet01 connects, and ltms
has passwordless sudo there. So fleet01's PID, uptime, jar and /healthz
can be reported over SSH even though its REST port is unreachable.
2026-08-28 09:06:54 +07:00
Dai Ha d867c87100 #184: correct the false ssh-agent premise in the URL-rewrite javadoc
CI / contract (push) Successful in 46s
CI / build (push) Successful in 1m37s
The javadoc said a member cannot authenticate at all once memberCredentials
blocks SSH_AUTH_SOCK, "there is no private key file on this host, only an
ssh-agent socket". That is wrong, and it was written after looking only in
~/.ssh, which holds nothing but Include lines.

Measured: ssh -G git.ltms.dev resolves an IdentityFile under the shared-env
directory. That file exists, is readable by this user, and has no passphrase.
A live member with SSH_AUTH_SOCK blanked pushed to the forge over SSH.

The rewrite itself is unchanged and still worth having. Only its stated reason
was wrong: it routes a member through its own scoped token instead of the
operator's ssh identity, which is what makes a member's pushes attributable
and revocable. It is not what stands between a member and the forge.
2026-08-28 06:35:46 +07:00
Dai Ha b0c4cedfab #157: redact remote-URL user-info before it reaches a log
CI / contract (push) Successful in 41s
CI / build (push) Successful in 1m25s
The four log lines added with the worktree HTTPS rewrite echoed the origin
URL verbatim, and one of them echoed the ssh:// authority, which carries
user-info. An ssh authority is normally just git@, so in practice this
changes nothing -- but a remote URL is not obviously a credential channel,
and that is precisely why one has leaked here three times (#157, #182).

Redact at the log call, not after it surprises someone.
2026-08-28 06:24:08 +07:00
ltms 85417d5215 #157: rewrite an SSH origin to HTTPS inside the provisioned worktree only
CI / contract (push) Successful in 46s
CI / build (push) Successful in 1m37s
Git never consults a credential.helper for an SSH transport, so #177's helper was inert on this repo — whose origin is ssh://. Once allow-list policy blocks SSH_AUTH_SOCK, a member on an SSH origin cannot authenticate at all: there is no private key file on this host, only an agent socket.

A worktree-scoped `url.<https>.insteadOf <ssh>` gives the member HTTPS for fetch and push while the primary checkout keeps SSH untouched. Host and port are parsed from the origin, never hardcoded — a test with a synthetic host proves it. The scp-like shorthand is left alone deliberately, since its host:path split is defined by ssh_config aliases rather than URI syntax.

Verified by the lead in an independent worktree: Tests run: 986, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 01:22:57 +02:00
Dai Ha 4accc746bd #157: rewrite SSH origin to HTTPS in the worktree so the credential helper is reachable
CI / build (pull_request) Successful in 1m10s
CI / contract (pull_request) Successful in 1m16s
2026-08-28 06:20:10 +07:00
ltms 21c4c8cbef #157: keep the forge token out of git config, via an environment credential helper
CI / build (push) Successful in 1m4s
CI / contract (push) Successful in 47s
The member credential scrub removes environment variables. It cannot remove a token written into git config inside the repo the member works in, so `git remote -v` handed a member a credential it was deliberately not given.

Provisioning now strips HTTPS user info from the origin before `git worktree add`, refuses the worktree if user info survives, and configures a per-worktree credential helper that reads WORKER_GITEA_TOKEN at call time. Nothing is persisted.

The helper emits BOTH username and password, and resets the inherited helper list first. An earlier revision emitted only `username=`, which made git fall through to the next helper — on a Mac that is osxkeychain, so a member would have authenticated with the operator's stored credential while every test passed and `git remote -v` looked clean. See #182.

`worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper` plants a synthetic operator helper in an isolated global config and proves the worktree helper wins. The worker confirmed it fails when the reset is removed. All credential tests pin GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM so they can neither read nor write real credentials.

Verified by the lead in an independent worktree: Tests run: 973, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 01:10:38 +02:00
ltms 430f5b0dae #164: never resolve a send with an empty scrape or a sub-floor turn
CI / contract (push) Successful in 53s
CI / build (push) Successful in 1m27s
A turn that dies on a backend error produces the same working -> idle transition as a real one, just faster and with nothing on screen. The resolver accepted that as a completed turn and handed the caller HTTP 200 with an empty reply, so a lost turn and a successful empty answer were indistinguishable.

Now: an empty or unreadable scrape fails, naming the member; and a BUSY -> DONE inside MIN_TURN_NANOS (2s) fails as a crash signature.

One existing test encoded the bug — it asserted a failed scrape resolved as a success carrying "" — and has been inverted rather than worked around.

Verified by the lead in an independent worktree: Tests run: 973, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 01:10:27 +02:00
ltms 42731833d0 CB-633: union memberCredentials.allow into the member env allow-list
CI / contract (push) Successful in 42s
CI / build (push) Successful in 1m38s
`policy: allow-list` silently ignored every name an operator wrote under `allow:` unless a profile happened to carry it too, so turning the policy on would have blanked credentials working members depend on. Derivation now unions the operator's list.

`SSH_AUTH_SOCK` stays governed only by `sshAuthSock`, even when listed under `allow:` — it is a live handle to the operator's ssh-agent, not a value.

Adds one INFO line per allow-list spawn, `member credentials: allowed N of M`, emitted only after the shell gate so it can never report coverage on a path where the scrub does not run.

Verified by the lead in an independent worktree: Tests run: 976, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 01:10:18 +02:00
ltms 65acf066ad #154: pin the AMQP reply inbox prefetch bound
CI / contract (push) Successful in 1m26s
CI / build (push) Successful in 1m39s
No behaviour change. #154 supposed that ownership drains a whole queue into the in-memory `held` map, making `x-max-length` and per-message TTL decorative. Measurement says otherwise: `basicConsume` is manual-ack, `deliverCallback` acks only duplicates, and `basicQos` is set on the one shared channel before any consumer starts — so total `held` is bounded by the prefetch window across all targets.

Adds a fake-broker test that drives the real `own()` path and fails if receipt ever starts acking, plus a javadoc line naming the prefetch window at the point of first mention.

Verified by the lead in an independent worktree: Tests run: 971, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 01:06:43 +02:00
Dai Ha ee8f570fd7 #157: isolate worktree credential helpers
CI / contract (pull_request) Successful in 1m5s
CI / build (pull_request) Successful in 2m46s
2026-08-28 06:06:12 +07:00
Dai Ha fa3f910d44 #154: pin AMQP reply inbox prefetch
CI / build (pull_request) Successful in 2m16s
CI / contract (pull_request) Successful in 2m18s
2026-08-28 06:03:54 +07:00
Dai Ha 46ac6e4e38 #157: use environment git credential helper
CI / contract (pull_request) Successful in 1m3s
CI / build (pull_request) Successful in 1m37s
2026-08-28 06:02:09 +07:00
Dai Ha 3bfa82839b fleetd#164: an empty or suspiciously fast scrape must fail, never resolve as a success
CI / contract (pull_request) Successful in 1m3s
CI / build (pull_request) Successful in 1m20s
CompletionResolver.resolve() used to hand the caller a successful "" reply whenever a
turn's scrape came back empty (whether the read failed, or genuinely produced nothing),
making a lost turn indistinguishable from a real empty answer. It also had no way to
tell a crashed backend's near-instant BUSY -> DONE transition apart from a genuine
completion.

Add MIN_TURN_NANOS (2s), a named floor below which a completed turn is treated as a
crash signature and failed rather than resolved as a reply. Fail on any empty scrape
(read failure or a clean-but-empty read) instead of resolving with "". Both failures
name the member and carry whatever is on the pane for context.

Thread an injectable LongSupplier clock through CompletionResolver (matching the
SessionManager/MessageService nowNanos pattern) so the floor is testable without a
real sleep.
2026-08-28 06:01:00 +07:00
Dai Ha 65ccf2e4ad CB-633 follow-up: only log allowed N of M when the scrub actually runs
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m44s
The coverage line was logged before the zsh gate, so a non-zsh
spawn (where nothing is scrubbed — overlayBlockedCredentials is the
fallback instead) printed 'allowed N of M' as if the derived
allow-list scrub had run. Move the log after the gate so it only
fires on the path that actually generates the ZDOTDIR scrub; the
non-zsh fallback keeps logCredentialGap's WARN as its only signal.

Added a test proving no 'allowed N of M' line is emitted on the
non-zsh fallback, through the real HerdrPeerLauncher#spawn path.
2026-08-28 06:00:38 +07:00
ltms 7a3b27f76f #150: report lead readiness from the delivery gate
CI / contract (push) Successful in 1m6s
CI / build (push) Successful in 1m7s
Share one deliverability predicate between Injector and FleetApp, so the status endpoint reports the same answer the injector acts on instead of re-deriving it from one of that predicate's two inputs.

Verified by the lead in an independent worktree: Tests run: 971, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.
2026-08-28 00:59:44 +02:00
Dai Ha 82e7be564c CB-633 follow-up: union memberCredentials.allow into the derived env allow-list
CI / build (pull_request) Successful in 1m7s
CI / contract (pull_request) Successful in 1m16s
MemberEnvAllowList.derive only ever looked at profile fields, so
memberCredentials.allow: was silently ignored under
policy: allow-list — turning the policy on would have blanked
credentials working members already depended on.

- derive(profiles, configuredAllow) unions memberCredentials.allow
  into the derived set, with SSH_AUTH_SOCK explicitly excluded from
  that union (it stays governed only by sshAuthSock: allow).
- HerdrPeerLauncher threads MemberCredentials.allowSet() into the
  derivation instead of calling the profiles-only overload.
- Added a per-spawn INFO log 'member credentials: allowed N of M'
  (N/M from the daemon's own env, the existing hostEnvNames proxy),
  never logging a blocked name or a value.
2026-08-28 05:56:27 +07:00
Dai Ha c5e24197bf #150: report lead readiness from delivery gate
CI / contract (pull_request) Successful in 46s
CI / build (pull_request) Successful in 1m37s
2026-08-28 05:54:40 +07:00
Dai Ha bcb402b688 #157: convert forge worktree origins to SSH
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 1m11s
2026-08-28 05:54:15 +07:00
Dai Ha 7c4170ff6d CB-643: join the message-layer evidence to the health monitor
CI / build (push) Successful in 1m8s
CI / contract (push) Successful in 1m9s
CB-640 published the three message-layer facts and CB-641 wired the herdr
and time ones. This joins them, so every HealthSnapshot field now carries
real evidence and the NOT_YET_OBSERVED placeholder is gone. That constant
is what made 8 of the 9 fault states unreachable, GONE and NEVER_READY
included, which is why CB-580's failTarget never fired.

hasOrphanedDelegation is a true snapshot, but it can read true for one
tick during an ordinary race: an async ticket exists before its virtual
thread reaches rendezvous.open, so for that instant nothing is accepted or
queued behind it. decide maps the field straight to DELEGATION_ORPHANED
with no smoothing, so one racy read would log a fault that clears on the
next tick. The monitor now requires two consecutive observations. That
costs one interval on a real orphan and removes the false positive.

Two tests drive real ticks against a genuinely orphaned ticket (an
unanswered fleet_ask that lapsed back to PENDING), not the seam: one tick
reports nothing, two report once, and a single clean tick in between
resets the streak.

Also correct two config comments. paneProbeIntervalSeconds is parsed and
read by nothing, so its "minimum 60" note promised a floor that does not
exist.

970 tests green.
2026-08-27 22:15:39 +07:00
Dai Ha a6095743f0 Merge CB-641: wire herdr and time evidence into the fleet health monitor 2026-08-27 22:06:50 +07:00
Dai Ha 66e776d178 CB-641: wire herdr health evidence
CI / contract (pull_request) Successful in 1m9s
CI / build (pull_request) Successful in 1m11s
2026-08-27 22:03:20 +07:00
Dai Ha 26f64cba45 Merge CB-640: MessageService evidence accessors for the health monitor
CI / build (push) Successful in 1m4s
CI / contract (push) Successful in 1m28s
2026-08-27 22:02:36 +07:00
Dai Ha 51047848f1 CB-642: make the fleets-status redaction global — a non-global sed leaks a second URI on the same line
CI / contract (push) Successful in 41s
CI / build (push) Successful in 1m29s
2026-08-27 22:01:15 +07:00
Dai Ha d8c0b657e8 Merge CB-642: /fleets-status skill — multi-fleet status over the shared LavinMQ 2026-08-27 22:00:49 +07:00
Dai Ha 312c0584ce CB-640: add MessageService message-layer health evidence accessors
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 1m11s
hasQueuedDelivery/hasStrandedReply/hasOrphanedDelegation surface three of the
message-layer facts FleetHealthMonitor needs but currently hardcodes to
NOT_YET_OBSERVED. Additive only — no existing public method's signature or
behavior changes.
2026-08-27 21:58:21 +07:00
Dai Ha da2625acfb CB-642: add shared fleets status skill
CI / contract (pull_request) Successful in 43s
CI / build (pull_request) Successful in 1m36s
2026-08-27 21:55:27 +07:00
Dai Ha 97d9cebc59 CB-634: skip the zsh scrub-report test when /bin/zsh is absent
CI / contract (push) Successful in 1m10s
CI / build (push) Successful in 1m27s
scrubWritesAnAllowedNofMReport spawned /bin/zsh with no guard, so it
errored on the Gitea CI runner (Linux ARM64 container, no zsh) while
the two sibling zsh tests already skipped there via assumeTrue. Add the
same guard so CI skips instead of failing; the Mac build still runs it.
2026-08-25 04:18:45 +02:00
Dai Ha 3ce76a5d69 CB-634: finish the bridge_* -> fleet_* tool rename in docs and comments
CI / contract (push) Successful in 1m5s
CI / build (push) Failing after 1m35s
The alias removal left bridge_* tool names in prose. Fix them:
- README no longer claims the old bridge_* names still answer (they were removed).
- pom + LeadTabScanner comments name fleet_* tools.
- FleetMcp comment no longer mentions the removed deprecated twin.
- docs/MCP-Contract.md and e2e swept bridge_* -> fleet_*; e2e ask files renamed.
The historical mcp__bridge__* mount-name note in CLAUDE.md is kept on purpose.
949 tests pass.
2026-08-25 04:06:43 +02:00
Dai Ha 2e138a199b CB-634: one shared "fleet" workspace + rename bridged -> fleetd cutover
Two changes ship together here.

1. One shared herdr workspace. The lead and every worker now live in one
   workspace called "fleet", so the operator sees one "session" with many
   windows, not two. Before, the lead sat in a "leads" workspace and workers
   in "bridged-workers", which read as two sessions. The lead is still told
   apart from workers by its exact tab label ("lead: <name>"), so putting them
   in one space is safe. LeadTabScanner keeps the exclude-by-label mechanism
   for split layouts; Fleetd now passes an empty exclude set.

2. Rename the daemon from "bridged" to "fleetd" (the binary, config, scripts,
   launchd/systemd units, module dir, and MCP mount).
   - Module dir bridged/ -> fleetd/; jar finalName -> fleetd.jar.
   - Log line, comments, docs, and CLAUDE.md updated to say fleetd.
   - Scripts renamed: redeploy-bridged.sh -> redeploy-fleetd.sh,
     bridged-launchd-wrapper.sh -> fleetd-launchd-wrapper.sh.
   - Deploy units renamed: dev.ltms.bridged.plist -> dev.ltms.fleetd.plist,
     bridged.service -> fleetd.service; launchd Label -> dev.ltms.fleetd.
   - Config default bridged.yaml -> fleetd.yaml; the legacy bridged.yaml is
     still read as a fallback, and still gitignored.
   - MCP: drop the deprecated bridge_* tool twins; only fleet_* remain. The
     server name is "fleet". The mount name in the local .mcp.json becomes
     "fleet" (gitignored, not in this commit).
   - Env var defaults BRIDGED_API_TOKEN -> FLEETD_API_TOKEN, fixture
     BRIDGED_WORKER_TOKEN -> FLEETD_WORKER_TOKEN.

Kept on purpose: the BRIDGED_MEMBER marker. Renaming it is a coupled change to
the credential-scrub security control (an operator secrets.sh may guard on it),
so it stays until that migration is done on its own.

Metrics were already fleet_* (CB-632); MetricNamesTest still guards that no
name says bridged_.

The canonical CLAUDE.md block and the wiki template stay byte-identical
(wiki working tree edited, committed to the wiki repo separately).

949 tests pass (mvn clean install). 4 fewer than before = the 4 removed
bridge_* alias tests.
2026-08-25 04:01:08 +02:00
Dai Ha 450a5ed9c5 Merge CB-634 + CB-636 + CB-637: members-IDE overlay, per-profile auto-compact window, cross-host lead coordination
CI / contract (push) Successful in 1m16s
CI / build (push) Failing after 1m30s
- CB-634: deliver IDE guidance as an on-disk CLAUDE.local.md / opencode overlay,
  pinned to the module dir, best-effort auto-open (opt-in per profile via ideMcpUrl).
- CB-636: per-profile autoCompactWindow -> --autocompact for claude-code,
  provider.<p>.models.<m>.limit.context for opencode. Range-validated [100000,1000000].
- CB-637: cross-host lead-to-lead over a shared AMQP coordination vhost
  (coordinator: block, fleet_send{coordId}, LeadMailbox + LeadCoordLoop).

952 unit tests + 5 LeadMailbox contract tests green. Live E2E verified on the Mac daemon.
2026-08-24 19:11:44 +02:00
Dai Ha edabccd885 CB-637: relabel lead-comms (was colliding CB-635) + document coordId route in the CLAUDE.md tool table
The lead-to-lead wiring shipped with CB-635 in its comments, but CB-635 is
already the broker.uriEnv / unreachable-broker work. Relabel the mailbox +
fleet_send{coordId} + receive loop to CB-637 so a ticket number names one
feature. Add the cross-host peer-lead row to the primary intent->tool table
(kept byte-identical with the wiki template).
2026-08-24 17:48:51 +02:00
Dai Ha 1dbe3a03fc Merge lead-comms-wiring: wire LeadMailbox into the daemon (fleet_send{coordId} + receive loop) 2026-08-24 17:42:11 +02:00
Dai Ha 6058b8472b lead comms: wire LeadMailbox into the daemon so lead-to-lead messages flow
CI / contract (pull_request) Successful in 41s
CI / build (pull_request) Failing after 1m22s
The sibling ticket landed the mechanism (LeadMailbox, LeadMessage, the
coordinator: config block) but nothing opened it, nothing sent through it, and
nothing read it. This is the wiring.

- LeadChannel: a small interface LeadMailbox now implements (publish/peek/ack
  plus a selfCoordId() accessor). It exists so FleetMcp and the receive loop can
  be tested with a fake instead of a live broker. LeadMailbox's AMQP logic is
  untouched — the diff is the implements clause, four @Override marks and the
  accessor.

- Fleetd.openLeadMailbox: opens this daemon's mailbox after the reply inbox is
  selected, with the same env-injected seam selectReplyInbox uses. Every "off"
  path returns null and the daemon still starts: no coordinator block (silent),
  a uriEnv that does not resolve (INFO), a configured broker with no selfId
  (WARN — a mailbox is named after the coord-id that owns it), or a broker that
  refuses at boot (WARN, credentials stripped). Closed in the ordered shutdown
  hook, after the loop that reads it has stopped.

- fleet_send{coordId}: publishes a LeadMessage(from=selfCoordId, to=coordId) to
  the peer's mailbox and returns the broker-confirmed receipt. coordId is
  mutually exclusive with sessionId/turnId and is rejected by name rather than
  resolved by precedence. An unroutable/nacked/timed-out publish comes back as a
  tool error naming the coordId, never a crash. The worker send/reply path is
  not touched.

- LeadCoordLoop: the receive half. Each tick peeks the mailbox, resolves the
  local lead pane, and — only at a turn boundary — injects "[lead <from>] <text>"
  and acks. Anything not delivered stays unacked and is retried, so a message is
  never dropped; one message per tick, so every delivery is gated on a status
  read that already saw the previous one.

- fleet_list reports {selfId, configured} when coordination is on, so an
  operator can find the coord-id a peer must use to reach them. Omitted
  entirely when it is off.

Tests: 20 new hermetic tests (no broker) across routing, delivery and startup
selection. mvn clean install: Tests run: 944, Failures: 0, Errors: 0, Skipped: 0
— BUILD SUCCESS.
2026-08-24 17:39:29 +02:00
Dai Ha f29968c334 Merge autocompact-window: per-profile autoCompactWindow for claude-code + opencode members 2026-08-24 17:32:53 +02:00
Dai Ha 7b98cca967 LeadMailbox: durable leader-to-leader mailbox over a shared coordination vhost
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Failing after 1m35s
Adds the broker-side mechanism for lead-to-lead messages across daemons/hosts
(unit 1 of 2): a LeadMessage envelope carrying from/to coord-ids, an
AMQP-backed LeadMailbox modeled closely on AmqpReplyInbox (consume-and-hold,
deferred manual ack, confirm-mode publish, recovery handling), and a new
optional coordinator: config block (separate vhost from broker:, leader
traffic only). Config parsing + accessors only — FleetMcp/Fleetd/Injector/
MessageService and the send path are untouched; wiring is a separate ticket.
2026-08-24 17:23:42 +02:00
257 changed files with 26917 additions and 6876 deletions
+2 -2
View File
@@ -1,6 +1,6 @@
{
"name": "claude-bridge",
"description": "Tooling for orchestrating a fleet of delegated coding agents through the bridged MCP gateway.",
"description": "Tooling for orchestrating a fleet of delegated coding agents through the fleetd MCP gateway.",
"owner": {
"name": "LTMS"
},
@@ -8,7 +8,7 @@
{
"name": "claude-bridge",
"source": "./plugin",
"description": "Make a project bridge-ready: mount the bridged MCP gateway and apply standard Claude Code settings so a session can orchestrate delegated workers. Ships no credentials.",
"description": "Make a project bridge-ready: mount the fleetd MCP gateway and apply standard Claude Code settings so a session can orchestrate delegated workers. Ships no credentials.",
"version": "0.1.0",
"author": {
"name": "LTMS"
+1 -1
View File
@@ -3,7 +3,7 @@ name: architect
description: Refine work into clear, independent units before implementation.
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You are an architect in this fleet. You refine work before anyone builds it: scope,
acceptance criteria, risks, and a unit split. You read the repo and write analysis.
+1 -1
View File
@@ -3,7 +3,7 @@ name: dev
description: Implement one assigned unit, test it, and open a pull request.
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You implement the one unit you were given and nothing else. Work in your assigned
git worktree and branch. Never check out, rebase onto, or push to `main`. Confirm
+1 -1
View File
@@ -3,7 +3,7 @@ name: reviewer
description: Review one assigned scope and report the most important real issue.
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You review the diff you were given. Report bugs, risks, and missing tests. You do
not change code.
+274
View File
@@ -0,0 +1,274 @@
---
name: fleets-status
description: Report the status of every fleet that shares one LavinMQ instance. Use for local daemon health, broker-wide fleet presence, and cross-host lead coordination checks.
---
# Status of every fleet on the shared LavinMQ instance
**The headline: always report what is missing.** This skill starts with the local fleet, then adds
broker-wide facts when its read-only credential exists. A missing fleet must appear as `unknown` or
`not reachable`, with the reason and the fix. Never leave it out.
The known topology has one LavinMQ instance on `10.10.20.13` (`fleet01`). AMQP uses port `5672`,
and the management API uses port `15672`. The Mac fleet owns vhost `/mac`. The fleet01 fleet owns
vhost `/fleet01`.
## 1. Protect credentials before any probe
**Hard rule — never print `LAVINMQ_URI`.** It is an AMQP URI with its password inline. It only
resolves in a login shell because `${SHARED_ENV}/tools/secrets.sh` supplies it. A non-login shell
can make every broker probe look empty.
- Never run `echo "$LAVINMQ_URI"`.
- Never put `${LAVINMQ_URI:-something}` in output. That form expands to the secret value when set.
- Parse the user, host, and password into shell or Python variables. Use them without printing them.
- Prefer `resolves` or `does not resolve` over any part of the value.
- Every command that can read `LAVINMQ_URI` must send all output through this redaction before it
reaches the report:
```bash
sed -E 's#://[^@]*@#://<redacted>@#g'
```
**The `g` flag is not optional.** Without it `sed` replaces only the first match on each line, so a
line carrying two URIs leaks the second one. `scripts/redeploy-fleetd.sh --check` prints lines like
that. Checked on 2026-08-27: without `g`, `amqp://u1:p1@h1/mac and http://u2:p2@h2:15672/api`
redacts the first pair and prints `u2:p2` in the clear.
Keep `pipefail` on when applying that filter. Otherwise the filter can hide a failed probe. Apply
the same no-print rule to the management password below, even though it is not in an AMQP URI.
## 2. Tier 1 — this fleet (always run)
Start here even when the broker tier is blocked. Work from the local fleetd checkout.
First run the read-only deployment check. It already checks the daemon process, deployed jar versus
the checkout `HEAD`, launchd state, and whether each configured token resolves in a login shell.
Do not copy those checks into new shell code. The script reads `LAVINMQ_URI`, so redact all output:
```bash
set -o pipefail
scripts/redeploy-fleetd.sh --check 2>&1 \
| sed -E 's#://[^@]*@#://<redacted>@#g'
git rev-parse HEAD
```
Treat jar drift as a top-level warning. A merge is not a deployment. State the running jar result
as `matches HEAD`, `drift`, or `unknown`; do not turn an unclear timestamp into a match.
Report the process identifier (PID) and uptime too:
```bash
PIDS="$(pgrep -f 'target/fleetd.jar' || true)"
if [ -z "$PIDS" ]; then
printf '%s\n' 'fleetd: not running'
else
for PID in $PIDS; do
ps -p "$PID" -o pid=,etime=,lstart=,command=
done
fi
```
Read the full health response. Keep the HTTP status because `503` means fleetd is running but herdr
is not reachable. Report both `herdr.version` and `herdr.protocol` when present:
```bash
curl -sS --max-time 5 -w '\nHTTP %{http_code}\n' http://127.0.0.1:8765/healthz
```
Call `fleet_whoami`, then call `fleet_list`. Preserve its sections in the report:
- `leads`, including which row is this lead;
- `members`, including state, role, profile, branch, and worktree when present;
- every per-profile `capacity` row, including `maxLoad`, `live`, `free`, and quarantine facts;
- the exact `healthCoverage` value.
Do not describe an empty `members` list as an empty fleet. It says only that no members are spawned.
Also do not hide a profile with `free: 0`; say whether load or credential quarantine caused it.
Show only WARN, ERROR, and SEVERE lines after the last `fleetd listening` line. This anchor stops an
old incident from looking current:
```bash
python3 - <<'PY'
from pathlib import Path
import re
path = Path("fleetd/fleetd.out")
if not path.exists():
print("cannot check current WARN/ERROR: fleetd/fleetd.out does not exist")
else:
lines = path.read_text(errors="replace").splitlines()
starts = [i for i, line in enumerate(lines) if "fleetd listening" in line]
if not starts:
print("cannot anchor WARN/ERROR: no 'fleetd listening' line exists")
else:
current = lines[starts[-1]:]
alerts = [line for line in current if re.search(r"\b(?:WARN|ERROR|SEVERE)\b", line)]
print(f"current WARN/ERROR/SEVERE count: {len(alerts)}")
for line in alerts[-50:]:
print(line)
PY
```
**What this tier cannot see:** it proves facts only about the Mac daemon at `127.0.0.1:8765`.
It cannot show the fleet01 daemon, broker queue depth, or broker consumers. The fleet01 REST service
at `10.10.20.13:8765` is not reachable from the Mac. Say this in the report rather than omitting
fleet01.
**But fleet01 IS reachable over SSH — checked 2026-08-28.** An older version of this line said SSH
was denied. That is true only for the user `dai.ha`. The host alias `fleet01` maps to user `ltms`,
and `ssh fleet01` works with key auth:
```bash
ssh -o BatchMode=yes -o ConnectTimeout=6 fleet01 'echo $(id -un)@$(hostname)'
```
So fleet01's daemon PID, uptime, jar and `/healthz` **can** be reported — over SSH, not over REST.
Do that rather than writing `not reachable`. `ltms` also has passwordless sudo there.
## 3. Tier 2 — the shared broker (run when management access exists)
**This tier is blocked today.** The AMQP user in `LAVINMQ_URI` can connect on port `5672`, but gets
HTTP `401` from the management API on port `15672`. An AMQP connection does not grant monitoring
access.
The operator must create a separate, read-only LavinMQ management user with the `monitoring` tag.
It needs access to inspect both `/mac` and `/fleet01`. Store its values as
`LAVINMQ_MANAGEMENT_USER` and `LAVINMQ_MANAGEMENT_PASSWORD` in
`${SHARED_ENV}/tools/secrets.sh`. Do not reuse or print the AMQP URI. Full multi-fleet status stays
blocked until this user exists.
When both variables resolve, run this from a login shell. It calls `GET /api/overview`,
`GET /api/vhosts`, `GET /api/queues`, and `GET /api/connections`. It prints selected status fields,
but never the user, password, Authorization header, or AMQP URI:
```bash
zsh -lc 'python3 - "$@"' -- <<'PY'
import base64
import json
import os
import sys
import urllib.error
import urllib.request
base = "http://10.10.20.13:15672"
user = os.environ.get("LAVINMQ_MANAGEMENT_USER", "")
password = os.environ.get("LAVINMQ_MANAGEMENT_PASSWORD", "")
if not user or not password:
print("broker tier: BLOCKED — management credential does not resolve in a login shell")
sys.exit(0)
token = base64.b64encode(f"{user}:{password}".encode()).decode()
def get(path):
request = urllib.request.Request(
base + path,
headers={"Authorization": "Basic " + token, "Accept": "application/json"},
)
with urllib.request.urlopen(request, timeout=5) as response:
return json.load(response)
try:
overview = get("/api/overview")
vhosts = get("/api/vhosts")
queues = get("/api/queues")
connections = get("/api/connections")
except urllib.error.HTTPError as error:
print(f"broker tier: BLOCKED — management API returned HTTP {error.code}")
sys.exit(0)
except Exception as error:
print(f"broker tier: BLOCKED — management API is not reachable: {type(error).__name__}")
sys.exit(0)
fleet_names = {"/mac": "Mac fleet", "/fleet01": "fleet01 fleet"}
print(json.dumps({
"overview": {
"lavinmq_version": overview.get("lavinmq_version"),
"rabbitmq_version": overview.get("rabbitmq_version"),
"queue_totals": overview.get("queue_totals", {}),
"object_totals": overview.get("object_totals", {}),
},
"fleets": [
{
"fleet": fleet_names.get(vhost.get("name"), "UNKNOWN FLEET"),
"vhost": vhost.get("name"),
"queues": [
{
"name": queue.get("name"),
"messages": queue.get("messages", 0),
"messages_ready": queue.get("messages_ready", 0),
"messages_unacknowledged": queue.get("messages_unacknowledged", 0),
"consumers": queue.get("consumers", 0),
}
for queue in queues if queue.get("vhost") == vhost.get("name")
],
"connections": [
{
"name": connection.get("name"),
"peer_host": connection.get("peer_host"),
"state": connection.get("state"),
}
for connection in connections if connection.get("vhost") == vhost.get("name")
],
}
for vhost in vhosts
],
}, indent=2, sort_keys=True))
PY
```
Map `/mac` to the Mac fleet and `/fleet01` to the fleet01 fleet. Keep any other vhost in the
report as `UNKNOWN FLEET`; do not drop it. For each vhost, total the ready, unacknowledged, and all
messages. Report every queue's consumer count and each live connection.
**A vhost with queues but zero consumers means that fleet's daemon is down while its durable state
survives. Call this out as a top-level warning.** This is the main reason to use the management API
instead of calling each remote daemon.
**What this tier cannot see:** without the new `monitoring` credential it cannot enumerate any
vhost, queue, depth, consumer, or connection. With the credential it still cannot report fleet01's
daemon PID, uptime, jar revision, `/healthz`, herdr version, or member capacity. Those need reachable
fleet01 REST or SSH access, which the Mac does not have today.
## 4. Tier 3 — cross-fleet lead coordination
Use the queue data from Tier 2. Select queues whose names match `lead.<coordId>.inbox`. Report each
queue's vhost, depth, consumer count, and the `coordId` between the prefix and suffix.
- A lead inbox with a consumer shows that a lead mailbox is live on that vhost.
- A durable lead inbox with zero consumers shows saved coordination state, but no live receiver.
- No lead inbox is not proof that coordination is disabled. The daemon may be down before declaring
its queue, or this account may not be allowed to see the vhost.
This Mac fleet currently sets both `broker.uriEnv` and `coordinator.uriEnv` to the same variable,
`LAVINMQ_URI`. Therefore its coordinator connects to `/mac`. Cross-host `fleet_send{coordId}` routes
only when both leads share the same coordinator vhost. If the fleet01 lead uses `/fleet01` for its
coordinator, the leads cannot see each other and the send will not route.
**Open question:** the fleet01 coordinator vhost has not been checked. Surface this question in
every report until a live `lead.<coordId>.inbox` consumer or fleet01's config proves the answer. Do
not claim that fleet01 uses `/fleet01` just because its member queues do.
Also compare these broker facts with the `leads` rows from local `fleet_list`. A missing remote lead
is `not visible from this coordinator`, not `down`, unless the broker consumer facts prove it.
**What this tier cannot see:** without Tier 2 management access it cannot list lead inboxes or their
consumers. Even with that access, a stopped fleet01 daemon leaves only durable queue history. That
history cannot prove which coordinator URI its current config would use after restart.
## 5. Report all fleets
Use one row per known or discovered fleet. Include blocked rows.
| Fleet | Daemon | Deployment | Herdr | Members/capacity | Queues/consumers | Lead coordination | Cannot check |
|---|---|---|---|---|---|---|---|
| Mac (`/mac`) | PID + uptime | jar vs `HEAD` | health + version + protocol | `fleet_list` + `healthCoverage` | facts or blocked reason | inbox facts or open question | exact missing facts |
| fleet01 (`/fleet01`) | reachable/down/unknown | value or `not reachable` | value or `not reachable` | value or `not reachable` | facts or blocked reason | inbox facts plus coordinator-vhost question | exact missing facts and fix |
Add rows for unknown vhosts. End with three short sections: `Current warnings`, `Checks that were
blocked`, and `Operator action`. Until the management user exists, `Operator action` must say:
> Create a read-only LavinMQ management user with the `monitoring` tag and access to `/mac` and
> `/fleet01`. Put its user and password in `${SHARED_ENV}/tools/secrets.sh` as
> `LAVINMQ_MANAGEMENT_USER` and `LAVINMQ_MANAGEMENT_PASSWORD`.
+15 -2
View File
@@ -1,6 +1,6 @@
---
name: implementer
description: Implementer-role procedure for a bridged worker — verify your worktree, implement the scope, commit, push, open your own PR, and hand off the PR URL. Load this when the lead delegates you an implementation task over bridged.
description: Implementer-role procedure for a fleetd worker — verify your worktree, implement the scope, commit, push, open your own PR, and hand off the PR URL. Load this when the lead delegates you an implementation task over fleetd.
---
# Implementer worker — procedure
@@ -39,6 +39,19 @@ a worker made all 59 of its edits in the primary's tree and never noticed.
test "$(git rev-parse --show-toplevel)" = "$PWD" || cd "$(git rev-parse --show-toplevel)"
```
**Never run `git stash` (or `git stash pop`/`apply`/`drop`).** Your worktree is isolated, but the
stash is **not**: `refs/stash` is one stack shared by the primary's checkout and every other
worker's worktree of this repo. Measured on 2026-09-04 — `git stash list` from a worker's worktree
and from the primary's tree returned byte-identical output. So a `git stash` you run can be popped
into someone else's tree, and a `git stash pop` you run can drop **another worker's** uncommitted
edits on top of yours. This has already happened here: two workers were running in parallel and one
of them had its in-progress edit silently overwritten by the other's stash.
The branch is your isolation, so use it instead. To set work aside, commit it on your own branch
(`git commit -m "wip: ..."`) and carry on; to try something and back out, use
`git diff > /tmp/<your-branch>.patch` then `git checkout -- <file>`. Both stay inside your worktree.
If you find a stash entry you did not create, leave it alone and say so in your report.
## 2. Implement
- Implement exactly the scope the lead named. Keep the diff focused; note anything out of scope
@@ -49,7 +62,7 @@ test "$(git rev-parse --show-toplevel)" = "$PWD" || cd "$(git rev-parse --show-t
your worktree*:
```bash
cd "$(git rev-parse --show-toplevel)/bridged" && mvn clean install
cd "$(git rev-parse --show-toplevel)/fleetd" && mvn clean install
echo "exit=$?"
```
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: reviewer
description: Reviewer-role procedure for a bridged worker — how to work a review scope and the exact shape of the finding to report. Load this when the lead delegates you a code review over bridged.
description: Reviewer-role procedure for a fleetd worker — how to work a review scope and the exact shape of the finding to report. Load this when the lead delegates you a code review over fleetd.
---
# Reviewer worker — procedure
+6 -6
View File
@@ -14,7 +14,7 @@ jobs:
# does not depend on the wiki repo being reachable.
- uses: actions/checkout@v4
# The runner image ships an older default-jdk; bridged sets maven.compiler.release=25, so
# The runner image ships an older default-jdk; fleetd sets maven.compiler.release=25, so
# provision the JDK explicitly rather than apt-installing whatever "default" means today.
- name: Set up JDK 25
uses: actions/setup-java@v4
@@ -32,7 +32,7 @@ jobs:
mvn -version
- name: Build and test
working-directory: bridged
working-directory: fleetd
# This IS the mock-socket surface CB-503 asks for: the pom's `default-excludes` profile
# already sets excludedGroups=contract, so the @Tag("contract") tests — which need a live
# herdr socket and a RabbitMQ container — are excluded without any flag here. Everything
@@ -46,7 +46,7 @@ jobs:
# the log instead, where they are actually readable.
- name: Failing test output
if: failure()
working-directory: bridged
working-directory: fleetd
run: |
for f in target/surefire-reports/*.txt; do
[ -f "$f" ] || continue
@@ -59,7 +59,7 @@ jobs:
# AmqpReplyInboxContractTest reads AMQP_URI (set below to the service's network alias) and binds
# straight to it — no Docker, no skipped tests. This separation (build job hermetic and
# Docker-free; contract job broker-provided) is deliberate — see the default-excludes/contract
# profiles in bridged/pom.xml. `setup-java` provides the JDK only; Maven is installed separately,
# profiles in fleetd/pom.xml. `setup-java` provides the JDK only; Maven is installed separately,
# exactly as in the build job above.
contract:
runs-on: ubuntu-latest
@@ -91,12 +91,12 @@ jobs:
# runs against the RabbitMQ service container (AMQP_URI). Pinned to the one contract test to
# avoid re-running the unit suite already covered by the `build` job.
- name: Contract tests
working-directory: bridged
working-directory: fleetd
run: mvn -B -Pcontract test -Dtest=AmqpReplyInboxContractTest
- name: Failing test output
if: failure()
working-directory: bridged
working-directory: fleetd
run: |
for f in target/surefire-reports/*.txt; do
[ -f "$f" ] || continue
+4 -4
View File
@@ -14,8 +14,8 @@
.env
.envrc
# Daemon runtime artefacts. bridged appends its log wherever it is launched from, so both the
# repo root and bridged/ collect one; neither belongs in git.
bridged.out
bridged/bridged.out
# Daemon runtime artefacts. fleetd appends its log wherever it is launched from, so both the
# repo root and fleetd/ collect one; neither belongs in git.
fleetd.out
fleetd/fleetd.out
logs/
+1 -1
View File
@@ -3,7 +3,7 @@ description: Refine work into clear, independent units before implementation.
mode: primary
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You are an architect in this fleet. You refine work before anyone builds it: scope,
acceptance criteria, risks, and a unit split. You read the repo and write analysis.
+1 -1
View File
@@ -3,7 +3,7 @@ description: Implement one assigned unit, test it, and open a pull request.
mode: primary
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You implement the one unit you were given and nothing else. Work in your assigned
git worktree and branch. Never check out, rebase onto, or push to `main`. Confirm
+1 -1
View File
@@ -3,7 +3,7 @@ description: Review one assigned scope and report the most important real issue.
mode: primary
---
<!-- CB-617: The model comes from bridged.yaml because the launch flag overrides model here on both backends. -->
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You review the diff you were given. Report bugs, risks, and missing tests. You do
not change code.
+46 -21
View File
@@ -10,7 +10,7 @@
If no `fleet_*` MCP tools are mounted in this session, this section does not apply — skip it.
`bridged` is the **sole communication gateway** between agents here. The orchestrating session (the
`fleetd` is the **sole communication gateway** between agents here. The orchestrating session (the
**primary**) and every delegated peer (a **member**) mount the *same* MCP server and talk only
through its `fleet_*` tools. No session addresses a peer, a broker, or the network directly.
@@ -112,7 +112,8 @@ the merge — and merging on a reviewer's word is delegating it by proxy.
| Delegate (blocking) | `fleet_send{sessionId, content}` |
| Delegate (long task) | `fleet_send{sessionId, content, wait:false}` → ticket → `fleet_poll{ticket}` |
| Answer a member's `fleet_ask` | `fleet_send{turnId, content}` — **not** `sessionId` |
| Message a **peer lead** | `fleet_send{sessionId: <their terminal>, content}` — `fleet_list` → `leads` reports it. Coordination only, **never** a task |
| Message a **peer lead** on this host | `fleet_send{sessionId: <their terminal>, content}` — `fleet_list` → `leads` reports it. Coordination only, **never** a task |
| Message a **peer lead** on another daemon or host | `fleet_send{coordId: <their coord-id>, content}` — needs a `coordinator:` block; your own coord-id is in `fleet_list`. Coordination only, **never** a task |
| Answer a peer lead that messaged you | `fleet_reply{content}` — the one case a lead replies |
| Collect a held reply | `fleet_poll{target}` · then `fleet_ack{target, msgId}` |
| Tear down a member | `fleet_stop{paneId}` |
@@ -184,24 +185,48 @@ must obey belongs in the charter, not here.
## Project addendum — claude-bridge (not part of the canonical block)
- **This repo is the bridge.** The daemon is `bridged`, its MCP mount is `http://127.0.0.1:8765/mcp`,
- **This repo is the bridge.** The daemon is `fleetd`, its MCP mount is `http://127.0.0.1:8765/mcp`,
and the code behind the rules above is `mcp/FleetMcp` (tools), `auth/Authz` (the role table),
`mcp/ConnectionIdentity` (connection→role), and `worker/*Launcher` (`REPLY_CHARTER`).
- **`fleet_profiles`/`fleet_list` report two separate outage states, and they are not the same
thing.** *Quarantined* (CB-578) means the backend told us it is out of capacity — a long,
1800s-default cooldown. *Cooling off* (fleetd #201/#227) means a profile's credential threw two
distinct backend errors (a non-exhaustion failure such as an HTTP 5xx) within 60 seconds — a
short, fixed 60s cooldown, not configurable per profile. Each check runs independently, so a
profile can show both at once. In the JSON: a cooling profile carries `credentialId` and
`coolingOffForSeconds`; a quarantined profile carries `quarantinedForSeconds`; a profile hit by
both carries all three fields, and either state alone already sets that profile's `free` to `0`.
A `fleet_spawn` naming a cooling-off profile is refused before it ever reaches the backend
adapter, with a message naming the credential and the remaining seconds ("cooling off after
repeated backend errors") — distinct wording from a quarantine refusal, so don't conflate the
two when reading a spawn failure.
- **Skills available to delegate:** `implementer` (worktree → commit → push → own PR) and
`reviewer` (scoped review → one structured finding). Name one in every delegation.
- **Primary-side skills** (not delegation playbooks — a worker cannot use them):
`port-to-opencode` (make an OpenCode session a participant in this workspace).
`port-to-opencode` (make an OpenCode session a participant in this workspace) and
`fleets-status` (report every fleet that shares one LavinMQ instance).
- **Never commit** `.mcp.json` (the primary's local copy, flagged `--skip-worktree`) or `wiki/`
(a submodule with its own remote).
- **Flows and the error model** — rendezvous, `fleet_ask`, detached delivery, turn-done fallback —
are diagrammed in `docs/MCP-Contract.md` **§6 only**. The rest of that page is a pre-build design
doc whose tool names, parameter names and REST paths never caught up with the code, so do not use
it as the tool reference (CB-609). Section 6 is kept out of this file because this file loads into
every session's context.
- **A provisioned worktree neutralizes `.mcp.json`, `opencode.json` and `.autoenv`** — the repo's
committed copies would otherwise mount the primary's IDE and forge servers (fleetd #134). The
worktree's copy of each is a stub, **not** the repo's real file, so a worker that reads one and
reports what it found is reporting on the stub. The daemon logs a per-spawn summary, but the
worker cannot see that log. From inside its own worktree a worker — or a lead debugging one —
reads the list with `git config --worktree --get-all fleet.neutralizedConfig`, and the
consequence with `git config --worktree --get fleet.neutralizedConfigNote`. Never brief a worker
to edit one of these files: the edit cannot be committed, and it will not tell you so.
- **Flows and the error model** — rendezvous, `fleet_ask`, detached delivery, the turn-done
fallback and status gating — are diagrammed in `docs/MCP-Contract.md`. That page is now flows
only: its pre-build tool catalogue, parameter tables and REST paths were deleted rather than
corrected, because a hand-maintained second copy of the tool surface is what drifted for a month
while this line pointed every session at it (CB-609 / #114). **The live MCP schema is the tool
reference**, with the intent→tool table above as the short form. `McpContractDocTest` fails if
that page names a `fleet_*` tool the server does not register. The flows are kept out of this
file because this file loads into every session's context.
### Redeploying the daemon — the lead may do this (primary only)
**A merge is not a deployment.** The running `bridged` holds the jar it was started with, so a
**A merge is not a deployment.** The running `fleetd` holds the jar it was started with, so a
feature merged to `main` does nothing until the daemon is rebuilt and restarted. Saying "shipped"
about code the live daemon has never loaded is a false report. The lead **may and should** redeploy
rather than hand the job back to the operator.
@@ -212,9 +237,9 @@ and stopping it kills the worker's own channel mid-turn.
**Use the script — do not hand-roll the steps.**
```bash
scripts/redeploy-bridged.sh --check # report state, change nothing
scripts/redeploy-bridged.sh # build, confirm drain, restart, verify
scripts/redeploy-bridged.sh --yes # skip the drain prompt (fleet already checked)
scripts/redeploy-fleetd.sh --check # report state, change nothing
scripts/redeploy-fleetd.sh # build, confirm drain, restart, verify
scripts/redeploy-fleetd.sh --yes # skip the drain prompt (fleet already checked)
```
It builds before it stops anything, so a failed build never leaves the fleet down; it waits for the
@@ -240,8 +265,8 @@ if the script is unavailable or a step fails, this is what it was protecting you
4. **Re-check identity afterwards.** Call `fleet_whoami` and confirm it still answers `primary`. The
lead is found by its tab label (`fleet.leaders.*.tab`), and a lead whose tab no longer matches is
demoted to worker, which refuses every orchestration call.
5. **Prove the new jar is the one running.** Confirm a *fresh* `bridged listening` line at the end of
`bridged/bridged.out`, dated after the restart. An old daemon that never died looks identical from
5. **Prove the new jar is the one running.** Confirm a *fresh* `fleetd listening` line at the end of
`fleetd/fleetd.out`, dated after the restart. An old daemon that never died looks identical from
the outside.
**Permission.** A `CLAUDE.md` rule grants intent, not tool permission — the command classifier
@@ -250,7 +275,7 @@ it is one auditable command, so the operator allow-lists it once instead of appr
start every time. The rule lives in the operator's Claude Code settings:
```json
{ "permissions": { "allow": ["Bash(scripts/redeploy-bridged.sh:*)"] } }
{ "permissions": { "allow": ["Bash(scripts/redeploy-fleetd.sh:*)"] } }
```
Granted by the operator on 2026-08-15. If a call is still refused, do **not** route around it by
@@ -276,7 +301,7 @@ Before you call any work done, check the row that matches what you touched:
| worktree provisioning or the parity overlay | the "both roles read this file" premise — it rests on the worker's worktree being a checkout of this repo |
| `.claude/skills/**` | the addendum's skill list, and the "name the playbook" rule |
| a new peer kind (non-Claude adapter) | what that peer can read — anything it must obey belongs in its charter, not in the block |
| **anything an operator can use, configure, or observe** — an MCP tool, a `bridged.yaml` knob, an endpoint, a visible behaviour | **[Features](wiki/11-Features.md)** — one entry: what it does · the knob that turns it on · **why it exists** · the gotcha |
| **anything an operator can use, configure, or observe** — an MCP tool, a `fleetd.yaml` knob, an endpoint, a visible behaviour | **[Features](wiki/11-Features.md)** — one entry: what it does · the knob that turns it on · **why it exists** · the gotcha |
That last row is not bookkeeping. Chapters 1–10 answer *how is this built* and *why this way*;
none of them has a home for *what can it do and how do I turn it on*, so for twenty tickets a
@@ -310,10 +335,10 @@ PY
Two IDE MCP servers are connected: **intellij-index** (semantic code intelligence) and
**jetbrains** (file problems, reformat, debugger). IntelliJ has multiple projects open; our
module is **`bridged`**. Always pass these to IDE MCP tools:
module is **`fleetd`**. Always pass these to IDE MCP tools:
- `project_path` = `/Users/dai.ha/LTMS/claude-bridge/bridged`
- IDE paths are relative to `bridged/` (e.g. `src/main/java/dev/ltms/bridged/...`)
- `project_path` = `/Users/dai.ha/LTMS/claude-bridge/fleetd`
- IDE paths are relative to `fleetd/` (e.g. `src/main/java/dev/ltms/fleet/...`)
### After editing any file — mandatory
@@ -327,7 +352,7 @@ module is **`bridged`**. Always pass these to IDE MCP tools:
whole-project gate before declaring work done or committing.
**Whenever dependencies change (or a `pom.xml` edit), validate CVEs with
`jetbrains get_file_problems{filePath: "bridged/pom.xml"}`** — its Mend.io check reflects the
`jetbrains get_file_problems{filePath: "fleetd/pom.xml"}`** — its Mend.io check reflects the
dependencies on disk. (Note: `ide_diagnostics` / intellij-index does NOT re-resolve dependencies
after a pom edit without a full Maven reimport, so it reports stale CVE results — don't trust it
for this.) Treat a CVE warning like any other: bump to a patched version and confirm
+2 -3
View File
@@ -56,9 +56,8 @@ flowchart LR
ever addresses a broker, a peer, or the network
directly**; any queue is `fleetd`-internal. MCP tool I/O never sets `ANTHROPIC_BASE_URL`, so
mounting the bridge is subscription-safe by construction.
**Tool naming:** the tools were renamed from `bridge_*` to `fleet_*` (CB-622). The daemon
still answers the old `bridge_*` names for one release, but they are deprecated — use the
`fleet_*` names.
**Tool naming:** the tools are `fleet_*` (renamed from `bridge_*` in CB-622). The old
`bridge_*` names were removed in CB-634 — only `fleet_*` answers now.
- **How the primary consumes a reply:** a single **blocking MCP call** (`fleet_send`);
`fleetd` holds it open until the worker calls `fleet_reply` or its turn hits
`agent_status=done`, then returns the reply as the tool result. No cross-turn busy-poll, so
@@ -1,21 +0,0 @@
package dev.ltms.fleet.auth;
import dev.ltms.fleet.peer.MemberRole;
/** Optional session lifecycle hook for live member-slot bindings. */
public interface MemberLifecycle {
MemberLifecycle NONE = new MemberLifecycle() {
@Override
public void acquired(MemberRole role, String profile, String terminal) {
}
@Override
public void released(String terminal) {
}
};
void acquired(MemberRole role, String profile, String terminal);
void released(String terminal);
}
@@ -1,151 +0,0 @@
package dev.ltms.fleet.health;
import dev.ltms.fleet.herdr.Agent;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.msg.MessageService;
import dev.ltms.fleet.session.MemberSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.BiConsumer;
import java.util.function.LongSupplier;
import java.util.function.Supplier;
/** Slow whole-fleet evidence collection. It is deliberately separate from the delivery poller. */
public final class FleetHealthMonitor {
private static final Logger log = LoggerFactory.getLogger(FleetHealthMonitor.class);
/** Bounded attempts to run {@link #failTarget} for one transition. Never retried tick-to-tick (CB-580). */
static final int MAX_FAIL_TARGET_ATTEMPTS = 3;
private final AgentControl agents;
private final Supplier<List<MemberSession>> roster;
private final MessageService messages;
private final ScheduledExecutorService scheduler;
private final LongSupplier clock;
private final long intervalSeconds;
private final BiConsumer<String, String> failTarget;
private final Map<String, HealthPrior> priors = new HashMap<>();
private final Map<String, HealthState> states = new HashMap<>();
// These facts need the evidence publishers introduced by later M4 units. They are not negatives.
private static final boolean NOT_YET_OBSERVED = false;
/**
* @param failTarget CB-568's idempotent target-wide failure operation (e.g. {@code messages::abandon}),
* invoked once when a member transitions into a terminal health state. Required —
* there is deliberately no defaulting overload; a caller that does not want the
* fail-tickets-on-terminal-health behavior must pass an explicit inert value (see
* {@code TestTurnTokens.inert} / {@code FleetMcp.CapacitySource.none()} for the pattern).
*/
public FleetHealthMonitor(AgentControl agents, Supplier<List<MemberSession>> roster, MessageService messages,
ScheduledExecutorService scheduler, LongSupplier clock, long intervalSeconds,
BiConsumer<String, String> failTarget) {
this.agents = agents;
this.roster = roster;
this.messages = messages;
this.scheduler = scheduler;
this.clock = clock;
this.intervalSeconds = intervalSeconds;
this.failTarget = Objects.requireNonNull(failTarget, "failTarget");
}
/** Pure per-member decision seam. */
static HealthDecision decide(HealthSnapshot snapshot, HealthPrior prior, long nowNanos) {
return FleetHealth.decide(snapshot, prior, nowNanos);
}
public void start() { scheduler.schedule(this::tick, intervalSeconds, TimeUnit.SECONDS); }
public void stop() { scheduler.shutdownNow(); }
// Package-private so tests can run one tick without waiting.
void tick() {
try {
List<Agent> agentsNow = agents.list(); // Exactly one list call for this complete observation.
List<MemberSession> rosterNow = roster.get(); // One in-memory roster snapshot for this tick.
Map<String, Agent> live = new HashMap<>();
for (Agent agent : agentsNow) live.put(agent.terminalId(), agent);
HashSet<String> current = new HashSet<>();
for (MemberSession session : rosterNow) {
current.add(session.terminalId());
Agent agent = live.get(session.terminalId());
AgentStatus status = agent == null ? AgentStatus.UNKNOWN : agent.status();
boolean accepted = messages.hasAcceptedDelivery(session.terminalId());
HealthSnapshot snapshot = new HealthSnapshot(session.state(), status, accepted, NOT_YET_OBSERVED,
messages.hasInboxMessage(session.terminalId()), agent != null, NOT_YET_OBSERVED,
NOT_YET_OBSERVED, NOT_YET_OBSERVED, NOT_YET_OBSERVED, NOT_YET_OBSERVED, NOT_YET_OBSERVED);
HealthDecision decision = decide(snapshot, priors.getOrDefault(session.terminalId(), HealthPrior.NONE),
clock.getAsLong());
priors.put(session.terminalId(), decision.prior());
reportTransition(session.terminalId(), decision.state());
}
priors.keySet().retainAll(current);
states.keySet().retainAll(current);
} catch (Throwable error) {
// A list failure is health evidence, and must never kill the monitor's only scheduler task.
log.warn("fleet health collection failed; will retry next tick", error);
} finally {
if (!scheduler.isShutdown()) {
scheduler.schedule(this::tick, intervalSeconds, TimeUnit.SECONDS);
}
}
}
void reportTransition(String target, HealthState next) {
HealthState previous = states.put(target, next);
if (previous == next) return;
if (fault(next)) {
log.warn("fleet health member={} state={} previous={}", target, next, previous);
} else if (previous != null && fault(previous)) {
log.info("fleet health member={} recovered state={} previous={}", target, next, previous);
}
// CB-580: a member entering GONE/NEVER_READY must not leave its waiting tickets pending
// forever. Fire exactly once per transition — never on a tick where the state is unchanged,
// which is what made the rejected commit call abandon() once per tick for as long as a
// member stayed terminal.
if (terminal(next)) {
failTerminalTarget(target, next);
}
}
private void failTerminalTarget(String target, HealthState state) {
String reason = "fleet health: member reached terminal state " + state.name();
RuntimeException last = null;
for (int attempt = 1; attempt <= MAX_FAIL_TARGET_ATTEMPTS; attempt++) {
try {
failTarget.accept(target, reason);
return;
} catch (RuntimeException error) {
last = error;
log.warn("fleet health: failTarget attempt {}/{} failed for member={} state={}",
attempt, MAX_FAIL_TARGET_ATTEMPTS, target, state, error);
}
}
log.warn("fleet health: giving up on failTarget for member={} state={} after {} attempts",
target, state, MAX_FAIL_TARGET_ATTEMPTS, last);
}
private static boolean terminal(HealthState state) {
return state == HealthState.GONE || state == HealthState.NEVER_READY;
}
private static boolean fault(HealthState state) {
return switch (state) {
case NEVER_READY, GONE, TURN_BOUNDARY_LOST, ERROR_ON_SCREEN, STALL_SUSPECTED,
MUTE, REPLY_STRANDED, DELEGATION_ORPHANED, CONTROL_LINK_DOWN -> true;
default -> false;
};
}
public static String coverage(boolean enabled, boolean notificationConfigured) {
return !enabled ? "off" : notificationConfigured ? "full" : "detection-only";
}
}
@@ -1,59 +0,0 @@
package dev.ltms.fleet.herdr;
import com.fasterxml.jackson.databind.JsonNode;
import java.util.Map;
/**
* Resolves which herdr pane a process belongs to — the herdr half of connection-based MCP
* identity (CB-105). Given the PID that opened an MCP connection, {@link #terminalForPid} finds
* the agent pane whose process tree contains it, so {@code bridged} can tell <em>which worker</em>
* is calling without the worker sending anything spoofable.
*
* <p>herdr owns the PID→pane truth: {@code pane.process_info} reports each pane's {@code shell_pid}
* and foreground process PIDs. This scans agent panes; a spawn-time {@code pid→terminal} cache is
* the obvious optimization once wired into {@code ClaudeCodeLauncher}.
*/
public final class PaneLocator {
private final HerdrClient herdr;
public PaneLocator(HerdrClient herdr) {
this.herdr = herdr;
}
/**
* The {@code terminal_id} of the agent pane whose process tree contains {@code pid}, or
* {@code null} if no agent pane owns it (e.g. the caller is the primary, or off-host).
*/
public String terminalForPid(long pid) {
if (pid <= 0) {
return null;
}
for (JsonNode pane : herdr.call("pane.list", Map.of()).path("panes")) {
String paneId = pane.path("pane_id").asText(null);
if (paneId != null && paneOwnsPid(paneId, pid)) {
return pane.path("terminal_id").asText(null);
}
}
return null;
}
private boolean paneOwnsPid(String paneId, long pid) {
JsonNode info;
try {
info = herdr.call("pane.process_info", Map.of("pane_id", paneId)).path("process_info");
} catch (HerdrException e) {
return false; // pane vanished mid-scan — just skip it
}
if (info.path("shell_pid").asLong(-1) == pid) {
return true;
}
for (JsonNode p : info.path("foreground_processes")) {
if (p.path("pid").asLong(-1) == pid) {
return true;
}
}
return false;
}
}
@@ -1,372 +0,0 @@
package dev.ltms.fleet.inject;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.TurnToken;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
/**
* The CB-106 completion fallback: bridges the {@link Injector}'s turn-completion signal to the
* {@link Rendezvous} so a blocking {@code fleet_send} resolves even when the worker finishes its
* task without ever calling {@code fleet_reply} — the common case for a real delegated coding task.
*
* <p>On a confirmed {@code working → idle} boundary it scrapes the worker's recent transcript and
* resolves the awaiting send with that tail (a {@link Rendezvous.Kind#COMPLETION} resolution, so the
* caller can tell a scrape from a structured reply). It scrapes only when a send is actually waiting
* — a fleet worker's own turns, or a send that already timed out, cost no herdr traffic. An explicit
* {@code fleet_reply} that raced in first wins; {@link Rendezvous#resolveCompletion} is then a no-op.
*
* <p>It also handles the CB-109 stall signal ({@link #onTurnFailed}): a worker that ran a turn then
* wedged in an {@code unknown} state resolves the send as a failure (with the error screen as
* context) rather than leaving it to time out.
*
* <p>The scrape is cleaned to the last {@code ⏺} assistant block (stripping TUI chrome) and guarded
* against misattribution (CB-115): the pane content is baselined on delivery ({@link #onDelivered}),
* and a completion whose scrape is unchanged from that baseline — the previous turn's wind-down
* sampled as this turn's boundary on a rapid back-to-back send — is suppressed rather than resolving
* the send with a stale answer.
*
* <p><strong>Waiter-specific resolution (CB-116).</strong> On delivery we also capture the exact
* {@link Rendezvous} waiter this turn belongs to, and the completion/failure fallbacks resolve
* <em>that</em> waiter — never "whatever send is waiting now". A completion fallback runs on a virtual
* thread and can land after the worker's {@code fleet_reply} already resolved the turn and the
* <em>next</em> send opened its own waiter on the same session; resolving the current waiter would
* then deliver turn N's stale scrape as turn N+1's answer. Targeting the captured waiter makes a late
* completion a harmless no-op (its waiter is already done) instead of a cross-turn stale reply.
*
* <p>Wired as the {@link Injector}'s {@link TurnListener}; the handlers hand off to a virtual thread
* so the scrape's herdr round-trip never stalls the status poller. The captured waiter is read on the
* poller thread (before any next-turn delivery can overwrite it) and passed into the virtual thread.
*/
public final class CompletionResolver implements TurnListener {
private static final Logger log = LoggerFactory.getLogger(CompletionResolver.class);
/**
* herdr {@code agent.read} source for the completion scrape. {@code recent} returns the tail of
* the transcript (the worker's last output), which is what a delegator wants when the worker
* didn't structure a reply.
*/
static final String SCRAPE_SOURCE = "recent";
/** Cap the scraped tail so a long transcript can't return an unbounded blob. */
static final int MAX_SCRAPE_CHARS = 4000;
private static final String CLIPPED_PANE_TAIL_MARKER =
"[Pane tail clipped: member did not call fleet_reply.]";
private final AgentControl agents;
private final Rendezvous rendezvous;
private final ExhaustedPatternLookup exhaustedPatterns;
private final ExhaustionSink exhaustionSink;
/**
* Per-target record of the turn currently in flight: the exact {@link Rendezvous} waiter its
* delivering send opened, plus the assistant block present when it was delivered.
*
* <p>The {@code waiter} is what makes a late fallback safe (CB-116): we resolve it, not "whoever
* is waiting now", so a completion that fires after the next send has opened its own waiter is a
* no-op rather than a cross-turn stale reply. The {@code baseline} is the CB-115 staleness
* reference: a completion scrape equal to it means the worker produced no new output (the previous
* turn's wind-down sampled as this boundary), so it is suppressed. Overwritten on each delivery;
* cleared when the turn resolves. Package-private so tests can capture and replay a specific turn.
*/
record InFlight(CompletableFuture<Rendezvous.Resolution> waiter, String baseline) {
}
private final ConcurrentHashMap<String, InFlight> inFlight = new ConcurrentHashMap<>();
/**
* @param exhaustedPatterns CB-578 stage A: per-target lookup for a profile's configured
* usage-limit refusal pattern. Required — there is deliberately no
* defaulting overload; a caller that does not want the classification
* must pass an explicit inert value ({@link ExhaustedPatternLookup#none()}).
* @param exhaustionSink CB-578 stage B: notified when a {@code BACKEND_EXHAUSTED}
* classification actually resolves a waiter. Required for the same
* reason as {@code exhaustedPatterns} — pass {@link ExhaustionSink#none()}
* to opt out.
*/
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink) {
this.agents = agents;
this.rendezvous = rendezvous;
this.exhaustedPatterns = Objects.requireNonNull(exhaustedPatterns, "exhaustedPatterns");
this.exhaustionSink = Objects.requireNonNull(exhaustionSink, "exhaustionSink");
}
@Override
public void onDelivered(String target, TurnToken token) {
// Capture the exact waiter this turn belongs to (CB-116) and snapshot the pane's pre-turn
// content — what it shows *before* the just-delivered turn produces output — as the staleness
// reference (CB-115). Done synchronously (like the delivering send itself) so both are in
// place before this turn's completion can fire.
captureBaseline(target, token);
}
/** Capture the in-flight turn: its waiter and pre-turn baseline (the testable core of {@link #onDelivered}). */
void captureBaseline(String target, TurnToken token) {
CompletableFuture<Rendezvous.Resolution> waiter = token.waiter();
if (waiter == null) {
inFlight.remove(target); // no send is waiting on this delivery — nothing to resolve later
return;
}
String baseline;
try {
// Clip to the same cap resolve() applies to the tail (line ~134): the CB-115 misattribution
// guard compares baseline.equals(tail), so both sides must be the same capped representation.
// An unclipped baseline vs a clipped tail would never match for a >MAX_SCRAPE_CHARS block,
// defeating the guard and letting a stale completion resolve the send.
baseline = clip(lastAssistantBlock(agents.read(target, SCRAPE_SOURCE)));
} catch (RuntimeException e) {
baseline = null; // fail open: no baseline ⇒ no suppression
log.debug("delivery baseline for {} failed: {}", target, e.getMessage());
}
inFlight.put(target, new InFlight(waiter, baseline));
}
/** The turn currently baselined for {@code target}, or {@code null} — a test hook for the captureBaseline path. */
InFlight inFlight(String target) {
return inFlight.get(target);
}
@Override
public void onTurnComplete(String target) {
// Read the in-flight turn on the poller thread — before any next-turn delivery can overwrite
// it — then off-load the scrape (a herdr round-trip we must not block polling on) to a vthread.
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("completion-" + target).start(() -> resolve(target, turn));
}
/**
* Resolve the completed turn before adapter housekeeping can erase its rendered output. This is
* intentionally synchronous and used only when a post-turn context reset is enabled; the normal
* path remains off-loaded so polling is not blocked by a scrape.
*/
public void resolveBeforePostAction(String target) {
resolve(target, inFlight.get(target));
}
@Override
public void onTurnFailed(String target) {
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("turn-failed-" + target).start(() -> fail(target, turn, null));
}
@Override
public void onTurnFailed(String target, String reason) {
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("turn-failed-" + target).start(() -> fail(target, turn, reason));
}
/** Synchronous resolve (the unit-testable core of {@link #onTurnComplete}). */
void resolve(String target, InFlight turn) {
CompletableFuture<Rendezvous.Resolution> waiter = turn == null ? null : turn.waiter();
if (waiter == null || waiter.isDone()) {
// Nobody is blocked on THIS turn (it had no send, or its fleet_reply already won). Skip
// the scrape; resolving the current waiter here would be the CB-116 cross-turn stale reply.
inFlight.remove(target, turn);
return;
}
String tail;
String assistantBlock = null;
int originalLength = 0;
boolean clipped = false;
boolean scrapeFailed = false;
try {
assistantBlock = lastAssistantBlock(agents.read(target, SCRAPE_SOURCE));
originalLength = assistantBlock.strip().length();
clipped = originalLength > MAX_SCRAPE_CHARS;
tail = clip(assistantBlock);
} catch (RuntimeException e) {
// The worker finished but we couldn't read its screen — still resolve the send so the
// caller unblocks; an empty tail beats hanging until the caller's timeout.
log.warn("completion scrape for {} failed; resolving with an empty tail: {}",
target, e.getMessage());
tail = "";
scrapeFailed = true;
}
// Misattribution guard (CB-115): if the scrape is byte-identical to the pane content at
// delivery, this turn produced no new output — the boundary belongs to the previous turn's
// wind-down (common on rapid back-to-back sends). Suppress rather than resolve the send with
// a stale answer; the real fleet_reply (or a later genuine completion) resolves it instead.
// A scrape that failed to read is exempt — an empty tail there is "couldn't see", not "no change".
String baseline = turn.baseline();
if (!scrapeFailed && baseline != null && baseline.equals(tail)) {
log.debug("suppressing misattributed completion for {} (no output change since delivery)",
target);
return; // keep the in-flight record: a later genuine completion still needs it
}
// CB-578 stage A: a turn that ended with no fleet_reply AND whose scrape matches the
// backend's configured usage-limit pattern is a refusal, not an answer. Classify it as
// BACKEND_EXHAUSTED rather than handing the caller a scrape that reads like a real reply.
if (!scrapeFailed) {
Pattern exhausted = exhaustedPatterns.patternFor(target);
String matchedLine = exhausted == null ? null : firstMatchingLine(assistantBlock, exhausted);
if (matchedLine != null) {
String reason = "backend exhausted (usage limit): " + matchedLine;
if (rendezvous.resolveExhausted(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("completion for {} classified BACKEND_EXHAUSTED (no fleet_reply; scrape "
+ "matched the profile's exhausted pattern): {}", target, reason);
// CB-578 stage B: only on the resolution that actually won the race — a late
// duplicate must never quarantine a credential twice for one refusal.
exhaustionSink.onExhausted(target, reason);
}
return;
}
}
String completion = clipped ? tail + "\n" + CLIPPED_PANE_TAIL_MARKER : tail;
if (rendezvous.resolveCompletion(waiter, completion)) {
inFlight.remove(target, turn);
if (clipped) {
log.warn("completion scrape for {} clipped from {} chars to the {} char cap; "
+ "member did not call fleet_reply, so the pane tail is partial",
target, originalLength, MAX_SCRAPE_CHARS);
}
log.debug("resolved send to {} via turn-completion fallback ({} chars scraped)",
target, tail.length());
}
}
/** Synchronous fail (the unit-testable core of {@link #onTurnFailed}). */
void fail(String target, InFlight turn) {
fail(target, turn, null);
}
/** Synchronous fail with an optional reason supplied by a dropped worker queue. */
void fail(String target, InFlight turn, String explicitReason) {
// A never-delivered readiness failure has no in-flight record but still has a blocked send;
// fall back to the currently-registered waiter (unambiguous — that send never completed, so
// no next turn exists to confuse it with).
CompletableFuture<Rendezvous.Resolution> waiter =
turn != null ? turn.waiter() : rendezvous.currentWaiter(target);
if (waiter == null || waiter.isDone()) {
inFlight.remove(target, turn); // nobody blocked on this worker — nothing to fail
return;
}
String reason = explicitReason;
if (reason == null || reason.isBlank()) {
try {
reason = clip(agents.read(target, SCRAPE_SOURCE));
} catch (RuntimeException e) {
reason = "";
}
if (reason.isBlank()) {
// No screen to scrape — either the worker is stuck (CB-109) or gone (CB-110).
reason = "worker did not reply; its turn ended in an unrecoverable state "
+ "(worker unreachable or stuck)";
}
}
if (rendezvous.resolveFailure(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("failing send to {} via turn-stall fallback: {}", target, reason);
}
}
/**
* The first line of {@code text} matching {@code pattern}, stripped — the CB-578 stage A
* evidence carried in a {@code BACKEND_EXHAUSTED} reason so the operator sees the real refusal
* text, never a generic label. {@code null} if no line matches.
*/
static String firstMatchingLine(String text, Pattern pattern) {
if (text == null || text.isEmpty()) return null;
for (String line : text.split("\n", -1)) {
if (pattern.matcher(line).find()) {
return line.strip();
}
}
return null;
}
/**
* Coverage summary for the CB-578 stage A exhausted-pattern classification, logged at startup
* the way {@link dev.ltms.fleet.health.FleetHealthMonitor#coverage} is — so an operator can
* see whether the classification is on, and for which profiles, without reading every
* profile's config by hand.
*
* @param allProfiles every configured profile name
* @param configuredProfiles the subset of {@code allProfiles} that carry an exhausted pattern
*/
public static String coverage(Set<String> allProfiles, Set<String> configuredProfiles) {
if (configuredProfiles.isEmpty()) {
return "off (no profile has an exhaustedPattern configured; profiles: " + sorted(allProfiles) + ")";
}
Set<String> unconfigured = new TreeSet<>(allProfiles);
unconfigured.removeAll(configuredProfiles);
return unconfigured.isEmpty()
? "full (all profiles configured: " + sorted(allProfiles) + ")"
: "partial (configured: " + sorted(configuredProfiles) + "; not configured: " + sorted(unconfigured) + ")";
}
private static List<String> sorted(Set<String> names) {
return names.stream().sorted().toList();
}
private static String clip(String s) {
if (s == null) return "";
String trimmed = s.strip();
return trimmed.length() <= MAX_SCRAPE_CHARS
? trimmed
: trimmed.substring(trimmed.length() - MAX_SCRAPE_CHARS);
}
/**
* Extract the last assistant message from a raw Claude Code pane scrape (CB-115). Claude Code
* prefixes each assistant turn with {@code ⏺}; the delegator wants that answer, not the TUI
* chrome around it. Take everything from the final {@code ⏺} onward and stop at the <em>first</em>
* hard interface boundary below it — the spinner/status line, input box, {@code ❯} prompt (which
* may echo the <em>next</em> turn's text), footer, or tips/warnings. Stopping at the first
* boundary (rather than trimming only trailing chrome) is what keeps a following turn's echoed
* prompt out of this reply. Blank lines are not boundaries, so a multi-paragraph answer survives;
* trailing blanks are trimmed at the end. With no {@code ⏺} marker (an unusual render) the whole
* text is scanned the same way, so we never lose the reply.
*
* <p>Package-private and pure so it is unit-testable without herdr.
*/
static String lastAssistantBlock(String raw) {
if (raw == null || raw.isBlank()) return "";
int marker = raw.lastIndexOf('⏺');
String block = marker >= 0 ? raw.substring(marker + 1) : raw;
StringBuilder out = new StringBuilder();
int kept = 0;
for (String line : block.split("\n", -1)) {
if (isBoundary(line)) break; // first TUI boundary ends the assistant message
if (kept++ > 0) out.append('\n');
out.append(line);
}
return out.toString().strip();
}
/**
* A hard TUI boundary line that marks the end of an assistant message and the start of interface
* chrome (input box, prompt, spinner, footer, tips/warnings). Blank lines are <em>not</em>
* boundaries — an answer may contain them — so they are kept and trimmed only if trailing.
*/
private static boolean isBoundary(String line) {
String t = line.strip();
if (t.isEmpty()) return false;
// A horizontal rule / all box-drawing separators (e.g. "──────").
if (t.chars().allMatch(c -> c == '─' || c == '—' || c == '━' || c == '═' || c == '-')) {
return true;
}
String lower = t.toLowerCase();
return t.startsWith("╭") || t.startsWith("│") || t.startsWith("╰") || t.startsWith("┌")
|| t.startsWith("└") || t.startsWith("❯") || t.startsWith("⏵")
|| t.startsWith("⎿") || t.startsWith("⚠")
// Status/spinner lines Claude Code renders below a settled or in-flight turn,
// e.g. "✻ Baked for 21s", "✶ Forming…".
|| t.startsWith("✻") || t.startsWith("✳") || t.startsWith("✽") || t.startsWith("·")
|| t.startsWith("●") || t.startsWith("◐") || t.startsWith("✢") || t.startsWith("✶")
|| lower.contains("auto mode") || lower.contains("for shortcuts")
|| lower.contains("esc to interrupt") || lower.contains("bypass permissions");
}
}
@@ -1,31 +0,0 @@
package dev.ltms.fleet.inject;
/**
* Notified when {@link CompletionResolver} actually delivers a {@code BACKEND_EXHAUSTED}
* classification to a waiting send (CB-578 stage B) — never on a race that lost (see
* {@link CompletionResolver#resolve}, which only calls this after
* {@code Rendezvous.resolveExhausted} returns {@code true}).
*
* <p>{@link CompletionResolver} knows only {@code target} (a herdr terminal id); it has no notion of
* profiles or credentials, so mapping {@code target} to whatever should be quarantined is entirely
* the sink's job — see {@code Fleetd.main}'s wiring, which resolves target → session → profile →
* {@code effectiveCredentialId()} and calls {@code BackendQuarantine.quarantine} on it.
*/
@FunctionalInterface
public interface ExhaustionSink {
/**
* @param target the herdr terminal id whose turn was classified {@code BACKEND_EXHAUSTED}
* @param reason the matched-line reason carried by the classification
*/
void onExhausted(String target, String reason);
/**
* Inert sink — nothing happens on exhaustion. The explicit stand-in a caller (or a test not
* exercising this feature) passes instead of a defaulting overload, exactly like
* {@link ExhaustedPatternLookup#none()}.
*/
static ExhaustionSink none() {
return (target, reason) -> { };
}
}
@@ -1,546 +0,0 @@
package dev.ltms.fleet.member;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.Agent;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.peer.Capability;
import dev.ltms.fleet.peer.PeerLauncher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import java.util.function.Function;
import java.util.function.LongSupplier;
import java.util.function.Supplier;
/**
* The {@link HerdrPeerLauncher} adapter for <strong>Claude Code</strong> — the safe path from a
* delegation request to a running off-subscription Claude.
*
* <p>Everything transport-related (tab/pane placement, the CB-306 spawn-readiness gate, unique
* naming, CB-117 orphan reap, teardown, listing, cwd resolution) lives in the base. This class
* supplies only the two Claude-specific seams:
* <ul>
* <li>the {@code claude} name prefix (so reap matches {@code claude-*} panes, never another
* adapter's), and</li>
* <li>{@link #buildLaunch}, which encodes the subscription boundary: build the worker env with
* {@code ANTHROPIC_BASE_URL}, assert that host is on the allowlist <em>before</em> touching
* herdr, and mount the bridge MCP + reply charter as inline launch flags. A worker's base_url
* lives in the env map handed to herdr and nowhere else; {@code bridged}'s own environment is
* never mutated, and nothing is written to the worker's profile.</li>
* </ul>
*/
public final class ClaudeCodeLauncher extends HerdrPeerLauncher {
/** Label prefix for this adapter's herdr agent names (drives naming + orphan reap). */
private static final String NAME_PREFIX = "claude";
private static final Logger log = LoggerFactory.getLogger(ClaudeCodeLauncher.class);
private final SubscriptionGuard guard;
/**
* Production constructor — disables the spawn-ready gate ({@code spawnReadyTimeoutMs == 0}) so
* existing deployments and tests keep the legacy non-blocking spawn semantics.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env) {
this(agents, spaces, guard, profiles, defaultProfile, env, 0,
System::currentTimeMillis, () -> sleepUninterruptibly(300));
}
/**
* Production constructor with spawn-ready gate enabled. The gate polls {@code agents.status()}
* until the pane reports an injectable state or {@code spawnReadyTimeoutMs} elapses.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs, long spawnReadyPollMs) {
this(agents, spaces, guard, profiles, defaultProfile, env,
spawnReadyTimeoutMs, spawnReadyPollMs, null);
}
/**
* Production constructor carrying the fleet-wide tab-label template (CB-557). The template comes
* from {@code fleet.tabLabel}, which a profile cannot know because it names the member's
* <em>role</em>; a profile may still override it with its own {@code tabLabel}.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs, long spawnReadyPollMs,
Supplier<FleetConfig.Fleet> fleet) {
this(agents, spaces, guard, profiles, defaultProfile, env,
spawnReadyTimeoutMs,
System::currentTimeMillis, () -> sleepUninterruptibly(spawnReadyPollMs),
fleet);
}
/**
* Production constructor, plus the CB-596 {@code memberCredentials} policy supplier.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs, long spawnReadyPollMs,
Supplier<FleetConfig.Fleet> fleet,
Supplier<FleetConfig.MemberCredentials> memberCredentials) {
this(agents, spaces, guard, profiles, defaultProfile, env,
spawnReadyTimeoutMs,
System::currentTimeMillis, () -> sleepUninterruptibly(spawnReadyPollMs),
fleet, memberCredentials);
}
/**
* Full testability constructor. Every injectable collaborator is explicit so unit tests supply
* fakes for the clock ({@code nowMillis}) and poll-loop wait ({@code sleeper}). The
* {@code sleeper} is never called when the gate is disabled ({@code spawnReadyTimeoutMs == 0}).
*
* @param agents herdr agent control (start, status, close)
* @param spaces workspace / tab control (ensure, create, close)
* @param guard subscription-boundary guard (checked before spawning)
* @param profiles configured worker profiles
* @param defaultProfile profile a no-argument spawn uses (nullable)
* @param env host env lookup (injectable for tests)
* @param spawnReadyTimeoutMs max ms to wait for injectable state (0 disables the gate)
* @param nowMillis monotonic clock source (e.g. {@code System::currentTimeMillis})
* @param sleeper sleep/wait hook (e.g. {@code () -> Thread.sleep(pollMs)}); it
* already encodes the poll interval, so the 8th positional argument
* (poll ms) is accepted for API symmetry but otherwise unused here
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs,
LongSupplier nowMillis, Runnable sleeper) {
this(agents, spaces, guard, profiles, defaultProfile, env,
spawnReadyTimeoutMs, nowMillis, sleeper, null);
}
/**
* Full testability constructor, plus the fleet-wide tab-label template (CB-557).
*
* @param fleet live fleet config, read once for each spawn
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs,
LongSupplier nowMillis, Runnable sleeper,
Supplier<FleetConfig.Fleet> fleet) {
super(NAME_PREFIX, agents, spaces, profiles, defaultProfile, env,
spawnReadyTimeoutMs, nowMillis, sleeper, fleet);
this.guard = guard;
}
/**
* Full testability constructor, plus the CB-596 {@code memberCredentials} policy supplier.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs,
LongSupplier nowMillis, Runnable sleeper,
Supplier<FleetConfig.Fleet> fleet,
Supplier<FleetConfig.MemberCredentials> memberCredentials) {
super(NAME_PREFIX, agents, spaces, profiles, defaultProfile, env,
spawnReadyTimeoutMs, nowMillis, sleeper, fleet, memberCredentials);
this.guard = guard;
}
/**
* Full testability constructor, plus an injectable host-env-names source for the CB-596
* criterion-4 gap detector. Test seam only — every production call site leaves this at the
* default (the real {@code System.getenv()} key set) via the constructor above.
*/
public ClaudeCodeLauncher(AgentControl agents, WorkspaceControl spaces, SubscriptionGuard guard,
Map<String, FleetConfig.Profile> profiles, String defaultProfile,
Function<String, String> env,
long spawnReadyTimeoutMs,
LongSupplier nowMillis, Runnable sleeper,
Supplier<FleetConfig.Fleet> fleet,
Supplier<FleetConfig.MemberCredentials> memberCredentials,
Supplier<Set<String>> hostEnvNames) {
super(NAME_PREFIX, agents, spaces, profiles, defaultProfile, env,
spawnReadyTimeoutMs, nowMillis, sleeper, fleet, memberCredentials, hostEnvNames);
this.guard = guard;
}
/**
* {@inheritDoc}
*
* <p>The spawn sequence encodes the subscription boundary: assert the profile's base_url is on
* the allowlist <em>before</em> any herdr call, then build the worker env with
* {@code ANTHROPIC_*}, the parity-neutral git-forge grant, and the bridge MCP + reply charter
* mounted as inline launch flags. When the request carries session identity (CB-547a) it is
* applied here — see {@link #applySessionIdentity}.
*/
@Override
protected Launch buildLaunch(FleetConfig.Profile cfg, LaunchSpec spec) {
// CB-539: a profile may deliberately opt into the subscription (subscription: true) when no
// off-subscription endpoint exists for it — e.g. `sonnet` on `ccs`. That profile gets no
// ANTHROPIC_BASE_URL/AUTH_TOKEN (there is nothing to point them at) and the guard's base_url
// requirement is skipped FOR IT ONLY. Every other profile keeps the hard boundary below.
boolean onSubscription = cfg.isSubscription();
String baseUrl = cfg.baseUrl();
if (onSubscription) {
// NO SILENT CONTRADICTION: subscription:true + a baseUrl state opposite intents; refuse
// loudly rather than pick a winner.
if (baseUrl != null && !baseUrl.isBlank()) {
throw new IllegalStateException("profile '" + cfg.profile()
+ "' sets both subscription: true and a baseUrl ('" + baseUrl + "') — the two "
+ "are contradictory: a subscription profile must not point at an endpoint. "
+ "Drop baseUrl, or drop subscription: true.");
}
// Visible without anyone going looking for it: this worker bills the subscription.
log.warn("spawning profile '{}' on the Claude subscription (subscription: true) — this "
+ "worker WILL bill the operator's subscription", cfg.profile());
} else {
guard.assertWorker(baseUrl); // hard stop before we spawn anything
}
// CB-634: the IDE guidance is delivered as an on-disk CLAUDE.local.md overlay, NOT through
// the charter — the charter returns to role -> reply only. Best-effort: a failed overlay
// must never fail the spawn, and `writeIdeOverlay` no-ops unless the cwd is a provisioned
// worktree (see its .git-file safety gate). The overlay pins, and the auto-open opens, the
// module dir (this repo's pom is in `bridged/`, not at the worktree root) — see ideProjectPath.
if (cfg.hasIdeMcp()) {
String projectPath = PeerLauncher.ideProjectPath(spec.cwd(), cfg.ideProjectDir());
writeIdeOverlay(spec.cwd(), projectPath);
PeerLauncher.openInIde(projectPath, cfg.ideOpenCommand(), log);
}
Map<String, String> workerEnv = baseEnv(cfg);
if (onSubscription) {
// CB-542 belt-and-braces: on the subscription path no guard vets these two keys, and the
// profile's env: is layered in by baseEnv — so strip any that rode in there. Config load
// already rejects this (loudly, naming the profile); this makes the boundary hold even
// for a profile built in code that never passed through that validation.
workerEnv.remove("ANTHROPIC_BASE_URL");
workerEnv.remove("ANTHROPIC_AUTH_TOKEN");
} else {
workerEnv.put("ANTHROPIC_BASE_URL", baseUrl);
putIfPresent(workerEnv, "ANTHROPIC_AUTH_TOKEN", env.apply(cfg.tokenEnv()));
}
putIfPresent(workerEnv, "ANTHROPIC_MODEL", cfg.model());
putIfPresent(workerEnv, "CLAUDE_CONFIG_DIR", cfg.configDir());
applyGitToken(workerEnv, cfg);
// CB-547a: Claude Code can MINT its own session id, so bridged chooses it — a fresh spawn
// gets a UUID we pass as --session-id and return from agentSessionId(), so the resume
// handle is known BEFORE the agent has written anything; a resume spawn adopts its prior
// id via -r and passes no --session-id (the two conflict). Both are injected before the
// model flag so --model keeps outranking the operator's own argv.
// mutableArgv: argvWithFleet may hand back the profile's own (immutable) List.of when it
// has neither MCP nor a charter — session flags must be added into a list we own.
List<String> argv = mutableArgv(argvWithFleet(cfg, spec));
String agentSessionId = applySessionIdentity(argv, spec.sessionName(), spec.resumeSessionId());
return new Launch(workerEnv, argvWithAutoCompact(argvWithModel(argv, cfg), cfg), agentSessionId);
}
/**
* Add the Claude-specific session-identity flags to {@code argv} and return the peer's OWN
* session id — the resume handle. A resume request passes the prior id via {@code -r} and
* returns that id; a fresh named session mints a new UUID, passes it via {@code --session-id},
* and returns the mint. The bridge's logical name rides along as {@code -n} when present. When
* <em>no</em> identity is requested (sessionName and resumeSessionId both blank) this adds
* nothing and returns {@code null}, keeping the legacy no-identity launch byte-identical.
*/
private static String applySessionIdentity(List<String> argv, String sessionName, String resumeSessionId) {
boolean resuming = resumeSessionId != null && !resumeSessionId.isBlank();
boolean named = sessionName != null && !sessionName.isBlank();
if (!resuming && !named) {
return null; // no identity requested — keep the legacy launch byte-identical
}
if (named) {
argv.add("-n");
argv.add(sessionName);
}
if (resuming) {
argv.add("-r");
argv.add(resumeSessionId);
return resumeSessionId;
}
String minted = UUID.randomUUID().toString();
argv.add("--session-id");
argv.add(minted);
return minted;
}
/**
* The launch argv, plus an inline {@code --mcp-config} when {@code worker.mcpUrl} is set, the
* CB-617 charter flags, and {@code --agent <role>} when the role has an agent-definition file
* under the worker's cwd. Neither touches the profile's config; all are pure command-line flags.
* This inline-flag mount is Claude Code specific — other adapters mount MCP and instructions
* their own way.
*
* <p>CB-617: the role charter is operator-authored and often multi-line, so it can never be a
* single inline argv element — herdr refuses to shell-encode a multi-line argument
* ({@code invalid_agent_argument}). It is written to a temp file instead and mounted with
* {@code --append-system-prompt-file}, which this host confirms Claude Code accepts for a
* multi-line file.
*
* <p>CB-618: Claude Code refuses to start when BOTH {@code --append-system-prompt} and
* {@code --append-system-prompt-file} are on the command line ("Cannot use both ... Please use
* only one"), so the two charters can never travel on separate flags. When both are present they
* are concatenated into the one file, role charter first and reply charter last — last is where
* the reply rule must sit, because it is the rule that must survive. When only the reply charter
* is present it keeps its proven inline {@code --append-system-prompt} delivery, which is also
* the only form that reaches a member with no repo checkout.
*/
private List<String> argvWithFleet(FleetConfig.Profile cfg, LaunchSpec spec) {
String roleCharter = nonBlank(spec.roleCharter());
// CB-634: the IDE guidance is delivered as an on-disk overlay (writeIdeOverlay), not through
// the charter. The charter file is role -> reply only.
String replyCharter = nonBlank(spec.replyCharter());
Path agentFile = agentDefinitionFile(spec.cwd(), spec.role(), ".claude", "agents");
if (!cfg.mountsAnyMcp() && roleCharter == null
&& replyCharter == null && agentFile == null) {
return cfg.argv();
}
List<String> argv = mutableArgv(cfg.argv());
if (cfg.mountsAnyMcp()) {
argv.add("--mcp-config");
argv.add(mcpConfigJson(cfg));
}
// Combine the charters in order role -> reply, dropping any that are absent. When two
// or more survive they must ride one --append-system-prompt-file (CB-618 forbids the inline
// flag and the file flag together). A lone reply charter keeps its proven inline delivery.
List<String> charters = new java.util.ArrayList<>(2);
if (roleCharter != null) charters.add(roleCharter);
if (replyCharter != null) charters.add(replyCharter);
if (charters.size() == 1 && replyCharter != null && roleCharter == null) {
argv.add("--append-system-prompt");
argv.add(replyCharter);
} else if (!charters.isEmpty()) {
argv.add("--append-system-prompt-file");
argv.add(writeCharterFile(String.join("\n\n", charters)).toString());
}
if (agentFile != null) {
argv.add("--agent");
argv.add(spec.role().wireName());
}
return argv;
}
/**
* The {@code --mcp-config} JSON for this member: always the bridge mount when {@link
* FleetConfig.Profile#hasMcp()}, plus the IDE Index MCP as a second server named {@code
* intellij} when {@link FleetConfig.Profile#hasIdeMcp()} (CB-634). At least one is present —
* the caller only reaches here when {@link FleetConfig.Profile#mountsAnyMcp()} is true.
*/
private static String mcpConfigJson(FleetConfig.Profile cfg) {
StringBuilder servers = new StringBuilder();
if (cfg.hasMcp()) {
servers.append('"').append(PeerLauncher.MCP_MOUNT_NAME)
.append("\":{\"type\":\"http\",\"url\":\"").append(cfg.mcpUrl()).append("\"}");
}
if (cfg.hasIdeMcp()) {
if (servers.length() > 0) servers.append(',');
servers.append("\"intellij\":{\"type\":\"http\",\"url\":\"")
.append(cfg.ideMcpUrl()).append("\"}");
}
return "{\"mcpServers\":{" + servers + "}}";
}
/**
* Deliver the shared IDE guidance ({@link PeerLauncher#ideOverlayText}) as an on-disk
* {@code CLAUDE.local.md} overlay beside the project's own {@code CLAUDE.md} (CB-634), and
* register the overlay in the repository's common {@code info/exclude} so it never shows as
* untracked (git reads a worktree's excludes from the common dir, not the per-worktree gitdir).
*
* <p><strong>Safety gate:</strong> the overlay is written ONLY when {@code cwd/.git} is a
* <em>regular file</em> — a provisioned worktree keeps a {@code .git} FILE holding a
* {@code gitdir: <path>} pointer, while the primary's real checkout has a {@code .git}
* DIRECTORY. Returning without writing when {@code .git} is a directory is the whole safety of
* the feature: it must never write into a non-worktree cwd, i.e. never clobber a project that
* does not want the overlay.
*
* <p>Best-effort: a failure is logged at debug and swallowed — a failed overlay must never fail
* the spawn.
*
* @param cwd the member's worktree root, where the {@code CLAUDE.local.md} file is written
* @param projectPath the module dir the overlay pins {@code project_path} to (see
* {@link PeerLauncher#ideProjectPath}); equals {@code cwd} when no module subdir
*/
private static void writeIdeOverlay(String cwd, String projectPath) {
try {
Path dotGit = Path.of(cwd, ".git");
if (!Files.isRegularFile(dotGit)) {
// Not a provisioned worktree (primary's real checkout has a .git directory, or the
// cwd is not a repo at all). Never write into it.
return;
}
// The overlay FILE lives at the worktree root (claude-code's cwd), but its CONTENT pins
// project_path to the module dir the IDE opened (projectPath), not the worktree root.
Files.writeString(Path.of(cwd, "CLAUDE.local.md"), PeerLauncher.ideOverlayText(projectPath));
String gitdirLine = Files.readString(dotGit).trim();
Path gitDir = Path.of(gitdirLine.replaceFirst("^gitdir:\\s*", ""));
if (!gitDir.isAbsolute()) {
gitDir = Path.of(cwd).resolve(gitDir).normalize();
}
// git reads info/exclude from the COMMON dir, never the per-worktree gitdir (only
// info/sparse-checkout is per-worktree). A provisioned worktree's gitdir is
// <common>/worktrees/<name>, so the common dir is two levels up; writing the entry into
// the per-worktree gitdir leaves it un-honoured and the overlay shows as untracked.
Path commonDir = gitDir;
if (gitDir.getParent() != null && gitDir.getParent().getFileName() != null
&& "worktrees".equals(gitDir.getParent().getFileName().toString())) {
commonDir = gitDir.getParent().getParent();
}
Path exclude = commonDir.resolve("info").resolve("exclude");
Files.createDirectories(exclude.getParent());
String overlayLine = "CLAUDE.local.md";
if (!Files.exists(exclude) || Files.readAllLines(exclude).stream().noneMatch(overlayLine::equals)) {
Files.writeString(exclude, (Files.exists(exclude) ? System.lineSeparator() : "")
+ overlayLine + System.lineSeparator());
}
} catch (Exception e) {
log.debug("cannot write IDE overlay into worktree '{}'", cwd, e);
}
}
/** {@code s}, or {@code null} when {@code s} is null/blank — the charter-presence test used above. */
private static String nonBlank(String s) {
return (s == null || s.isBlank()) ? null : s;
}
/**
* Write the role charter to a fresh temp file so it can be mounted with
* {@code --append-system-prompt-file} instead of riding inline in argv (CB-617). Best-effort
* cleaned via {@code deleteOnExit} — the same disposable-worker-config cleanup
* {@link OpenCodeLauncher#writeConfig} already uses for its charter file, since the process that
* reads this file (the spawned peer) outlives this JVM call and there is no spawn-scoped teardown
* hook to delete it synchronously.
*/
private static Path writeCharterFile(String charterText) {
try {
Path file = Files.createTempFile("bridged-role-charter-", ".md");
Files.writeString(file, charterText);
file.toFile().deleteOnExit();
return file;
} catch (IOException e) {
throw new UncheckedIOException("cannot write role charter temp file", e);
}
}
/**
* Pin the model on the command line as well as in {@code ANTHROPIC_MODEL} (CB-533).
*
* <p>The env var alone is not a reliable pin for this adapter, because the argv is usually a
* launcher rather than {@code claude} itself — {@code ["ccs", "<profile>"]} — and {@code ccs}
* exports its profile's own model family ({@code ANTHROPIC_MODEL}, {@code DEFAULT_OPUS/SONNET/
* HAIKU}, {@code CLAUDE_CODE_SUBAGENT_MODEL}) over whatever it inherited. A worker profile that
* set {@code model:} therefore got silently overruled by its own launcher. Claude Code's
* {@code --model} flag outranks the environment, and {@code ccs <profile> [claude-args...]}
* passes trailing arguments through, so the flag survives the wrapper.
*
* <p>Appended last so it also outranks anything in the operator's own {@code argv}. Profiles
* that deliberately leave {@code model:} unset (letting {@code ccs} own model selection, as
* {@code gx10} does) are untouched — this adds nothing when there is nothing to add. This is
* the {@code kind: claude} counterpart of the opencode adapter's {@code -m provider/model}.
*/
private static List<String> argvWithModel(List<String> argv, FleetConfig.Profile cfg) {
if (cfg.model() == null || cfg.model().isBlank()) {
return argv;
}
List<String> withModel = mutableArgv(argv);
withModel.add("--model");
withModel.add(cfg.model());
return withModel;
}
/**
* Pin a bounded auto-compaction window on the command line via {@code --autocompact <tokens>},
* opt-in per profile (CB-634's sibling ticket: a member that runs out of context dies mid-turn
* and its {@code fleet_reply} — the whole point of the turn — is lost with it; opencode already
* forces {@code compaction.auto: true} unconditionally, CB-523, but Claude Code has no equivalent
* and runs at the backend's own default window).
*
* <p>Mirrors {@link #argvWithModel}: appended after it, so it survives the {@code ccs <profile>}
* wrapper the same way {@code --model} does, and outranks env/settings and the operator's own
* {@code argv}. Verified: {@code claude 2.1.241 --help} lists {@code --autocompact <auto|tokens>}
* (either the literal {@code auto}, or an integer 100k–1M) — {@link FleetConfig#load} rejects a
* configured value outside that band before this ever runs, so the flag Claude Code receives here
* is always in range.
*/
private static List<String> argvWithAutoCompact(List<String> argv, FleetConfig.Profile cfg) {
if (cfg.autoCompactWindow() == null) {
return argv;
}
List<String> withAutoCompact = mutableArgv(argv);
withAutoCompact.add("--autocompact");
withAutoCompact.add(String.valueOf(cfg.autoCompactWindow()));
return withAutoCompact;
}
// --- Agent-returning convenience spawns (used by callers/tests that want the herdr Agent) ---
/** Spawn a worker for the default profile in the resolved default cwd. */
public Agent spawn() {
return spawnInternal(null, null, null);
}
/** Spawn a worker for a named profile (null → default) in the resolved default cwd. */
public Agent spawn(String profileName) {
return spawnInternal(profileName, null, null);
}
/** Spawn a worker for a named profile with an explicit requested/caller cwd (CB-112). */
public Agent spawn(String profileName, String requestedCwd, String callerCwd) {
return spawnInternal(profileName, requestedCwd, callerCwd);
}
// --- capabilities --------------------------------------------------------------------------
@Override
public Set<Capability> capabilities() {
Set<Capability> caps = EnumSet.of(Capability.MID_TURN_ASK, Capability.WORKTREE,
Capability.CONTEXT_RESET, Capability.ORPHAN_REAP,
Capability.SESSION_NAME, Capability.SESSION_RESUME);
if (hasGitTokenProfile()) {
caps.add(Capability.SELF_PR);
}
return Set.copyOf(caps);
}
@Override
public boolean clearContext(String id) {
String target = agentTarget(id);
if (target == null) {
return false;
}
// This deliberately bypasses Injector: /clear is housekeeping, not a delegated turn.
agents().send(target, "/clear");
return true;
}
/** Whether any configured profile opts into a git-forge token (required for {@link Capability#SELF_PR}). */
private boolean hasGitTokenProfile() {
return profileConfigs().stream().anyMatch(FleetConfig.Profile::hasGitToken);
}
// --- CB-117 reap predicate (Claude prefix), kept for direct unit testing -------------------
/**
* Whether {@code name} is a Claude Code bridge worker started by a <em>different</em> process
* than {@code currentNonce}. A thin {@code claude}-prefix binding of
* {@link HerdrPeerLauncher#isForeignWorker(String, String, String)}.
*/
static boolean isForeignWorker(String name, String currentNonce) {
return HerdrPeerLauncher.isForeignWorker(NAME_PREFIX, name, currentNonce);
}
}
@@ -1,122 +0,0 @@
package dev.ltms.fleet.member;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
/**
* Resolves the opencode session id for a bridged worker from opencode's on-disk storage — the
* only place this adapter touches opencode's private layout, and deliberately the <em>only</em>
* class that does.
*
* <p><strong>Why this is isolated behind one seam.</strong> The layout is version-coupled and not a
* stable contract: opencode writes one JSON file per session under
* {@code <storageRoot>/session/<projectID>/<ses_*.json>}, and each record carries a
* {@code "version"} field (e.g. {@code "1.1.31"}), so the exact directory shape, file naming, and
* field names can move between opencode releases. opencode also ships a headless HTTP server that
* may supersede file scanning entirely. Everything this adapter knows about that private storage —
* its shape, naming, and field names — lives here, so a layout change, or a switch to the HTTP
* server, changes exactly one class and nothing in {@link OpenCodeLauncher}.
*
* <p>The determinism that makes this useful is structural, not a guess: every bridged worker runs
* in its own unique git worktree, so the record's {@code directory} (its project root) equals the
* worker's cwd identifies <em>its</em> session unambiguously. We match on {@code directory} rather
* than diffing {@code opencode session list} before/after — that races under concurrent spawns, and
* the CLI listing does not even show the directory.
*
* <p>All reads are best-effort and never throw: a missing or unreadable storage root, a record that
* fails to parse, or a directory with no record yet all yield {@code null}, and the caller (the
* session handle) treats that as "identity not resolved yet" and retries later.
*/
final class OpenCodeSessionDiscovery {
private final Path storageRoot; // e.g. ~/.local/share/opencode (injectable for tests)
private final ObjectMapper json;
OpenCodeSessionDiscovery(Path storageRoot) {
this.storageRoot = storageRoot;
this.json = new ObjectMapper();
}
/**
* The opencode session id whose record references {@code directory} (the worker's cwd), or
* {@code null} when no record matches yet. When several records share the directory — e.g.
* repeated spawns into the same worktree — the <em>most recently modified</em> one wins: it is
* the session the pane most likely corresponds to.
*
* <p>Never throws: a missing {@code storageRoot}, an unreadable/malformed record, or a
* directory that has not been persisted yet all resolve to {@code null} rather than failing a
* spawn. A bridged worker's session record is written lazily (when the session is first
* persisted), so {@code null} here is the normal answer right after the pane is ready, and the
* caller retries later.
*
* @param directory the worker's cwd, as resolved for this spawn
* @return the matching session id, or {@code null} if none is known yet
*/
String sessionIdForDirectory(String directory) {
if (directory == null || directory.isBlank()) {
return null;
}
Path sessionRoot = storageRoot.resolve("session");
if (!Files.isDirectory(sessionRoot)) {
return null;
}
String best = null;
long bestMtime = Long.MIN_VALUE;
try (Stream<Path> projectDirs = Files.list(sessionRoot)) {
for (Path projectDir : projectDirs.filter(Files::isDirectory).toList()) {
try (Stream<Path> records = Files.list(projectDir)) {
for (Path record : records.toList()) {
String id = matchId(record, directory);
if (id == null) {
continue;
}
long mtime = lastModifiedEpochMillis(record);
if (mtime > bestMtime) {
bestMtime = mtime;
best = id;
}
}
} catch (IOException ignored) {
// one project dir unreadable — skip it; another may still match
}
}
} catch (IOException ignored) {
// storage root vanished or became unreadable — "no session known yet"
return null;
}
return best;
}
/**
* The record's session id when it references {@code directory}, else {@code null}. A record
* that is not JSON, lacks {@code id}/{@code directory}, or points at a different directory is
* simply not our session; a malformed one is skipped, never fatal.
*/
private String matchId(Path record, String directory) {
try {
JsonNode node = json.readTree(record.toFile());
JsonNode id = node == null ? null : node.get("id");
JsonNode dir = node == null ? null : node.get("directory");
if (id == null || dir == null || !directory.equals(dir.asText())) {
return null;
}
return id.asText();
} catch (IOException e) {
return null;
}
}
/** The record's last-modified epoch ms, or {@code Long.MIN_VALUE} if unreadable (never wins). */
private static long lastModifiedEpochMillis(Path record) {
try {
return Files.getLastModifiedTime(record).toMillis();
} catch (IOException e) {
return Long.MIN_VALUE;
}
}
}
@@ -1,510 +0,0 @@
package dev.ltms.fleet.session;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
/**
* Production {@link Worktrees} implementation that shells {@code git} via {@link ProcessBuilder}.
* Non-zero exits become {@link WorktreeException}. Worktree directories live under a configurable
* root (default: a sibling {@code .bridged-worktrees} of the repo root) so they are never nested
* inside the primary working tree.
*/
public final class GitWorktrees implements Worktrees {
private static final Logger log = LoggerFactory.getLogger(GitWorktrees.class);
/** Project-level MCP config. Present in the repo, so every worktree would otherwise inherit the
* primary's IDE server mounts (a CB-523 worker edited the primary checkout; see the isolation
* javadoc). Neutralized unconditionally. */
private static final String MCP_CONFIG = ".mcp.json";
/** What {@link #isolateToolSurface} writes for {@code .mcp.json}: a valid, explicitly empty server map. */
private static final String NEUTRAL_MCP_CONFIG = "{\n \"mcpServers\": {}\n}\n";
/** OpenCode's repo-level config. Tracked here, so it lands in every worktree, and it mounts the
* primary's gitea and context7 servers with the primary's credentials. Neutralized so the worker
* gets only the config its launcher writes via {@code OPENCODE_CONFIG}.
*
* <p>The reason has changed shape and is now stronger. It used to be a crash: the file carried
* {@code {file:.secrets/...}} references to gitignored files that never reached a worktree, and
* opencode refuses to start on a dangling reference (CB-543). Those credentials now live in one
* shell-level store and the file reads them as {@code {env:...}}, so in a worktree the reference
* resolves instead of failing. That is worse, not better: a member would silently inherit the
* primary's admin-scoped {@code GITEA_ACCESS_TOKEN}. A loud crash became a quiet privilege leak,
* so this entry protects a boundary now rather than papering over a startup error. */
private static final String OPENCODE_CONFIG = "opencode.json";
/** What {@link #isolateToolSurface} writes for {@code opencode.json}: a valid, empty JSON object. */
private static final String NEUTRAL_OPENCODE_CONFIG = "{}\n";
/** Autoenv's repo-level config. Not tracked today, but re-landing it must stay safe: autoenv
* authorizes by path, so a fresh worktree path is always unauthorized and its interactive prompt
* would block every spawn — neutralize it so it can never be committed. */
private static final String AUTOENV_CONFIG = ".autoenv";
/** What {@link #isolateToolSurface} writes for {@code .autoenv}: a valid, empty env file. */
private static final String NEUTRAL_AUTOENV_CONFIG = "";
/**
* A tracked project config that is hostile in a provisioned worktree, and what to replace it
* with. {@link #file} is the repo-relative path; {@link #stub} is a neutral but VALID payload for
* that file's format — a malformed stub would only trade one crash for another;
* {@link #createIfAbsent} keeps {@code .mcp.json}'s long-standing behaviour of writing its stub
* even when the repo carries no such file, whereas the others are only touched when present.
*/
private record WorktreeHostileConfig(String file, String stub, boolean createIfAbsent) {}
/** The worktree-hostile configs neutralized in every provisioned worktree, in order. */
private static final List<WorktreeHostileConfig> WORKTREE_HOSTILE_CONFIGS = List.of(
new WorktreeHostileConfig(MCP_CONFIG, NEUTRAL_MCP_CONFIG, true),
new WorktreeHostileConfig(OPENCODE_CONFIG, NEUTRAL_OPENCODE_CONFIG, false),
new WorktreeHostileConfig(AUTOENV_CONFIG, NEUTRAL_AUTOENV_CONFIG, false)
);
private final String configuredRoot;
private final SecureRandom random = new SecureRandom();
private final AtomicLong seq = new AtomicLong();
/** Default constructor: worktree root is derived per-repo as {@code <repoRoot>/../.bridged-worktrees}. */
public GitWorktrees() {
this(null);
}
/** @param configuredRoot nullable absolute or relative path; null/blank derives a sibling of the repo root. */
public GitWorktrees(String configuredRoot) {
this.configuredRoot = configuredRoot;
}
@Override
public String add(String repoRoot, String branch, String baseRef) {
String base = (baseRef == null || baseRef.isBlank()) ? "HEAD" : baseRef;
String nonce = nonce();
Path root = resolveRoot(repoRoot);
Path path = root.resolve(nonce);
try {
Files.createDirectories(root);
} catch (IOException e) {
throw new WorktreeException("cannot create worktree root " + root + ": " + e.getMessage(), e);
}
String wt = path.toAbsolutePath().toString();
log.info("adding worktree branch={} path={} base={}", branch, wt, base);
exec("git", "-C", repoRoot, "worktree", "add", wt, "-b", branch, base);
isolateToolSurface(wt);
return wt;
}
/**
* Neutralize the worktree's worktree-hostile project configs so a worker inherits only the tools
* and environment its launcher mounts (the bridge via {@code --mcp-config}, the opencode config
* via {@code OPENCODE_CONFIG}) — never the primary's.
*
* <p>This is unconditional, and it is not the same job as the parity overlay. The repo's own
* committed {@code .mcp.json} declares the primary's IDE servers, so a fresh checkout mounts them
* whether or not the overlay copies anything; a worker that inherits them navigates and edits
* through tools bound to the <em>primary's</em> IntelliJ project, which silently hands it absolute
* paths outside its own worktree. That is not hypothetical: a CB-523 worker made all 59 of its
* edits in the primary checkout while compiling its worktree, so every build it ran was of code
* that did not contain its changes. {@code opencode.json} is the same trap one tool over — tracked,
* so it lands in every worktree, and it mounts gitea and context7 with the primary's own
* credentials, which a member must never hold. {@code .autoenv} extends the principle to a
* config that is not tracked today: autoenv authorizes by path, so a fresh worktree path is always
* unauthorized and its interactive prompt would block every spawn, so re-landing one must be safe.
*
* <p>Where a config exists it is replaced by a valid neutral stub (an explicitly empty
* map/object, or an empty env file — never a deletion, which would still let a later
* {@code git checkout} restore the hostile copy). The {@code --skip-worktree} bit keeps the
* neutralized copy from ever showing up as a local modification the worker might commit. A config
* the repo does not carry is skipped silently — no stub is invented for a file the repo does not
* have, and one missing file must never fail provisioning.
*/
private void isolateToolSurface(String worktreePath) {
Path root = Path.of(worktreePath).toAbsolutePath().normalize();
for (WorktreeHostileConfig cfg : WORKTREE_HOSTILE_CONFIGS) {
neutralize(root, worktreePath, cfg);
}
}
private void neutralize(Path root, String worktreePath, WorktreeHostileConfig cfg) {
Path target = root.resolve(cfg.file());
if (!Files.exists(target) && !cfg.createIfAbsent()) {
log.debug("{} absent in the worktree — skipping (repo does not carry it)", cfg.file());
return;
}
try {
Files.writeString(target, cfg.stub());
} catch (IOException e) {
throw new WorktreeException("cannot neutralize " + cfg.file() + " in the worktree: "
+ e.getMessage(), e);
}
if (isTracked(root, cfg.file())) {
exec("git", "-C", worktreePath, "update-index", "--skip-worktree", cfg.file());
}
log.debug("neutralized {} — worker tool surface is launcher-mounted only", cfg.file());
}
@Override
public void remove(String repoRoot, String worktreePath) {
Path p = Path.of(worktreePath);
if (!Files.exists(p)) {
log.debug("worktree {} already gone — nothing to remove", worktreePath);
return;
}
log.info("removing worktree {}", worktreePath);
exec("git", "-C", repoRoot, "worktree", "remove", "--force", worktreePath);
}
@Override
public boolean hasUncommitted(String worktreePath) {
// A worktree that is already gone holds no work to lose, and it must not break teardown:
// git -C <missing-dir> status exits non-zero and would throw where release() is mid-way
// through stopping a pane. Mirror remove()'s already-gone tolerance by treating it as clean.
Path p = Path.of(worktreePath);
if (!Files.exists(p)) {
log.debug("worktree {} already gone — nothing can be uncommitted", worktreePath);
return false;
}
// No --untracked-files=no: the exact shape of the work lost in CB-576 was a new file
// that was never added, so an untracked-only worktree is still dirty.
String out = exec("git", "-C", worktreePath, "status", "--porcelain");
return !out.isBlank();
}
@Override
public void overlayParity(String repoRoot, String worktreePath, List<String> overlay) {
if (overlay == null || overlay.isEmpty()) {
return;
}
Path srcRoot = Path.of(repoRoot).toAbsolutePath().normalize();
Path dstRoot = Path.of(worktreePath).toAbsolutePath().normalize();
for (String rel : overlay) {
Path src = srcRoot.resolve(rel).normalize();
if (!Files.exists(src)) {
log.debug("parity overlay source missing — skipping {}", rel);
continue;
}
Path dst = dstRoot.resolve(rel).normalize();
try {
Files.createDirectories(dst.getParent());
Files.copy(src, dst, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.COPY_ATTRIBUTES);
log.debug("copied parity overlay {}", rel);
} catch (IOException e) {
throw new WorktreeException("cannot copy overlay " + rel + ": " + e.getMessage(), e);
}
if (isTracked(dstRoot, rel)) {
exec("git", "-C", worktreePath, "update-index", "--skip-worktree", rel);
log.debug("marked overlay --skip-worktree {}", rel);
}
}
}
@Override
public String repoRoot(String cwd) {
String out = exec("git", "-C", cwd, "rev-parse", "--show-toplevel");
return Path.of(out.trim()).toAbsolutePath().normalize().toString();
}
/**
* CB-578 stage C, fixed by CB-587. Stages into a <em>temporary</em> index (never the worktree's
* real one, which the worker may still be writing to) — but that temp index is first <em>seeded</em>
* from the worktree's real one, rather than starting empty:
*
* <pre>
* cp $(git -C worktree rev-parse --git-path index) &lt;temp&gt;
* GIT_INDEX_FILE=&lt;temp&gt; git -C worktree add -A
* tree=$(GIT_INDEX_FILE=&lt;temp&gt; git -C worktree write-tree)
* commit=$(git -C worktree commit-tree $tree -p HEAD -m message)
* git -C worktree update-ref refs/wip/branch $commit
* </pre>
*
* A fresh empty index carries none of the real index's {@code --skip-worktree} /
* {@code --assume-unchanged} bits, so {@code add -A} into it stages a skip-worktree file's local
* on-disk content even though {@code git status --porcelain} correctly hides that file (CB-587).
* Seeding from the real index preserves those bits, so {@code add -A} then skips exactly what
* {@code git status} skips. {@code add -A} (never {@code -f}) also still respects
* {@code .gitignore} exactly as it would in the real index — a gitignored file staying ignored is
* what keeps secrets and local config out of the snapshot's tree. The temporary index file is
* removed afterwards regardless of outcome; the worker's real index is never opened for writing.
*/
@Override
public Optional<String> snapshot(String worktreePath, String branch, String message) {
if (!Files.exists(Path.of(worktreePath))) {
log.debug("worktree {} already gone — nothing to snapshot", worktreePath);
return Optional.empty();
}
Path tempIndex;
try {
tempIndex = Files.createTempFile("bridged-wip-index-", ".tmp");
} catch (IOException e) {
throw new WorktreeException("cannot create a temporary index for snapshot: " + e.getMessage(), e);
}
Map<String, String> indexEnv = Map.of("GIT_INDEX_FILE", tempIndex.toAbsolutePath().toString());
try {
Path realIndex = resolveRealIndex(worktreePath);
try {
Files.copy(realIndex, tempIndex, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new WorktreeException("cannot copy the worktree's real index (" + realIndex
+ ") into the temporary snapshot index: " + e.getMessage(), e);
}
exec(indexEnv, "git", "-C", worktreePath, "add", "-A");
String tree = exec(indexEnv, "git", "-C", worktreePath, "write-tree").trim();
String commit = exec("git", "-C", worktreePath, "commit-tree", tree, "-p", "HEAD", "-m", message).trim();
exec("git", "-C", worktreePath, "update-ref", "refs/wip/" + branch, commit);
log.info("snapshotted worktree {} to refs/wip/{} commit={}", worktreePath, branch, commit);
return Optional.of(commit);
} finally {
try {
Files.deleteIfExists(tempIndex);
} catch (IOException e) {
log.debug("could not delete temporary snapshot index {}: {}", tempIndex, e.getMessage());
}
}
}
/**
* Resolve the path of {@code worktreePath}'s real index. Never assume {@code <worktree>/.git/index}:
* in a linked worktree {@code .git} is a <em>file</em> pointing at the main repo's
* {@code worktrees/<name>/} directory, and that is where the real per-worktree index lives.
* {@code git rev-parse --git-path index} resolves this correctly for both a linked worktree and
* the main checkout. Throws {@link WorktreeException} — same as every other failure in this
* class — if the command fails or the resolved path does not exist, rather than silently
* snapshotting from an empty index.
*/
private Path resolveRealIndex(String worktreePath) {
String out = exec("git", "-C", worktreePath, "rev-parse", "--git-path", "index").trim();
Path index = Path.of(out);
if (!index.isAbsolute()) {
index = Path.of(worktreePath).resolve(index).normalize();
}
if (!Files.exists(index)) {
throw new WorktreeException("worktree's real index not found at resolved path " + index
+ " (git rev-parse --git-path index reported '" + out + "')");
}
return index;
}
/**
* One {@code refs/wip/<branch>} snapshot ref as read by {@link #listWipRefs}: its full ref name,
* the snapshot commit's sha, and that commit's committer time in unix millis (the age of the
* snapshot — a snapshot is written once and never rewritten, so the commit date is the ref's).
*/
private record WipRef(String refName, String sha, long committerMillis) {
String branch() {
return refName.substring("refs/wip/".length());
}
}
@Override
public WipRefStats wipRefs(String repoRoot) {
List<WipRef> refs = listWipRefs(repoRoot);
long costBytes = 0;
for (WipRef ref : refs) {
costBytes += treeSize(repoRoot, ref.sha());
}
return new WipRefStats(refs.size(), costBytes);
}
@Override
public int pruneWipRefs(String repoRoot, long minAgeMillis) {
// The rule is documented on Worktrees#pruneWipRefs: delete only a snapshot whose tree
// content is already reachable from main AND that is older than minAgeMillis. Reachability
// is the floor that keeps a worker's last copy; the age floor keeps a just-written snapshot
// from being swept while a lead may still be looking at it.
List<WipRef> refs = listWipRefs(repoRoot);
if (refs.isEmpty()) {
return 0;
}
long nowMillis = System.currentTimeMillis();
// Resolve what main carries once per sweep, not once per ref.
Set<String> mainObjects = reachableObjectsFromMain(repoRoot);
int deleted = 0;
for (WipRef ref : refs) {
long ageMillis = nowMillis - ref.committerMillis();
if (ageMillis <= minAgeMillis) {
continue; // too recent — never swept, even if it looks recoverable (CB-586)
}
String tree = exec("git", "-C", repoRoot, "rev-parse", ref.sha() + "^{tree}").trim();
if (!mainObjects.contains(tree)) {
// Last copy of the snapshot's content — the worker's work exists nowhere else.
// Never delete automatically (CB-586 criterion 2).
continue;
}
exec("git", "-C", repoRoot, "update-ref", "-d", ref.refName());
deleted++;
log.info("pruned snapshot ref refs/wip/{} commit={} (age {}h): its tree is already "
+ "reachable from main, so the work is preserved; recover from reflog via "
+ "git update-ref refs/wip/{} {}",
ref.branch(), ref.sha(), TimeUnit.MILLISECONDS.toHours(ageMillis),
ref.branch(), ref.sha());
}
return deleted;
}
/**
* Every {@code refs/wip/*} ref (see {@link WipRef}). The committer date is read as a unix
* count of seconds and converted to millis. {@code %00} (NUL) separates the fields because a
* branch name may contain spaces.
*/
private List<WipRef> listWipRefs(String repoRoot) {
String out = exec("git", "-C", repoRoot, "for-each-ref",
"--format=%(refname)%00%(objectname)%00%(committerdate:unix)", "refs/wip/");
List<WipRef> refs = new ArrayList<>();
for (String line : out.split("\\R")) {
if (line.isBlank()) {
continue;
}
String[] parts = line.split("\u0000", -1);
if (parts.length == 3 && !parts[1].isBlank()) {
refs.add(new WipRef(parts[0], parts[1], Long.parseLong(parts[2]) * 1000L));
}
}
return refs;
}
/**
* The set of object shas reachable from {@code main}, or an empty set when {@code main} cannot
* be resolved. An empty set is the safe direction: the retention sweep then concludes nothing
* is recoverable, so it deletes nothing — a repo with no {@code main} must never cause a
* worker's last copy of a snapshot to be dropped on a reachability misreading.
*/
private Set<String> reachableObjectsFromMain(String repoRoot) {
if (exitCode("git", "-C", repoRoot, "rev-parse", "--verify", "main") != 0) {
log.debug("refs/wip retention: no 'main' ref in {} — treating nothing as reachable", repoRoot);
return Set.of();
}
String out = exec("git", "-C", repoRoot, "rev-list", "--objects", "main");
Set<String> objects = new HashSet<>();
for (String line : out.split("\\R")) {
if (line.isBlank()) {
continue;
}
int sp = line.indexOf(' ');
objects.add(sp < 0 ? line : line.substring(0, sp));
}
return objects;
}
/** Approximate cost of a snapshot: the sum of every blob's size in its committed tree. */
private long treeSize(String repoRoot, String sha) {
String out = exec("git", "-C", repoRoot, "ls-tree", "-r", "-l", sha);
long total = 0;
for (String line : out.split("\\R")) {
if (line.isBlank()) {
continue;
}
// ls-tree -l row: "<mode> <type> <object> <size>\t<path>"; the size is only numeric for
// blobs (trees read "-"), so gate on the type token and take the 4th whitespace field.
String[] parts = line.split("\\s+");
if (parts.length >= 4 && "blob".equals(parts[1])) {
try {
total += Long.parseLong(parts[3]);
} catch (NumberFormatException ignored) {
// a '-' size (or any anomaly) contributes nothing to the rough figure
}
}
}
return total;
}
/** Resolve the directory that will hold per-session worktree checkouts. */
private Path resolveRoot(String repoRoot) {
if (configuredRoot != null && !configuredRoot.isBlank()) {
return Path.of(configuredRoot).toAbsolutePath().normalize();
}
Path repo = Path.of(repoRoot).toAbsolutePath().normalize();
return repo.resolveSibling(".bridged-worktrees");
}
private String nonce() {
return String.format("%06x", random.nextInt(1 << 24)) + "-" + seq.incrementAndGet();
}
private boolean isTracked(Path worktreeRoot, String rel) {
return exitCode("git", "-C", worktreeRoot.toString(), "ls-files", "--error-unmatch", rel) == 0;
}
/**
* Run a command and return its stdout. Non-zero exit → {@link WorktreeException} with both
* stdout and stderr (merged by redirectErrorStream).
*/
private String exec(String... command) {
return exec(Map.of(), command);
}
/** Same as {@link #exec(String...)}, with extra environment variables set on the child process. */
private String exec(Map<String, String> extraEnv, String... command) {
String out;
int code;
Process p;
try {
ProcessBuilder pb = new ProcessBuilder(command).redirectErrorStream(true);
if (extraEnv != null && !extraEnv.isEmpty()) {
pb.environment().putAll(extraEnv);
}
p = pb.start();
} catch (IOException e) {
throw new WorktreeException("failed to start " + command[0] + ": " + e.getMessage(), e);
}
try (BufferedReader r = new BufferedReader(new InputStreamReader(p.getInputStream(), StandardCharsets.UTF_8))) {
out = r.lines().collect(Collectors.joining("\n"));
} catch (IOException e) {
p.destroyForcibly();
throw new UncheckedIOException(e);
}
try {
if (!p.waitFor(30, TimeUnit.SECONDS)) {
p.destroyForcibly();
throw new WorktreeException("command timed out: " + String.join(" ", command) + "\n" + out);
}
code = p.exitValue();
} catch (InterruptedException e) {
p.destroyForcibly();
Thread.currentThread().interrupt();
throw new WorktreeException("interrupted waiting for command: " + String.join(" ", command), e);
}
if (code != 0) {
throw new WorktreeException("exit " + code + " for: " + String.join(" ", command)
+ (out.isBlank() ? "" : "\n" + out));
}
return out;
}
private int exitCode(String... command) {
Process p;
try {
p = new ProcessBuilder(command).redirectErrorStream(true).start();
} catch (IOException e) {
throw new WorktreeException("failed to start " + command[0] + ": " + e.getMessage(), e);
}
try {
if (!p.waitFor(30, TimeUnit.SECONDS)) {
p.destroyForcibly();
throw new WorktreeException("command timed out: " + String.join(" ", command));
}
return p.exitValue();
} catch (InterruptedException e) {
p.destroyForcibly();
Thread.currentThread().interrupt();
throw new WorktreeException("interrupted waiting for command: " + String.join(" ", command), e);
}
}
}
@@ -1,170 +0,0 @@
package dev.ltms.fleet.health;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.inject.Injector;
import dev.ltms.fleet.msg.InMemoryReplyInbox;
import dev.ltms.fleet.msg.MessageService;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.session.SessionManager;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.config.FleetConfig;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.Executors;
import java.util.function.BiConsumer;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
class FleetHealthMonitorTest {
@Test void oneTickUsesOneFleetListForAnyRosterSize() {
FakeHerdr herdr = new FakeHerdr();
AgentControl agents = new AgentControl(herdr);
FleetConfig.Profile profile = new FleetConfig.Profile("test", "http://test:1", null,
null, null, null, null, null, null, null, null, null);
ClaudeCodeLauncher launcher = new ClaudeCodeLauncher(agents, new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("test")), Map.of("test", profile), "test", _ -> "token");
SessionManager sessions = new SessionManager(launcher);
sessions.acquire("test", null, null, null);
sessions.acquire("test", null, null, null);
herdr.calls.clear();
MessageService messages = new MessageService(agents, new Injector(agents), new Rendezvous(), new InMemoryReplyInbox());
var scheduler = Executors.newSingleThreadScheduledExecutor();
FleetHealthMonitor monitor = new FleetHealthMonitor(agents, sessions::roster, messages, scheduler, () -> 1, 60,
(_, _) -> { });
monitor.tick();
monitor.stop();
assertEquals(1, herdr.calls.stream().filter(call -> call.method().equals("agent.list")).count());
}
@Test void failedTickDoesNotStopTheNextTick() {
FakeHerdr herdr = new FakeHerdr().healthy(false);
AgentControl agents = new AgentControl(herdr);
var scheduler = Executors.newSingleThreadScheduledExecutor();
FleetHealthMonitor monitor = new FleetHealthMonitor(agents, java.util.List::of,
new MessageService(agents, new Injector(agents), new Rendezvous(), new InMemoryReplyInbox()),
scheduler, () -> 1, 60, (_, _) -> { });
monitor.tick();
herdr.healthy(true);
monitor.tick();
monitor.stop();
assertEquals(2, herdr.calls.stream().filter(call -> call.method().equals("agent.list")).count());
}
@Test void faultTransitionLogsOnlyOnceUntilItChanges() {
Logger logger = (Logger) LoggerFactory.getLogger(FleetHealthMonitor.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
try {
FakeHerdr herdr = new FakeHerdr();
AgentControl agents = new AgentControl(herdr);
var scheduler = Executors.newSingleThreadScheduledExecutor();
FleetHealthMonitor monitor = new FleetHealthMonitor(agents, java.util.List::of,
new MessageService(agents, new Injector(agents), new Rendezvous(), new InMemoryReplyInbox()),
scheduler, () -> 1, 60, (_, _) -> { });
monitor.reportTransition("term_a", HealthState.TURN_BOUNDARY_LOST);
monitor.reportTransition("term_a", HealthState.TURN_BOUNDARY_LOST);
monitor.stop();
assertEquals(1, appender.list.stream().filter(event -> event.getFormattedMessage()
.contains("member=term_a state=TURN_BOUNDARY_LOST")).count());
} finally {
logger.detachAppender(appender);
}
}
// --- CB-580: a member that reaches GONE/NEVER_READY must fail its waiting tickets
private static FleetHealthMonitor monitorWith(BiConsumer<String, String> failTarget) {
FakeHerdr herdr = new FakeHerdr();
AgentControl agents = new AgentControl(herdr);
var scheduler = Executors.newSingleThreadScheduledExecutor();
return new FleetHealthMonitor(agents, java.util.List::of,
new MessageService(agents, new Injector(agents), new Rendezvous(), new InMemoryReplyInbox()),
scheduler, () -> 1, 60, failTarget);
}
@Test void terminalTransitionFailsTheTargetOnce() {
RecordingFailTarget failTarget = new RecordingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.stop();
assertEquals(1, failTarget.calls.size());
assertEquals("term_a", failTarget.calls.get(0).target());
assertTrue(failTarget.calls.get(0).reason().contains("GONE"));
}
@Test void neverReadyNamesItselfAsTheReason() {
RecordingFailTarget failTarget = new RecordingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.NEVER_READY);
monitor.stop();
assertEquals(1, failTarget.calls.size());
assertTrue(failTarget.calls.get(0).reason().contains("NEVER_READY"));
}
@Test void stayingInATerminalStateProducesOneFailureNotN() {
RecordingFailTarget failTarget = new RecordingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.stop();
assertEquals(1, failTarget.calls.size());
}
@Test void aNonTerminalFaultStateDoesNotFailTheTarget() {
RecordingFailTarget failTarget = new RecordingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.TURN_BOUNDARY_LOST);
monitor.stop();
assertEquals(0, failTarget.calls.size());
}
@Test void failTargetRetryIsBounded() {
AlwaysThrowingFailTarget failTarget = new AlwaysThrowingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.GONE);
monitor.stop();
assertEquals(FleetHealthMonitor.MAX_FAIL_TARGET_ATTEMPTS, failTarget.calls);
}
@Test void exhaustedRetryStillDoesNotRefireOnAnUnchangedTick() {
AlwaysThrowingFailTarget failTarget = new AlwaysThrowingFailTarget();
FleetHealthMonitor monitor = monitorWith(failTarget);
monitor.reportTransition("term_a", HealthState.GONE);
int afterFirstTransition = failTarget.calls;
monitor.reportTransition("term_a", HealthState.GONE);
monitor.stop();
assertEquals(afterFirstTransition, failTarget.calls);
}
private record RecordedCall(String target, String reason) { }
private static final class RecordingFailTarget implements BiConsumer<String, String> {
final java.util.List<RecordedCall> calls = new java.util.ArrayList<>();
@Override public void accept(String target, String reason) {
calls.add(new RecordedCall(target, reason));
}
}
private static final class AlwaysThrowingFailTarget implements BiConsumer<String, String> {
int calls = 0;
@Override public void accept(String target, String reason) {
calls++;
throw new RuntimeException("boom");
}
}
}
@@ -1,27 +0,0 @@
package dev.ltms.fleet.herdr;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
/** Unit tests for PID → pane resolution (the herdr half of connection-based MCP identity). */
class PaneLocatorTest {
private final PaneLocator loc = new PaneLocator(new FakeHerdr());
@Test
void resolvesTerminalForAForegroundPid() {
assertEquals("term_a", loc.terminalForPid(FakeHerdr.WORKER_PID));
}
@Test
void nullForAPidInNoPane() {
assertNull(loc.terminalForPid(999_999));
}
@Test
void nullForNonPositivePid() {
assertNull(loc.terminalForPid(0));
assertNull(loc.terminalForPid(-1));
}
}
@@ -1,501 +0,0 @@
package dev.ltms.fleet.inject;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.TestTurnTokens;
import dev.ltms.fleet.msg.TurnToken;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import java.util.Set;
import java.util.regex.Pattern;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Unit behaviour of the CB-106 completion resolver in isolation from the injector. */
class CompletionResolverTest {
@Test
void skipsTheScrapeWhenNoSendIsWaiting() {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous(); // no waiter opened
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
resolver.resolve("term_a", null); // no in-flight turn captured for this target
assertFalse(herdr.called("agent.read"),
"a turn nobody is blocked on must not cost a transcript scrape");
}
@Test
void failSkipsTheScrapeWhenNoSendIsWaiting() {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous(); // no waiter opened
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
resolver.fail("term_a", null); // no in-flight turn, and no registered waiter to fall back to
assertFalse(herdr.called("agent.read"),
"a wedge nobody is blocked on must not cost a transcript scrape");
}
@Test
void captureBaselineSkipsTheReadWhenNoSendIsWaiting() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ X\n❯ ");
Rendezvous rendezvous = new Rendezvous(); // no waiter opened
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
resolver.captureBaseline("term_a", TestTurnTokens.inert("term_a")); // no send to attribute a later completion to
assertFalse(herdr.called("agent.read"),
"with no waiting send there is no turn to baseline — skip the scrape");
}
// --- CB-115 clean scrape: extract the last assistant block ----------------
@Test
void extractsTheLastAssistantBlockStrippingChrome() {
String raw = """
⏺ Reading the file…
⏺ Done. The bug was an off-by-one in the loop bound.
╭──────────────────────────────────────╮
│ > │
╰──────────────────────────────────────╯
⏵⏵ auto mode on · ? for shortcuts
""";
assertEquals("Done. The bug was an off-by-one in the loop bound.",
CompletionResolver.lastAssistantBlock(raw));
}
@Test
void keepsMultiLineAssistantContent() {
String raw = "⏺ Line one.\nLine two.\n❯ ";
assertEquals("Line one.\nLine two.", CompletionResolver.lastAssistantBlock(raw));
}
@Test
void fallsBackToRawTextWhenThereIsNoMarker() {
String raw = "plain worker output with no glyph";
assertEquals("plain worker output with no glyph", CompletionResolver.lastAssistantBlock(raw));
}
@Test
void blankScrapeYieldsEmpty() {
assertTrue(CompletionResolver.lastAssistantBlock("").isEmpty());
assertTrue(CompletionResolver.lastAssistantBlock(null).isEmpty());
}
@Test
void stripsSpinnerAndRuleChrome() {
String raw = """
⏺ Channel check confirmed — your message got through.
✻ Brewed for 11s
─────────────────────────────────────
""";
assertEquals("Channel check confirmed — your message got through.",
CompletionResolver.lastAssistantBlock(raw));
}
@Test
void cutsANextTurnPromptEchoAndTrailingTipsFromTheBlock() {
// The exact turn-2 leak: the scrape captured the settled answer, then a "✻ Cooked" spinner,
// then the NEXT turn's echoed prompt, then a "✶ Forming…" spinner and trailing tips/warnings
// whose lines (⎿, ⚠) are not themselves chrome-terminated. Stopping at the first boundary
// (the ✻ spinner) is what keeps every one of those interface lines out of the reply.
String raw = """
⏺ Channel confirmed — the bridge reply delivered successfully.
✻ Cooked for 9s
❯ Thanks. Now a small task: what is 17 * 23? Show just the number.
✶ Forming…
⎿ Tip: Name your conversations with /rename
⚠ claude.ai connectors are disabled because ANTHROPIC_API_KEY is set
""";
assertEquals("Channel confirmed — the bridge reply delivered successfully.",
CompletionResolver.lastAssistantBlock(raw));
}
// --- CB-115 misattribution guard: suppress a stale (unchanged) completion -------
@Test
void suppressesACompletionWhoseScrapeIsUnchangedFromDelivery() {
// Rapid back-to-back turn: the pane still shows the PREVIOUS turn's answer when this turn's
// (misattributed) completion boundary fires. The scrape == the delivery baseline, so the
// send must NOT be resolved with the stale answer.
FakeHerdr herdr = new FakeHerdr().readText("⏺ 391\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a"); // a send is blocked on this turn
// The turn as captured at delivery: its waiter, and the previous turn's answer still on screen.
var turn = new CompletionResolver.InFlight(waiter, "391");
resolver.resolve("term_a", turn); // scrape still "391" == baseline → suppress
assertFalse(waiter.isDone(), "a completion with no output change must not resolve the send");
assertTrue(rendezvous.isWaiting("term_a"), "the send stays waiting for a real reply");
}
@Test
void resolvesACompletionWhoseScrapeChangedSinceDelivery() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ No, 391 = 17 × 23.\n❯ "); // the worker's real answer
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
// Delivery baseline was the previous turn's "391"; the scrape now differs → resolve.
var turn = new CompletionResolver.InFlight(waiter, "391");
resolver.resolve("term_a", turn);
assertTrue(waiter.isDone(), "a completion with new output must resolve the send");
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind());
assertEquals("No, 391 = 17 × 23.", waiter.getNow(null).text());
}
@Test
void marksAClippedCompletionPaneTail() {
String block = "⏺ " + "x".repeat(CompletionResolver.MAX_SCRAPE_CHARS + 1) + "\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals("x".repeat(CompletionResolver.MAX_SCRAPE_CHARS)
+ "\n[Pane tail clipped: member did not call fleet_reply.]",
waiter.getNow(null).text());
}
@Test
void leavesAnUnclippedCompletionPaneTailUnmarked() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ complete report\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals("complete report", waiter.getNow(null).text());
}
@Test
void resolvesSynchronouslyBeforePostTurnContextClearing() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ previous answer\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiter));
herdr.readText("⏺ answer that /clear would erase\n❯ ");
resolver.resolveBeforePostAction("term_a");
assertTrue(waiter.isDone(), "the answer is captured before the adapter sends /clear");
assertEquals("answer that /clear would erase", waiter.getNow(null).text());
}
@Test
void suppressesAnUnchangedCompletionEvenWhenTheBlockExceedsTheScrapeCap() {
// The fan-out issue-hunt finding: captureBaseline once stored the RAW (unclipped) assistant
// block while resolve compares against a clip()'d tail. For a block longer than MAX_SCRAPE_CHARS
// the two capped representations differ even when the pane never changed, so the CB-115
// byte-identical guard failed to fire and a stale completion could resolve the send. Both sides
// must clip identically. The returned-text marker is added only after this comparison, so an
// unchanged >cap block on rapid back-to-back turns still stays suppressed.
String longBlock = "⏺ " + "x".repeat(CompletionResolver.MAX_SCRAPE_CHARS + 500) + "\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(longBlock);
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a"); // a send is blocked on this turn
resolver.captureBaseline("term_a", new TurnToken("term_a", waiter)); // baseline is the clipped >cap block
var turn = resolver.inFlight("term_a");
assertEquals(CompletionResolver.MAX_SCRAPE_CHARS, turn.baseline().length(),
"the delivery baseline is clipped to the same cap resolve() applies to the tail");
resolver.resolve("term_a", turn); // scrape unchanged → clipped tail == baseline → suppress
assertFalse(waiter.isDone(),
"an unchanged >cap block must still be recognised as stale and suppressed");
assertTrue(rendezvous.isWaiting("term_a"), "the send stays waiting for a real reply");
}
@Test
void resolvesWhenThereIsNoBaseline() {
// No delivery baseline (e.g. the pre-turn read failed) ⇒ never suppress; the completion resolves.
FakeHerdr herdr = new FakeHerdr().readText("⏺ hello\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertTrue(waiter.isDone(), "with no baseline a completion resolves as before");
assertEquals("hello", waiter.getNow(null).text());
}
@Test
void resolvesWhenTheScrapeItselfFailsEvenWithABaselinePresent() {
// The most important branch of the CB-115 guard: a failed read means the resolver could not
// SEE the screen — "couldn't see", not "no change". It must still resolve the send (an empty
// tail beats hanging until the caller's timeout), even though a baseline was captured. The
// baseline here is "" (an empty pane at delivery), so without the !scrapeFailed clause the
// byte-identical guard would wrongly match the empty tail and suppress.
FakeHerdr herdr = new FakeHerdr().healthy(false); // agent.read throws HerdrException
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
var turn = new CompletionResolver.InFlight(waiter, ""); // empty pane baselined at delivery
resolver.resolve("term_a", turn);
assertTrue(waiter.isDone(),
"a failed scrape must still resolve the send, not hang until the caller's timeout");
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind());
assertEquals("", waiter.getNow(null).text(), "the tail is empty because the screen was unreadable");
}
// --- CB-115/CB-116 fail guard: an already-done or absent waiter is left alone ---------
@Test
void failLeavesAnAlreadyResolvedWaiterUntouchedAndSkipsTheScrape() {
// The send was already resolved (e.g. by the worker's explicit reply) before fail fired.
// fail must not overwrite that value, and must not even scrape the worker — nobody needs it.
FakeHerdr herdr = new FakeHerdr().readText("an error screen");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
var turn = new CompletionResolver.InFlight(waiter, null);
assertTrue(rendezvous.resolveCompletion(waiter, "already replied"));
resolver.fail("term_a", turn);
assertFalse(herdr.called("agent.read"),
"fail must not scrape a waiter that is already done");
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind(),
"fail must not overwrite the existing resolution");
assertEquals("already replied", waiter.getNow(null).text());
}
@Test
void failFallsBackToTheRegisteredWaiterWhenThereIsNoInFlightTurn() {
// A never-delivered readiness failure has no in-flight record but still has a blocked send;
// fail falls back to the waiter currently registered on the Rendezvous and fails it.
FakeHerdr herdr = new FakeHerdr().readText("stuck on an error screen");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a"); // send registered, but no captureBaseline ever ran
resolver.fail("term_a", null); // no in-flight turn → fall back to the registered waiter
assertTrue(waiter.isDone(), "fail falls back to the registered waiter when no turn is in flight");
assertEquals(Rendezvous.Kind.FAILED, waiter.getNow(null).kind());
assertEquals("stuck on an error screen", waiter.getNow(null).text());
}
@Test
void failIsLoggedAtWarnWithTheReason() {
// CB-564: this used to be a bare DEBUG "failed send to X via turn-stall fallback" — a symptom
// with no cause, and below the level anyone watching for member health would see. A fail that
// resolves a caller's blocked send is at least WARN and must carry the reason.
LoggerContext ctx = (LoggerContext) LoggerFactory.getILoggerFactory();
ch.qos.logback.classic.Logger resolverLog =
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger(CompletionResolver.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.setContext(ctx);
appender.start();
resolverLog.addAppender(appender);
resolverLog.setLevel(Level.WARN);
try {
FakeHerdr herdr = new FakeHerdr().readText("stuck on an error screen");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.fail("term_a", null);
String warn = appender.list.stream()
.filter(e -> e.getLevel().equals(Level.WARN))
.map(ILoggingEvent::getFormattedMessage)
.findFirst()
.orElse("no turn-stall WARN logged");
assertTrue(warn.contains("term_a"), "the log names the target: " + warn);
assertTrue(warn.contains("stuck on an error screen"), "the log carries the reason: " + warn);
assertTrue(waiter.isDone());
} finally {
resolverLog.detachAppender(appender);
}
}
// --- CB-116 waiter identity: a late completion never crosses into the next turn ---------
@Test
void aLateCompletionForOneTurnNeverResolvesTheNextTurnsWaiter() {
// The cross-turn stale reply the conversation test surfaced: turn N's completion fallback
// fires AFTER turn N was resolved by an explicit fleet_reply and turn N+1 has opened its own
// waiter on the same session. Resolving "whatever is waiting now" would hand turn N's stale
// scrape to turn N+1; targeting turn N's captured waiter makes the late completion a no-op.
FakeHerdr herdr = new FakeHerdr().readText("⏺ turn N answer\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterN = rendezvous.open("term_a"); // turn N's send
// The turn as the injector captured it at delivery (waiter + pre-turn baseline).
var turnN = new CompletionResolver.InFlight(waiterN, "an earlier answer");
// Turn N is resolved by the worker's explicit reply, and its send deregisters the waiter.
assertTrue(rendezvous.resolve("term_a", "N replied"));
rendezvous.close("term_a", waiterN); // the sender's finally, before the next turn opens
// Turn N+1's send opens its own waiter on the same session (CB-548: open fails if the
// previous waiter is still registered, so a clean turn deregisters it first as above).
var waiterN1 = rendezvous.open("term_a");
resolver.resolve("term_a", turnN); // turn N's completion fallback finally fires
assertFalse(waiterN1.isDone(), "turn N's late completion must not resolve turn N+1's waiter");
assertEquals(Rendezvous.Kind.REPLY, waiterN.getNow(null).kind(),
"turn N stays resolved by its own reply");
assertTrue(rendezvous.isWaiting("term_a"), "turn N+1 is still awaiting its own resolution");
}
// --- CB-578 stage A: backend-exhausted classification ---------------------------------
@Test
void classifiesAMatchingScrapeAsBackendExhaustedInsteadOfACompletedReply() {
String block = "⏺ Working on it...\nThe usage limit has been reached. Try again later.\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
ExhaustedPatternLookup patterns = target -> Pattern.compile("usage limit has been reached");
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, patterns, ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertTrue(waiter.isDone(), "a matching scrape still resolves the blocked send");
assertEquals(Rendezvous.Kind.BACKEND_EXHAUSTED, waiter.getNow(null).kind(),
"not reported as a completed reply — the classification is distinct");
}
@Test
void theExhaustedReasonCarriesTheMatchedLine() {
String block = "⏺ Working on it...\nThe usage limit has been reached. Try again later.\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
ExhaustedPatternLookup patterns = target -> Pattern.compile("usage limit has been reached");
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, patterns, ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals("backend exhausted (usage limit): The usage limit has been reached. Try again later.",
waiter.getNow(null).text(), "the reason names the real cause and carries the matched line");
}
@Test
void aWinningBackendExhaustedClassificationNotifiesTheExhaustionSink() {
String block = "⏺ Working on it...\nThe usage limit has been reached. Try again later.\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
ExhaustedPatternLookup patterns = target -> Pattern.compile("usage limit has been reached");
java.util.List<String> notified = new java.util.ArrayList<>();
ExhaustionSink sink = (target, reason) -> notified.add(target + ": " + reason);
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, patterns, sink);
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals(1, notified.size(), "the sink is notified exactly once for the winning classification");
assertTrue(notified.get(0).startsWith("term_a: "), "the sink is told which target exhausted");
assertTrue(notified.get(0).contains("The usage limit has been reached"),
"the sink is told the matched reason: " + notified.get(0));
}
@Test
void aLosingBackendExhaustedClassificationNeverNotifiesTheExhaustionSink() {
// The waiter was already resolved (e.g. by the worker's own reply) before this scrape landed —
// resolveExhausted loses the race and must return false, so the sink must not fire either.
String block = "⏺ Working on it...\nThe usage limit has been reached. Try again later.\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
ExhaustedPatternLookup patterns = target -> Pattern.compile("usage limit has been reached");
java.util.List<String> notified = new java.util.ArrayList<>();
ExhaustionSink sink = (target, reason) -> notified.add(target + ": " + reason);
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, patterns, sink);
var waiter = rendezvous.open("term_a");
var turn = new CompletionResolver.InFlight(waiter, null);
assertTrue(rendezvous.resolveCompletion(waiter, "already replied"));
resolver.resolve("term_a", turn);
assertTrue(notified.isEmpty(), "a classification that loses the race must not quarantine anything");
assertEquals("already replied", waiter.getNow(null).text(), "the earlier resolution stands untouched");
}
@Test
void aNonMatchingScrapeResolvesAsAnOrdinaryCompletion() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ complete report\n❯ ");
Rendezvous rendezvous = new Rendezvous();
ExhaustedPatternLookup patterns = target -> Pattern.compile("usage limit has been reached");
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous, patterns, ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind(),
"a scrape that does not match the pattern is an ordinary completion");
assertEquals("complete report", waiter.getNow(null).text());
}
@Test
void aProfileWithNoConfiguredPatternKeepsTodaysCompletionFallbackUnchanged() {
// Even a scrape that WOULD have matched some other profile's pattern must resolve as a
// plain completion when this target's own profile has none configured (CB-578 criterion 4).
String block = "⏺ The usage limit has been reached.\n❯ ";
FakeHerdr herdr = new FakeHerdr().readText(block);
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver =
new CompletionResolver(new AgentControl(herdr), rendezvous, ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiter = rendezvous.open("term_a");
resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind(),
"no pattern configured for this target's profile ⇒ unchanged completion-fallback behaviour");
assertEquals("The usage limit has been reached.", waiter.getNow(null).text());
}
@Test
void coverageIsOffWhenNoProfileHasAPatternConfigured() {
assertEquals("off (no profile has an exhaustedPattern configured; profiles: [terra])",
CompletionResolver.coverage(Set.of("terra"), Set.of()));
}
@Test
void coverageIsFullWhenEveryProfileHasAPatternConfigured() {
assertEquals("full (all profiles configured: [gx10, terra])",
CompletionResolver.coverage(Set.of("terra", "gx10"), Set.of("terra", "gx10")));
}
@Test
void coverageIsPartialAndNamesWhichProfilesAreConfigured() {
assertEquals("partial (configured: [terra]; not configured: [gx10])",
CompletionResolver.coverage(Set.of("terra", "gx10"), Set.of("terra")));
}
}
File diff suppressed because it is too large Load Diff
@@ -1,170 +0,0 @@
package dev.ltms.fleet.member;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.peer.Capability;
import dev.ltms.fleet.peer.MemberRole;
import dev.ltms.fleet.peer.SpawnRequest;
import org.junit.jupiter.api.Test;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.Supplier;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* CB-633: proves the allow-list scrub is actually WIRED INTO the spawn path — not merely that its
* pieces work when a test calls them directly.
*
* <p>Why this test exists, and why it is separate from {@link EnvAllowListScrubTest}. Every other
* test of this feature calls {@code EnvAllowListScrub} or {@code MemberEnvAllowList} itself. Those
* prove the scrub is correct. None of them proves anyone runs it: deleting the single
* {@code applyEnvironmentAllowListPolicy(cfg, launch)} line from {@code spawnInternal} left all 896
* tests green while turning the control completely off. That is the recurring shape in this
* codebase — a feature behind one call, with every test on the far side of it (CB-586, CB-611).
*
* <p>So this test starts a real spawn through {@link HerdrPeerLauncher#spawn} and asserts on what
* reached herdr. It deliberately checks the pane-creation parameters rather than the launcher's own
* map, because the map is an intermediate: {@code ZDOTDIR} only protects anything if it is in the
* env herdr uses to create the pane, and that is the last point we can observe before the shell
* starts.
*/
class HerdrPeerLauncherAllowListWiringTest {
/** A name the daemon itself injects — it must survive its own scrub, so it must be allowed. */
private static final String INJECTED = "ANTHROPIC_BASE_URL";
@Test
void spawningUnderAllowListPolicyGivesThePaneAGeneratedZdotdir() {
FakeHerdr herdr = new FakeHerdr();
WiringLauncher launcher = new WiringLauncher(herdr, allowList());
launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
String paneParams = String.valueOf(herdr.lastCall("pane.split").params());
assertTrue(paneParams.contains("ZDOTDIR"),
"the spawn must hand herdr a ZDOTDIR so the pane's zsh reads our generated startup "
+ "files; without it the scrub never runs and the member inherits the whole "
+ "host environment. pane.split params were: " + paneParams);
Path dir = Path.of(launcher.env.get("ZDOTDIR"));
assertTrue(Files.isDirectory(dir), "ZDOTDIR must point at a directory that exists: " + dir);
// Both startup files must exist and both must source the scrub: .zlogin covers macOS panes
// (login shells), .zshrc covers Linux panes (interactive, NOT login). Checking only one
// would pass on the platform it was written for and ship a dead control on the other.
for (String file : List.of(".zshrc", ".zlogin")) {
Path f = dir.resolve(file);
assertTrue(Files.isRegularFile(f), file + " must be generated: " + f);
assertTrue(readAll(f).contains(EnvAllowListScrub.SCRUB_FILE),
file + " must source " + EnvAllowListScrub.SCRUB_FILE + " — a scrub only one of "
+ "them runs is dead on the platform that reads the other");
}
assertTrue(readAll(dir.resolve(EnvAllowListScrub.SCRUB_FILE)).contains(INJECTED),
"the allow-list must include the names this very launch injects (" + INJECTED
+ "), or the daemon's own configuration is blanked by its own control");
}
/** The default policy must not generate anything — an upgrade changes nothing until asked. */
@Test
void spawningUnderTheDefaultPolicyGeneratesNoZdotdir() {
FakeHerdr herdr = new FakeHerdr();
WiringLauncher launcher = new WiringLauncher(herdr, () -> new FleetConfig.MemberCredentials(
null, List.of(), List.of(), null));
launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
assertFalse(launcher.env.containsKey("ZDOTDIR"),
"policy=deny-by-default is the shipped default; it must not silently start "
+ "rewriting members' shell startup files");
}
/**
* A non-zsh shell cannot read {@code ZDOTDIR} at all. The launcher must fall back rather than
* generate a directory nothing will ever read — a directory that would look like protection.
*/
@Test
void aNonZshShellGeneratesNothingAndFallsBack() {
FakeHerdr herdr = new FakeHerdr();
WiringLauncher launcher = new WiringLauncher(herdr, allowList(), "/bin/bash");
launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
assertFalse(launcher.env.containsKey("ZDOTDIR"),
"bash ignores ZDOTDIR; setting it would be protection theatre");
}
private static Supplier<FleetConfig.MemberCredentials> allowList() {
return () -> new FleetConfig.MemberCredentials(
FleetConfig.MemberCredentials.POLICY_ALLOW_LIST, List.of(), List.of(), null);
}
private static String readAll(Path p) {
try {
return Files.readString(p);
} catch (java.io.IOException e) {
throw new AssertionError("cannot read " + p, e);
}
}
private static FleetConfig.Profile profile() {
return new FleetConfig.Profile("test", "http://gx00.gw:8000", null, null,
"BRIDGED_WORKER_TOKEN", List.of("test"), "pane", null, null, null, null, null);
}
/**
* A minimal peer launcher whose {@code buildLaunch} returns a MUTABLE env map holding one name
* the daemon injects. Mutable on purpose: the policy adds {@code ZDOTDIR} to this very map, so
* an immutable one would throw and the test would pass for the wrong reason.
*/
private static final class WiringLauncher extends HerdrPeerLauncher {
private final Map<String, String> env = new HashMap<>(Map.of(INJECTED, "http://gateway"));
WiringLauncher(FakeHerdr herdr, Supplier<FleetConfig.MemberCredentials> creds) {
this(herdr, creds, "/bin/zsh");
}
WiringLauncher(FakeHerdr herdr, Supplier<FleetConfig.MemberCredentials> creds, String shell) {
super("test", new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of("test", profile()), "test",
name -> "SHELL".equals(name) ? shell : null,
0, () -> 0L, () -> { }, null, creds);
}
@Override
protected Launch buildLaunch(FleetConfig.Profile cfg, LaunchSpec spec) {
return new Launch(env, List.of("test"));
}
@Override
public Set<Capability> capabilities() {
return Set.of();
}
}
/** The generated directory is a temp directory; make sure the test does not leave a pile. */
@Test
void theGeneratedDirectoryIsRemovedWhenThePaneIsStopped() {
FakeHerdr herdr = new FakeHerdr();
WiringLauncher launcher = new WiringLauncher(herdr, allowList());
var spawned = launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
Path dir = Path.of(launcher.env.get("ZDOTDIR"));
assertNotNull(spawned, "spawn returned nothing");
assertTrue(Files.isDirectory(dir));
launcher.stop(spawned.id());
assertEquals(false, Files.exists(dir),
"stopping the pane must remove its generated ZDOTDIR: " + dir);
}
}
@@ -1,614 +0,0 @@
package dev.ltms.fleet.member;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.peer.Capability;
import dev.ltms.fleet.peer.CharterReceipt;
import dev.ltms.fleet.peer.MemberRole;
import dev.ltms.fleet.peer.PeerHandle;
import dev.ltms.fleet.peer.PeerUnreachableException;
import dev.ltms.fleet.peer.SpawnRequest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.function.Supplier;
import static org.junit.jupiter.api.Assertions.*;
/**
* The opencode adapter's launch build: a file-based MCP mount + reply-charter instructions (no
* inline flags, no {@code ANTHROPIC_*}, no guard), the {@code -m} model flag, and the shared base
* transport (naming, reap, readiness gate) proving the {@link HerdrPeerLauncher} SPI is neutral.
*/
class OpenCodeLauncherTest {
private static FleetConfig.Profile opencodeCfg(String model, String mcpUrl, String gitTokenEnv) {
return new FleetConfig.Profile("gemini", null, model, null, "BRIDGED_WORKER_TOKEN",
List.of("opencode"), "tab", "bridged-workers", "opencode: {model} #{n}", mcpUrl,
null, null, gitTokenEnv, null, FleetConfig.Profile.KIND_OPENCODE);
}
/** Gate-disabled launcher whose per-spawn config dirs land under an inspectable temp root. */
private static OpenCodeLauncher service(FakeHerdr herdr, Path configRoot, FleetConfig.Profile cfg) {
return new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of(cfg.profile(), cfg), cfg.profile(), k -> "GITEA_ACCESS_TOKEN".equals(k) ? "tok" : null,
0, System::currentTimeMillis, () -> { }, configRoot, configRoot);
}
private static OpenCodeLauncher service(FakeHerdr herdr, Path configRoot, FleetConfig.Profile cfg,
Supplier<FleetConfig.Fleet> fleet) {
return new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null,
0, System::currentTimeMillis, () -> { }, configRoot, configRoot, fleet);
}
private static OpenCodeLauncher serviceWithCredentials(FakeHerdr herdr, Path configRoot,
FleetConfig.Profile cfg,
FleetConfig.MemberCredentials creds) {
return new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of(cfg.profile(), cfg), cfg.profile(), k -> "GITEA_ACCESS_TOKEN".equals(k) ? "tok" : null,
0, System::currentTimeMillis, () -> { }, configRoot, configRoot, null, () -> creds);
}
@SuppressWarnings("unchecked")
private static Map<String, Object> lastStart(FakeHerdr herdr) {
return (Map<String, Object>) herdr.lastCall("agent.start").params();
}
/** Protocol 19: the worker's env is injected at pane creation (tab.create), not agent.start. */
@SuppressWarnings("unchecked")
private static Map<String, String> startEnv(FakeHerdr herdr) {
Map<String, String> env =
(Map<String, String>) ((Map<String, Object>) herdr.lastCall("tab.create").params()).get("env");
return env == null ? Map.of() : env;
}
/** Protocol 19: agent.start carries only the args after the kind-resolved executable. */
@SuppressWarnings("unchecked")
private static List<String> startArgs(FakeHerdr herdr) {
return (List<String>) lastStart(herdr).get("args");
}
@Test
void writesRemoteMcpConfigAndCharterInstructionsWhenMcpUrlSet(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Fleet fleet = new FleetConfig.Fleet(Map.of(), Map.of(), Map.of(), Map.of(),
Map.of("dev", "role rule"), null);
service(herdr, root, opencodeCfg("google/gemini-2.5-pro", "http://127.0.0.1:8765/mcp", null),
() -> fleet).spawn();
Map<String, String> env = startEnv(herdr);
assertNull(env.get("ANTHROPIC_BASE_URL"), "opencode carries no ANTHROPIC_* / subscription boundary");
String cfgPath = env.get("OPENCODE_CONFIG");
assertNotNull(cfgPath, "OPENCODE_CONFIG points the worker at the generated config file");
assertTrue(Path.of(cfgPath).startsWith(root), "config file is generated under the injected root");
// Assert on parsed structure, not substrings: the generated config is real JSON and its
// whitespace is the formatter's business, not the contract's.
JsonNode json = new ObjectMapper().readTree(Path.of(cfgPath).toFile());
assertTrue(json.path("compaction").path("auto").asBoolean(),
"spawned opencode peers explicitly enable automatic compaction");
JsonNode mount = json.path("mcp").path("fleet");
assertEquals("remote", mount.path("type").asText(), "the fleet MCP server is mounted as remote");
assertEquals("http://127.0.0.1:8765/mcp", mount.path("url").asText(),
"the profile's fleet MCP url is present");
assertTrue(mount.path("enabled").asBoolean(), "the fleet server is enabled");
assertTrue(json.path("mcp").path("bridge").isMissingNode(),
"the mount is named fleet since CB-632, not bridge");
assertTrue(json.path("instructions").isArray() && !json.path("instructions").isEmpty(),
"the member charter is mounted via instructions");
// The instructions entry is a real file path holding the composed member charter.
Path charter = Path.of(cfgPath).resolveSibling("member-charter.md");
assertTrue(Files.exists(charter), "the charter file the config references was written");
assertEquals("role rule\n\n" + HerdrPeerLauncher.REPLY_CHARTER, Files.readString(charter),
"the composed charter keeps the role rule first and the reply rule last");
assertEquals(charter.toAbsolutePath().toString(), json.path("instructions").get(0).asText(),
"instructions names the charter file by its absolute path");
}
@Test
void noConfigFileWhenMcpUrlAbsent(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg("google/gemini-2.5-pro", null, null)).spawn();
assertNull(startEnv(herdr).get("OPENCODE_CONFIG"),
"no bridge MCP url → no config file and no OPENCODE_CONFIG");
}
@Test
void roleCharterWithoutMcpOrCustomProviderStillWritesAConfig(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Fleet fleet = new FleetConfig.Fleet(Map.of(), Map.of(), Map.of(), Map.of(),
Map.of("dev", "role rule"), null);
Path configRoot = Files.createDirectory(root.resolve("configs"));
Path checkout = Files.createDirectory(root.resolve("checkout"));
service(herdr, configRoot, opencodeCfg("google/gemini-2.5-pro", null, null), () -> fleet).spawn();
String cfgPath = startEnv(herdr).get("OPENCODE_CONFIG");
assertNotNull(cfgPath, "a role charter needs a config even without MCP or custom provider");
JsonNode json = new ObjectMapper().readTree(Path.of(cfgPath).toFile());
Path charter = Path.of(json.path("instructions").get(0).asText());
assertEquals("role rule", Files.readString(charter), "the base-composed role charter is unchanged");
assertTrue(json.path("mcp").isMissingNode(), "a charter does not add an MCP mount");
assertTrue(charter.startsWith(configRoot), "the charter is written under the temp config root");
try (var files = Files.walk(checkout)) {
assertFalse(files.anyMatch(path -> path.getFileName().toString().equals("member-charter.md")),
"the worker checkout receives no charter file");
}
}
@Test
void nullCharterWritesNoCharterFileOrInstructions(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, pinnedCfg("local-vllm/model", "http://127.0.0.1:8000", null),
() -> new FleetConfig.Fleet(Map.of(), Map.of(), Map.of(), Map.of(), Map.of(), null)).spawn();
String config = startEnv(herdr).get("OPENCODE_CONFIG");
assertNotNull(config, "the custom provider still needs a config");
JsonNode json = new ObjectMapper().readTree(Path.of(config).toFile());
assertTrue(json.path("instructions").isMissingNode(), "a null charter adds no instructions entry");
try (var files = Files.walk(root)) {
assertFalse(files.anyMatch(path -> path.getFileName().toString().equals("member-charter.md")),
"a null charter creates no charter file");
}
}
@Test
void concurrentSpawnsWriteSeparateCharterDirectories(@TempDir Path root) throws Exception {
FleetConfig.Profile cfg = opencodeCfg("google/gemini-2.5-pro", "http://127.0.0.1:8765/mcp", null);
ExecutorService executor = Executors.newFixedThreadPool(2);
try {
Future<String> first = executor.submit(() -> spawnConfigPath(root, cfg));
Future<String> second = executor.submit(() -> spawnConfigPath(root, cfg));
Path firstCharter = Path.of(first.get()).resolveSibling("member-charter.md");
Path secondCharter = Path.of(second.get()).resolveSibling("member-charter.md");
assertNotEquals(firstCharter.getParent(), secondCharter.getParent(),
"each concurrent spawn owns a separate config directory");
assertTrue(Files.exists(firstCharter));
assertTrue(Files.exists(secondCharter));
} finally {
executor.shutdownNow();
}
}
private static String spawnConfigPath(Path root, FleetConfig.Profile cfg) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, cfg).spawn();
return startEnv(herdr).get("OPENCODE_CONFIG");
}
@Test
void passesTheModelAsDashMFlagAlongsideAutoApprove(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg("google/gemini-2.5-pro", null, null)).spawn();
List<String> args = startArgs(herdr);
assertTrue(args.contains("--auto"),
"--auto is present alongside -m so a spawned peer never blocks on approval");
int m = args.indexOf("-m");
assertTrue(m >= 0, "model is selected with -m");
assertEquals("google/gemini-2.5-pro", args.get(m + 1), "the provider/model selector follows -m");
}
@Test
void autoApproveIsUnconditionalWhenModelBlank(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg(null, null, null)).spawn();
assertEquals(List.of("--auto"), startArgs(herdr),
"--auto is unconditional: a model-less worker still must never block on approval");
}
// --- CB-617: --agent <role> when the role has an agent-definition file --------------------
@Test
void agentFlagIsPassedWhenTheRoleAgentDefinitionFileExists(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
Path agentsDir = Files.createDirectories(root.resolve(".opencode/agent"));
Files.writeString(agentsDir.resolve("dev.md"), "You are a dev.");
OpenCodeLauncher svc = service(herdr, root, opencodeCfg(null, null, null));
svc.spawn(new SpawnRequest(null, root.toString(), null, null, null, MemberRole.DEV));
List<String> args = startArgs(herdr);
int flag = args.indexOf("--agent");
assertTrue(flag >= 0, "--agent is passed when the role's agent file exists: " + args);
assertEquals("dev", args.get(flag + 1));
}
@Test
void noAgentFlagWhenTheRoleAgentDefinitionFileIsAbsent(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
OpenCodeLauncher svc = service(herdr, root, opencodeCfg(null, null, null));
PeerHandle handle = svc.spawn(new SpawnRequest(null, root.toString(), null, null, null, MemberRole.DEV));
assertNotNull(handle, "the member still spawns with no agent-definition file");
assertFalse(startArgs(herdr).contains("--agent"),
"no --agent flag when the role has no agent-definition file");
}
@Test
void injectsForgeTokenWhenProfileGrantsIt(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg(null, null, "GITEA_ACCESS_TOKEN")).spawn();
assertEquals("tok", startEnv(herdr).get("GITEA_TOKEN"),
"a git-token profile gets the peer-neutral GITEA_TOKEN grant, same as Claude");
}
/**
* CB-596: the config-driven shadow lives in {@link HerdrPeerLauncher#baseEnv}, shared by every
* adapter — this pins that the opencode path gets it too, not just Claude's. See the matching
* tests in {@code ClaudeCodeLauncherTest} for the full rationale (gitea #82, superseding CB-592's
* single hardcoded name).
*/
@Test
void aKnownNameNotAllowedIsShadowedWithTheSentinel(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.MemberCredentials creds = new FleetConfig.MemberCredentials(
null, List.of("AI_GATEWAY_TOKEN"), List.of("AI_GATEWAY_TOKEN", "GITEA_ACCESS_TOKEN"));
serviceWithCredentials(herdr, root, opencodeCfg(null, null, null), creds).spawn();
String shadowed = startEnv(herdr).get("GITEA_ACCESS_TOKEN");
assertNotNull(shadowed, "GITEA_ACCESS_TOKEN must be explicitly overlaid, not left unmentioned");
assertFalse(shadowed.isBlank(), "a blank overlay value's override behaviour is unverified — must be non-blank");
assertFalse(startEnv(herdr).containsKey("AI_GATEWAY_TOKEN"),
"an allow-listed name must get no overlay entry at all");
}
/** No {@code memberCredentials} configured (the pre-CB-596 constructor overloads) blocks nothing. */
@Test
void noMemberCredentialsConfiguredBlocksNothing(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg(null, null, null)).spawn();
assertNull(startEnv(herdr).get("GITEA_ACCESS_TOKEN"),
"with no memberCredentials configured, nothing is shadowed — config must supply the policy");
}
@Test
void capabilitiesDeclareOrphanReapAndMcpAskAndConditionalSelfPr(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
assertEquals(java.util.Set.of(Capability.MID_TURN_ASK, Capability.WORKTREE, Capability.ORPHAN_REAP,
Capability.SESSION_RESUME),
service(herdr, root, opencodeCfg(null, null, null)).capabilities(),
"opencode can be resumed by its own session id, so SESSION_RESUME is always declared");
assertFalse(service(herdr, root, opencodeCfg(null, null, null))
.capabilities().contains(Capability.SESSION_NAME),
"opencode has no display-name flag, so SESSION_NAME must NOT be declared");
assertTrue(service(herdr, root, opencodeCfg(null, null, "GITEA_ACCESS_TOKEN"))
.capabilities().contains(Capability.SELF_PR),
"a git-token profile adds SELF_PR");
}
// --- CB-547: resume + post-hoc session discovery --------------------------------------------
@Test
void aResumeSpawnPassesTheSessionIdAsDashS(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg("google/gemini-2.5-pro", null, null))
.spawn(new SpawnRequest(null, null, null, null, "ses_41b79fc90ffeI9E8uZv6VprUn2"));
List<String> args = startArgs(herdr);
int s = args.indexOf("-s");
assertTrue(s >= 0, "a resumed spawn carries opencode's -s flag");
assertEquals("ses_41b79fc90ffeI9E8uZv6VprUn2", args.get(s + 1),
"the resume target id follows -s");
}
@Test
void aFreshSpawnCarriesNoSessionFlag(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg(null, null, null))
.spawn(new SpawnRequest(null, null, null, null, null));
assertFalse(startArgs(herdr).contains("-s"),
"no resume target → a fresh session with no -s flag");
}
@Test
void theHandleDiscoversTheSessionIdForTheWorkersCwdOnlyAfterItAppears(@TempDir Path root,
@TempDir Path discRoot)
throws Exception {
FakeHerdr herdr = new FakeHerdr();
OpenCodeLauncher launcher = new OpenCodeLauncher(new AgentControl(herdr),
new WorkspaceControl(herdr), Map.of("gemini", opencodeCfg(null, null, null)),
"gemini", _ -> null, 0, System::currentTimeMillis, () -> { }, root, discRoot);
PeerHandle handle = launcher.spawn(new SpawnRequest(null, "/work/dir", null));
// opencode writes the record only when the session is first persisted — the instant the
// pane is ready it does not exist, so agentSessionId() is null (never a spawn failure).
assertNull(handle.agentSessionId(), "no record yet → null, not a spawn-time block");
// Once the record appears (here: same cwd), lazy discovery resolves it — the handle's
// session id matches its own worktree, not another's.
OpenCodeSessionDiscoveryTest.writeRecord(discRoot, "p1", "ses_a.json",
"ses_resolved", "/work/dir", 1000L);
assertEquals("ses_resolved", handle.agentSessionId(),
"agentSessionId() re-scans and picks up a record that has since been written");
}
@Test
void foreignWorkerMatchesOpencodePrefixButNotClaude() {
String nonce = "abc123";
assertTrue(OpenCodeLauncher.isForeignWorker("opencode-gemini-def456-1", nonce),
"an opencode pane from another process is foreign");
assertFalse(OpenCodeLauncher.isForeignWorker("opencode-gemini-" + nonce + "-1", nonce),
"our own opencode pane (same nonce) is not foreign");
assertFalse(OpenCodeLauncher.isForeignWorker("claude-ltms-local-def456-1", nonce),
"a claude pane is never reaped by the opencode adapter");
}
@Test
void productionConstructorsWireThroughToTheBase() {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = opencodeCfg(null, null, null);
// 5-arg (gate disabled) and 7-arg (gate enabled) production constructors both expose the profile.
OpenCodeLauncher disabled = new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
OpenCodeLauncher gated = new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null, 5000, 100);
assertEquals(java.util.Set.of("gemini"), disabled.profiles());
assertEquals("gemini", gated.defaultProfile());
}
@Test
void spawnGateThrowsPeerUnreachableWhenNeverInjectable(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
herdr.agentStatus("unknown"); // never injectable
long[] clock = {0};
OpenCodeLauncher svc = new OpenCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
Map.of("gemini", opencodeCfg(null, null, null)), "gemini", _ -> null,
1000, () -> clock[0], () -> clock[0] += 50, root, root);
PeerUnreachableException ex = assertThrows(PeerUnreachableException.class,
() -> svc.spawn(new SpawnRequest(null, null, null)));
assertTrue(clock[0] >= 1000, "the fake clock advanced past the timeout: " + clock[0]);
long closes = herdr.calls.stream()
.filter(c -> c.method().equals("pane.close"))
.filter(c -> "w9:pRoot_1".equals(((Map<?, ?>) c.params()).get("pane_id")))
.count();
assertEquals(1, closes, "the worker pane was reaped on timeout (no orphan)");
assertNotNull(ex.getMessage());
}
@Test
void spawnReturnsHandleWhenGateDisabled(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
PeerHandle handle = service(herdr, root, opencodeCfg(null, null, null))
.spawn(new SpawnRequest(null, null, null));
assertNotNull(handle, "spawn returns a handle when the gate is disabled");
assertFalse(herdr.called("agent.get"), "no polling when the gate is disabled");
}
@Test
void handleCarriesTheRealCharterReceiptNotTheInterfaceDefault(@TempDir Path root) {
// The base's WorkerHandle computes a real CharterReceipt (CB-571), but the opencode adapter
// wraps it in SessionAwareHandle for lazy session discovery. Before this fix that decorator
// did not override charterReceipt(), so it silently inherited PeerHandle's `null` default
// and the real receipt sitting on its delegate was lost.
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Fleet fleet = new FleetConfig.Fleet(Map.of(), Map.of(), Map.of(), Map.of(),
Map.of("dev", "role rule"), null);
PeerHandle handle = service(herdr, root,
opencodeCfg("google/gemini-2.5-pro", "http://127.0.0.1:8765/mcp", null), () -> fleet)
.spawn(new SpawnRequest(null, null, null));
assertNotNull(handle.charterReceipt(),
"an opencode spawn's charterReceipt() must not silently be null");
String composed = "role rule\n\n" + HerdrPeerLauncher.REPLY_CHARTER;
assertEquals(CharterReceipt.digestOf(composed), handle.charterReceipt().charterSha256(),
"the receipt on the wrapped handle must match the exact composed charter bytes");
}
// --- CB-508: pinned OpenAI-compatible endpoint (e.g. a local vLLM) ---------------------------
/** A profile with a baseUrl but no model provider prefix cannot be resolved — fail loudly. */
private static FleetConfig.Profile pinnedCfg(String model, String baseUrl, String mcpUrl) {
return new FleetConfig.Profile("local", baseUrl, model, null, "BRIDGED_WORKER_TOKEN",
List.of("opencode"), "tab", "bridged-workers", "opencode: {model} #{n}", mcpUrl,
null, null, null, null, FleetConfig.Profile.KIND_OPENCODE);
}
@Test
void baseUrlDeclaresACustomOpenAiCompatibleProvider(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, pinnedCfg("local-vllm/deepseek-v4-flash", "http://127.0.0.1:8000", null))
.spawn();
String cfgPath = startEnv(herdr).get("OPENCODE_CONFIG");
assertNotNull(cfgPath, "a pinned endpoint needs a config file even with no bridge MCP url");
JsonNode provider = new ObjectMapper().readTree(Path.of(cfgPath).toFile())
.path("provider").path("local-vllm");
assertFalse(provider.isMissingNode(), "the provider id comes from the model selector");
assertEquals("@ai-sdk/openai-compatible", provider.path("npm").asText());
assertEquals("http://127.0.0.1:8000/v1", provider.path("options").path("baseURL").asText(),
"a bare host:port gets /v1 appended — that is where these servers mount the API");
assertFalse(provider.path("options").path("apiKey").asText().isBlank(),
"the AI SDK requires a non-empty key even when the server ignores it");
assertFalse(provider.path("models").path("deepseek-v4-flash").isMissingNode(),
"the model half of the selector is declared under the provider");
}
@Test
void aBaseUrlThatAlreadyCarriesAPathIsUsedVerbatim(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, pinnedCfg("local-vllm/m", "http://127.0.0.1:8000/openai/v1", null)).spawn();
JsonNode json = new ObjectMapper()
.readTree(Path.of(startEnv(herdr).get("OPENCODE_CONFIG")).toFile());
assertEquals("http://127.0.0.1:8000/openai/v1",
json.path("provider").path("local-vllm").path("options").path("baseURL").asText(),
"an endpoint mounted on a custom path must not have /v1 bolted on");
}
@Test
void aPinnedEndpointRejectsAModelWithNoProviderPrefix(@TempDir Path root) {
FakeHerdr herdr = new FakeHerdr();
OpenCodeLauncher launcher =
service(herdr, root, pinnedCfg("deepseek-v4-flash", "http://127.0.0.1:8000", null));
// Silently falling back to the default gateway would point the worker at the wrong LLM
// while looking healthy — the one failure mode worth being loud about.
IllegalArgumentException e = assertThrows(IllegalArgumentException.class, launcher::spawn);
assertTrue(e.getMessage().contains("<provider>/<model>"), "the error says how to fix it");
}
@Test
void aPinnedEndpointAndTheFleetMcpCoexistInOneConfig(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, pinnedCfg("local-vllm/deepseek-v4-flash",
"http://127.0.0.1:8000", "http://127.0.0.1:8766/mcp")).spawn();
JsonNode json = new ObjectMapper()
.readTree(Path.of(startEnv(herdr).get("OPENCODE_CONFIG")).toFile());
assertEquals("remote", json.path("mcp").path("fleet").path("type").asText(),
"pinning an endpoint must not drop the fleet MCP mount");
assertFalse(json.path("provider").path("local-vllm").isMissingNode(),
"and the provider block is still declared alongside it");
assertTrue(json.path("instructions").isArray() && !json.path("instructions").isEmpty(),
"the reply charter survives too");
}
@Test
void noBaseUrlDeclaresNoProviderSoTheDefaultGatewayIsUsed(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg("opencode/some-free-model", "http://127.0.0.1:8766/mcp", null))
.spawn();
JsonNode json = new ObjectMapper()
.readTree(Path.of(startEnv(herdr).get("OPENCODE_CONFIG")).toFile());
assertTrue(json.path("provider").isMissingNode(),
"without a baseUrl opencode resolves its own provider as before");
}
// --- autoCompactWindow: opencode has no absolute compact-at-N knob, so this is applied as the
// model's own limit.context, only when model: resolves to "provider/model" -----------------
private static FleetConfig.Profile opencodeCfgWithAutoCompactWindow(String model, String baseUrl,
Integer window) {
return new FleetConfig.Profile("gemini", baseUrl, model, null, "BRIDGED_WORKER_TOKEN",
List.of("opencode"), "tab", "bridged-workers", "opencode: {model} #{n}", null,
null, null, null, null, FleetConfig.Profile.KIND_OPENCODE, Map.of(), null, null,
null, null, null, null, null, null, window);
}
@Test
void autoCompactWindowIsAppliedAsThePerModelContextLimit(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfgWithAutoCompactWindow("openai/gpt-5", null, 250_000)).spawn();
String cfgPath = startEnv(herdr).get("OPENCODE_CONFIG");
assertNotNull(cfgPath, "autoCompactWindow alone must trigger config generation, with no MCP"
+ " and no custom provider set");
JsonNode limit = new ObjectMapper().readTree(Path.of(cfgPath).toFile())
.path("provider").path("openai").path("models").path("gpt-5").path("limit");
assertEquals(250_000, limit.path("context").asInt());
assertEquals(16384, limit.path("output").asInt(),
"opencode's limit schema requires both keys; output gets a safe documented default");
}
@Test
void autoCompactWindowMergesIntoACustomProviderRatherThanOverwritingIt(@TempDir Path root)
throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfgWithAutoCompactWindow(
"local-vllm/deepseek-v4-flash", "http://127.0.0.1:8000", 300_000)).spawn();
JsonNode provider = new ObjectMapper()
.readTree(Path.of(startEnv(herdr).get("OPENCODE_CONFIG")).toFile())
.path("provider").path("local-vllm");
assertEquals("@ai-sdk/openai-compatible", provider.path("npm").asText(),
"addCustomProvider's own fields must survive the later limit merge");
assertEquals(300_000, provider.path("models").path("deepseek-v4-flash")
.path("limit").path("context").asInt());
assertEquals("deepseek-v4-flash", provider.path("models").path("deepseek-v4-flash")
.path("name").asText(),
"the model's pre-existing 'name' field must survive the limit merge too");
}
@Test
void autoCompactWindowWithNoProviderSlashInModelGetsNoLimitAndAWarn(@TempDir Path root)
throws Exception {
FakeHerdr herdr = new FakeHerdr();
// mcpUrl set too, only so a config file gets written at all to inspect; a bare model name
// with no other config-triggering knob would leave OPENCODE_CONFIG unset entirely, which is
// also correct (nothing to write) but not what this test is asserting.
service(herdr, root, new FleetConfig.Profile("gemini", null, "some-free-model", null,
"BRIDGED_WORKER_TOKEN", List.of("opencode"), "tab", "bridged-workers",
"opencode: {model} #{n}", "http://127.0.0.1:8765/mcp", null, null, null, null,
FleetConfig.Profile.KIND_OPENCODE, Map.of(), null, null, null, null, null, null,
null, null, 250_000)).spawn();
String cfgPath = startEnv(herdr).get("OPENCODE_CONFIG");
JsonNode json = new ObjectMapper().readTree(Path.of(cfgPath).toFile());
assertTrue(json.path("provider").isMissingNode(),
"a bare model name cannot be targeted at a specific provider/model limit entry — "
+ "no silent no-op, but also no broken partial write");
}
// --- CB-634: IDE Index MCP + guidance overlay (opencode does not read CLAUDE.local.md) -------
private static FleetConfig.Profile opencodeIdeCfg(String mcpUrl, String ideUrl, String cwd) {
return new FleetConfig.Profile("gemini", null, null, null, "BRIDGED_WORKER_TOKEN",
List.of("opencode"), "tab", "bridged-workers", "opencode: {model} #{n}", mcpUrl,
cwd, null, null, null, FleetConfig.Profile.KIND_OPENCODE,
null, null, null, null, null, null, ideUrl);
}
@Test
void ideMcpUrlAddsTheIntellijServerAndAnInstructionsRulesEntry(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
Path cwd = Files.createDirectory(root.resolve("checkout"));
service(herdr, root, opencodeIdeCfg("http://127.0.0.1:8765/mcp",
"http://127.0.0.1:29170/index-mcp/streamable-http", cwd.toString())).spawn();
String cfgPath = startEnv(herdr).get("OPENCODE_CONFIG");
assertNotNull(cfgPath, "an IDE profile needs a config file");
JsonNode json = new ObjectMapper().readTree(Path.of(cfgPath).toFile());
JsonNode ide = json.path("mcp").path("intellij");
assertEquals("remote", ide.path("type").asText(),
"the IDE server uses the same remote shape as the bridge mount");
assertEquals("http://127.0.0.1:29170/index-mcp/streamable-http", ide.path("url").asText());
assertTrue(ide.path("enabled").asBoolean(), "the IDE server is enabled");
assertEquals("remote", json.path("mcp").path("fleet").path("type").asText(),
"the bridge mount still coexists with the IDE server");
// The instructions array gains an entry pointing at a real rules file pinning the worktree.
String rulesContent = null;
for (JsonNode n : json.path("instructions")) {
Path p = Path.of(n.asText());
if (p.getFileName().toString().equals("ide-rules.md")) {
rulesContent = Files.readString(p);
}
}
assertNotNull(rulesContent, "an ide-rules.md instructions entry is present");
assertTrue(rulesContent.contains("project_path: \"" + cwd + "\""),
"the rules pin every ide_* call to the worker's own cwd");
}
@Test
void noIdeServerWhenIdeMcpUrlUnset(@TempDir Path root) throws Exception {
FakeHerdr herdr = new FakeHerdr();
service(herdr, root, opencodeCfg("google/gemini-2.5-pro", "http://127.0.0.1:8765/mcp", null))
.spawn();
JsonNode json = new ObjectMapper()
.readTree(Path.of(startEnv(herdr).get("OPENCODE_CONFIG")).toFile());
assertTrue(json.path("mcp").path("intellij").isMissingNode(),
"no IDE server when ideMcpUrl is unset");
}
}
@@ -1,90 +0,0 @@
package dev.ltms.fleet.member;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileTime;
import static org.junit.jupiter.api.Assertions.*;
/**
* {@link OpenCodeSessionDiscovery} matches an opencode session record by the worker's cwd (its
* {@code directory}) against opencode's on-disk storage. These tests populate a TEMP storage root
* themselves — never the operator's real {@code ~/.local/share/opencode}.
*/
class OpenCodeSessionDiscoveryTest {
/**
* Write a session record {@code {"id":..., "directory":...}} under
* {@code <root>/session/<projectID>/<fileName>} and stamp it with a known last-modified time,
* so "most recently modified wins" is deterministic. Static so the launcher test can reuse it.
*/
static void writeRecord(Path root, String projectId, String fileName, String id,
String directory, long lastModifiedEpochMillis) throws Exception {
Path dir = root.resolve("session").resolve(projectId);
Files.createDirectories(dir);
Path file = dir.resolve(fileName);
Files.writeString(file, "{\"id\":\"" + id + "\",\"directory\":\"" + directory
+ "\",\"projectID\":\"" + projectId + "\",\"version\":\"1.1.31\"}");
Files.setLastModifiedTime(file, FileTime.fromMillis(lastModifiedEpochMillis));
}
@Test
void findsTheRecordWhoseDirectoryEqualsTheCwd(@TempDir Path root) throws Exception {
writeRecord(root, "p1", "ses_a.json", "ses_aaa", "/w/a", 1000L);
writeRecord(root, "p2", "ses_b.json", "ses_bbb", "/w/b", 2000L);
assertEquals("ses_bbb", new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/b"),
"the record whose directory equals the cwd is the one found");
assertEquals("ses_aaa", new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/a"));
}
@Test
void aNonMatchingDirectoryYieldsNullRatherThanAMismatch(@TempDir Path root) throws Exception {
writeRecord(root, "p1", "ses_a.json", "ses_aaa", "/w/a", 1000L);
assertNull(new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/other"),
"no record for this cwd yet → null, not a wrong session");
}
@Test
void prefersTheMostRecentlyModifiedRecordWhenSeveralMatch(@TempDir Path root) throws Exception {
writeRecord(root, "p1", "old.json", "ses_old", "/w/a", 1000L);
writeRecord(root, "p2", "new.json", "ses_new", "/w/a", 5000L);
assertEquals("ses_new", new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/a"),
"the freshest record for the cwd wins");
}
@Test
void aMissingOrEmptyStorageRootYieldsNullWithoutThrowing(@TempDir Path root) throws Exception {
// Missing: no session dir at all under the root.
assertNull(new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/a"));
// Present but empty: a session dir with nothing in it produces no match, not a throw.
Path emptyRoot = root.resolve("empty");
Files.createDirectories(emptyRoot.resolve("session"));
assertNull(new OpenCodeSessionDiscovery(emptyRoot).sessionIdForDirectory("/w/a"));
}
@Test
void aBlankOrNullDirectoryYieldsNull(@TempDir Path root) {
OpenCodeSessionDiscovery discovery = new OpenCodeSessionDiscovery(root);
assertNull(discovery.sessionIdForDirectory(null));
assertNull(discovery.sessionIdForDirectory(" "));
}
@Test
void aMalformedRecordIsSkippedRatherThanFatal(@TempDir Path root) throws Exception {
// A record that fails to parse must not abort the scan of its siblings.
Path dir = root.resolve("session").resolve("p1");
Files.createDirectories(dir);
Files.writeString(dir.resolve("broken.json"), "{not valid json");
writeRecord(root, "p1", "good.json", "ses_good", "/w/a", 1000L);
assertEquals("ses_good", new OpenCodeSessionDiscovery(root).sessionIdForDirectory("/w/a"),
"an unreadable record is skipped; a later valid one still matches");
}
}
@@ -1,625 +0,0 @@
package dev.ltms.fleet.session;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HashSet;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import static org.junit.jupiter.api.Assertions.*;
/**
* CB-525 acceptance test for tool-surface isolation. This is one of the few tests that drives real
* {@code git} — the behaviour under test is precisely what {@link GitWorktrees} does to a checkout,
* so a fake would assert nothing. Everything happens inside a {@link TempDir} throwaway repo.
*/
class GitWorktreesTest {
/** A project MCP config with servers in it — what this repo actually commits. */
private static final String WITH_SERVERS = """
{
"mcpServers": {
"jetbrains": { "type": "sse", "url": "http://localhost:64342/sse" }
}
}
""";
/** An opencode config carrying a {@code {file:.secrets/...}} reference — CB-543's crash repro. */
private static final String OPENCODE_WITH_FILE_REF = """
{
"env": {
"CONTEXT7_TOKEN": "{file:.secrets/context7-token}"
}
}
""";
/** A non-empty autoenv file — the form that would prompt for authorization in a worktree. */
private static final String AUTOENV_WITH_DIRECTIVE = "export HELLO=world\n";
private static Path initRepo(Path dir) throws Exception {
Files.createDirectories(dir);
git(dir, "init", "-q", "-b", "main");
git(dir, "config", "user.email", "test@example.invalid");
git(dir, "config", "user.name", "Test");
Files.writeString(dir.resolve(".mcp.json"), WITH_SERVERS);
Files.writeString(dir.resolve("README.md"), "seed\n");
git(dir, "add", ".mcp.json", "README.md");
git(dir, "commit", "-q", "-m", "seed");
return dir;
}
private static void git(Path cwd, String... args) throws Exception {
List<String> cmd = new java.util.ArrayList<>(List.of("git"));
cmd.addAll(List.of(args));
Process p = new ProcessBuilder(cmd).directory(cwd.toFile()).redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git timed out: " + String.join(" ", cmd));
assertEquals(0, p.exitValue(), "git " + String.join(" ", args) + " failed:\n" + out);
}
/** Pending changes to {@code file} in {@code cwd}, empty when git considers it unmodified. */
private static String status(Path cwd, String file) throws Exception {
Process p = new ProcessBuilder("git", "status", "--porcelain", "--", file)
.directory(cwd.toFile()).redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
return out;
}
/** Every pending change in {@code cwd} — the whole-tree porcelain status, unlike {@link #status}. */
private static String fullStatus(Path cwd) throws Exception {
Process p = new ProcessBuilder("git", "status", "--porcelain")
.directory(cwd.toFile()).redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
return out;
}
private static String revParse(Path cwd, String ref) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "rev-parse", ref)
.redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git rev-parse timed out");
assertEquals(0, p.exitValue(), "git rev-parse " + ref + " failed:\n" + out);
return out;
}
/** The recursive file list of a commit's tree — used to check what a snapshot actually committed. */
private static String lsTree(Path cwd, String ref) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "ls-tree", "-r", "--name-only", ref)
.redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git ls-tree timed out");
assertEquals(0, p.exitValue(), "git ls-tree " + ref + " failed:\n" + out);
return out;
}
/** The set of paths in {@code git diff --name-only from..to} — used to check exactly what a
* snapshot's tree changed relative to its parent, the same shape {@code git status --porcelain}
* reports for the worktree it was taken from. */
private static Set<String> diffNameOnly(Path cwd, String from, String to) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "diff", "--name-only", from, to)
.redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git diff timed out");
assertEquals(0, p.exitValue(), "git diff " + from + ".." + to + " failed:\n" + out);
Set<String> paths = new HashSet<>();
for (String line : out.split("\\R")) {
if (!line.isBlank()) {
paths.add(line.trim());
}
}
return paths;
}
/** The set of paths a {@code git status --porcelain} listing names, stripping the two-char status
* code prefix each line carries. */
private static Set<String> porcelainPaths(String porcelain) {
Set<String> paths = new HashSet<>();
for (String line : porcelain.split("\\R")) {
if (!line.isBlank()) {
paths.add(line.substring(3).trim());
}
}
return paths;
}
private static String forEachRef(Path cwd, String pattern) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "for-each-ref", pattern)
.redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git for-each-ref timed out");
assertEquals(0, p.exitValue(), "git for-each-ref " + pattern + " failed:\n" + out);
return out;
}
/** Write {@code content} as a blob into the object database; returns its sha. */
private static String blobOf(Path cwd, String content) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "hash-object", "-w", "--stdin")
.redirectErrorStream(true).start();
p.getOutputStream().write(content.getBytes(StandardCharsets.UTF_8));
p.getOutputStream().close();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git hash-object timed out");
assertEquals(0, p.exitValue(), "git hash-object failed:\n" + out);
return out;
}
/** Build a single-file tree object from {@code blob}; returns the tree's sha. */
private static String treeOf(Path cwd, String path, String blob) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "mktree")
.redirectErrorStream(true).start();
p.getOutputStream().write(("100644 blob " + blob + "\t" + path + "\n").getBytes(StandardCharsets.UTF_8));
p.getOutputStream().close();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git mktree timed out");
assertEquals(0, p.exitValue(), "git mktree failed:\n" + out);
return out;
}
/** {@code git commit-tree} rooted at {@code tree} with a chosen committer date; returns the sha. */
private static String commitTree(Path cwd, String tree, String parent, String committerDate,
String message) throws Exception {
ProcessBuilder pb = new ProcessBuilder("git", "-C", cwd.toString(), "commit-tree",
tree, "-p", parent, "-m", message);
pb.environment().put("GIT_COMMITTER_DATE", committerDate);
Process p = pb.redirectErrorStream(true).start();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git commit-tree timed out");
assertEquals(0, p.exitValue(), "git commit-tree failed:\n" + out);
return out;
}
/** {@code git update-ref <ref> <sha>} — create the snapshot ref directly. */
private static void updateRef(Path cwd, String ref, String sha) throws Exception {
git(cwd, "update-ref", ref, sha);
}
/** True when {@code ref} exists in the repo (for-each-ref on a missing ref is empty, not an error). */
private static boolean refExists(Path cwd, String ref) throws Exception {
return !forEachRef(cwd, ref).trim().isEmpty();
}
/**
* The heart of CB-525: a provisioned worktree must not inherit the primary's MCP servers. Without
* the isolation step the checked-out {@code .mcp.json} carries them in, and a worker navigating
* through the primary's IDE servers edits the primary's tree while building its own.
*/
@Test
void aProvisionedWorktreeInheritsNoMcpServers(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-525-a", "HEAD");
Path mcp = Path.of(wt).resolve(".mcp.json");
assertTrue(Files.exists(mcp), ".mcp.json must still exist — present and explicitly empty");
String body = Files.readString(mcp);
assertFalse(body.contains("jetbrains"), "worktree inherited the primary's MCP servers:\n" + body);
assertTrue(body.replaceAll("\\s+", "").contains("\"mcpServers\":{}"),
"expected an explicitly empty server map, got:\n" + body);
}
/** Neutralizing must not look like work in progress, or a worker would commit it into its PR. */
@Test
void theNeutralizedConfigIsNotAPendingLocalModification(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-525-b", "HEAD");
assertEquals("", status(Path.of(wt), ".mcp.json"),
"the neutralized .mcp.json shows as modified — --skip-worktree did not take");
}
/** Isolation is the worktree's business only; the primary's own checkout must be untouched. */
@Test
void thePrimaryCheckoutIsLeftAlone(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
new GitWorktrees(tmp.resolve("wts").toString()).add(repo.toString(), "cb-525-c", "HEAD");
assertEquals(WITH_SERVERS, Files.readString(repo.resolve(".mcp.json")),
"the primary's .mcp.json was rewritten — isolation reached out of the worktree");
}
/** A repo that commits no {@code .mcp.json} still gets one, so nothing can be inherited later. */
@Test
void aRepoWithoutAnMcpConfigStillGetsANeutralOne(@TempDir Path tmp) throws Exception {
Path repo = tmp.resolve("repo");
Files.createDirectories(repo);
git(repo, "init", "-q", "-b", "main");
git(repo, "config", "user.email", "test@example.invalid");
git(repo, "config", "user.name", "Test");
Files.writeString(repo.resolve("README.md"), "seed\n");
git(repo, "add", "README.md");
git(repo, "commit", "-q", "-m", "seed");
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-525-d", "HEAD");
// Untracked is the normal case here, so the --skip-worktree branch must be skipped rather
// than run and fail: `update-index --skip-worktree` on an unknown path exits non-zero.
String body = Files.readString(Path.of(wt).resolve(".mcp.json"));
assertTrue(body.replaceAll("\\s+", "").contains("\"mcpServers\":{}"), body);
}
/**
* CB-543's repro: a tracked {@code opencode.json} carries a {@code {file:.secrets/...}} reference
* to a gitignored secret that never reaches a worktree, and opencode refuses to start on it. The
* worktree's copy must be neutralized and hidden like {@code .mcp.json}.
*/
@Test
void aTrackedOpencodeConfigIsNeutralizedAndHidden(@TempDir Path tmp) throws Exception {
Path repo = tmp.resolve("repo");
Files.createDirectories(repo);
git(repo, "init", "-q", "-b", "main");
git(repo, "config", "user.email", "test@example.invalid");
git(repo, "config", "user.name", "Test");
Files.writeString(repo.resolve(".mcp.json"), WITH_SERVERS);
Files.writeString(repo.resolve("opencode.json"), OPENCODE_WITH_FILE_REF);
Files.writeString(repo.resolve("README.md"), "seed\n");
git(repo, "add", ".mcp.json", "opencode.json", "README.md");
git(repo, "commit", "-q", "-m", "seed");
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-543-a", "HEAD");
String body = Files.readString(Path.of(wt).resolve("opencode.json"));
assertFalse(body.contains(".secrets"),
"worktree kept a dangling {file:...} secret reference:\n" + body);
assertEquals("{}", body.replaceAll("\\s+", ""),
"expected an empty JSON object stub, got:\n" + body);
assertEquals("", status(Path.of(wt), "opencode.json"),
"the neutralized opencode.json shows as modified — --skip-worktree did not take");
}
/** A config the repo does not carry must be skipped — no stub invented, provisioning still succeeds. */
@Test
void anAbsentConfigIsSkippedWithoutError(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo")); // only .mcp.json + README are committed
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-543-b", "HEAD");
assertFalse(Files.exists(Path.of(wt).resolve("opencode.json")),
"a stub was invented for a config the repo does not carry");
assertFalse(Files.exists(Path.of(wt).resolve(".autoenv")),
"a stub was invented for a config the repo does not carry");
// .mcp.json's long-standing create-always behaviour must be unchanged.
assertTrue(Files.exists(Path.of(wt).resolve(".mcp.json")), ".mcp.json stub was dropped");
}
/**
* CB-576. {@code hasUncommitted} must treat a freshly-provisioned worktree as clean, but a
* worktree holding a brand-new, never-added file as dirty. The untracked-file-only shape is
* exactly the work lost in the incident — a worker's draft that compiled but was never
* committed because it stopped to ask its lead a question.
*/
@Test
void anUntrackedOnlyWorktreeCountsAsDirty(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String wt = gitWorktrees.add(repo.toString(), "cb-576-u", "HEAD");
assertFalse(gitWorktrees.hasUncommitted(wt),
"a freshly provisioned worktree must read as clean");
Files.writeString(Path.of(wt).resolve("brand-new.txt"), "draft that was never added\n");
assertTrue(gitWorktrees.hasUncommitted(wt),
"an untracked-only file must count as dirty");
Files.writeString(Path.of(wt).resolve("README.md"), "edited tracked file\n");
assertTrue(gitWorktrees.hasUncommitted(wt),
"a tracked modification must also count as dirty");
}
/**
* CB-576 review. {@code hasUncommitted} must tolerate a missing worktree exactly like
* {@code remove}: an already-gone directory holds no work to lose, and throwing here would
* break teardown — SessionManager.release() calls it before stopping the pane, so an
* exception would orphan a live pane and skip the release notification (CB-516).
*/
@Test
void hasUncommittedOnAMissingWorktreeReturnsFalseWithoutThrowing(@TempDir Path tmp) {
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String gone = tmp.resolve("wts").resolve("does-not-exist").toString();
assertFalse(gitWorktrees.hasUncommitted(gone),
"a missing worktree is reported clean, not an error");
}
/** All three protected configs are covered: each one present in a worktree is neutralized and hidden. */
@Test
void allThreeConfigsAreNeutralizedWhenPresent(@TempDir Path tmp) throws Exception {
Path repo = tmp.resolve("repo");
Files.createDirectories(repo);
git(repo, "init", "-q", "-b", "main");
git(repo, "config", "user.email", "test@example.invalid");
git(repo, "config", "user.name", "Test");
Files.writeString(repo.resolve(".mcp.json"), WITH_SERVERS);
Files.writeString(repo.resolve("opencode.json"), OPENCODE_WITH_FILE_REF);
Files.writeString(repo.resolve(".autoenv"), AUTOENV_WITH_DIRECTIVE);
Files.writeString(repo.resolve("README.md"), "seed\n");
git(repo, "add", ".mcp.json", "opencode.json", ".autoenv", "README.md");
git(repo, "commit", "-q", "-m", "seed");
String wt = new GitWorktrees(tmp.resolve("wts").toString())
.add(repo.toString(), "cb-543-c", "HEAD");
assertTrue(Files.readString(Path.of(wt).resolve(".mcp.json"))
.replaceAll("\\s+", "").contains("\"mcpServers\":{}"),
".mcp.json was not neutralized");
assertEquals("{}", Files.readString(Path.of(wt).resolve("opencode.json")).replaceAll("\\s+", ""),
"opencode.json was not neutralized");
assertEquals("", Files.readString(Path.of(wt).resolve(".autoenv")),
".autoenv was not neutralized");
assertEquals("", status(Path.of(wt), ".mcp.json"), ".mcp.json still shows as modified");
assertEquals("", status(Path.of(wt), "opencode.json"), "opencode.json still shows as modified");
assertEquals("", status(Path.of(wt), ".autoenv"), ".autoenv still shows as modified");
}
/**
* CB-578 stage C, acceptance criterion 1. A dirty worktree — a tracked edit plus a brand-new
* untracked file, exactly the shape lost in CB-576 — must land in {@code refs/wip/<branch>}'s
* tree, and that ref must live outside {@code refs/heads} so it never shows up in
* {@code git branch} or gets swept by a branch cleanup.
*/
@Test
void dirtySnapshotCreatesARefWhoseTreeContainsUntrackedAndTrackedChanges(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-578-c-a";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
Files.writeString(Path.of(wt).resolve("untracked.txt"), "draft that was never added\n");
Files.writeString(Path.of(wt).resolve("README.md"), "edited tracked file\n");
Optional<String> ref = gitWorktrees.snapshot(wt, branch, "test snapshot");
assertTrue(ref.isPresent(), "a dirty worktree snapshot returns a commit sha");
String tree = lsTree(repo, "refs/wip/" + branch);
assertTrue(tree.contains("untracked.txt"), "snapshot tree must include the untracked file:\n" + tree);
assertTrue(tree.contains("README.md"), "snapshot tree must include the tracked edit:\n" + tree);
String heads = forEachRef(repo, "refs/heads");
assertFalse(heads.contains("refs/wip/"), "the snapshot ref must not live under refs/heads:\n" + heads);
}
/**
* CB-578 stage C, acceptance criterion 2. Building the commit through a temporary
* {@code GIT_INDEX_FILE} must leave the worker's own index, working tree, and HEAD exactly as
* they were — the worker may still be mid-write, and staging into the real index would corrupt
* that.
*/
@Test
void snapshotDoesNotTouchTheWorkersOwnIndexWorkingTreeOrHead(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-578-c-b";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
Files.writeString(Path.of(wt).resolve("untracked.txt"), "draft that was never added\n");
Files.writeString(Path.of(wt).resolve("README.md"), "edited tracked file\n");
String headBefore = revParse(Path.of(wt), "HEAD");
String statusBefore = fullStatus(Path.of(wt));
gitWorktrees.snapshot(wt, branch, "test snapshot");
assertEquals(headBefore, revParse(Path.of(wt), "HEAD"), "snapshot must not move HEAD");
assertEquals(statusBefore, fullStatus(Path.of(wt)),
"snapshot must not change the worker's own index or working tree status");
}
/**
* CB-578 stage C, acceptance criterion 3. {@code add -A} (never {@code -f}) respects
* {@code .gitignore}, and that is the only thing keeping a gitignored file (secrets, local
* config) out of a snapshot commit — an untracked file that is NOT ignored must still be
* included, so this isn't just "untracked files are dropped".
*/
@Test
void gitignoredFileIsExcludedButOtherUntrackedFilesAreNot(@TempDir Path tmp) throws Exception {
Path repo = tmp.resolve("repo");
Files.createDirectories(repo);
git(repo, "init", "-q", "-b", "main");
git(repo, "config", "user.email", "test@example.invalid");
git(repo, "config", "user.name", "Test");
Files.writeString(repo.resolve(".gitignore"), ".env\n");
Files.writeString(repo.resolve("README.md"), "seed\n");
git(repo, "add", ".gitignore", "README.md");
git(repo, "commit", "-q", "-m", "seed");
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-578-c-c";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
Files.writeString(Path.of(wt).resolve(".env"), "SECRET=shh\n");
Files.writeString(Path.of(wt).resolve("untracked.txt"), "draft that was never added\n");
Optional<String> ref = gitWorktrees.snapshot(wt, branch, "test snapshot");
assertTrue(ref.isPresent());
String tree = lsTree(repo, "refs/wip/" + branch);
assertFalse(tree.contains(".env"), "a gitignored file must never enter the snapshot:\n" + tree);
assertTrue(tree.contains("untracked.txt"),
"a non-ignored untracked file must still be included:\n" + tree);
}
/** CB-578 stage C. A clean worktree still produces a valid, if tree-identical, commit — the caller
* (SessionManager) is the one that decides not to call this on a clean worktree. */
@Test
void snapshotOfAMissingWorktreeReturnsEmptyWithoutThrowing(@TempDir Path tmp) {
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String gone = tmp.resolve("wts").resolve("does-not-exist").toString();
assertTrue(gitWorktrees.snapshot(gone, "some-branch", "msg").isEmpty(),
"a missing worktree has nothing to snapshot, and must not throw");
}
/**
* CB-587, acceptance criteria 1 and 2. A file with a REAL {@code --skip-worktree} bit set in the
* worktree's own index must never enter the snapshot, even though its on-disk content has locally
* diverged from what is committed — that is exactly the divergence {@code --skip-worktree} exists
* to hide from {@code git status}, and a snapshot built from a fresh empty temp index (the bug)
* stages that local content anyway because the fresh index carries none of the real index's flags.
* The snapshot's diff against its parent must list exactly what {@code git status --porcelain}
* reports for the worktree — no more, no less.
*/
@Test
void dirtySnapshotHonoursARealSkipWorktreeBit(@TempDir Path tmp) throws Exception {
Path repo = tmp.resolve("repo");
Files.createDirectories(repo);
git(repo, "init", "-q", "-b", "main");
git(repo, "config", "user.email", "test@example.invalid");
git(repo, "config", "user.name", "Test");
Files.writeString(repo.resolve("protected.cfg"), "committed-value\n");
Files.writeString(repo.resolve("README.md"), "seed\n");
git(repo, "add", "protected.cfg", "README.md");
git(repo, "commit", "-q", "-m", "seed");
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-587-a";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
git(Path.of(wt), "update-index", "--skip-worktree", "protected.cfg");
Files.writeString(Path.of(wt).resolve("protected.cfg"), "locally-diverged-never-commit\n");
Files.writeString(Path.of(wt).resolve("untracked.txt"), "draft that was never added\n");
Files.writeString(Path.of(wt).resolve("README.md"), "edited tracked file\n");
String porcelain = fullStatus(Path.of(wt));
assertFalse(porcelain.contains("protected.cfg"),
"test setup invalid — protected.cfg must not show in git status once skip-worktree is set:\n"
+ porcelain);
Optional<String> ref = gitWorktrees.snapshot(wt, branch, "test snapshot");
assertTrue(ref.isPresent(), "a dirty worktree snapshot returns a commit sha");
Set<String> diffPaths = diffNameOnly(repo, "HEAD", "refs/wip/" + branch);
assertFalse(diffPaths.contains("protected.cfg"),
"a --skip-worktree file's local drift leaked into the snapshot:\n" + diffPaths);
assertEquals(porcelainPaths(porcelain), diffPaths,
"snapshot diff must list exactly what git status --porcelain reports, no more, no less");
}
/**
* CB-587, acceptance criterion 6. If the worktree's real index cannot be resolved/read, snapshot
* must fail loudly (throw) rather than silently falling back to an empty temp index and producing
* a wrong snapshot. SessionManager's caller already catches and WARNs on any exception here — this
* only needs to confirm the failure is not swallowed inside snapshot() itself.
*/
@Test
void snapshotThrowsWhenTheWorktreesRealIndexCannotBeResolved(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-587-b";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
// Break git's ability to resolve the worktree's real index by removing the linked worktree's
// `.git` file (which normally points at the main repo's worktrees/<name>/ directory).
Files.delete(Path.of(wt).resolve(".git"));
assertThrows(WorktreeException.class, () -> gitWorktrees.snapshot(wt, branch, "test snapshot"),
"an unresolvable real index must fail loudly, not silently snapshot from an empty index");
}
/**
* CB-586, criterion 2. A snapshot whose content is NOT reachable from {@code main} is the last
* copy of a worker's work, and must never be deleted automatically — even when it is old and
* even when the caller passes a zero age floor. Uses the real snapshot path on a dirty worktree,
* so the unreachable tree is exactly the shape CB-576/CB-578 stage C exist to protect.
*/
@Test
void anUnreachableSnapshotIsNeverPruned(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String branch = "cb-586-unreachable";
String wt = gitWorktrees.add(repo.toString(), branch, "HEAD");
Files.writeString(Path.of(wt).resolve("worker-draft.txt"), "work that exists nowhere else\n");
Optional<String> ref = gitWorktrees.snapshot(wt, branch, "snapshot with unreachable content");
assertTrue(ref.isPresent());
// Age floor 0 makes age a non-issue: only reachability can save it — and it must.
assertEquals(0, gitWorktrees.pruneWipRefs(repo.toString(), 0),
"the unreachable snapshot is the last copy and must not be pruned");
assertTrue(refExists(repo, "refs/wip/" + branch),
"an unreachable snapshot must survive the sweep");
}
/**
* CB-586, criterion 1 (the reachable half). A snapshot whose tree content IS already reachable
* from {@code main} and which is older than the age floor is pure duplication — the work is
* recovered — so it must be pruned.
*/
@Test
void aReachableSnapshotOlderThanTheFloorIsPruned(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
// A snapshot whose tree is exactly main's current tree: fully reachable from main.
String mainTree = revParse(repo, "main^{tree}");
String old = commitTree(repo, mainTree, revParse(repo, "HEAD"), "2020-01-01T00:00:00", "snapshot");
updateRef(repo, "refs/wip/recovered", old);
assertEquals(1, gitWorktrees.pruneWipRefs(repo.toString(), TimeUnit.HOURS.toMillis(24)),
"an old, main-reachable snapshot must be pruned");
assertFalse(refExists(repo, "refs/wip/recovered"),
"the reachable snapshot's ref must be gone after the sweep");
}
/**
* CB-586, the age floor. A snapshot whose content IS reachable from {@code main} but which is
* younger than the age floor must not be swept — a lead may still be looking at it.
*/
@Test
void aReachableButRecentSnapshotIsNotPruned(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
// Reachable from main, but committed "now" — a fresh snapshot. The 24h floor must protect it.
String mainTree = revParse(repo, "main^{tree}");
String fresh = commitTree(repo, mainTree, revParse(repo, "HEAD"),
"2038-01-01T00:00:00", "snapshot just taken");
updateRef(repo, "refs/wip/fresh", fresh);
assertEquals(0, gitWorktrees.pruneWipRefs(repo.toString(), TimeUnit.HOURS.toMillis(24)),
"a recent snapshot must be kept even when reachable");
assertTrue(refExists(repo, "refs/wip/fresh"),
"the recent reachable snapshot must survive the sweep");
}
/**
* CB-586, criterion 5. A fleet that has never snapshotted anything has no {@code refs/wip/*},
* so a sweep is a no-op and the census reports none — identical to before CB-586 existed.
*/
@Test
void aFleetWithNoSnapshotsPrunesNothingAndReportsNothing(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
assertEquals(0, gitWorktrees.pruneWipRefs(repo.toString(), 0),
"no snapshot refs means nothing to prune");
Worktrees.WipRefStats stats = gitWorktrees.wipRefs(repo.toString());
assertEquals(0, stats.count(), "a never-snapshotted fleet has zero refs/wip refs");
assertEquals(0L, stats.costBytes(), "a never-snapshotted fleet costs zero bytes");
}
/** CB-586, criterion 4: the census reports how many refs exist and roughly what they cost. */
@Test
void wipRefsReportsCountAndCost(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
GitWorktrees gitWorktrees = new GitWorktrees(tmp.resolve("wts").toString());
String blob = blobOf(repo, "a recoverable snapshot's worth of content");
String tree = treeOf(repo, "snapshot.txt", blob);
updateRef(repo, "refs/wip/one", commitTree(repo, tree, revParse(repo, "HEAD"),
"2020-01-01T00:00:00", "snapshot"));
updateRef(repo, "refs/wip/two", commitTree(repo, tree, revParse(repo, "HEAD"),
"2020-01-02T00:00:00", "snapshot"));
Worktrees.WipRefStats stats = gitWorktrees.wipRefs(repo.toString());
assertEquals(2, stats.count(), "two snapshot refs are reported");
assertTrue(stats.costBytes() > 0, "the cost of the snapshots is a positive byte count");
}
}
-61
View File
@@ -1,61 +0,0 @@
# CB-504 — systemd unit for bridged (Linux).
#
# The macOS launchd agent (deploy/dev.ltms.bridged.plist) is the supervision target for the
# current single-host deployment. This unit exists for the per-host gateways CB-308 introduces,
# which will run on Linux.
#
# Install (user service — bridged drives the user's herdr, not a system daemon):
# mkdir -p ~/.config/systemd/user
# cp deploy/bridged.service ~/.config/systemd/user/
# # edit ExecStart / WorkingDirectory / Environment below, then:
# systemctl --user daemon-reload
# systemctl --user enable --now bridged
# journalctl --user -u bridged -f
[Unit]
Description=bridged — claude-bridge message server
Documentation=https://git.ltms.dev/fleet/fleetd/wiki
# Ordering only: herdr is a user process and its socket may appear after us. This is advisory —
# bridged retries the herdr socket rather than exiting, which is what actually makes a late
# socket survivable. Do NOT add Requires=: a herdr restart must not take bridged down with it.
After=herdr.service
Wants=herdr.service
[Service]
Type=simple
WorkingDirectory=%h/src/claude-bridge/bridged
ExecStart=/usr/lib/jvm/temurin-25-jdk/bin/java -jar target/bridged.jar bridged.yaml
Environment=HERDR_SOCKET_PATH=%h/.config/herdr/herdr.sock
# PATH matters more than it looks (CB-511): bridged propagates its own PATH to every worker it
# spawns, so this line decides whether the fleet can run a build at all. systemd does not source a
# login shell, so without it the daemon — and every worker — gets a bare default with no JDK/Maven.
Environment=PATH=/usr/lib/jvm/temurin-25-jdk/bin:/usr/share/maven/bin:/usr/local/bin:/usr/bin:/bin
# Secrets are NOT set here — this file is committed. Put the API/worker tokens in a private
# drop-in that systemd reads with restrictive permissions:
# systemctl --user edit bridged → [Service] / Environment=BRIDGED_API_TOKEN=...
# or point EnvironmentFile at a 0600 file:
# EnvironmentFile=%h/.config/bridged/env
Restart=on-failure
RestartSec=10s
# A bad config (e.g. a non-loopback bind without token auth) makes bridged fail fast by design.
# Give up rather than restart-loop on a permanent error.
StartLimitBurst=5
StartLimitIntervalSec=120
# The daemon reads the repo, writes worktrees, and talks to a Unix socket — it needs no more.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-write
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
StandardOutput=journal
StandardError=journal
SyslogIdentifier=bridged
[Install]
WantedBy=default.target
@@ -1,36 +1,36 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<!--
CB-504 / CB-594 — launchd agent for bridged (macOS).
CB-504 / CB-594 — launchd agent for fleetd (macOS).
This is the real supervision target today: the dogfooded daemon runs on macOS, where there is
no systemd. A systemd unit ships alongside (deploy/bridged.service) for the Linux gateways
no systemd. A systemd unit ships alongside (deploy/fleetd.service) for the Linux gateways
CB-308 introduces.
Install:
cp deploy/dev.ltms.bridged.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/dev.ltms.bridged.plist
launchctl list | grep bridged
cp deploy/dev.ltms.fleetd.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/dev.ltms.fleetd.plist
launchctl list | grep fleetd
The paths below are already filled in for this host (resolved 2026-08-16 from
`/usr/libexec/java_home`... except that reported the system Applet-plugin JVM, not the jenv-
managed JDK 25 actually used to build/run bridged, so JAVA_HOME here is the real one:
managed JDK 25 actually used to build/run fleetd, so JAVA_HOME here is the real one:
`JENV_VERSION=25.0.3 java -XshowSettings:properties -version 2>&1 | grep java.home`; `which mvn`;
`echo $HOME`). If this file is copied to a different host, re-resolve all three paths and check
no placeholder path is left behind; scripts/redeploy-bridged.sh's check mode does not (and
no placeholder path is left behind; scripts/redeploy-fleetd.sh's check mode does not (and
cannot) check this file for you.
CB-594 — launchd cannot run a login shell (see the PATH comment on EnvironmentVariables below,
and scripts/bridged-launchd-wrapper.sh for the fix): ProgramArguments below execs THAT wrapper,
and scripts/fleetd-launchd-wrapper.sh for the fix): ProgramArguments below execs THAT wrapper,
not java directly, so WORKER_GITEA_TOKEN and AI_GATEWAY_TOKEN still get sourced from
${SHARED_ENV}/tools/secrets.sh even though launchd itself never sources anything.
Note on ordering: launchd has no "start after herdr" primitive for user agents, and neither
does systemd in a way that survives a socket appearing late. bridged retries the herdr socket
does systemd in a way that survives a socket appearing late. fleetd retries the herdr socket
on startup instead, so an agent that comes up before herdr converges rather than dying — that
retry is the actual fix; KeepAlive below is the backstop.
CB-594 — KeepAlive vs. scripts/redeploy-bridged.sh: a bare SIGTERM makes this JVM exit 143 even
CB-594 — KeepAlive vs. scripts/redeploy-fleetd.sh: a bare SIGTERM makes this JVM exit 143 even
with its shutdown hook running to completion (measured, see the CB-594 report), which
SuccessfulExit:false below reads as a crash and races to restart the OLD jar. The redeploy
script now detects a loaded agent and uses `launchctl unload`/`load` instead of a raw kill, so
@@ -40,20 +40,20 @@
<plist version="1.0">
<dict>
<key>Label</key>
<string>dev.ltms.bridged</string>
<string>dev.ltms.fleetd</string>
<key>ProgramArguments</key>
<array>
<string>/Users/dai.ha/LTMS/claude-bridge/scripts/bridged-launchd-wrapper.sh</string>
<string>/Users/dai.ha/LTMS/claude-bridge/scripts/fleetd-launchd-wrapper.sh</string>
<string>/Users/dai.ha/Softwares/jdks/jdk-25.0.3.jdk/Contents/Home/bin/java</string>
<string>-jar</string>
<string>/Users/dai.ha/LTMS/claude-bridge/bridged/target/bridged.jar</string>
<string>bridged.yaml</string>
<string>/Users/dai.ha/LTMS/claude-bridge/fleetd/target/fleetd.jar</string>
<string>fleetd.yaml</string>
</array>
<!-- Config path in ProgramArguments is relative, so the working directory must be the module. -->
<key>WorkingDirectory</key>
<string>/Users/dai.ha/LTMS/claude-bridge/bridged</string>
<string>/Users/dai.ha/LTMS/claude-bridge/fleetd</string>
<key>EnvironmentVariables</key>
<dict>
@@ -62,7 +62,7 @@
<key>HERDR_SOCKET_PATH</key>
<string>/Users/dai.ha/.config/herdr/herdr.sock</string>
<!--
PATH matters more than it looks (CB-511): bridged propagates its own PATH to every worker
PATH matters more than it looks (CB-511): fleetd propagates its own PATH to every worker
it spawns, so this line decides whether the fleet can run a build at all. launchd does NOT
source .zprofile/.zshrc, so without this the daemon (and therefore every worker) gets a
bare /usr/bin:/bin and no JDK or Maven. Keep the toolchain entries first.
@@ -71,10 +71,10 @@
<string>/Users/dai.ha/Softwares/jdks/jdk-25.0.3.jdk/Contents/Home/bin:/Users/dai.ha/Softwares/apache-maven/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
<!--
Worker/API tokens are NOT set here: this file is committed. CB-594 —
scripts/bridged-launchd-wrapper.sh (named in ProgramArguments above) is what supplies
scripts/fleetd-launchd-wrapper.sh (named in ProgramArguments above) is what supplies
them, by execing a login shell that sources ${SHARED_ENV}/tools/secrets.sh before the
daemon itself starts. bridged also reads the API token from the env var named by
auth.tokenEnv (default BRIDGED_API_TOKEN) and only in auth.mode: token — the wrapper
daemon itself starts. fleetd also reads the API token from the env var named by
auth.tokenEnv (default FLEETD_API_TOKEN) and only in auth.mode: token — the wrapper
covers that one too, since it is the same login shell.
-->
</dict>
@@ -84,21 +84,21 @@
<!--
CB-600 — read this before assuming ThrottleInterval bounds anything. It paces restarts to at
most one per 10s; it does NOT cap how many times launchd retries. If bridged fails fast on
every start — a bad bridged.yaml, for example auth.mode: token with the token env var unset,
most one per 10s; it does NOT cap how many times launchd retries. If fleetd fails fast on
every start — a bad fleetd.yaml, for example auth.mode: token with the token env var unset,
which throws in main() before the daemon ever binds a port — launchd restarts it forever,
once every 10s, until a human intervenes. LaunchAgents have no "give up after N attempts"
primitive, so this is not something a config change here can fix.
That loop stops only two ways: (1) `launchctl unload -w ~/Library/LaunchAgents/dev.ltms.bridged.plist`,
That loop stops only two ways: (1) `launchctl unload -w ~/Library/LaunchAgents/dev.ltms.fleetd.plist`,
or (2) the underlying cause gets fixed, so the process starts successfully and stays up (no
more exits to restart). scripts/redeploy-bridged.sh does not add a third way — it does not
make bridged self-disable on a config error, on purpose: a fail-fast exit path that
more exits to restart). scripts/redeploy-fleetd.sh does not add a third way — it does not
make fleetd self-disable on a config error, on purpose: a fail-fast exit path that
sometimes decides "this is unrecoverable, stop trying" is one more thing that can misfire,
and a wrongly self-disabled daemon needs the exact same manual `launchctl load -w` recovery
this comment already names — so it buys nothing an operator watching for the crash loop
doesn't already have, at the cost of a new way to be silently down. Watch for it with
`launchctl list dev.ltms.bridged` (a high restart count) or by tailing bridged.out for the
`launchctl list dev.ltms.fleetd` (a high restart count) or by tailing fleetd.out for the
same startup error repeating every ~10s.
-->
<key>KeepAlive</key>
@@ -110,16 +110,16 @@
<integer>10</integer>
<!--
CB-594 — same file scripts/redeploy-bridged.sh already tails ($BRIDGED/bridged.out), and both
CB-594 — same file scripts/redeploy-fleetd.sh already tails ($BRIDGED/fleetd.out), and both
streams point at it, not two separate log files: the script's fresh-line / ERROR-count checks
after a restart read this one path regardless of whether launchd or the script started the
process, and a stdout/stderr split would make half of what happens during a launchd-driven
restart invisible to it.
-->
<key>StandardOutPath</key>
<string>/Users/dai.ha/LTMS/claude-bridge/bridged/bridged.out</string>
<string>/Users/dai.ha/LTMS/claude-bridge/fleetd/fleetd.out</string>
<key>StandardErrorPath</key>
<string>/Users/dai.ha/LTMS/claude-bridge/bridged/bridged.out</string>
<string>/Users/dai.ha/LTMS/claude-bridge/fleetd/fleetd.out</string>
<key>ProcessType</key>
<string>Background</string>
+77
View File
@@ -0,0 +1,77 @@
# CB-504 — systemd unit for fleetd (Linux).
#
# The macOS launchd agent (deploy/dev.ltms.fleetd.plist) is the supervision target for the
# current single-host deployment. This unit exists for the per-host gateways CB-308 introduces,
# which will run on Linux.
#
# Install (user service — fleetd drives the user's herdr, not a system daemon):
# mkdir -p ~/.config/systemd/user
# cp deploy/fleetd.service ~/.config/systemd/user/
# # edit ExecStart / WorkingDirectory / Environment below, then:
# systemctl --user daemon-reload
# systemctl --user enable --now fleetd
# journalctl --user -u fleetd -f
[Unit]
Description=fleetd — claude-bridge message server
Documentation=https://git.ltms.dev/fleet/fleetd/wiki
# Ordering only: herdr is a user process and its socket may appear after us. This is advisory —
# fleetd retries the herdr socket rather than exiting, which is what actually makes a late
# socket survivable. Do NOT add Requires=: a herdr restart must not take fleetd down with it.
After=herdr.service
Wants=herdr.service
[Service]
Type=simple
WorkingDirectory=%h/src/claude-bridge/fleetd
ExecStart=/usr/lib/jvm/temurin-25-jdk/bin/java -jar target/fleetd.jar fleetd.yaml
Environment=HERDR_SOCKET_PATH=%h/.config/herdr/herdr.sock
# PATH matters more than it looks (CB-511): fleetd propagates its own PATH to every worker it
# spawns, so this line decides whether the fleet can run a build at all. systemd does not source a
# login shell, so without it the daemon — and every worker — gets a bare default with no JDK/Maven.
Environment=PATH=/usr/lib/jvm/temurin-25-jdk/bin:/usr/share/maven/bin:/usr/local/bin:/usr/bin:/bin
# Secrets are NOT set here — this file is committed. Put ALL three tokens in a private drop-in
# that systemd reads with restrictive permissions. In `systemctl --user edit fleetd`, add:
# [Service]
# Environment=FLEETD_API_TOKEN=...
# Environment=WORKER_GITEA_TOKEN=...
# Environment=AI_GATEWAY_TOKEN=...
# FLEETD_API_TOKEN protects fleetd's API. WORKER_GITEA_TOKEN lets members open pull requests; if
# it is missing, fleetd still starts, but a member fails when it later tries to open a pull request.
# AI_GATEWAY_TOKEN authenticates gateway profiles; if it is missing, fleetd still starts, but a
# gateway profile later returns HTTP 401. Or, put the same three variables in a 0600 file and add:
# EnvironmentFile=%h/.config/fleetd/env
# After starting, check which of them actually resolved. The daemon reports every secret a
# configured profile references, by name, never by value:
# journalctl --user -u fleetd | grep 'startup secret'
# A resolved one logs "startup secret NAME: set (profile 'x' tokenEnv)". A missing one logs
# "startup secret NAME: MISSING" at WARN — and the daemon starts anyway, which is the whole
# problem: without this grep the first sign is a member that cannot open a pull request, hours
# later and in a different component.
# Note what the report can and cannot tell you. It lists only names some profile actually
# references (tokenEnv, gitTokenEnv, and the broker uriEnv). A secret nothing references is never
# reported, because nothing needs it.
Restart=on-failure
RestartSec=10s
# A bad config (e.g. a non-loopback bind without token auth) makes fleetd fail fast by design.
# Give up rather than restart-loop on a permanent error.
StartLimitBurst=5
StartLimitIntervalSec=120
# The daemon reads the repo, writes worktrees, and talks to a Unix socket — it needs no more.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-write
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
StandardOutput=journal
StandardError=journal
SyslogIdentifier=fleetd
[Install]
WantedBy=default.target
+9 -9
View File
@@ -1,8 +1,8 @@
# LavinMQ — the AMQP broker behind bridged's durable ReplyInbox (CB-307 Stage 2).
# LavinMQ — the AMQP broker behind fleetd's durable ReplyInbox (CB-307 Stage 2).
#
# Why this file exists: the broker was previously run ad hoc and simply vanished from the host,
# which takes bridged down with it — AmqpReplyInbox.open throws on an unreachable broker and
# Bridged.java:187 does not guard it, so a missing broker is a hard startup failure, not a
# which takes fleetd down with it — AmqpReplyInbox.open throws on an unreachable broker and
# Fleetd.java:187 does not guard it, so a missing broker is a hard startup failure, not a
# degraded mode. This pins the version, keeps the data, and brings itself back after a reboot.
#
# Usage:
@@ -14,27 +14,27 @@
#
# Management UI: http://127.0.0.1:15672 (guest / guest)
#
# This is bridged's OWN broker. Do not point bridged at any other AMQP server on this host —
# This is fleetd's OWN broker. Do not point fleetd at any other AMQP server on this host —
# notably not the `local-rabbitmq` container, which belongs to a different project and would end
# up carrying this project's queues.
name: bridged-broker
name: fleetd-broker
services:
lavinmq:
# Pinned deliberately: :latest silently moves the broker under a running daemon.
image: cloudamqp/lavinmq:2.9.1
container_name: bridged-lavinmq
container_name: fleetd-lavinmq
# The failure this deployment exists to prevent — survive reboots and Docker restarts, but
# stay down if it was stopped on purpose.
restart: unless-stopped
# Loopback-bound on purpose. LavinMQ ships a default guest/guest account, which is only
# acceptable because nothing off-host can reach it. bridged connects over 127.0.0.1, and
# acceptable because nothing off-host can reach it. fleetd connects over 127.0.0.1, and
# binding 0.0.0.0 here would expose a broker with default credentials to the network.
ports:
- "127.0.0.1:5672:5672" # AMQP — bridged.yaml broker.uri points here
- "127.0.0.1:5672:5672" # AMQP — fleetd.yaml broker.uri points here
- "127.0.0.1:15672:15672" # HTTP management API + UI
# The whole point of Stage 2. Held-but-unacked replies live here; without a named volume a
@@ -57,4 +57,4 @@ services:
volumes:
lavinmq-data:
name: bridged-lavinmq-data
name: fleetd-lavinmq-data
+531
View File
@@ -0,0 +1,531 @@
# CB-201 and CB-227 refinement
Date: 2026-09-03
## Decision
#201 and #227 are one delivery program, but they are not one implementation unit.
#201 has a real seam: `CompletionResolver` can publish a typed backend-error event only after its
waiter resolution wins. #227 can consume that event without knowing any pane text. The classifier
must land before the final #227 wiring. However, the policy engine, roster state, and lead nudge can
be built in parallel with the classifier.
I propose five units. Units 1 to 4 own separate files and can run in parallel. Unit 5 owns all
composition files and lands after them. It also depends on the #234 defect 2 fix named in the task.
```mermaid
flowchart LR
U1["Unit 1: typed backend-error classification"] --> U5["Unit 5: wire policy, spawn gate, and fleet views"]
U2["Unit 2: credential outage policy"] --> U5
U3["Unit 3: lead outage nudge"] --> U5
U4["Unit 4: durable member outcome"] --> U5
D234["#234 defect 2: fail-loud target resolution"] --> U5
```
*Figure 1. Four file-disjoint foundations feed one composition unit.*
This split keeps `Fleetd.java` under one owner. It also keeps every other production file under one
unit in this plan.
## Evidence checked in the current branch
I read both issue pages in full. Each page reports zero comments.
| Evidence | What the code says now |
|---|---|
| `inject/CompletionResolver.java:229-237` | A turn below two seconds fails before normal scrape classification. A matching fast backend error is therefore only a generic failure today. |
| `inject/CompletionResolver.java:260-276` and `:332-367` | #211 already added raw-screen classification when `lastAssistantBlock` is empty. The “dead-code question” in #201 is stale on this branch. |
| `inject/CompletionResolver.java:288-317` | Exhaustion wins before the hard-coded `API Error:` match. A backend error then goes through generic `fail(...)`. |
| `inject/CompletionResolver.java:311-316` | The code admits that the pattern is a heuristic. A member report which quotes an API error may match it. |
| `inject/CompletionResolver.java:449-467` | Startup coverage exists only for `exhaustedPattern`. |
| `inject/ExhaustedPatternLookup.java:13-25` | The current lookup and explicit `none()` value are a good shape for the new classifier seam. |
| `Fleetd.java:196-207` | One `BackendQuarantine` is shared by placement and the exhaustion sink. Its cooldown comes from `quarantineCooldownSeconds`. |
| `Fleetd.java:322-363` | Pattern compilation, target-to-profile lookup, and the live `ExhaustionSink` are composed in `Fleetd.main`. The sink on this branch still ends in `.ifPresent(...)`. This plan assumes #234 replaces that silent path. |
| `placement/BackendQuarantine.java:60-87` | A repeated exhaustion restarts one long quarantine. The store is credential-keyed and uses an injected monotonic clock. |
| `member/CompositePeerLauncher.java:260-317` | Explicit and policy-selected spawns have separate gates. Both paths must learn about outage cool-off. |
| `member/CompositePeerLauncher.java:347-379` | Exhaustion refusal already checks a credential for explicit spawns and filters policy candidates. Its error text says “exhausted”. |
| `placement/PlacementContext.java:10-22` and `PlacementPolicyUtil.java:14-83` | Automatic placement has only one transient exclusion set named `quarantined`. Reusing it would make outage errors say “backend exhausted”. |
| `mcp/FleetMcp.java:913-1025` | `fleet_list` sets `free: 0` and adds `credentialId` plus `quarantinedForSeconds` when quarantine is active. |
| `session/MemberSession.java:51-59` | The roster has `DONE` and generic `FAILED`, but no backend-error state or stored reason. |
| `session/SessionManager.java:695-773` | A normal boundary moves `BUSY` to `DONE`. A failure moves any non-released session to `FAILED`. The async completion resolver can race the `DONE` update. |
| `session/SessionManager.java:648-687` | `rosterView` reports the session state, but it reports no terminal reason. |
| `msg/MessageService.java:922-940` | CB-588 already nudges for every terminal async ticket, including failures. Current code would report failed tickets, but it would not report one correlated outage. |
| `msg/ReplyPushLoop.java:20-48` | Replies, terminal tickets, and questions share one per-lead schedule. This prevents two push sources from injecting competing turns. |
| `msg/ReplyPushLoop.java:305-395` | Each push entry point resolves the owning lead through `PrimaryRegistry`. Missing ownership is logged and the durable or pending item remains the backstop. |
| `msg/ReplyPushLoop.java:496-547` | One tick builds one combined nudge. Pending items have separate reminder counts. |
| `health/FleetHealthMonitor.java:91-143` | Health is a slow periodic observer of members and message-layer facts. It does not receive completion classifications. |
| `health/FleetHealthMonitor.java:206-208` | `healthCoverage` means health enabled plus webhook configured. It does not describe lead-pane alerts. |
| `Fleetd.java:465-486` | Health stays `detection-only` without the webhook notification setting. |
I also read the related unit tests for `CompletionResolver`, `ReplyPushLoop`, `BackendQuarantine`,
`CompositePeerLauncher`, `PlacementPolicyUtil`, `SessionManager`, `MessageService`, and `FleetMcp`.
I did not inspect the in-progress #234 branch. I only used the two measured facts in the task. No
peer architect was named, so I did not exchange a design with one.
## Required behaviour
The policy should use these first values:
- Threshold: **2** classified backend errors.
- Window: **60 seconds**, measured from the first error to the second.
- Cool-off: **60 seconds**, starting when the threshold is reached.
- Correlation key: `credentialId`, never profile name and never error text.
- Incident rule: one active incident per credential. Errors during its cool-off do not extend it and
do not create more lead notices.
- Rearm rule: after cool-off ends, two fresh errors are needed for another incident.
Two errors are the smallest threshold which protects the honest one-turn failure. A 60-second window
fits the measured two-member outage. A 60-second cool-off blocks immediate repeat spawns without
turning a short backend fault into the default 1,800-second exhaustion quarantine.
A single classified error still fails its send and marks its member `backend_error`. It does not
cool a credential and does not send an outage notice. This is what “a single error changes nothing”
must mean at the credential level. It cannot mean that the failed member still looks successful.
```mermaid
sequenceDiagram
participant R1 as Resolver for member A
participant R2 as Resolver for member B
participant P as Outage policy
participant S as Spawn gate
participant N as Lead push loop
participant L as Lead pane
R1->>P: backend error for credential C
Note over P: Count 1, no cool-off
R2->>P: backend error for credential C within 60s
P->>P: Start one 60s incident
P->>S: Credential C is cooling off
P->>N: Queue one incident notice
N->>L: Inject when lead is idle, blocked, or done
L->>S: Request another spawn on credential C
S-->>L: Refuse and report remaining cool-off
```
*Figure 2. The second independent classification creates the fleet-level event.*
Against the 2026-09-01 case, the second failed member would start cool-off. `fleet_list` would show
zero free capacity and both members as `backend_error`. The push loop would inject one outage notice
even if the lead had not polled either ticket yet. The design reports the outage. It does not recover
uncommitted work from the members.
## Unit 1 — Typed backend-error classification
### Scope
Replace the direct hard-coded check inside `CompletionResolver` with a lookup and a sink. Keep the
public send result as a failed send. The typed internal event is the seam #227 consumes.
The lookup returns the pattern for a target. The sink receives the target, matched line, and full
failure reason. It fires only after `Rendezvous.resolveFailure(...)` wins for that exact captured
waiter. This copies the race rule already used by `ExhaustionSink`.
The classifier must run in all three current paths:
1. a normal non-empty assistant block;
2. the #211 raw scrape fallback;
3. a turn inside `MIN_TURN_NANOS`, before it becomes a generic too-fast failure.
In every path, the order stays: stale-baseline guard, exhaustion, backend error, then generic
failure or completion. A fast turn still fails when no configured pattern matches.
Keep `(?i)\bAPI Error\s*:` as a compatibility pattern for profiles without `errorPattern` until the
operator config is updated. Do not call this full coverage. Startup reporting in Unit 5 must name
profiles using this weaker legacy default.
### Files owned
- Add `fleetd/src/main/java/dev/ltms/fleet/inject/BackendErrorPatternLookup.java`.
- Add `fleetd/src/main/java/dev/ltms/fleet/inject/BackendErrorSink.java`.
- Change `fleetd/src/main/java/dev/ltms/fleet/inject/CompletionResolver.java`.
- Change `fleetd/src/test/java/dev/ltms/fleet/inject/CompletionResolverTest.java`.
No other unit may edit these files.
### Acceptance criteria
1. A target-specific error pattern matches a normal assistant block and resolves the send as failed.
2. The same match calls `BackendErrorSink` exactly once after the waiter resolution wins.
3. A late classification which loses to `fleet_reply` does not call the sink.
4. An exhausted line that also matches the generic error pattern stays `BACKEND_EXHAUSTED`. It calls
only `ExhaustionSink`.
5. A raw pane with leading Terminal User Interface (TUI) chrome and no assistant marker still uses
the #211 fallback and calls the backend-error sink.
6. A matching error inside the two-second floor is typed and sent to the sink. A non-matching fast
turn stays a generic failure.
7. An unchanged delivery baseline which contains old backend-error text is suppressed. It never
increments outage evidence.
8. A non-match keeps the existing completion result and text.
9. Constructors used by current callers keep compiling. They use the legacy default lookup and an
explicit inert sink until Unit 5 supplies the production objects.
10. Unit tests pass. The developer runs the focused test first, then `mvn clean install` from
`fleetd/`.
### Dependencies
None. Unit 1 can run with Units 2, 3, and 4.
Unit 5 depends on its new lookup, sink, and constructor.
### What to report back
- The exact classifier order in all three paths.
- The focused test command and result.
- The test which proves a losing waiter race does not publish an event.
- The test which proves a fast matching failure is typed.
- The final `mvn clean install` result.
- Any constructor kept only for transition and where Unit 5 replaces it.
## Unit 2 — Credential outage policy
### Scope
Build a small credential-keyed state machine. It accepts already-classified backend-error events.
It does not read pane text, profiles, sessions, or lead state.
Use an injected monotonic clock. A call records `credentialId`, target, and reason. It returns a new
incident only on the threshold crossing. The incident contains a stable event id, credential id,
the distinct affected targets, evidence count, window, and remaining cool-off.
This class owns both correlation and short cool-off. Keeping them together makes threshold crossing
and the cool-off deadline one atomic state change.
### Files owned
- Add `fleetd/src/main/java/dev/ltms/fleet/placement/BackendOutagePolicy.java`.
- Add `fleetd/src/test/java/dev/ltms/fleet/placement/BackendOutagePolicyTest.java`.
No other unit may edit these files.
### Acceptance criteria
1. One error creates no incident and no cool-off.
2. Two errors for one credential within 60 seconds create exactly one incident and a 60-second
cool-off.
3. Two errors more than 60 seconds apart do not create an incident.
4. The exact 60-second boundary has a pinned result. Use inclusive `<= 60s` so scheduler delay does
not discard evidence at the boundary.
5. Different credentials never share evidence.
6. Different profiles which supply the same credential id do share evidence. The policy itself only
sees the credential id.
7. More errors during active cool-off do not extend its deadline and do not return another incident.
8. After expiry, old evidence is cleared. Two fresh errors are needed to create the next incident.
9. Remaining seconds round up, matching `BackendQuarantine` reporting.
10. Concurrent second and third errors cannot return two incidents.
11. The focused tests and `mvn clean install` pass.
### Dependencies
None. Unit 2 can run with Units 1, 3, and 4.
Unit 5 depends on the policy API.
### What to report back
- The state transition table and locking method.
- The exact threshold, window, cool-off, and boundary rule.
- The test which proves one incident under concurrent calls.
- The focused test and `mvn clean install` results.
## Unit 3 — Lead outage nudge
### Scope
Add backend incidents as a fourth pending source in `ReplyPushLoop`. Do not create another scheduler
or call `AgentControl.send` from `Fleetd`. The existing combined per-lead schedule is the control
which prevents competing injected turns.
The entry point takes an incident id, affected worker targets, credential id, affected profile
names, and remaining cool-off. It resolves distinct owning leads through `PrimaryRegistry`.
Each `(incidentId, lead)` item is one-shot. It waits while the lead is not injectable. After one
successful `agents.send`, remove it. A send exception keeps it pending for a bounded retry. It never
uses the repeated reminder behaviour of an uncollected ticket.
Also add a fail-loud entry point for a classified target that Unit 5 cannot map to a credential. It
uses `PrimaryRegistry.nudgeTargetFor(target)` and says that correlation could not run. If no lead is
known, log at `WARN`, not `DEBUG`.
### Files owned
- Change `fleetd/src/main/java/dev/ltms/fleet/msg/ReplyPushLoop.java`.
- Change `fleetd/src/test/java/dev/ltms/fleet/msg/ReplyPushLoopTest.java`.
No other unit may edit these files.
### Acceptance criteria
1. One incident affecting two workers owned by one lead causes one successful pane injection.
2. Two affected workers owned by two leads cause one successful injection per affected lead. This
is one notice per event per lead, not one notice per member.
3. Repeating the same incident id is idempotent.
4. A busy or unknown lead is not injected. The item stays pending until the lead becomes injectable
or its attempt cap is reached.
5. After one successful injection, later ticks do not mention that incident again.
6. A failed `agents.send` is retried within the existing bound. A successful retry still gives only
one successful send.
7. A pending ticket and an outage incident for one lead appear in one combined nudge, not two
competing turns.
8. The text names the credential, profiles, affected workers, and remaining cool-off. It tells the
lead to run `fleet_list`.
9. An unmapped target produces a direct warning notice when a lead is known. If no lead is known,
the code logs a `WARN` naming the target and reason.
10. `stop()` clears incident state as it clears other push state.
11. Existing reply, ticket, and question tests stay green. The focused tests and
`mvn clean install` pass.
### Dependencies
None. The API uses plain values, not the Unit 2 incident class. This lets Unit 3 run in parallel.
Unit 5 adapts the Unit 2 incident into this entry point.
### What to report back
- The exact one-shot and retry rules.
- The test showing one combined nudge with a failed ticket.
- The test showing one successful send for two affected workers.
- The focused test and `mvn clean install` results.
## Unit 4 — Durable member backend-error outcome
### Scope
Make a classified backend failure remain visible after its ticket is collected or expires.
Add `BACKEND_ERROR` to `MemberSession.State`. Add a nullable failure detail to `MemberSession` and
render it as `failureReason` in `SessionManager.rosterView`. Add
`SessionManager.onBackendError(target, reason)`.
The transition must handle both completion orderings:
- `BUSY -> BACKEND_ERROR` when classification wins before the normal completion state update;
- `DONE -> BACKEND_ERROR` when the async resolver runs after `SessionManager.onTurnComplete`.
It must use a compare-and-set retry or another atomic update. `RELEASED` must never return to the
roster. A backend-error member is terminal and cannot accept another delivery.
### Files owned
- Change `fleetd/src/main/java/dev/ltms/fleet/session/MemberSession.java`.
- Change `fleetd/src/main/java/dev/ltms/fleet/session/SessionManager.java`.
- Change `fleetd/src/test/java/dev/ltms/fleet/session/SessionManagerTest.java`.
No other unit may edit these files.
### Acceptance criteria
1. `onBackendError` moves a `BUSY` member to `BACKEND_ERROR` and stores the reason.
2. It also moves `DONE` to `BACKEND_ERROR`, covering the resolver race.
3. A later normal `onTurnComplete` cannot change `BACKEND_ERROR` back to `DONE`.
4. A released or unknown member is not recreated. The unknown case logs at `WARN` and returns an
explicit false result to its caller.
5. `onDelivered` refuses a `BACKEND_ERROR` member, just as it refuses generic `FAILED`.
6. `rosterView` reports `state: backend_error` and `failureReason` after the send ticket is gone.
7. Ordinary members do not gain a blank or invented `failureReason` field.
8. Existing constructors keep source compatibility for tests and adapters.
9. The focused tests and `mvn clean install` pass.
### Dependencies
None. Unit 4 can run with Units 1, 2, and 3.
Unit 5 calls the new session method from the production sink.
### What to report back
- The two race orderings and the tests for both.
- The exact roster JSON shape.
- The unknown-target result and log level.
- The focused test and `mvn clean install` results.
## Unit 5 — Production wiring, spawn gate, and fleet views
### Scope
Compose Units 1 to 4 in production. This is the only unit which edits `Fleetd.java`.
Add per-profile `errorPattern` config beside `exhaustedPattern`. Compile both once at startup. A
configured pattern wins over the legacy default. Report configured profiles and legacy-default
profiles separately at startup. A bad regex must stop startup with the profile and key in the
message.
Wire one production `BackendErrorSink` with this order:
1. mark the member `backend_error` with its reason;
2. resolve the profile and its current `effectiveCredentialId()` through the fail-loud #234 seam;
3. record the error in `BackendOutagePolicy`;
4. on a new incident, submit one event to `ReplyPushLoop`.
If target metadata cannot be resolved, do not end in `Optional.ifPresent`. Log an error and call the
Unit 3 unmapped-target notice. The failed send still reaches its ticket through CB-588.
Teach both spawn paths about a separate cool-off source. Exhaustion quarantine has priority when
both states are active. Automatic placement needs a distinct `coolingOff` set so its refusal does
not say “exhausted”.
Extend the MCP (Model Context Protocol) views:
- A cooling profile has `free: 0`, `credentialId`, and `coolingOffForSeconds` in `fleet_list`.
- It does not have `quarantinedForSeconds` unless exhaustion quarantine is also active.
- `fleet_profiles` has a separate `coolingOff` map, not an entry in `quarantined`.
- A direct spawn refusal says the credential is cooling off after repeated backend errors and gives
the remaining seconds.
Do not change `FleetHealthMonitor.coverage`. It still describes the periodic health webhook path.
Lead-pane outage delivery is a separate capability.
### Files owned
- `fleetd/src/main/java/dev/ltms/fleet/Fleetd.java`
- `fleetd/src/main/java/dev/ltms/fleet/config/FleetConfig.java`
- `fleetd/src/main/java/dev/ltms/fleet/config/ConfigRef.java`
- `fleetd/src/main/java/dev/ltms/fleet/member/CompositePeerLauncher.java`
- `fleetd/src/main/java/dev/ltms/fleet/placement/PlacementContext.java`
- `fleetd/src/main/java/dev/ltms/fleet/placement/PlacementPolicyUtil.java`
- `fleetd/src/main/java/dev/ltms/fleet/mcp/FleetMcp.java`
- `fleetd/src/test/java/dev/ltms/fleet/config/FleetConfigTest.java`
- `fleetd/src/test/java/dev/ltms/fleet/config/ConfigRefTest.java`
- `fleetd/src/test/java/dev/ltms/fleet/member/CompositePeerLauncherTest.java`
- `fleetd/src/test/java/dev/ltms/fleet/placement/PlacementPolicyTest.java`
- `fleetd/src/test/java/dev/ltms/fleet/mcp/FleetMcpTest.java`
- Add `fleetd/src/test/java/dev/ltms/fleet/BackendOutageFlowTest.java`.
- `fleetd/fleetd.example.yaml`
- `CLAUDE.md`
No earlier unit edits these files.
The lead, not a worker, must update `wiki/7-Use-Cases.md`, `wiki/9-Implementation.md`, and
`wiki/11-Features.md`. Project rules forbid workers from committing `wiki/`. The portable block in
`CLAUDE.md` and `wiki/7-Use-Cases.md` must remain byte-identical.
### Acceptance criteria
1. `errorPattern` binds per profile. Blank uses the legacy default and is reported as degraded
coverage. A config reload which changes it is reported as deferred because patterns are compiled
at startup.
2. A malformed `errorPattern` stops startup and names `profiles.<name>.errorPattern`.
3. The real production sink never silently drops an unknown target. A test captures its error log
and the fallback notice call.
4. One real backend-error classification through `CompletionResolver` marks only its member. It does
not cool the credential and does not send an outage notice.
5. Two real classifications for one credential within 60 seconds start one incident.
6. The integration test then calls the real explicit-profile spawn gate. It is refused before any
adapter spawn call, with “cooling off” and remaining seconds in the message.
7. The same test calls an automatic placement path. A cooling candidate is skipped. If every
candidate is cooling, the error names cool-off rather than exhaustion.
8. Profiles sharing the credential are all blocked. A profile on another credential stays usable.
9. `fleet_list` from the same fixture shows both members as `backend_error`, preserves each failure
reason, and reports `free: 0`, the credential, and `coolingOffForSeconds`.
10. `fleet_profiles` reports cool-off separately from quarantine.
11. The real `ReplyPushLoop` receives one incident and makes one successful lead-pane send. Existing
failed-ticket notice content may share that same combined send.
12. Exhaustion still wins when a line matches both patterns. A simultaneous exhaustion quarantine
also wins in spawn errors and fleet views.
13. After the 60-second cool-off, spawn is allowed again. A new incident needs two fresh errors.
14. `healthCoverage` has the same value before and after this change for the same health config.
15. `fleetd.example.yaml` explains `errorPattern`, the legacy fallback, 2/60/60 policy, and the
difference between cool-off and exhaustion quarantine.
16. `CLAUDE.md` tells leads how `fleet_profiles` and `fleet_list` report cool-off. The lead later
applies the matching wiki updates and runs the documented byte-sync check.
17. The developer records the new end-to-end test failing before implementation, then passing. The
focused suites and `mvn clean install` pass.
### Dependencies
Unit 5 starts only after Units 1 to 4 are merged or rebased into its branch. It also starts after the
#234 defect 2 fix lands, because both areas touch the same target-resolution control path.
### What to report back
- The exact commits used for Units 1 to 4 and #234.
- The startup coverage line with one configured and one legacy-default profile.
- The red test output before the implementation and its green result after.
- The explicit and automatic spawn refusal text.
- Sample `fleet_list` and `fleet_profiles` JSON for cool-off and exhaustion.
- The number and text of lead-pane sends in the real-path test.
- The focused test commands and final `mvn clean install` result.
- The exact `CLAUDE.md` change and the wiki edits the lead must apply.
## File ownership summary
| Area | Unit | Shared edit risk |
|---|---:|---|
| Completion classification | 1 | Only Unit 1 edits `CompletionResolver` and its test. |
| Correlation and cool-off state | 2 | New files only. |
| Lead push scheduling | 3 | Only Unit 3 edits `ReplyPushLoop` and its test. |
| Member terminal state | 4 | Only Unit 4 edits `MemberSession`, `SessionManager`, and their test. |
| Main composition, config, placement, MCP views, shipped prompt | 5 | Only Unit 5 edits `Fleetd`, `FleetConfig`, `CompositePeerLauncher`, placement context, `FleetMcp`, and `CLAUDE.md`. |
| Wiki propagation | Lead after Unit 5 | Workers do not commit the wiki submodule. |
## What I would not build
1. **Do not reuse `BackendQuarantine` for outages.** Its repeat call restarts a long credential
quarantine. Its fields and errors say “exhausted”. That is wrong for a short outage.
2. **Do not merge the exhaustion and generic error patterns.** Exhaustion must win because it has a
different policy and duration.
3. **Do not group by error string.** One outage can produce different text. The shared operational
limit is the credential.
4. **Do not mark a profile unusable until config changes.** The current classifier cannot safely
tell a permanent malformed request from a transient service fault. A permanent state would need
a stronger error taxonomy first.
5. **Do not quarantine on the first generic backend error.** That would turn one bad request or one
false pattern match into a fleet-wide capacity loss.
6. **Do not add this to `FleetHealthMonitor`.** The monitor samples slow member health. The exact
backend event already exists at completion resolution, and moving it to polling would lose type
and time.
7. **Do not add another direct lead injector.** `ReplyPushLoop` already owns status gating,
per-lead coalescing, retry bounds, and heartbeat stand-down.
8. **Do not change `healthCoverage` to `full`.** That field still means a webhook notification sink
exists for periodic health. A backend outage nudge does not make every health event visible.
9. **Do not persist incident history across daemon restart in this work.** Existing exhaustion
quarantine is also in memory. A 60-second state does not justify a new durable store.
10. **Do not build work recovery.** The PR-body survival story proves why checkpoint-first work is
useful, but these tickets are about detection, capacity, and signalling.
11. **Do not remove the legacy `API Error:` fallback in the first release.** Doing so would turn an
unedited config back into a false successful completion. Report it as degraded coverage instead.
12. **Do not reorder or add the old `visibleTurn` fallback from #201.** #211 already implemented the
narrow raw-scrape fallback at `CompletionResolver.classifyRawScrapeFallback`.
## Riskiest assumption and cheapest experiment
The riskiest assumption is that a configured error regex means “the backend failed this turn”. The
current code and test already show the counterexample: a worker may quote `API Error:` while writing
a valid report. Two such false matches on one credential would now remove capacity for 60 seconds.
The cheapest experiment is a replay corpus before Unit 5 ships:
1. Save the full pane text from the measured 2026-09-01 outage.
2. Produce one safe failure per backend with a disposable invalid endpoint or request.
3. Save one valid member report which quotes each error line.
4. Replay all samples through the real `CompletionResolver` test fixture.
5. Require outage samples to match and quoted-report samples not to match after assistant-block
extraction and baseline checks.
This costs no outage deployment and no real sleep. If quoted reports still match, narrow the profile
patterns before enabling correlation. Do not raise the threshold to hide a bad classifier.
## Sequencing with three developers
First wave:
1. Developer A: Unit 1, typed classification.
2. Developer B: Unit 2, credential outage policy.
3. Developer C: Unit 3, lead outage nudge.
As soon as one slot is free, start Unit 4. It is file-disjoint from every first-wave unit. Merge and
review Units 1 to 4 independently.
Start Unit 5 only after all four foundations and #234 are available. Unit 5 is the only high-conflict
integration branch, so no other active unit should touch its file list.
## Checks performed for this refinement
- Read issue #201 and issue #227 through their Gitea pages. Both showed zero comments.
- Read the source and tests named in the evidence section.
- Ran `git status --short --branch`; the branch was clean before this document was added.
- Ran `git log --oneline -12` to identify the branch base.
- I did not run Maven because this change adds only a design document.
- Rendered both Mermaid blocks with `npx @mermaid-js/mermaid-cli`; both commands succeeded.
+1 -1
View File
@@ -79,7 +79,7 @@ Unit tests (add to the existing `ClaudeCodeLauncher` test):
## 5. Config
Add to the launcher-level config (a bridged-level knob, not per-profile) in `bridged.yaml` +
Add to the launcher-level config (a fleetd-level knob, not per-profile) in `fleetd.yaml` +
`FleetConfig`:
```yaml
+1 -1
View File
@@ -140,7 +140,7 @@ Stage-2 AMQP adapter: absent → in-memory, present → AMQP).
## 5. Build & verification (worker side)
- Build with Maven from the worktree's `bridged/` dir. **Capture the exit code without a masking pipe**
- Build with Maven from the worktree's `fleetd/` dir. **Capture the exit code without a masking pipe**
(`mvn clean install; echo "MVN_EXIT=$?"` — never `mvn … | tail`, which hides failures).
- Read the real test totals from `target/surefire-reports/TEST-*.xml`, not from stdout scroll.
- You do **not** have IDE MCP access — do not claim `ide_diagnostics` results. The **primary** runs the
+3 -3
View File
@@ -208,7 +208,7 @@ they overlap (notably: the envelope is no longer optional, and dedup is split by
never an argument* — across the broker: cross-host, identity comes from the key. Complements
(not replaces) per-gateway broker logins over TLS.
2. **Profiles are owned by the worker's host.** `fleet_spawn(profile, host)` resolves the name in
the *target* gateway's `bridged.yaml`. Gateways advertise their profile names in presence
the *target* gateway's `fleetd.yaml`. Gateways advertise their profile names in presence
heartbeats, so a leader sees what each host offers before spawning; an unknown name is a clear
error from the target. Secrets (base URLs, tokens) never leave the host that uses them.
3. **Repo provisioning — clone from the forge, pinned.** A cross-host spawn names the repo URL and
@@ -275,11 +275,11 @@ they overlap (notably: the envelope is no longer optional, and dedup is split by
- **Gateway discovery:** how gateways find the broker and each other (static config vs. discovery).
- **Control authorization — THE GATE ON U4.** Signing (§7.1) settles *who sent it*; authorization
is *who may do what*. **Cross-host spawn must not land before the minimal version exists**: a
per-host allowlist in `bridged.yaml` — beside the peer public keys — of gateway ids permitted to
per-host allowlist in `fleetd.yaml` — beside the peer public keys — of gateway ids permitted to
publish control to this host, checked against the verified signature. A few lines of config and
check; without them, any principal holding broker credentials can start processes on every host
in the fleet.
- **Key distribution & rotation:** static config (host → public key in each `bridged.yaml`) is
- **Key distribution & rotation:** static config (host → public key in each `fleetd.yaml`) is
fine at the current 2–3 host scale; rotation is manual. A refinement, not a blocker.
- **Gateway death mid-turn:** the roster reaps it by missed heartbeat, and in-flight primary-bound
messages survive by broker durability; still open is reconciling *worker* state when the dead
+1 -1
View File
@@ -11,7 +11,7 @@ All five increments of §4 are done, including increment 5 (the §5 live checkli
## 1. Goal
Prove the [`PeerLauncher`](../bridged/src/main/java/dev/ltms/fleet/peer/PeerLauncher.java) SPI
Prove the [`PeerLauncher`](../fleetd/src/main/java/dev/ltms/fleet/peer/PeerLauncher.java) SPI
actually holds for a **non-Claude** coding agent by shipping a second, first-class in-tree
adapter: **opencode** (`opencode` 1.1.31, a provider-agnostic terminal coding agent).
+8 -8
View File
@@ -9,7 +9,7 @@ with no terminator, and our third-party members cannot detect it (§7.2).
· **Upstream issue:** [systems/vms#31](https://git.ltms.dev/systems/vms/issues/31)
The gateway went live on 2026-08-15 and replaced Bifrost. This plan says what that means for a
**member definition** in `bridged.yaml`, because that is the part of this repo the change actually
**member definition** in `fleetd.yaml`, because that is the part of this repo the change actually
touches.
---
@@ -134,16 +134,16 @@ own provider credentials. So 3b needs **no allowlist change**; only 3a does.
`HerdrPeerLauncher.resolveEnv` → `env.apply(name)`, which reads the **daemon's own process
environment**. The running `fleetd` inherited its environment when it started, so a variable added to
`secrets.sh` afterwards is simply not there — the launcher would inject an empty token and the
gateway would answer 401. This is the same failure as trap 1 in `scripts/redeploy-bridged.sh`
gateway would answer 401. This is the same failure as trap 1 in `scripts/redeploy-fleetd.sh`
(`WORKER_GITEA_TOKEN`), and it has the same fix: **restart from a login shell**, and use
`scripts/redeploy-bridged.sh --check` to confirm the name resolves before restarting anything.
`scripts/redeploy-fleetd.sh --check` to confirm the name resolves before restarting anything.
### 3c. What this does to `ccs`
Once a profile carries `baseUrl`, `tokenEnv` and `model` itself, the ccs instance stops being what
routes a member. Be precise about what is left, though: `configDir` still supplies **folder trust**
and `settings.json`, and dropping it is what produced the trust dialog and the wrong-model error
recorded in `bridged.yaml`. So ccs goes from *deciding where the tokens go* to *holding client-side
recorded in `fleetd.yaml`. So ccs goes from *deciding where the tokens go* to *holding client-side
state*. Less load-bearing, not removable.
### Why `/anthropic` and never `/v1/chat/completions`
@@ -217,7 +217,7 @@ defensible; picking one is not mine to do.
### D3 — token scope
One consumer, `claude-bridge`, its token in `${SHARED_ENV}/tools/secrets.sh` as `AI_GATEWAY_TOKEN`,
referenced by name only. Never the literal value in `bridged.yaml` — `tokenEnv` exists for this.
referenced by name only. Never the literal value in `fleetd.yaml` — `tokenEnv` exists for this.
---
@@ -226,7 +226,7 @@ referenced by name only. Never the literal value in `bridged.yaml` — `tokenEnv
```mermaid
flowchart TB
U1["U1 · consumer token<br/>issue via cockpit, add to secrets.sh"]
U2["U2 · profile + guard<br/>bridged.yaml, restart"]
U2["U2 · profile + guard<br/>fleetd.yaml, restart"]
U3["U3 · verify live<br/>spawn, prove thinking survives"]
U4["U4 · context7 via gateway<br/>.mcp.json + opencode.json"]
U5["U5 · docs<br/>CLAUDE.md, wiki 11-Features"]
@@ -238,7 +238,7 @@ flowchart TB
| # | Scope | Who | Why |
|---|---|---|---|
| U1 | Issue the `claude-bridge` consumer at `auth.ltms.dev`; store as `AI_GATEWAY_TOKEN` | **operator** | touches secrets and a host we do not own |
| U2a | New `gx` opencode profile on `/v1` — pure config, no guard change | **lead** | `bridged.yaml` is gitignored, so a worker cannot see or edit it |
| U2a | New `gx` opencode profile on `/v1` — pure config, no guard change | **lead** | `fleetd.yaml` is gitignored, so a worker cannot see or edit it |
| U2b | `local` → `/anthropic`; add `local-direct` weight 0; add `llm.ltms.dev` to the guard allowlist | **lead** | same |
| U2c | One restart from a **login shell**, after U2a and U2b | **lead** | picks up `AI_GATEWAY_TOKEN` into the daemon env *and* the guard allowlist, in one stop |
| U3 | Live spawn on both new profiles; confirm reasoning survives on each surface | **lead** | needs real spawns and the running daemon |
@@ -372,7 +372,7 @@ one turn; this one costs the whole task and is indistinguishable from a slow wor
> **Diagnosing a stuck opencode member.** Do not read its pane. The launcher writes its config to a
> temp dir and passes it as `OPENCODE_CONFIG` — find it with
> `ls -dt /var/folders/*/*/T/bridged-opencode-* | head -1`, check the provider block and the key's
> `ls -dt /var/folders/*/*/T/fleetd-opencode-* | head -1`, check the provider block and the key's
> length and prefix (never its value), then reproduce with `opencode run --auto -m <provider>/<model>`
> using the same `OPENCODE_CONFIG`. That is what turned "it hangs" into a one-line error.
+2 -2
View File
@@ -67,7 +67,7 @@ it will be disabled and the stage is wasted. So:
auth:
mode: loopback-trust # default — behaves exactly like today: loopback ⇒ PRIMARY, no token needed
# mode: token # every non-worker caller must present a valid bearer token
# tokenEnv: BRIDGED_API_TOKEN # host env var holding the token; never the literal value
# tokenEnv: FLEETD_API_TOKEN # host env var holding the token; never the literal value
```
`mode: loopback-trust` is the current behaviour, named honestly and now *chosen* rather than
@@ -114,7 +114,7 @@ not found), and the daemon that has been dogfooded for weeks runs as a bare fore
- `deploy/dev.ltms.fleet.plist` — launchd agent, the *actual* runtime here, with `KeepAlive` and
ordered start after herdr.
- `deploy/bridged.service` — systemd unit for the Linux gateways CB-308 introduces.
- `deploy/fleetd.service` — systemd unit for the Linux gateways CB-308 introduces.
Ordering after herdr is advisory in both: the herdr socket may not exist at boot, so the daemon
must **retry the socket rather than exit** — supervision ordering is a nicety, socket-retry is the
+1 -1
View File
@@ -807,7 +807,7 @@ Checked against `main` at `e09cac6` on 2026-08-15. Unit 2 was written as one blo
have since been built by separate CB tickets. Read this before planning the rest, or that work gets
done twice.
The check was a symbol survey of `bridged/src/main/java` plus the merge history. It tells you whether
The check was a symbol survey of `fleetd/src/main/java` plus the merge history. It tells you whether
the machinery exists at all. It is **not** a line-by-line audit of whether each criterion is fully
met, and I did not run one.
+123 -324
View File
@@ -1,311 +1,162 @@
# MCP Contract — `fleetd`'s unified gateway
# MCP flows and error model — `fleetd`
> **Status: 🔴 HISTORICAL DESIGN — do NOT use as the tool reference.** Written 2026-07-14, before
> any MCP code existed. The system shipped and this page never caught up, so **its tool names,
> parameter names and REST paths are wrong today**. Audited 2026-08-17; the specific drift:
> **What this page is.** The **flows**: how a delegation, a clarification, a detached task and a
> silent member each travel through `fleetd`. These shapes are what shipped, and they are hard to
> read off the code because they span the MCP face, the rendezvous registry, the `Injector` and
> herdr.
>
> - **Tools it names that do not exist:** `bridge_read`, `bridge_cancel`.
> - **Shipped tools it omits:** `bridge_poll`, `bridge_ack`, `bridge_profiles`, `bridge_whoami`.
> - **Parameter names are wrong nearly everywhere** — it says `message`/`target`/`timeout_seconds`/
> `block` where the code takes `content`/`sessionId`/`timeoutMs`/`wait`; `text` where
> `bridge_reply` takes `content`; `target` where `bridge_stop` takes `paneId`.
> - **REST paths are wrong:** it says `POST /workers` and `DELETE /workers/{paneId}`; the daemon
> serves `POST /members` and `DELETE /members/{paneId}`.
> **What this page is NOT: a tool reference.** It deliberately holds no tool catalogue, no
> parameter tables and no REST paths. **The live MCP schema is the authority** — each tool's own
> description and parameters, as mounted — with the intent→tool table in `CLAUDE.md` as the short
> form.
>
> **The authoritative tool surface is the live MCP schema** (each tool's own description and
> parameters, as mounted), with the intent→tool table in `CLAUDE.md` as the short form. Both were
> checked against `mcp/FleetMcp.java` on 2026-08-17 and are accurate.
> That absence is the fix for fleetd #114 (CB-609), and it is worth stating why. This page used to
> carry a full tool catalogue written in July 2026, before any MCP code existed. The code shipped;
> the page did not follow. By August it named two tools that do not exist, omitted five that do,
> had the wrong name for nearly every parameter, pointed at REST paths the daemon does not serve,
> and — worst — still described an identity model (*"any connection that does not map to a known
> worker is treated as a primary"*) that was a real privilege bug, fixed since by the ancestry
> walk in fleetd #161. Every one of those errors is the same error: **a second, hand-maintained
> copy of something the code already states**. So the second copy is gone rather than corrected.
> Only the flows remain, because a flow is a shape rather than a name, and shapes are what this
> page was ever good for.
>
> What is still worth reading here is **§6 — the flows and the error model** (rendezvous,
> `fleet_ask`, detached delivery, the turn-done fallback). The shapes it describes are the ones
> that shipped; only the names around them drifted. Rewriting this page is tracked as **CB-609**.
`fleetd` is the **sole communication gateway** for every Claude session in the bridge. Both
the **primary** (Opus, on subscription) and every **worker** (off-subscription Claude Code)
mount the *same* MCP server with a single `claude mcp add` line, and talk only through its
tools. No Claude session ever addresses a broker, a peer, or the network directly.
This document defines every MCP tool that face must expose, who may call it, its blocking
semantics, and how it maps onto the code already in the tree.
> The names that do appear below are checked by `McpContractDocTest`, which fails if this page
> names a `fleet_*` tool the server does not register. That test is the whole reason it is safe to
> write a tool name here at all.
---
## 1. Design constraints (non-negotiable)
## 1. Rendezvous flows
These come from the project's core invariants and bound every decision below.
### 1.1 Delegation — happy path
1. **One server, both roles.** The primary and all workers mount an identical server. The
catalog must serve both, and `fleetd` must decide *who is calling* from the connection —
never from a caller-supplied argument that could be spoofed.
2. **Subscription-safe by construction.** No MCP tool ever reads, sets, or forwards
`ANTHROPIC_BASE_URL`. Mounting the bridge cannot move a session off subscription.
Enforced today by [`SubscriptionGuard`](1-Architecture).
3. **Blocking rendezvous, no busy-poll.** The primary consumes a worker's reply through a
*single* MCP call that `fleetd` holds open — never a cross-turn poll loop that would burn
subscription quota.
4. **Status-gated delivery.** Anything that puts text into a worker flows through the existing
[`Injector`](1-Architecture): delivered only when the worker is `idle`/`blocked`, at most
one message per turn.
5. **`fleetd` owns policy; herdr owns PTYs.** MCP tools express *intent*; `fleetd`
translates it into guard checks, rendezvous bookkeeping, and herdr `agent.*` calls.
---
## 2. Topology
Both faces live in the one daemon. The **north face** is MCP (this document); the **south
face** is the herdr Unix socket. REST/SSE remains only for non-Claude clients and dashboards.
```mermaid
flowchart LR
OPUS["Opus — primary<br/>(Claude Code, env CLEAN)<br/>MCP client"]
subgraph BD["fleetd — standalone daemon"]
MCP["MCP server (north face)<br/>bridge_send · bridge_reply<br/>bridge_ask · bridge_status · lifecycle"]
RDV["rendezvous registry<br/>(blocking-call waiters)"]
INJ["Injector + StatusPoller<br/>(status-gated writer)"]
SOCK["herdr socket client (south face)"]
MCP --> RDV
RDV --> INJ
INJ --> SOCK
MCP --> SOCK
end
HERDR["herdr<br/>panes · agent-status"]
W["worker claude pane<br/>ANTHROPIC_BASE_URL set<br/>MCP client"]
OPUS -->|"bridge_send (blocks)"| MCP
W -.->|"bridge_reply / bridge_ask"| MCP
SOCK -->|"agent.start · agent.send<br/>agent.get · pane.close"| HERDR
HERDR -->|"drives PTY"| W
classDef ext fill:#2b6cb0,stroke:#1a365d,color:#ffffff;
classDef core fill:#2f855a,stroke:#22543d,color:#ffffff;
class OPUS,W ext
class MCP,RDV,INJ,SOCK core
```
---
## 3. Identity & addressing
Because the same server is mounted by everyone, `fleetd` resolves the caller's role on every
request — this is the linchpin of the whole contract and has no code yet.
- **Workers are known.** `fleetd` spawns every worker
([`WorkerService`](1-Architecture)) and records its herdr session UUID / `terminal_id` on
the returned [`Agent`]. When a call arrives on a connection that maps to a known worker,
the caller is *that* worker — so **workers never pass a target**; routing is implicit.
- **The primary is "not a worker".** Any connection that does not map to a known worker is
treated as a primary. It addresses workers **explicitly** by `target` — a session UUID,
a `terminal_id`, or a friendly `profile` name.
- **Turn correlation.** A blocking `bridge_send` registers a *waiter* keyed by worker
identity. A worker's later `bridge_reply` / `bridge_ask` on the same identity resolves that
waiter. A `turn_id` is minted per exchange so a clarification round-trip
(§6.2) rejoins the right turn.
---
## 4. Transport
`fleetd` is a long-lived daemon serving **multiple** concurrent clients (one primary + N
workers), so a per-client stdio child is the wrong shape. The recommended transport is
**streamable-HTTP / SSE** on the same bind as the REST face:
```bash
# identical on primary and every worker
claude mcp add --transport http fleetd http://127.0.0.1:8080/mcp
```
This adds an MCP-server dependency the pom does not yet carry. See [Open decisions](#10-open-decisions).
---
## 5. Tool catalog
| Tool | Caller | Blocks? | Backing (exists today?) |
|---|---|---|---|
| [`bridge_send`](#bridge_send) | primary | yes (default) | `Injector.enqueue` ✅ · rendezvous registry ❌ (CB-104) |
| [`bridge_reply`](#bridge_reply) | worker | no | rendezvous ❌ · pane injection via `Injector` ✅ |
| [`bridge_ask`](#bridge_ask) | worker | yes | reverse rendezvous ❌ |
| [`bridge_status`](#bridge_status) | either | no | `AgentControl.status` ✅ · `Injector.activeTargets` ✅ |
| [`bridge_spawn`](#lifecycle) | primary | no | `WorkerService.spawn` ✅ (`POST /workers`) |
| [`bridge_list`](#lifecycle) | either | no | `WorkerService.list` ✅ (`/agents`) |
| [`bridge_stop`](#lifecycle) | primary | no | `WorkerService.stop` ✅ (`DELETE /workers/{paneId}`) |
| [`bridge_read`](#bridge_read) | primary | no | `AgentControl.read` ✅ |
| [`bridge_cancel`](#bridge_cancel) | primary | no | — ❌ (future) |
### Core: delegation & rendezvous
#### `bridge_send`
*(primary → worker — the headline tool, CB-104)*
- **Params:** `message` (required); `target` (optional — defaults to the sole worker / default
profile); `timeout_seconds` (default 600); `block` (default `true`); `auto_spawn`
(default `true`); `turn_id` (optional — supplied when answering a worker's `bridge_ask`).
- **Blocking (`block:true`):** enqueue `message` via the `Injector`, then hold the call open
until exactly one of:
- worker calls `bridge_reply` → `{ outcome:"reply", text }`
- worker calls `bridge_ask` → `{ outcome:"question", text, turn_id }`
- worker's `agent_status` reaches done/idle with no reply → `{ outcome:"turn_done", text:<terminal tail> }`
- deadline elapses → `{ outcome:"timeout" }`
- worker gone → error `worker_gone`
- **Detached (`block:false`):** enqueue and return `{ outcome:"dispatched", dispatch_id }`
immediately. The eventual reply is injected into the primary's idle pane (§6.3), or drained
via `bridge_status` on a split-host primary.
#### `bridge_reply`
*(worker → primary)*
- **Params:** `text` (required); `final` (default `true`).
- **Behavior:** resolve the primary waiter registered against this worker with `text`. If no
waiter exists (detached delegation), `fleetd` **injects the primary's idle pane** instead.
Returns `{ delivered:true, mode:"resolved"|"injected" }`. No `target` — identity is implicit.
#### `bridge_ask`
*(worker → primary — the reverse rendezvous)*
- **Params:** `question` (required); `timeout_seconds`.
- **Behavior:** blocks the *worker's* call. Surfaces the question to the primary (resolving its
open `bridge_send` with `outcome:"question"`, or injecting its pane). When the primary
answers — a `bridge_send` carrying the matching `turn_id` — that unblocks this call and
returns `{ answer }` to the worker, which continues **in the same turn**.
### Worker lifecycle
<a id="lifecycle"></a>
Thin adapters over [`WorkerService`](1-Architecture) — parity with the existing REST routes.
- **`bridge_spawn`** — `{ profile? }` → worker view (`sessionId`, `terminalId`, `paneId`,
`status`). Guard-checked; a boundary breach returns error `subscription_boundary` (the
REST `403`).
- **`bridge_list`** — no params → all workers + `agent_status`. Read-only, either role.
- **`bridge_stop`** — `{ target }` → tears down the pane and its dedicated tab. Idempotent.
### Observability
#### `bridge_status`
*(either role — the README's 4th named tool)*
- **Params:** `target?`.
- **Behavior:** per-worker `agent_status`, queue depth (`Injector.activeTargets`), whether a
rendezvous is open, and ids. For the *calling* session it also reports/drains **pending
messages addressed to me** — the path a split-host primary's `Stop`-hook uses to wake and
collect replies without being injectable. Read-only, non-blocking.
#### `bridge_read`
*(primary)*
- **Params:** `target`; `source` ∈ `visible | recent | recent_unwrapped | detection`.
- **Behavior:** returns the worker's terminal text so the primary can peek at a *detached*
worker's progress. Adapter over `AgentControl.read`.
### Control (future)
#### `bridge_cancel`
*(primary)*
- **Params:** `target`. Interrupt the worker's current turn / abandon the rendezvous. No
backing code yet.
---
## 6. Rendezvous flows
### 6.1 Delegation — happy path
One blocking call, zero polls.
One blocking call, zero polls. The lead's call is held open by `fleetd` until the member answers.
```mermaid
sequenceDiagram
participant P as Primary (Opus)
participant B as fleetd (MCP + Injector)
participant P as "Lead (primary)"
participant B as "fleetd (MCP + Injector)"
participant H as herdr
participant W as Worker (Claude)
participant W as "Member"
P->>B: fleet_send("do X", target=w) — blocks
B->>B: register waiter(w)
B->>H: agent.send(w, "do X") (idle window)
H-->>W: prompt injected
W->>W: works the turn
W->>B: fleet_reply("result")
B->>B: resolve waiter(w)
B-->>P: { outcome:"reply", text:"result" }
P->>B: "fleet_send{sessionId, content} — blocks"
B->>B: "register waiter(sessionId)"
B->>H: "agent.send — only in an injectable window"
H-->>W: "prompt injected"
W->>W: "works the turn"
W->>B: "fleet_reply{content}"
B->>B: "resolve waiter"
B-->>P: "{ outcome: reply }"
```
### 6.2 Clarification — reverse rendezvous (`fleet_ask`)
**The cap that matters:** a blocking `fleet_send` is bounded by the *caller's own* MCP client
timeout, about 60 seconds — not by the task. Anything slower than that must use the detached flow
in §1.3, or the lead's call returns while the member is still working.
The worker pauses mid-turn to ask; the primary answers; the worker resumes in the same turn.
### 1.2 Clarification — reverse rendezvous
The member pauses mid-turn to ask, the lead answers, and the member resumes **the same turn** with
its context intact.
```mermaid
sequenceDiagram
participant P as Primary
participant P as "Lead"
participant B as fleetd
participant W as Worker
participant W as "Member"
P->>B: fleet_send("do X", target=w) — blocks
B-->>W: "do X" (injected)
W->>B: fleet_ask("which config?") — worker blocks
B-->>P: { outcome:"question", text:"which config?", turn_id }
P->>B: fleet_send("config.yaml", target=w, turn_id) — blocks again
B-->>W: resolve fleet_ask → { answer:"config.yaml" }
W->>W: resumes same turn
W->>B: fleet_reply("done")
B-->>P: { outcome:"reply", text:"done" }
P->>B: "fleet_send{sessionId, content} — blocks"
B-->>W: "content injected"
W->>B: "fleet_ask{question} — member blocks"
B-->>P: "{ outcome: question, turnId }"
P->>B: "fleet_send{turnId, content} — answers THIS turn"
B-->>W: "fleet_ask returns the answer"
W->>W: "resumes the same turn"
W->>B: "fleet_reply{content}"
B-->>P: "{ outcome: reply }"
```
### 6.3 Detached delegation — pane injection
**Answer with `turnId`, never `sessionId`.** A `sessionId` send starts a new turn; it does not
resolve the waiting `fleet_ask`.
The primary does not block; the reply arrives later in its idle pane.
**The window is about 55 seconds and no nudge extends it.** So never brief a member to "ask me":
decide before delegating, or give the member an explicit default to fall back on.
### 1.3 Detached delegation — the lead does not block
The lead gets a ticket immediately and collects the answer later. This is the flow for any real
task, because of the ~60s cap in §1.1.
```mermaid
sequenceDiagram
participant P as Primary
participant P as "Lead"
participant B as fleetd
participant W as Worker
participant W as "Member"
P->>B: fleet_send("do X", target=w, block=false)
B-->>P: { outcome:"dispatched", dispatch_id }
P->>P: continues its own work
W->>B: fleet_reply("result")
Note over B: no waiter → detached path
B->>B: Injector.enqueue(primary_pane, "result")
B-->>P: injected into idle pane (status-gated)
P->>B: "fleet_send{sessionId, content, wait:false}"
B-->>P: "accepted — ticket"
P->>P: "continues its own work"
W->>B: "fleet_reply{content}"
Note over B: "no waiter is blocked — the reply is held"
B->>B: "nudge the lead's own pane (status-gated)"
P->>B: "fleet_poll{ticket}"
B-->>P: "the member's report"
P->>B: "fleet_ack{target, msgId}"
```
### 6.4 Uncooperative worker — turn-done fallback
A terminal ticket nudges the lead's pane by itself, so a detached task does not need watching. The
nudge needs an injectable lead pane and is capped, so it is a convenience rather than a guarantee.
A worker that never calls `fleet_reply` still returns a result: `fleetd` reads its terminal
tail when the turn completes.
### 1.4 The member never replies — turn-done fallback
A member that ends its turn without `fleet_reply` still produces something: `fleetd` reads its
pane tail. This is a **fallback, not a channel** — it is lossy in three separate ways, and every
one of them has produced a wrong answer in practice.
```mermaid
sequenceDiagram
participant P as Primary
participant P as "Lead"
participant B as fleetd
participant W as Worker
participant W as "Member"
P->>B: fleet_send("do X", target=w) — blocks
B-->>W: "do X" (injected)
W->>W: works, never calls fleet_reply
B->>B: StatusPoller sees agent_status → idle/done
B->>B: AgentControl.read(w, "recent")
B-->>P: { outcome:"turn_done", text:<terminal tail> }
P->>B: "fleet_send — blocks or detaches"
B-->>W: "content injected"
W->>W: "works, never calls fleet_reply"
B->>B: "StatusPoller sees the turn end"
B->>B: "read the pane tail"
B->>B: "classify: exhausted? echoed brief? real report?"
B-->>P: "{ outcome: turn_done } or a named failure"
```
The three ways it goes wrong, and what each looks like now:
| What happened | What the lead used to get | What it gets today |
|---|---|---|
| The report is longer than the scrape window | The **end** silently cut off | Still clipped, but marked partial |
| The member never started — spent credential | The lead's **own brief** echoed back as a report | A named failure: backend exhausted |
| The member is simply slow | A tail of work in progress | Unchanged — read it as a hint, not a result |
The echoed-brief case is the one to remember: it reads as a long, on-topic report with nothing in
it from the member. It is suppressed now, but the general rule stands — **check the member's
worktree with `git log` before believing a report you did not watch arrive.**
---
## 7. Status gating
## 2. Status gating
Delivery only happens in a safe window. This is the state machine the `Injector` already
enforces via `AgentStatus.injectable()`; MCP `bridge_send` is simply its producer.
Delivery only happens in a safe window. `fleet_send` is a producer for the `Injector`, which
already enforces this through `AgentStatus.injectable()`.
```mermaid
stateDiagram-v2
[*] --> IDLE
IDLE --> WORKING: message delivered / picks up
WORKING --> IDLE: turn done
WORKING --> BLOCKED: awaits input
BLOCKED --> WORKING: input delivered
IDLE --> UNKNOWN: detection glitch
BLOCKED --> UNKNOWN: detection glitch
UNKNOWN --> IDLE: re-detected
IDLE --> WORKING: "message delivered, picked up"
WORKING --> IDLE: "turn done"
WORKING --> BLOCKED: "awaits input"
BLOCKED --> WORKING: "input delivered"
IDLE --> UNKNOWN: "detection glitch"
BLOCKED --> UNKNOWN: "detection glitch"
UNKNOWN --> IDLE: "re-detected"
note right of IDLE
injectable — deliver head of FIFO
@@ -321,69 +172,17 @@ stateDiagram-v2
end note
```
At most one message is delivered per turn: after a send the `Injector` waits for a `WORKING`
pickup before delivering the next, with a `PICKUP_GRACE_POLLS` fallback for turns faster than
the poll interval. A herdr `events.subscribe` stream can later replace the sampling without
touching this state machine.
**At most one message per turn.** After a send, the `Injector` waits for a `WORKING` pickup before
delivering the next, with a grace-poll fallback for turns that finish faster than the poll
interval.
---
Two consequences a lead feels directly:
## 8. Error model
- **A second send to a busy member never lands.** It reports as queued and times out. The member
is fine; the message simply waits, and then restarts the member when it next goes idle.
- **A spawned member is not deliverable until it has mounted the MCP.** Until then a send waits on
that gate for about 60 seconds and then fails without ever reaching the pane.
| Condition | `bridge_send` result | Notes |
|---|---|---|
| Worker replies | `{ outcome:"reply" }` | normal |
| Worker asks | `{ outcome:"question", turn_id }` | answer with `bridge_send(turn_id)` |
| Turn ends, no reply | `{ outcome:"turn_done" }` | terminal tail as text |
| Deadline elapsed | `{ outcome:"timeout" }` | message may still be queued/delivered |
| Worker vanished | error `worker_gone` | `Injector.drop` fails the queued future |
| Guard breach on spawn | error `subscription_boundary` | REST `403` parity |
| Delivery failed at herdr | error, message dropped | poisoned message not left blocking the FIFO |
`bridge_reply` from a worker with no open waiter is **not** an error — it falls through to
detached pane injection (§6.3).
---
## 9. Mapping to existing code
The MCP face is a thin adapter layer; nearly every capability already exists behind the REST
seam. Only the **rendezvous registry** and the **caller-identity resolver** are new.
| MCP tool | Existing collaborator | New work |
|---|---|---|
| `bridge_send` | `Injector.enqueue`, `AgentControl.send` | waiter registry, timeout, outcome mux (CB-104) |
| `bridge_reply` / `bridge_ask` | `Injector` (pane injection) | reverse rendezvous, identity resolver |
| `bridge_status` | `AgentControl.status`, `Injector.activeTargets` | pending-drain projection |
| `bridge_spawn` / `list` / `stop` | `WorkerService.{spawn,list,stop}` | MCP adapter only |
| `bridge_read` | `AgentControl.read` | MCP adapter only |
Because the REST routes in `FleetApp` already exercise the collaborators, MCP tools are
validated by **parity** against those routes, not by re-testing behavior.
---
## 10. Open decisions
1. **`bridge_ask` direction.** This page defines it as *worker-asks-primary* (a genuine reverse
channel, matching the "inject the primary's pane" language). The alternative — a synonym for
a blocking primary→worker send — is weaker and produces different plumbing. **Recommend
worker-asks-primary.**
2. **Detached delivery shape.** A `block:false` param on `bridge_send` (keeps the catalog
small) vs. a separate `bridge_dispatch` tool. **Recommend the param.**
3. **Auto-spawn on send.** `bridge_send` provisions a worker per profile when none exists
(simplest primary UX) vs. requiring an explicit `bridge_spawn` first. **Recommend
auto-spawn, defaulting on.**
4. **Transport & SDK.** Streamable-HTTP/SSE co-located with the REST bind (recommended) vs.
stdio. Requires choosing a Java MCP server SDK and adding it to the pom.
---
## 11. Implementation staging
- **CB-104** — blocking `bridge_send` + rendezvous registry + caller-identity resolver
(the producer that finally drives the inert `StatusPoller`).
- **CB-1xx** — `bridge_reply` / `bridge_ask` reverse rendezvous + detached pane injection.
- **CB-1xx** — lifecycle + observability adapters (`bridge_spawn/list/stop/status/read`).
- **CB-1xx** — transport wiring + `claude mcp add` docs; parity tests vs. REST.
- **Later** — `bridge_cancel`; swap `StatusPoller` for herdr `events.subscribe`.
`UNKNOWN` is deliberately neither injectable nor a pickup. A pane whose status cannot be read is
not a pane that is safe to write to — see fleetd #176 for what happens when a gate treats an
unreadable pane as a ready one.
+208
View File
@@ -0,0 +1,208 @@
# Wiki audit for #168
**Source checked:** `.wiki-snapshot/` at `68e32c6` (2026-08-31). I did not use
`wiki/`. Code references below are from the current `fleetd` source tree. A quoted
line is a concrete claim that needs correction, unless the table says `KEEP`.
| Page | Verdict | One-line reason |
|---|---|---|
| `Home.md` | REVISE | Good overview, but it still names the retired product. |
| `_Sidebar.md` | REVISE | The heading still says `claude-bridge`. |
| `1-Architecture.md` | REBUILD | Its component contract mixes current names with removed tools, routes, and planned backends. |
| `2-Message-Server.md` | REBUILD | The claimed MCP schema, mount command, REST/SSE surface, and fallback paths are pre-build design. |
| `3-Approaches.md` | REVISE | Useful research history, but it presents unbuilt AgentAPI as a selectable fallback. |
| `4-Setup.md` | RETIRE | It is an intentional stub that only redirects to chapter 13. |
| `5-Operations.md` | RETIRE | It is an intentional stub that only redirects to chapter 13. |
| `6-Team.md` | REBUILD | It teaches role-addressed sends and a Claude-only team model that the shipped API does not have. |
| `7-Use-Cases.md` | REBUILD | Its flagship flow depends on removed `ccs` profiles and removed send parameters. |
| `8-Roadmap.md` | REBUILD | It is a historical plan, but it presents old implementation choices and planned work as the current stack. |
| `9-Implementation.md` | REBUILD | Its package, class, endpoint, and outcome map has drifted from the source. |
| `10-Cross-Host-Messaging.md` | REVISE | It labels most federation work proposed, but misses the shipped `coordinator:` lead channel. |
| `11-Features.md` | REVISE | It is the right catalogue, but code-path names are old and it misses the second-herdr-daemon capability. |
| `12-Claude-to-OpenCode.md` | REVISE | The porting guide is mostly current, but calls the product and spawned-member path a bridge. |
| `13-User-Guide.md` | REVISE | It is the best operator page, but needs the product rename and the second-herdr-daemon setup. |
## Pages needing work
### `Home.md` — REVISE
- Quote: `# claude-bridge` (line 1) and `` `claude-bridge` keeps`` (line 11).
The product is `fleet` / `fleetd`. The MCP server identifies itself as `fleet` in
`fleetd/src/main/java/dev/ltms/fleet/mcp/FleetMcp.java:313-315`.
- Quote: `AgentAPI ... swappable fallback injector` (lines 73-76).
There is no AgentAPI implementation under `fleetd/src/main/java`; the actual
launchers are selected by `Profile.kind` in
`fleetd/src/main/java/dev/ltms/fleet/config/FleetConfig.java:265-270`.
### `_Sidebar.md` — REVISE
- Quote: `### 📖 claude-bridge` (line 1).
Rename it to `fleet`. `FleetMcp` registers the current product-facing tool set at
`fleetd/src/main/java/dev/ltms/fleet/mcp/FleetMcp.java:301-326`.
### `1-Architecture.md` — REBUILD
- Quote: `` `claude-bridge` lets`` (line 3). The product was renamed; the MCP
server name is `fleet` (`FleetMcp.java:313-315`).
- Quote: ``fleet_read`` in the tool list (line 102). No such tool is registered.
The complete registered list is `fleet_send` through `fleet_whoami` at
`FleetMcp.java:301-326`; `fleet_read` is absent.
- Quote: `SSE (GET /events)` (line 143). `FleetApp.build()` registers no `/events`
route; its routes are listed at `FleetApp.java:143-159`.
- Quote: `Redis Streams / NATS JetStream, or an embedded queue` (line 106).
The shipped durable inbox is AMQP, configured by `broker`, at
`FleetConfig.java:49-50` and `FleetConfig.java:655-714`.
- Quote: `AgentAPI (fallback)` (line 107). No AgentAPI adapter exists; shipped
launcher kinds are `claude-code` and `opencode` (`FleetConfig.java:265-270`).
### `2-Message-Server.md` — REBUILD
- Quote: `claude mcp add --transport http bridge http://127.0.0.1:8080/mcp`
(line 67). The daemon defaults to port `8765` in `FleetConfig.java:183-187`,
and identifies its server as `fleet` at `FleetMcp.java:313-315`.
- Quote: ``fleet_send(message, target?, {block, timeout_seconds, auto_spawn,
turn_id})`` (line 80). The real parameters are `sessionId`, `content`,
`timeoutMs`, `wait`, `turnId`, and `coordId` (`FleetMcp.java:1096-1108`).
- Quote: ``fleet_read(target, source)`` (line 85). It is not registered; see the
complete registration at `FleetMcp.java:301-326`.
- Quote: `docs/MCP-Contract.md ... normative` (lines 87-88). That is not a valid
reference: only §6 is current, as the current operator guide itself says at
`.wiki-snapshot/13-User-Guide.md:466`.
- Quote: `SSE (GET /events)` (line 45). No route exists in the built REST surface,
`FleetApp.java:143-159`.
### `3-Approaches.md` — REVISE
- Quote: `AgentAPI ... remains a swappable fallback injector` (lines 78-84).
It was never built. The shipped adapter selection is only `claude-code` or
`opencode` (`FleetConfig.java:265-270`). Keep it as discarded research, not an
operational fallback.
- Quote: `claude-bridge` (line 109). Rename the product to `fleet`; the runtime
package is `dev.ltms.fleet`, for example `FleetMcp.java:1`.
### `4-Setup.md` — RETIRE
It is a 25-line redirect and says its procedure was never written (lines 3-9).
Chapter 13 is the maintained install procedure. Keeping a second navigation page
adds no working documentation.
### `5-Operations.md` — RETIRE
It is a 35-line redirect and says its runbook was never written (lines 3-14).
Chapter 13 now owns run and recovery instructions.
### `6-Team.md` — REBUILD
- Quote: `fleet_send {role: w-claude, prompt: A}` (line 98). `fleet_send` accepts
`sessionId` and `content`, not `role` or `prompt` (`FleetMcp.java:1096-1108`).
- Quote: `some on Claude, some on the remote local LLM` (lines 3-5) and `Every
worker is ... Claude Code` (line 25). `opencode` is a first-class launcher kind,
not a Claude worker (`FleetConfig.java:265-270`).
- Quote: `fleetd's concurrency policy` (line 121). The configured capacity control
is per-profile `maxLoad` (`FleetConfig.java:251-264`), not the role routing model
described here.
### `7-Use-Cases.md` — REBUILD
- Quote: `ccs profile` (line 10), `ccs + herdr` (line 22), and `ccs-spawn`
(line 45). The configuration has `profiles` and `fleet`, not `ccs`:
`FleetConfig.java:34-58` and `FleetConfig.java:81-101`.
- Quote: `fleet_send({"to", "kind", "body", "block"})` (lines 55-62).
None of those are the shipped send parameters. The schema is
`FleetMcp.java:1096-1108`.
- Quote: `fleet_list() → { "profiles": ... }` (lines 74-80). `fleet_list` is a
roster view; `fleet_profiles` is the configured-backend view, as registered at
`FleetMcp.java:307-311` and described at `FleetMcp.java:1176-1182`.
### `8-Roadmap.md` — REBUILD
- Quote: `Java 21+` (line 43). The current project guidance and source use Java 25;
the `FleetConfig` source itself uses Java 25 unnamed lambda parameters, for
example `FleetConfig.java:102`.
- Quote: `herdr 0.7.0 / protocol 14` (line 46). The current REST health endpoint
reports the live protocol returned by herdr (`FleetApp.java:240-244`), while the
current operator guide records protocol 19 at
`.wiki-snapshot/13-User-Guide.md:76-85`.
- Quote: `ccs <profile> claude` and `ccs env <profile>` (lines 47-48). Shipped
configuration uses `Profile` records and launcher `kind`,
`FleetConfig.java:313-330` and `FleetConfig.java:265-270`.
- Quote: `Redis Streams via Lettuce` (line 50). The actual durable inbox is AMQP
`broker`, `FleetConfig.java:655-714`.
### `9-Implementation.md` — REBUILD
- Quote: `rest.FleetdApp` and `mcp.BridgeMcp` (lines 29-30). The classes are
`rest.FleetApp` and `mcp.FleetMcp` (`FleetApp.java:46`; `FleetMcp.java:67`).
- Quote: `dev.ltms.fleetd` (line 67). The source package is `dev.ltms.fleet`
(`FleetMcp.java:1`).
- Quote: `WorkerPresence` (line 110). The current class is `MemberPresence`, as
imported and used by `FleetMcp` at `FleetMcp.java:12` and `465-469`.
- Quote: the outcome list ending in `STALE_TURN` (lines 128-131). The code also
has `BACKEND_EXHAUSTED` (`FleetMcp.java:550-554`) and async `ASKING` handling
(`FleetMcp.java:664-668`).
- Quote: `FleetdApp` (line 207) and `FleetdConfig` (line 211). These names do not
resolve; current classes are `FleetApp` and `FleetConfig`.
### `10-Cross-Host-Messaging.md` — REVISE
- Quote: the chapter says the cross-host fabric is proposed except for the
single-host inbox (lines 3-8). Cross-host **lead-to-lead** delivery shipped:
`fleet_send` accepts `coordId` (`FleetMcp.java:1094-1107`) and publishes it at
`FleetMcp.java:616-641`; configuration has `coordinator` at
`FleetConfig.java:74-78` and `99-101`.
- Quote: `bridge.dlx` (line 90). This product name is stale. The shipped lead path
uses `LeadChannel`, not the proposed exchange flow (`FleetMcp.java:95-96` and
`616-641`). Keep the proposed federation design, but add a clear shipped/proposed
boundary for CB-637.
### `11-Features.md` — REVISE
- Quote: `mcp/BridgeMcp` (line 22), `config/FleetdConfig` (lines 25-27), and other
index references. These paths no longer resolve; the source classes are
`mcp/FleetMcp` (`FleetMcp.java:67`) and `config/FleetConfig`
(`FleetConfig.java:81`).
- Quote: `fleet_whoami` returns only `primary` or `worker` (lines 99-100).
It also returns `architect` (`FleetMcp.java:1235-1244`).
- The page needs the missing separate member-herdr-daemon feature listed below.
### `12-Claude-to-OpenCode.md` — REVISE
- Quote: `same bridge mount` (line 5) and `a bridge-spawned worker` (line 94).
Rename the product path to `fleet`. The daemon exposes the MCP server as `fleet`
(`FleetMcp.java:313-315`), and profiles select OpenCode with `kind: opencode`
(`FleetConfig.java:332-335`).
- Quote: the sample mount name is `fleetd` (line 67). The server name is `fleet`;
update the sample to avoid teaching a second product name.
### `13-User-Guide.md` — REVISE
- Quote: `The bridge is the only channel` (line 63). The invariant is correct, but
the product term needs the `fleet` rename. The daemon's MCP server name is
`fleet` (`FleetMcp.java:313-315`).
- Quote: it describes one herdr socket (lines 72-85). It needs the optional
`memberHerdrSocket` setup and two-daemon health meaning. The config key is in
`FleetConfig.java:34-37`, and `/healthz` checks both daemons when configured at
`FleetApp.java:210-245`.
## MISSING
`11-Features.md` has a body section for **routing members through a separate herdr daemon**
(`## memberHerdrSocket`, line 2174), but **no row in the index table** at the top of the page
(lines 20-95). That table is how the page is meant to be read, so a capability absent from it is
effectively undiscoverable. Lead note: this is my own omission — I added the section on 2026-08-31
and did not add the matching row. Fixed in the wiki at `68e32c6`'s successor.
The original audit stated the feature had no entry at all. That was wrong: the section exists. The
gap is the index row. Recorded here rather than silently corrected, because the difference matters —
"undocumented" and "documented but unindexed" are different jobs.
Evidence for the feature itself: `FleetConfig.java:34-37` and `FleetApp.java:103-115`, `210-245`,
and `247-263`.
## Audit method and coverage
I checked all 15 pages. I checked concrete tool, route, config, class, file, and
product-name claims claim-by-claim on 11 pages: Home, Sidebar, 1, 2, 4, 5, 6, 7, 9,
11, and 13. I skimmed the remaining four long historical or research pages (3, 8, 10,
12), then checked their concrete claims that affect the verdict. This is an audit of
the supplied snapshot, not a wiki rewrite.
+3 -3
View File
@@ -2,7 +2,7 @@
A standard, repeatable **live** end-to-end test of the two-way channel: it drives a real
multi-turn conversation between a primary and an off-subscription worker **through the
running `bridged` daemon**, captures the full transcript, and grades the channel.
running `fleetd` daemon**, captures the full transcript, and grades the channel.
This is the committed form of the ad-hoc channel test that discovered the CB-115 gaps
(herdr `unknown` misclassification wedging delivery, dirty completion scrapes, and workers
@@ -20,7 +20,7 @@ construction** — it only calls the bridge's loopback REST face.
```mermaid
sequenceDiagram
participant T as conversation_test.py
participant B as bridged (REST)
participant B as fleetd (REST)
participant W as worker (off-sub)
T->>B: POST /workers (spawn)
T->>B: GET /sessions/{id}/status (await ready)
@@ -37,7 +37,7 @@ sequenceDiagram
## Prerequisites
- `bridged` is running (default REST on `http://127.0.0.1:8765`) with at least one worker
- `fleetd` is running (default REST on `http://127.0.0.1:8765`) with at least one worker
profile configured and its backend reachable.
- herdr is up (the daemon needs it).
- Python 3 (standard library only — no pip installs).
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""Sustained back-and-forth bridge test — ONE primary, ONE worker, many dependent turns
over a fixed wall-clock window (default 5 minutes), through the running `bridged` daemon.
over a fixed wall-clock window (default 5 minutes), through the running `fleetd` daemon.
Where conversation_test.py proves a handful of turns work and issue_hunt_test.py proves
fan-out isolation, this proves the channel stays healthy under a *sustained, stateful*
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""Standard bridge conversation test — a multi-turn primary↔worker exchange through
the running `bridged` daemon, fully captured, with automatic gap analysis.
the running `fleetd` daemon, fully captured, with automatic gap analysis.
This is the repeatable form of the ad-hoc channel test that surfaced the CB-115 gaps
(herdr `unknown` misclassification, dirty completion scrape, workers not calling
@@ -24,7 +24,7 @@ Like the rest of the suite it talks ONLY to the bridge's REST face on loopback
ANTHROPIC_BASE_URL and never touches herdr, so it is subscription-safe by construction.
Usage:
python3 bridge_ask_test.py [--base URL] [--profile NAME] [--repo DIR] [--out DIR]
python3 fleet_ask_test.py [--base URL] [--profile NAME] [--repo DIR] [--out DIR]
[--send-timeout SECS] [--answer-timeout SECS] [--keep-worker]
--base bridge REST base URL (default http://127.0.0.1:8765)
@@ -1,12 +1,12 @@
# Live bridge_ask — reverse rendezvous — 2026-07-16 16:30
# Live fleet_ask — reverse rendezvous — 2026-07-16 16:30
One worker paused its delegated turn to ask the primary, then resumed with the answer (profile `default`). Result: **`OK`**.
## Round-trip
1. **primary → worker** (delegation): the ask-forcing task.
2. **worker → primary** (`bridge_ask`, 6.6s): 'PICK A COLOR: red or blue?' — surfaced on the primary's blocked send as a `question` with `turnId=term_656bb47d2c42a9e#1`.
2. **worker → primary** (`fleet_ask`, 6.6s): 'PICK A COLOR: red or blue?' — surfaced on the primary's blocked send as a `question` with `turnId=term_656bb47d2c42a9e#1`.
3. **primary → worker** (answer on that turn): `blue`.
4. **worker → primary** (`bridge_reply`, 7.9s, source=reply): 'CHOSEN=BLUE'
4. **worker → primary** (`fleet_reply`, 7.9s, source=reply): 'CHOSEN=BLUE'
> **OK:** asked, resumed the same turn, and the reply reflected the primary's answer
+4 -4
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""Standard bridge fan-out test — ONE primary vs MANY workers, concurrently, for
issue hunting through the running `bridged` daemon, fully captured, with gap analysis.
issue hunting through the running `fleetd` daemon, fully captured, with gap analysis.
Where conversation_test.py exercises a single worker over multiple turns, this drives
the path that only appears under fan-out: the primary spawns N workers, sends each a
@@ -53,11 +53,11 @@ REPO_ROOT = HERE.parent
# hot files this project has been iterating on, so a real issue is plausible to find.
DEFAULT_ASSIGNMENTS = [
{"id": "completion", "probe": "CompletionResolver",
"target": "bridged/src/main/java/dev/ltms/bridged/inject/CompletionResolver.java"},
"target": "fleetd/src/main/java/dev/ltms/fleet/inject/CompletionResolver.java"},
{"id": "worker", "probe": "WorkerService",
"target": "bridged/src/main/java/dev/ltms/bridged/worker/WorkerService.java"},
"target": "fleetd/src/main/java/dev/ltms/fleet/worker/WorkerService.java"},
{"id": "rendezvous", "probe": "Rendezvous",
"target": "bridged/src/main/java/dev/ltms/bridged/msg/Rendezvous.java"},
"target": "fleetd/src/main/java/dev/ltms/fleet/msg/Rendezvous.java"},
]
PROMPT_TMPL = (
+2 -2
View File
@@ -2,8 +2,8 @@
target/
dependency-reduced-pom.xml
# Local runtime config (copy from fleetd.example.yaml). Both names: the live file is still
# bridged.yaml until the cutover, and Fleetd reads either one.
# Local runtime config (copy from fleetd.example.yaml). Both names are ignored: fleetd.yaml is
# the current name, and bridged.yaml is the legacy name Fleetd still falls back to.
fleetd.yaml
bridged.yaml
@@ -1,10 +1,10 @@
# bridged configuration (example). Copy to bridged.yaml and adjust.
# fleetd configuration (example). Copy to fleetd.yaml and adjust.
#
# bridged is the sole gateway between primary/worker Claude sessions and herdr.
# fleetd is the sole gateway between primary/worker Claude sessions and herdr.
# It is NOT a Claude process and must never carry ANTHROPIC_BASE_URL.
# REST + MCP listen address. Keep it on loopback unless you also switch auth.mode to `token`
# below — bridged REFUSES TO START on a non-loopback bind under loopback-trust (see auth).
# below — fleetd REFUSES TO START on a non-loopback bind under loopback-trust (see auth).
bind:
host: 127.0.0.1
port: 8765
@@ -21,7 +21,7 @@ bind:
# bind — the daemon fails fast otherwise, because "unauthenticated ⇒
# primary" on a reachable port would hand spawn/stop/send to anyone.
# tokenEnv → host env var holding the token (never the literal value). Default
# BRIDGED_API_TOKEN. Read only in token mode; empty ⇒ startup fails.
# FLEETD_API_TOKEN. Read only in token mode; empty ⇒ startup fails.
#
# TLS is deliberately NOT terminated in the daemon (CB-501 D3): run a reverse proxy in front and
# let it own certificate lifecycle, e.g.
@@ -29,7 +29,7 @@ bind:
# The broker link gets TLS from its own URI (amqps://…) — see `broker` below.
# auth:
# mode: token
# tokenEnv: BRIDGED_API_TOKEN
# tokenEnv: FLEETD_API_TOKEN
# Optional pinned primary terminal (CB-307). Names the herdr pane the PRIMARY itself runs in:
# a caller whose connection maps to this pane resolves as the primary (no credential needed —
@@ -48,10 +48,10 @@ bind:
# every orchestration call. List each lead's pane here and all of them resolve as leads.
#
# tab → the ONLY field identity depends on (CB-579); the exact label of the tab hosting the lead.
# Label the tab yourself, or let bridged label one it launches — see `fleet.leaders:` below.
# Label the tab yourself, or let fleetd label one it launches — see `fleet.leaders:` below.
# kind/model → descriptive; they document what runs in the pane and are echoed by fleet_whoami
#
# A lead's tab must already carry its label (or be launched by bridged, which labels it) — there is
# A lead's tab must already carry its label (or be launched by fleetd, which labels it) — there is
# no terminal id to paste in and nothing to re-pin when the session restarts: the tab survives, so
# the same label resolves the same lead again on the next scan.
# `fleet_whoami` reports `{"role":"primary","leader":"<name>"}`; role stays "primary" because a lead
@@ -63,7 +63,7 @@ bind:
# Leads are configured under `fleet.leaders:` — see THE FLEET further down.
#
# Two things stop the tab-name convention from becoming a way to claim leadership: the configured
# member spaces are excluded from the scan, so nothing bridged places can land in a matching tab;
# member spaces are excluded from the scan, so nothing fleetd places can land in a matching tab;
# and startup REFUSES a `tabPrefix` that the fleet tabLabel template, or any per-profile `tabLabel`
# override, also matches — so the two namespaces cannot overlap by accident. The label is a NAME,
# never a capability: what a pane may do is decided by the role the daemon resolves for it.
@@ -93,21 +93,20 @@ bind:
# intervalSeconds → how often a tick runs (default 30). ENFORCED floor of 15: the code computes
# Math.max(15, intervalSeconds), so a lower value is silently raised, not
# rejected.
# workingSuspectAfterSeconds, paneProbeIntervalSeconds → accepted and parsed, but NOT YET READ by
# anything — the dormant monitor only consumes intervalSeconds today (CB-573
# shipped ahead of the evidence publishers these two knobs are for). Setting
# them changes nothing right now, and no minimum is enforced on either, because
# nothing reads them to enforce one. They exist so a later build can start
# honouring them without another config-shape change.
# workingSuspectAfterSeconds → age before a BUSY member is suspected of a stall (default 600).
# ENFORCED floor of 300: a lower value is silently raised.
# paneProbeIntervalSeconds → accepted and parsed, but NOT YET READ by anything. Setting it changes
# nothing right now. It exists so a later build can start honouring it without
# another config-shape change.
# notifications.mode → "webhook" flips what fleet_list REPORTS (healthCoverage: "full" instead
# of "detection-only") — it does NOT make bridged send any webhook call; no
# of "detection-only") — it does NOT make fleetd send any webhook call; no
# delivery mechanism is implemented yet. Any other value, or omitting the
# block, reports "detection-only".
# health:
# enabled: true
# intervalSeconds: 30
# workingSuspectAfterSeconds: 600
# paneProbeIntervalSeconds: 60
# intervalSeconds: 30 # floor 15
# workingSuspectAfterSeconds: 600 # floor 300 — how long BUSY with no activity means STALL_SUSPECTED
# paneProbeIntervalSeconds: 60 # parsed, but nothing reads it yet — changing it changes nothing
# notifications:
# mode: disabled
@@ -115,6 +114,18 @@ bind:
# (${HERDR_SOCKET_PATH:-~/.config/herdr/herdr.sock}).
herdrSocket: ~/.config/herdr/herdr.sock
# Optional socket for member panes. Omit this to use herdrSocket for both leads and members.
# memberHerdrSocket: /Users/member/.config/herdr/herdr.sock
# fleetd #213: the login shell the member OS user (memberHerdrSocket above) actually runs. ONLY
# read when memberHerdrSocket is set — fleetd's own $SHELL says nothing about a pane running
# under a different OS user, and there is no channel to ask herdr for that user's shell, so this
# must be told rather than guessed. Absent, blank, or anything not ending in "zsh" is treated the
# same as "not zsh": the memberCredentials.policy: allow-list ZDOTDIR scrub (see worktreeGroup
# below) is skipped in favour of the weaker CB-596 sentinel overlay — a degraded control, never a
# refusal to spawn. When memberHerdrSocket is absent this key is never consulted at all.
# memberLoginShell: /bin/zsh
# How member sessions are spawned. Define one or more named profiles (backends) under
# `profiles`; each key is the profile name (also the ccs profile). A profile says only WHICH
# BACKEND — model, CLI adapter, credentials, cost. It says nothing about what a member spawned on
@@ -124,16 +135,16 @@ herdrSocket: ~/.config/herdr/herdr.sock
# Shared knobs (placement/workspace) can be repeated per profile; they usually match.
# placement: tab → each worker lands in its OWN tab in a dedicated worker space (default).
# Use `pane` for the legacy behaviour (split the focused tab).
# mcpUrl → bridged mounts the bridge MCP (--mcp-config, inline) + reply charter
# mcpUrl → fleetd mounts the bridge MCP (--mcp-config, inline) + reply charter
# (--append-system-prompt) as launch flags; nothing is written to the profile.
# ideMcpUrl → opt-in (CB-634), default off. When set, bridged mounts the IDE Index MCP as a
# ideMcpUrl → opt-in (CB-634), default off. When set, fleetd mounts the IDE Index MCP as a
# second inline server named `intellij`, and adds an IDE charter that pins every
# ide_* call to the member's own worktree. A URL, not a boolean — host and port
# are host-specific. Set it only on a host where the IDE actually runs.
# ideProjectDir → repo-relative module dir the IDE opens and the overlay pins (CB-634). Only read
# when ideMcpUrl is set. This repo's Maven pom lives in `bridged/`, not at the
# when ideMcpUrl is set. This repo's Maven pom lives in `fleetd/`, not at the
# worktree root, so opening the root imports no module and ide_* resolves nothing;
# set this to `bridged`. Omit for a repo whose project is the worktree root.
# set this to `fleetd`. Omit for a repo whose project is the worktree root.
# ideOpenCommand → host command that opens ideProjectDir in the IDE at spawn (CB-634 auto-open).
# Only read when ideMcpUrl is set. `{dir}` is replaced with the absolute module
# dir and the command runs through `/bin/sh -c`, so set env inline if needed —
@@ -151,7 +162,7 @@ herdrSocket: ~/.config/herdr/herdr.sock
# already), so this is instead applied as the model's `limit.context` in the
# generated opencode.json — the member compacts WITHIN this window, not exactly
# at it — and only when this profile's `model:` is in `provider/model` form; if it
# isn't, bridged logs a WARN naming the profile rather than silently doing nothing.
# isn't, fleetd logs a WARN naming the profile rather than silently doing nothing.
# tokenEnv → host env var holding the worker's auth token (value never stored in config);
# omit for a backend that needs no token (e.g. a local ollama).
# cwd → pin this profile's working directory (CB-112). Omit to inherit the primary's
@@ -161,7 +172,11 @@ herdrSocket: ~/.config/herdr/herdr.sock
# skills/MCP/hooks. Omit to leave the worker on the host default.
# parityOverlay → repo-relative paths copied primary→worktree so a worker in a provisioned
# worktree sees the same local config (CB-301-ext). Omit for the default set:
# [.env, .envrc]. (.claude/settings.local.json is NOT in the default — it
# [.env] only (CB-148). .envrc is left out of the default on purpose: it is
# executable shell that direnv runs on every cd, so copying it carries
# behaviour into the worker, not just values, unlike .env. An operator who
# wants it copied can still write parityOverlay: [.env, .envrc] explicitly.
# (.claude/settings.local.json is NOT in the default — it
# pre-approves IDE/tool grants a member must not hold ambiently; CB-525/CB-634.)
#
# Do NOT add .mcp.json (CB-525). A worker's tools are whatever its launcher
@@ -170,7 +185,7 @@ herdrSocket: ~/.config/herdr/herdr.sock
# checkout, so its navigation returned paths OUTSIDE its own worktree: one
# worker made all 59 of its edits in the primary tree while compiling its
# worktree, and every build it ran was of code that did not contain them.
# bridged neutralizes a provisioned worktree's .mcp.json for this reason;
# fleetd neutralizes a provisioned worktree's .mcp.json for this reason;
# listing it here would copy the primary's back over that.
# gitTokenEnv → host env var holding the git-forge API token. When set, its value is injected
# as GITEA_TOKEN so the worker can open its OWN PR at checkpoint (CB-302).
@@ -183,7 +198,7 @@ herdrSocket: ~/.config/herdr/herdr.sock
# refused on a subscription usage limit, rather than a real answer. Opt-in —
# omit and this profile's completion fallback behaves exactly as before.
# Every backend words its refusal differently, so this is config, never a
# vendor string baked into bridged itself.
# vendor string baked into fleetd itself.
# DEFERRED: compiled once into a startup pattern map — editing it needs a
# daemon restart, same as this profile's model/baseUrl/argv.
# credentialId → CB-578 stage B: the credential this profile quarantines WITH when a
@@ -195,18 +210,51 @@ herdrSocket: ~/.config/herdr/herdr.sock
# profile quarantines alone, under its own name, exactly as if the field did
# not exist. Cooldown length is the top-level quarantineCooldownSeconds below.
# HOT: read live at every spawn/exhaustion check — no restart needed.
# errorPattern → fleetd #201 / #227: regex matched against a completion-fallback scrape to
# classify a turn that ended with no fleet_reply as a BACKEND ERROR — a
# credential outage or a provider 5xx — rather than a real answer or a
# usage-limit exhaustion (exhaustedPattern above always wins when a line
# matches both). Opt-in. Omit it and this profile falls back to fleetd's
# built-in legacy pattern `(?i)\bAPI Error\s*:` — classification still
# happens, just without a profile-specific match; every backend words its
# failure differently, so a hardcoded sentence would only ever match one
# of them.
# DEFERRED: compiled once into a startup pattern map, same as exhaustedPattern
# — editing it needs a daemon restart.
# # errorPattern: "503 Service Unavailable" # opt-in: classify a backend outage
#
# What happens once a match fires (BackendOutagePolicy, credentialId-keyed,
# SEPARATE from the CB-578 stage B quarantine above and never merged with it):
# - threshold 2 — TWO DISTINCT TARGETS (never raw events) on the same
# effective credential inside a 60-second window start an "incident" and a
# 60-second cool-off for that credential. One member repeating the same
# classified line twice never cools anything off — a real outage hits
# every target on that credential, so requiring a second, independent
# target loses nothing against the case this guards against, while
# protecting against a heuristic misfire on one flaky member.
# - a fresh error while a credential is already cooling off is ignored
# outright: it neither extends the 60s deadline nor starts a new incident.
# - `fleet_list`/`fleet_profiles` report a cooling credential with
# `coolingOffForSeconds` (never `quarantinedForSeconds`, unless CB-578
# exhaustion quarantine is ALSO independently active for the same
# credential — the two checks can both fire at once). A spawn onto a
# cooling profile is refused with a message naming the credential and
# remaining seconds — "cooling off", never "exhausted", so an operator can
# tell a short transient fault from a spent subscription at a glance.
# - the lead gets ONE nudge per incident (not one per affected target), via
# the same push loop that already delivers ticket/question reminders.
# env → extra environment for this profile's workers, as a literal key/value map
# (CB-511). Use it to give workers a toolchain.
#
# A worker's environment does NOT come from your shell. bridged hands herdr an
# A worker's environment does NOT come from your shell. fleetd hands herdr an
# explicit env map and herdr merges it into ITS OWN process env — so before
# CB-511 a worker inherited whatever PATH the herdr server happened to be
# started with, which on a long-lived herdr can predate your toolchain entirely
# and leave workers unable to run `mvn` or `java` at all.
# bridged now propagates ITS OWN PATH to every worker by default; set `env:`
# fleetd now propagates ITS OWN PATH to every worker by default; set `env:`
# only to override that or add more (JAVA_HOME, …). Since the default is the
# daemon's PATH, make sure the daemon is started with a good one — see the PATH
# lines in deploy/dev.ltms.fleet.plist and deploy/bridged.service.
# lines in deploy/dev.ltms.fleet.plist and deploy/fleetd.service.
#
# Adapter-owned variables always win over `env:`: ANTHROPIC_BASE_URL and the
# rest of the ANTHROPIC_*/CLAUDE_* wiring are applied after it, so an `env:`
@@ -219,10 +267,10 @@ profiles:
baseUrl: http://gx01.gw:8000 # the vLLM host this profile targets (gx00.gw / gx01.gw)
model: coder
placement: tab
workspace: bridged-workers
workspace: fleetd-workers
# tabLabel: an optional per-profile override; the fleet template usually covers it
mcpUrl: http://127.0.0.1:8765/mcp
tokenEnv: BRIDGED_WORKER_TOKEN
tokenEnv: FLEETD_WORKER_TOKEN
argv: ["ccs", "gx10"]
# weight: relative selection weight for automatic placement (weighted, round-robin, and
# fixed's fallback walk). Absent defaults to 1.0. An explicit 0 or negative value means
@@ -258,21 +306,57 @@ profiles:
# GOTCHA 2 — `maxLoad` is the ONLY throttle you have here. There is no metering, no budget
# and no refusal on cost; the cap on live members is the single thing standing between a
# fan-out and your monthly limit. Set it deliberately and keep it small.
#
# GOTCHA 3 (fleetd #176, corrected by fleetd #257) — `maxLoad` counts members, never the lead
# itself. The lead is a live `claude` session on this SAME account (a lead is never moved
# off-subscription, whatever its own profile says), so it already holds one seat before any
# member spawns. If a lead's `fleet.leaders.<name>.profile` names THIS profile — or ANY OTHER
# `subscription: true` profile that shares this one's account (see THE SENTINEL, just below,
# next to `credentialId:`) — `fleet_list` reports that seat count under `leadSeats`; see
# `profile:` under THE FLEET below. `free` itself is NEVER reduced by `leadSeats`: `free` means
# "what the real placement gate (`CompositePeerLauncher#enforceMaxLoad`) will actually grant a
# fresh `fleet_spawn` right now", and that gate only ever compares live members against
# `maxLoad` — it has no notion of the lead's own seat. An earlier cut of this feature
# subtracted `leadSeats` from `free` on the theory it made `free` describe the true ceiling on
# the account, but no backend seat ceiling shared with the lead has ever actually been
# measured, and the subtraction just made `free` disagree with the one thing it is supposed to
# describe — the fleetd #257 fix. `maxLoad: 3` means 3 member slots, full stop; a lead sharing
# the account is a fact you can see in `leadSeats`, not a reason `free` undercounts spawns that
# will, in practice, succeed.
#
# THE SENTINEL (fleetd #176 stage 2, correcting an inert stage 1 fix): every `subscription:
# true` profile that leaves `credentialId` unset shares ONE implicit account-wide credential
# id with every other such profile on this host — because a subscription profile doesn't
# authenticate with a credential of its own, it authenticates as the operator's own Claude
# login, and there is exactly one of those. So on a typical host, `opus` (the lead's profile)
# and `sonnet` (the members' profile) are linked automatically, with NOTHING to set here — that
# is what makes GOTCHA 3 above work without also writing matching `credentialId:` values on
# both. This linkage is not just cosmetic: it is the same key `BackendQuarantine`/cool-off use,
# so a usage-limit hit on `opus` now quarantines `sonnet` too (and vice versa) — correct, since
# they are one Claude account, but worth knowing before you wonder why an unrelated-looking
# profile went quarantined.
#
# WHEN TO OVERRIDE — set explicit, DIFFERENT `credentialId:` values on two `subscription: true`
# profiles only when they are genuinely two separate Claude logins on the same host (a real,
# if unusual, setup). An explicit `credentialId` always wins over the sentinel, so this is the
# one way to keep two subscription profiles from being treated as one account for lead-seat
# counting AND for quarantine/cool-off grouping alike.
# gitTokenEnv: GITEA_TOKEN # opt-in: let this profile's workers open their own PR (CB-302)
# gitHostEnv: GITEA_HOST # defaults to GITEA_HOST; injected only with gitTokenEnv
# exhaustedPattern: "usage limit has been reached" # opt-in: classify a usage-limit refusal (CB-578)
# credentialId: shared-openai # opt-in: quarantine together with every other profile sharing this id (CB-578)
# errorPattern: "503 Service Unavailable" # opt-in: classify a backend outage (fleetd #201/#227) — see the key doc above
# configDir: /Users/me/.ccs/instances/gx10 # CLAUDE_CONFIG_DIR — inherit that profile's skills/MCP
# cwd: /Users/me/src/myrepo # pin the working dir; omit to inherit the primary's
# parityOverlay: [".env", ".envrc"] # the default; never add .mcp.json or .claude/settings.local.json — see above
# parityOverlay: [".env"] # the default; add ".envrc" explicitly if you want it copied too (CB-148) — never add .mcp.json or .claude/settings.local.json — see above
# ideMcpUrl: http://127.0.0.1:29170/index-mcp/streamable-http # opt-in (CB-634): IDE code intelligence, pinned to the worktree
# ideProjectDir: bridged # CB-634: module dir the IDE opens + the overlay pins (this repo's pom is in bridged/)
# ideProjectDir: fleetd # CB-634: module dir the IDE opens + the overlay pins (this repo's pom is in fleetd/)
# ideOpenCommand: env DISPLAY=:10.0 idea {dir} # CB-634 auto-open: opens {dir} in the IDE at spawn; omit to open by hand
# autoCompactWindow: 250000 # opt-in: bound member context; claude-code compacts AT this, opencode within it (model limit.context)
gx11: # a second backend, so `placement: weighted` has a choice
baseUrl: http://gx01.gw:8000 # self-hosted; ccs handles the model + token
placement: tab
workspace: bridged-workers
workspace: fleetd-workers
# tabLabel: an optional per-profile override; the fleet template usually covers it
mcpUrl: http://127.0.0.1:8765/mcp
argv: ["ccs", "gx11"]
@@ -298,7 +382,7 @@ profiles:
# kind: opencode
# model: opencode/north-mini-code-free # `provider/model` selector, injected as `-m`
# placement: tab
# workspace: bridged-workers
# workspace: fleetd-workers
# tabLabel: "opencode: {profile} #{n}"
# mcpUrl: http://127.0.0.1:8765/mcp
# argv: ["opencode"]
@@ -323,7 +407,7 @@ profiles:
# baseUrl: http://127.0.0.1:8000
# model: local-vllm/deepseek-v4-flash
# placement: tab
# workspace: bridged-workers
# workspace: fleetd-workers
# tabLabel: "opencode: {profile} #{n}"
# mcpUrl: http://127.0.0.1:8765/mcp
# argv: ["opencode"]
@@ -358,11 +442,17 @@ placement: weighted
# DEFERRED: baked once into the BackendQuarantine built at startup — a running quarantine keeps
# its original cooldown regardless; a new value only applies to a quarantine that starts after a
# restart. Editing this needs a daemon restart to take effect.
#
# This does NOT govern the fleetd #201 / #227 backend-error cool-off documented under errorPattern
# above — that mechanism is a separate, shorter-lived, NOT-configurable policy (threshold 2 distinct
# targets, 60-second window, 60-second cool-off), on purpose: it exists to survive a brief transient
# fault, not to replace this 30-minute exhaustion quarantine. Do not conflate the two when reading
# fleet_list/fleet_profiles — coolingOffForSeconds and quarantinedForSeconds are independent facts.
# quarantineCooldownSeconds: 1800
# Re-read this file without restarting the daemon (CB-559). Off unless you add this block, so an
# upgraded bridged keeps the old behaviour: the file is read once at boot and never again.
# enabled → turn the watch on. bridged checks the file's modified time on a timer and
# upgraded fleetd keeps the old behaviour: the file is read once at boot and never again.
# enabled → turn the watch on. fleetd checks the file's modified time on a timer and
# reloads when it moves.
# intervalSeconds → how often to check (default 10). One `stat` per tick, so this is cheap.
#
@@ -384,8 +474,10 @@ placement: weighted
# stage B — baked once into the quarantine tracker built at startup), ADDING or
# REMOVING a profile (a new backend needs its own launcher, and launchers are built
# once), AND an existing profile's launch settings — model, baseUrl, argv, env,
# configDir, mcpUrl, tabLabel, exhaustedPattern. The launcher takes a copy of
# `profiles:` at startup and resolves every spawn out of that copy, so those never
# configDir, mcpUrl, tabLabel, exhaustedPattern, errorPattern (fleetd #201 / #227 —
# compiled once into a startup pattern map the same way exhaustedPattern is). The
# launcher takes a copy of `profiles:` at startup and resolves every spawn out of
# that copy, so those never
# reach a launch until you restart. The reload logs them by name rather than
# pretending they applied.
# COLD → cannot change at all: `bind:`, `herdrSocket:`, `broker:` and `auth:`. The socket is
@@ -446,6 +538,23 @@ fleet:
# recognised: give it a `profile:` and the daemon launches the shortfall when fewer than
# `instances` are live. Omit `profile:` and it is recognise-only, as before.
#
# `profile:` has a SECOND job as of fleetd #176, even for a recognise-only lead you never want
# auto-launched: it is also how fleetd learns which account this lead's own session shares. A
# `subscription: true` profile bills the operator's Claude account, and the lead itself is always
# a live `claude` session on that same account — `maxLoad` never counted that seat. If a lead
# entry here names a profile that shares a worker profile's account, `fleet_list` reports the
# lead's live seat(s) on that worker profile under `leadSeats` — informational only, as of fleetd
# #257 it is NEVER subtracted from `free` (see GOTCHA 3, next to `maxLoad:`, in THE WORKERS above,
# for why). "Shares the account" is decided by matching `effectiveCredentialId()`, which (fleetd
# #176 stage 2 — see THE SENTINEL, next to `credentialId:`, in THE WORKERS above) means: an
# explicit, matching `credentialId:` on both, OR — the common case, needing NO extra config — both
# being `subscription: true` with `credentialId` left unset, since those all share one implicit
# account-wide id. A lead on `opus` and workers on `sonnet` link automatically this way; they do
# NOT need the same profile name. Setting `profile:` on an already-running, recognise-only lead is
# safe — the daemon only launches the SHORTFALL below `instances`, so naming a profile here does
# not, by itself, start anything. Omit it and fleetd has no way to derive the sharing — there is
# no other reliable signal on the daemon's side — so that lead's seat never appears in `leadSeats`.
#
# `tab:` (CB-579) is REQUIRED and is the only field identity depends on — the exact label of the
# tab hosting the lead, matched case-insensitively. Label the tab yourself and put that same
# string here, and the pane is recognised on the next rescan. Reopen the tab later, or the session
@@ -477,7 +586,7 @@ fleet:
# workspace: leads # where a launched lead's tab is created (default "leads").
# # MUST NOT be a member workspace — those are excluded from the
# # scan, so a lead placed in one is never found again.
# cwd: /path/to/repo # the launched lead's working directory (default: bridged's own)
# cwd: /path/to/repo # the launched lead's working directory (default: fleetd's own)
# kind: claude # descriptive; reported by fleet_whoami
# gpt-sol-5.6:
# tab: "lead: gpt-sol-5.6"
@@ -506,7 +615,7 @@ guard:
# Member credential policy (CB-596, gitea issue #82). A herdr pane runs a LOGIN shell, and that
# shell re-sources the operator's own secret store — so a spawned member inherits every credential
# the operator's shell holds, not just the ones bridged means to give it. Measured on this host:
# the operator's shell holds, not just the ones fleetd means to give it. Measured on this host:
# 31 credential names, all set, with only ONE (GITEA_ACCESS_TOKEN) blocked before this — and that
# block was a single name hardcoded in HerdrPeerLauncher.java, not driven by this file. This block
# replaces that hardcoded shadow with a config-driven list of names.
@@ -567,20 +676,35 @@ guard:
# every name here NOT also in `allow` is overlaid with a non-secret sentinel value before
# the pane's login shell runs — real protection only for names that shell does not itself
# re-export (see the ROUND-2 CORRECTION note above). Under allow-list: reporting only.
# sshAuthSock → whether SSH_AUTH_SOCK may pass through under allow-list ("allow") or must be
# blanked like any other non-derived name ("block", the default). This is a decision you
# have to make explicitly: SSH_AUTH_SOCK is a handle to YOUR ssh-agent, and a member
# holding it can sign with your keys — it sits in no secret file and looks like no
# credential, which is why it slipped past three earlier tickets (gitea #110). Blocking
# it breaks git over SSH inside members (push/fetch authenticate as you); use HTTPS
# remotes or scoped deploy keys instead of allowing it lightly.
# sshAgentEnv → whether SSH_AUTH_SOCK may pass through under allow-list ("inherit") or is omitted
# from the member environment ("omit", the default). Omitting it only omits the
# inherited ssh-agent path. It discourages automatic use of the operator's agent.
# It does not deny same-user access to that socket. It also does not block SSH keys that
# are readable on disk. Git over SSH may still work from inside a member. Keep the block:
# it is correct and costs nothing, but it is not a control. A member runs as the same OS
# user as the lead. Inside one uid, ordinary Unix permissions provide no meaningful
# confidentiality boundary. A real boundary needs a different OS user or OS-level
# confinement, such as a container or VM. That is the open question in fleetd #184.
#
# Still do not set this to "inherit" casually. SSH_AUTH_SOCK is a live handle to YOUR
# ssh-agent, so a member holding it can sign with EVERY key the agent holds. It sits in
# no secret file and looks like no credential, which is why it slipped past three
# earlier tickets (gitea #110). Blocking it does not contain a member, but allowing it
# hands one a signing capability for no gain — the block costs nothing, so keep it.
#
# Both halves of this are measured, not argued. 2026-08-28: a member with
# SSH_AUTH_SOCK blanked pushed to the forge over SSH successfully, because `ssh -G`
# resolves an IdentityFile outside ~/.ssh that is readable and has no passphrase. An
# earlier version of this comment claimed blocking the socket BREAKS git over SSH. It
# does not. That claim came from looking only in ~/.ssh, which holds nothing but four
# `Include` lines — looking in one place and concluding about the whole host.
#
# HOT-RELOADABLE the same way `fleet:` is (CB-559): read fresh on every spawn, so editing this list
# and reloading config (or restarting) changes what the NEXT spawn inherits; already-running members
# are unaffected either way.
# memberCredentials:
# policy: deny-by-default # or "deny-list", or "allow-list" (CB-633) — see above
# sshAuthSock: block # allow-list only; see the sshAuthSock note above
# sshAgentEnv: omit # allow-list only; see the sshAgentEnv note above
# allow:
# - AI_GATEWAY_TOKEN # named in a profile's tokenEnv (local/gx) — a member reaching the
# # gateway is by design, not a leak
@@ -632,6 +756,24 @@ guard:
# to a sibling directory of the repo root.
# worktreeRoot: /Users/me/src/.bridged-worktrees
# Worktree group sharing (fleetd #185 stage 3). OPTIONAL, off by default. Names an OS group
# that a provisioned worktree's repo is made group-writable for (git config
# core.sharedRepository group, plus a one-time chgrp/chmod/setgid fix-up), so a member spawned
# under a DIFFERENT OS user (see memberHerdrSocket) can write its own worktree, its
# per-worktree git metadata, and its own commit objects — without it, every file GitWorktrees
# creates is owned by fleetd's own uid and unwritable by another user.
# CAUTION: this isolates credentials, not the repository — a member in the group can still
# write the operator's git objects and refs in the shared repo. The operator running fleetd
# must already be a member of the named group, or every provisioning spawn fails loudly.
#
# fleetd #213: this is also the ONE group the memberCredentials.policy: allow-list ZDOTDIR scrub
# reuses when memberHerdrSocket is set — deliberately not a second config key. Under
# memberHerdrSocket, the scrub directory is generated under worktreeRoot (never java.io.tmpdir,
# which the member OS user cannot reach) and shared read-only with this group. If worktreeGroup
# is unset while memberHerdrSocket is set, the scrub cannot be guaranteed reachable by the member,
# so fleetd falls back to the weaker CB-596 sentinel overlay instead (a WARN names the gap).
# worktreeGroup: fleet-workers
# Session lifecycle limits (CB-303). All knobs are opt-in; omit or set to null to keep
# the feature disabled. By default the daemon never reaps, caps, or drains sessions.
# idleTtlSeconds → reap READY/DONE sessions idle longer than this (never BUSY/SPAWNING)
@@ -667,6 +809,23 @@ guard:
# uriEnv: LAVINMQ_URI
# prefetch: 32
# Shared cross-host LEADER coordination broker. OMIT this block to leave lead-to-lead messaging
# off entirely (config-only in this ticket — nothing here wires it into a live LeadMailbox yet).
# This is a SEPARATE AMQP vhost from `broker:` above: member/worker inboxes always stay on the
# per-fleet `broker:` vhost, and this vhost carries only leader-to-leader traffic, so two fleets
# whose members must never see each other can still share one coordination vhost for their leads.
# uriEnv → name of a host env var holding the coordination AMQP URI, same convention as
# broker.uriEnv (keeps the credential out of fleetd.yaml). Wins over `uri` when set.
# selfId → this daemon's own lead coord-id — the name its mailbox is owned under
# (lead.<selfId>.inbox), e.g. "mac-opus" or "fleet01-lead". Must be globally unique
# across every daemon sharing this vhost.
# prefetch → consumer basicQos, capping how many unacked messages the mailbox holds in-heap.
# Default 32 when omitted.
# coordinator:
# uriEnv: LEAD_COORD_URI
# selfId: mac-opus
# prefetch: 32
# Active push-to-primary (CB-307 Stage 3). When a worker reply lands with no open fleet_send,
# the ReplyPushLoop injects a *drain nudge* (never the payload) into the primary's own herdr
# pane — status-gated (only when injectable, never mid-turn) and bounded. Ack = drain: the loop
@@ -678,7 +837,7 @@ guard:
#
# REQUIRED (CB-522) if the primary itself runs inside a herdr pane. Caller
# identity resolves a loopback PID to its herdr pane, and PaneLocator scans
# EVERY pane — not just bridged-spawned ones — so such a primary is otherwise
# EVERY pane — not just fleetd-spawned ones — so such a primary is otherwise
# classified as a WORKER and refused SPAWN/SEND/STOP. That failure is
# self-locking: the learned terminal is populated by the very orchestration
# calls being refused, so only this pinned value can break the cycle. Read the
+33 -6
View File
@@ -28,6 +28,8 @@
<testcontainers.version>1.20.4</testcontainers.version>
<commons-compress.version>1.27.1</commons-compress.version>
<commons-lang3.version>3.18.0</commons-lang3.version>
<sqlite-jdbc.version>3.53.4.0</sqlite-jdbc.version>
<archunit.version>1.5.0</archunit.version>
</properties>
<!--
@@ -44,6 +46,12 @@
3.0-rc5; bumping Jackson 3 to the patched 3.2.x breaks the SDK (annotation mismatch).
Only the loopback /mcp endpoint parses this JSON, from trusted local Claude clients.
The 11.0.23 -> 11.0.25 bump did clear jetty CVE-2024-8184 (5.9) and CVE-2024-6763.
fleetd #206: org.xerial:sqlite-jdbc 3.53.4.0 (added for OpenCodeSessionDiscovery) — the
only known advisory against this artifact is CVE-2023-32697 (RCE via an attacker-controlled
JDBC URL), fixed in 3.41.2.2; 3.53.4.0 is well past that fix and OSV.dev reports no open
advisory against it. Checked via the OSV.dev API (no Mend.io/JetBrains IDE MCP mount
available from this worktree) on 2026-08-31.
-->
<!-- Force the latest patched Jetty 11.x across all Javalin-pulled Jetty modules (no version
@@ -106,7 +114,7 @@
</dependency>
<!-- MCP server: the SERVER face. Streamable-HTTP servlet mounted on Javalin's Jetty at
/mcp, exposing bridge_send/bridge_reply/bridge_status as thin adapters over REST. -->
/mcp, exposing fleet_send/fleet_reply/fleet_status as thin adapters over REST. -->
<dependency>
<groupId>io.modelcontextprotocol.sdk</groupId>
<artifactId>mcp</artifactId>
@@ -123,6 +131,17 @@
<version>${amqp.version}</version>
</dependency>
<!-- fleetd #206: opencode moved its session store from a JSON tree to SQLite
(opencode.db). This is the JDBC driver OpenCodeSessionDiscovery uses to read it
read-only. Ships bundled native libraries (linux/mac/windows, several archs), so it
is a heavier jar than most deps here — see the pom's dependency-security note below
for the size/CVE tradeoff actually measured. -->
<dependency>
<groupId>org.xerial</groupId>
<artifactId>sqlite-jdbc</artifactId>
<version>${sqlite-jdbc.version}</version>
</dependency>
<!-- Logging -->
<dependency>
<groupId>org.slf4j</groupId>
@@ -158,13 +177,21 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<!-- fleetd #131: package-boundary and cycle enforcement (PackageCyclesTest). -->
<dependency>
<groupId>com.tngtech.archunit</groupId>
<artifactId>archunit-junit5</artifactId>
<version>${archunit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<!-- CB-632: stays 'bridged' until the cutover renames the module dir and the launchd
plist together. The installed plist names bridged/target/bridged.jar and
KeepAlive is armed, so renaming the jar alone strands a restart. -->
<finalName>bridged</finalName>
<!-- CB-634: the cutover renamed the module dir (bridged/ -> fleetd/), the jar, and the
launchd plist together. The installed plist names fleetd/target/fleetd.jar and
KeepAlive is armed, so this name, the plist, and the wrapper must move as one. -->
<finalName>fleetd</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -206,7 +233,7 @@
</configuration>
</plugin>
<!-- Runnable fat jar: java -jar target/bridged.jar -->
<!-- Runnable fat jar: java -jar target/fleetd.jar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
@@ -7,11 +7,14 @@ import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.herdr.HerdrRouter;
import dev.ltms.fleet.herdr.LeadTabScanner;
import dev.ltms.fleet.lead.LeadLauncher;
import dev.ltms.fleet.herdr.PaneLocator;
import dev.ltms.fleet.herdr.UnixSocketHerdrClient;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.inject.BackendErrorPatternLookup;
import dev.ltms.fleet.inject.BackendErrorSink;
import dev.ltms.fleet.inject.CompletionResolver;
import dev.ltms.fleet.inject.ExhaustedPatternLookup;
import dev.ltms.fleet.inject.ExhaustionSink;
@@ -31,6 +34,9 @@ import dev.ltms.fleet.mcp.LsofPeerPidLookup;
import dev.ltms.fleet.mcp.LsofProcessCwdLookup;
import dev.ltms.fleet.msg.AmqpReplyInbox;
import dev.ltms.fleet.msg.InMemoryReplyInbox;
import dev.ltms.fleet.msg.LeadChannel;
import dev.ltms.fleet.msg.LeadCoordLoop;
import dev.ltms.fleet.msg.LeadMailbox;
import dev.ltms.fleet.msg.MessageService;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.ReplyInbox;
@@ -45,7 +51,9 @@ import dev.ltms.fleet.session.SessionReaper;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import dev.ltms.fleet.member.CompositePeerLauncher;
import dev.ltms.fleet.member.HerdrPeerLauncher;
import dev.ltms.fleet.member.MemberCredentialPolicyView;
import dev.ltms.fleet.member.OpenCodeLauncher;
import dev.ltms.fleet.placement.BackendOutagePolicy;
import dev.ltms.fleet.placement.BackendQuarantine;
import io.javalin.Javalin;
import org.slf4j.Logger;
@@ -57,21 +65,21 @@ import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.function.Supplier;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
/**
* {@code bridged} entry point. Wires the real herdr socket client to the REST app and
* {@code fleetd} entry point. Wires the real herdr socket client to the REST app and
* starts listening. Before anything else it asserts its own environment is clean —
* {@code bridged} is not a Claude process and must never carry a base_url.
* {@code fleetd} is not a Claude process and must never carry a base_url.
*/
public final class Fleetd {
@@ -79,12 +87,21 @@ public final class Fleetd {
/** CB-504: how long to wait at startup for herdr's socket before serving degraded. */
private static final long HERDR_WAIT_SECONDS = 30;
/**
* CB-637: how often the lead coordination loop looks for peer messages. A few seconds — slow
* enough that an idle fleet is not polling a broker in a tight loop, fast enough that a peer
* lead's message is not left sitting once the local lead reaches a turn boundary. The mailbox
* pushes into the loop's held set on its own consumer thread, so this interval bounds only the
* pane delivery, never the receive.
*/
private static final long LEAD_COORD_INTERVAL_MS = 3_000L;
private static final long HERDR_WAIT_POLL_MILLIS = 500;
/**
* CB-632: prefer {@code fleetd.yaml} in {@code dir}; fall back to {@code bridged.yaml} when
* the new name is not there. The operator's live file is still named {@code bridged.yaml},
* so the old name keeps working until that file moves.
* CB-632/CB-634: prefer {@code fleetd.yaml} in {@code dir}; fall back to the legacy
* {@code bridged.yaml} when the new name is not there. The product renamed to {@code fleetd},
* but a deployment whose local config file is still {@code bridged.yaml} keeps working until
* that file is renamed.
*/
static Path chooseDefaultConfigFile(Path dir) {
Path fleetd = dir.resolve("fleetd.yaml");
@@ -96,14 +113,15 @@ public final class Fleetd {
static void main(String[] args) {
Path configPath = args.length > 0 ? Path.of(args[0]) : chooseDefaultConfigFile(Path.of(""));
// CB-632: the config file is being renamed bridged.yaml -> fleetd.yaml. Name the file we
// actually loaded, whichever of the two names it carries.
// The config file was renamed bridged.yaml -> fleetd.yaml. Name the file we actually
// loaded, whichever of the two names it carries.
log.info("Using configuration file {}", configPath);
FleetConfig cfg = FleetConfig.load(configPath);
// CB-594: report which secret env vars the config actually needs, by name, before anything
// else can fail on a silently-empty one. A daemon started without a login shell (launchd)
// boots fine either way — this is the only thing that says so out loud.
reportRequiredSecrets(cfg);
reportMemberTrustModel(cfg);
// CB-596: an absent (or empty) memberCredentials: block blocks NOTHING — no credential
// name is hardcoded any more to fall back on. Say so loudly, the same way a missing
// secret is reported above, so upgrading past this commit never silently drops CB-592's
@@ -115,7 +133,7 @@ public final class Fleetd {
// contract; adding a reader here does not make a key reloadable by itself.
ConfigRef config = new ConfigRef(configPath, cfg);
// The primary/host env that launched bridged must not be tainted.
// The primary/host env that launched fleetd must not be tainted.
SubscriptionGuard guard = new SubscriptionGuard(cfg.guard().hostSet());
guard.assertPrimaryClean(System.getenv());
@@ -138,9 +156,12 @@ public final class Fleetd {
: UnixSocketHerdrClient.defaultSocketPath();
UnixSocketHerdrClient herdr = UnixSocketHerdrClient.connect(socket, new com.fasterxml.jackson.databind.ObjectMapper());
AgentControl agents = new AgentControl(herdr);
WorkspaceControl spaces = new WorkspaceControl(herdr);
UnixSocketHerdrClient memberHerdr = cfg.memberHerdrSocket() != null && !cfg.memberHerdrSocket().isBlank()
? UnixSocketHerdrClient.connect(Path.of(cfg.memberHerdrSocket()), new com.fasterxml.jackson.databind.ObjectMapper())
: herdr;
AtomicReference<Supplier<Map<String, String>>> leadsRef = new AtomicReference<>(Map::of);
HerdrRouter router = new HerdrRouter(herdr, memberHerdr,
target -> leadsRef.get().get().containsKey(target));
// CB-402: one adapter per configured peer kind, fronted by a composite router. A profile's
// `kind:` selects its adapter — claude-code (the default) and opencode partition the profile
// set — and the composite dispatches each SPI call to the adapter that owns the profile/pane.
@@ -154,22 +175,34 @@ public final class Fleetd {
}
});
List<HerdrPeerLauncher> adapters = new ArrayList<>();
// fleetd #175: the daemon's real ExhaustionSink can only be built once `sessions` exists
// (below), but `sessions` needs `workers`, which needs the adapters built right here — a
// genuine cycle. Break it exactly like liveCountRef below: a forwarding sink built now,
// pointed at the real one once it exists.
AtomicReference<ExhaustionSink> exhaustionSinkRef = new AtomicReference<>(ExhaustionSink.none());
// fleetd #234, round 4: routed through the shared ExhaustionSink.forwardingTo factory
// rather than written inline here — not because a lambda at this call site is unsafe
// anymore (it is not: the 3-arg overload is now the interface's single abstract method, so
// there is no 2-arg overload left for any lambda to silently bind to instead), but so a
// test can call the exact same object this line builds, instead of asserting a copy of its
// shape (round 3's lesson).
ExhaustionSink forwardingExhaustionSink = ExhaustionSink.forwardingTo(exhaustionSinkRef::get);
// The claude-code adapter is the always-present default; keep it even with no profiles (so a
// bridge configured with no workers, or opencode-only, still has a well-defined base adapter)
// unless opencode is the only kind configured.
if (!claudeProfiles.isEmpty() || opencodeProfiles.isEmpty()) {
adapters.add(new ClaudeCodeLauncher(agents, spaces, guard,
adapters.add(new ClaudeCodeLauncher(router.memberAgents(), router.memberSpaces(), guard,
claudeProfiles, cfg.effectiveDefaultProfile(), System::getenv,
cfg.spawnReadyTimeoutMs(), cfg.spawnReadyPollMs(),
() -> config.get().fleet(),
() -> config.get().memberCredentials()));
() -> config.get().memberCredentials(), null, config::get));
}
if (!opencodeProfiles.isEmpty()) {
adapters.add(new OpenCodeLauncher(agents, spaces,
adapters.add(new OpenCodeLauncher(router.memberAgents(), router.memberSpaces(),
opencodeProfiles, cfg.effectiveDefaultProfile(), System::getenv,
cfg.spawnReadyTimeoutMs(), cfg.spawnReadyPollMs(),
() -> config.get().fleet(),
() -> config.get().memberCredentials()));
() -> config.get().memberCredentials(), config::get, forwardingExhaustionSink));
}
AtomicReference<Function<String, Integer>> liveCountRef = new AtomicReference<>(_ -> 0);
// CB-578 stage B: one quarantine tracker for the whole daemon, shared between the launcher
@@ -178,13 +211,19 @@ public final class Fleetd {
// here, at startup, and a config reload only changes it for a daemon restart.
BackendQuarantine quarantine = new BackendQuarantine(System::nanoTime,
TimeUnit.SECONDS.toNanos(cfg.quarantineCooldownSeconds()));
// fleetd #201 Unit 5: one outage-cool-off tracker for the whole daemon, shared between the
// launcher (checked at spawn, like `quarantine` above) and the backend-error sink wired in
// below (written on a classified backend error). A SEPARATE, shorter-lived mechanism from
// `quarantine` — see BackendOutagePolicy's class doc — never merged with it.
BackendOutagePolicy outagePolicy = new BackendOutagePolicy(System::nanoTime);
PeerLauncher workers = new CompositePeerLauncher(
adapters,
cfg.effectiveDefaultProfile(),
config,
profileName -> liveCountRef.get().apply(profileName),
quarantine);
// CB-504: under supervision (launchd/systemd) bridged can start before herdr's socket
quarantine,
outagePolicy);
// CB-504: under supervision (launchd/systemd) fleetd can start before herdr's socket
// exists. The client itself is lazy — it connects per call — but the orphan reap below is
// the first thing that actually talks to herdr, so without this wait a boot-order race
// would crash the daemon into a restart loop. Wait, then degrade rather than die: serving
@@ -209,11 +248,9 @@ public final class Fleetd {
contextCap = cfg.lifecycle().contextCap();
}
boolean clearAfterTurn = cfg.lifecycle() != null && cfg.lifecycle().clearAfterTurn();
SessionManager sessions = new SessionManager(workers, new GitWorktrees(cfg.worktreeRoot()),
SessionManager sessions = new SessionManager(workers, new GitWorktrees(cfg.worktreeRoot(), cfg.worktreeGroup()),
System::nanoTime, contextCap, clearAfterTurn);
liveCountRef.set(profileName -> (int) sessions.roster().stream()
.filter(s -> profileName.equals(s.profile()))
.count());
liveCountRef.set(profileName -> liveSessionCount(sessions.roster(), profileName));
// CB-303 part 1: idle-ttl reaper — only when configured, defaults to disabled.
final SessionReaper reaper;
@@ -243,34 +280,39 @@ public final class Fleetd {
final Supplier<Map<String, String>> leads;
var leaders = cfg.fleet().leaders();
if (!leaders.isEmpty()) {
Set<String> memberSpaces = cfg.profiles().values().stream()
.map(FleetConfig.Profile::workspace)
.filter(Objects::nonNull)
.collect(Collectors.toSet());
Map<String, String> tabToName = new LinkedHashMap<>();
leaders.forEach((name, leader) -> {
if (leader != null && leader.tab() != null && !leader.tab().isBlank()) {
tabToName.put(leader.tab(), name);
}
});
// The lead and the members now share ONE workspace (the operator asked for a single
// "session" with many tabs), so no workspace can be excluded — the lead lives in the
// members' space by design. A lead is told from a member by its exact tab label alone:
// a lead carries its configured `tab`, a member its `worker: {profile} #{n}` template,
// and the two never collide. (The scanner still supports an exclusion set for a split
// layout; the fleet's policy here is simply not to use one.)
//
// One shared rescan cadence: still taken from the first entry, as before — it is an
// operational cadence, not identity, so there is no correctness reason to give every
// lead its own scanner.
int scanIntervalSeconds = leaders.values().iterator().next().scanIntervalSeconds();
leads = new LeadTabScanner(herdr, tabToName, memberSpaces,
// This must use the lead daemon: scanning member tabs would demote the lead to a worker.
leads = new LeadTabScanner(herdr, tabToName, Set.of(),
TimeUnit.SECONDS.toNanos(scanIntervalSeconds), System::nanoTime);
log.info("lead scan: tabs {} host a lead (rescan every {}s, member spaces {} excluded)",
tabToName.keySet(), scanIntervalSeconds, memberSpaces);
log.info("lead scan: tabs {} host a lead (rescan every {}s, shared fleet space)",
tabToName.keySet(), scanIntervalSeconds);
} else {
leads = () -> leadTerminals;
}
leadsRef.set(leads);
// CB-558: start any declared lead that is not already running. After the scanner is built,
// because both read the same tab labels and the ordering makes that dependency visible; and
// only when herdr answered, because the launcher's whole safety property is that it can
// count live leads first — it must never guess and risk a second orchestrator.
if (herdrUp && !leaders.isEmpty()) {
int launched = new LeadLauncher(agents, spaces, cfg).ensureLeads();
int launched = new LeadLauncher(router.leadAgents(), router.leadSpaces(), cfg).ensureLeads();
if (launched > 0) {
log.info("lead auto-launch: {} lead(s) started", launched);
}
@@ -309,24 +351,100 @@ public final class Fleetd {
.map(session -> exhaustedPatternsByProfile.get(session.profile()))
.orElse(null);
log.info("backend-exhausted classification (CB-578 stage A): {}",
CompletionResolver.coverage(cfg.profiles().keySet(), exhaustedPatternsByProfile.keySet()));
CompletionResolver.coverage("exhaustedPattern", cfg.profiles().keySet(),
exhaustedPatternsByProfile.keySet()));
// fleetd #201 Unit 5: classify a completion-fallback scrape that matches a profile's
// configured backend-error refusal (a credential outage, a provider 5xx) as a backend error
// rather than handing it back as a real answer. Compiled once at startup, keyed by profile
// name, mirroring exhaustedPatternsByProfile above — a profile with no configured
// errorPattern is simply absent here, so CompletionResolver falls back to its built-in
// narrow {@code (?i)\bAPI Error\s*:} compatibility pattern for that profile's targets
// (BackendErrorPatternLookup#legacy's contract — see backendErrorPatterns below).
Map<String, Pattern> errorPatternsByProfile = new LinkedHashMap<>();
cfg.profiles().forEach((name, profile) -> {
if (profile.hasErrorPattern()) {
errorPatternsByProfile.put(name, Pattern.compile(profile.errorPattern()));
}
});
// fleetd #248: extracted to a static factory (see backendErrorPatternLookup below) so a
// test can prove main() actually PASSES this into CompletionResolver, not only that the
// lookup itself behaves correctly — the exact gap fleetd #248 exists to close.
BackendErrorPatternLookup backendErrorPatterns = backendErrorPatternLookup(sessions::roster,
errorPatternsByProfile);
log.info("backend-error classification (fleetd #201 Unit 5): {}",
CompletionResolver.coverage("errorPattern", cfg.profiles().keySet(),
errorPatternsByProfile.keySet()));
// CB-578 stage B: on a classification that actually wins, quarantine the exhausted profile's
// CREDENTIAL — not the profile name — so a profile sharing that credential (e.g. two models
// on one OpenAI account) is refused too, not just the one that happened to report it. Reads
// the profile config live off `config`, so a credentialId edit is hot: no restart needed.
ExhaustionSink exhaustionSink = (target, reason) -> sessions.roster().stream()
.filter(session -> target.equals(session.terminalId()))
.findFirst()
.map(MemberSession::profile)
.map(profileName -> config.get().profiles().get(profileName))
.ifPresent(profile -> {
String credentialId = profile.effectiveCredentialId();
quarantine.quarantine(credentialId);
log.warn("credential '{}' quarantined for {}s (profile '{}' classified "
+ "BACKEND_EXHAUSTED): {}", credentialId,
cfg.quarantineCooldownSeconds(), profile.profile(), reason);
});
CompletionResolver completion = new CompletionResolver(agents, rendezvous, exhaustedPatterns, exhaustionSink);
//
// fleetd #175: this sink is now also the quarantine target for OpenCodeLauncher's
// model-mismatch check — it needs nothing profile-specific from the caller beyond `target`
// (a herdr terminal id) and `reason`, so reusing it here is exactly "the existing
// ExhaustionSink path", not a new mechanism.
//
// fleetd #234: that check fires from SessionAwareHandle.agentSessionId(), which runs during
// SessionManager.acquire() BEFORE this session is registered in sessions.roster() — so the
// roster-only lookup below used to find nothing, .ifPresent silently no-op'd, and the
// ERROR the check had just logged ("quarantining this profile's credential") was a lie:
// nothing was quarantined, and nothing said so. Two changes: (1) OpenCodeLauncher now
// passes its OWN profile name via ExhaustionSink's 3-arg overload — it already has the
// FleetConfig.Profile in hand and does not need the roster at all — used here as a
// fallback whenever the roster lookup misses; (2) if a profile still cannot be resolved
// (neither the roster nor the hint names a configured one), this logs loudly at ERROR
// instead of silently doing nothing — a control that cannot act must say so.
// fleetd #234, round 4: the 3-arg overload is now ExhaustionSink's single abstract method,
// so this is safely a lambda — there is no separate 2-arg overload left for it to bind to
// instead and silently drop profileHint (that was rounds 1-3's whole hazard).
ExhaustionSink exhaustionSink = (target, reason, profileHint) -> {
String profileName = sessions.roster().stream()
.filter(session -> target.equals(session.terminalId()))
.findFirst()
.map(MemberSession::profile)
.orElse(profileHint);
FleetConfig.Profile profile = profileName == null ? null : config.get().profiles().get(profileName);
if (profile == null) {
log.error("quarantine requested for target '{}' ({}) but no profile could be "
+ "resolved — the target is not (yet) in the roster, and {} — "
+ "credential NOT quarantined (fleetd #234)",
target, reason,
profileHint == null ? "no profile hint was given"
: "the hinted profile '" + profileHint + "' is not configured");
return;
}
String credentialId = profile.effectiveCredentialId();
quarantine.quarantine(credentialId);
log.warn("credential '{}' quarantined for {}s (profile '{}'): {}", credentialId,
cfg.quarantineCooldownSeconds(), profile.profile(), reason);
};
// fleetd #175: point the forwarding sink handed to OpenCodeLauncher above at the real one,
// now that `sessions` exists to resolve target -> session -> profile.
exhaustionSinkRef.set(exhaustionSink);
// fleetd #201 Unit 5: the production BackendErrorSink needs `pushLoop` (built further below,
// after `sessions`) to tell a lead about an incident or an unmapped target — the same
// construction-order cycle `exhaustionSinkRef` breaks above, broken the same way: a mutable
// holder set once `pushLoop` exists, read lazily from inside the lambda built here.
AtomicReference<ReplyPushLoop> pushLoopRef = new AtomicReference<>();
// fleetd #248: extracted to a static factory (see backendErrorSink below), public rather
// than package-private like the other two factories here, so
// dev.ltms.fleet.inject.BackendOutageFlowTest can exercise the REAL production sink
// directly instead of a hand-mirrored copy of this lambda — that copy was precisely the
// gap fleetd #248 exists to close (see that test's class doc for the history).
BackendErrorSink backendErrorSink = backendErrorSink(sessions, () -> config.get().profiles(),
outagePolicy, pushLoopRef::get);
AgentControl agents = router.memberAgents();
// Both fleetd#201 Unit 5 (backend-error patterns + sink) and fleetd#241 (the worktree/branch
// lookup the fallback report names) land on this one call. The full constructor takes both,
// so neither feature is dropped; nowNanos must be passed explicitly to reach it.
//
// fleetd #248: every argument built specifically for this call (backendErrorPatterns and
// backendErrorSink above, and the worktree/branch lookup right here) now comes from a
// static factory tested on its own; FleetdCompletionResolverWiringTest source-asserts that
// THIS call actually passes them, which is the coverage that was missing before.
CompletionResolver completion = new CompletionResolver(agents, rendezvous, exhaustedPatterns,
exhaustionSink, backendErrorPatterns, backendErrorSink, System::nanoTime,
worktreeBranchLookup(sessions::roster));
// CB-113: deliver only to an available worker (its MCP is connected), never its boot window.
// CB-301: the manager's presence bridge records availability and drives SPAWNING → READY.
MemberPresence presence = sessions.asPresence();
@@ -366,15 +484,22 @@ public final class Fleetd {
sessions.onTurnFailed(target);
}
};
Injector injector = new Injector(agents, turnListener, deliverableTo(presence, leads),
Predicate<String> deliverable = deliverableTo(presence, leads);
Injector injector = new Injector(router, turnListener, deliverable,
presence::forget);
StatusPoller poller = new StatusPoller(agents, injector, Injector.POLL_INTERVAL_MILLIS);
StatusPoller poller = new StatusPoller(router, injector, Injector.POLL_INTERVAL_MILLIS);
poller.start();
// CB-307: reply inbox. A broker: block selects the AMQP-backed durable adapter; absent (or
// unusable), bridged stays soft-state on the in-memory inbox. The AMQP inbox owns a broker
// unusable), fleetd stays soft-state on the in-memory inbox. The AMQP inbox owns a broker
// connection, so keep the reference to close it in the ordered shutdown hook.
final ReplyInbox replyInbox = selectReplyInbox(cfg.broker(), System.getenv(), AmqpReplyInbox::open);
// CB-637: this daemon's lead-to-lead mailbox on the SHARED coordination vhost — a separate
// broker from the reply inbox by design (see FleetConfig.Coordinator). Absent a coordinator:
// block this is null and every lead path below is simply not wired, which is exactly the
// behaviour before this ticket. It owns a broker connection, so keep the reference for the
// ordered shutdown hook.
final LeadMailbox leadMailbox = openLeadMailbox(cfg.coordinator(), System.getenv(), LeadMailbox::open);
// CB-307: learn the primary's terminal from orchestration tool calls (or pin from config).
// The pin also feeds CallerResolver below: a primary running inside a herdr pane would
// otherwise resolve as a worker and be refused every orchestration tool.
@@ -399,8 +524,11 @@ public final class Fleetd {
// are counted at their single funnel rather than at each of the two caller-facing surfaces.
// CB-512: the push loop takes it too, so nudge outcomes (delivered|exhausted) are counted.
Metrics metrics = FleetMetrics.create(sessions, replyInbox);
var pushLoop = new ReplyPushLoop(primaryRegistry, agents, replyInbox,
var pushLoop = new ReplyPushLoop(primaryRegistry, router.leadAgents(), replyInbox,
pushScheduler, maxReminders, backoffMs, metrics);
// fleetd #201 Unit 5: point the forwarding holder captured by the backendErrorSink lambda
// above at the real push loop, now that it exists.
pushLoopRef.set(pushLoop);
// CB-551: idle-lead heartbeat. Opt-in; absent `leadHeartbeat:` this is never constructed, so
// an upgraded daemon cannot silently start spending subscription on nudging an idle lead.
// It has its own single-thread scheduler and holds its own scheduler shutdown via close().
@@ -409,7 +537,7 @@ public final class Fleetd {
Thread.ofVirtual().name("bridge-heartbeat-").unstarted(r));
if (cfg.leadHeartbeat() != null) {
var hb = cfg.leadHeartbeat();
heartbeat = new LeadHeartbeatLoop(primaryRegistry, agents, replyInbox, sessions::roster,
heartbeat = new LeadHeartbeatLoop(primaryRegistry, router.leadAgents(), replyInbox, sessions::roster,
pushLoop, heartbeatScheduler, System::nanoTime,
TimeUnit.SECONDS.toNanos(hb.idleAfterSeconds()), hb.backoffMs(), hb.quietNudgeCap(),
metrics);
@@ -418,7 +546,7 @@ public final class Fleetd {
heartbeat = null;
heartbeatScheduler.shutdownNow();
}
MessageService messages = new MessageService(agents, injector, rendezvous, replyInbox,
MessageService messages = new MessageService(router, injector, rendezvous, replyInbox,
pushLoop, metrics);
// Health is a slow whole-fleet observer. Keep it separate from the 250ms delivery poller.
@@ -429,7 +557,8 @@ public final class Fleetd {
// CB-580: a member found GONE/NEVER_READY must fail whatever ticket is waiting on it,
// through the same idempotent target-wide operation CB-516 already uses on release.
healthMonitor = new FleetHealthMonitor(agents, sessions::roster, messages, healthScheduler,
System::nanoTime, cfg.health().intervalOrDefault(), messages::abandon);
System::nanoTime, cfg.health().intervalOrDefault(),
cfg.health().workingSuspectAfterOrDefault(), messages::abandon);
String coverage = FleetHealthMonitor.coverage(true,
cfg.health().notifications() != null && cfg.health().notifications().configured());
if ("detection-only".equals(coverage)) {
@@ -462,15 +591,22 @@ public final class Fleetd {
if (detail.agentSessionId() != null) {
reason += " agentSessionId=" + detail.agentSessionId();
}
messages.abandon(detail.terminalId(), reason);
// fleetd #275: this is an explicit teardown (fleet_stop, or the idle reaper) — the
// worker's pane is being stopped right now, so an open fleet_ask has no turn left to
// resume into. Sweep it too, unlike FleetHealthMonitor's health-classification call
// (see MessageService.abandon's javadoc for why those two must differ).
messages.abandon(detail.terminalId(), reason, true);
replyInbox.release(detail.terminalId());
primaryRegistry.forgetDelegation(detail.terminalId()); // CB-532: don't leak the lead binding
});
// MCP server face (CB-105): fleet_send/fleet_reply/fleet_status, mounted at /mcp.
// Caller identity is resolved from the connection (peer PID → herdr pane), not arguments.
// CB-185: a caller's pane can live on either daemon (a lead's on the lead daemon, a
// member's on the member daemon) — search both, lead first. Collapses to one scan when
// memberHerdrSocket is unset (herdr == memberHerdr).
ConnectionIdentity identity = new ConnectionIdentity(
new PaneLocator(herdr), new LsofPeerPidLookup(), new LsofProcessCwdLookup());
new PaneLocator(herdr, memberHerdr), new LsofPeerPidLookup(), new LsofProcessCwdLookup());
// CB-501: one resolver behind both entry paths. Worker identity still comes from the
// connection and is never token-gated, so enabling token mode cannot lock the fleet out.
@@ -479,7 +615,7 @@ public final class Fleetd {
String token = System.getenv(cfg.auth().tokenEnv());
if (token == null || token.isBlank()) {
throw new IllegalStateException("auth.mode=token but env var " + cfg.auth().tokenEnv()
+ " is unset or empty — export it before starting bridged");
+ " is unset or empty — export it before starting fleetd");
}
callers = CallerResolver.withLeadsAndMembers(identity, true, token, leads, members);
log.info("auth: token mode (bearer required for non-worker callers, env {})",
@@ -489,6 +625,19 @@ public final class Fleetd {
log.info("auth: loopback-trust (any loopback non-worker caller is the primary)");
}
// fleetd #297: named once and reused verbatim below for FleetApp's GET /profiles, rather than
// built a second time — two independently-constructed sources reading the SAME BackendQuarantine
// / BackendOutagePolicy would still be able to drift (e.g. a future edit to the credentialIdFor
// closure in only one of the two places), exactly the shape #284 was.
FleetMcp.QuarantineSource quarantineSource = new FleetMcp.QuarantineSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
}, quarantine);
FleetMcp.OutageSource outageSource = new FleetMcp.OutageSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
}, outagePolicy);
FleetMcp mcp = new FleetMcp(messages, workers, sessions, identity, presence,
primaryRegistry, callers, metrics, new FleetMcp.CapacitySource(profile -> liveCountRef.get().apply(profile),
profile -> {
@@ -500,10 +649,29 @@ public final class Fleetd {
return FleetHealthMonitor.coverage(health != null && health.isEnabled(),
health != null && health.notifications() != null && health.notifications().configured());
}),
new FleetMcp.QuarantineSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
}, quarantine));
quarantineSource,
leadMailbox,
outageSource,
new FleetMcp.LeadSeatSource(leadSeatLookup(() -> config.get().profiles(), leaders, leads)));
// CB-637: the receive half. Only constructed when a lead mailbox actually opened — with no
// coordinator (or an unreachable one) there is nothing to deliver, so no scheduler is
// created and no thread runs. It reads the SAME live lead supplier the injector's
// deliverability gate does, so a lead found by the tab scan after startup is reachable
// without a restart.
final LeadCoordLoop leadCoordLoop;
final ScheduledExecutorService leadCoordSchedulerRef;
if (leadMailbox != null) {
var leadCoordScheduler = Executors.newSingleThreadScheduledExecutor(r ->
Thread.ofVirtual().name("bridge-leadcoord-").unstarted(r));
leadCoordLoop = new LeadCoordLoop(leadMailbox, router.leadAgents(), leads, leadCoordScheduler,
LEAD_COORD_INTERVAL_MS);
leadCoordLoop.start();
leadCoordSchedulerRef = leadCoordScheduler;
} else {
leadCoordLoop = null;
leadCoordSchedulerRef = null;
}
// CB-559: opt-in config reload. With no `configReload:` block nothing is constructed, so an
// upgraded daemon behaves exactly as before — the file is read once at boot and never again.
@@ -524,6 +692,8 @@ public final class Fleetd {
messages.close();
pushLoop.close();
if (heartbeat != null) heartbeat.close(); // CB-551: stop the idle-lead heartbeat scheduler
if (leadCoordLoop != null) leadCoordLoop.close(); // CB-637: stop delivering peer-lead messages
if (leadCoordSchedulerRef != null) leadCoordSchedulerRef.shutdownNow();
if (healthMonitor != null) healthMonitor.stop();
if (configWatcher != null) configWatcher.stop(); // CB-559: stop polling the config file
mcp.close();
@@ -536,13 +706,30 @@ public final class Fleetd {
log.debug("reply inbox close: {}", e.toString());
}
}
herdr.close();
// CB-637: the coordination connection goes with it — after the loop that reads it has
// stopped, so no tick can be mid-ack against a closed channel.
if (leadMailbox != null) {
try {
leadMailbox.close();
} catch (Exception e) {
log.debug("lead mailbox close: {}", e.toString());
}
}
router.close();
}));
Javalin app = new FleetApp(herdr, workers, sessions, messages, presence, mcp.servlet(),
callers, metrics).build();
// CB-185: give FleetApp both daemons — /healthz must require both to answer and
// GET /sessions must merge across both, or a down/unpolled member daemon is invisible.
// fleetd #111: live (re-read-per-request) memberCredentials view for GET /member-credentials —
// same hot-reload shape as the memberCredentials supplier passed to ClaudeCodeLauncher above.
// fleetd #297: quarantineSource/outageSource are the SAME instances passed to FleetMcp above —
// GET /profiles must report the identical quarantine/cool-off facts as fleet_profiles.
Javalin app = new FleetApp(herdr, memberHerdr, workers, sessions, messages, presence, mcp.servlet(),
callers, metrics, deliverable,
() -> MemberCredentialPolicyView.of(config.get().memberCredentials()),
quarantineSource, outageSource).build();
app.start(cfg.bind().host(), cfg.bind().port());
log.info("bridged listening on {}:{}, herdr socket {}",
log.info("fleetd listening on {}:{}, herdr socket {}",
cfg.bind().host(), cfg.bind().port(), socket);
}
@@ -569,12 +756,264 @@ public final class Fleetd {
return target -> presence.isPresent(target) || leads.get().containsKey(target);
}
/**
* fleetd #248: package-private factory for the member worktree/branch lookup {@link
* CompletionResolver} uses to name a fallback report's worktree and branch (fleetd#241).
*
* <p>Before this ticket the lookup was an anonymous lambda built inline inside {@code main}'s
* {@code CompletionResolver} constructor call — provably untested wiring, the whole reason
* fleetd #248 exists: dropping that one argument (passing {@code _ -> null} instead) compiled
* clean and left every test green. Extracted here, {@code main} now calls this factory instead
* of building the lambda inline, and a source assertion on that call site
* ({@code FleetdCompletionResolverWiringTest}) proves the argument is still actually passed.
*
* <p>Takes the roster as a plain {@link Supplier} — not a {@link SessionManager} — so this is
* directly testable with a hand-built session list; no real {@code SessionManager} (launcher,
* worktrees, …) needs constructing. Follows the same {@code static} factory pattern as
* {@link #deliverableTo} above.
*
* @param roster the live member roster, normally {@code sessions::roster}
*/
static Function<String, CompletionResolver.WorktreeBranch> worktreeBranchLookup(
Supplier<List<MemberSession>> roster) {
return target -> roster.get().stream()
.filter(session -> target.equals(session.terminalId()))
.findFirst()
.map(session -> new CompletionResolver.WorktreeBranch(session.worktree(), session.branch()))
.orElse(null);
}
/**
* fleetd #176: per-profile factory for {@link FleetMcp.LeadSeatSource} — how many seats a
* profile's own live LEAD session(s) hold on the same Claude subscription.
*
* <p>{@code maxLoad} counts only members; the lead itself is a live {@code claude} session that
* is never moved off-subscription ({@code LeadLauncher} strips {@code ANTHROPIC_BASE_URL}/
* {@code AUTH_TOKEN} from a lead's env whatever its profile says), so a {@code subscription:
* true} profile's real ceiling is lower than its configured {@code maxLoad} by exactly the
* number of lead seats sharing that same account.
*
* <p><b>The derivation, and why this route was chosen over a new config key.</b> The link is
* {@code fleet.leaders.<name>.profile} — the field the operator already sets to name which
* {@code profiles:} entry a lead runs on (CB-557; see {@code fleetd.example.yaml}) — matched
* against the profile passed in here via {@link FleetConfig.Profile#effectiveCredentialId()},
* the same grouping key {@link dev.ltms.fleet.placement.BackendQuarantine} already uses to say
* two profiles share one account. Nothing new is added to the config schema: this reuses a field
* that already exists and already means "the profile this lead's own session runs on". A lead
* entry that names no {@code profile:} (recognise-only, CB-558) says nothing about which account
* it shares, and there is no other reliable signal on the daemon's side to derive that from — so
* such a lead contributes no seats, exactly as before this ticket. Making that lead's seat count
* requires the operator to add one line (`profile: sonnet` under its {@code fleet.leaders} entry)
* — a config statement, not a code change, and the smallest one available given the field
* already exists for a closely related purpose.
*
* <p>Only counts leads {@code liveLeadTerminals} currently reports — CB-531's live tab scan (or
* the legacy {@code primary.terminal} pin) — never every configured lead: an entry whose
* {@code instances} nobody has actually started is not really competing for a seat, and must not
* shrink capacity for one that was never live.
*
* @param profiles the live profile map, normally {@code () -> config.get().profiles()}
* in {@code main} — hot, like every other {@code maxLoad}/
* {@code credentialId} read {@link FleetMcp.CapacitySource} already does
* @param leaders {@code fleet.leaders}, read once at startup like the rest of that
* block ({@code fleetd.example.yaml} notes it is not hot) — passed as a
* plain map, never re-read from {@code config.get()}
* @param liveLeadTerminals terminal_id → lead name for every CURRENTLY recognised lead, normally
* the same supplier {@link dev.ltms.fleet.auth.CallerResolver#leads()}
* and {@code LeadCoordLoop} already consult
*/
static Function<String, Integer> leadSeatLookup(Supplier<Map<String, FleetConfig.Profile>> profiles,
Map<String, FleetConfig.Leader> leaders, Supplier<Map<String, String>> liveLeadTerminals) {
return profileName -> {
FleetConfig.Profile target = profiles.get().get(profileName);
if (target == null || !target.isSubscription()) {
return 0;
}
String targetCredential = target.effectiveCredentialId();
int seats = 0;
for (String leadName : liveLeadTerminals.get().values()) {
FleetConfig.Leader lead = leaders.get(leadName);
if (lead == null || lead.profile() == null || lead.profile().isBlank()) {
continue;
}
FleetConfig.Profile leadProfile = profiles.get().get(lead.profile());
if (leadProfile != null && targetCredential.equals(leadProfile.effectiveCredentialId())) {
seats++;
}
}
return seats;
};
}
/**
* fleetd #248 / fleetd#201 Unit 5: package-private factory for the per-target backend-error
* pattern lookup {@link CompletionResolver} classifies a pane scrape against. Closes over the
* live roster (to resolve a target to a profile) and {@code errorPatternsByProfile} (each
* profile's configured {@code errorPattern}, already compiled by the caller — the same map also
* feeds the coverage log next to where this is called) — nothing else, so it is directly
* testable. See {@link #worktreeBranchLookup} above for why this ticket exists and why the
* factory takes a roster {@link Supplier} rather than a {@link SessionManager}.
*
* @param roster the live member roster, normally {@code sessions::roster}
* @param errorPatternsByProfile every profile that has an {@code errorPattern} configured,
* keyed by profile name
*/
static BackendErrorPatternLookup backendErrorPatternLookup(Supplier<List<MemberSession>> roster,
Map<String, Pattern> errorPatternsByProfile) {
return target -> roster.get().stream()
.filter(session -> target.equals(session.terminalId()))
.findFirst()
.map(session -> errorPatternsByProfile.get(session.profile()))
.orElse(null);
}
/**
* Count sessions that occupy a profile's spawn capacity. A {@code BACKEND_ERROR} or
* {@code FAILED} session stays in the roster so {@code fleet_list} can show its failure, but a
* member that cannot accept another delivery does not use a seat.
*/
static int liveSessionCount(List<MemberSession> roster, String profileName) {
return (int) roster.stream()
.filter(session -> profileName.equals(session.profile()))
.filter(session -> session.state() != MemberSession.State.BACKEND_ERROR)
.filter(session -> session.state() != MemberSession.State.FAILED)
.count();
}
/**
* fleetd #248 / fleetd#201 Unit 5: factory for the production {@link BackendErrorSink} — the
* collaborator {@link CompletionResolver} notifies when a pane-scrape classification actually
* resolves a waiter as a backend error. Order: (1) mark the member BACKEND_ERROR; (2) resolve
* profile/credential through the roster — fail loud (never {@code Optional.ifPresent}, the
* fleetd #234 lesson applied to this sink) and notify the lead via {@code
* onBackendTargetUnmapped} when it cannot be resolved; (3) record the error in {@code
* outagePolicy}; (4) on a NEW incident (the {@code record()} call that actually crosses the
* threshold), tell the lead via {@code onBackendIncident}.
*
* <p>{@code public}, unlike {@link #worktreeBranchLookup} and {@link #backendErrorPatternLookup}
* above: {@code dev.ltms.fleet.inject.BackendOutageFlowTest} exercises this exact object as the
* real, wired production path, replacing what its own class doc used to call out as a
* hand-mirrored copy of this lambda ("mirrors {@code Fleetd.main}'s {@code backendErrorSink}
* lambda line-for-line") — that copy proved only itself, never that {@code main} still wires the
* real thing. That was precisely the gap fleetd #248 exists to close.
*
* @param sessions the session registry; both read (roster) and written (onBackendError)
* @param profiles the live profile map, normally {@code () -> config.get().profiles()} in
* {@code main}, or a fixed test map via {@code () -> profiles}
* @param pushLoop the lead-nudge loop, read lazily: {@code main} builds this sink before the
* real {@link ReplyPushLoop} exists (a genuine construction-order cycle, broken
* the same way {@code exhaustionSinkRef} is a few lines above it), so a
* {@link Supplier} reads whatever {@code main} has filled in by the time a real
* backend error fires
*/
public static BackendErrorSink backendErrorSink(SessionManager sessions,
Supplier<Map<String, FleetConfig.Profile>> profiles, BackendOutagePolicy outagePolicy,
Supplier<ReplyPushLoop> pushLoop) {
return (target, matchedLine, reason) -> {
sessions.onBackendError(target, reason);
String profileName = sessions.roster().stream()
.filter(session -> target.equals(session.terminalId()))
.findFirst()
.map(MemberSession::profile)
.orElse(null);
FleetConfig.Profile profile = profileName == null ? null : profiles.get().get(profileName);
if (profile == null) {
log.error("backend error on target '{}' ({}) but no profile could be resolved — the "
+ "target is not (yet) in the roster — no cool-off applied (fleetd #201 Unit 5)",
target, reason);
ReplyPushLoop loop = pushLoop.get();
if (loop != null) {
loop.onBackendTargetUnmapped(target, reason);
}
return;
}
String credentialId = profile.effectiveCredentialId();
Optional<BackendOutagePolicy.Incident> incident = outagePolicy.record(credentialId, target, reason);
incident.ifPresent(inc -> {
List<String> affectedProfiles = profiles.get().values().stream()
.filter(p -> credentialId.equals(p.effectiveCredentialId()))
.map(FleetConfig.Profile::profile)
.sorted()
.toList();
log.warn("credential '{}' cooling off for {}s after backend errors on {} distinct "
+ "target(s) (profile '{}'): {}", credentialId,
inc.remainingCoolOffSeconds(), inc.evidenceCount(), profile.profile(), reason);
ReplyPushLoop loop = pushLoop.get();
if (loop != null) {
loop.onBackendIncident(inc.id(), inc.targets(), credentialId, affectedProfiles,
(int) inc.remainingCoolOffSeconds());
}
});
};
}
/** Injection seam for {@link #selectReplyInbox}: production binds {@link AmqpReplyInbox#open}. */
@FunctionalInterface
interface AmqpOpener {
ReplyInbox open(String uri, int prefetch);
}
/** Injection seam for {@link #openLeadMailbox}: production binds {@link LeadMailbox#open}. */
@FunctionalInterface
interface LeadMailboxOpener {
LeadMailbox open(String uri, String selfCoordId, int prefetch);
}
/**
* CB-637: open this daemon's lead-to-lead mailbox, or return {@code null} to leave the feature
* off. Package-private and env-injected for the same reason as {@link #selectReplyInbox}: the
* selection is then testable without a broker or a mutable process environment.
*
* <p>Every "off" path returns {@code null}, and each says why at the level it deserves:
*
* <ul>
* <li>no {@code coordinator:} block — silent. Lead coordination is opt-in; an operator who
* never configured it does not need to be told it is off on every boot.</li>
* <li>a block whose {@code uriEnv} does not resolve — INFO, the same "you moved to the secret
* store and the variable is not there" case {@code selectReplyInbox} warns about.</li>
* <li>a configured broker but no {@code selfId} — WARN. This one is a half-finished config: a
* mailbox is named after the coord-id that owns it, so with no id there is no queue to own
* and no {@code from} to send as. Loud, because the operator plainly intended the feature.</li>
* <li>the broker refuses at boot — WARN, and carry on. Mirrors {@code openAmqpOrFallback}: a
* coordination broker that is down must never take a whole fleet's daemon with it, and the
* fleet still works exactly as it did before this feature existed.</li>
* </ul>
*/
static LeadMailbox openLeadMailbox(FleetConfig.Coordinator coordinator, Map<String, String> env,
LeadMailboxOpener opener) {
if (coordinator == null) {
return null; // opt-in: nothing configured, nothing to say
}
String uri = coordinator.effectiveUri(env);
if (uri == null) {
log.info("lead coordination: OFF — coordinator{} has no usable broker uri",
coordinator.uriEnv() == null ? "" : ".uriEnv=" + coordinator.uriEnv());
return null;
}
if (coordinator.selfId() == null || coordinator.selfId().isBlank()) {
log.warn("coordinator.selfId is unset — lead coordination is OFF. A lead mailbox is the "
+ "queue named after the coord-id that owns it, so with no id there is nothing to "
+ "own and no sender identity to publish as. Set coordinator.selfId to a name that "
+ "is unique across every daemon sharing {} and restart fleetd.",
stripCredentials(uri));
return null;
}
try {
LeadMailbox mailbox = opener.open(uri, coordinator.selfId(), coordinator.prefetchOrDefault());
log.info("lead coordination: ON as coord-id {} (prefetch={})",
coordinator.selfId(), coordinator.prefetchOrDefault());
return mailbox;
} catch (IllegalStateException e) {
log.warn("cannot reach the AMQP coordination broker ({}) — lead-to-lead messaging is OFF "
+ "for this process lifetime. fleet_send{{coordId}} will report it as "
+ "not configured, and peer messages already queued stay on the broker until "
+ "a restart picks them up. Reason: {}",
stripCredentials(uri), reasonOf(e));
return null;
}
}
/**
* CB-151/152: pick the reply inbox. A usable broker — a literal {@code uri}, or a {@code
* uriEnv} whose variable resolves (both read from {@code env}) — selects the durable AMQP inbox.
@@ -599,7 +1038,7 @@ public final class Fleetd {
if (effectiveUri == null) {
log.warn("broker.uriEnv={} is unset or blank — durable AMQP reply inbox DISABLED. "
+ "Replies are soft-state and will not survive a restart. Set {} in the "
+ "daemon's environment (see scripts/redeploy-bridged.sh) and restart to "
+ "daemon's environment (see scripts/redeploy-fleetd.sh) and restart to "
+ "use the durable broker inbox.",
broker.uriEnv(), broker.uriEnv());
log.info("reply inbox: in-memory (soft-state)");
@@ -657,7 +1096,7 @@ public final class Fleetd {
/**
* CB-594: which env vars the loaded config actually needs, and why — every non-{@code
* subscription} profile's {@code tokenEnv} (a subscription profile never reads one, see
* subscription} profile's explicitly configured {@code tokenEnv} (a subscription profile never reads one, see
* {@link FleetConfig.Profile#isSubscription()}), plus every profile's {@code gitTokenEnv}
* where set (opt-in), plus a configured {@code broker.uriEnv} (CB-151). Derived from the
* config, not hard-coded, so a new profile is covered for free. A var required by more than one
@@ -674,7 +1113,9 @@ public final class Fleetd {
static Map<String, List<String>> requiredSecretEnvVars(FleetConfig cfg) {
Map<String, List<String>> requiredBy = new LinkedHashMap<>();
cfg.profiles().forEach((name, profile) -> {
if (!profile.isSubscription()) {
// All kinds are checked when tokenEnv is explicit. OpenCode may use provider credentials,
// but an explicit tokenEnv still declares a required host secret for its configured provider.
if (!profile.isSubscription() && profile.hasTokenEnv()) {
requiredBy.computeIfAbsent(profile.tokenEnv(), _ -> new ArrayList<>())
.add("profile '" + name + "' tokenEnv");
}
@@ -700,7 +1141,7 @@ public final class Fleetd {
* <p>A missing entry only warns — it must never refuse to start. A daemon that boots and says
* what is wrong is strictly more useful than one that will not boot at all.
*/
private static void reportRequiredSecrets(FleetConfig cfg) {
static void reportRequiredSecrets(FleetConfig cfg) {
Map<String, List<String>> requiredBy = requiredSecretEnvVars(cfg);
if (requiredBy.isEmpty()) {
log.info("startup secrets: no profile references a token env var — nothing to check");
@@ -714,13 +1155,31 @@ public final class Fleetd {
} else {
log.warn("startup secret {}: MISSING ({}) — the daemon will start anyway, and this "
+ "failure stays invisible until a worker actually needs it. Fix "
+ "${SHARED_ENV}/tools/secrets.sh and restart bridged from a LOGIN "
+ "shell (see scripts/redeploy-bridged.sh).",
+ "${SHARED_ENV}/tools/secrets.sh and restart fleetd from a LOGIN "
+ "shell (see scripts/redeploy-fleetd.sh).",
varName, String.join(", ", sources));
}
});
}
/**
* fleetd #184: state the member trust model at startup. Environment controls and worktrees do
* not make a sandbox when fleetd and its members use the same OS user. A separate herdr may
* provide that boundary, but fleetd cannot inspect the uid at the other end of its socket.
*/
static void reportMemberTrustModel(FleetConfig cfg) {
if (cfg.memberHerdrSocket() != null && !cfg.memberHerdrSocket().isBlank()) {
log.info("member trust model: members are routed to a separate herdr through "
+ "memberHerdrSocket. fleetd cannot see that herdr's uid, so confirm it runs "
+ "as a different OS user before treating it as a boundary.");
return;
}
log.info("member trust model: members run as the same OS user as fleetd, not in a sandbox. "
+ "A member can read any file this user can read, including SSH keys and credential "
+ "stores, whatever memberCredentials says. To add a real boundary, route members to "
+ "a second herdr under a different OS user with memberHerdrSocket.");
}
/**
* CB-596: {@code known:} empty (block absent entirely, or present but empty) means {@link
* FleetConfig.MemberCredentials#blockedSet()} is empty too — every member pane inherits the
@@ -734,12 +1193,14 @@ public final class Fleetd {
* #requiredSecretEnvVars} is exposed for {@link #reportRequiredSecrets}'s own test.
*/
static void reportMemberCredentialsGap(FleetConfig cfg) {
FleetConfig.MemberCredentials creds = cfg.memberCredentials();
if (creds != null && !creds.known().isEmpty()) {
// fleetd #111: the counts below come from MemberCredentialPolicyView, the same class the
// live GET /member-credentials endpoint reads — one place computes them, not two.
MemberCredentialPolicyView view = MemberCredentialPolicyView.of(cfg.memberCredentials());
if (view.present()) {
log.info("memberCredentials: policy={}, {} known name(s), {} allowed — blocking {} on "
+ "every spawn{}",
creds.policy(), creds.known().size(), creds.allow().size(), creds.blockedSet().size(),
creds.isAllowList()
view.policy(), view.knownCount(), view.allowedCount(), view.blockedCount(),
cfg.memberCredentials().isAllowList()
? " (allow-list: known/allow are reporting only — the control is the derived ZDOTDIR scrub)"
: "");
return;
@@ -747,7 +1208,7 @@ public final class Fleetd {
log.warn("memberCredentials: absent or empty — the daemon will start anyway, and every "
+ "member pane inherits the operator's WHOLE secret store, unblocked (CB-592's "
+ "protection is lost). Add a memberCredentials: block (policy/allow/known) to "
+ "bridged.yaml — see fleetd.example.yaml — and restart.");
+ "fleetd.yaml — see fleetd.example.yaml — and restart.");
}
/**
@@ -262,11 +262,15 @@ public final class CallerResolver {
return token.isEmpty() ? null : token;
}
/**
* fleetd #305: delegates to {@link ConnectionIdentity#isLoopback}. This used to be a second,
* independent copy of the same rule, and the two drifted: this one accepted all of
* {@code 127.0.0.0/8}, {@code ConnectionIdentity}'s accepted only {@code 127.0.0.1}. A caller
* from {@code 127.0.0.2} therefore had its identity skipped (so it had no terminal) and was
* then read as loopback here — which under loopback-trust is the primary. Sharing the inputs
* would not have prevented that; only sharing the computation does.
*/
private static boolean isLoopback(String remoteAddr) {
if (remoteAddr == null) {
return false;
}
return remoteAddr.equals("127.0.0.1") || remoteAddr.equals("::1")
|| remoteAddr.equals("0:0:0:0:0:0:0:1") || remoteAddr.startsWith("127.");
return ConnectionIdentity.isLoopback(remoteAddr);
}
}
@@ -0,0 +1,74 @@
package dev.ltms.fleet.auth;
import dev.ltms.fleet.peer.MemberRole;
/** Optional session lifecycle hook for live member-slot bindings. */
public interface MemberLifecycle {
/** A slot held before a member process starts. */
record SlotReservation(String slot, String profile) { }
MemberLifecycle NONE = new MemberLifecycle() {
@Override
public MemberRole acquired(MemberRole role, String profile, String terminal) {
return role; // no registry configured — nothing to bind against, so the request stands
}
@Override
public void released(String terminal) {
}
@Override
public void requireSlotFor(MemberRole role, String profile) {
// no registry configured — nothing to validate against, so nothing is refused
}
@Override
public SlotReservation reserve(MemberRole role, String profile) {
return null;
}
@Override
public boolean bind(SlotReservation reservation, String terminal) {
return false;
}
@Override
public void release(SlotReservation reservation) {
}
};
/**
* Try to bind a newly spawned {@code terminal} into the role it was granted.
*
* @return the role this session actually holds: {@code role} unchanged for a role with no
* live slot-binding semantics (dev, reviewer), or when the bind succeeded; a fallback
* role — never {@code role} — when a slot-bound role (architect) could not be bound.
* Callers must record THIS value on the session, never the requested {@code role}, so
* a later roster read never reports a role the session does not hold (CB-619). In
* normal operation this fallback should not happen once a reservation has been bound.
* It remains the honest answer if a caller has no reservation, or if binding a
* reservation unexpectedly fails.
*/
MemberRole acquired(MemberRole role, String profile, String terminal);
void released(String terminal);
/**
* Refuse an acquire before anything spawns when {@code role} requires a live slot binding and
* no configured slot carries {@code profile} (CB-619 / fleetd #123). A no-op for a role with
* no slot-binding semantics.
*
* @throws IllegalArgumentException naming the role, the profile, and the pools that do carry it
*/
void requireSlotFor(MemberRole role, String profile);
/** Reserve a matching slot before launch, or refuse before a charter can be delivered. */
SlotReservation reserve(MemberRole role, String profile);
/** Convert a reservation into a live terminal binding. */
boolean bind(SlotReservation reservation, String terminal);
/** Return an unbound reservation after a failed launch. */
void release(SlotReservation reservation);
}
@@ -8,6 +8,7 @@ import org.slf4j.LoggerFactory;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -55,8 +56,10 @@ public final class MemberRegistry implements MemberLifecycle {
}
private final Map<String, Entry> slots;
/** Live {@code terminal_id → qualified slot key}; guarded by {@code this}. */
/** Live {@code terminal_id → qualified slot key}; guarded by {@code terminalToSlot}. */
private final Map<String, String> terminalToSlot = new HashMap<>();
/** Slot keys held between reservation and the terminal binding. Guarded by terminalToSlot. */
private final java.util.Set<String> reservedSlots = new java.util.HashSet<>();
/** Flatten every role pool in {@code fleet} into one registry. Leaders are not members. */
public MemberRegistry(FleetConfig.Fleet fleet) {
@@ -166,7 +169,7 @@ public final class MemberRegistry implements MemberLifecycle {
if (existingSlot != null) {
return slot.equals(existingSlot); // already this slot (idempotent) or a different one
}
if (terminalToSlot.containsValue(slot)) {
if (terminalToSlot.containsValue(slot) || reservedSlots.contains(slot)) {
return false; // slot already hosts a terminal — no second one
}
terminalToSlot.put(terminal, slot);
@@ -206,19 +209,116 @@ public final class MemberRegistry implements MemberLifecycle {
*
* <p>The role check is lifecycle policy. {@link CallerResolver} repeats it when resolving a
* binding, so a later lifecycle regression cannot turn a worker into an architect.
*
* <p>CB-619 / fleetd #123: the return value is the role this session actually holds, and the
* caller is required to record THAT — never the requested {@code role} — on the session. Before
* this fix the caller kept the requested role regardless of whether the bind below succeeded, so
* a demoted session's {@code GET /members} row still said {@code "architect"} while
* {@code fleet_whoami} (which reads the live binding, not the request) correctly said
* {@code "worker"} — three sources of truth that disagreed about one live member, silently.
*/
@Override
public void acquired(MemberRole role, String profile, String terminal) {
public MemberRole acquired(MemberRole role, String profile, String terminal) {
if (role != MemberRole.ARCHITECT || terminal == null || terminal.isBlank()) {
return;
return role;
}
// slotsFor preserves definition order, so duplicate-profile slots use the first free one.
for (Entry entry : slotsFor(MemberRole.ARCHITECT).values()) {
if (Objects.equals(profile, entry.profile()) && bind(entry.key(), terminal)) {
return;
return MemberRole.ARCHITECT;
}
}
// fleetd #123: at least WARN — a role downgrade that the roster must now also reflect is
// not routine bookkeeping. requireSlotFor already refuses the config-gap case (no slot at
// all carries this profile) before a process ever spawns; reaching here means the config DID
// carry a matching slot but every one of them was already bound to a different terminal — a
// race this pre-spawn check cannot close on its own (see requireSlotFor's javadoc).
log.warn("member slot: no free architect slot for profile={} terminal={}; holding the session "
+ "as {} instead of the architect it asked for — every configured slot for this "
+ "profile is already bound to a different terminal", profile, terminal,
MemberRole.DEV.wireName());
return MemberRole.DEV;
}
/**
* CB-619 / fleetd #123: refuse an architect acquire before anything spawns when no configured
* slot carries {@code profile} — the config-gap case from the original defect report (a spawn
* asked for {@code role=architect, profile=sonnet}, and {@code fleet.architects} carried only
* {@code opus} and {@code sol}). A dev/reviewer acquire is always a no-op: those pools are
* placement candidates only (see {@code CompositePeerLauncher}), never a live identity binding,
* so there is nothing here to refuse — an explicit profile outside the pool for those roles is a
* documented operator override, not a defect.
*
* <p>This closes the config-gap case, not the live-capacity case: a profile that DOES carry a
* slot can still lose the race to a concurrent spawn between this check and the actual
* {@link #bind}, which is why {@link #acquired} must still answer honestly even after this
* check has passed.
*/
@Override
public void requireSlotFor(MemberRole role, String profile) {
if (role != MemberRole.ARCHITECT) {
return;
}
boolean hasSlot = slotsFor(MemberRole.ARCHITECT).values().stream()
.anyMatch(e -> Objects.equals(profile, e.profile()));
if (hasSlot) {
return;
}
List<String> pools = slotsFor(MemberRole.ARCHITECT).values().stream()
.map(Entry::profile)
.distinct()
.toList();
throw new IllegalArgumentException(
"no " + role.wireName() + " slot for profile '" + profile + "' — an architect's "
+ "identity IS the slot it is bound to, so there is nothing to bind this "
+ "session's identity to. fleet." + role.configKey() + " carries profiles: "
+ (pools.isEmpty() ? "(none configured)" : String.join(", ", pools))
+ "; add profile '" + profile + "' there, or spawn " + role.wireName()
+ " on one of those profiles instead");
}
@Override
public SlotReservation reserve(MemberRole role, String profile) {
if (role != MemberRole.ARCHITECT) {
return null;
}
synchronized (terminalToSlot) {
for (Entry entry : slotsFor(MemberRole.ARCHITECT).values()) {
if ((profile == null || profile.isBlank() || Objects.equals(profile, entry.profile()))
&& !terminalToSlot.containsValue(entry.key()) && reservedSlots.add(entry.key())) {
return new SlotReservation(entry.key(), entry.profile());
}
}
}
throw new IllegalArgumentException("no free architect slot for profile '" + profile
+ "' — every matching slot is already bound or reserved");
}
@Override
public boolean bind(SlotReservation reservation, String terminal) {
if (reservation == null || terminal == null || terminal.isBlank()) {
return false;
}
synchronized (terminalToSlot) {
if (!reservedSlots.remove(reservation.slot())) {
return false;
}
if (!isSlot(reservation.slot()) || terminalToSlot.containsKey(terminal)
|| terminalToSlot.containsValue(reservation.slot())) {
return false;
}
terminalToSlot.put(terminal, reservation.slot());
return true;
}
}
@Override
public void release(SlotReservation reservation) {
if (reservation != null) {
synchronized (terminalToSlot) {
reservedSlots.remove(reservation.slot());
}
}
log.info("member slot: no free architect slot for profile={}; session remains a worker", profile);
}
/** Unbind a released terminal using the compare-safe registry operation. */
@@ -43,7 +43,9 @@ import java.util.function.Supplier;
* which is constructed once), <em>and an existing profile's launch settings</em> —
* {@code model}, {@code baseUrl}, {@code argv}, {@code env}, {@code mcpUrl},
* {@code exhaustedPattern} (CB-578 stage A — compiled once into {@code Fleetd.main}'s
* pattern map at startup), and the rest. {@code credentialId} (CB-578 stage B) is NOT on
* pattern map at startup), {@code errorPattern} (fleetd #201 Unit 5 — compiled once into
* {@code Fleetd.main}'s backend-error pattern map at startup, the same way), and the rest.
* {@code credentialId} (CB-578 stage B) is NOT on
* this list — it is read live off the config supplier at every quarantine check and
* exhaustion event, exactly like {@code weight} / {@code maxLoad}, so it is hot instead.
* {@code HerdrPeerLauncher} takes {@code Map.copyOf(profiles)} at construction and resolves
@@ -71,7 +73,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
/** Keys that cannot change under a running daemon — see the class doc. */
private static final Set<String> COLD_KEYS =
Set.of("bind", "herdrSocket", "broker", "auth");
Set.of("bind", "herdrSocket", "memberHerdrSocket", "broker", "auth");
private final Path path;
private final AtomicReference<FleetConfig> current;
@@ -128,7 +130,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
}
if (!applied) {
return "config reload refused — these keys cannot change under a running daemon: "
+ String.join(", ", coldKeys) + ". Restart bridged to apply them.";
+ String.join(", ", coldKeys) + ". Restart fleetd to apply them.";
}
if (!deferred.isEmpty()) {
return "config reloaded; these changes need a restart to take effect: "
@@ -190,6 +192,9 @@ public final class ConfigRef implements Supplier<FleetConfig> {
if (!Objects.equals(old.herdrSocket(), fresh.herdrSocket())) {
changed.add("herdrSocket");
}
if (!Objects.equals(old.memberHerdrSocket(), fresh.memberHerdrSocket())) {
changed.add("memberHerdrSocket");
}
if (!Objects.equals(old.broker(), fresh.broker())) {
changed.add("broker");
}
@@ -289,6 +294,10 @@ public final class ConfigRef implements Supplier<FleetConfig> {
// CB-578 stage B: exhaustedPattern is compiled once into Fleetd.main's pattern map
// at startup (see ExhaustedPatternLookup wiring) — a reload never re-reads it, so a
// changed pattern must be reported as deferred, exactly like model/baseUrl/argv.
&& Objects.equals(a.exhaustedPattern(), b.exhaustedPattern());
&& Objects.equals(a.exhaustedPattern(), b.exhaustedPattern())
// fleetd #201 Unit 5: errorPattern is compiled once into Fleetd.main's backend-error
// pattern map at startup (see BackendErrorPatternLookup wiring), the same way
// exhaustedPattern is — a reload never re-reads it either.
&& Objects.equals(a.errorPattern(), b.errorPattern());
}
}
@@ -11,7 +11,7 @@ import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
/**
* Polls {@code bridged.yaml}'s modified time and asks {@link ConfigRef} to reload when it moves
* Polls {@code fleetd.yaml}'s modified time and asks {@link ConfigRef} to reload when it moves
* (CB-559). Opt-in through {@code configReload.enabled}.
*
* <p><strong>Why polling and not a filesystem watch.</strong> {@code WatchService} on macOS has no
@@ -1,11 +1,14 @@
package dev.ltms.fleet.config;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import dev.ltms.fleet.msg.AmqpReplyInbox;
import dev.ltms.fleet.msg.LeadMailbox;
import dev.ltms.fleet.peer.MemberRole;
import dev.ltms.fleet.placement.PlacementPolicies;
import org.slf4j.Logger;
@@ -23,15 +26,18 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
/**
* {@code bridged} configuration, loaded from a YAML file (see
* {@code fleetd} configuration, loaded from a YAML file (see
* {@code fleetd.example.yaml}). Unknown keys are ignored so config can grow ahead
* of the code — but an unknown <em>top-level</em> key is logged as a WARN at load (CB-530), because
* silently dropping a whole block is indistinguishable from honouring it.
*
* @param bind REST/MCP listen host:port
* @param herdrSocket path to herdr's Unix socket ({@code null} → client default)
* @param herdrSocket path to the lead herdr Unix socket ({@code null} → client default)
* @param memberHerdrSocket optional member herdr Unix socket ({@code null}/blank → lead socket)
* @param profiles named backend profiles, keyed by profile name (multi-backend fleet). A
* profile answers <em>which backend</em> — model, CLI adapter, credentials,
* cost. It says nothing about what the member spawned on it is for; that is
@@ -69,11 +75,43 @@ import java.util.Set;
* @param memberCredentials deny-by-default policy (CB-596) for which of the operator's own host
* credentials a spawned member's pane inherits. {@code null} (the block
* omitted) blocks nothing — see {@link MemberCredentials}.
* @param coordinator shared cross-host leader coordination broker: a SEPARATE AMQP vhost from
* {@link #broker} used only for lead-to-lead traffic (member/worker inboxes
* stay on {@code broker}'s vhost). {@code null} → no lead mailbox is opened.
* Config parsing + accessors only — nothing here wires it into a live
* {@code LeadMailbox}; that is a separate ticket. See {@link Coordinator}.
* @param worktreeGroup optional OS group name (fleetd #185 stage 3) that makes a provisioned
* worktree's repo group-shared, so a member running as a different OS user
* (see {@code memberHerdrSocket}) can write its own worktree, its per-worktree
* git metadata, and its own commit objects. {@code null}/blank/empty ⇒ off,
* today's behaviour unchanged (every file stays owned by fleetd's own uid).
* <strong>This isolates credentials, not the repository</strong>: a member in
* the group can still write the operator's git objects and refs in the shared
* repo. See {@link dev.ltms.fleet.session.Worktrees#shareWithGroup}.
* @param memberLoginShell fleetd #213: the login shell the member's OS user actually runs, ONLY
* meaningful (and only ever read) when {@code memberHerdrSocket} is
* configured — that mode spawns member panes under a different OS user than
* fleetd's own process, so fleetd's own {@code $SHELL} says nothing about what
* that pane runs. There is no channel to ask herdr for another user's shell, so
* this must be told, never guessed. {@code null}/blank (or a value not ending
* in {@code zsh}) is treated the same as "not zsh". fleetd #155: what that means
* now depends on {@code memberCredentials.policy}. Under {@code deny-by-default}
* it stays a degraded control, never a refusal to spawn — the pane-creation
* overlay is unaffected by shell type, so the spawn proceeds with a WARN naming
* the shell. Under {@code allow-list} the spawn is REFUSED instead: that policy's
* whole point is a control a sourced file cannot undo, so silently falling back
* to the weaker overlay would be the same "control silently does nothing" defect
* #155 exists to remove — configure this field (or move the account to zsh, or
* switch policy back to {@code deny-by-default}) to unblock the spawn.
* When {@code memberHerdrSocket} is NOT configured this field is never
* consulted at all; fleetd keeps reading its own {@code $SHELL}, exactly as
* before this field existed.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record FleetConfig(
Bind bind,
String herdrSocket,
String memberHerdrSocket,
Map<String, Profile> profiles,
Guard guard,
String worktreeRoot,
@@ -89,7 +127,46 @@ public record FleetConfig(
Auth auth,
ConfigReload configReload,
Integer quarantineCooldownSeconds,
MemberCredentials memberCredentials) {
MemberCredentials memberCredentials,
Coordinator coordinator,
String worktreeGroup,
String memberLoginShell) {
/** Back-compat form before the {@code memberLoginShell} key was added. */
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
Guard guard, String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload, Integer quarantineCooldownSeconds,
MemberCredentials memberCredentials, Coordinator coordinator, String worktreeGroup) {
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup, null);
}
/** Back-compat form before the {@code worktreeGroup} key was added. */
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
Guard guard, String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload, Integer quarantineCooldownSeconds,
MemberCredentials memberCredentials, Coordinator coordinator) {
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, null, null);
}
/** Back-compat form before the {@code coordinator:} block was added. */
public FleetConfig(Bind bind, String herdrSocket, Map<String, Profile> profiles, Guard guard,
String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload, Integer quarantineCooldownSeconds,
MemberCredentials memberCredentials) {
this(bind, herdrSocket, null, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, null, null);
}
/** Back-compat form before the CB-596 {@code memberCredentials:} block was added. */
public FleetConfig(Bind bind, String herdrSocket, Map<String, Profile> profiles, Guard guard,
@@ -97,9 +174,9 @@ public record FleetConfig(
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload, Integer quarantineCooldownSeconds) {
this(bind, herdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
this(bind, herdrSocket, null, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, null);
configReload, quarantineCooldownSeconds, null, null);
}
/** Default cooldown (CB-578 stage B) when {@code quarantineCooldownSeconds} is absent/non-positive. */
@@ -110,8 +187,8 @@ public record FleetConfig(
String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, String placement, Auth auth) {
this(bind, herdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, null, placement, auth, null, null);
this(bind, herdrSocket, null, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, null, placement, auth, null, null, null, null);
}
/** Back-compat form before the optional {@code health:} block was added. */
@@ -119,8 +196,8 @@ public record FleetConfig(
String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, String placement, Auth auth, ConfigReload configReload) {
this(bind, herdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, null, placement, auth, configReload, null);
this(bind, herdrSocket, null, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, null, placement, auth, configReload, null, null, null);
}
/** Back-compat form before the CB-578 stage B {@code quarantineCooldownSeconds} field was added. */
@@ -129,9 +206,9 @@ public record FleetConfig(
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload) {
this(bind, herdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
this(bind, herdrSocket, null, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, null);
configReload, null, null, null);
}
/**
@@ -173,13 +250,14 @@ public record FleetConfig(
* @param configDir {@code CLAUDE_CONFIG_DIR} so the worker inherits the profile's
* skills/MCP/hooks (may be {@code null})
* @param tokenEnv name of the host env var holding the worker's auth token; its value
* is injected as {@code ANTHROPIC_AUTH_TOKEN} (never stored in config)
* is injected as {@code ANTHROPIC_AUTH_TOKEN} (never stored in config).
* {@code null}/blank means this profile needs no token.
* @param argv launch command; defaults to {@code ["claude"]}
* @param placement where a worker lands: {@code "tab"} (default — its own tab in the
* worker space) or {@code "pane"} (legacy — split the focused tab)
* @param workspace label of the dedicated worker space; found-or-created on first
* spawn (default {@code "bridged-workers"}). A future per-session
* layout is just a distinct label here — the shared space is default.
* @param workspace label of the shared fleet space; found-or-created on first spawn
* (default {@code "fleet"}). The lead and every member share this one
* space so the operator sees a single "session" with many tabs.
* @param tabLabel template for a worker tab's label; {@code {profile}}/{@code {model}}
* and {@code {n}} (per-worker number, to keep sibling tabs distinct)
* are substituted (default {@code "worker: {profile} #{n}"})
@@ -189,7 +267,13 @@ public record FleetConfig(
* @param cwd fixed working directory for this profile's workers (CB-112 "told otherwise");
* {@code null}/blank → inherit the primary's cwd, else the daemon's
* @param parityOverlay repo-relative paths copied primary→worktree for config parity; null/empty
* defaults to a sensible set of local config files.
* defaults to {@code [.env]} only (CB-148 point 2). {@code .env} is data — a
* copy of it can only carry values. {@code .envrc} is executable shell that
* {@code direnv} evaluates on every {@code cd}, so copying it moves behaviour
* into the worker, not just values, and that is a different risk from copying
* data. It is deliberately left out of the default for that reason. The knob
* still supports it: an operator who wants it copied writes
* {@code parityOverlay: [.env, .envrc]} explicitly and owns that choice.
* <p><strong>Every overlay path must be gitignored or tracked-and-skipped.</strong>
* CB-576 made {@code release()} preserve a worktree that {@code git status
* --porcelain} reports as dirty, and it deliberately counts untracked files —
@@ -199,11 +283,10 @@ public record FleetConfig(
* worktrees then accumulate with no error anywhere.
* <p>Checked on 2026-08-15 (CB-581): inert as configured. Tracked overlay
* files carry {@code --skip-worktree} so {@code --porcelain} cannot see them,
* {@code bridged.yaml} is gitignored, and the default pair {@code .env} /
* {@code .envrc} does not exist in this repo. Note the default applies to
* <em>every</em> profile, so creating either file at the repo root is enough
* to make it live. Add a new overlay path to {@code .gitignore} in the same
* change that adds it here.
* {@code fleetd.yaml} is gitignored, and {@code .env} does not exist in this
* repo. Note the default applies to <em>every</em> profile, so creating
* {@code .env} at the repo root is enough to make it live. Add a new overlay
* path to {@code .gitignore} in the same change that adds it here.
* <p>CB-578 stage C raised the stakes: a preserved dirty worktree is now also
* committed to {@code refs/wip/<branch>} via {@code git add -A}. The overlay
* carries the primary's own environment files into the worktree, so a
@@ -239,8 +322,9 @@ public record FleetConfig(
* statement that does not stop being true just because the profile was
* named directly. A negative value has no sane meaning (there is no
* "excluded" to degrade to below zero) and is refused at config load
* instead, naming the profile and the key. Live means any session the
* registry still owns (acquired and not yet released), in any state.
* instead, naming the profile and the key. Live means a session that can
* receive another delivery. The roster keeps terminal {@code BACKEND_ERROR}
* and {@code FAILED} sessions for diagnostics, but they do not use capacity.
* @param kind which peer launcher spawns this profile: {@code "claude-code"} (default —
* the {@link dev.ltms.fleet.member.ClaudeCodeLauncher}) or {@code "opencode"}.
* The {@code CompositePeerLauncher} routes {@code spawn}/reap by this value, so
@@ -288,6 +372,16 @@ public record FleetConfig(
* {@code limit.context} instead, which bounds the window opencode compacts
* <em>within</em>, and only when {@code model} resolves to a
* {@code provider/model} pair.
* @param errorPattern regex matched against a completion-fallback scrape (fleetd #201 Unit 5)
* to classify a turn that ended with no {@code fleet_reply} as a backend
* error (credential outage, provider 5xx) rather than a real answer or an
* exhausted usage limit. {@code null}/blank ⇒ this profile relies on
* {@link dev.ltms.fleet.inject.CompletionResolver}'s built-in
* {@code (?i)\bAPI Error\s*:} compatibility pattern instead — classification
* still happens, just without a profile-specific match (reported separately
* at startup as legacy-default coverage, never full coverage). Compiled once
* at startup ({@code Fleetd.main}), like {@code exhaustedPattern}, so a
* reload of this key is DEFERRED (see {@code ConfigRef}).
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record Profile(String profile, String baseUrl, String model,
@@ -306,7 +400,8 @@ public record FleetConfig(
String ideMcpUrl,
String ideProjectDir,
String ideOpenCommand,
Integer autoCompactWindow) {
Integer autoCompactWindow,
String errorPattern) {
/** Peer kind spawned by {@link dev.ltms.fleet.member.ClaudeCodeLauncher} (the default). */
public static final String KIND_CLAUDE_CODE = "claude-code";
@@ -322,9 +417,9 @@ public record FleetConfig(
? (KIND_CLAUDE_CODE.equals(k) ? List.of("claude") : List.of(k))
: List.copyOf(argv);
kind = k;
tokenEnv = (tokenEnv == null || tokenEnv.isBlank()) ? "BRIDGED_WORKER_TOKEN" : tokenEnv;
tokenEnv = (tokenEnv == null || tokenEnv.isBlank()) ? null : tokenEnv;
placement = (placement == null || placement.isBlank()) ? "tab" : placement.toLowerCase();
workspace = (workspace == null || workspace.isBlank()) ? "bridged-workers" : workspace;
workspace = (workspace == null || workspace.isBlank()) ? "fleet" : workspace;
// CB-557: no per-profile default any more. A label is generated from the member's ROLE
// ("dev: sonnet #2"), which a profile cannot know, so the template lives on `fleet:` and
// this field is only an override for a profile that wants its own. Blank ⇒ defer.
@@ -340,7 +435,7 @@ public record FleetConfig(
// servers do not exist there to be enabled. This is defence in depth, not a live fix: the
// worker stays isolated only because this separate mechanism already removes the servers.
parityOverlay = (parityOverlay == null || parityOverlay.isEmpty())
? List.of(".env", ".envrc")
? List.of(".env")
: List.copyOf(parityOverlay);
// gitTokenEnv stays null when unset (opt-in). gitHostEnv defaults so operators enabling
// checkpoints need only set gitTokenEnv; it is injected only alongside a resolved token.
@@ -370,7 +465,7 @@ public record FleetConfig(
// (pinned to its own worktree via the charter). Blank ⇒ off.
ideMcpUrl = (ideMcpUrl == null || ideMcpUrl.isBlank()) ? null : ideMcpUrl;
// CB-634 auto-open: both are only read when hasIdeMcp(). ideProjectDir is the repo-relative
// module dir IntelliJ must open (this repo's pom lives in `bridged/`, not at the root), and
// module dir IntelliJ must open (this repo's pom lives in `fleetd/`, not at the root), and
// it is also the project_path the overlay pins. Blank ⇒ the worktree root (unchanged before
// auto-open). ideOpenCommand is the host command that opens that dir in the IDE, with {dir}
// substituted; blank ⇒ no auto-open (the operator opens the module by hand).
@@ -380,6 +475,32 @@ public record FleetConfig(
// there is no blank-string form to normalize (it's an Integer), and the [100000, 1000000]
// range is enforced eagerly at config load (rejectAutoCompactWindowOutOfRange), naming the
// profile, rather than silently clamped here. A profile that never sets it keeps null.
// errorPattern stays null when unset/blank (opt-in) — same rule as exhaustedPattern: no
// defaulting, no vendor wording. A profile that never sets it relies on
// CompletionResolver's built-in compatibility pattern instead (never "off").
errorPattern = (errorPattern == null || errorPattern.isBlank()) ? null : errorPattern;
}
/**
* Backward-compatible constructor without the fleetd #201 {@code errorPattern} field — the
* profile relies on {@code CompletionResolver}'s built-in compatibility pattern instead
* (legacy-default coverage). This is the shape the canonical constructor had before the
* field was added; every pre-existing Java call site (and any YAML that omits the key) keeps
* compiling and behaving identically. Jackson binds the canonical (longest) constructor, so
* YAML omitting {@code errorPattern:} still lands here as {@code null} via that path too.
*/
public Profile(String profile, String baseUrl, String model,
String configDir, String tokenEnv, List<String> argv,
String placement, String workspace, String tabLabel, String mcpUrl,
String cwd, List<String> parityOverlay, String gitTokenEnv, String gitHostEnv,
String kind, Map<String, String> env, Float weight, Integer maxLoad,
Boolean subscription, String exhaustedPattern, String credentialId,
String ideMcpUrl, String ideProjectDir, String ideOpenCommand,
Integer autoCompactWindow) {
this(profile, baseUrl, model, configDir, tokenEnv, argv, placement, workspace, tabLabel,
mcpUrl, cwd, parityOverlay, gitTokenEnv, gitHostEnv, kind, env, weight, maxLoad,
subscription, exhaustedPattern, credentialId, ideMcpUrl, ideProjectDir, ideOpenCommand,
autoCompactWindow, null);
}
/**
@@ -424,7 +545,8 @@ public record FleetConfig(
public Profile withProfile(String p) {
return new Profile(p, baseUrl, model, configDir, tokenEnv, argv, placement, workspace, tabLabel,
mcpUrl, cwd, parityOverlay, gitTokenEnv, gitHostEnv, kind, env, weight, maxLoad, subscription,
exhaustedPattern, credentialId, ideMcpUrl, ideProjectDir, ideOpenCommand, autoCompactWindow);
exhaustedPattern, credentialId, ideMcpUrl, ideProjectDir, ideOpenCommand, autoCompactWindow,
errorPattern);
}
/**
@@ -522,14 +644,55 @@ public record FleetConfig(
}
/**
* The credential group this profile quarantines with (CB-578 stage B): the configured
* {@link #credentialId} when set, else this profile's own name — so an unconfigured profile
* quarantines alone, exactly as it did before this field existed. Two profiles that set the
* same non-blank {@code credentialId} share one quarantine: a {@code BACKEND_EXHAUSTED}
* classification on either one quarantines both.
* True when this profile configures its own fleetd #201 Unit 5 backend-error classification
* pattern. {@code false} means this profile relies on {@code CompletionResolver}'s built-in
* {@code (?i)\bAPI Error\s*:} compatibility pattern instead (legacy-default coverage).
*/
public boolean hasErrorPattern() {
return errorPattern != null;
}
/**
* The shared credential id every {@code subscription: true} profile falls back to when it
* sets no explicit {@link #credentialId} (fleetd #176 stage 2, correcting an inert first cut
* of that ticket). A subscription profile has no credential of its own to fall back to its
* name for: it authenticates as the operator's own Claude login, and a host has exactly one
* of those, whatever names the operator gives the profiles running on it. Falling back to the
* profile's own name (the way an ordinary off-subscription profile does) would keep two
* subscription profiles on one login apart from each other, which is the opposite of what
* "one account" means.
*
* <p>Measured live and what it broke: a lead on profile {@code opus}, members on profile
* {@code sonnet}, same Claude login, neither setting {@code credentialId}. Before this
* sentinel, {@code opus.effectiveCredentialId()} was {@code "opus"} and {@code sonnet
* .effectiveCredentialId()} was {@code "sonnet"} — so fleetd #176's lead-seat matcher (and,
* this sentinel now also fixes, {@code CompositePeerLauncher}'s quarantine/cool-off spawn
* refusal and {@code BackendOutagePolicy}'s incident grouping) silently never linked them: the
* fix shipped, and stayed inert on the one host it was written for.
*/
public static final String SUBSCRIPTION_CREDENTIAL_ID = "<subscription>";
/**
* The credential group this profile quarantines with (CB-578 stage B; extended fleetd #176
* stage 2 — see {@link #SUBSCRIPTION_CREDENTIAL_ID}): the configured {@link #credentialId}
* when set — that always wins, so an operator with two separate Claude logins on one host can
* still keep them apart. Otherwise, a {@code subscription: true} profile falls back to
* {@link #SUBSCRIPTION_CREDENTIAL_ID} rather than its own name; an ordinary off-subscription
* profile falls back to its own name, exactly as it did before this field existed, so an
* unconfigured off-subscription profile still quarantines alone.
*
* <p>A {@code BACKEND_EXHAUSTED} (or repeated backend-error) classification on any profile
* sharing the result quarantines/cools off every profile that shares it — including, now,
* every {@code subscription: true} profile with no explicit {@code credentialId}. That is
* intended, not incidental: one Claude subscription hitting a usage limit really does take out
* every profile running on it, the same way {@code credentialId: openai-shared} already lets
* two OpenAI-backed profiles share one quarantine.
*/
public String effectiveCredentialId() {
return (credentialId == null || credentialId.isBlank()) ? profile : credentialId;
if (credentialId != null && !credentialId.isBlank()) {
return credentialId;
}
return isSubscription() ? SUBSCRIPTION_CREDENTIAL_ID : profile;
}
/** True when this profile's workers are granted a forge token to open their own PR (CB-302). */
@@ -537,6 +700,11 @@ public record FleetConfig(
return gitTokenEnv != null && !gitTokenEnv.isBlank();
}
/** True when this profile explicitly names a host token environment variable. */
public boolean hasTokenEnv() {
return tokenEnv != null && !tokenEnv.isBlank();
}
/**
* True when this profile's {@code env:} block names a worker-side Anthropic binding variable.
* Those two keys are the adapter's, never the operator's: on a claude-code worker
@@ -623,6 +791,9 @@ public record FleetConfig(
Integer paneProbeIntervalSeconds, Notifications notifications) {
public boolean isEnabled() { return Boolean.TRUE.equals(enabled); }
public int intervalOrDefault() { return Math.max(15, intervalSeconds == null ? 30 : intervalSeconds); }
public int workingSuspectAfterOrDefault() {
return Math.max(300, workingSuspectAfterSeconds == null ? 600 : workingSuspectAfterSeconds);
}
public record Notifications(String mode) {
public boolean configured() { return "webhook".equalsIgnoreCase(mode); }
}
@@ -630,7 +801,7 @@ public record FleetConfig(
/**
* External AMQP broker for durable, cross-restart reply delivery (CB-307 Stage 2). Its mere
* presence swaps the in-memory {@code ReplyInbox} for the AMQP-backed adapter; absent, bridged
* presence swaps the in-memory {@code ReplyInbox} for the AMQP-backed adapter; absent, fleetd
* stays soft-state. Production default is LavinMQ; a stock RabbitMQ speaks the same AMQP 0-9-1
* and is a URI-only swap.
*
@@ -690,6 +861,82 @@ public record FleetConfig(
}
}
/**
* Shared cross-host leader coordination broker: a {@link LeadMailbox} lets two leads on
* different daemons — possibly different hosts — exchange durable messages, which a herdr pane
* injection (how {@code fleet_send} reaches a lead today) cannot do at all. Its mere presence is
* config only in this ticket: nothing here opens a live {@code LeadMailbox} yet, that wiring is
* a separate ticket.
*
* <p>Deliberately a SEPARATE vhost from {@link Broker}, not a reuse of it. {@link Broker} is
* per-fleet — its queues are named by worker session id, and two fleets sharing one broker stay
* isolated by vhost (see {@code Two fleets share one LavinMQ}). Leader coordination is meant to
* cross exactly that boundary: two independently-owned fleets' leads talking to each other. Using
* the same vhost would either leak member traffic across the fleet boundary this is meant to
* cross, or force every fleet's members onto one shared vhost to get leader coordination — a
* second, dedicated vhost keeps "member inboxes stay fleet-local" true while still letting leads
* reach across fleets.
*
* @param uri AMQP connection URI for the coordination vhost, e.g.
* {@code amqp://guest:guest@127.0.0.1:5672/coord}. Blank/{@code null} ⇒ the
* coordinator block is treated as unconfigured. Ignored when {@code uriEnv} is set.
* @param uriEnv name of a host env var holding the AMQP URI, same convention as
* {@link Broker#uriEnv()} — keeps the credential out of the config file. Wins
* over {@code uri} whenever set. Blank/{@code null} ⇒ ignored.
* @param selfId this daemon's own lead coord-id — the name its {@code LeadMailbox} is owned
* under ({@code lead.<selfId>.inbox}), e.g. {@code "mac-opus"}. Blank/
* {@code null} ⇒ kept as {@code null} (no self id configured).
* @param prefetch the consumer's {@code basicQos} prefetch count. {@code null}/non-positive ⇒
* {@link LeadMailbox#DEFAULT_PREFETCH}.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record Coordinator(String uri, String uriEnv, String selfId, Integer prefetch) {
public Coordinator {
selfId = (selfId == null || selfId.isBlank()) ? null : selfId;
}
/** True when a {@code uriEnv} is configured by name, whether or not its variable resolves. */
public boolean hasUriEnv() {
return uriEnv != null && !uriEnv.isBlank();
}
/**
* True when a usable coordination broker URI is configured (an empty block does not enable
* it). Honors {@code uriEnv} first: if it names a variable that is unset or blank, the
* coordinator is <em>not</em> configured — a bare {@code uri} is only consulted when no
* {@code uriEnv} is set.
*/
public boolean isConfigured() {
return effectiveUri() != null;
}
/**
* The effective AMQP URI to connect with. {@code uriEnv} wins when set (both over
* {@code uri} and alone). When {@code uriEnv} names a variable that is unset or blank,
* returns {@code null} rather than falling back to {@code uri} — an operator who moved to
* the secret store must not silently drop back onto a stale clear-text URI. Returns the
* literal {@code uri} when no {@code uriEnv} is configured.
*/
public String effectiveUri() {
return effectiveUri(System.getenv());
}
/** As {@link #effectiveUri()}, reading the variable value from {@code env} (the injection seam). */
public String effectiveUri(Map<String, String> env) {
if (hasUriEnv()) {
String value = env.get(uriEnv);
return (value != null && !value.isBlank()) ? value : null;
}
return (uri != null && !uri.isBlank()) ? uri : null;
}
/** The prefetch to use, defaulting to {@link LeadMailbox#DEFAULT_PREFETCH} when unset. */
public int prefetchOrDefault() {
return (prefetch != null && prefetch > 0) ? prefetch : LeadMailbox.DEFAULT_PREFETCH;
}
}
/**
* Optional pinned primary terminal config (CB-307). When present with a non-blank
* {@code terminal}, the bridge uses this as the primary's herdr identity instead of
@@ -740,7 +987,15 @@ public record FleetConfig(
* survives restarts of the agent inside it — so identity is now the tab label alone.
*
* @param profile the {@code profiles:} entry to launch this lead on when one must
* be created; {@code null} ⇒ recognise-only, never create
* be created; {@code null} ⇒ recognise-only, never create.
* <p>fleetd #176: also the field {@code Fleetd.leadSeatLookup} reads
* to learn which account this lead's own live session shares — set it
* (safely, even on an already-running recognise-only lead: naming a
* profile here never starts anything beyond {@code instances}) so a
* {@code subscription: true} worker profile sharing its
* {@code effectiveCredentialId()} has this lead's seat subtracted from
* {@code fleet_list}'s {@code free}. {@code null} here also means this
* lead's seat cannot be derived and is not counted.
* @param tab the exact tab label hosting this lead, matched case-insensitively;
* the only field identity depends on. Required — a lead with no
* {@code tab} can never be discovered, launched or not
@@ -762,11 +1017,12 @@ public record FleetConfig(
String workspace, String cwd) {
/**
* Where an auto-launched lead's tab is created (CB-558). It must NOT be a member workspace:
* {@code LeadTabScanner} excludes those wholesale, so a lead placed in one would never be
* discovered and the daemon would relaunch it on every boot.
* Where an auto-launched lead's tab is created (CB-558). It defaults to the SAME shared
* {@code "fleet"} space the members use, so the operator sees one "session" with many tabs.
* The scanner no longer excludes member spaces — it tells a lead from a member by the exact
* tab label, so a lead sharing the members' space is still discovered (see LeadLauncher).
*/
public static final String DEFAULT_WORKSPACE = "leads";
public static final String DEFAULT_WORKSPACE = "fleet";
public Leader {
instances = (instances == null || instances < 0) ? 1 : instances;
@@ -965,7 +1221,7 @@ public record FleetConfig(
}
/**
* Watch {@code bridged.yaml} and re-read it when it changes (CB-559).
* Watch {@code fleetd.yaml} and re-read it when it changes (CB-559).
*
* <p>Opt-in, like every other block that acts on its own initiative. A daemon that reloads
* whenever a file is saved would apply a half-finished edit the moment an editor writes it, and
@@ -1023,7 +1279,7 @@ public record FleetConfig(
* a caller must present {@code Authorization: Bearer <token>} or it is
* {@code ANONYMOUS} and authorized for nothing.
* @param tokenEnv name of the host env var holding the bearer token; the literal value is
* never stored in config. Defaults to {@code BRIDGED_API_TOKEN}. Only read
* never stored in config. Defaults to {@code FLEETD_API_TOKEN}. Only read
* when {@code mode} is {@code token}.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
@@ -1036,7 +1292,7 @@ public record FleetConfig(
public Auth {
mode = (mode == null || mode.isBlank()) ? MODE_LOOPBACK_TRUST : mode.toLowerCase();
tokenEnv = (tokenEnv == null || tokenEnv.isBlank()) ? "BRIDGED_API_TOKEN" : tokenEnv;
tokenEnv = (tokenEnv == null || tokenEnv.isBlank()) ? "FLEETD_API_TOKEN" : tokenEnv;
}
/** True when a bearer token is required of every non-worker caller. */
@@ -1104,19 +1360,19 @@ public record FleetConfig(
* deny-list/deny-by-default every name here that is NOT also in {@link #allow} is
* overlaid with a non-secret sentinel value. Under allow-list this list is
* reporting only.
* @param sshAuthSock whether the member may inherit {@code SSH_AUTH_SOCK} under the allow-list
* policy ({@code "allow"}) or must have it blanked ({@code "block"}, the default).
* @param sshAgentEnv whether the member may inherit {@code SSH_AUTH_SOCK} under the allow-list
* policy ({@code "inherit"}) or must have it omitted ({@code "omit"}, the default).
* This is a DECISION, never a default: {@code SSH_AUTH_SOCK} is a handle to the
* operator's ssh-agent, and a member holding it can sign with the operator's own
* keys — but it appears in no secret file and is credential-shaped like nothing on
* any list, which is why three earlier tickets missed it (gitea #110 / CB-607).
* Blocking it breaks git over SSH inside the member; allow it only when members do
* Omitting it does not prevent git over SSH inside the member; inherit it only when members do
* not need to authenticate as the operator over SSH. Ignored under deny-list /
* deny-by-default, which never touch the name.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record MemberCredentials(String policy, List<String> allow, List<String> known,
String sshAuthSock) {
String sshAgentEnv) {
/** Default policy: block every {@code known} name not in {@code allow}, via the env overlay. */
public static final String POLICY_DENY_BY_DEFAULT = "deny-by-default";
@@ -1133,11 +1389,25 @@ public record FleetConfig(
*/
public static final String POLICY_ALLOW_LIST = "allow-list";
/** The pre-CB-633 three-field form — {@code sshAuthSock} defaults to blocked. */
/** The pre-CB-633 three-field form — {@code sshAgentEnv} defaults to omitted. */
public MemberCredentials(String policy, List<String> allow, List<String> known) {
this(policy, allow, known, null);
}
/**
* Reads both the current {@code sshAgentEnv} key and the compatible {@code sshAuthSock} key.
* When both keys are present, {@code sshAgentEnv} wins, even if its value is unrecognised.
*/
@JsonCreator
public static MemberCredentials fromYaml(@JsonProperty("policy") String policy,
@JsonProperty("allow") List<String> allow,
@JsonProperty("known") List<String> known,
@JsonProperty("sshAgentEnv") String sshAgentEnv,
@JsonProperty("sshAuthSock") String sshAuthSock) {
return new MemberCredentials(policy, allow, known,
sshAgentEnv != null ? sshAgentEnv : sshAuthSock);
}
public MemberCredentials {
String normalizedPolicy = (policy == null || policy.isBlank())
? POLICY_DENY_BY_DEFAULT : policy.toLowerCase(java.util.Locale.ROOT);
@@ -1146,8 +1416,9 @@ public record FleetConfig(
policy = POLICY_DENY_LIST.equals(normalizedPolicy) ? POLICY_DENY_BY_DEFAULT : normalizedPolicy;
allow = allow == null ? List.of() : List.copyOf(allow);
known = known == null ? List.of() : List.copyOf(known);
sshAuthSock = (sshAuthSock != null && "allow".equalsIgnoreCase(sshAuthSock.trim()))
? "allow" : "block";
sshAgentEnv = (sshAgentEnv != null
&& ("inherit".equalsIgnoreCase(sshAgentEnv.trim())
|| "allow".equalsIgnoreCase(sshAgentEnv.trim()))) ? "inherit" : "omit";
}
/** True when this block selects the CB-633 derived-allow-list policy. */
@@ -1156,8 +1427,8 @@ public record FleetConfig(
}
/** True when {@code SSH_AUTH_SOCK} may pass through under the allow-list policy. Default: no. */
public boolean sshAuthSockAllowed() {
return "allow".equals(sshAuthSock);
public boolean sshAgentEnvInherited() {
return "inherit".equals(sshAgentEnv);
}
/** {@link #allow} as a set, for membership checks. */
@@ -1218,13 +1489,13 @@ public record FleetConfig(
*
* <p>Package-private (not {@code private}) so a test can assert every key here is documented in
* {@code fleetd.example.yaml} — the only committed description of the config schema, since
* {@code bridged.yaml} itself is gitignored.
* {@code fleetd.yaml} itself is gitignored.
*/
static final Set<String> KNOWN_TOP_LEVEL_KEYS = Set.of(
"bind", "herdrSocket", "profiles", "guard", "worktreeRoot",
"bind", "herdrSocket", "memberHerdrSocket", "profiles", "guard", "worktreeRoot",
"lifecycle", "spawnReadyTimeoutMs", "spawnReadyPollMs", "broker", "primary", "fleet",
"leadHeartbeat", "health", "placement", "auth", "configReload", "quarantineCooldownSeconds",
"memberCredentials");
"memberCredentials", "coordinator", "worktreeGroup", "memberLoginShell");
/** Load and validate config from {@code path}. */
public static FleetConfig load(Path path) {
@@ -1236,6 +1507,7 @@ public record FleetConfig(
rejectDuplicateMemberSlots(yaml);
rejectNegativeMaxLoad(yaml);
rejectAutoCompactWindowOutOfRange(yaml);
rejectMalformedProfilePatterns(yaml);
rejectUnknownKind(yaml);
rejectUnknownAuthMode(yaml);
rejectUnknownPlacement(yaml);
@@ -1247,7 +1519,7 @@ public record FleetConfig(
rejectUnknownPlacementPolicy(cfg.placement());
return cfg.withDefaults();
} catch (IOException e) {
throw new UncheckedIOException("cannot read bridged config at " + path, e);
throw new UncheckedIOException("cannot read fleetd config at " + path, e);
}
}
@@ -1584,6 +1856,59 @@ public record FleetConfig(
}
}
/**
* Reject a profile whose {@code errorPattern} (fleetd #201 Unit 5) or {@code exhaustedPattern}
* (CB-578 stage A) is not a valid Java regex, naming the profile, the key, and the parser's own
* message.
*
* <p>Unset/{@code null} means, for {@code errorPattern}, "use {@code CompletionResolver}'s
* built-in {@code (?i)\bAPI Error\s*:} compatibility pattern", and for {@code exhaustedPattern},
* "opt out of that classification" — either way it passes silently. A profile that DOES set
* either key gets it compiled once at daemon startup ({@code Fleetd.main}) — an uncaught
* {@link java.util.regex.PatternSyntaxException} there crashes startup without naming which
* profile or key is at fault (fleetd #273: this happened for {@code exhaustedPattern}, which had
* no validator here even though its sibling {@code errorPattern} did). Validate eagerly here
* instead, at config load, the same "fail loud at load, not lazily later" reasoning as
* {@link #rejectAutoCompactWindowOutOfRange}. Both keys are checked from a single load, and any
* failures from either are collected together into one message.
*
* @param yaml the raw config text
* @throws IllegalStateException when any profile's {@code errorPattern} or
* {@code exhaustedPattern} fails to compile
*/
static void rejectMalformedProfilePatterns(String yaml) {
Map<?, ?> raw;
try {
raw = YAML.readValue(yaml, Map.class);
} catch (IOException | IllegalArgumentException e) {
return; // a malformed file is reported by the real parse, not here
}
if (raw == null || !(raw.get("profiles") instanceof Map<?, ?> profiles)) {
return;
}
List<String> bad = new ArrayList<>();
for (Map.Entry<?, ?> e : profiles.entrySet()) {
if (!(e.getValue() instanceof Map<?, ?> p)) {
continue;
}
for (String key : List.of("errorPattern", "exhaustedPattern")) {
if (!(p.get(key) instanceof String pattern) || pattern.isBlank()) {
continue;
}
try {
Pattern.compile(pattern);
} catch (PatternSyntaxException ex) {
bad.add("profiles." + e.getKey() + "." + key + " (\"" + pattern + "\"): " + ex.getMessage());
}
}
}
bad.sort(String::compareTo);
if (!bad.isEmpty()) {
throw new IllegalStateException("refusing to start: malformed pattern — "
+ String.join("; ", bad));
}
}
/** The peer kinds this build has an adapter for — {@link Profile#kind()}'s only valid values. */
private static final Set<String> KNOWN_KINDS = Set.of(Profile.KIND_CLAUDE_CODE, Profile.KIND_OPENCODE);
@@ -1834,15 +2159,22 @@ public record FleetConfig(
// guard/lifecycle, an absent block is not a safe "feature off" default here, it is a gap. It
// is deliberately not pre-populated with a Java-side name list (that would just reintroduce
// the hardcoded-list defect this record replaces); the block must be configured in
// bridged.yaml to protect anything. See fleetd.example.yaml's memberCredentials: comment.
// fleetd.yaml to protect anything. See fleetd.example.yaml's memberCredentials: comment.
// CB-633: policy stays deny-by-default here — the allow-list scrub is opt-in, because it is
// stricter than today's behaviour (it blanks every non-derived name, not just known ones)
// and an upgrade must not change what a running deployment's members inherit.
MemberCredentials mc = memberCredentials != null ? memberCredentials
: new MemberCredentials(null, List.of(), List.of());
return new FleetConfig(b, herdrSocket, profiles, g, worktreeRoot, l, timeout, pollMs,
// coordinator is left as-is, like broker/primary above: null keeps no LeadMailbox opened,
// and this ticket's Coordinator is config-only anyway (nothing yet reads it at startup).
// worktreeGroup is left as-is (fleetd #185 stage 3): null/blank is "off", and there is no
// sane non-null default — an OS group name is operator-specific.
// memberLoginShell is left as-is (fleetd #213), like worktreeGroup: null/blank is "not
// configured", and there is no sane non-null default — a member's login shell is
// operator-specific and only meaningful when memberHerdrSocket is also set.
return new FleetConfig(b, herdrSocket, memberHerdrSocket, profiles, g, worktreeRoot, l, timeout, pollMs,
broker, primary, f, leadHeartbeat, health, placementOrDefault, a, configReload,
quarantineCooldown, mc);
quarantineCooldown, mc, coordinator, worktreeGroup, memberLoginShell);
}
/**
@@ -1872,7 +2204,7 @@ public record FleetConfig(
/**
* Reject a lead-scan convention that a worker tab would also satisfy (CB-531).
*
* <p>The scan reads a tab label and concludes "a lead lives here". bridged also <em>writes</em>
* <p>The scan reads a tab label and concludes "a lead lives here". fleetd also <em>writes</em>
* tab labels — every member gets one rendered into its tab. Choose a lead {@code tabPrefix} that
* a member template matches and the daemon starts labelling its own members as leads, promoting
* the entire fleet to {@link dev.ltms.fleet.auth.Role#PRIMARY} with no message and no diff.
@@ -16,7 +16,7 @@ import java.util.Set;
* means its traffic would leave the subscription. That is a hard stop.</li>
* </ul>
*
* Both checks throw {@link GuardException} on violation. {@code bridged} calls
* Both checks throw {@link GuardException} on violation. {@code fleetd} calls
* {@link #assertWorker} before spawning a worker and {@link #assertPrimaryClean}
* against its own environment at startup.
*/
@@ -0,0 +1,270 @@
package dev.ltms.fleet.health;
import dev.ltms.fleet.herdr.Agent;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.msg.MessageService;
import dev.ltms.fleet.session.MemberSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.BiConsumer;
import java.util.function.LongSupplier;
import java.util.function.Supplier;
/** Slow whole-fleet evidence collection. It is deliberately separate from the delivery poller. */
public final class FleetHealthMonitor {
private static final Logger log = LoggerFactory.getLogger(FleetHealthMonitor.class);
/** Bounded attempts to run {@link #failTarget} for one transition. Never retried tick-to-tick (CB-580). */
static final int MAX_FAIL_TARGET_ATTEMPTS = 3;
// CB-641: Match the injector's 60s readiness gate so health allows a full first boot.
static final long READINESS_GRACE_NANOS = TimeUnit.SECONDS.toNanos(60);
/**
* fleetd #280: how long after a terminal transition to wait before the one bounded re-check
* fires. Must exceed the worst-case reverse-rendezvous {@code fleet_ask} window (55-115s, see
* {@code FleetMcp.ASK_DEFAULT_TIMEOUT_MS} / {@code FleetApp.MAX_ASK_TIMEOUT_MS}) so that, if the
* target was genuinely {@code ASKING} when {@code state} was first observed, its own ask has had
* time to lapse (clearing {@code Task#question} back to {@code null}) before this fires.
*/
static final long ASK_LAPSE_RECHECK_DELAY_SECONDS = 120;
private final AgentControl agents;
private final Supplier<List<MemberSession>> roster;
private final MessageService messages;
private final ScheduledExecutorService scheduler;
private final LongSupplier clock;
private final long intervalSeconds;
private final long workingSuspectAfterNanos;
private final BiConsumer<String, String> failTarget;
private final Map<String, HealthPrior> priors = new HashMap<>();
/**
* The live classification per member, and the only one of this class's three maps that more
* than one scheduler task touches. {@code tick} writes it (and prunes it to the roster);
* fleetd #280's delayed {@link #recheckTerminalTarget} reads it from its own separate scheduled
* task. Both run on the single-threaded scheduler {@code Fleetd} passes in today, so they are
* serialised — but nothing in this class enforces that, and an unsynchronised {@link HashMap}
* read racing a resize can spin a CPU forever rather than fail visibly. {@code priors} and
* {@code orphanStreaks} stay plain maps because {@code tick} is still their only toucher.
*/
private final Map<String, HealthState> states = new ConcurrentHashMap<>();
/**
* CB-643: consecutive ticks on which a target looked like an orphaned delegation. The fact
* {@link MessageService#hasOrphanedDelegation} reports is a true snapshot, but it can read true
* for one tick during an ordinary race — an async ticket exists before its virtual thread has
* reached {@code rendezvous.open()}, so for that instant nothing is accepted or queued behind
* it. {@code decide} maps the field straight to {@code DELEGATION_ORPHANED} with no cross-tick
* smoothing of its own, so a single racy read would log a fault that clears on the next tick.
* Requiring two consecutive observations costs one interval of latency on a real orphan and
* removes that false positive entirely.
*/
private final Map<String, Integer> orphanStreaks = new HashMap<>();
/** How many consecutive ticks a target must look orphaned before health reports it (CB-643). */
static final int ORPHAN_CONFIRM_TICKS = 2;
// CB-643: every HealthSnapshot field now carries real evidence. The NOT_YET_OBSERVED placeholder
// that stood in for 7 of the 12 is gone, and with it the reason 8 of the 9 fault states were
// unreachable — GONE and NEVER_READY included, which is what kept CB-580's failTarget from ever
// firing. Do not reintroduce a constant here: a field with no publisher is a dead state, and the
// tests pass either way, so nothing else will tell you.
/**
* @param failTarget CB-568's idempotent target-wide failure operation (e.g. {@code messages::abandon}),
* invoked once when a member transitions into a terminal health state. Required —
* there is deliberately no defaulting overload; a caller that does not want the
* fail-tickets-on-terminal-health behavior must pass an explicit inert value (see
* {@code TestTurnTokens.inert} / {@code FleetMcp.CapacitySource.none()} for the pattern).
*/
public FleetHealthMonitor(AgentControl agents, Supplier<List<MemberSession>> roster, MessageService messages,
ScheduledExecutorService scheduler, LongSupplier clock, long intervalSeconds,
long workingSuspectAfterSeconds, BiConsumer<String, String> failTarget) {
this.agents = agents;
this.roster = roster;
this.messages = messages;
this.scheduler = scheduler;
this.clock = clock;
this.intervalSeconds = intervalSeconds;
this.workingSuspectAfterNanos = TimeUnit.SECONDS.toNanos(workingSuspectAfterSeconds);
this.failTarget = Objects.requireNonNull(failTarget, "failTarget");
}
/** Pure per-member decision seam. */
static HealthDecision decide(HealthSnapshot snapshot, HealthPrior prior, long nowNanos) {
return FleetHealth.decide(snapshot, prior, nowNanos);
}
public void start() { scheduler.schedule(this::tick, intervalSeconds, TimeUnit.SECONDS); }
public void stop() { scheduler.shutdownNow(); }
// Package-private so tests can run one tick without waiting.
void tick() {
try {
List<MemberSession> rosterNow = roster.get(); // One in-memory roster snapshot for this tick.
List<Agent> agentsNow;
boolean controlLinkDown = false;
try {
agentsNow = agents.list(); // Exactly one list call for this complete observation.
} catch (HerdrException error) {
agentsNow = List.of();
controlLinkDown = true;
log.warn("fleet health control link unavailable; classifying roster", error);
}
Map<String, Agent> live = new HashMap<>();
for (Agent agent : agentsNow) live.put(agent.terminalId(), agent);
HashSet<String> current = new HashSet<>();
long nowNanos = clock.getAsLong();
for (MemberSession session : rosterNow) {
current.add(session.terminalId());
Agent agent = live.get(session.terminalId());
AgentStatus status = agent == null ? AgentStatus.UNKNOWN : agent.status();
boolean accepted = messages.hasAcceptedDelivery(session.terminalId());
boolean present = agent != null;
boolean targetNotFound = !controlLinkDown && !present
&& session.state() != MemberSession.State.SPAWNING;
boolean readinessGraceElapsed = nowNanos - session.spawnedAtNanos() >= READINESS_GRACE_NANOS;
boolean stalled = session.state() == MemberSession.State.BUSY
&& nowNanos - session.lastActivityAtNanos() >= workingSuspectAfterNanos;
// CB-643: the three message-layer facts CB-640 published. Read them here rather than
// leaving them false — that constant is what made 8 of the 9 fault states dead.
boolean queuedDelivery = messages.hasQueuedDelivery(session.terminalId());
boolean replyStranded = messages.hasStrandedReply(session.terminalId());
boolean orphanedDelegation = confirmOrphan(session.terminalId(),
messages.hasOrphanedDelegation(session.terminalId()));
HealthSnapshot snapshot = new HealthSnapshot(session.state(), status, accepted, queuedDelivery,
messages.hasInboxMessage(session.terminalId()), present, targetNotFound, controlLinkDown,
readinessGraceElapsed, orphanedDelegation, replyStranded, stalled);
HealthDecision decision = decide(snapshot, priors.getOrDefault(session.terminalId(), HealthPrior.NONE),
nowNanos);
priors.put(session.terminalId(), decision.prior());
reportTransition(session.terminalId(), decision.state());
}
priors.keySet().retainAll(current);
states.keySet().retainAll(current);
orphanStreaks.keySet().retainAll(current);
} catch (Throwable error) {
// Any unclassified collection failure must never kill the monitor's only scheduler task.
log.warn("fleet health collection failed; will retry next tick", error);
} finally {
if (!scheduler.isShutdown()) {
scheduler.schedule(this::tick, intervalSeconds, TimeUnit.SECONDS);
}
}
}
/**
* Debounce {@link MessageService#hasOrphanedDelegation} across ticks (CB-643). Returns true only
* once {@code observed} has held for {@link #ORPHAN_CONFIRM_TICKS} consecutive ticks; a single
* false reading resets the streak, so a transient race never reaches the classifier.
*/
private boolean confirmOrphan(String target, boolean observed) {
if (!observed) {
orphanStreaks.remove(target);
return false;
}
int streak = orphanStreaks.merge(target, 1, Integer::sum);
return streak >= ORPHAN_CONFIRM_TICKS;
}
void reportTransition(String target, HealthState next) {
HealthState previous = states.put(target, next);
if (previous == next) return;
if (fault(next)) {
log.warn("fleet health member={} state={} previous={}", target, next, previous);
} else if (previous != null && fault(previous)) {
log.info("fleet health member={} recovered state={} previous={}", target, next, previous);
}
// CB-580: a member entering GONE/NEVER_READY must not leave its waiting tickets pending
// forever. Fire exactly once per transition — never on a tick where the state is unchanged,
// which is what made the rejected commit call abandon() once per tick for as long as a
// member stayed terminal.
if (terminal(next)) {
failTerminalTarget(target, next);
scheduleTerminalRecheck(target, next);
}
}
private void failTerminalTarget(String target, HealthState state) {
String reason = "fleet health: member reached terminal state " + state.name();
RuntimeException last = null;
for (int attempt = 1; attempt <= MAX_FAIL_TARGET_ATTEMPTS; attempt++) {
try {
failTarget.accept(target, reason);
return;
} catch (RuntimeException error) {
last = error;
log.warn("fleet health: failTarget attempt {}/{} failed for member={} state={}",
attempt, MAX_FAIL_TARGET_ATTEMPTS, target, state, error);
}
}
log.warn("fleet health: giving up on failTarget for member={} state={} after {} attempts",
target, state, MAX_FAIL_TARGET_ATTEMPTS, last);
}
/**
* fleetd #280: schedule the one bounded, delayed follow-up for a terminal transition — never a
* per-tick retry (CB-580 rejected that shape; {@link #reportTransition} still fires
* {@link #failTerminalTarget} exactly once per transition, unconditionally on the tick loop).
* This is a single one-shot task, scheduled once per transition into GONE/NEVER_READY, so a
* member stuck terminal for the rest of its life gets exactly one extra attempt, not one per
* tick. See {@link #recheckTerminalTarget} for why the extra attempt is safe.
*/
private void scheduleTerminalRecheck(String target, HealthState state) {
if (scheduler.isShutdown()) return;
try {
scheduler.schedule(() -> recheckTerminalTarget(target, state),
ASK_LAPSE_RECHECK_DELAY_SECONDS, TimeUnit.SECONDS);
} catch (RuntimeException e) {
log.warn("fleet health: could not schedule terminal re-check for member={} state={}",
target, state, e);
}
}
/**
* fleetd #280: the delayed re-check {@link #scheduleTerminalRecheck} scheduled for one terminal
* transition. By now, a {@code fleet_ask} that was still open when {@code state} was first
* observed has had time to lapse on its own (see {@link #ASK_LAPSE_RECHECK_DELAY_SECONDS}),
* clearing {@code Task#question} back to {@code null} — which is exactly what
* {@link MessageService#abandon(String, String, boolean)}'s {@code sweepAsking=false} filter
* needs to finally match it. Calling {@link #failTerminalTarget} again is safe only because
* {@code sweepAsking} stays {@code false}: a task genuinely still {@code ASKING} is skipped
* exactly as it was on the very first attempt — this never fails a ticket whose ask has not yet
* lapsed.
*
* <p><strong>Guarded on "target is still classified {@code state}."</strong> Without this guard,
* a member that recovered (or was released and dropped from the roster) between the transition
* and this re-check would still take a blind {@code failTarget} call — reaching into whatever
* brand-new, unrelated turn it has since picked up and failing it too. {@link #states} already
* carries the live classification (updated every tick, pruned to the current roster on release),
* so a stale or recovered target simply reads as a mismatch here and this is a no-op.
*/
void recheckTerminalTarget(String target, HealthState state) {
if (states.get(target) != state) return;
failTerminalTarget(target, state);
}
private static boolean terminal(HealthState state) {
return state == HealthState.GONE || state == HealthState.NEVER_READY;
}
private static boolean fault(HealthState state) {
return switch (state) {
case NEVER_READY, GONE, TURN_BOUNDARY_LOST, ERROR_ON_SCREEN, STALL_SUSPECTED,
MUTE, REPLY_STRANDED, DELEGATION_ORPHANED, CONTROL_LINK_DOWN -> true;
default -> false;
};
}
public static String coverage(boolean enabled, boolean notificationConfigured) {
return !enabled ? "off" : notificationConfigured ? "full" : "detection-only";
}
}
@@ -38,6 +38,11 @@ public final class AgentControl {
this.herdr = herdr;
}
/** The herdr daemon this control object sends its agent calls to. */
public HerdrClient herdr() {
return herdr;
}
/** One agent-targeted call, translating a terminal id to its pane id (retrying once fresh). */
private JsonNode agentCall(String method, String target, Map<String, Object> extra) {
String resolved = resolveTarget(target);
@@ -32,7 +32,7 @@ public enum AgentStatus {
};
}
/** Whether {@code bridged} may inject a message now without stepping on a live turn. */
/** Whether {@code fleetd} may inject a message now without stepping on a live turn. */
public boolean injectable() {
return this == IDLE || this == BLOCKED || this == DONE;
}
@@ -5,13 +5,13 @@ import com.fasterxml.jackson.databind.JsonNode;
/**
* Client face onto the herdr daemon (protocol 14, herdr 0.7.0).
*
* <p>This is the ONLY thing in {@code bridged} that speaks to herdr. Every method
* <p>This is the ONLY thing in {@code fleetd} that speaks to herdr. Every method
* maps to a herdr JSON-RPC call over its Unix domain socket. Requests are
* newline-delimited JSON with a <em>string</em> id; responses carry either a
* {@code result} object (whose {@code type} field discriminates the payload) or an
* {@code error} object.
*
* <p>Higher layers ({@code bridged}'s policy brain, REST endpoints, MCP adapters)
* <p>Higher layers ({@code fleetd}'s policy brain, REST endpoints, MCP adapters)
* depend on this interface, not on the socket. Tests substitute a fake; the
* {@code contract}-tagged suite exercises the real implementation against a running
* herdr to catch protocol drift.
@@ -0,0 +1,40 @@
package dev.ltms.fleet.herdr;
import java.util.Objects;
import java.util.function.Predicate;
/** Routes lead operations and member operations to their owning herdr daemon. */
public final class HerdrRouter implements AutoCloseable {
private final HerdrClient lead;
private final HerdrClient member;
private final AgentControl leadAgents;
private final AgentControl memberAgents;
private final WorkspaceControl leadSpaces;
private final WorkspaceControl memberSpaces;
private final Predicate<String> isLead;
public HerdrRouter(HerdrClient lead, HerdrClient member, Predicate<String> isLead) {
this.lead = Objects.requireNonNull(lead, "lead");
this.member = member != null ? member : lead;
this.isLead = Objects.requireNonNull(isLead, "isLead");
leadAgents = new AgentControl(this.lead);
memberAgents = this.member == this.lead ? leadAgents : new AgentControl(this.member);
leadSpaces = new WorkspaceControl(this.lead);
memberSpaces = this.member == this.lead ? leadSpaces : new WorkspaceControl(this.member);
}
public AgentControl leadAgents() { return leadAgents; }
public WorkspaceControl leadSpaces() { return leadSpaces; }
public AgentControl memberAgents() { return memberAgents; }
public WorkspaceControl memberSpaces() { return memberSpaces; }
public AgentControl agentsFor(String targetId) { return isLead.test(targetId) ? leadAgents : memberAgents; }
HerdrClient leadClient() { return lead; }
HerdrClient memberClient() { return member; }
@Override
public void close() {
lead.close();
if (member != lead) member.close();
}
}
@@ -39,18 +39,18 @@ import java.util.function.Supplier;
* <li>Worker spaces are excluded wholesale ({@code excludedWorkspaceLabels}), so a worker cannot
* become a lead by being placed — as a split, say — inside a matching tab.</li>
* <li>A worker cannot rename a tab: {@code tab.rename} is reachable only through
* {@link WorkspaceControl}, which no {@code bridge_*} tool exposes. The label is writable by
* {@link WorkspaceControl}, which no {@code fleet_*} tool exposes. The label is writable by
* the human at the terminal and by nobody the bridge is defending against.</li>
* <li>The label is a <em>name</em>, not a capability. What a pane may do is decided by
* {@code Authz} against the role {@code CallerResolver} returns; a tab that calls itself a
* lead still cannot act as one unless the daemon's own registry agrees.</li>
* </ol>
*
* <p><strong>CB-558 — bridged now writes lead labels too.</strong> This class used to be able to say
* that bridged never renames a lead tab, so the label was always the human's own writing and there
* <p><strong>CB-558 — fleetd now writes lead labels too.</strong> This class used to be able to say
* that fleetd never renames a lead tab, so the label was always the human's own writing and there
* was no round-trip from the daemon's rename back into its next decision.
* {@code dev.ltms.fleet.lead.LeadLauncher} ends that: an auto-launched lead is labelled by the
* daemon and found again by this scan. The trust direction above is unaffected — bridged writing a
* daemon and found again by this scan. The trust direction above is unaffected — fleetd writing a
* name for a lead it just started is not a pane promoting itself — but <em>staleness</em> becomes
* real: a label left behind by a session that has since died would read as a live lead forever.
* This scanner does not solve that (its job is naming, and a stale name costs nothing here); the
@@ -0,0 +1,147 @@
package dev.ltms.fleet.herdr;
import com.fasterxml.jackson.databind.JsonNode;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.OptionalLong;
import java.util.Set;
/**
* Resolves which herdr pane a process belongs to — the herdr half of connection-based MCP
* identity (CB-105). Given the PID that opened an MCP connection, {@link #terminalForPid} finds
* the agent pane whose process tree contains it, so {@code fleetd} can tell <em>which worker</em>
* is calling without the worker sending anything spoofable.
*
* <p>herdr owns the PID→pane truth: {@code pane.process_info} reports each pane's {@code shell_pid}
* and foreground process PIDs. A pid that is neither of those directly — e.g. a grandchild a
* worker spawned, such as a {@code python3} or {@code curl} helper that opens its own MCP
* connection — is resolved by walking its ancestry (via {@link ParentResolver}) up to the root and
* matching any ancestor against a pane's {@code shell_pid} or foreground pids (CB-161). Without
* this walk such a pid matches no pane, and the caller falls through to loopback-trust and is
* resolved as the primary — a worker→primary privilege escalation.
*
* <p>This scans agent panes; a spawn-time {@code pid→terminal} cache is the obvious optimization
* once wired into {@code ClaudeCodeLauncher}.
*
* <p>CB-185 split the fleet across two herdr daemons — lead operations on one, members on the
* other ({@code memberHerdrSocket}). A caller's pane can live on <em>either</em> daemon (a lead's
* MCP connection resolves against the lead daemon; a member's against the member daemon), so this
* must be able to search more than one client. {@link #PaneLocator(HerdrClient, HerdrClient)}
* searches the lead client first, then the member client, and collapses to a single scan when the
* two are the same object (the historical single-daemon deployment). The caller's ancestor set is
* computed once per {@link #terminalForPid} call and reused across every client searched — it
* does not depend on which daemon a pane happens to live on.
*/
public final class PaneLocator {
/**
* Bound on how many ancestor generations {@link #ancestorsOf} walks. This runs on every MCP
* call, so a cycle or a pathologically deep process tree must not hang identity resolution;
* 32 generations is far more than any real worker→helper process tree needs.
*/
private static final int MAX_ANCESTRY_DEPTH = 32;
private final List<HerdrClient> herdrs;
private final ParentResolver parentResolver;
/** Search only this client — the single-daemon deployment. */
public PaneLocator(HerdrClient herdr) {
this(herdr, ParentResolver.PROCESS_HANDLE);
}
/** Search only this client, resolving ancestry through {@code parentResolver} — for tests. */
public PaneLocator(HerdrClient herdr, ParentResolver parentResolver) {
this.herdrs = List.of(herdr);
this.parentResolver = parentResolver;
}
/**
* Search {@code lead} first, then {@code member} — the two-daemon deployment (CB-185). When
* the caller passes the same client for both (no {@code memberHerdrSocket} configured), this
* collapses to one client and one scan, exactly {@link #PaneLocator(HerdrClient)}'s behaviour.
*/
public PaneLocator(HerdrClient lead, HerdrClient member) {
this(lead, member, ParentResolver.PROCESS_HANDLE);
}
/** Two-daemon deployment, resolving ancestry through {@code parentResolver} — for tests. */
public PaneLocator(HerdrClient lead, HerdrClient member, ParentResolver parentResolver) {
this.herdrs = lead == member ? List.of(lead) : List.of(lead, member);
this.parentResolver = parentResolver;
}
/**
* The {@code terminal_id} of the agent pane whose process tree contains {@code pid}, or
* {@code null} if no agent pane on any searched daemon owns it (e.g. the caller is the
* primary, or off-host).
*/
public String terminalForPid(long pid) {
if (pid <= 0) {
return null;
}
Set<Long> ancestry = ancestorsOf(pid);
for (HerdrClient herdr : herdrs) {
String terminal = terminalForPid(herdr, ancestry);
if (terminal != null) {
return terminal;
}
}
return null;
}
/**
* {@code pid} itself plus its ancestor chain, walked through {@link #parentResolver} up to
* {@link #MAX_ANCESTRY_DEPTH} generations or pid 1, whichever comes first. A vanished ancestor
* ({@link ParentResolver#parentOf} returning empty) ends the walk without error — it just means
* the chain is shorter than the bound. A cycle in a fake resolver is caught by the "already
* seen" check and also ends the walk, so this can never loop.
*/
private Set<Long> ancestorsOf(long pid) {
Set<Long> ancestry = new LinkedHashSet<>();
long current = pid;
for (int depth = 0; depth < MAX_ANCESTRY_DEPTH; depth++) {
if (current <= 0 || !ancestry.add(current)) {
break; // vanished/invalid pid, or a cycle back to a pid already recorded
}
if (current == 1) {
break; // reached the root of the process tree
}
OptionalLong parent = parentResolver.parentOf(current);
if (parent.isEmpty()) {
break; // vanished ancestor — not an error, just the end of the chain
}
current = parent.getAsLong();
}
return ancestry;
}
private static String terminalForPid(HerdrClient herdr, Set<Long> ancestry) {
for (JsonNode pane : herdr.call("pane.list", Map.of()).path("panes")) {
String paneId = pane.path("pane_id").asText(null);
if (paneId != null && paneOwnsAnyOf(herdr, paneId, ancestry)) {
return pane.path("terminal_id").asText(null);
}
}
return null;
}
private static boolean paneOwnsAnyOf(HerdrClient herdr, String paneId, Set<Long> ancestry) {
JsonNode info;
try {
info = herdr.call("pane.process_info", Map.of("pane_id", paneId)).path("process_info");
} catch (HerdrException e) {
return false; // pane vanished mid-scan — just skip it
}
if (ancestry.contains(info.path("shell_pid").asLong(-1))) {
return true;
}
for (JsonNode p : info.path("foreground_processes")) {
if (ancestry.contains(p.path("pid").asLong(-1))) {
return true;
}
}
return false;
}
}
@@ -0,0 +1,21 @@
package dev.ltms.fleet.herdr;
import java.util.OptionalLong;
/**
* Resolves a pid's parent pid — the seam {@link PaneLocator} walks a process's ancestry through,
* so its tests can drive the walk from a fake pid→parent map instead of spawning real processes.
*
* <p>{@link #PROCESS_HANDLE} is the production implementation, backed by {@link ProcessHandle}.
*/
public interface ParentResolver {
/** The parent pid of {@code pid}, or empty if {@code pid} is gone or has no known parent. */
OptionalLong parentOf(long pid);
/** Production resolver: asks the JVM's {@link ProcessHandle} view of the OS process tree. */
ParentResolver PROCESS_HANDLE = pid -> ProcessHandle.of(pid)
.flatMap(ProcessHandle::parent)
.map(parent -> OptionalLong.of(parent.pid()))
.orElse(OptionalLong.empty());
}
@@ -7,7 +7,7 @@ import com.fasterxml.jackson.databind.JsonNode;
* dedicated worker space so they never split or clutter the user's real work spaces.
*
* @param workspaceId herdr's stable id (e.g. {@code "w4"})
* @param label display label shown in herdr's UI (e.g. {@code "bridged-workers"})
* @param label display label shown in herdr's UI (e.g. {@code "fleetd-workers"})
* @param activeTabId the workspace's currently-focused tab, or {@code null}
*/
public record Workspace(String workspaceId, String label, String activeTabId) {
@@ -0,0 +1,35 @@
package dev.ltms.fleet.inject;
import java.util.regex.Pattern;
/**
* Per-target lookup for a profile's configured backend-error pattern (fleetd#201 / #227): how
* {@link CompletionResolver} recognizes a backend that failed a turn outright (a credential
* outage, a provider 5xx) from whatever ends up on the member's pane, apart from a genuine
* completion.
*
* <p>The pattern is always profile config, never a vendor string in Java source: every backend
* words its failure differently, so a hardcoded sentence would only ever match one of them. A
* target with no configured pattern is not "off" the way {@link ExhaustedPatternLookup#none()}
* is — {@link CompletionResolver} falls back to its narrow {@code (?i)\bAPI Error\s*:}
* compatibility pattern instead, so classification still happens, just without a profile-specific
* match. {@link #legacy()} is the explicit stand-in existing (pre-fleetd#201) callers pass to get
* exactly that fallback-only behavior until a caller wires a real, profile-driven lookup
* (fleetd#201 Unit 5).
*/
@FunctionalInterface
public interface BackendErrorPatternLookup {
/** The compiled pattern configured for {@code target}'s profile, or {@code null} if none. */
Pattern patternFor(String target);
/**
* Legacy lookup — no profile has a configured pattern, so {@link CompletionResolver} classifies
* every target using only its built-in {@code (?i)\bAPI Error\s*:} compatibility pattern. The
* explicit stand-in existing constructors pass so their behavior is unchanged until a caller
* wires a real, profile-driven lookup.
*/
static BackendErrorPatternLookup legacy() {
return target -> null;
}
}
@@ -0,0 +1,35 @@
package dev.ltms.fleet.inject;
/**
* Notified when {@link CompletionResolver} actually delivers a typed backend-error classification
* to a waiting send (fleetd#201 / #227) — never on a race that lost. {@link CompletionResolver}
* calls this only after {@code Rendezvous.resolveFailure} returns {@code true} for that exact
* waiter, mirroring the win-only race rule {@link ExhaustionSink} already uses.
*
* <p>The public send result is unchanged by this classification — it is still a failed send
* ({@code Rendezvous.Kind#FAILED}); this sink is the internal seam a later stage (fleetd#201 Unit
* 5, the policy that cools a credential off after two such failures in 60 seconds and tells the
* lead) consumes. {@link CompletionResolver} knows only {@code target} (a herdr terminal id); it
* has no notion of profiles or credentials, so mapping {@code target} to whatever should be
* quarantined is entirely the sink's job — exactly like {@link ExhaustionSink}.
*/
@FunctionalInterface
public interface BackendErrorSink {
/**
* @param target the herdr terminal id whose turn was classified as a backend error
* @param matchedLine the single pane line that matched the backend-error pattern
* @param reason the full failure reason carried by the classification (the matched line
* plus whatever pane context the classifying path attaches)
*/
void onBackendError(String target, String matchedLine, String reason);
/**
* Inert sink — nothing happens on a backend-error classification. The explicit stand-in a
* caller (or a test not exercising this feature) passes instead of a defaulting overload,
* exactly like {@link ExhaustionSink#none()}.
*/
static BackendErrorSink none() {
return (target, matchedLine, reason) -> { };
}
}
@@ -0,0 +1,696 @@
package dev.ltms.fleet.inject;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.TurnToken;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.LongSupplier;
import java.util.function.Function;
import java.util.regex.Pattern;
/**
* The CB-106 completion fallback: bridges the {@link Injector}'s turn-completion signal to the
* {@link Rendezvous} so a blocking {@code fleet_send} resolves even when the worker finishes its
* task without ever calling {@code fleet_reply} — the common case for a real delegated coding task.
*
* <p>On a confirmed {@code working → idle} boundary it scrapes the worker's recent transcript and
* resolves the awaiting send with that tail (a {@link Rendezvous.Kind#COMPLETION} resolution, so the
* caller can tell a scrape from a structured reply). It scrapes only when a send is actually waiting
* — a fleet worker's own turns, or a send that already timed out, cost no herdr traffic. An explicit
* {@code fleet_reply} that raced in first wins; {@link Rendezvous#resolveCompletion} is then a no-op.
*
* <p>It also handles the CB-109 stall signal ({@link #onTurnFailed}): a worker that ran a turn then
* wedged in an {@code unknown} state resolves the send as a failure (with the error screen as
* context) rather than leaving it to time out.
*
* <p>The scrape is cleaned to the last {@code ⏺} assistant block (stripping TUI chrome) and guarded
* against misattribution (CB-115): the pane content is baselined on delivery ({@link #onDelivered}),
* and a completion whose scrape is unchanged from that baseline — the previous turn's wind-down
* sampled as this turn's boundary on a rapid back-to-back send — is suppressed rather than resolving
* the send with a stale answer.
*
* <p><strong>Waiter-specific resolution (CB-116).</strong> On delivery we also capture the exact
* {@link Rendezvous} waiter this turn belongs to, and the completion/failure fallbacks resolve
* <em>that</em> waiter — never "whatever send is waiting now". A completion fallback runs on a virtual
* thread and can land after the worker's {@code fleet_reply} already resolved the turn and the
* <em>next</em> send opened its own waiter on the same session; resolving the current waiter would
* then deliver turn N's stale scrape as turn N+1's answer. Targeting the captured waiter makes a late
* completion a harmless no-op (its waiter is already done) instead of a cross-turn stale reply.
*
* <p>Wired as the {@link Injector}'s {@link TurnListener}; the handlers hand off to a virtual thread
* so the scrape's herdr round-trip never stalls the status poller. The captured waiter is read on the
* poller thread (before any next-turn delivery can overwrite it) and passed into the virtual thread.
*/
public final class CompletionResolver implements TurnListener {
private static final Logger log = LoggerFactory.getLogger(CompletionResolver.class);
/**
* herdr {@code agent.read} source for the completion scrape. {@code recent} returns the tail of
* the transcript (the worker's last output), which is what a delegator wants when the worker
* didn't structure a reply.
*/
static final String SCRAPE_SOURCE = "recent";
/** Cap the scraped tail so a long transcript can't return an unbounded blob. */
static final int MAX_SCRAPE_CHARS = 4000;
/**
* fleetd#164: the floor below which a {@code BUSY -> DONE} transition cannot be real work. A
* backend that rejects a turn outright (e.g. an HTTP 400 from the model, before the worker read
* a single file or produced a token) drives the exact same confirmed {@code working -> idle}
* transition a genuine completion does — just in about a second instead of the many seconds a
* real turn costs. {@link #onTurnComplete} cannot tell those two cases apart from the transition
* alone, so a turn that settles inside this floor is treated as a crash signature and resolved
* as a failure, never as a (possibly empty) success.
*/
public static final long MIN_TURN_NANOS = Duration.ofSeconds(2).toNanos();
/**
* fleetd#164 (part 2): one stable, explicit backend-failure marker seen on a Claude Code pane
* when the backend itself rejected the turn (e.g. {@code "API Error: 400 invalid request body"}).
* Kept deliberately narrow — a growing list of ad-hoc error strings rots as backends change their
* wording; broader backend-error surfacing is out of scope here (fleetd#164 point 3).
*/
private static final Pattern BACKEND_ERROR = Pattern.compile("(?i)\\bAPI Error\\s*:");
private static final String CLIPPED_PANE_TAIL_MARKER =
"[Pane tail clipped: member did not call fleet_reply.]";
/** A pane echo must be this large before it can replace a completion report. */
static final int ECHO_MIN_CHARS = 400;
/** Normalised TUI chrome may add this many characters to an otherwise echoed brief. */
static final int MAX_ECHO_EXCESS_CHARS = 160;
/** The explicit result returned instead of a lead's echoed injected brief. */
public static final String NO_REPORT_PREFIX = "[no report — the member ended its turn without fleet_reply, "
+ "and the pane still shows the injected brief. Nothing was produced on the pane. Check the "
+ "member's worktree and branch for committed work before re-delegating.";
private final AgentControl agents;
private final Rendezvous rendezvous;
private final ExhaustedPatternLookup exhaustedPatterns;
private final ExhaustionSink exhaustionSink;
private final BackendErrorPatternLookup backendErrorPatterns;
private final BackendErrorSink backendErrorSink;
private final LongSupplier nowNanos;
private final Function<String, WorktreeBranch> worktreeBranches;
/** Known member location, used only to guide a lead after an echoed brief. */
public record WorktreeBranch(String worktree, String branch) {
}
/**
* Per-target record of the turn currently in flight: the exact {@link Rendezvous} waiter its
* delivering send opened, plus the assistant block present when it was delivered.
*
* <p>The {@code waiter} is what makes a late fallback safe (CB-116): we resolve it, not "whoever
* is waiting now", so a completion that fires after the next send has opened its own waiter is a
* no-op rather than a cross-turn stale reply. The {@code baseline} is the CB-115 staleness
* reference: a completion scrape equal to it means the worker produced no new output (the previous
* turn's wind-down sampled as this boundary), so it is suppressed. Overwritten on each delivery;
* cleared when the turn resolves. Package-private so tests can capture and replay a specific turn.
*
* <p>{@code deliveredAtNanos} (fleetd#164) is the {@link #nowNanos} reading taken at delivery —
* the other half of the {@link #MIN_TURN_NANOS} floor check, compared against a fresh reading at
* resolution time.
*/
record InFlight(CompletableFuture<Rendezvous.Resolution> waiter, String baseline, long deliveredAtNanos,
String injectedText) {
/**
* Convenience for tests exercising scrape/suppression logic that don't care about turn
* timing: back-dates the delivery far enough that {@link #MIN_TURN_NANOS} can never fire.
* Not used by production code — {@link #captureBaseline} always records a real reading.
*/
InFlight(CompletableFuture<Rendezvous.Resolution> waiter, String baseline) {
this(waiter, baseline, Long.MIN_VALUE / 2, null);
}
InFlight(CompletableFuture<Rendezvous.Resolution> waiter, String baseline, long deliveredAtNanos) {
this(waiter, baseline, deliveredAtNanos, null);
}
}
private final ConcurrentHashMap<String, InFlight> inFlight = new ConcurrentHashMap<>();
/**
* @param exhaustedPatterns CB-578 stage A: per-target lookup for a profile's configured
* usage-limit refusal pattern. Required — there is deliberately no
* defaulting overload; a caller that does not want the classification
* must pass an explicit inert value ({@link ExhaustedPatternLookup#none()}).
* @param exhaustionSink CB-578 stage B: notified when a {@code BACKEND_EXHAUSTED}
* classification actually resolves a waiter. Required for the same
* reason as {@code exhaustedPatterns} — pass {@link ExhaustionSink#none()}
* to opt out.
*
* <p>Transition constructor (fleetd#201 Unit 1): delegates to the full constructor below with
* {@link BackendErrorPatternLookup#legacy()} and {@link BackendErrorSink#none()}, so every
* existing caller keeps today's behavior — the narrow built-in {@code API Error:} match, no
* sink notified — until a caller wires a real, profile-driven backend-error lookup and sink
* (fleetd#201 Unit 5).
*/
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink) {
this(agents, rendezvous, exhaustedPatterns, exhaustionSink,
BackendErrorPatternLookup.legacy(), BackendErrorSink.none(), System::nanoTime);
}
/** Production constructor with a lookup for the member worktree and branch. */
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink, Function<String, WorktreeBranch> worktreeBranches) {
this(agents, rendezvous, exhaustedPatterns, exhaustionSink,
BackendErrorPatternLookup.legacy(), BackendErrorSink.none(), System::nanoTime, worktreeBranches);
}
/**
* Transition constructor (fleetd#201 Unit 1): same legacy backend-error defaults as the 4-arg
* constructor above, but with the injectable clock. Kept so existing fleetd#164 timing tests
* (and any other caller that wants a controllable clock but not the backend-error feature)
* keep compiling unchanged.
*/
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink, LongSupplier nowNanos) {
this(agents, rendezvous, exhaustedPatterns, exhaustionSink,
BackendErrorPatternLookup.legacy(), BackendErrorSink.none(), nowNanos);
}
/**
* Production constructor (fleetd#201 Unit 1): adds the target-keyed backend-error pattern
* lookup and sink alongside the existing exhaustion pair. Required, like {@code exhaustedPatterns}
* and {@code exhaustionSink} — pass {@link BackendErrorPatternLookup#legacy()} and
* {@link BackendErrorSink#none()} to opt out.
*
* @param backendErrorPatterns fleetd#201: per-target lookup for a profile's configured
* backend-error pattern; a target with none configured is matched
* against the built-in compatibility pattern instead (never "off").
* @param backendErrorSink fleetd#201: notified when a backend-error classification actually
* resolves a waiter — never on a race that lost.
*/
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink, BackendErrorPatternLookup backendErrorPatterns,
BackendErrorSink backendErrorSink) {
this(agents, rendezvous, exhaustedPatterns, exhaustionSink, backendErrorPatterns, backendErrorSink,
System::nanoTime, _ -> null);
}
/**
* Test constructor with an injectable clock (fleetd#164), matching the {@code LongSupplier}
* pattern {@link dev.ltms.fleet.session.SessionManager} and {@link dev.ltms.fleet.msg.MessageService}
* already use: lets a test place a turn's delivery and its resolution at an exact, controllable
* distance apart around the {@link #MIN_TURN_NANOS} floor, without a real sleep. Public (rather
* than package-private like those two) because callers that wire a full {@code MessageService}
* fixture — e.g. {@code MessageServiceTest} — construct this resolver directly from another
* package.
*/
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink, BackendErrorPatternLookup backendErrorPatterns,
BackendErrorSink backendErrorSink, LongSupplier nowNanos) {
this(agents, rendezvous, exhaustedPatterns, exhaustionSink, backendErrorPatterns, backendErrorSink,
nowNanos, _ -> null);
}
/** Full constructor with injectable clock and member location lookup. */
public CompletionResolver(AgentControl agents, Rendezvous rendezvous, ExhaustedPatternLookup exhaustedPatterns,
ExhaustionSink exhaustionSink, BackendErrorPatternLookup backendErrorPatterns,
BackendErrorSink backendErrorSink, LongSupplier nowNanos,
Function<String, WorktreeBranch> worktreeBranches) {
this.agents = agents;
this.rendezvous = rendezvous;
this.exhaustedPatterns = Objects.requireNonNull(exhaustedPatterns, "exhaustedPatterns");
this.exhaustionSink = Objects.requireNonNull(exhaustionSink, "exhaustionSink");
this.backendErrorPatterns = Objects.requireNonNull(backendErrorPatterns, "backendErrorPatterns");
this.backendErrorSink = Objects.requireNonNull(backendErrorSink, "backendErrorSink");
this.nowNanos = Objects.requireNonNull(nowNanos, "nowNanos");
this.worktreeBranches = Objects.requireNonNull(worktreeBranches, "worktreeBranches");
}
@Override
public void onDelivered(String target, TurnToken token) {
// Capture the exact waiter this turn belongs to (CB-116) and snapshot the pane's pre-turn
// content — what it shows *before* the just-delivered turn produces output — as the staleness
// reference (CB-115). Done synchronously (like the delivering send itself) so both are in
// place before this turn's completion can fire.
captureBaseline(target, token);
}
/** Capture the in-flight turn: its waiter and pre-turn baseline (the testable core of {@link #onDelivered}). */
void captureBaseline(String target, TurnToken token) {
CompletableFuture<Rendezvous.Resolution> waiter = token.waiter();
if (waiter == null) {
inFlight.remove(target); // no send is waiting on this delivery — nothing to resolve later
return;
}
String baseline;
try {
// Clip to the same cap resolve() applies to the tail (line ~134): the CB-115 misattribution
// guard compares baseline.equals(tail), so both sides must be the same capped representation.
// An unclipped baseline vs a clipped tail would never match for a >MAX_SCRAPE_CHARS block,
// defeating the guard and letting a stale completion resolve the send.
baseline = clip(lastAssistantBlock(agents.read(target, SCRAPE_SOURCE)));
} catch (RuntimeException e) {
baseline = null; // fail open: no baseline ⇒ no suppression
log.debug("delivery baseline for {} failed: {}", target, e.getMessage());
}
inFlight.put(target, new InFlight(waiter, baseline, nowNanos.getAsLong(), token.injectedText()));
}
/** The turn currently baselined for {@code target}, or {@code null} — a test hook for the captureBaseline path. */
InFlight inFlight(String target) {
return inFlight.get(target);
}
@Override
public void onTurnComplete(String target) {
// Read the in-flight turn on the poller thread — before any next-turn delivery can overwrite
// it — then off-load the scrape (a herdr round-trip we must not block polling on) to a vthread.
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("completion-" + target).start(() -> resolve(target, turn));
}
/**
* Resolve the completed turn before adapter housekeeping can erase its rendered output. This is
* intentionally synchronous and used only when a post-turn context reset is enabled; the normal
* path remains off-loaded so polling is not blocked by a scrape.
*/
public void resolveBeforePostAction(String target) {
resolve(target, inFlight.get(target));
}
@Override
public void onTurnFailed(String target) {
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("turn-failed-" + target).start(() -> fail(target, turn, null));
}
@Override
public void onTurnFailed(String target, String reason) {
InFlight turn = inFlight.get(target);
Thread.ofVirtual().name("turn-failed-" + target).start(() -> fail(target, turn, reason));
}
/** Synchronous resolve (the unit-testable core of {@link #onTurnComplete}). */
void resolve(String target, InFlight turn) {
CompletableFuture<Rendezvous.Resolution> waiter = turn == null ? null : turn.waiter();
if (waiter == null || waiter.isDone()) {
// Nobody is blocked on THIS turn (it had no send, or its fleet_reply already won). Skip
// the scrape; resolving the current waiter here would be the CB-116 cross-turn stale reply.
inFlight.remove(target, turn);
return;
}
// fleetd#164: a BUSY -> DONE transition inside the floor cannot be real work — it's a crash
// signature (e.g. a backend HTTP 400 before the worker did anything), not a fast answer. Fail
// it before spending a scrape on the ordinary path; the reason still carries whatever is on
// screen, since that is usually the backend's own error.
long elapsedNanos = nowNanos.getAsLong() - turn.deliveredAtNanos();
if (elapsedNanos < MIN_TURN_NANOS) {
failTooFast(target, turn, waiter, elapsedNanos);
return;
}
String tail;
String assistantBlock = null;
String rawScrape = null;
int originalLength = 0;
boolean clipped = false;
boolean scrapeFailed = false;
try {
rawScrape = agents.read(target, SCRAPE_SOURCE);
assistantBlock = lastAssistantBlock(rawScrape);
originalLength = assistantBlock.strip().length();
clipped = originalLength > MAX_SCRAPE_CHARS;
tail = clip(assistantBlock);
} catch (RuntimeException e) {
log.warn("completion scrape for {} failed: {}", target, e.getMessage());
tail = "";
scrapeFailed = true;
}
// fleetd#164: a scrape nobody could read, and a scrape that read cleanly but produced nothing,
// both used to resolve the send as a SUCCESS carrying "" — indistinguishable from a worker that
// genuinely finished with nothing to say. That is the defect: fail loudly instead, naming the
// member, so a caller (including a lead deciding whether to delegate again) can tell a lost
// turn from a real empty answer.
if (scrapeFailed || tail.isEmpty()) {
// fleetd#211: lastAssistantBlock() found nothing usable — most often a pane with no ⏺
// marker at all, whose boundary scan then starts at the top of the raw screen and breaks
// immediately on the first line of TUI chrome (╭, │, ❯, …). Before giving up as a lost
// turn, run the same exhaustion/backend-error classification against the RAW scrape as a
// fallback, ONLY here. A pane that already yielded a usable block never reaches this
// branch, so the narrow (trimmed) match on the normal path below is completely unchanged
// — zero new false positives there. Every pane this fallback examines was already headed
// for the empty-scrape failure, so a wrong label here is strictly less bad than silently
// losing an exhaustion signal: the alternative outcome is already a failure, just one that
// never quarantines the credential. lastAssistantBlock stays the source of the reply
// TEXT everywhere else; only classification ever consults the raw scrape, and only here.
if (rawScrape != null && classifyRawScrapeFallback(target, turn, waiter, rawScrape)) {
return;
}
fail(target, turn, emptyScrapeReason(target, scrapeFailed));
return;
}
// Misattribution guard (CB-115): if the scrape is byte-identical to the pane content at
// delivery, this turn produced no new output — the boundary belongs to the previous turn's
// wind-down (common on rapid back-to-back sends). Suppress rather than resolve the send with
// a stale answer; the real fleet_reply (or a later genuine completion) resolves it instead.
String baseline = turn.baseline();
if (baseline != null && baseline.equals(tail)) {
log.debug("suppressing misattributed completion for {} (no output change since delivery)",
target);
return; // keep the in-flight record: a later genuine completion still needs it
}
// CB-578 stage A: a turn that ended with no fleet_reply AND whose scrape matches the
// backend's configured usage-limit pattern is a refusal, not an answer. Classify it as
// BACKEND_EXHAUSTED rather than handing the caller a scrape that reads like a real reply.
Pattern exhausted = exhaustedPatterns.patternFor(target);
String matchedLine = exhausted == null ? null : firstMatchingLine(assistantBlock, exhausted);
if (matchedLine != null) {
String reason = "backend exhausted (usage limit): " + matchedLine;
if (rendezvous.resolveExhausted(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("completion for {} classified BACKEND_EXHAUSTED (no fleet_reply; scrape "
+ "matched the profile's exhausted pattern): {}", target, reason);
// CB-578 stage B: only on the resolution that actually won the race — a late
// duplicate must never quarantine a credential twice for one refusal.
exhaustionSink.onExhausted(target, reason);
}
return;
}
// fleetd#164 (part 2) / fleetd#201: a scrape that read cleanly and produced content still
// isn't a real reply when that content is the backend's own rejection (e.g. an HTTP 400
// before the worker did any work). Classify it as a failure naming the member, rather than
// handing the caller a scrape that reads like a completed answer, and — only on the
// resolution that actually wins the race, mirroring the exhaustion sink above — notify the
// typed backend-error sink so a later stage can act on repeated failures.
String backendError = firstMatchingLine(assistantBlock, backendErrorPatternOrFallback(target));
if (backendError != null) {
// Carry the whole scrape, not just the matched line. The pattern is a heuristic: a member
// that forgot fleet_reply while reporting *about* a backend error matches it too. Failing
// is still right — the caller must not read a scrape as an answer — but dropping the rest
// of the pane would destroy the report, which is the same defect fleetd#164 is about.
String reason = "member " + target + " ended on a backend error: " + backendError
+ "\n--- pane tail ---\n" + tail;
if (rendezvous.resolveFailure(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("failing send to {} via turn-stall fallback: {}", target, reason);
// fleetd#201 Unit 1: only on the resolution that actually won the race — a late
// duplicate must never double-count one backend failure.
backendErrorSink.onBackendError(target, backendError, reason);
}
return;
}
String completion = clipped ? tail + "\n" + CLIPPED_PANE_TAIL_MARKER : tail;
if (echoesInjectedBrief(tail, turn.injectedText())) {
completion = noReportMessage(target) + (clipped ? "\n" + CLIPPED_PANE_TAIL_MARKER : "");
}
if (rendezvous.resolveCompletion(waiter, completion)) {
inFlight.remove(target, turn);
if (clipped) {
log.warn("completion scrape for {} clipped from {} chars to the {} char cap; "
+ "member did not call fleet_reply, so the pane tail is partial",
target, originalLength, MAX_SCRAPE_CHARS);
}
log.debug("resolved send to {} via turn-completion fallback ({} chars scraped)",
target, tail.length());
}
}
/**
* A full echoed brief is at least 400 normalised characters. A scrape that contains the brief may
* add no more than 160 normalised characters of TUI chrome. This accepts harmless status text, but
* preserves a real report that restates the full brief before adding substantive content.
*/
static boolean echoesInjectedBrief(String scrape, String injectedText) {
String normalScrape = normalize(scrape);
String normalInjected = normalize(injectedText);
if (normalScrape.length() < ECHO_MIN_CHARS || normalInjected.length() < ECHO_MIN_CHARS) {
return false;
}
if (normalInjected.contains(normalScrape)) {
return true;
}
return normalScrape.contains(normalInjected)
&& normalScrape.length() - normalInjected.length() <= MAX_ECHO_EXCESS_CHARS;
}
private static String normalize(String text) {
return text == null ? "" : text.toLowerCase().replaceAll("[^a-z0-9]+", "");
}
private String noReportMessage(String target) {
WorktreeBranch location = worktreeBranches.apply(target);
if (location == null || (location.worktree() == null && location.branch() == null)) {
return NO_REPORT_PREFIX + "]";
}
String locationText = location.worktree() == null ? "" : " worktree=" + location.worktree();
if (location.branch() != null) {
locationText += " branch=" + location.branch();
}
return NO_REPORT_PREFIX + locationText + "]";
}
/**
* fleetd#211: the raw-scrape fallback classification, run only when {@link #lastAssistantBlock}
* found nothing usable (see the call site in {@link #resolve}). Mirrors the two classifications
* the normal path already applies to the trimmed assistant block — exhaustion first, then the
* narrow {@link #BACKEND_ERROR} pattern — against {@code raw} instead, and reports whether one of
* them handled the turn (resolved the waiter or failed it) so the caller skips the empty-scrape
* failure. Never runs on the normal (non-empty-block) path, and never touches the reply text.
*/
private boolean classifyRawScrapeFallback(String target, InFlight turn,
CompletableFuture<Rendezvous.Resolution> waiter, String raw) {
Pattern exhausted = exhaustedPatterns.patternFor(target);
String matchedLine = exhausted == null ? null : firstMatchingLine(raw, exhausted);
if (matchedLine != null) {
String reason = "backend exhausted (usage limit): " + matchedLine;
if (rendezvous.resolveExhausted(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("completion for {} classified BACKEND_EXHAUSTED from the raw scrape (no "
+ "usable assistant block; no fleet_reply): {}", target, reason);
// CB-578 stage B: only on the resolution that actually won the race — a late
// duplicate must never quarantine a credential twice for one refusal.
exhaustionSink.onExhausted(target, reason);
}
return true;
}
String backendError = firstMatchingLine(raw, backendErrorPatternOrFallback(target));
if (backendError != null) {
// Carry the pane, not just the matched line — the same fleetd#164 rule the normal path
// above applies. Here it matters more, not less: the trimmed block was empty, so the raw
// scrape is the ONLY copy of whatever the member managed to say. Clipped to the same cap
// the normal path uses, since a raw screen has no boundary trimming to bound it.
String reason = "member " + target + " ended on a backend error: " + backendError
+ "\n--- pane tail ---\n" + clip(raw);
if (rendezvous.resolveFailure(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("failing send to {} via turn-stall fallback from the raw scrape: {}", target, reason);
// fleetd#201 Unit 1: only on the resolution that actually won the race.
backendErrorSink.onBackendError(target, backendError, reason);
}
return true;
}
return false;
}
/** Synchronous fail (the unit-testable core of {@link #onTurnFailed}). */
void fail(String target, InFlight turn) {
fail(target, turn, null);
}
/** Synchronous fail with an optional reason supplied by a dropped worker queue. */
void fail(String target, InFlight turn, String explicitReason) {
// A never-delivered readiness failure has no in-flight record but still has a blocked send;
// fall back to the currently-registered waiter (unambiguous — that send never completed, so
// no next turn exists to confuse it with).
CompletableFuture<Rendezvous.Resolution> waiter =
turn != null ? turn.waiter() : rendezvous.currentWaiter(target);
if (waiter == null || waiter.isDone()) {
inFlight.remove(target, turn); // nobody blocked on this worker — nothing to fail
return;
}
String reason = explicitReason;
if (reason == null || reason.isBlank()) {
try {
reason = clip(agents.read(target, SCRAPE_SOURCE));
} catch (RuntimeException e) {
reason = "";
}
if (reason.isBlank()) {
// No screen to scrape — either the worker is stuck (CB-109) or gone (CB-110).
reason = "worker did not reply; its turn ended in an unrecoverable state "
+ "(worker unreachable or stuck)";
}
}
if (rendezvous.resolveFailure(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("failing send to {} via turn-stall fallback: {}", target, reason);
}
}
/**
* fleetd#164 (floor) / fleetd#201 (classification): fail a turn that settled inside
* {@link #MIN_TURN_NANOS} — a crash signature (e.g. a backend HTTP 400 before the worker did
* anything) that a bare {@code BUSY -> DONE} transition cannot be told apart from a genuinely
* fast completion. Runs the same backend-error classification the normal and raw-scrape paths
* apply, against whatever is on screen right now: a match is a typed failure that notifies
* {@link #backendErrorSink} (only on the resolution that wins the race); a non-match stays the
* original generic too-fast failure, naming the member and both timings, with whatever the pane
* shows appended so the caller sees the cause, not just "it failed".
*/
private void failTooFast(String target, InFlight turn, CompletableFuture<Rendezvous.Resolution> waiter,
long elapsedNanos) {
String scrape;
try {
scrape = agents.read(target, SCRAPE_SOURCE);
} catch (RuntimeException e) {
scrape = "";
}
String clippedScrape = clip(scrape);
String baseReason = String.format(
"member %s went BUSY -> DONE in %dms (floor %dms) — too fast to be real work, most "
+ "likely a backend error before any work started",
target, elapsedNanos / 1_000_000, MIN_TURN_NANOS / 1_000_000);
String backendError = firstMatchingLine(scrape, backendErrorPatternOrFallback(target));
if (backendError != null) {
String reason = baseReason + ": " + clippedScrape;
if (rendezvous.resolveFailure(waiter, reason)) {
inFlight.remove(target, turn);
log.warn("failing send to {} via turn-stall fallback: {}", target, reason);
// fleetd#201 Unit 1: only on the resolution that actually won the race.
backendErrorSink.onBackendError(target, backendError, reason);
}
return;
}
fail(target, turn, clippedScrape.isBlank() ? baseReason : baseReason + ": " + clippedScrape);
}
/**
* fleetd#201: the pattern to classify a backend error against for {@code target} — its
* profile's configured {@link BackendErrorPatternLookup} entry when there is one, else the
* built-in {@link #BACKEND_ERROR} compatibility pattern. A target with no configured pattern is
* never "off": it always falls back to this narrow default, exactly like the classification
* behaved before fleetd#201 (Unit 5 reports a target relying on this fallback separately, as
* legacy-default coverage rather than full per-profile coverage).
*/
private Pattern backendErrorPatternOrFallback(String target) {
Pattern configured = backendErrorPatterns.patternFor(target);
return configured != null ? configured : BACKEND_ERROR;
}
/**
* fleetd#164: the failure reason for a scrape that produced zero characters — names the member
* and says plainly that the turn produced nothing, so a caller (a lead deciding whether to
* delegate again included) never mistakes a lost turn for a genuinely empty reply.
*/
private static String emptyScrapeReason(String target, boolean scrapeFailed) {
return "member " + target + " turn completed with an empty scrape (0 chars) — "
+ (scrapeFailed ? "its pane could not be read; " : "")
+ "treating as a lost turn, not a real answer";
}
/**
* The first line of {@code text} matching {@code pattern}, stripped — the CB-578 stage A
* evidence carried in a {@code BACKEND_EXHAUSTED} reason so the operator sees the real refusal
* text, never a generic label. {@code null} if no line matches.
*/
static String firstMatchingLine(String text, Pattern pattern) {
if (text == null || text.isEmpty()) return null;
for (String line : text.split("\n", -1)) {
if (pattern.matcher(line).find()) {
return line.strip();
}
}
return null;
}
/**
* Coverage summary for the CB-578 stage A exhausted-pattern classification, logged at startup
* the way {@link dev.ltms.fleet.health.FleetHealthMonitor#coverage} is — so an operator can
* see whether the classification is on, and for which profiles, without reading every
* profile's config by hand.
*
* @param allProfiles every configured profile name
* @param configuredProfiles the subset of {@code allProfiles} that carry an exhausted pattern
*/
public static String coverage(String patternKey, Set<String> allProfiles, Set<String> configuredProfiles) {
if (configuredProfiles.isEmpty()) {
return "off (no profile has an " + patternKey + " configured; profiles: " + sorted(allProfiles) + ")";
}
Set<String> unconfigured = new TreeSet<>(allProfiles);
unconfigured.removeAll(configuredProfiles);
return unconfigured.isEmpty()
? "full (all profiles configured: " + sorted(allProfiles) + ")"
: "partial (configured: " + sorted(configuredProfiles) + "; not configured: " + sorted(unconfigured) + ")";
}
private static List<String> sorted(Set<String> names) {
return names.stream().sorted().toList();
}
private static String clip(String s) {
if (s == null) return "";
String trimmed = s.strip();
return trimmed.length() <= MAX_SCRAPE_CHARS
? trimmed
: trimmed.substring(trimmed.length() - MAX_SCRAPE_CHARS);
}
/**
* Extract the last assistant message from a raw Claude Code pane scrape (CB-115). Claude Code
* prefixes each assistant turn with {@code ⏺}; the delegator wants that answer, not the TUI
* chrome around it. Take everything from the final {@code ⏺} onward and stop at the <em>first</em>
* hard interface boundary below it — the spinner/status line, input box, {@code ❯} prompt (which
* may echo the <em>next</em> turn's text), footer, or tips/warnings. Stopping at the first
* boundary (rather than trimming only trailing chrome) is what keeps a following turn's echoed
* prompt out of this reply. Blank lines are not boundaries, so a multi-paragraph answer survives;
* trailing blanks are trimmed at the end. With no {@code ⏺} marker (an unusual render) the whole
* text is scanned the same way, so we never lose the reply.
*
* <p>Package-private and pure so it is unit-testable without herdr.
*/
static String lastAssistantBlock(String raw) {
if (raw == null || raw.isBlank()) return "";
int marker = raw.lastIndexOf('⏺');
String block = marker >= 0 ? raw.substring(marker + 1) : raw;
StringBuilder out = new StringBuilder();
int kept = 0;
for (String line : block.split("\n", -1)) {
if (isBoundary(line)) break; // first TUI boundary ends the assistant message
if (kept++ > 0) out.append('\n');
out.append(line);
}
return out.toString().strip();
}
/**
* A hard TUI boundary line that marks the end of an assistant message and the start of interface
* chrome (input box, prompt, spinner, footer, tips/warnings). Blank lines are <em>not</em>
* boundaries — an answer may contain them — so they are kept and trimmed only if trailing.
*/
private static boolean isBoundary(String line) {
String t = line.strip();
if (t.isEmpty()) return false;
// A horizontal rule / all box-drawing separators (e.g. "──────").
if (t.chars().allMatch(c -> c == '─' || c == '—' || c == '━' || c == '═' || c == '-')) {
return true;
}
String lower = t.toLowerCase();
return t.startsWith("╭") || t.startsWith("│") || t.startsWith("╰") || t.startsWith("┌")
|| t.startsWith("└") || t.startsWith("❯") || t.startsWith("⏵")
|| t.startsWith("⎿") || t.startsWith("⚠")
// Status/spinner lines Claude Code renders below a settled or in-flight turn,
// e.g. "✻ Baked for 21s", "✶ Forming…".
|| t.startsWith("✻") || t.startsWith("✳") || t.startsWith("✽") || t.startsWith("·")
|| t.startsWith("●") || t.startsWith("◐") || t.startsWith("✢") || t.startsWith("✶")
|| lower.contains("auto mode") || lower.contains("for shortcuts")
|| lower.contains("esc to interrupt") || lower.contains("bypass permissions");
}
}
@@ -0,0 +1,92 @@
package dev.ltms.fleet.inject;
import java.util.function.Supplier;
/**
* Notified when {@link CompletionResolver} actually delivers a {@code BACKEND_EXHAUSTED}
* classification to a waiting send (CB-578 stage B) — never on a race that lost (see
* {@link CompletionResolver#resolve}, which only calls this after
* {@code Rendezvous.resolveExhausted} returns {@code true}).
*
* <p>{@link CompletionResolver} knows only {@code target} (a herdr terminal id); it has no notion of
* profiles or credentials, so mapping {@code target} to whatever should be quarantined is entirely
* the sink's job — see {@code Fleetd.main}'s wiring, which resolves target → session → profile →
* {@code effectiveCredentialId()} and calls {@code BackendQuarantine.quarantine} on it.
*
* <p><strong>The 3-arg overload is the single abstract method — fleetd #234, round 4.</strong> Two
* earlier rounds each shipped a caller that silently dropped the profile hint (see {@link
* #onExhausted(String, String, String)}): a lambda written against this interface can only ever
* implement whichever overload is abstract, and while the 2-arg form held that position, EVERY
* lambda site — a call-site forwarder in {@code Fleetd.java}, a hand-built test double — bound to
* it and silently inherited the profile-dropping default, whether or not its author remembered the
* hazard. Making the 3-arg form abstract instead removes the shape entirely: a lambda declared
* against this interface today is <em>forced</em> by the compiler to take {@code (target, reason,
* profile)}, so there is no overload left for it to bind to that can drop the hint. This is a type
* change, not a test — it holds even for a caller that has never heard of fleetd #234.
*/
@FunctionalInterface
public interface ExhaustionSink {
/**
* @param target the herdr terminal id whose turn was classified {@code BACKEND_EXHAUSTED}
* @param reason the matched-line reason carried by the classification
* @param profile the profile the caller already knows should be quarantined, or {@code null}
* when the caller has no better answer than {@code target} alone (fleetd #234):
* a caller whose {@code target} is not yet resolvable through whatever roster
* the sink's implementation consults — {@link
* dev.ltms.fleet.member.OpenCodeLauncher}'s model-mismatch check fires from
* {@code SessionAwareHandle.agentSessionId()}, which runs during {@code
* SessionManager.acquire()} <em>before</em> that session is registered, so a
* target -> session -> profile lookup finds nothing at that point. That launcher
* already has its own {@code FleetConfig.Profile} in hand and does not need the
* roster to know which profile to quarantine, so it supplies this directly
* instead of leaving the sink to guess.
*/
void onExhausted(String target, String reason, String profile);
/**
* Convenience for a caller with no profile to offer — every existing call site that predates
* the hint (fleetd #234): {@link dev.ltms.fleet.inject.CompletionResolver}'s two call sites
* always call with a {@code target} that IS live in the roster at the time of the call, so they
* need no hint and keep working exactly as before, unchanged by this default.
*
* @param target as {@link #onExhausted(String, String, String)}
* @param reason as {@link #onExhausted(String, String, String)}
*/
default void onExhausted(String target, String reason) {
onExhausted(target, reason, null);
}
/**
* Inert sink — nothing happens on exhaustion. The explicit stand-in a caller (or a test not
* exercising this feature) passes instead of a defaulting overload, exactly like
* {@link ExhaustedPatternLookup#none()}. Safe as a lambda: the 3-arg form is now the interface's
* single abstract method, so a lambda here has no other overload to silently bind to instead —
* it simply does nothing with all three arguments.
*/
static ExhaustionSink none() {
return (target, reason, profile) -> { };
}
/**
* A sink that forwards to whatever {@code target} currently supplies (fleetd #234). Exists to
* break a genuine construction-order cycle: {@code Fleetd.main} builds its adapters (including
* {@link dev.ltms.fleet.member.OpenCodeLauncher}) before {@code sessions} exists, so it cannot
* hand them the real sink yet — it hands them a forwarder pointed at an {@code
* AtomicReference<ExhaustionSink>} that starts at {@link #none()} and gets {@code .set()} to the
* real sink once {@code sessions} is built. {@code target} is evaluated on every call, never
* cached, so the forwarder keeps working after the reference is repointed.
*
* <p>Now safe as a one-line lambda (round 4): forwarding the single 3-arg abstract method
* forwards everything a caller can supply — there is no separate 2-arg overload left for a
* forwarder to bind to instead and silently lose the hint. Kept as a named factory rather than
* written inline at each call site anyway, so a test can call the exact object {@code
* Fleetd.java} builds instead of asserting a rebuilt copy of its shape (round 3's lesson).
*
* @param target supplies the sink to forward to, evaluated fresh on every call
* @return a sink whose call delegates to {@code target.get()}
*/
static ExhaustionSink forwardingTo(Supplier<ExhaustionSink> target) {
return (t, r, p) -> target.get().onExhausted(t, r, p);
}
}
@@ -2,6 +2,7 @@ package dev.ltms.fleet.inject;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.HerdrRouter;
import dev.ltms.fleet.msg.TurnToken;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -89,6 +90,7 @@ public final class Injector {
public static final long POLL_INTERVAL_MILLIS = 250;
private final AgentControl agents;
private final HerdrRouter router;
private final TurnListener turnListener;
private final Predicate<String> ready; // CB-113: a target is deliverable only when available
private final Consumer<String> forget; // CB-114: clear a gone worker's readiness/presence
@@ -124,11 +126,25 @@ public final class Injector {
public Injector(AgentControl agents, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget) {
this.agents = agents;
this.router = null;
this.turnListener = turnListener;
this.ready = ready;
this.forget = forget;
}
public Injector(HerdrRouter router, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget) {
this.agents = null;
this.router = router;
this.turnListener = turnListener;
this.ready = ready;
this.forget = forget;
}
private AgentControl agentsFor(String target) {
return router != null ? router.agentsFor(target) : agents;
}
/** A pending message and the future that completes when it has been delivered. */
private record Pending(String text, TurnToken token, CompletableFuture<Void> delivered) {
}
@@ -141,6 +157,7 @@ public final class Injector {
boolean awaitingCompletion; // a delivered message's turn is not yet known-complete
boolean turnObserved; // saw a real `working` sample since that delivery (turn ran)
int unknownSinceTurn; // consecutive `unknown` samples while a delegation is outstanding (CB-109)
int unknownSincePostTurn; // the same, for the post-turn housekeeping phase (fleetd #306)
int notReadySincePoll; // consecutive injectable samples a queued message waited on the readiness gate (CB-114)
boolean postTurnPending; // completion observed; adapter housekeeping has not started yet
boolean awaitingPostTurnPickup;
@@ -201,10 +218,12 @@ public final class Injector {
t.awaitingPickup = false;
t.injectableSincePickup = 0;
t.unknownSinceTurn = 0;
t.unknownSincePostTurn = 0;
t.notReadySincePoll = 0;
if (t.awaitingCompletion) t.turnObserved = true;
} else if (status.injectable()) { // IDLE or BLOCKED
t.unknownSinceTurn = 0;
t.unknownSincePostTurn = 0;
if (t.awaitingPostTurnPickup) {
if (++t.injectableSincePostTurnPickup >= PICKUP_GRACE_POLLS) {
t.awaitingPostTurnPickup = false;
@@ -253,7 +272,7 @@ public final class Injector {
if (p != null && ready.test(target)) {
t.notReadySincePoll = 0;
try {
agents.send(target, p.text());
agentsFor(target).send(target, p.text());
t.queue.poll();
t.awaitingPickup = true;
t.awaitingCompletion = true;
@@ -299,6 +318,24 @@ public final class Injector {
t.unknownSinceTurn = 0;
turnFailed = true;
}
// fleetd #306: the same escape for the post-turn housekeeping phase. Four latches
// gate delivery (awaitingCompletion, postTurnPending, awaitingPostTurnPickup,
// postTurnObserved) and only the first had a way out of a sustained unknown streak —
// a gate that closed one direction only. The other two below are released here as
// well; postTurnPending needs no escape because it is cleared unconditionally on the
// line after the listener call that sets it.
//
// This does NOT set turnFailed. The delegated turn already completed and its waiter
// already resolved — what is outstanding is adapter housekeeping (the `/clear`).
// Reporting a turn failure here would drive SessionManager.onFailed on a session
// that genuinely finished its work, which is a worse lie than the wedge.
if ((t.awaitingPostTurnPickup || t.postTurnObserved)
&& ++t.unknownSincePostTurn >= TURN_STALL_GRACE_POLLS) {
t.awaitingPostTurnPickup = false;
t.postTurnObserved = false;
t.injectableSincePostTurnPickup = 0;
t.unknownSincePostTurn = 0;
}
}
// Reclaim the entry once the worker is fully quiescent (nothing queued, no pickup or
@@ -313,7 +350,7 @@ public final class Injector {
// thread while it holds the target lock.
if (resubmit) {
try {
agents.submit(target); // nudge a raced Enter so the pending paste submits
agentsFor(target).submit(target); // nudge a raced Enter so the pending paste submits
} catch (RuntimeException e) {
log.debug("resubmit to {} failed (will retry next poll): {}", target, e.getMessage());
}
@@ -3,6 +3,7 @@ package dev.ltms.fleet.inject;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.herdr.HerdrRouter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -22,6 +23,7 @@ public final class StatusPoller {
private static final Logger log = LoggerFactory.getLogger(StatusPoller.class);
private final AgentControl agents;
private final HerdrRouter router;
private final Injector injector;
private final StatusRefiner refiner;
private final long intervalMillis;
@@ -35,11 +37,24 @@ public final class StatusPoller {
public StatusPoller(AgentControl agents, Injector injector, StatusRefiner refiner,
long intervalMillis) {
this.agents = agents;
this.router = null;
this.injector = injector;
this.refiner = refiner;
this.intervalMillis = intervalMillis;
}
public StatusPoller(HerdrRouter router, Injector injector, long intervalMillis) {
this.agents = null;
this.router = router;
this.injector = injector;
// CB-185: this refiner's own AgentControl (member) is only a default for the legacy 2-arg
// refine() overload — the loop below always calls the 3-arg refine(target, raw, control)
// with the per-target control from router.agentsFor(target), so a lead target is refined
// against the LEAD daemon even though this field points at the member one.
this.refiner = new StatusRefiner(router.memberAgents());
this.intervalMillis = intervalMillis;
}
/** Start the polling loop on a virtual thread. Idempotent. */
public synchronized void start() {
if (running) return;
@@ -56,7 +71,11 @@ public final class StatusPoller {
try {
// herdr's agent_status can misreport a settled worker as `unknown`; refine it
// against the pane content before it drives delivery/completion (CB-115).
AgentStatus status = refiner.refine(target, agents.status(target));
// CB-185: refine THROUGH the same control the raw status came from — a router
// splits lead/member targets across two herdr daemons, and reading a lead's pane
// through the (fixed) member refiner never finds it, wedging that lead at UNKNOWN.
AgentControl control = router != null ? router.agentsFor(target) : agents;
AgentStatus status = refiner.refine(target, control.status(target), control);
injector.onStatus(target, status);
} catch (HerdrException e) {
// The worker's agent is gone — stop trying and unblock its waiters.
@@ -41,15 +41,32 @@ public final class StatusRefiner {
}
/**
* Return a trustworthy status for {@code target}. Any non-{@code UNKNOWN} {@code raw} is returned
* unchanged; an {@code UNKNOWN} triggers a pane read and content classification. A read failure
* leaves it {@code UNKNOWN} (the safe default: no delivery, and the stall path still applies).
* Return a trustworthy status for {@code target}, reading its pane through this refiner's own
* {@link AgentControl}. Equivalent to {@link #refine(String, AgentStatus, AgentControl)} with
* that control — kept for callers that only ever talk to one herdr daemon.
*/
public AgentStatus refine(String target, AgentStatus raw) {
return refine(target, raw, agents);
}
/**
* Return a trustworthy status for {@code target}. Any non-{@code UNKNOWN} {@code raw} is returned
* unchanged; an {@code UNKNOWN} triggers a pane read (through {@code control}) and content
* classification. A read failure leaves it {@code UNKNOWN} (the safe default: no delivery, and
* the stall path still applies).
*
* <p>CB-185: {@code control} must be the {@link AgentControl} for the <em>same</em> daemon the
* raw status was sampled from — a router splits lead and member targets across two herdr
* daemons, and reading a lead's pane through the member client (or vice versa) fails to find
* the pane and leaves the target wedged at {@code UNKNOWN} forever. Callers that route per
* target (e.g. {@code StatusPoller}) must pass that target's control explicitly rather than
* relying on the control fixed at construction.
*/
public AgentStatus refine(String target, AgentStatus raw, AgentControl control) {
if (raw != AgentStatus.UNKNOWN) return raw;
String pane;
try {
pane = agents.read(target, PROBE_SOURCE);
pane = control.read(target, PROBE_SOURCE);
} catch (RuntimeException e) {
log.debug("status refine read for {} failed; leaving UNKNOWN: {}", target, e.getMessage());
return AgentStatus.UNKNOWN;
@@ -15,8 +15,6 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import dev.ltms.fleet.peer.PeerLauncher;
/**
@@ -41,7 +39,7 @@ import dev.ltms.fleet.peer.PeerLauncher;
* broken later, quietly.
*
* <p><strong>Liveness, and the label round-trip.</strong> {@code LeadTabScanner} used to be able to
* promise that bridged never writes a lead label. That is no longer true — an auto-launched lead is
* promise that fleetd never writes a lead label. That is no longer true — an auto-launched lead is
* labelled by this class, and the scanner reads that label back. The risk this opens is not
* privilege escalation (the tab label never granted anything a pane could take for itself; see that
* class's javadoc), but <em>staleness</em>: a label left behind by a crashed session would otherwise
@@ -106,7 +104,7 @@ public final class LeadLauncher {
// A lead with a `tab:` but no `profile:` is recognise-only by design: the operator
// opens it by hand. Say so once rather than looking like a silent failure.
log.info("lead '{}' is not live, and names no profile — it can be recognised but not "
+ "launched. Add `profile:` under fleet.leaders.{} to have bridged start it.",
+ "launched. Add `profile:` under fleet.leaders.{} to have fleetd start it.",
name, name);
continue;
}
@@ -140,15 +138,15 @@ public final class LeadLauncher {
* auto-launched one is — by labelling its tab to match.
*/
private Map<String, Integer> liveLeads(Map<String, FleetConfig.Leader> leaders) {
Set<String> memberSpaces = cfg.profiles().values().stream()
.map(FleetConfig.Profile::workspace)
.filter(w -> w != null && !w.isBlank())
.collect(Collectors.toSet());
// tabId → the lead name its label declares.
// A lead and the members share ONE workspace now (the operator asked for a single "session"
// with many tabs), so a workspace can no longer be excluded wholesale — the lead lives in the
// member workspace by design. The sole discriminator is the exact tab label: a lead carries
// its configured `fleet.leaders.<name>.tab` ("lead: opus"), while a member carries its
// profile's `worker: {profile} #{n}` template. These never collide, so an exact-label match
// separates them without needing to know which workspace anyone is in.
Map<String, String> nameByTab = new LinkedHashMap<>();
for (Workspace ws : spaces.listWorkspaces()) {
if (ws.workspaceId() == null || memberSpaces.contains(ws.label())) {
if (ws.workspaceId() == null) {
continue;
}
for (Tab tab : spaces.listTabs(ws.workspaceId())) {

Some files were not shown because too many files have changed in this diff Show More