Compare commits

...

22 Commits

Author SHA1 Message Date
Dai Ha 2dff4b84a4 Audit: async ticket / rendezvous lifecycle in fleetd msg package 2026-09-04 10:35:01 +07:00
Dai Ha 66e5247b6d Merge #275 (PR #279): sweep an ASKING ticket on a definite teardown
CI / contract (push) Successful in 49s
CI / build (push) Successful in 1m39s
A member torn down while parked in fleet_ask left its async ticket pending
for good. resolveQuestion had already closed the forward waiter, so
abandon()'s waiter branch found nothing; the 'question == null' guard then
excluded the task from the matching loop. By the time the worker's own ask
lapsed (~55-115s), the released session was gone from the roster, so
nothing was left to call abandon() on that target again. fleet_poll{ticket}
reported PENDING forever.

The ticket told the worker to drop the 'question == null' guard. That was
wrong, and the worker said so with evidence: an existing test
(abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer) deliberately pins that
an ASKING ticket must SURVIVE abandon(), because the primary may be mid
answer() for that same turn. Widening the shared method would have traded
this bug for a worse one — a health guess killing a live conversation.

So the fix splits the two callers by what they actually know:

  - sessions.onRelease (fleet_stop / idle reaper) knows the pane is being
    stopped right now, so it sweeps: sweepAsking=true.
  - FleetHealthMonitor keeps sweepAsking=false. GONE/NEVER_READY is a
    classification from the live agent list, not a teardown it performed.

I verified the reachability chain myself rather than taking it on trust.
FleetHealth.decide returns GONE before it can ever return
DELEGATION_ORPHANED; FleetHealthMonitor.reportTransition returns early when
previous == next; and terminal() is GONE/NEVER_READY only. So after the one
GONE transition fires and no-ops, nothing re-fires. Every link holds.

Verified: the real merge into current main builds green (1283 tests), the
protective test still passes untouched, and my own mutation — reverting the
sweepAsking widening — fails the new test with 'a released target's open ask
can never resume, so it must fail right here'.
2026-09-04 10:18:59 +07:00
Dai Ha 1e41bd63b4 Merge #274 (PR #277): clean up the worktree when add() fails after creating it
CI / contract (push) Successful in 59s
CI / build (push) Successful in 1m34s
GitWorktrees.add() created the worktree and branch, then ran more steps that
can throw — requireCredentialFreeHttpsOrigin among them, which is an
intended security refusal, not an IO accident. Any throw meant add() never
returned, so SessionManager.acquireWithWorktree never learned the path, its
'if (path != null)' cleanup could not fire, and the worktree and branch
leaked with nothing tracking them. Every OTHER exit from that method was
cleaned up correctly; only the exits inside add() were uncounted.

add() now cleans up what it created before rethrowing, reusing remove() and
additionally deleting the branch — a branch that never finished provisioning
has no session and no PR behind it. Worktree first, since a checked-out
branch cannot be deleted. Cleanup failure is logged and never masks the
original exception.

Verified by me: the real merge into current main builds green (1281 tests),
and I reran the mutation myself without git stash — dropping the cleanup
call fails the new test with 'the worktree directory leaked after a
post-creation step threw'.

The test drives add() itself through the existing afterWorktreeAdded seam,
so the failure happens after the worktree exists rather than downstream in
another caller.
2026-09-04 10:14:05 +07:00
Dai Ha 4887d03d88 #275: abandon() sweeps an ASKING ticket only on a definite teardown
CI / contract (pull_request) Successful in 1m21s
CI / build (pull_request) Successful in 2m5s
Confirmed reachable: a target torn down for good (fleet_stop / the idle
reaper) while its async ticket sits in fleet_ask (Phase.ASKING) got
permanently stuck. resolveQuestion already closes the forward waiter, the
question == null guard excluded the task from abandon()'s sweep, and by the
time the worker's own fleet_ask lapses (~55-115s) the released session no
longer appears in FleetHealthMonitor's roster, so nothing ever calls
abandon() again. fleet_poll{ticket} then reports PENDING forever.

Add abandon(target, reason, sweepAsking) — sessions.onRelease (a definite
teardown: the pane is being stopped right now) passes true and now fails the
ASKING ticket and closes its reverse-rendezvous ask. FleetHealthMonitor's
health-classification call keeps the 2-arg overload (sweepAsking=false):
a GONE/NEVER_READY reading is a guess from the live agent list, not a
teardown it performed, and abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer
already covers why an active ask must survive that guess (the primary may
be mid-answer for the same turn). hasOrphanedDelegation is left unchanged
for the same reason — it must not flag a live, active ask as orphaned.

Proven with a test driving the real public sequence (sendAsync -> ask ->
abandon(..., true)), not a hand-built task map; reverted the widening to
confirm it goes red, then restored it.
2026-09-04 10:13:56 +07:00
Dai Ha 7d5434455d implementer: never git stash — the stash stack is shared across worktrees
CI / build (push) Successful in 1m41s
CI / contract (push) Successful in 1m48s
A worker's worktree is isolated; refs/stash is not. It is one stack shared
by the primary's checkout and every worker worktree of this repo.

This bit a real worker today. Two ran in parallel; one called git stash
while the other was mid-edit, and the second worker's in-progress change
was silently overwritten by the first's stashed content. It recovered by
retyping the edit and diffing to confirm, and pushed the other worker's
change back onto the stack untouched — but nothing warned either of them,
and nothing would have.

Measured before writing this: 'git stash list' from a worker worktree and
from the primary's checkout return byte-identical output, and refs/stash
is a single common ref, not a per-worktree one.

The branch already IS the isolation, so the skill now points at committing
a wip commit or writing a patch file instead.
2026-09-04 10:10:08 +07:00
Dai Ha f71ee4926e Merge #273 (PR #278): validate exhaustedPattern at config load
CI / contract (push) Successful in 44s
CI / build (push) Successful in 1m44s
A malformed exhaustedPattern passed FleetConfig.load and then crashed the
daemon at startup, in Fleetd.main's unguarded Pattern.compile, with a
message naming neither the profile nor the key. Its sibling errorPattern
had a load-time validator whose own javadoc explains exactly why that is
bad. The validator was correct; its coverage was not.

rejectMalformedErrorPattern becomes rejectMalformedProfilePatterns and now
compiles both keys, reporting failures from either in one message.

Verified by me, not taken on the worker's word: the actual merge of this
branch into main builds green (1280 tests), and I reran the mutation myself
— narrowing the loop back to errorPattern turns exactly the two new tests
red, one of them with 'Expected IllegalStateException to be thrown, but
nothing was thrown', which is the defect stated out loud.
2026-09-04 10:09:15 +07:00
Dai Ha 282a2fc2b8 fleetd #274: clean up the worktree and branch when add() fails after creating them
CI / contract (pull_request) Successful in 56s
CI / build (pull_request) Successful in 1m22s
GitWorktrees.add() created the worktree and branch, then ran several more
steps that can throw (requireCredentialFreeHttpsOrigin — an intended
security refusal, not only an IO accident — plus the credential-helper and
tool-surface isolation steps). Any exception there meant add() never
returned, so its caller (SessionManager#acquireWithWorktree) never learned
the path: its local `path` stayed null, the `if (path != null)` cleanup
guard never ran, and the worktree directory and branch leaked on disk
forever with nothing tracking them.

Wrap those steps in try/catch; on failure, clean up via the same
`git worktree remove --force` path remove() already uses, additionally
force-delete the new branch (remove() alone deliberately leaves a
released session's branch behind, but a branch that never finished
provisioning has nothing else pointing at it), log the cleanup outcome,
and rethrow the original exception so it is never masked.

Test drives add() itself via the existing afterWorktreeAdded seam with a
mutation that trips requireCredentialFreeHttpsOrigin after the worktree
exists, then asserts both the worktree directory and the branch are gone.
Reverting the fix (git stash on GitWorktrees.java, test unchanged) turns
it red: "the worktree directory leaked after a post-creation step threw
==> expected: <false> but was: <true>". Restored afterward.

mvn clean install: BUILD SUCCESS, Tests run: 1275, Failures: 0, Errors: 0
2026-09-04 10:05:59 +07:00
Dai Ha f04e934b94 fleetd #273: validate exhaustedPattern regex at load, like errorPattern
CI / build (pull_request) Successful in 1m36s
CI / contract (pull_request) Successful in 2m11s
FleetConfig.rejectMalformedErrorPattern only compiled errorPattern eagerly
at config load. exhaustedPattern was compiled unguarded in Fleetd.main,
so profiles.<name>.exhaustedPattern: "[" passed load() and then crashed
the whole daemon at boot with a raw PatternSyntaxException naming neither
the profile nor the key.

Rename the validator to rejectMalformedProfilePatterns and extend it to
also compile every non-blank exhaustedPattern, reporting
profiles.<name>.exhaustedPattern ("<value>"): <message> in the same style
as errorPattern. Both keys are collected and reported together from a
single load. Fleetd.java's compile site is left as-is per scope — it is
now safe because load already rejects a bad value.

Added tests covering: a bad exhaustedPattern is refused; a bad pattern in
each key is reported together in one message; valid patterns still load;
a blank/absent exhaustedPattern is ignored.
2026-09-04 10:05:36 +07:00
Dai Ha 3fae35c357 #276: say whose environment the "allowed N of M" line counted
CI / contract (push) Successful in 55s
CI / build (push) Successful in 1m42s
A gap in my own #269 fix. That ticket stopped four sites claiming things
about the member's environment that fleetd cannot see when memberHerdrSocket
is configured, and gave the WARN in logCredentialGap a guard. The INFO line
called two lines earlier never got one:

    logAllowListCoverage(allowed);      // no guard
    logCredentialGap(creds, allowed);   // guarded since #269

Read plainly, "member credentials: allowed 7 of 39" is a statement about the
member's credentials. Under memberHerdrSocket the pane is routed to a second
herdr whose environment fleetd has no channel to inspect, so those counts
come from fleetd's own process instead. Same overclaim #269 existed to
remove, in the line next door.

The method's javadoc does carry the caveat, by cross-reference to another
field's javadoc. That does not help the operator reading fleetd.out.

The counts stay useful, so this is not a WARN and not a refusal — only the
claim is narrowed. The unguarded path keeps its exact original wording, so
the existing assertion on "member credentials: allowed 1 of 3" still holds.

The new test pins the pair together so a later edit cannot fix one line and
leave the other. Mutation-proved: with the guard removed it fails printing
the old line verbatim.

Also worth recording: no test covered #269's own guard — that WARN wording
shipped unverified, and still has no coverage.
2026-09-04 10:01:23 +07:00
Dai Ha 18aecbfe67 #272: fleet_poll{target} is a drain, so gate it as one
CI / build (push) Successful in 1m46s
CI / contract (push) Successful in 2m29s
fleet_poll is two operations behind one tool name. With `ticket` it observes
an async delegation and changes nothing. With `target` it calls
MessageService.drainReplies, which REMOVES the replies — a second call
returns nothing.

The handler gated both branches with a constant Authz.Action.READ, and did
not pass the target at all. READ is open to every authenticated role, so any
worker could read a peer's sessionId out of fleet_list and destroy the
replies that peer had queued for the primary. The gate failed open, and a
drained reply is not recoverable.

Three things already said the tight gate was intended:

  - fleet_ack, four lines below, gates the same drain as DRAIN, with a
    comment giving the exact reasoning missed here ("Acking removes a reply
    from the inbox, so it is a drain, not a read").
  - the REST path checks DRAIN in FleetApp.drainReplies.
  - wiki/2-Message-Server.md lists fleet_poll as lead-only, and the tool
    schema says "drain that worker's inbox".

Nothing that works today breaks: the documented flow is fleet_poll{target}
then fleet_ack{target,msgId}, and fleet_ack is already primary-only. A
worker could never complete that flow — only destroy its first half.

The required action is a function of the arguments, but the handler chose it
before looking at them. pollAction(target) makes that choice explicit. The
ticket branch stays READ on purpose: an architect may fleet_send, so it owns
tickets and must be able to poll them.

Why the suite missed it: FleetMcpAuthzTest checks every Action against every
Role, including "a worker may not DRAIN", and passed the whole time. The
policy table was right; the action fed to it was wrong, and nothing tested
that mapping. The new tests assert against pollAction itself, so the handler
keeps no private copy of the rule.

Mutation-proved: reverting pollAction to a constant READ turns exactly the
two new defect tests red and leaves the ticket-branch test green.

Introduced in 9daf1ec, where Authz.READ's own javadoc ("...task polling")
describes only the ticket half.
2026-09-04 09:54:52 +07:00
Dai Ha 5d75f72473 Merge #271: warn when the model check cannot run for a no-worktree opencode spawn (#267)
CI / contract (push) Successful in 1m18s
CI / build (push) Successful in 1m40s
2026-09-04 08:38:05 +07:00
Dai Ha e028a0ae54 fleetd #267: warn once per profile when the model check can't run
CI / contract (pull_request) Successful in 1m6s
CI / build (pull_request) Successful in 1m50s
OpenCodeLauncher.SessionAwareHandle.agentSessionId() is the only caller of
checkModelMatch (fleetd #175), and it sits behind the fleetd #249 worktree
gate. A spawn with no worktree:true — the ordinary shape of most opencode
spawns — never reached the check at all, and the gap was totally silent.

The check cannot be decoupled from agentSessionId()'s resolved id: doing so
would re-derive 'whatever is newest in the shared directory' and reintroduce
the false-positive risk fleetd #234 fixed (a sibling's differently-configured
model looking like a mismatch for a profile that never actually ran it). The
#249 gate is correct and stays as-is.

Instead, log once per profile at WARN, naming the profile, the same
treatment discoveryUnavailable already gets a few lines above — a logged
UNKNOWN beats a check that silently never runs.
2026-09-04 08:36:12 +07:00
Dai Ha 2fa673d4c0 Merge #270: ArchUnit package-cycle test with explicit accepted exceptions (#131)
CI / contract (push) Successful in 1m0s
CI / build (push) Successful in 1m57s
2026-09-04 08:22:35 +07:00
Dai Ha 9020d01b40 Merge #268: rename sshAuthSock values to omit/inherit with a read-both shim (#266) 2026-09-03 20:27:55 +07:00
Dai Ha 1006805027 fleetd #131: enforce package boundaries with an ArchUnit cycle test
CI / contract (pull_request) Successful in 1m10s
CI / build (pull_request) Successful in 2m1s
Adds PackageCyclesTest, which fails the build on any new cycle between
the top-level dev.ltms.fleet.* packages. Today's five real cycles are
recorded as narrow, explicit exceptions (ignoreDependency per named
pair, both directions), each commented with the ticket step (or a note
that it needs its own) that removes it. No package moves in this PR.

archunit-junit5 1.5.0 (current stable, newer than an earlier 1.4.1
draft). Main code only (DO_NOT_INCLUDE_TESTS) and importPackages(...)
instead of a working-directory-relative target/classes path.
2026-09-03 20:22:46 +07:00
Dai Ha 27aefbf9a0 Merge #269: stop claiming memberHerdrSocket proves a different OS user (#184 item 5)
CI / contract (push) Successful in 1m26s
CI / build (push) Successful in 1m41s
2026-09-03 20:14:44 +07:00
Dai Ha d42c2bc204 fleetd #266: rename SSH agent environment setting
CI / contract (pull_request) Successful in 1m29s
CI / build (pull_request) Successful in 2m1s
2026-09-03 20:12:46 +07:00
Dai Ha 3916adc372 fleetd #184: stop claiming memberHerdrSocket proves a different OS user
CI / contract (pull_request) Successful in 1m18s
CI / build (pull_request) Successful in 1m19s
HerdrPeerLauncher asserted, as established fact, that member panes run under
a different OS user whenever memberHerdrSocket is configured. fleetd has no
channel to see the uid at the other end of a herdr unix socket — an operator
may point memberHerdrSocket at a second herdr under the SAME user for pane
isolation, in which case members do inherit fleetd's environment and the
count this WARN told them to disregard is the real gap.

Reworded the class javadoc on hostEnvNames, the WARN in
warnUnknownMemberEnvironment, the javadoc on memberHerdrSocketConfigured(),
and warnCannotShareScrubDirectory's "unreadable by another uid" claim to say
what is actually true: fleetd cannot confirm what OS user the second herdr
runs as, so the member credential gap is UNKNOWN, not known-clean or
known-dirty. No behaviour change — the fallback paths and the honest
UNKNOWN conclusion stay the same, only the stated reason changes.

Matches the framing already used by Fleetd.reportMemberTrustModel on main.

Added unknownEnvironmentWarnStatesUncertaintyNotAnAssertedDifferentUser to
HerdrPeerLauncherAllowListWiringTest asserting the new WARN wording and that
it no longer claims a different OS user as fact.
2026-09-03 20:10:22 +07:00
Dai Ha fa97f598dd Merge #265: state the member trust model at startup (#184)
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m26s
2026-09-03 16:48:44 +07:00
Dai Ha ea9aa4fd77 fleetd #184: report member trust model
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 1m59s
2026-09-03 16:48:13 +07:00
Dai Ha a2b8caf6b5 #184: keep the reason SSH_AUTH_SOCK matters, and the measurement
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m23s
The correction removed a false claim (blocking the socket breaks git over
SSH) but took a true one with it: the socket is a live handle to the agent,
so a member holding it can sign with every key the agent holds. Without that,
the entry reads as if the setting does not matter, and an operator has no
reason left not to set it to allow. Fixing an overclaim must not leave an
underclaim.

Also record the measurement and the mistake behind the old claim, so the next
person does not re-argue it from scratch.
2026-09-03 16:42:11 +07:00
Dai Ha b5ddbe5757 Merge #264: sshAuthSock is not a control, and the config now says so (#184) 2026-09-03 16:41:52 +07:00
21 changed files with 1139 additions and 103 deletions
+13
View File
@@ -39,6 +39,19 @@ a worker made all 59 of its edits in the primary's tree and never noticed.
test "$(git rev-parse --show-toplevel)" = "$PWD" || cd "$(git rev-parse --show-toplevel)"
```
**Never run `git stash` (or `git stash pop`/`apply`/`drop`).** Your worktree is isolated, but the
stash is **not**: `refs/stash` is one stack shared by the primary's checkout and every other
worker's worktree of this repo. Measured on 2026-09-04 — `git stash list` from a worker's worktree
and from the primary's tree returned byte-identical output. So a `git stash` you run can be popped
into someone else's tree, and a `git stash pop` you run can drop **another worker's** uncommitted
edits on top of yours. This has already happened here: two workers were running in parallel and one
of them had its in-progress edit silently overwritten by the other's stash.
The branch is your isolation, so use it instead. To set work aside, commit it on your own branch
(`git commit -m "wip: ..."`) and carry on; to try something and back out, use
`git diff > /tmp/<your-branch>.patch` then `git checkout -- <file>`. Both stay inside your worktree.
If you find a stash entry you did not create, leave it alone and say so in your report.
## 2. Implement
- Implement exactly the scope the lead named. Keep the diff focused; note anything out of scope
+96
View File
@@ -0,0 +1,96 @@
# Audit: async ticket / rendezvous lifecycle (`fleetd/src/main/java/dev/ltms/fleet/msg/`)
Scope: `Rendezvous.java` and `MessageService.java` — the lifecycle of an async ticket
(`Task`) and a rendezvous waiter: create, send, ask, answer, resolve, timeout, abandon, prune.
## Main finding
```
1. fleetd/src/main/java/dev/ltms/fleet/msg/MessageService.java:938
2. issue: answer() completes an async ticket's future with a QUESTION outcome when the worker
asks a second fleet_ask in the same resumed turn, permanently mislabeling a live delegation
as failed and losing its real reply to fleet_poll.
3. fix: guard the finishAsyncTask(turnId, result) call at line 938 the same way sendAsync's
lambda already guards its own call (lines 999-1005): skip it when result.outcome() ==
Outcome.QUESTION, and instead re-associate the task with the new turnId (as markAsyncQuestion
does on the first ask).
4. severity: high
```
### Call sequence that reaches it
1. Lead: `fleet_send{sessionId: W, content: "task", wait:false}` → `sendAsync` creates `task1`
/ `ticket1`. Its worker thread calls `send(W, content, ASYNC_TIMEOUT_MS, onAccepted, task1)`,
which does `asyncTasksByWaiter.put(reply, task1)` (line 802) before blocking on
`reply.get()`.
2. Worker `W` calls `fleet_ask{"Q1"}` → `ask(W, "Q1", t)`. `markAsyncQuestion` finds `task1`
via `asyncTasksByWaiter`, stamps `task1.turnId = turnId1`,
`asyncTasksByTurn[turnId1] = task1`. `resolveQuestion` wakes step 1's `send()`, which returns
`Outcome.QUESTION`; `sendAsync`'s lambda sees `QUESTION` and deliberately does **not** call
`finishAsyncTask` (lines 1000-1005) — `ticket1` correctly polls `Phase.ASKING`.
3. Lead polls, sees `ASKING`, answers: `fleet_send{turnId: turnId1, content: "A1"}` →
`answer(turnId1, "A1", t)`. This opens a **new** waiter via `rendezvous.open(workerSession)`
(line 929) but — unlike `send()` — never puts it into `asyncTasksByWaiter`.
`answerAsk(turnId1, "A1")` unblocks the worker's `ask()` call.
`clearAsyncQuestion(turnId1, false)` clears `task1.question` but keeps
`asyncTasksByTurn[turnId1] = task1` (deliberate, per its own javadoc). `answer()` then blocks
on its own `reply.get()` (line 936).
4. Worker `W`, still in the same resumed turn, calls `fleet_ask{"Q2"}` again before replying →
a second `ask(W, "Q2", t)`. `openAsk` mints `turnId2`. `markAsyncQuestion` looks up
`asyncTasksByWaiter.get(waiter)` for the waiter `answer()` opened in step 3 — **not found**
(never registered), so `task == null`; `task1.turnId` stays `turnId1`, no
`asyncTasksByTurn[turnId2]` entry is ever created. `resolveQuestion` still succeeds (it only
needs a live waiter, not a `Task`) and wakes `answer(turnId1,...)`'s blocked `reply.get()`
with `Resolution(QUESTION, "Q2", turnId2)`.
5. `answer(turnId1,...)` (line 936-939): `result = Reply(Outcome.QUESTION, "Q2", turnId2)`;
`finishAsyncTask(turnId1, result)` looks up `task1` by the **original** `turnId1` (still
stamped from step 3) and unconditionally does `task1.future.complete(result)` — completing
`ticket1`'s future with a **QUESTION** outcome, then removes `asyncTasksByTurn[turnId1]`.
`answer()` returns `Outcome.QUESTION` to the lead's own `fleet_send{turnId1,...}` call
(correct, and separately answerable via `turnId2`), but `ticket1` is now terminally done.
### What goes wrong
- `fleet_poll{ticket1}` now hits the `f.isDone()` branch in `poll()` permanently. `Outcome.QUESTION`
is not `REPLIED`/`COMPLETED_UNREPLIED` (`r.completed()` is false) and carries no
`WORKER_FAILED`/`BACKEND_EXHAUSTED` reason, so it falls through to
`Phase.FAILED`, `detail = "no reply — question"` — even though the worker is alive and only
waiting on `turnId2`.
- `asyncTasksByTurn` no longer has any entry for `task1`/`target`, so
`hasAsyncQuestion(target)` goes back to `false` immediately, and `hasOrphanedDelegation`
no longer excludes this target's real state correctly either.
- If the worker's eventual real `fleet_reply` (after `turnId2` is answered, or times out and it
finishes on its own) is not captured by a chained direct `answer(turnId2,...)` call,
`reply()`'s fast path (`rendezvous.resolve`) finds no live waiter, `askAnsweredAsyncTasks`
finds no candidate (`task1.future.isDone()` is already true, so it is excluded), and the reply
is silently dropped into the inbox as a **stranded reply** — unreachable from `ticket1` and
from `abandon()`'s stranded-reply recovery (no open `matching` task exists any more).
### Confidence
High. I traced this with no races or interleavings assumed beyond the documented, deterministic
CB-205 chained-ask protocol that `outcomeOf`/`answer()` already generically support (mapping
`Kind.QUESTION` through `answer()`'s own return value is clearly intentional — see the
`Reply.turnId()` javadoc). I did not run the suite, but grepped
`src/test/java/dev/ltms/fleet/msg/MessageServiceTest.java` for a test exercising a *second*
`fleet_ask` inside one resumed (answered) turn and found none — `asyncQuestionBelongsToTheTaskThatOwnsItsForwardWaiter`
and `askSurfacesAsAQuestionAndTheAnswerResumesTheSameTurn` both cover only a single ask per
turn. A `git log -p` on this file also turned up the CB-588 comment (now at
`sendAsync`, describing the `whenComplete` hook) which explicitly frames
`answer()`'s `finishAsyncTask(turnId, result)` as firing "once a QUESTION is resolved" —
i.e. the author modeled that call as inherently terminal, which is exactly the assumption this
bug violates when the resumed turn asks again.
## Secondary (much shorter)
1. **Root-cause detail, same defect as above** — `answer()` (line 929) never puts its freshly
opened waiter into `asyncTasksByWaiter`, unlike `send()` (line 802). Even if the outcome
guard above is added, a chained second ask still can't be re-attached to `task1` via the
normal `markAsyncQuestion` path without also fixing this registration gap.
2. **Low / shape only** — `pruneTerminalTickets()` (line ~1092) is only invoked from inside
`sendAsync()`. A fleet whose sessions stop receiving new async sends (e.g. everything now
goes through blocking `send()`, or the target churns and workers are torn down) never prunes
its already-terminal `tasks` entries past `TICKET_TTL_NANOS`. Not reachable as a "stuck"
ticket (tickets still resolve correctly), only as unbounded `tasks`/`asyncTasksByWaiter`-adjacent
memory growth over a long-lived daemon with no further `sendAsync` traffic; did not verify
this is realistic in production traffic patterns, flagging as a shape only.
+16 -3
View File
@@ -676,8 +676,8 @@ guard:
# every name here NOT also in `allow` is overlaid with a non-secret sentinel value before
# the pane's login shell runs — real protection only for names that shell does not itself
# re-export (see the ROUND-2 CORRECTION note above). Under allow-list: reporting only.
# sshAuthSock → whether SSH_AUTH_SOCK may pass through under allow-list ("allow") or is omitted
# from the member environment ("block", the default). Blocking it only omits the
# sshAgentEnv → whether SSH_AUTH_SOCK may pass through under allow-list ("inherit") or is omitted
# from the member environment ("omit", the default). Omitting it only omits the
# inherited ssh-agent path. It discourages automatic use of the operator's agent.
# It does not deny same-user access to that socket. It also does not block SSH keys that
# are readable on disk. Git over SSH may still work from inside a member. Keep the block:
@@ -686,12 +686,25 @@ guard:
# confidentiality boundary. A real boundary needs a different OS user or OS-level
# confinement, such as a container or VM. That is the open question in fleetd #184.
#
# Still do not set this to "inherit" casually. SSH_AUTH_SOCK is a live handle to YOUR
# ssh-agent, so a member holding it can sign with EVERY key the agent holds. It sits in
# no secret file and looks like no credential, which is why it slipped past three
# earlier tickets (gitea #110). Blocking it does not contain a member, but allowing it
# hands one a signing capability for no gain — the block costs nothing, so keep it.
#
# Both halves of this are measured, not argued. 2026-08-28: a member with
# SSH_AUTH_SOCK blanked pushed to the forge over SSH successfully, because `ssh -G`
# resolves an IdentityFile outside ~/.ssh that is readable and has no passphrase. An
# earlier version of this comment claimed blocking the socket BREAKS git over SSH. It
# does not. That claim came from looking only in ~/.ssh, which holds nothing but four
# `Include` lines — looking in one place and concluding about the whole host.
#
# HOT-RELOADABLE the same way `fleet:` is (CB-559): read fresh on every spawn, so editing this list
# and reloading config (or restarting) changes what the NEXT spawn inherits; already-running members
# are unaffected either way.
# memberCredentials:
# policy: deny-by-default # or "deny-list", or "allow-list" (CB-633) — see above
# sshAuthSock: block # allow-list only; see the sshAuthSock note above
# sshAgentEnv: omit # allow-list only; see the sshAgentEnv note above
# allow:
# - AI_GATEWAY_TOKEN # named in a profile's tokenEnv (local/gx) — a member reaching the
# # gateway is by design, not a leak
+9
View File
@@ -29,6 +29,7 @@
<commons-compress.version>1.27.1</commons-compress.version>
<commons-lang3.version>3.18.0</commons-lang3.version>
<sqlite-jdbc.version>3.53.4.0</sqlite-jdbc.version>
<archunit.version>1.5.0</archunit.version>
</properties>
<!--
@@ -176,6 +177,14 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<!-- fleetd #131: package-boundary and cycle enforcement (PackageCyclesTest). -->
<dependency>
<groupId>com.tngtech.archunit</groupId>
<artifactId>archunit-junit5</artifactId>
<version>${archunit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
@@ -121,6 +121,7 @@ public final class Fleetd {
// else can fail on a silently-empty one. A daemon started without a login shell (launchd)
// boots fine either way — this is the only thing that says so out loud.
reportRequiredSecrets(cfg);
reportMemberTrustModel(cfg);
// CB-596: an absent (or empty) memberCredentials: block blocks NOTHING — no credential
// name is hardcoded any more to fall back on. Say so loudly, the same way a missing
// secret is reported above, so upgrading past this commit never silently drops CB-592's
@@ -592,7 +593,11 @@ public final class Fleetd {
if (detail.agentSessionId() != null) {
reason += " agentSessionId=" + detail.agentSessionId();
}
messages.abandon(detail.terminalId(), reason);
// fleetd #275: this is an explicit teardown (fleet_stop, or the idle reaper) — the
// worker's pane is being stopped right now, so an open fleet_ask has no turn left to
// resume into. Sweep it too, unlike FleetHealthMonitor's health-classification call
// (see MessageService.abandon's javadoc for why those two must differ).
messages.abandon(detail.terminalId(), reason, true);
replyInbox.release(detail.terminalId());
primaryRegistry.forgetDelegation(detail.terminalId()); // CB-532: don't leak the lead binding
});
@@ -1136,6 +1141,24 @@ public final class Fleetd {
});
}
/**
* fleetd #184: state the member trust model at startup. Environment controls and worktrees do
* not make a sandbox when fleetd and its members use the same OS user. A separate herdr may
* provide that boundary, but fleetd cannot inspect the uid at the other end of its socket.
*/
static void reportMemberTrustModel(FleetConfig cfg) {
if (cfg.memberHerdrSocket() != null && !cfg.memberHerdrSocket().isBlank()) {
log.info("member trust model: members are routed to a separate herdr through "
+ "memberHerdrSocket. fleetd cannot see that herdr's uid, so confirm it runs "
+ "as a different OS user before treating it as a boundary.");
return;
}
log.info("member trust model: members run as the same OS user as fleetd, not in a sandbox. "
+ "A member can read any file this user can read, including SSH keys and credential "
+ "stores, whatever memberCredentials says. To add a real boundary, route members to "
+ "a second herdr under a different OS user with memberHerdrSocket.");
}
/**
* CB-596: {@code known:} empty (block absent entirely, or present but empty) means {@link
* FleetConfig.MemberCredentials#blockedSet()} is empty too — every member pane inherits the
@@ -1,6 +1,8 @@
package dev.ltms.fleet.config;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -1357,19 +1359,19 @@ public record FleetConfig(
* deny-list/deny-by-default every name here that is NOT also in {@link #allow} is
* overlaid with a non-secret sentinel value. Under allow-list this list is
* reporting only.
* @param sshAuthSock whether the member may inherit {@code SSH_AUTH_SOCK} under the allow-list
* policy ({@code "allow"}) or must have it blanked ({@code "block"}, the default).
* @param sshAgentEnv whether the member may inherit {@code SSH_AUTH_SOCK} under the allow-list
* policy ({@code "inherit"}) or must have it omitted ({@code "omit"}, the default).
* This is a DECISION, never a default: {@code SSH_AUTH_SOCK} is a handle to the
* operator's ssh-agent, and a member holding it can sign with the operator's own
* keys — but it appears in no secret file and is credential-shaped like nothing on
* any list, which is why three earlier tickets missed it (gitea #110 / CB-607).
* Blocking it breaks git over SSH inside the member; allow it only when members do
* Omitting it does not prevent git over SSH inside the member; inherit it only when members do
* not need to authenticate as the operator over SSH. Ignored under deny-list /
* deny-by-default, which never touch the name.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record MemberCredentials(String policy, List<String> allow, List<String> known,
String sshAuthSock) {
String sshAgentEnv) {
/** Default policy: block every {@code known} name not in {@code allow}, via the env overlay. */
public static final String POLICY_DENY_BY_DEFAULT = "deny-by-default";
@@ -1386,11 +1388,25 @@ public record FleetConfig(
*/
public static final String POLICY_ALLOW_LIST = "allow-list";
/** The pre-CB-633 three-field form — {@code sshAuthSock} defaults to blocked. */
/** The pre-CB-633 three-field form — {@code sshAgentEnv} defaults to omitted. */
public MemberCredentials(String policy, List<String> allow, List<String> known) {
this(policy, allow, known, null);
}
/**
* Reads both the current {@code sshAgentEnv} key and the compatible {@code sshAuthSock} key.
* When both keys are present, {@code sshAgentEnv} wins, even if its value is unrecognised.
*/
@JsonCreator
public static MemberCredentials fromYaml(@JsonProperty("policy") String policy,
@JsonProperty("allow") List<String> allow,
@JsonProperty("known") List<String> known,
@JsonProperty("sshAgentEnv") String sshAgentEnv,
@JsonProperty("sshAuthSock") String sshAuthSock) {
return new MemberCredentials(policy, allow, known,
sshAgentEnv != null ? sshAgentEnv : sshAuthSock);
}
public MemberCredentials {
String normalizedPolicy = (policy == null || policy.isBlank())
? POLICY_DENY_BY_DEFAULT : policy.toLowerCase(java.util.Locale.ROOT);
@@ -1399,8 +1415,9 @@ public record FleetConfig(
policy = POLICY_DENY_LIST.equals(normalizedPolicy) ? POLICY_DENY_BY_DEFAULT : normalizedPolicy;
allow = allow == null ? List.of() : List.copyOf(allow);
known = known == null ? List.of() : List.copyOf(known);
sshAuthSock = (sshAuthSock != null && "allow".equalsIgnoreCase(sshAuthSock.trim()))
? "allow" : "block";
sshAgentEnv = (sshAgentEnv != null
&& ("inherit".equalsIgnoreCase(sshAgentEnv.trim())
|| "allow".equalsIgnoreCase(sshAgentEnv.trim()))) ? "inherit" : "omit";
}
/** True when this block selects the CB-633 derived-allow-list policy. */
@@ -1409,8 +1426,8 @@ public record FleetConfig(
}
/** True when {@code SSH_AUTH_SOCK} may pass through under the allow-list policy. Default: no. */
public boolean sshAuthSockAllowed() {
return "allow".equals(sshAuthSock);
public boolean sshAgentEnvInherited() {
return "inherit".equals(sshAgentEnv);
}
/** {@link #allow} as a set, for membership checks. */
@@ -1489,7 +1506,7 @@ public record FleetConfig(
rejectDuplicateMemberSlots(yaml);
rejectNegativeMaxLoad(yaml);
rejectAutoCompactWindowOutOfRange(yaml);
rejectMalformedErrorPattern(yaml);
rejectMalformedProfilePatterns(yaml);
rejectUnknownKind(yaml);
rejectUnknownAuthMode(yaml);
rejectUnknownPlacement(yaml);
@@ -1839,20 +1856,26 @@ public record FleetConfig(
}
/**
* Reject a profile whose {@code errorPattern} (fleetd #201 Unit 5) is not a valid Java regex,
* naming the profile, the key, and the parser's own message.
* Reject a profile whose {@code errorPattern} (fleetd #201 Unit 5) or {@code exhaustedPattern}
* (CB-578 stage A) is not a valid Java regex, naming the profile, the key, and the parser's own
* message.
*
* <p>Unset/{@code null} means "use {@code CompletionResolver}'s built-in {@code (?i)\bAPI
* Error\s*:} compatibility pattern" and passes silently. A profile that DOES set the key gets it
* compiled once at daemon startup ({@code Fleetd.main}, mirroring {@code exhaustedPattern}) — an
* uncaught {@link java.util.regex.PatternSyntaxException} there crashes startup without naming
* which profile or key is at fault. Validate eagerly here instead, at config load, the same
* "fail loud at load, not lazily later" reasoning as {@link #rejectAutoCompactWindowOutOfRange}.
* <p>Unset/{@code null} means, for {@code errorPattern}, "use {@code CompletionResolver}'s
* built-in {@code (?i)\bAPI Error\s*:} compatibility pattern", and for {@code exhaustedPattern},
* "opt out of that classification" — either way it passes silently. A profile that DOES set
* either key gets it compiled once at daemon startup ({@code Fleetd.main}) — an uncaught
* {@link java.util.regex.PatternSyntaxException} there crashes startup without naming which
* profile or key is at fault (fleetd #273: this happened for {@code exhaustedPattern}, which had
* no validator here even though its sibling {@code errorPattern} did). Validate eagerly here
* instead, at config load, the same "fail loud at load, not lazily later" reasoning as
* {@link #rejectAutoCompactWindowOutOfRange}. Both keys are checked from a single load, and any
* failures from either are collected together into one message.
*
* @param yaml the raw config text
* @throws IllegalStateException when any profile's {@code errorPattern} fails to compile
* @throws IllegalStateException when any profile's {@code errorPattern} or
* {@code exhaustedPattern} fails to compile
*/
static void rejectMalformedErrorPattern(String yaml) {
static void rejectMalformedProfilePatterns(String yaml) {
Map<?, ?> raw;
try {
raw = YAML.readValue(yaml, Map.class);
@@ -1867,18 +1890,20 @@ public record FleetConfig(
if (!(e.getValue() instanceof Map<?, ?> p)) {
continue;
}
if (!(p.get("errorPattern") instanceof String pattern) || pattern.isBlank()) {
continue;
}
try {
Pattern.compile(pattern);
} catch (PatternSyntaxException ex) {
bad.add("profiles." + e.getKey() + ".errorPattern (\"" + pattern + "\"): " + ex.getMessage());
for (String key : List.of("errorPattern", "exhaustedPattern")) {
if (!(p.get(key) instanceof String pattern) || pattern.isBlank()) {
continue;
}
try {
Pattern.compile(pattern);
} catch (PatternSyntaxException ex) {
bad.add("profiles." + e.getKey() + "." + key + " (\"" + pattern + "\"): " + ex.getMessage());
}
}
}
bad.sort(String::compareTo);
if (!bad.isEmpty()) {
throw new IllegalStateException("refusing to start: malformed errorPattern — "
throw new IllegalStateException("refusing to start: malformed pattern — "
+ String.join("; ", bad));
}
}
@@ -319,10 +319,12 @@ public final class FleetMcp {
};
BiFunction<McpSyncServerExchange, McpSchema.CallToolRequest, McpSchema.CallToolResult> pollHandler =
(exchange, req) -> {
McpSchema.CallToolResult denied = deny(exchange, Authz.Action.READ, null);
if (denied != null) return denied;
Map<String, Object> a = req.arguments();
return poll(messages, str(a, "ticket"), str(a, "target"));
String target = str(a, "target");
// The action depends on the ARGUMENTS, not on the tool name -- see pollAction.
McpSchema.CallToolResult denied = deny(exchange, pollAction(target), target);
if (denied != null) return denied;
return poll(messages, str(a, "ticket"), target);
};
// CB-307 Increment 3: per-msgId ack (not needed in v1 but supported by the inbox).
// Acking removes a reply from the inbox, so it is a drain, not a read.
@@ -724,6 +726,34 @@ public final class FleetMcp {
return text("delivered to peer lead " + coordId + " (msgId " + msg.msgId() + ")");
}
/**
* Which authorization action a {@code fleet_poll} call needs, decided by its arguments
* (fleetd #272).
*
* <p>{@code fleet_poll} is <strong>two operations behind one tool name</strong>. With {@code
* ticket} it observes an async delegation and changes nothing, which is a {@link
* Authz.Action#READ}. With {@code target} it calls {@link MessageService#drainReplies} on that
* session -- the replies are removed from the inbox and a second call returns nothing -- so it
* is a {@link Authz.Action#DRAIN}, the same gate {@code fleet_ack} already uses for removing a
* single message, and the same one the REST path uses at {@code FleetApp.drainReplies}.
*
* <p>Until this method existed the handler passed a constant {@code READ} for both branches.
* {@code READ} is open to every authenticated role, so any worker could read a peer's id out of
* {@code fleet_list} and destroy the replies that peer had queued for the primary. The gate
* failed open, and it did so because the required action is a function of the arguments while
* the handler chose it before looking at them.
*
* <p>The choice lives in this method, and not inline in the handler, so that a test can assert
* the mapping the handler actually uses. {@code FleetMcpAuthzTest} already checked every
* {@link Authz.Action} against every {@link Role} and passed throughout -- it tested the policy
* table, which was correct, while the defect was in which action the caller handed it.
*
* @param target the {@code target} argument of the call, or {@code null}/blank when absent
*/
static Authz.Action pollAction(String target) {
return isBlank(target) ? Authz.Action.READ : Authz.Action.DRAIN;
}
/** {@code fleet_poll}: check an async delegation by ticket, or drain a worker's inbox by target. */
static McpSchema.CallToolResult poll(MessageService messages, String ticket, String target) {
if (!isBlank(target)) {
@@ -117,9 +117,11 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
*
* <p>fleetd #185 stage 2: that mirroring assumption holds only while the member pane runs under
* the SAME OS user as the daemon. When {@code memberHerdrSocket:} is configured, member panes
* run on a second herdr owned by a different user — different {@code $HOME}, different {@code
* secrets.sh}, different environment entirely — so this field's data no longer describes what a
* member pane inherits. See {@link #logCredentialGap} for how that mode is handled.
* are routed to a second herdr, and fleetd has no channel to confirm what OS user that herdr
* runs as — it may be a different user with a different {@code $HOME} and {@code secrets.sh},
* or the same one the daemon runs as. Either way this field's data can no longer be trusted to
* describe what a member pane inherits. See {@link #logCredentialGap} for how that mode is
* handled.
*/
private final Supplier<Set<String>> hostEnvNames;
@@ -1471,9 +1473,9 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
Set<String> brokerUriEnvNames = brokerUriEnvNames();
Set<String> allowed = new java.util.TreeSet<>(
MemberEnvAllowList.derive(profiles.values(), creds.allowSet(), brokerUriEnvNames));
if (creds.sshAuthSockAllowed()) {
if (creds.sshAgentEnvInherited()) {
allowed.add(SSH_AUTH_SOCK);
} // blocked by default: absent from the set ⇒ blanked by the scrub like any other name
} // omitted by default: absent from the set ⇒ blanked by the scrub like any other name
allowed.addAll(launch.env().keySet());
allowed.removeAll(brokerUriEnvNames);
return allowed;
@@ -1495,10 +1497,27 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
* survive {@code allowed} (including the {@code LC_*} prefix rule). Neither number is a constant:
* both come from the actual derived set and the actual environment this spawn sees. Never logs a
* variable NAME or VALUE — only the counts.
*
* <p><strong>Under {@code memberHerdrSocket} the counts describe fleetd's own process, not the
* member's</strong> (fleetd #269 follow-up), so the message says so rather than leaving the
* reader to infer it from this javadoc, which the operator reading the log never sees.
*/
private void logAllowListCoverage(Set<String> allowed) {
Set<String> hostNames = hostEnvNames.get();
long kept = hostNames.stream().filter(name -> MemberEnvAllowList.keeps(allowed, name)).count();
if (memberHerdrSocketConfigured()) {
// fleetd #269 covered the sibling line below (logCredentialGap) and stopped there.
// This line has the same problem: read plainly, "allowed 7 of 39" is a statement about
// the member's pane, and under memberHerdrSocket it is not -- the pane is routed to a
// second herdr whose environment fleetd cannot inspect. The counts stay useful, so
// this is not a WARN and not a refusal; only the claim is narrowed to what is true.
log.info("member credentials: allowed {} of {} names in fleetd's OWN environment — "
+ "memberHerdrSocket is configured, so member panes are routed to a "
+ "second herdr whose environment fleetd has no channel to inspect. "
+ "These counts describe fleetd's process, NOT the member pane's.",
kept, hostNames.size());
return;
}
log.info("member credentials: allowed {} of {}", kept, hostNames.size());
}
@@ -1533,21 +1552,24 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
/**
* fleetd #213: {@code memberHerdrSocket} is configured and the member login shell IS zsh, but
* {@code worktreeRoot} and/or {@code worktreeGroup} is missing, so the generated ZDOTDIR cannot
* be placed anywhere the member's OS user can reach — {@code java.io.tmpdir} is fleetd's own
* 0700 temp dir, unreadable by another uid, which is the exact gap this ticket exists to close.
* Say so once per launcher instance, instead of either generating a directory nothing can read
* (protection theatre) or refusing to spawn (turning a degraded credential control into an
* outage for an opt-in feature).
* be placed anywhere fleetd can be sure the member's OS user can reach — {@code java.io.tmpdir}
* is fleetd's own 0700 temp dir, which is unreadable if the member pane runs as a different OS
* user, and fleetd has no channel to confirm whether it does or not. Rather than gamble on that,
* this treats memberHerdrSocket as reason enough to require an explicitly shared location, which
* is the exact gap this ticket exists to close. Say so once per launcher instance, instead of
* either generating a directory that might not be readable (protection theatre) or refusing to
* spawn (turning a degraded credential control into an outage for an opt-in feature).
*/
private void warnCannotShareScrubDirectory() {
if (cannotShareScrubDirWarned.compareAndSet(false, true)) {
log.warn("memberCredentials policy=allow-list: memberHerdrSocket is configured and the "
+ "member login shell is zsh, but worktreeRoot and/or worktreeGroup is not "
+ "configured — the generated ZDOTDIR cannot be placed where the member's OS "
+ "user can read it (java.io.tmpdir is fleetd's own, unreadable by another uid), "
+ "so the scrub cannot be guaranteed to run. Falling back to the CB-596 sentinel "
+ "overlay. Configure both worktreeRoot and worktreeGroup to enable the "
+ "allow-list scrub under memberHerdrSocket.");
+ "configured — the generated ZDOTDIR cannot be placed where fleetd can be sure "
+ "the member's OS user can read it (java.io.tmpdir is fleetd's own 0700 dir, "
+ "unreadable if the member runs as a different OS user — fleetd has no channel "
+ "to confirm whether it does), so the scrub cannot be guaranteed to run. Falling "
+ "back to the CB-596 sentinel overlay. Configure both worktreeRoot and "
+ "worktreeGroup to enable the allow-list scrub under memberHerdrSocket.");
}
}
@@ -1626,10 +1648,12 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
private final AtomicBoolean unknownMemberEnvironmentWarned = new AtomicBoolean();
/**
* fleetd #185 stage 2: whether {@code memberHerdrSocket:} is configured, i.e. member panes run
* on a second herdr owned by a different OS user than the daemon's own process. Re-read from the
* live config on every call (same hot-reload shape as {@link #memberCredentials}), never cached,
* so a config reload takes effect on the next spawn without a restart.
* fleetd #185 stage 2: whether {@code memberHerdrSocket:} is configured, i.e. member panes are
* routed to a second herdr. This tests only that the config key is set — fleetd has no channel
* to confirm what OS user that second herdr runs as, so a {@code true} result means "member
* panes may run under a different OS user," not that they do. Re-read from the live config on
* every call (same hot-reload shape as {@link #memberCredentials}), never cached, so a config
* reload takes effect on the next spawn without a restart.
*
* <p>{@link #config} is {@code null} on any call site that never threaded the full config
* through (every production {@code HerdrPeerLauncher} does; a handful of older tests do not) —
@@ -1652,14 +1676,15 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
* fleetd #185 stage 2: the single replacement WARN for {@link #logCredentialGap}'s usual
* conclusions when {@code memberHerdrSocket:} is configured. {@link #hostEnvNames} (and
* everything derived from it — {@code known}/{@code allow} coverage, the allow-list scrub's
* derived set) describes the DAEMON's own environment; under this config key member panes run as
* a different OS user with a different environment entirely, so neither "every member pane
* inherits them UNBLOCKED" nor "the scrub blanks them" is evidence-backed here — both would be
* reporting on the wrong process. Logged once, names the config key, and states the honest
* conclusion: the gap for member panes is UNKNOWN, not clean, so {@code memberCredentials} cannot
* be verified from this daemon. The one count it does report is scoped explicitly to fleetd's own
* environment, never presented as if it said anything about the member's — see {@link
* #logCredentialGap}'s javadoc for why this branch exists.
* derived set) describes the DAEMON's own environment; under this config key member panes are
* routed to a second herdr, and fleetd has no channel to confirm what OS user that herdr runs
* as or to read its environment, so neither "every member pane inherits them UNBLOCKED" nor
* "the scrub blanks them" is evidence-backed here — both would be reporting on the wrong
* process. Logged once, names the config key, and states the honest conclusion: the gap for
* member panes is UNKNOWN, not clean, so {@code memberCredentials} cannot be verified from this
* daemon. The one count it does report is scoped explicitly to fleetd's own environment, never
* presented as if it said anything about the member's — see {@link #logCredentialGap}'s javadoc
* for why this branch exists.
*/
private void warnUnknownMemberEnvironment(FleetConfig.MemberCredentials creds) {
if (!unknownMemberEnvironmentWarned.compareAndSet(false, true)) {
@@ -1672,13 +1697,13 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
.filter(name -> CREDENTIAL_SHAPED_NAME.matcher(name).matches())
.filter(name -> !covered.contains(name))
.count();
log.warn("memberCredentials gap: memberHerdrSocket is configured, so member panes run under "
+ "a different OS user than fleetd's own process, with a different environment "
+ "entirely — fleetd has no channel to read that user's environment. {} of the "
+ "{} names in fleetd's OWN environment are credential-shaped and not on "
+ "known:/allow:, but that count describes fleetd's process, not the member "
+ "herdr's. The credential gap for member panes is UNKNOWN, not clean, and "
+ "memberCredentials cannot be verified from here.",
log.warn("memberCredentials gap: memberHerdrSocket is configured, so member panes are routed "
+ "to a second herdr — fleetd has no channel to confirm what OS user that herdr "
+ "runs as, so it cannot tell whether those panes inherit its own environment or "
+ "a different one entirely. {} of the {} names in fleetd's OWN environment are "
+ "credential-shaped and not on known:/allow:, but that count describes fleetd's "
+ "process, not the member herdr's. The credential gap for member panes is "
+ "UNKNOWN, not clean, and memberCredentials cannot be verified from here.",
gapInFleetdsOwnEnv, hostNames.size());
}
@@ -34,7 +34,7 @@ import java.util.TreeSet;
*
* <p>{@code SSH_AUTH_SOCK} is deliberately NOT here. It is a handle to the operator's ssh-agent — a
* member holding it can sign with the operator's keys — so keeping it is a config decision
* ({@code memberCredentials.sshAuthSock: allow}), not a derivation default.
* ({@code memberCredentials.sshAgentEnv: inherit}), not a derivation default.
*
* <p><b>CB-633 follow-up:</b> the union also includes {@code memberCredentials.allow:} — the
* operator's own explicit list. Before this, {@code policy: allow-list} silently ignored every name
@@ -42,7 +42,7 @@ import java.util.TreeSet;
* turning the policy on could blank credentials working members already depended on. {@code
* SSH_AUTH_SOCK} and configured broker URI environment names are exceptions: even when the operator
* lists them under {@code allow:}, they are excluded here. {@code SSH_AUTH_SOCK} is added back ONLY
* by the caller when {@code sshAuthSock: allow} is explicitly set
* by the caller when {@code sshAgentEnv: inherit} is explicitly set
* (see {@link #SSH_AUTH_SOCK}'s javadoc) — it is a live handle to the operator's own ssh-agent, not
* a value, so treating it like any other allow-listed name would hand a member every key the
* operator's agent holds the moment they typed the name under {@code allow:} for an unrelated
@@ -53,7 +53,7 @@ public final class MemberEnvAllowList {
/**
* The operator's ssh-agent socket path. Deliberately excluded from {@link #derive}'s union of
* {@code memberCredentials.allow:} — see the class javadoc's CB-633 follow-up note. Governed
* ONLY by {@code memberCredentials.sshAuthSock}, never by appearing in {@code allow:}.
* ONLY by {@code memberCredentials.sshAgentEnv}, never by appearing in {@code allow:}.
*/
public static final String SSH_AUTH_SOCK = "SSH_AUTH_SOCK";
@@ -21,6 +21,7 @@ import java.util.EnumSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.BooleanSupplier;
@@ -669,6 +670,16 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
private final AtomicBoolean discoveryUnavailableWarned =
new AtomicBoolean();
/**
* fleetd #267: one WARN per PROFILE (not per launcher instance — several profiles can each hit
* this gap independently) for the model-mismatch check (fleetd #175) never getting to run
* because the spawn was not given a fleetd-provisioned worktree (fleetd #249). Profile names
* accumulate here for the life of this launcher instance and are never removed — the same
* one-shot treatment {@link #discoveryUnavailableWarned} already gets, just keyed per profile
* instead of globally.
*/
private final Set<String> modelCheckSkippedWarned = ConcurrentHashMap.newKeySet();
/** Add lazy on-disk session discovery to the base handle. */
@Override
public PeerHandle spawn(SpawnRequest req) {
@@ -695,7 +706,8 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
// be running.
FleetConfig.Profile cfg = requireProfile(req.profileName());
return new SessionAwareHandle(inner, discovery, cwd, cfg,
this::memberHerdrSocketConfigured, discoveryUnavailableWarned, exhaustionSink);
this::memberHerdrSocketConfigured, discoveryUnavailableWarned,
modelCheckSkippedWarned, exhaustionSink);
}
/**
@@ -720,6 +732,7 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
private final FleetConfig.Profile cfg;
private final BooleanSupplier discoveryUnavailable;
private final AtomicBoolean discoveryUnavailableWarned;
private final Set<String> modelCheckSkippedWarned;
private final ExhaustionSink exhaustionSink;
/** CAS'd true the first (and only) time a model mismatch is reported for this handle. */
private final AtomicBoolean modelMismatchReported = new AtomicBoolean();
@@ -750,6 +763,7 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
FleetConfig.Profile cfg,
BooleanSupplier discoveryUnavailable,
AtomicBoolean discoveryUnavailableWarned,
Set<String> modelCheckSkippedWarned,
ExhaustionSink exhaustionSink) {
this.delegate = delegate;
this.discovery = discovery;
@@ -757,6 +771,7 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
this.cfg = cfg;
this.discoveryUnavailable = discoveryUnavailable;
this.discoveryUnavailableWarned = discoveryUnavailableWarned;
this.modelCheckSkippedWarned = modelCheckSkippedWarned;
this.exhaustionSink = exhaustionSink;
this.worktreeProvisioned = isProvisionedWorktree(cwd);
}
@@ -807,9 +822,29 @@ public final class OpenCodeLauncher extends HerdrPeerLauncher {
// member's row apart from a sibling's in that case (measured: a three-day-old row from
// a different profile). Refuse to guess — absent is the honest answer, and it is what
// this codebase already returns elsewhere for absent evidence (fleetd #175's UNKNOWN).
// No WARN here: unlike discoveryUnavailable above, this is the ordinary, expected shape
// of the large majority of spawns (no worktree requested), not a configuration gap.
// This IS the ordinary, expected shape of the large majority of spawns (no worktree
// requested), not a configuration gap — but fleetd #267 found that same shape silently
// switches off the fleetd #175 model-mismatch check for those spawns too, since
// checkModelMatch's only call site is right below this gate. The check cannot be moved
// off agentSessionId()'s resolved id: the id is the only safe way to key
// actualModelForSessionId to THIS session's own row rather than "whatever is newest in
// the shared directory" (fleetd #234) — re-deriving a second, independent answer via
// `directory` here would reintroduce exactly the false-positive risk #234 fixed (a
// sibling's differently-configured model looking like THIS profile's mismatch). So the
// model genuinely is unknowable without a provisioned worktree, and unlike the silence
// this branch used to keep, that gap now gets the same one-time, per-profile WARN
// treatment discoveryUnavailable already gets above — but keyed by profile, since
// several profiles can each hit this independently.
if (!worktreeProvisioned) {
if (cfg.model() != null && !cfg.model().isBlank()
&& modelCheckSkippedWarned.add(cfg.profile())) {
log.warn("opencode model-mismatch check (fleetd #175) cannot run for profile "
+ "'{}': it was spawned without a fleetd-provisioned worktree (fleetd "
+ "#249), so its cwd may be shared with other sessions and the actual "
+ "model it is running cannot be safely told apart from a sibling's — "
+ "spawn with worktree:true to enable the check for this profile.",
cfg.profile());
}
return null;
}
// fleetd #234: once resolved, stay resolved. Re-deriving from `directory` on every call
@@ -360,6 +360,18 @@ public final class MessageService {
* {@link #ask} clears the ticket's question and returns it to {@code PENDING}, but {@link #send}
* already closed the forward waiter the instant the question surfaced, so the target has
* neither an accepted nor a queued delivery left to show for it.
*
* <p><strong>Deliberately still {@code question == null} only (fleetd #275).</strong> This
* method must not also report a still-{@link Phase#ASKING} task as orphaned: the worker may
* genuinely be waiting on a live primary that is about to (or already mid-{@link #answer})
* answer it, and {@link dev.ltms.fleet.health.FleetHealthMonitor} would classify that as
* {@code DELEGATION_ORPHANED} on nothing more than an active, healthy conversation. {@link
* #abandon(String, String, boolean)}'s {@code sweepAsking} path fixes the actual reachable gap
* (a target torn down for good while genuinely {@code ASKING}) at the point of teardown itself,
* by completing the task's future right there — so by the time this method would ever see it,
* {@code task.future.isDone()} is already {@code true} and it is excluded regardless of this
* guard. Widening this check instead of that one would trade a real fix for false positives on
* every ordinary in-flight question.
*/
public boolean hasOrphanedDelegation(String target) {
if (target == null || hasAcceptedDelivery(target) || hasQueuedDelivery(target)) {
@@ -580,6 +592,39 @@ public final class MessageService {
* reply — see the note above)
*/
public boolean abandon(String target, String reason) {
return abandon(target, reason, false);
}
/**
* As {@link #abandon(String, String)}, with control over whether a task still paused in
* {@code fleet_ask} ({@link Phase#ASKING}) is swept too (fleetd #275).
*
* <p>{@code sweepAsking} must be {@code true} only when the caller has independent, certain
* knowledge that {@code target} can never resume its turn — today that is only
* {@code sessions.onRelease}'s teardown (an explicit {@code fleet_stop}, or the idle reaper):
* the worker's pane is being stopped right now, so whatever it was mid-{@code fleet_ask} about
* has no turn left to resume into. {@link dev.ltms.fleet.health.FleetHealthMonitor}'s
* health-classification call keeps passing {@code false} (via {@link #abandon(String, String)}):
* a GONE/NEVER_READY reading is the daemon's best guess from the live agent list, not a teardown
* it performed itself, and {@code abandonDoesNotFailAnAsyncTicketWaitingForAnAnswer} documents
* why an active ask must survive that guess — the primary may already be mid-{@link #answer} for
* the very same turn, and completing it here first would preempt a real answer with a misleading
* failure.
*
* <p><strong>Without {@code sweepAsking} on the release path, a target torn down while
* genuinely {@code ASKING} was unrecoverable.</strong> {@link #resolveQuestion} had already
* closed the forward waiter the instant the question surfaced (so the {@code waiter} branch
* below finds nothing to fail), the {@code question == null} guard excluded the task from
* {@code matching} (so the loop below skipped it too), and the worker's own {@code fleet_ask}
* clears {@link Task#question} back to {@code null} only once it lapses (the reverse-rendezvous
* window — up to {@code FleetMcp.ASK_DEFAULT_TIMEOUT_MS} / {@code FleetApp.MAX_ASK_TIMEOUT_MS},
* 55–115s) — by which point the released session no longer appears in {@code sessions.roster()}
* for {@link dev.ltms.fleet.health.FleetHealthMonitor} to ever re-observe, so nothing was ever
* left to call {@link #abandon} on this target again. The ticket then sat in {@link #tasks}
* forever: not terminal, so {@link #pruneTerminalTickets} never dropped it, and
* {@code fleet_poll} reported it stuck at {@link Phase#PENDING} for good.
*/
public boolean abandon(String target, String reason, boolean sweepAsking) {
boolean hadStrandedReply = hasStrandedReply(target);
// CB-640: the session is gone — nothing will ever accept or deliver into it now.
strandedReplies.remove(target);
@@ -590,7 +635,8 @@ public final class MessageService {
List<Task> matching = new ArrayList<>();
for (Task task : tasks.values()) {
if (target.equals(task.target) && task.question == null && !task.future.isDone()) {
if (target.equals(task.target) && (sweepAsking || task.question == null)
&& !task.future.isDone()) {
matching.add(task);
}
}
@@ -607,11 +653,20 @@ public final class MessageService {
for (Task task : matching) {
boolean isRecovery = task == recoveryTask && recovered != null;
Reply outcome = isRecovery ? recovered : new Reply(Outcome.WORKER_FAILED, reason);
String turnId = task.turnId;
if (task.future.complete(outcome)) {
if (outcome.outcome() == Outcome.WORKER_FAILED) {
asyncFailed = true;
} else if (task.turnId != null) {
asyncTasksByTurn.remove(task.turnId, task);
}
if (turnId != null) {
// #275: whether this task was swept out of ASKING or was already answered and
// only waiting on its resumed turn's real reply (#137), nothing will ever
// complete this turnId now — drop it from this class's own bookkeeping AND the
// reverse-rendezvous itself, so hasAsyncQuestion(target) stops reporting a turn
// that is actually done, and a late answer() sees it as lapsed rather than
// resolving a question nothing is listening for any more.
asyncTasksByTurn.remove(turnId, task);
rendezvous.closeAsk(turnId);
}
} else if (isRecovery) {
// The recovered reply was already drained out of the inbox, but this task resolved
@@ -167,14 +167,59 @@ public final class GitWorktrees implements Worktrees {
log.info("adding worktree branch={} path={} base={}", branch, wt, base);
removeUserInfoFromHttpsOrigin(repoRoot);
exec("git", "-C", repoRoot, "worktree", "add", wt, "-b", branch, base);
afterWorktreeAdded.accept(wt);
requireCredentialFreeHttpsOrigin(wt);
configureEnvironmentCredentialHelper(repoRoot, wt);
configureHttpsUrlRewriteForSshOrigin(repoRoot, wt);
isolateToolSurface(wt);
try {
afterWorktreeAdded.accept(wt);
requireCredentialFreeHttpsOrigin(wt);
configureEnvironmentCredentialHelper(repoRoot, wt);
configureHttpsUrlRewriteForSshOrigin(repoRoot, wt);
isolateToolSurface(wt);
} catch (RuntimeException e) {
cleanupAfterAddFailure(repoRoot, wt, branch, e);
throw e;
}
return wt;
}
/**
* {@code add()} has already created the worktree and its branch by the time any step from
* {@link #afterWorktreeAdded} through {@link #isolateToolSurface} can throw — including
* {@link #requireCredentialFreeHttpsOrigin}, an intended security refusal, not only an IO
* accident. Without this, {@code add()} never returns, so its caller
* ({@code SessionManager#acquireWithWorktree}) never receives a path to register or clean up:
* its local {@code path} stays null, the {@code if (path != null)} guard in its own catch block
* never runs, and the worktree directory and branch leak on disk forever with nothing tracking
* them (fleetd #274).
*
* <p>Reuses {@link #remove} — the same {@code git worktree remove --force} path every other
* cleanup exit in this class already goes through — rather than a bespoke removal. It
* additionally deletes {@code branch}: {@link #remove} alone deliberately leaves a released
* session's branch behind (a worker's branch is expected to outlive its worktree, for PRs and
* recovery), but a branch that never finished provisioning has no session, no PR, and nothing
* else pointing at it, so leaving it behind would just trade one leak for a smaller one. Forced
* (`-D`) because the branch is new and unmerged by construction. The worktree is removed first:
* a branch checked out by a worktree cannot be deleted until the worktree that holds it is gone.
*
* <p>Cleanup failure must never mask {@code original} — that is the exception that explains
* what actually went wrong — so a failure here is only logged, matching the pattern already
* used in {@code SessionManager#acquireWithWorktree}'s own catch block.
*/
private void cleanupAfterAddFailure(String repoRoot, String worktreePath, String branch, RuntimeException original) {
log.warn("provisioning failed for branch={} path={}: {} — cleaning up before rethrowing",
branch, worktreePath, original.getMessage());
try {
remove(repoRoot, worktreePath);
} catch (RuntimeException cleanup) {
log.warn("failed to remove leaked worktree {} after provisioning error: {}",
worktreePath, cleanup.getMessage());
}
try {
exec("git", "-C", repoRoot, "branch", "-D", branch);
} catch (RuntimeException cleanup) {
log.warn("failed to remove leaked branch {} after provisioning error: {}",
branch, cleanup.getMessage());
}
}
/**
* A linked worktree shares its primary checkout's git config. Remove HTTPS user info before
* adding one, so a credential accidentally embedded in that config cannot reach the member.
@@ -0,0 +1,73 @@
package dev.ltms.fleet;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
import dev.ltms.fleet.config.FleetConfig;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.slf4j.LoggerFactory;
import java.nio.file.Files;
import java.nio.file.Path;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* fleetd #184: startup must state whether members share fleetd's OS user or use a separate herdr.
*/
class MemberTrustModelReportTest {
private static FleetConfig load(Path dir, String yaml) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, yaml);
return FleetConfig.load(f);
}
private static ListAppender<ILoggingEvent> attach() {
Logger logger = (Logger) LoggerFactory.getLogger(Fleetd.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
return appender;
}
private static void detach(ListAppender<ILoggingEvent> appender) {
((Logger) LoggerFactory.getLogger(Fleetd.class)).detachAppender(appender);
}
private static String report(FleetConfig cfg) {
Logger logger = (Logger) LoggerFactory.getLogger(Fleetd.class);
ch.qos.logback.classic.Level original = logger.getLevel();
logger.setLevel(ch.qos.logback.classic.Level.INFO);
ListAppender<ILoggingEvent> appender = attach();
try {
Fleetd.reportMemberTrustModel(cfg);
} finally {
detach(appender);
logger.setLevel(original);
}
return appender.list.getFirst().getFormattedMessage();
}
@Test
void unsetMemberHerdrSocketStatesThatMembersAreNotSandboxed(@TempDir Path dir) throws Exception {
FleetConfig cfg = load(dir, "bind:\n host: 127.0.0.1\n port: 8765\n");
assertEquals("member trust model: members run as the same OS user as fleetd, not in a sandbox. "
+ "A member can read any file this user can read, including SSH keys and credential "
+ "stores, whatever memberCredentials says. To add a real boundary, route members to "
+ "a second herdr under a different OS user with memberHerdrSocket.",
report(cfg));
}
@Test
void configuredMemberHerdrSocketStatesThatFleetdCannotConfirmTheBoundary(@TempDir Path dir) throws Exception {
FleetConfig cfg = load(dir, "memberHerdrSocket: /tmp/member-herdr.sock\n");
assertEquals("member trust model: members are routed to a separate herdr through "
+ "memberHerdrSocket. fleetd cannot see that herdr's uid, so confirm it runs "
+ "as a different OS user before treating it as a boundary.",
report(cfg));
}
}
@@ -0,0 +1,96 @@
package dev.ltms.fleet;
import com.tngtech.archunit.base.DescribedPredicate;
import com.tngtech.archunit.core.domain.JavaClass;
import com.tngtech.archunit.core.domain.JavaClass.Predicates;
import com.tngtech.archunit.core.importer.ClassFileImporter;
import com.tngtech.archunit.core.importer.ImportOption;
import com.tngtech.archunit.library.dependencies.SliceRule;
import com.tngtech.archunit.library.dependencies.SlicesRuleDefinition;
import org.junit.jupiter.api.Test;
/**
* fleetd #131 (CB-627): enforce package boundaries with an ArchUnit test instead of a
* Maven module split.
*
* <p>This test fails the build the moment a NEW cycle appears between the top-level
* {@code dev.ltms.fleet.*} packages. Today's cycles are recorded below as explicit,
* narrow exceptions: each one ignores dependencies between exactly the two named
* packages, in both directions, and nothing else. A cycle through any other pair of
* packages -- or a brand new pair -- still fails this test.
*
* <p><b>Main code only.</b> The import excludes test classes
* ({@link ImportOption.Predefined#DO_NOT_INCLUDE_TESTS}). Test code legitimately wires
* across many packages for setup and mocking; that is not part of the shipped
* architecture this rule protects. Verified: importing test classes too pulls in a much
* larger, noisier cycle set -- {@code herdr}, {@code member}, {@code peer}, {@code
* config}, {@code guard} and {@code placement} all show up in cycles that disappear the
* moment test classes are excluded. Scanning off the classpath via {@code
* importPackages(...)} (not a hardcoded {@code target/classes} path) also keeps this
* test correct regardless of the working directory the build is invoked from.
*
* <p><b>No package moves here</b> -- ticket #131 is explicit that removing a cycle is
* its own, later PR. See the comment on each exception below for which ticket step
* removes it.
*/
class PackageCyclesTest {
@Test
void packagesAreFreeOfCycles() {
var classes = new ClassFileImporter()
.withImportOption(ImportOption.Predefined.DO_NOT_INCLUDE_TESTS)
.importPackages("dev.ltms.fleet");
SliceRule rule = SlicesRuleDefinition.slices()
.matching("dev.ltms.fleet.(*)..")
.should().beFreeOfCycles();
// fleetd #131 step 1: move ConnectionIdentity so authz stops depending on the
// MCP layer. Evidence: auth/CallerResolver.java:3 imports mcp.ConnectionIdentity;
// mcp/FleetMcp.java:3-7 imports auth.AuditLog, Authz, CallerResolver, Principal,
// Role.
rule = ignoreCycle(rule, "auth", "mcp");
// fleetd #131 step 2: PrimaryRegistry is used by loops in msg; move it, or put
// an interface between msg and mcp. Evidence: msg/ReplyPushLoop.java:5 and
// msg/LeadHeartbeatLoop.java:5 import mcp.PrimaryRegistry; mcp/FleetMcp.java:15-18
// imports msg.LeadChannel, LeadMessage, MessageService, Rendezvous.
rule = ignoreCycle(rule, "mcp", "msg");
// fleetd #131 -- found while implementing this test, NOT one of the ticket's
// original three; it names its own follow-up step before removal. Evidence:
// inject/CompletionResolver.java:4-5, inject/Injector.java:6 and
// inject/TurnListener.java:3 import msg.Rendezvous / msg.TurnToken;
// msg/MessageService.java:6 imports inject.Injector.
rule = ignoreCycle(rule, "inject", "msg");
// fleetd #131 -- same as above, its own follow-up. Evidence:
// metrics/FleetMetrics.java:3 imports msg.ReplyInbox; msg/MessageService.java:7-8,
// msg/LeadHeartbeatLoop.java:6-7 and msg/ReplyPushLoop.java:6-7 import
// metrics.FleetMetrics / metrics.Metrics.
rule = ignoreCycle(rule, "metrics", "msg");
// fleetd #131 -- same as above, its own follow-up. Evidence:
// session/SessionManager.java:7 imports msg.TurnToken;
// msg/LeadHeartbeatLoop.java:8 imports session.MemberSession.
rule = ignoreCycle(rule, "msg", "session");
rule.check(classes);
}
/**
* Accepts today's known cycle between two top-level packages, and nothing else.
* Ignoring both directions removes exactly this pair from cycle detection; every
* other dependency -- including any new one added later, between these same two
* packages or any other pair -- is still checked.
*/
private static SliceRule ignoreCycle(SliceRule rule, String packageA, String packageB) {
return rule
.ignoreDependency(residesIn(packageA), residesIn(packageB))
.ignoreDependency(residesIn(packageB), residesIn(packageA));
}
private static DescribedPredicate<JavaClass> residesIn(String topLevelPackage) {
return Predicates.resideInAPackage("dev.ltms.fleet." + topLevelPackage + "..");
}
}
@@ -155,6 +155,90 @@ class FleetConfigTest {
assertTrue(e.getMessage().contains("errorPattern"), "the offending key is named: " + e.getMessage());
}
// ── fleetd #273: exhaustedPattern gets the same load-time validation as its sibling errorPattern ──
@Test
void aProfileWithAMalformedExhaustedPatternIsRejectedAtLoadNamingTheProfileAndKey(@TempDir Path dir)
throws Exception {
Path f = dir.resolve("malformed-exhausted-pattern.yaml");
Files.writeString(f, """
profiles:
ltms-local:
baseUrl: http://gx00.gw:8000
exhaustedPattern: "["
""");
IllegalStateException e = assertThrows(IllegalStateException.class, () -> FleetConfig.load(f));
assertTrue(e.getMessage().contains("ltms-local"), "the offending profile is named: " + e.getMessage());
assertTrue(e.getMessage().contains("exhaustedPattern"), "the offending key is named: " + e.getMessage());
}
@Test
void aMalformedErrorPatternAndAMalformedExhaustedPatternAreBothReportedFromOneLoad(@TempDir Path dir)
throws Exception {
Path f = dir.resolve("both-malformed.yaml");
Files.writeString(f, """
profiles:
sonnet:
baseUrl: http://gx00.gw:8000
errorPattern: "(unterminated["
terra:
baseUrl: http://gx01.gw:8000
exhaustedPattern: "["
""");
IllegalStateException e = assertThrows(IllegalStateException.class, () -> FleetConfig.load(f));
assertTrue(e.getMessage().contains("sonnet"), "the errorPattern profile is named: " + e.getMessage());
assertTrue(e.getMessage().contains("errorPattern"), e.getMessage());
assertTrue(e.getMessage().contains("terra"), "the exhaustedPattern profile is named: " + e.getMessage());
assertTrue(e.getMessage().contains("exhaustedPattern"), e.getMessage());
}
@Test
void validErrorPatternAndExhaustedPatternBothLoadFine(@TempDir Path dir) throws Exception {
Path f = dir.resolve("both-valid.yaml");
Files.writeString(f, """
profiles:
ltms-local:
baseUrl: http://gx00.gw:8000
errorPattern: "credential outage"
exhaustedPattern: "usage limit has been reached"
""");
FleetConfig.Profile w = FleetConfig.load(f).profiles().get("ltms-local");
assertEquals("credential outage", w.errorPattern());
assertEquals("usage limit has been reached", w.exhaustedPattern());
}
@Test
void aBlankExhaustedPatternNormalizesToNullJustLikeUnset(@TempDir Path dir) throws Exception {
Path f = dir.resolve("blank-exhausted-pattern.yaml");
Files.writeString(f, """
profiles:
ltms-local:
baseUrl: http://gx00.gw:8000
exhaustedPattern: " "
""");
FleetConfig.Profile w = FleetConfig.load(f).profiles().get("ltms-local");
assertNull(w.exhaustedPattern());
assertFalse(w.hasExhaustedPattern());
}
@Test
void aProfileWithNoExhaustedPatternLoadsFine(@TempDir Path dir) throws Exception {
Path f = dir.resolve("no-exhausted-pattern.yaml");
Files.writeString(f, """
profiles:
ltms-local:
baseUrl: http://gx00.gw:8000
""");
FleetConfig.Profile w = FleetConfig.load(f).profiles().get("ltms-local");
assertNull(w.exhaustedPattern());
assertFalse(w.hasExhaustedPattern());
}
@Test
void withProfileCarriesErrorPatternThrough(@TempDir Path dir) throws Exception {
Path f = dir.resolve("with-profile-error-pattern.yaml");
@@ -2010,22 +2094,93 @@ class FleetConfigTest {
"deny-list normalizes onto the canonical deny-by-default value");
}
/**
* CB-633: {@code SSH_AUTH_SOCK} is a decision, never a default — absent, blank, or misspelled,
* it stays BLOCKED; only the literal "allow" (any case) passes it through. A typo like "alow"
* failing safe here is the whole point of making it a knob.
*/
@Test
void sshAuthSockDefaultsToBlockedAndOnlyExplicitAllowUnblocksIt() {
assertTrue(new FleetConfig.MemberCredentials("allow-list", List.of(), List.of()).sshAuthSock().equals("block"),
"absent knob blocks SSH_AUTH_SOCK");
assertFalse(new FleetConfig.MemberCredentials("allow-list", List.of(), List.of()).sshAuthSockAllowed());
assertFalse(new FleetConfig.MemberCredentials(null, null, null, "").sshAuthSockAllowed(),
"blank knob blocks SSH_AUTH_SOCK");
assertFalse(new FleetConfig.MemberCredentials(null, null, null, "alow").sshAuthSockAllowed(),
"a misspelled value fails SAFE, not open");
assertTrue(new FleetConfig.MemberCredentials(null, null, null, "ALLOW").sshAuthSockAllowed(),
"the literal allow (case-insensitive) unblocks SSH_AUTH_SOCK");
void sshAgentEnvAcceptsEveryCompatibleKeyAndValuePair(@TempDir Path dir) throws Exception {
String[][] spellings = {
{"sshAuthSock", "block", "omit"},
{"sshAuthSock", "allow", "inherit"},
{"sshAuthSock", "omit", "omit"},
{"sshAuthSock", "inherit", "inherit"},
{"sshAgentEnv", "block", "omit"},
{"sshAgentEnv", "allow", "inherit"},
{"sshAgentEnv", "omit", "omit"},
{"sshAgentEnv", "inherit", "inherit"}
};
for (int i = 0; i < spellings.length; i++) {
Path file = dir.resolve("member-credentials-ssh-agent-" + i + ".yaml");
Files.writeString(file, """
bind:
port: 8080
memberCredentials:
policy: allow-list
""" + " " + spellings[i][0] + ": " + spellings[i][1] + "\n");
FleetConfig.MemberCredentials credentials = FleetConfig.load(file).memberCredentials();
assertEquals(spellings[i][2], credentials.sshAgentEnv(),
spellings[i][0] + ": " + spellings[i][1] + " must normalize correctly");
assertEquals("inherit".equals(spellings[i][2]), credentials.sshAgentEnvInherited());
}
}
/** Protects the live {@code sshAuthSock: block} allow-list configuration during the rename. */
@Test
void legacySshAuthSockBlockKeepsLiveAllowListConfigOmitted(@TempDir Path dir) throws Exception {
Path file = dir.resolve("live-member-credentials.yaml");
Files.writeString(file, """
bind:
port: 8080
memberCredentials:
policy: allow-list
sshAuthSock: block
""");
FleetConfig.MemberCredentials credentials = FleetConfig.load(file).memberCredentials();
assertEquals("omit", credentials.sshAgentEnv());
assertFalse(credentials.sshAgentEnvInherited(), "the live config must omit SSH_AUTH_SOCK");
}
@Test
void sshAgentEnvWinsWhenBothCompatibleKeysArePresent(@TempDir Path dir) throws Exception {
Path file = dir.resolve("both-ssh-agent-keys.yaml");
Files.writeString(file, """
bind:
port: 8080
memberCredentials:
policy: allow-list
sshAuthSock: allow
sshAgentEnv: omit
""");
FleetConfig.MemberCredentials credentials = FleetConfig.load(file).memberCredentials();
assertEquals("omit", credentials.sshAgentEnv());
assertFalse(credentials.sshAgentEnvInherited());
}
@Test
void sshAgentEnvDefaultsToOmitAndUnknownValuesFailClosed(@TempDir Path dir) throws Exception {
Path absent = dir.resolve("member-credentials-ssh-agent-absent.yaml");
Files.writeString(absent, """
bind:
port: 8080
memberCredentials:
policy: allow-list
""");
assertEquals("omit", FleetConfig.load(absent).memberCredentials().sshAgentEnv());
Path unknown = dir.resolve("member-credentials-ssh-agent-unknown.yaml");
Files.writeString(unknown, """
bind:
port: 8080
memberCredentials:
policy: allow-list
sshAgentEnv: inhert
""");
FleetConfig.MemberCredentials credentials = FleetConfig.load(unknown).memberCredentials();
assertEquals("omit", credentials.sshAgentEnv());
assertFalse(credentials.sshAgentEnvInherited(), "an unknown value must fail closed");
}
/**
@@ -179,6 +179,53 @@ class FleetMcpAuthzTest {
"no CallerResolver supplied ⇒ authorization not enforced (legacy behaviour)");
}
// --- which action each tool hands the gate (fleetd #272) ------------------------------------
/**
* fleetd #272: {@code fleet_poll{target}} drains a session's reply inbox, so it needs
* {@link Authz.Action#DRAIN} -- not the {@link Authz.Action#READ} the handler passed for both
* of its branches until this ticket.
*
* <p>This asserts against {@link FleetMcp#pollAction}, the method the handler itself calls, so
* the handler holds no separate copy of the rule that this test could miss. Every other test in
* this class checks the policy table (is a worker allowed to DRAIN?) and all of them passed for
* the whole time the defect was live -- the table was right, the action fed to it was wrong.
*/
@Test
void pollingByTargetIsADrainAndPollingByTicketIsARead() {
assertEquals(Authz.Action.DRAIN, FleetMcp.pollAction("term_b"),
"poll by target removes the replies — that is a drain, not an observation");
assertEquals(Authz.Action.READ, FleetMcp.pollAction(null),
"poll by ticket changes nothing");
assertEquals(Authz.Action.READ, FleetMcp.pollAction(" "),
"a blank target is an absent target");
}
@Test
void aWorkerMayNotDrainAnotherSessionsInboxByPolling() {
FleetMcp m = mcp(true);
assertNotNull(m.denyFor(WORKER_A, FleetMcp.pollAction("term_b"), "term_b"),
"a worker draining a peer's inbox would destroy replies queued for the primary");
assertNotNull(m.denyFor(ARCH_DESIGN, FleetMcp.pollAction("term_b"), "term_b"),
"an architect has no lifecycle rights either — same gate as fleet_ack");
assertNull(m.denyFor(PRIMARY, FleetMcp.pollAction("term_b"), "term_b"),
"collecting a held reply is the primary's job");
}
/**
* The tightening must not close the branch that legitimately serves non-primary callers: an
* architect may {@code fleet_send}, so it owns tickets and must be able to poll them.
*/
@Test
void pollingAnOwnTicketStaysOpenToWorkersAndArchitects() {
FleetMcp m = mcp(true);
assertNull(m.denyFor(WORKER_A, FleetMcp.pollAction(null), null));
assertNull(m.denyFor(ARCH_DESIGN, FleetMcp.pollAction(null), null),
"an architect delegates with wait:false, so it must be able to poll its ticket");
}
// --- identity reconstruction from the transport context ------------------------------------
@Test
@@ -170,10 +170,10 @@ class HerdrPeerLauncherAllowListWiringTest {
/**
* {@code SSH_AUTH_SOCK} is a live ssh-agent handle, not a value — it must stay blocked under
* {@code allow-list} even when the operator lists it under {@code allow:}, because {@code
* sshAuthSock} defaults to blocked. Governed ONLY by {@code memberCredentials.sshAuthSock}.
* sshAgentEnv} defaults to omit. Governed ONLY by {@code memberCredentials.sshAgentEnv}.
*/
@Test
void sshAuthSockStaysBlockedEvenWhenListedInMemberCredentialsAllow() {
void sshAgentEnvStaysOmittedEvenWhenListedInMemberCredentialsAllow() {
FakeHerdr herdr = new FakeHerdr();
WiringLauncher launcher = new WiringLauncher(herdr,
allowListWithAllow(List.of("SSH_AUTH_SOCK")));
@@ -184,7 +184,7 @@ class HerdrPeerLauncherAllowListWiringTest {
String scrub = readAll(dir.resolve(EnvAllowListScrub.SCRUB_FILE));
assertFalse(scrub.contains("'SSH_AUTH_SOCK'"),
"SSH_AUTH_SOCK must not be on the derived allow-list just because the operator put "
+ "it under allow: — sshAuthSock is unset here, so it defaults to block");
+ "it under allow: — sshAgentEnv is unset here, so it defaults to omit");
}
@Test
@@ -250,6 +250,57 @@ class HerdrPeerLauncherAllowListWiringTest {
+ appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList());
}
/**
* fleetd #269 follow-up: the same overclaim the WARN in {@code logCredentialGap} was fixed for,
* in the INFO line beside it. With {@code memberHerdrSocket} configured, member panes are routed
* to a second herdr whose environment fleetd has no channel to inspect, so the counts come from
* fleetd's OWN environment. The bare line "member credentials: allowed 1 of 3" reads as a fact
* about the member's pane, and there it is not one.
*
* <p>#269 reworded four sites and stopped at the sibling below; this pins the pair together so
* a future edit cannot fix one and leave the other. Real path: asserted after a real {@link
* HerdrPeerLauncher#spawn}, reading the log production actually emits.
*/
@Test
void theAllowedCountLineSaysWhoseEnvironmentItCountedWhenMemberHerdrSocketIsSet(@TempDir Path worktreeRoot)
throws IOException {
String group = currentUserGroup();
FakeHerdr herdr = new FakeHerdr();
Set<String> hostEnvNames = Set.of(INJECTED, "SOME_UNRELATED_NAME", "ANOTHER_UNRELATED_NAME");
WiringLauncher launcher = new WiringLauncher(herdr, allowList(), "/bin/bash-should-be-ignored",
() -> hostEnvNames,
() -> configWithMemberHerdrSocketRootAndGroup("/tmp/other-user-herdr.sock", "/bin/zsh",
worktreeRoot.toString(), group));
Logger logger = (Logger) LoggerFactory.getLogger(HerdrPeerLauncher.class);
Level original = logger.getLevel();
logger.setLevel(Level.INFO);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
try {
launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
} finally {
logger.detachAppender(appender);
logger.setLevel(original);
}
List<String> lines = appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList();
String coverage = lines.stream()
.filter(l -> l.startsWith("member credentials: allowed "))
.findFirst()
.orElse(null);
assertNotNull(coverage, "the coverage line must still be logged — narrowing the claim must "
+ "not silently delete the line: " + lines);
assertTrue(coverage.contains("fleetd's OWN environment"),
"the line must say whose environment it counted: " + coverage);
assertTrue(coverage.contains("NOT the member pane's"),
"and must say plainly that it is not the member's: " + coverage);
// The counts themselves stay real — narrowing the claim must not turn them into constants.
assertTrue(coverage.startsWith("member credentials: allowed 1 of 3"),
"the real counts must survive the rewording: " + coverage);
}
/**
* Lead-review fix: on a NON-zsh shell no scrub ever runs (bash ignores {@code ZDOTDIR}), so the
* "allowed N of M" line — which describes what the scrub does — must not be printed there either.
@@ -387,6 +438,33 @@ class HerdrPeerLauncherAllowListWiringTest {
+ appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList());
}
/**
* fleetd #184 item 5: the unknown-environment WARN must state the honest reason for the
* UNKNOWN conclusion — fleetd has no channel to confirm what OS user the second herdr runs
* as — and must NOT assert as fact that member panes run under a different OS user just
* because {@code memberHerdrSocket} is configured. An operator may point it at a second herdr
* running as the SAME user, for pane isolation alone; in that case members DO inherit fleetd's
* environment, and asserting otherwise would tell the operator to disregard a real, known gap.
*/
@Test
void unknownEnvironmentWarnStatesUncertaintyNotAnAssertedDifferentUser() {
FakeHerdr herdr = new FakeHerdr();
Set<String> hostEnvNames = Set.of("FLEETD_WORKER_TOKEN", "SOME_UNKNOWN_SECRET_TOKEN");
WiringLauncher launcher = new WiringLauncher(herdr, allowList(), "/bin/zsh", () -> hostEnvNames,
() -> configWithMemberHerdrSocket("/tmp/other-user-herdr.sock", "/bin/zsh"));
List<String> messages = spawnAndCaptureLogs(launcher);
assertTrue(messages.stream().anyMatch(m -> m.contains("memberHerdrSocket")
&& m.contains("no channel to confirm what OS user that herdr runs as")),
"expected the WARN to name the actual uncertainty (no channel to confirm the "
+ "herdr's uid), got: " + messages);
assertFalse(messages.stream().anyMatch(m -> m.contains("member panes run under a different OS "
+ "user than fleetd's own process")),
"the WARN must not assert as fact that members run under a different OS user just "
+ "because memberHerdrSocket is configured — got: " + messages);
}
/**
* Hard constraint: the gap detector must never log an env var VALUE, only its NAME. {@code
* SOME_UNKNOWN_SECRET_TOKEN} resolves to a distinctive canary value through the same {@code env}
@@ -109,11 +109,11 @@ class MemberEnvAllowListTest {
/**
* {@code SSH_AUTH_SOCK} is a live handle to the operator's ssh-agent, never a value — so it must
* stay excluded from the derived set even when the operator lists it under {@code allow:} for an
* unrelated reason. It is governed ONLY by {@code memberCredentials.sshAuthSock}, applied
* unrelated reason. It is governed ONLY by {@code memberCredentials.sshAgentEnv}, applied
* separately by the caller ({@code HerdrPeerLauncher}).
*/
@Test
void sshAuthSockInMemberCredentialsAllowIsStillExcluded() {
void sshAgentEnvInMemberCredentialsAllowIsStillExcluded() {
Set<String> derived = MemberEnvAllowList.derive(List.of(), Set.of("SSH_AUTH_SOCK", "OTHER_NAME"));
assertFalse(derived.contains("SSH_AUTH_SOCK"),
@@ -1440,4 +1440,131 @@ class OpenCodeLauncherTest {
"the profile hint must survive the Fleetd-style forwarding hop and reach the real "
+ "sink — a lambda forwarder drops it and this must go red");
}
// --- fleetd #267: the #175 check never ran for the ordinary (no-worktree) spawn shape --------
/**
* fleetd #267 acceptance criterion 2, half 1 — a regression guard for the NEW code path only:
* a spawn WITH a fleetd-provisioned worktree must keep running the fleetd #175 model check
* exactly as before (already proven thoroughly above), and must now ALSO never emit the new
* fleetd #267 "cannot run" WARN, since the check is not skipped in this shape. Driven through
* the real {@code SessionManager.acquire()}/{@code get()} late-resolve path (fleetd #209),
* the same path the existing #175 tests already exercise.
*/
@Test
void aProvisionedWorktreeSpawnRunsTheModelCheckThroughSessionManagerAndNeverLogsTheSkipWarn(
@TempDir Path configRoot, @TempDir Path discRoot) throws Exception {
String workDir = provisionedWorkDir(configRoot);
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = opencodeCfg("opencode/nemotron-3-ultra-free", null, null);
List<String> exhausted = new ArrayList<>();
ExhaustionSink sink = (target, reason, profile) -> exhausted.add(reason);
OpenCodeLauncher launcher = serviceWithSink(herdr, configRoot, discRoot, cfg, sink);
SessionManager sessions = new SessionManager(launcher);
Logger logger = (Logger) LoggerFactory.getLogger(OpenCodeLauncher.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
try {
MemberSession acquired = sessions.acquire(cfg.profile(), workDir, null, null);
assertNull(acquired.agentSessionId(), "no opencode row yet");
OpenCodeSessionDiscoveryTest.writeRecord(discRoot, "ses_x", workDir, 1000L,
"{\"id\":\"gpt-5.6-sol\",\"providerID\":\"openai\"}");
Optional<MemberSession> after = sessions.get(acquired.paneId());
assertEquals("ses_x", after.get().agentSessionId());
} finally {
logger.detachAppender(appender);
}
assertEquals(1, exhausted.size(),
"the mismatch check still runs on the real path with a provisioned worktree: " + exhausted);
boolean cannotRunWarn = appender.list.stream()
.filter(e -> e.getLevel() == Level.WARN)
.anyMatch(e -> e.getFormattedMessage().contains("cannot run"));
assertFalse(cannotRunWarn, "a provisioned-worktree spawn must never log the fleetd #267 "
+ "'cannot run' WARN — the check ran, it was not skipped: " + appender.list);
}
/**
* fleetd #267's central defect, reproduced and fixed: {@code OpenCodeLauncher.SessionAwareHandle
* .agentSessionId()} is the ONLY caller of {@code checkModelMatch}, and it sits behind the
* fleetd #249 worktree gate — so a plain {@code fleet_spawn} with no {@code worktree:true}
* (the ticket's "ordinary, expected shape of the large majority of spawns") never reached
* {@code checkModelMatch} at all. A test that called {@code checkModelMatch} directly, or built
* a {@link OpenCodeLauncher.SessionAwareHandle}/{@link PeerHandle} in isolation, would have
* passed on every single day this gap existed — it never drives {@code agentSessionId()}
* through the worktree gate the way production does. This test instead drives the REAL
* late-resolve path: {@code SessionManager.acquire()} (which calls {@code handle
* .agentSessionId()} to build the very first {@code MemberSession}) and a re-poll via {@code
* SessionManager.get()} (fleetd #209's retained-handle mechanism) — the exact sequence a live
* pane goes through.
*
* <p>The fix chosen (see {@code OpenCodeLauncher}'s javadoc on the {@code !worktreeProvisioned}
* branch) is the WARN path, not a decoupled check: {@code actualModelForSessionId} can only be
* keyed safely by a RESOLVED session id (fleetd #234's fix for exactly this false-positive
* risk), and without a provisioned worktree no id can ever be safely resolved (fleetd #249) —
* re-deriving "whatever is newest in this shared directory" here would silently reintroduce the
* false-positive risk #234 fixed. This test proves both halves: a plausible-looking mismatch
* row for the shared, non-provisioned cwd never quarantines anything, AND the new one-time,
* per-profile WARN replaces the old total silence.
*/
@Test
void aSpawnWithoutAProvisionedWorktreeNeverRunsTheModelCheckButWarnsOncePerProfile(
@TempDir Path configRoot, @TempDir Path discRoot) throws Exception {
// Deliberately NOT provisionedWorkDir(...) / markAsProvisionedWorktree(...): a plain
// directory with no .git marker — the exact "fleet_spawn with no worktree:" shape fleetd
// #267 is about, and the ordinary shape the ticket says most spawns actually take.
Path workDir = Files.createDirectories(configRoot.resolve("shared-cwd"));
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = opencodeCfg("opencode/nemotron-3-ultra-free", null, null);
List<String> exhausted = new ArrayList<>();
ExhaustionSink sink = (target, reason, profile) -> exhausted.add(reason);
OpenCodeLauncher launcher = serviceWithSink(herdr, configRoot, discRoot, cfg, sink);
SessionManager sessions = new SessionManager(launcher);
Logger logger = (Logger) LoggerFactory.getLogger(OpenCodeLauncher.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
try {
// The real production entrypoint: acquire() calls handle.agentSessionId() itself to
// build the very first MemberSession, BEFORE any row exists.
MemberSession acquired = sessions.acquire(cfg.profile(), workDir.toString(), null, null);
assertNull(acquired.agentSessionId(),
"still refuses to guess an identity for a shared, non-provisioned cwd (fleetd #249)");
// A row for this exact (shared) directory appears, running a model that WOULD look like
// a mismatch against cfg.model() if fleetd trusted the shared-directory heuristic —
// exactly the false-positive shape fleetd #234 fixed for the id-resolved case.
OpenCodeSessionDiscoveryTest.writeRecord(discRoot, "ses_sibling", workDir.toString(), 1000L,
"{\"id\":\"gpt-5.6-sol\",\"providerID\":\"openai\"}");
// Re-drive the SAME real late-resolve path (fleetd #209) — repeatedly, to also prove
// the new WARN fires at most once per profile, not once per poll.
Optional<MemberSession> resolved = sessions.get(acquired.paneId());
assertNull(resolved.get().agentSessionId(), "still no identity — the gate never opens");
sessions.get(acquired.paneId());
} finally {
logger.detachAppender(appender);
}
assertTrue(exhausted.isEmpty(),
"must never quarantine off a shared-directory row it cannot trust as this session's "
+ "own — fleetd #234's exact concern, now also for the model check: " + exhausted);
List<String> skipWarnings = appender.list.stream()
.filter(e -> e.getLevel() == Level.WARN)
.map(ILoggingEvent::getFormattedMessage)
.filter(m -> m.contains("cannot run"))
.toList();
assertEquals(1, skipWarnings.size(),
"exactly one 'cannot run' WARN across acquire() + two get() re-polls — the old code "
+ "logged NOTHING here, which is the bug this ticket fixes; got: " + skipWarnings);
assertTrue(skipWarnings.get(0).contains(cfg.profile()),
"the WARN must name the profile, same treatment discoveryUnavailable already gets: "
+ skipWarnings.get(0));
}
}
@@ -828,6 +828,56 @@ class MessageServiceTest {
assertEquals(MessageService.Outcome.REPLIED, answer.get(5, TimeUnit.SECONDS).outcome());
}
// --- fleetd #275: a target torn down FOR GOOD while genuinely ASKING must not orphan --------
//
// sessions.onRelease (fleet_stop, or the idle reaper) is the one abandon() caller that knows
// for certain the target can never resume: its pane is being stopped right now. Unlike the
// health-classification caller above (a GONE/NEVER_READY guess, not a teardown it performed),
// it must sweep an ASKING ticket right here — see MessageService.abandon(String, String,
// boolean)'s javadoc for the full reachability chain this closes: without this, the forward
// waiter is already closed by the time the question surfaces, the ASKING guard skips the task,
// and by the time the worker's own fleet_ask lapses (~55-115s later) the released session no
// longer appears in FleetHealthMonitor's roster for anything to ever sweep it again — leaving
// fleet_poll{ticket} stuck PENDING forever.
@Test
void abandonWithSweepAskingFailsATornDownTargetsAskingTicket() throws Exception {
String ticket = messages.sendAsync(T, "task that asks");
awaitWaiting();
injectDelivery();
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config?", 300));
MessageService.TaskView asking = awaitTicketPhase(ticket, MessageService.Phase.ASKING);
assertTrue(messages.abandon(T, "the worker session was released before it replied", true),
"a released target's open ask can never resume, so it must fail right here");
MessageService.TaskView failed = awaitTicketPhase(ticket, MessageService.Phase.FAILED);
assertEquals("the worker session was released before it replied", failed.detail());
// The reverse-rendezvous ask is torn down too: the worker's still-blocked fleet_ask rides
// out its own timeout (nothing completed its answer future), and a late answer() for the
// same turnId must see it as lapsed rather than resolving a question nobody is waiting on.
assertEquals(MessageService.AskOutcome.TIMED_OUT, ask.get(5, TimeUnit.SECONDS).outcome());
assertEquals(MessageService.Outcome.STALE_TURN,
messages.answer(asking.turnId(), "config.yaml", 200).outcome());
}
@Test
void abandonWithoutSweepAskingBehavesLikeTheTwoArgOverload() throws Exception {
String ticket = messages.sendAsync(T, "task that asks");
awaitWaiting();
injectDelivery();
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config?", 5000));
awaitTicketPhase(ticket, MessageService.Phase.ASKING);
assertFalse(messages.abandon(T, "agent target term_a not found", false),
"sweepAsking=false must match the plain abandon(target, reason) overload");
assertEquals(MessageService.Phase.ASKING, messages.poll(ticket).phase());
}
// --- #137: a fleet_ask round-trip must not orphan the ticket's own reply -------------------
//
// The primary's fleet_send{turnId} answer call is itself bounded (a real MCP call, capped well
@@ -21,6 +21,7 @@ import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.*;
@@ -362,6 +363,46 @@ class GitWorktreesTest {
assertEquals("worktree origin contains HTTPS user info; refusing provision", error.getMessage());
}
/**
* fleetd #274. {@code add()} creates the worktree and its branch, then runs several more steps
* that can throw — {@code requireCredentialFreeHttpsOrigin} among them, an intended security
* refusal, not an IO accident. Before the fix, any exception from those later steps left
* {@code add()} never returning, so its caller never learned the path and the worktree
* directory plus its branch leaked on disk forever with nothing tracking them.
*
* <p>This drives the exact same {@code afterWorktreeAdded} test seam as
* {@link #provisioningRefusesAWorktreeWhoseOriginStillHasHttpsUserInfo} — a mutation applied
* right after {@code git worktree add}, so the step that throws
* ({@code requireCredentialFreeHttpsOrigin}, reached moments later inside {@code add()} itself)
* runs strictly after the worktree and branch already exist, not downstream of {@code add()}
* in some other caller. {@code afterWorktreeAdded} also hands back the created path, so the
* assertions below don't have to guess the generated nonce.
*/
@Test
void addCleansUpTheWorktreeAndBranchWhenAPostCreationStepThrows(@TempDir Path tmp) throws Exception {
Path repo = initRepo(tmp.resolve("repo"));
git(repo, "remote", "add", "origin", "https://git.ltms.dev/akb/kb.git");
String branch = "cb-274-leak";
AtomicReference<String> createdPath = new AtomicReference<>();
GitWorktrees worktrees = new GitWorktrees(tmp.resolve("wts").toString(), worktreePath -> {
createdPath.set(worktreePath);
try {
git(Path.of(worktreePath), "remote", "set-url", "origin",
"https://synthetic-test-token@git.ltms.dev/akb/kb.git");
} catch (Exception e) {
throw new RuntimeException(e);
}
});
assertThrows(WorktreeException.class, () -> worktrees.add(repo.toString(), branch, "HEAD"));
assertNotNull(createdPath.get(), "afterWorktreeAdded must have run with the created path");
assertFalse(Files.exists(Path.of(createdPath.get())),
"the worktree directory leaked after a post-creation step threw");
String heads = forEachRef(repo, "refs/heads/" + branch);
assertTrue(heads.isBlank(), "the branch leaked after a post-creation step threw:\n" + heads);
}
// ---- CB-189: broader remote-URL coverage — every remote, both fetch and push URLs, any
// non-SSH scheme. Reporting only, additive to the origin/https strip-and-refuse tests above. ----