Compare commits

...

64 Commits

Author SHA1 Message Date
Dai Ha 2e349139e9 #568: add hunter member role
CI / shell-tests (pull_request) Successful in 6s
CI / contract (pull_request) Successful in 1m21s
CI / build (pull_request) Successful in 1m44s
2026-09-19 15:09:07 +07:00
Dai Ha a639969a9a CLAUDE.md: a blocked lead consults architects, not the operator
CI / shell-tests (push) Successful in 6s
CI / contract (push) Successful in 1m32s
CI / build (push) Successful in 1m42s
The operator set this rule on 2026-09-19: when a decision blocks a lead, it
consults one or more architect members, who are authorized to agree on one
decision and unblock. The operator is not asked. Escalation stays open only
for things outside the fleet's authority -- money, credentials, or a promise
made to someone else.

The paragraph also carries the reason the ticket record is mandatory rather
than optional. The operator's old notification channel was the block itself:
work stopped, so they found out. Taking the operator out of the loop removes
that signal with it, so the decision goes on the ticket, which reaches them
whether or not they are at a terminal when it is made.

The rule has a second half aimed at architects, which lives in
fleet.charters.architect and is applied per daemon -- filed as #591, because
a charter can never reach a lead and charters do not travel between hosts.
The missing notification event is #592.

Block verified byte-identical with wiki 7-Use-Cases.md at 1d9bd1b.
2026-09-19 14:57:04 +07:00
Dai Ha 17c3a69c57 docs: CB-591 gateway page had two claims that went stale
CI / shell-tests (push) Successful in 5s
CI / contract (push) Successful in 1m23s
CI / build (push) Failing after 1m37s
The gateway's chat model is served under the stable alias `acoder`, and the
model behind that alias changed on 2026-08-28 — it is Qwen3.8-27B now, not
DeepSeek-V4-Flash. The old name is still served, so nothing broke, but it
names a model this is not.

Two claims on the page were wrong as a result, and both were written as
current facts rather than dated measurements:

- `/v1/models` returns exactly `["deepseek-v4-flash"]` — it returns 6 ids
  now. This sat under a heading saying it needs no re-testing.
- the status banner said `local` and `gx` are both at `weight: 100` —
  `local` is at 0.

Measured today against the live gateway: /v1/models returns acoder,
qwen3.8-27b-nvfp4, deepseek-v4-flash and three embedding names; a completion
sent as `deepseek-v4-flash` comes back reporting `"model": "acoder"`, which
is the alias in plain sight. /v1/deployment reports generation
2026-08-28-qwen3.8-27b-nvfp4.

§2 and §3 are left alone. They are the August plan, and rewriting them would
destroy the record of the migration.

fleetd.yaml moved to `acoder` in the same change. It is not tracked here.
2026-09-13 07:01:15 +07:00
ltms 49a5875586 Merge #583: fleetd #582 — assert pending message-id cleanup at every publish cleanup site
CI / shell-tests (push) Successful in 9s
CI / contract (push) Successful in 48s
CI / build (push) Failing after 2m1s
All ten assertions proven live: six by the implementer, the last four by the lead.
One contract build with four deleted removal lines produced exactly four named failures,
one per site, with the total unchanged at 1825.
2026-09-12 15:52:25 +02:00
Dai Ha 634d33b50b Merge worker/562-loop-health-wiring-test-99611c-5
CI / shell-tests (push) Successful in 9s
CI / contract (push) Successful in 1m17s
CI / build (push) Successful in 1m45s
2026-09-12 20:28:14 +07:00
Dai Ha 1db79bcaa9 Merge worker/581-completionresolver-cas-sites-0542b7-6 2026-09-12 20:28:14 +07:00
Dai Ha 4507bc5a70 Merge worker/571-attempted-outcome-5739f7-2 2026-09-12 20:28:14 +07:00
Dai Ha d7239ed23b fleetd #571: pin FleetMcp.formatReply's TIMED_OUT_UNCONFIRMED wording
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 1m42s
CORRECTION 5 on the ticket: mutating the new arm's message text to the
queued/working arm's text survived every existing test, because nothing
asserted the specific wording. This adds one test that asserts the
unconfirmed-delivery message and asserts it does NOT carry the
queued/working arm's retry invitation — the distinction #571 exists for.

No production code changes; formatReply's TIMED_OUT_UNCONFIRMED arm was
already correct.
2026-09-12 20:19:37 +07:00
Dai Ha dfeb9340b4 fleetd #581: cover completion CAS removals
CI / shell-tests (pull_request) Successful in 5s
CI / build (pull_request) Failing after 1m31s
CI / contract (pull_request) Successful in 1m35s
2026-09-12 20:19:20 +07:00
Dai Ha 1513d4f260 fleetd #562 follow-up: extract loopHealthSource factory, pin its wiring
CI / shell-tests (pull_request) Successful in 6s
CI / contract (pull_request) Successful in 59s
CI / build (pull_request) Successful in 1m43s
PR #579's inline `new FleetMcp.LoopHealthSource(poller::health, ...)` in
Fleetd.main had nothing a test could call directly. Measured: replacing
poller::health with a constant () -> RUNNING compiled clean and left all
1771 tests green (see issue #562 comment "HOLD on PR #579").

Extracts the inline construction to a package-private Fleetd.loopHealthSource
factory, the same style as the sibling capacitySource/healthCoverageSource
factories, and adds FleetdLoopHealthSourceWiringTest with three separate
assertions: the statusPoller half, the sessionReaper half, and the
reaper == null branch (still STOPPED).
2026-09-12 20:16:23 +07:00
Dai Ha 4ca7d72303 #582: assert pending message-id cleanup
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 50s
CI / build (pull_request) Successful in 2m32s
2026-09-12 20:15:48 +07:00
Dai Ha c0545d003d fleetd #571: make FleetApp.writeReply's inner Outcome switch exhaustive, no default
CI / shell-tests (pull_request) Successful in 8s
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 1m48s
Ticket comments (17126, 17127) corrected the original acceptance criterion after this
unit was already in flight: a hand-listed grep for the enum's constant names goes stale
silently the moment a new constant lands, so the compiler must be the enumeration instead.
sendOutcomeLabel (MessageService.java) and formatReply (FleetMcp.java) were already
default-free switch expressions. The one gap was writeReply's inner "status" switch, which
had `default -> "done"` — the exact value that would have lied about TIMED_OUT_UNCONFIRMED.
Remove the default and list every Outcome constant explicitly; REPLIED, COMPLETED_UNREPLIED,
QUESTION and STALE_TURN get an arm too even though the outer switch always dispatches them
first, so the inner switch stays exhaustive on its own. The outer switch (a statement, not
an expression) keeps its own default — Java does not require exhaustiveness there regardless,
and "everything not terminal is a 202" is an intentional catch-all.

Verified with the proof the ticket asked for: added a scratch 11th Outcome constant after
deleting all default arms and confirmed all three switch-expression sites (and no test file)
fail to compile without an arm for it, one at a time, then removed the scratch constant.
2026-09-12 19:28:59 +07:00
Dai Ha 275ac0d251 fleetd #562: surface loop health
CI / shell-tests (pull_request) Successful in 12s
CI / contract (pull_request) Successful in 1m31s
CI / build (pull_request) Successful in 2m11s
2026-09-12 19:26:37 +07:00
Dai Ha c1e06c9e12 fleetd #571: add TIMED_OUT_UNCONFIRMED so an ATTEMPTED delivery is not reported as never-arriving
MessageService.send's TimeoutException branch collapsed Injector.Cancellation.ATTEMPTED
(fleetd #551 — the send call was made but its outcome is unknown) into
Outcome.TIMED_OUT_QUEUED, which promises the caller the message will never arrive. On this
route agent.prompt may already have pasted and submitted the text, so a caller's natural
recovery (resend) risks a double delivery.

Add Outcome.TIMED_OUT_UNCONFIRMED and route ATTEMPTED to it. Update the three readers found
by searching for the enum's constant names (not `Outcome.`, which misses FleetMcp's
unqualified `case REPLIED ->` switches and would false-positive on ConfigRef's unrelated
Outcome record):
 - MessageService.sendOutcomeLabel: add it to the "timeout" metric label group.
 - FleetMcp.formatReply: its own case, warning against a blind retry (distinct from the
   generic "retry or poll status" message the other timeouts get).
 - FleetApp.writeReply: its own "unconfirmed" status and detail text, so it no longer falls
   through the switch's default -> "done" arm, which would have reported "the delegation
   completed" for the one case where delivery is unconfirmed.
2026-09-12 19:21:10 +07:00
Dai Ha 204da67d66 Merge #576: fleetd #575 — one finally covers answer()'s STALE_TURN exit
CI / shell-tests (push) Successful in 16s
CI / build (push) Successful in 1m42s
CI / contract (push) Successful in 1m57s
Third instance of the #572 shape on this file: one invariant kept at N sites,
asserted at fewer than N. Here answer()'s inner try opened AFTER the Task
lookup/registration and the STALE_TURN early return, so that return was covered
only by a hand-rolled copy of the finally's cleanup pair. The fix widens the try
upward and deletes the copy, so every exit runs the one finally exactly once.
rendezvous.open(workerSession) correctly stays outside it — nothing to clean if
it never opened.

This fixed no live leak, and the code comment says so: neither
rendezvous.answerAsk nor clearAsyncQuestion(turnId, false) can throw, so nothing
ever left through the old gap uncovered. It is a structure fix, the ticket's own
fallback case.

Verified here, not taken from the worker's report:
- baseline on the branch: 1766 tests, 0 failures, from Maven and from an
  independent sum over target/surefire-reports/*.txt.
- my own mutation, located fresh: move the inner `try {` back down below the
  STALE_TURN return — the exact pre-fix structure, minus the hand-rolled pair.
  Result: 1766 run, exactly 1 failure, and it is the new test —
  MessageServiceTest.answerLosingTheRaceToAnAlreadyAnsweredAskStillReturnsStaleTurnAndCleansUpOnce:545
  "the forward waiter this answer() call opened must be closed after a STALE_TURN
  return". One failure, not a crowd: the new test is the only thing holding this
  path.
- the worker's own mutation removed the whole finally and took 8 other tests with
  it. That proves the finally runs; it does not prove the STALE_TURN path reaches
  it. Mine does.

The new test hook answerAskLapseRaceHookForTest follows the file's existing
askTimeoutRaceHookForTest convention.

Closes #575.
2026-09-12 19:05:00 +07:00
Dai Ha b091c51eee fleetd #575: widen answer()'s try so one finally covers its STALE_TURN exit
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 58s
CI / build (pull_request) Successful in 2m30s
The waiter cleanup pair (asyncTasksByWaiter.remove + rendezvous.close) was
duplicated: two sites sit in a finally, the third was hand-rolled inline
before answer()'s early STALE_TURN return, structurally outside any finally.

Both rendezvous.answerAsk and clearAsyncQuestion(turnId, false) are total
(cannot throw), so the gap never leaked in practice. But the duplicate was
untested: mutating it away left all 1765 tests green, while the two
finally-protected sites are each killed by 8-36 tests. Same shape as #572.

Fix: widen the try to wrap the Task registration and the STALE_TURN check,
so the single finally covers every exit and the hand-rolled copy is gone.
Added a race hook + regression test that deterministically reproduces the
'ask lapsed between the lookup and the unblock' case and proves the fix
still returns STALE_TURN and cleans up exactly once.
2026-09-12 18:56:02 +07:00
ltms 84d631b030 Merge #572: answer()'s session-lock release is pinned on all four exits
CI / shell-tests (push) Successful in 9s
CI / contract (push) Successful in 1m27s
CI / build (push) Successful in 1m44s
fleetd #572. MessageService releases its per-session lock in a finally at two sites. Removing the
send() one failed 22 tests and errored 1. Removing the answer() one left the whole suite green.
Skip that unlock and the thread holds the session lock forever, so every later send or answer to
that session blocks permanently — no exception, no log line.

Tests only. Against current main the diff is one file, MessageServiceTest.java, +178 lines.

Four new tests, one per exit of answer(): normal REPLIED, TIMED_OUT_WORKING, the ExecutionException
rethrow, and the InterruptedException rethrow. Each proves REACQUISITION rather than the return
value — a bounded follow-up send on the SAME session must not come back BUSY, and send reports BUSY
only when tryLock itself timed out, so a non-BUSY probe is specifically evidence the lock was free.

Lead verification, re-running rather than accepting the worker's numbers, on the branch merged with
main at 3f8c38f:
  baseline   -> exit 0, 1765 tests from Maven and an independent sum over 131 reports (1761 + 4)
  mutation L -> BUILD FAILURE, exactly 4 failures, all four new tests by name

THE TRAP THE WORKER CAUGHT ITSELF, which is the most valuable thing in this PR. Their first draft of
the TIMED_OUT_WORKING test called answer() inline on the test thread and then probed with send() on
that same thread. It FALSE-PASSED under the mutation: lock is a ReentrantLock, so the same thread
reenters for free whether or not unlock() ran. A same-thread probe proves reentrancy, not release.
They found it only because they actually ran the mutation instead of trusting a green baseline, moved
answer() onto a background thread, and re-proved the kill. The pitfall is now documented in the test's
own javadoc, with the measurement that exposed it.

NOTE FOR ANYONE RE-RUNNING THE TICKET'S RECIPE: the ticket records `sed '1218s|...'`, measured before
#551 merged. #551 edited MessageService.java, and the line is now 1231. Locate it fresh — a
line-anchored sed against a stale number mutates the wrong line and reports a meaningless green. The
anchor count is the check that catches this: `grep -Fxc '            lock.unlock();'` must read 2
pristine and 1 after.

Shape reported by the worker, not investigated and not fixed: the same two-line cleanup
`asyncTasksByWaiter.remove(reply); rendezvous.close(...)` appears at three sites — send()'s finally,
answer()'s inner finally, and answer()'s inline STALE_TURN early return, which is NOT inside a
finally. Same maintained-at-N-sites shape as this finding. Filed separately.
2026-09-12 13:30:22 +02:00
ltms 3f8c38fc54 Merge #567: pin LeadMailbox.inspect's probe-channel close
CI / shell-tests (push) Successful in 8s
CI / contract (push) Successful in 1m1s
CI / build (push) Successful in 1m49s
fleetd #567. LeadMailbox.inspect opens a probe channel and closes it in a finally. The production
code was already correct; nothing asserted it, so a future refactor could drop the close and leak an
AMQP channel per inspect() call with the suite green.

Test only. LeadMailbox.java is untouched — sha256 a2cd99be77345b7e... before and after.

The test asserts the CONSEQUENCE rather than the return value: it caps the connection at three
channels (LeadMailbox uses two, consume and publish), runs a successful inspect, then requires a
replacement channel. If the probe is left open, the broker has no channel number left and
createChannel() returns null. A test that only checked inspect()'s MailboxState would pass under the
mutation, which is the whole reason this hole existed.

Lead verification, re-running rather than accepting the worker's numbers, on the branch merged with
main at ed2fd66:
  mvn -o clean install -Pcontract  ->  exit 0, 1792 tests from Maven and from an independent sum
                                       over 137 surefire reports.
  1792 = 1761 (main) + 30 (contract-only) + 1 (new), which also confirms the default-profile count
  is untouched: the new test is in an @Tag("contract") class.

My own mutation, located fresh rather than assuming the reported line number: delete
LeadMailbox.java:338 `probe.close();`, anchor by grep -Fxc 1 -> 0. Result under -Pcontract: exactly
1 failure, inspectClosesItsSuccessfulProbeChannel:226, "the replacement channel was null". Restored
to a2cd99be77345b7e..., git status --short empty.

THE PROFILE TRAP THIS TICKET EXISTS BECAUSE OF. An earlier sweep worker instrumented this line, saw
zero hits under the DEFAULT profile, and concluded "no test executes this line". That was false:
fleetd/pom.xml:264 sets excludedGroups=contract, so the covering tests were excluded from the run,
not absent. A surviving mutation has THREE causes — never executes, executes with nothing asserted,
or the covering tests were excluded from the profile — and only naming the profile tells them apart.
The true finding was covered-but-unasserted.

Known fragility, recorded rather than fixed: the test's channel cap of three assumes LeadMailbox
holds exactly two channels. If it ever holds more, this test fails loudly, which is fine. If it ever
holds fewer, a leak would no longer exhaust the cap and the test would go vacuous silently. Worth
re-checking if LeadMailbox's channel usage changes.

Not covered, and stated rather than faked: the defensive catch (RuntimeException) around the passive
declare, which needs a connection dying between createChannel() and the declare landing. The
method's own javadoc already admits that branch is unproven; the worker did not invent a test for it.
2026-09-12 13:26:30 +02:00
Dai Ha a4dbc8f8b7 fleetd #572: pin answer()'s session-lock release across all four exits
CI / shell-tests (pull_request) Successful in 7s
CI / contract (pull_request) Successful in 47s
CI / build (pull_request) Successful in 2m10s
MessageService.answer() releases its per-session lock in an outer finally
(MessageService.java:1218) that mutation testing showed was covered but
unasserted: removing that line left all 1750 existing tests green, because
every existing test on this path checks answer()'s return value, never that
the lock it took is actually reacquirable afterward. If it leaked, a session
would be wedged forever with no exception and no log line.

Adds four tests, one per exit of answer() (normal REPLIED reply,
TIMED_OUT_WORKING, ExecutionException rethrow, InterruptedException
rethrow), each proving the lock is reacquirable via a bounded (300ms)
follow-up send on the same session rather than merely checking answer()'s
own outcome. No production change.
2026-09-12 18:23:54 +07:00
ltms ed2fd6646a Merge #561: the completion/session listener fan-out survives either half throwing, and both sites are pinned
CI / shell-tests (push) Successful in 6s
CI / contract (push) Successful in 50s
CI / build (push) Successful in 2m7s
fleetd #561. Fleetd composed two TurnListener halves as `completion.X(); sessions.X();`, so a throw
from the first half skipped the second. The anonymous class is now a package-private factory,
Fleetd.turnListener(completion, sessions), built on two helpers that always attempt both halves and
rethrow whatever escaped — a second failure attached with addSuppressed rather than dropped, so it
still reaches StatusPoller's catch (Throwable).

The wiring at Fleetd.java:498 calls that factory, so the seam under test is the real caller.

Lead verification, on a merged tree, re-running the checks rather than accepting the worker's:
exit 0, 1761 tests from Maven and from an independent sum over 131 surefire reports.

The interesting part is what the first round MISSED. Two helpers maintain one invariant — "the
second half always runs" — and the first round's five tests asserted it at only one site. Measured:

  bothMustRun                      reverted to the pre-fix bug -> 1 named failure   (pinned)
  bothMustRunKeepingSecondResult   the SAME bug                -> 1755/1755 GREEN   (unpinned)
  failure.addSuppressed(t) deleted                             -> 1755/1755 GREEN   (unpinned)

Both survivors are now killed by new tests, re-verified by the lead after the fix:
sessionHalfStillRunsWhenTheCompletionHalfThrowsSynchronouslyForPostAction and
bothFailuresEscapeWhenBothHalvesThrowDistinctExceptions, each failing alone under its own mutation.

The rule this cost us, worth carrying: COUNT ASSERTIONS PER SITE, NOT PER INVARIANT. The total being
non-zero is what hides a zero at one site, and extracting a shared helper makes it worse rather than
better — it does not reduce the number of sites, only how many are visible. Credit to the fleet01
lead, who predicted this shape before an instance was found.

onDelivered stays deliberately unguarded. Its comment now gives the real reason —
CompletionResolver.captureBaseline already catches RuntimeException around its scrape and fails open,
so that half does not realistically throw — instead of the previous reason, which was true but about
registration rather than about this pair. A correct conclusion resting on a wrong premise reads
exactly like a verified one.
2026-09-12 13:20:56 +02:00
Dai Ha 5441a2b321 fleetd #567: assert inspect closes probe channel
CI / shell-tests (pull_request) Successful in 9s
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 2m23s
2026-09-12 18:19:43 +07:00
ltms 384867dfa3 Merge #551: ATTEMPTED is its own cancellation answer, and the javadoc stops claiming a timed-out send never arrived
CI / shell-tests (push) Successful in 6s
CI / contract (push) Successful in 1m11s
CI / build (push) Successful in 1m44s
fleetd #551. The injector polls a queued entry off the queue and marks it ATTEMPTED BEFORE the
irreversible AgentControl#send call, not after. So a Throwable escaping that call can never leave
the entry QUEUED at the head (the #546 re-send hazard) and can never be recorded as a confident
NOT_DELIVERED for text that may already be in the pane.

Cancellation.ATTEMPTED is added as a third answer. NOT_DELIVERED stays reserved for confirmed
absence: the readiness grace expiring, drop(), or a herdr *_not_found error, which the rest of this
codebase already reads as definitely-absent rather than inconclusive.

Also fixes three javadoc/comment sites that claimed a timed-out send definitely did not arrive:
TIMED_OUT_QUEUED, hasQueuedDelivery, the queuedDeliveries field, and the comment in send()'s timeout
branch. Two claims were wrong, not merely stale: "on every route it will not arrive later" is false
on the ATTEMPTED route, and "may already hold a partial paste" understates it — agent.prompt pastes
AND SUBMITS in one call, so the target may hold a complete, running turn.

Verified by the lead on a merged tree: mvn -o clean install from fleetd/, exit 0, 1754 tests from
Maven and from an independent sum over 130 surefire reports. Mutation: folding ATTEMPTED back into
NOT_DELIVERED in cancellationOf gives 3 red, each naming the property
(anErrorFromSendRemovesTheMessageAndMarksItAttempted:740,
aHerdrExceptionFromSendStillSurfacesButNowReportsAttempted:781,
aHerdrExceptionAfterThePasteIsNeverRecordedAsConfidentlyNotDelivered:806).

The final round is comment-only, proven mechanically rather than by reading: stripping every comment
from MessageService.java before and after and collapsing whitespace gives byte-identical code.
2026-09-12 13:19:20 +02:00
Dai Ha f40c19ecf0 fleetd #551 shape sweep: fix stale ATTEMPTED-route javadoc/comments in MessageService
CI / shell-tests (pull_request) Successful in 6s
CI / contract (pull_request) Successful in 50s
CI / build (pull_request) Successful in 2m26s
hasQueuedDelivery's javadoc, the queuedDeliveries field javadoc, and a code
comment in send()'s timeout path all still made two claims that ATTEMPTED
(fleetd #551) falsifies: a blanket "the message will not arrive later" across
every route, and "the terminal may already hold a partial paste" — but
agent.prompt pastes AND submits in one call, so the target may hold a
complete, already-submitted turn. Each now names the three Injector.Cancellation
routes (CANCELLED, NOT_DELIVERED, ATTEMPTED) and says plainly that only the
first two establish the message will not arrive later.

Comment/javadoc only. No behaviour change: Injector.java is untouched
(sha256 0c689b6cf36275c0da45497a74b2bd4f5a3d80c4dbda77d46670c66004e53b69) and
no test was added. mvn -o clean install: BUILD SUCCESS, Tests run: 1754,
Failures: 0, Errors: 0 (unchanged from before this commit).
2026-09-12 18:13:06 +07:00
Dai Ha 034e17bb32 fleetd #561 follow-up: pin the session half of bothMustRunKeepingSecondResult
CI / shell-tests (pull_request) Successful in 7s
CI / contract (pull_request) Successful in 1m14s
CI / build (pull_request) Successful in 1m41s
Two helpers maintain one invariant (the second callback half always runs,
even when the first throws): bothMustRun and bothMustRunKeepingSecondResult.
Only bothMustRun's "session half still runs" direction was asserted
(sessionHalfStillRunsWhenTheCompletionHalfThrowsSynchronously, via
onTurnComplete). bothMustRunKeepingSecondResult — the helper
onTurnCompleteWithPostAction uses — could be reverted to the pre-#561
broken shape and the suite stayed green.

Adds two tests to FleetdTurnListenerCompositionTest:
- sessionHalfStillRunsWhenTheCompletionHalfThrowsSynchronouslyForPostAction:
  mirrors the existing onTurnComplete case for onTurnCompleteWithPostAction/
  bothMustRunKeepingSecondResult.
- bothFailuresEscapeWhenBothHalvesThrowDistinctExceptions: proves a second,
  distinct failure from the session half is preserved via addSuppressed
  rather than silently dropped when both halves of bothMustRun throw.

Also rewords the onDelivered comment in Fleetd.turnListener: it previously
said this pair is safe because registration survives a throw via #556's
Injector wiring, which is true but is not why THIS pair is unguarded.
CompletionResolver.captureBaseline already catches RuntimeException around
its scrape read and fails open, so completion.onDelivered does not
realistically throw. Comment text only, no logic change.
2026-09-12 18:10:31 +07:00
Dai Ha 68b428c484 fleetd #551 rework (comment 17058): fix TIMED_OUT_QUEUED javadoc for ATTEMPTED
CI / shell-tests (pull_request) Successful in 13s
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 2m21s
Javadoc-only change. #551 added Injector.Cancellation.ATTEMPTED, which
MessageService.send's timeout path folds into Outcome.TIMED_OUT_QUEUED
alongside CANCELLED and NOT_DELIVERED (the collapse itself is unchanged
behaviour and is being tracked as a separate follow-up ticket).

TIMED_OUT_QUEUED's javadoc — landed by #513 to state the routes that
reach it — named only two routes and said "on every route it will not
arrive later", with a "may already hold a partial paste" caveat. Both
claims are now stale: ATTEMPTED is a third route, and because
agent.prompt pastes AND submits in one call, that route may mean the
target holds a complete, already-submitted turn and is working on it
right now.

Names all three routes, says which one is uncertain, and drops the
now-false blanket claim. No behaviour change.
2026-09-12 17:51:26 +07:00
Dai Ha e20ccab1eb fleetd #561: harden the completion/session TurnListener fan-out
CI / shell-tests (pull_request) Successful in 8s
CI / contract (pull_request) Successful in 1m4s
CI / build (pull_request) Successful in 1m56s
Fleetd's turnListener composition had four callbacks (onTurnComplete,
onTurnCompleteWithPostAction, and both onTurnFailed overloads) built from two
bare, unguarded statements each. onDelivered's registration was already fixed
structurally by #556; these four had the identical fragility and were still
untested: nothing enforced that the completion resolver's half ran before the
session half beyond call order in the source, so a future reorder (or a
throwing session listener sequenced first) could silently skip the
completion resolver's effect and strand a caller for its full timeout.

Extracted the composition to a package-private static factory,
Fleetd.turnListener(completion, sessions), and hardened it with
bothMustRun/bothMustRunKeepingSecondResult: both callback halves are always
attempted regardless of whether the other throws, and whatever escapes is
rethrown afterward (never swallowed) so it still reaches StatusPoller's
catch (Throwable) and logs at ERROR.

FleetdTurnListenerCompositionTest builds this real composition from a real
CompletionResolver and a throwing fake sessions half, and asserts the
completion resolver's effect (the waiter resolving) survives the session
half throwing, for all four callbacks, plus a mirror case showing the
session half still runs when the completion half throws first.

onTurnCompleteWithPostAction keeps completion-before-session as a functional
requirement (resolveBeforePostAction must run before the context-reset
housekeeping can erase the pane), not just fault tolerance, so it is not
reorder-symmetric like the other three — documented in Fleetd.turnListener's
javadoc.
2026-09-12 17:47:53 +07:00
Dai Ha d83821bbce fleetd #551: record the delivery attempt before the irreversible send
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 57s
CI / build (pull_request) Successful in 2m23s
The Injector wrote the delivery outcome AFTER calling AgentControl.send(),
so a HerdrException thrown from the response half of that call (herdr
already replied, or may have) was recorded as a confident NOT_DELIVERED
for text that may already be sitting in the worker's pane.

Poll the queue entry and mark it Pending.State.ATTEMPTED before send() is
called, not after. On success it is upgraded to DELIVERED; on an ordinary
failure it stays ATTEMPTED (honest uncertainty), except a herdr
*_not_found error, which the rest of this codebase already treats as a
confirmed absence and which now still writes NOT_DELIVERED.

cancellationOf gets a matching third answer (Cancellation.ATTEMPTED)
instead of folding the new state into NOT_DELIVERED, so a caller that
cancels an already-attempted delivery is told the truth too.
2026-09-12 17:43:21 +07:00
ltms ba2f4d16f8 Merge #555: the redeploy main flow is lifted into tested predicates, and the guard now catches functions below the SOURCED line
CI / shell-tests (push) Successful in 9s
CI / contract (push) Successful in 1m0s
CI / build (push) Successful in 1m52s
fleetd #555. Eight main-flow decisions in scripts/redeploy-fleetd.sh move into
predicate and dispatch functions the suite can source and test. The guard test
test_no_untested_main_flow_conditionals stops new bare conditionals reappearing.

The rework closes a hole the lead found (comment 17012): a conditional wrapped in
a function defined BELOW the SOURCED guard was invisible to the guard, and such a
function can never be sourced, so it can never be tested. The guard now fails on
any function definition after the guard line, on its own.

Verified by the lead on a tree with main merged in, redeploy-fleetd.sh at its
pristine sha 4ffacc5185807d39720a3484d85b922413806eb5347318265bd8897dfd61e8d9:
  bash 5.3.9      -> exit 0, 0 lines matching ^FAIL:
  /bin/bash 3.2.57 -> exit 0, 0 lines matching ^FAIL:

Three mutations, each restored to the pristine sha afterwards:
  a bare conditional appended to the main flow      -> exit 1, reported by line
  the same conditional wrapped in a function below
    the SOURCED guard (the found hole)              -> exit 1, "function defined
    after the SOURCED guard (line 1038) - it cannot be sourced, so it cannot be tested"
  the new FUNC emission deleted from the guard      -> that same case returns to
    exit 0, so the new assertion is what catches it. Anchor count 1 -> 0,
    test-redeploy-fleetd.sh restored to sha 0d713a3092a0c0ea8c05663ffa0cb1595e21b9d73872e662eb99b5035fd38151.
2026-09-12 12:19:47 +02:00
ltms db4c98ac60 Merge #556: the Injector owns turn registration, and the #553 backstop is pinned too
CI / shell-tests (push) Successful in 6s
CI / contract (push) Successful in 51s
CI / build (push) Successful in 2m12s
fleetd #556. Registration moves off the TurnListener fan-out onto its own narrow
TurnRegistrar seam, wired directly to CompletionResolver::register in Fleetd.java,
so it survives any listener throwing regardless of call order.

Verified by the lead on a tree with main (f4f5f31) merged in:
  mvn -o clean install exit 0; 1750 tests, agreed by Maven's own summary and an
  independent sum over 130 surefire report files.

Both registrar.register call sites are independently pinned:
  Injector.java:703 (the #553 finally backstop) removed -> exactly 1 failure, the
    new aRuntimeExceptionFromOnTurnCompleteStillLeavesTheNextDeliveryRegisteredOnTheRecoveryPath
  Injector.java:660 (the ordinary path) removed -> exactly 2 failures, the two
    original tests; the backstop test stays green
Anchor count 1 -> 0 on each, sha restored to
8fcb698afccc254b0c99d3a4bf9e960c0e7c85e542024e870c1e815dce6d62a9 both times.

The :703 line survived the full suite before this rework while being executed
twice — covered but unasserted. That gap is what the added test closes.
2026-09-12 12:18:15 +02:00
Dai Ha 8f80d267a0 fleetd #555 rework: catch function definitions after the SOURCED guard
CI / shell-tests (pull_request) Successful in 9s
CI / contract (pull_request) Successful in 1m20s
CI / build (pull_request) Successful in 3m3s
Comment 17012 on #555 found a hole in test_no_untested_main_flow_conditionals:
the guard's function-body detection treats anything inside a function as
"fine, out of scope for this scan" — but a function DEFINED after the
SOURCED guard line can never be reached by sourcing this script (sourcing
stops before the main flow runs), so its body is untestable by construction
while still reading to the guard as safely inside a function.

mainflow_bare_conditionals now also emits a FUNC record for every function
opened after the guard line (reusing the same open-brace detection already
used for depth tracking), and test_no_untested_main_flow_conditionals treats
any such record as a violation on its own, independent of what the function's
body contains or whether the allowlist would otherwise excuse a bare
conditional inside it.

Proof (redeploy-fleetd.sh restored to 4ffacc5185807d39720a3484d85b922413806eb5347318265bd8897dfd61e8d9
after each):

- CONTROL — a bare conditional appended to the main flow is still caught:
  EXIT=1, "found 1 untested main-flow if/elif/case line(s) ... line 1341:
  if [ "$MY_CONTROL_BARE" = 1 ]; then :; fi"
- CANDIDATE — the same conditional wrapped in a function defined after the
  boundary, previously invisible (EXIT=0), is now caught: EXIT=1, "line 1341:
  function defined after the SOURCED guard (line 1038) — it cannot be
  sourced, so it cannot be tested: newfunc_below_the_boundary() {"

Full suite re-run green on both bash 5.3.9 and /bin/bash 3.2.57 (macOS
system bash): exit 0, 0 FAIL lines, reached the final PASS line, on both.

No change to redeploy-fleetd.sh; the 8 lifted decisions, their mutation
proofs, and the allowlist all stand as before.
2026-09-12 17:11:11 +07:00
Dai Ha 738d34a609 fleetd #556 rework: pin registration on the #553 finally backstop path
CI / shell-tests (pull_request) Successful in 7s
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 2m13s
Comment 17009: there are TWO registrar.register(target, sent.token())
call sites in Injector's delivery method — the ordinary path inside
`if (sent != null)`, and the fleetd #553 finally backstop, reached only
when an earlier block throws before the ordinary path ever runs. The
lead's mutation on Injector.java:703 (the backstop call) survived the
full suite: the existing #553 regression test for this exact scenario
(aRuntimeExceptionFromOnTurnCompleteStillCompletesTheNextDelivery)
asserts only that the delivered future completes, never that the turn
is registered with CompletionResolver — so a redesign that dropped
registration from the backstop would reopen this ticket's own defect on
precisely the path #553 exists for, with every existing test green.

Adds aRuntimeExceptionFromOnTurnCompleteStillLeavesTheNextDeliveryRegisteredOnTheRecoveryPath:
drives the same construction as the existing #553 test (onTurnComplete
throws for a previous turn, forcing the next turn's delivery down the
finally backstop) and additionally asserts the new turn is registered
with CompletionResolver and carries the correct waiter — the same
assertion the ordinary-path test makes, now made on the recovery path.

Proven by mutation: removing Injector.java:703 alone (exact-line anchor
1 -> 0) turns the new test red with its own assertion message; restored
and confirmed byte-identical (sha256 8fcb698afccc254b0c99d3a4bf9e960c0e7c85e542024e870c1e815dce6d62a9,
matching the pre-mutation tree); re-run green as a control. Full suite
after restore: 1750 tests, 0 failures, 0 errors, 0 skipped (Maven's own
summary and an independent sum over surefire-reports/*.txt agree),
BUILD SUCCESS.
2026-09-12 17:08:44 +07:00
Dai Ha a46e4058ac fleetd #556: make turn registration structural, independent of any TurnListener
CI / shell-tests (pull_request) Successful in 4s
CI / contract (pull_request) Successful in 50s
CI / build (pull_request) Successful in 2m29s
The Injector owns the invariant "every delivered turn has a registered
waiter," but before this the only thing that satisfied it was
CompletionResolver.captureBaseline, called from inside a TurnListener
callback wired in Fleetd.java. Any TurnListener that throws (from
onDelivered or elsewhere) could break the invariant with no way for the
Injector to detect it. #553 only made the one reachable listener behave
via a try/finally backstop; it did not remove this structural dependency.

Add a narrow TurnRegistrar functional interface, decoupled from
TurnListener, whose only job is registering a delivered turn's waiter.
CompletionResolver now implements it via a new register() method
(extracted from captureBaseline's registration half; captureBaseline
keeps its own full body unchanged, so existing direct callers/tests are
untouched). Injector gets an explicit registrar field/constructor family
(auto-derived from the TurnListener via instanceof where that still
works, explicit where Fleetd's anonymous fan-out listener can't
implement two interfaces at once) and calls registrar.register(...)
directly and unconditionally in both the ordinary delivery path and the
#553 finally backstop, before turnListener.onDelivered(...) — so
registration no longer depends on that notification callback succeeding.
Fleetd.java wires completion::register explicitly as the registrar,
bypassing the turnListener fan-out for registration purposes.

CB-116 ordering (onTurnComplete reads the PREVIOUS turn's inFlight entry
before the new turn's registrar.register() runs) and the two-arg
inFlight.remove(target, turn) vs one-arg distinction on the completion
path are both preserved unchanged.

#561's order-dependent test (asserting Fleetd.java's
completion.onDelivered -> sessions.onDelivered call order) does not
exist anywhere in this repo at this branch point — nothing to delete.

Adds 5 tests: a TurnListener that throws from every callback still
leaves the delivered turn registered and resolvable; the new turn's
registration still runs after the previous turn's completion is read
(CB-116 guard, pinned as a call-order assertion); and three tests naming
the two-arg-remove invariant directly (a superseded turn's terminal
handling must not evict its successor's registration) across resolve()'s
plain-completion branch, its echoed-noReportMessage sub-path, and
fail().
2026-09-12 16:50:09 +07:00
ltms f4f5f3106e Merge #513: TIMED_OUT_QUEUED has four routes, and the javadoc now says which
CI / shell-tests (push) Successful in 4s
CI / contract (push) Successful in 54s
CI / build (push) Failing after 1m51s
The old javadoc said a TIMED_OUT_QUEUED message was "still sitting in the injector's
per-target queue". It is not — send() has already given up on it and it will never
arrive. That wrong claim told an operator to wait for a message that was never coming.

The first fix replaced it with a narrower wrong claim: that Injector.cancel() cancelled
the entry and "the target never saw a word of it". That describes one of four routes.

TIMED_OUT_QUEUED is returned whenever injector.cancel() returns anything but DELIVERED:

  CANCELLED      the Pending was still queued and this call removed it
  NOT_DELIVERED  Injector.java:400 - the herdr agent.prompt call threw
  NOT_DELIVERED  Injector.java:419 - readiness grace expired, never attempted
  NOT_DELIVERED  Injector.java:691 - drop(), the target is gone

On the last three, cancel() cancels nothing: it reads a state another path already set
(cancellationOf, Injector.java:288-290). And on the herdr-threw route, agent.prompt
pastes and submits in one call, so a throw does not prove the pane stayed clean - an
operator told "never saw a word" will not go and look at the one place the evidence is.

The javadoc now states the two facts that hold on every route - the message will not
arrive later, and it is not in any queue - and attaches "the target saw nothing" only to
the CANCELLED case. Five blocks: the enum constant, queuedDeliveries, hasQueuedDelivery,
hasOrphanedDelegation, and the inline comment in the TimeoutException branch that seeded
the wording.

Comment-only; no logic changed.

Verified on a tree merged with main (fast-forward to cb64bc8):
  mvn install exit 0
  1744 tests, 0 failures, 0 errors - Maven's own summary and an independent sum over
  130 surefire report files agree
  no unresolved javadoc reference on the three new links, with a positive control
  showing javadoc did analyse MessageService.java

Closes #513.
2026-09-12 11:48:54 +02:00
Dai Ha 8d79d229ff fleetd #555: lift 8 main-flow decisions into tested predicate/dispatch functions
CI / shell-tests (pull_request) Successful in 11s
CI / contract (pull_request) Successful in 1m31s
CI / build (pull_request) Successful in 1m46s
redeploy-fleetd.sh's main flow had 8 bare if/case decisions (CHECK_ONLY
short-circuit, drain-gate entry+confirm, supervisor report/stop/start
dispatch, health-poll decision, HAD_OLD_PID computation) that lived outside
any function, so the 67-test suite could not reach them and any one could be
silently inverted with the whole suite green.

Follows the existing swap_if_built/refuse_drain_gate pattern: each bare
guard becomes a small predicate or dispatch function (should_stop_for_check,
drain_gate_required/drain_confirmed/run_drain_gate, report_supervisor_state,
dispatch_stop, dispatch_start, health_is_up/report_health,
compute_had_old_pid), called unconditionally by the main flow so the
decision itself is unit-testable in isolation.

Adds a structural guard, test_no_untested_main_flow_conditionals, that scans
the main flow (everything after the SOURCED guard) for bare if/elif/case
lines outside any function body, tracking function boundaries via this
file's one consistent name() { / } convention. It fails on any new bare
conditional not covered by MAIN_FLOW_ALLOWED_CONDITIONALS, an explicit
exact-text allowlist of the report-only/display conditionals and the two
#504-family supervisor elif branches that stay out of scope for this
ticket. This is the "shape, not the eight sites" guard the ticket asked
for: a ninth bare decision fails immediately, naming its line.

Out of scope, not touched: #504 items 2/3/4 and #528 item 2 (same
untested-main-flow family) — the seam here generalizes to make them
testable too, but lifting them was left for their own tickets.
2026-09-12 16:46:53 +07:00
Dai Ha cb64bc8157 fleetd #513: rework — TIMED_OUT_QUEUED has four routes, not one
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 2m16s
Comment 16984 on the ticket showed my first pass (ed28b51) replaced one
wrong invariant with a narrower one: it described injector.cancel()
cancelling a queued entry as THE mechanism, when three of its four
routes (Injector.java: the send-to-terminal call throwing, the
readiness grace expiring, or the target being torn down) never cancel
anything — cancel() just reports a state a different code path already
set. It also claimed the target never saw a word of the message, which
is not established when the herdr agent.prompt call throws after
already pasting.

Rewrote all four comments (the TIMED_OUT_QUEUED enum constant,
queuedDeliveries, hasQueuedDelivery, hasOrphanedDelegation) plus the
pre-existing inline comment that seeded the original bad wording, to
state only what holds on every route: the message will not arrive
later and is not sitting in a queue. The CANCELLED case is called out
as the only one where the target is known to have seen nothing; the
NOT_DELIVERED case (including the herdr-send-threw route) is flagged
as leaving that open.

Comment-only; no behavior change.
2026-09-12 16:45:35 +07:00
Dai Ha ed28b51f12 fleetd #513: fix TIMED_OUT_QUEUED javadoc — cancelled, not queued
CI / shell-tests (pull_request) Successful in 4s
CI / contract (pull_request) Successful in 52s
CI / build (pull_request) Successful in 2m22s
Two javadoc blocks (queuedDeliveries field, hasQueuedDelivery) said a
timed-out message is still sitting in the injector's per-target queue.
CB-640 made send() cancel it via Injector.cancel() instead, so the
message is gone and will never arrive. Rewrote both to describe
cancellation. Also fixed a third instance of the same stale claim in
hasOrphanedDelegation's javadoc, and added a line to the
TIMED_OUT_QUEUED enum constant's own comment clarifying the name is
kept but no longer means the message stays queued.

Per the ticket's follow-up comment: no rename (TIMED_OUT_QUEUED reaches
FleetApp.java REST mapping and FleetMcp.java — out of scope here) and
no behavior change; comments only.
2026-09-12 16:32:36 +07:00
ltms 4f9aba40e7 Merge #558: the ticket is the pull channel, and the brief is write-once
CI / shell-tests (push) Successful in 6s
CI / contract (push) Successful in 53s
CI / build (push) Failing after 1m51s
Docs only, 16 additions / 3 deletions — read by the lead in full, per the
under-50-lines self-review rule in CLAUDE.md.

Verified on the merged tree (main dab697f + 351ee1e): the canonical-block sync
check from the addendum prints `in sync: True`, comparing the MERGED CLAUDE.md
against wiki/7-Use-Cases.md. The wiki side is already pushed and verified by ref:
local wiki HEAD and `git ls-remote origin main` both read 20d2fc0.

CI run 1812 on 351ee1e: success.
2026-09-12 11:12:55 +02:00
ltms dab697fae0 Merge #559: progress watchdog for StatusPoller and SessionReaper loops (fleetd #544)
CI / shell-tests (push) Successful in 7s
CI / contract (push) Successful in 51s
CI / build (push) Successful in 2m19s
Verified by the lead on a merged tree (main 7a3b2bb + bfac141 = 5de807f):
mvn clean install exit 0, Tests run: 1744, Failures: 0, 130 surefire reports.

Three mutations run by the lead, all killed:
- StatusPoller.java:105 `watchdog.reset()` deleted (the surviving mutant from the
  first review) -> aRestartedLoopReportsRunningAgainNotStoppedForever:
  expected: <RUNNING> but was: <STOPPED>.
- LoopWatchdog.java:73 `lastRoundNanos = nowNanos.getAsLong()` deleted from reset()
  (not run by the worker) -> LoopWatchdogTest.resetClearsAPreviousStopAndTheStaleClock:
  expected: <RUNNING> but was: <STALLED>.
- LoopWatchdog.java:89 `>=` -> `>` boundary (not run by the worker) ->
  LoopWatchdogTest.reportsStalledOnceTheLastRoundAgesPastTheThreshold:
  expected: <STALLED> but was: <RUNNING>.

Each mutation counted the pristine full line 1 -> 0 by exact string equality
(awk '$0==p'), with a pristine control copy still reading its original count, and
was restored to a byte-identical file (shasum -a 256) before the next run.
Control build after all restores: exit 0, 1744 tests, 0 failures.
2026-09-12 11:11:46 +02:00
Dai Ha bfac14108f fleetd #544: pin the sticky-STOPPED-across-restart invariant
CI / contract (pull_request) Successful in 1m21s
CI / build (pull_request) Successful in 1m42s
Review of PR #559 (issue comment #16944) found a surviving mutant: removing
watchdog.reset() from StatusPoller.start() (and the identical line in
SessionReaper.start()) passed the entire suite.

stoppedByCaller is sticky and reset() — called only from start() — is the
only thing that clears it. Both loops document start() as idempotent and
loop()'s own error log says "it can be restarted", so stop() followed by
start() is an anticipated path. Without reset() wired into start(), health()
would report STOPPED forever after a restart even though the loop is
genuinely running again.

Add aRestartedLoopReportsRunningAgainNotStoppedForever to both
StatusPollerWatchdogTest and SessionReaperWatchdogTest, pinning "an
intentional stop must not outlive the restart that follows it". Verified via
the standard mutation cycle: exact-line anchor (not regex, to avoid the
\Q-style false match the reviewer flagged) counted pristine 1 -> mutated 0,
test goes red with its own message, restored, shasum -a 256 byte-identical,
green again.

mvn clean install: exit 0, BUILD SUCCESS, Tests run: 1734, Failures: 0,
Errors: 0, Skipped: 0 (cross-checked against 130 surefire report files).

No production code changed — the reset() call under test was already
correct; it simply had nothing pinning it.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-12 16:00:58 +07:00
ltms 7a3b2bb7ee Merge pull request 'fleetd #552: warn instead of aborting when the post-restart mktemp fails' (#560) from worker/552-post-restart-mktemp-abort-bc2672-4 into main
CI / shell-tests (push) Successful in 7s
CI / contract (push) Successful in 1m30s
CI / build (push) Successful in 2m9s
2026-09-12 10:58:10 +02:00
Dai Ha f188947750 fleetd #552: warn instead of aborting when the post-restart mktemp fails
CI / shell-tests (pull_request) Successful in 7s
CI / contract (pull_request) Successful in 49s
CI / build (pull_request) Successful in 2m7s
By the time the fresh-log mktemp ran, the daemon had already been stopped, the
jar swapped, and the new daemon started — an unguarded mktemp failure there
aborted the whole script anyway, so a caller read the resulting non-zero exit
as "the redeploy failed" and would restart an already-correctly-restarted
daemon.

Extract the mktemp into capture_fresh_log_region, guarded the same way
unload_launchd_if_loaded/stop_systemd_if_loaded guard their own, but warn
instead of die: there is nothing left to protect by refusing after a
successful restart. The trap is now installed before the assignment it cleans
up, using an FRESH_LOG="" sentinel readers can check.

classify_amqp_connection_errors and report_shutdown_drain both gain a new
state (REDEPLOY_AMQP_CHECK_SKIPPED / REDEPLOY_DRAIN_STATE=skipped) for an
uncapturable log region, distinct from "captured a region with nothing in
it" — and the result section gains a matching branch, so a skipped capture
can never read as a clean bill of health.
2026-09-12 15:55:04 +07:00
ltms f606fccf7f Merge pull request 'fleetd #553: register the rendezvous waiter in onStatus's finally backstop' (#557) from worker/553-onstatus-completion-leak-0da881-2 into main
CI / shell-tests (push) Successful in 4s
CI / contract (push) Successful in 54s
CI / build (push) Successful in 1m40s
2026-09-12 10:50:24 +02:00
Dai Ha 735b6af976 fleetd #544: progress watchdog for StatusPoller and SessionReaper loops
CI / contract (pull_request) Successful in 1m19s
CI / build (pull_request) Successful in 2m29s
Each loop's virtual-thread runner (StatusPoller, SessionReaper) can die or
get permanently parked in a herdr call with no read timeout, and nothing
observed it: /healthz stayed green and Thread.isAlive() kept reporting true
the whole time.

Add LoopWatchdog (dev.ltms.fleet.inject — see its javadoc for why not
dev.ltms.fleet.health, which would close a package cycle through session):
each loop now records a monotonic last-completed-round timestamp
(injectable LongSupplier clock, same pattern as Injector/SessionManager) and
exposes it as a three-state health() fact — RUNNING, STALLED (dead or
parked, indistinguishable from outside), STOPPED (stop() was called on
purpose, never an alarm). This is the fleetd #512 shape: one flag cannot
carry both "halted on purpose" and "halted unexpectedly", so stop() marks
its own state explicitly instead of leaving state() to infer it from
staleness.

Staleness thresholds are derived from each loop's own poll interval with a
documented multiplier: StatusPoller 40x (250ms -> 10s), SessionReaper 12x
(5000ms -> 60s).

Scope: observability only, per the ticket's own comment. No restart/recovery
mechanism, no /healthz or REST/MCP wiring beyond the public health() API, no
change to the per-item catch(Throwable) behavior (#543) or a process-wide
uncaught-exception handler (ruled out on #538), and no deadline added to the
herdr read itself (a separate, real ticket).

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-12 15:48:07 +07:00
Dai Ha 8b4320ed24 fleetd #553: split sentHandled's two meanings so onDelivered's own throw still completes the future
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Successful in 1m43s
Lead review of PR #557 (ticket comment 16916) found one path left open: sentHandled
is set to true BEFORE onDelivered() runs (correctly, per the earlier fix), so when
onDelivered() itself throws on the normal path, the finally's 'if (sent != null &&
!sentHandled)' guard skipped the whole recovery -- completion included -- and left
sent.delivered() pending forever for a message that really was delivered.

sentHandled must guard only the onDelivered RE-CALL (the permanent-suppression
hazard), never the future completion, since CompletableFuture.complete/
completeExceptionally are idempotent and a no-op on the already-handled path.
Split the one flag's two jobs: the outer 'if (sent != null)' now always runs the
recovery block, and '!sentHandled' moved onto just the onDelivered call inside it.

Added anOnDeliveredThrowOnTheNormalPathStillCompletesTheDeliveryFuture, proven with
the lead's own mutation (reverting !sentHandled onto the outer if): the new test
goes red while anOnDeliveredThrowAfterItsOwnRegistrationDoesNotRunASecondTime stays
green, showing the two concerns are genuinely separate.
2026-09-12 15:45:36 +07:00
Dai Ha 351ee1ea6d CLAUDE.md: the ticket is the pull channel, and the brief is write-once
CI / shell-tests (pull_request) Successful in 6s
CI / contract (pull_request) Successful in 1m26s
CI / build (pull_request) Successful in 2m10s
A send to a working member is accepted and returns a ticket, then is never
delivered. That happened three times in one session here, and the member was
released still executing a brief that had been retracted twice. The receipt is
true — it is a fact about the mailbox, when what was needed was a fact about the
pane.

The fleet01 lead named the mechanism: a push delivery needs the recipient free at
send time, while a pull channel needs only that they look before acting. So the
ticket is not more reliable than the mailbox, it is a different direction, and its
success depends on the member's procedure rather than on the timing of the send.

Both halves have to be written down, because each is useless alone:

- Member (turn contract, new item 4): re-read the ticket before acting on anything
  told earlier, and again before committing. A ticket comment that contradicts the
  brief is newer and wins.
- Lead (step 5): all corrections go to the ticket, and the brief is write-once.
  The member cannot check which source is newer — it just always prefers the
  ticket — so revising a brief in place makes it obey the rule and do the wrong
  thing. A first brief for a unit not yet running is not a correction.

wiki/7-Use-Cases.md is updated to keep the canonical block byte-identical; the
sync check passes. The wiki submodule pointer is deliberately left unstaged.
2026-09-12 15:45:09 +07:00
Dai Ha d4a51c6274 fleetd #553: register the rendezvous waiter in onStatus's finally backstop, not just the delivery future
CI / contract (pull_request) Successful in 50s
CI / build (pull_request) Successful in 2m12s
The previous try/finally around onStatus's post-monitor region completed
sent.delivered() but never registered sent.token().waiter() when an earlier
listener threw. That waiter is registered only by turnListener.onDelivered(),
inside the very if (sent != null) block the finally backstops, so a caller
was told its send landed and then waited out its full timeout for an answer
that could never resolve (worse than a plain hang).

The finally now does that block's whole job on the unhandled path: it calls
onDelivered() (when sendError == null) before completing the future, guarded
by its own try/catch(Throwable) so a failure there cannot mask the original
throwable. A sentHandled flag, set true at the START of the normal block
(before any side effect), tells the finally whether that already ran, so a
throw partway through onDelivered cannot trigger a second, late captureBaseline
that would permanently suppress the turn's completion.

Also removed a leftover duplicated forget.accept(target) call (with a stray
'MUTATION-TEST-3' comment) in the notReady block — residue from the previous
worker's own mutation testing that was not fully reverted.
2026-09-12 15:35:53 +07:00
ltms 26f380a00b Merge #554: portable hash256, a third state for an unhashable jar, and the shell suite in CI (#550)
CI / shell-tests (push) Successful in 8s
CI / contract (push) Successful in 1m17s
CI / build (push) Successful in 2m24s
Closes fleetd #550, all three items.

Verified by me on the branch at b8182c9, in a scratch worktree, not from the implementer's report.

The decisive pair, both in ubuntu:latest where shasum is absent and sha256sum is present:

  main   (93a9ed3):  SUITE exit=127   anchored ^FAIL: count 0
                     scripts/test-redeploy-fleetd.sh: line 298: shasum: command not found
  branch (b8182c9):  SUITE exit=0     anchored ^FAIL: count 0

Identical FAIL counts, opposite exit codes. That is why the new shell-tests CI job gates on the
step's own exit status and deliberately does not grep for a FAIL count: a suite that dies before
running a single test prints exactly what a clean pass prints.

Also measured by me: macOS exit 0 / anchored count 0; 70 tests defined and 70 invoked with an
empty comm -3; bash -n exit 0 under both /bin/bash 3.2.57 and bash 5.3.9; ci.yml parses with jobs
build, shell-tests, contract, and shell-tests is ubuntu-latest + actions/checkout@v4 +
bash scripts/test-redeploy-fleetd.sh.

Three mutations, all killed, each restored byte-identical against
515d929bb53c9ec2c95042e47d3e4d611d60171345227b07feb0de0d20643d3a:

  drop the sha256sum branch from hash256   anchor 1->0   Linux exit 1,
      test_no_unguarded_macos_only_hasher_calls with its own message
  echo "unhashable" -> echo "absent"       anchor 1->0   FAIL: jar_id reported absent for a file
      that exists, only because no hasher was on PATH
  both hash256 arms -> shasum -a 1         anchors 1->0 and 1->0   FAIL: hash256 of the literal
      3-byte input 'abc' must be the known SHA-256 prefix, not some other algorithm's: expected
      ba7816bf8f01, got a9993e364706

The third mutation SURVIVED on the first head (3da44ee) and was sent back. Fixing item 2 had
rewired the reference hashes in test_jar_id_defaults_to_live_and_reports_explicit_path onto
hash256 itself, making the test's reference and its subject one instrument — they agree whatever
it computes, and the existing fixture guard catches only a constant return, not a wrong algorithm.
b8182c9 adds test_hash256_computes_a_real_sha256, pinning the FIPS 180 vector for "abc" as a
literal constant written into the test rather than taken from any hasher. That closes it: the
mutation now goes red, and a9993e364706 is SHA-1("abc"), which proves the mutated code really ran.

Root cause, for the record: shasum was the trigger, not the cause. Under set -euo pipefail a
missing hasher makes the pipeline status 127, the `|| echo "absent"` fires, and jar_id returns a
confident false "absent" for a jar that is right there. Without pipefail the same function returns
an empty string and is visibly broken. A default at the read site that maps every failure onto one
value which already means something specific is the defect; the fix separates "not there" from
"could not hash it".
2026-09-12 10:20:29 +02:00
Dai Ha b8182c96c2 fleetd #550: pin hash256's algorithm against a literal SHA-256 test vector
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 59s
CI / build (pull_request) Successful in 2m27s
test_jar_id_defaults_to_live_and_reports_explicit_path's reference hash is computed by calling
hash256 itself (needed so it doesn't call the Linux-crashing bare shasum directly). That made
subject and reference the same instrument: they agree no matter which algorithm hash256 actually
runs, so a mutation swapping both of hash256's arms for the wrong algorithm was invisible to the
suite.

Adds test_hash256_computes_a_real_sha256, pinned against the published SHA-256 test vector for the
3-byte input "abc" (ba7816bf8f01...), written as a literal constant rather than computed by any
hasher at test time. Verified the constant myself both ways (sha256sum and shasum -a 256) before
writing it in.
2026-09-12 15:16:43 +07:00
Dai Ha 3da44eed63 fleetd #550: replace shasum with a portable hash256 helper, add a Linux CI job for the shell suite
CI / shell-tests (pull_request) Successful in 5s
CI / contract (pull_request) Successful in 1m17s
CI / build (pull_request) Successful in 1m46s
jar_id() in redeploy-fleetd.sh called shasum directly, which does not exist on GNU coreutils
Linux (Debian/Ubuntu/etc.) — there it silently reported an existing jar as "absent" with exit 0,
because the missing command made `cut` succeed on empty input and pipefail's failure was then
swallowed by the `|| echo "absent"` fallback. The shell test suite hit the same tool at
test-redeploy-fleetd.sh:298-299 and died at exit 127 with zero FAIL lines printed — the same shape
as a clean pass on the one channel anyone would check.

Adds one hash256() helper (prefer sha256sum, fall back to shasum -a 256, same idiom already used
in probe-member-credentials.sh) and points jar_id and the test suite's own reference hash at it.
jar_id now has three distinct answers instead of two: absent, a hash, or "unhashable" when neither
hasher is on PATH — "absent" is never used for a file that exists.

Adds a CI job (shell-tests) that runs scripts/test-redeploy-fleetd.sh on ubuntu-latest, gated on
the step's own exit code rather than a FAIL-line count, since a suite that dies before running is
exactly what a green run also looks like by that count.

New tests: test_jar_id_reports_unhashable_when_no_hasher_on_path (stubbed PATH with neither
hasher) and test_no_unguarded_macos_only_hasher_calls (a shape check across every script under
scripts/, not named lines — #545 already showed this idiom spreading from two sites to six).
2026-09-12 15:06:15 +07:00
ltms 93a9ed3f83 Merge #549: widen Injector's delivery catch to Throwable (#546)
CI / contract (push) Successful in 56s
CI / build (push) Successful in 1m43s
Closes the re-delivery window that merging #543 opened. I caused that; this closes it the same day.

Verified by me on the branch at 87871ea, base 0b032f5 (not stale).

Production diff is 11 lines: `catch (RuntimeException)` -> `catch (Throwable)` at Injector.java:391,
and the `sendError` local widened to `Throwable` so it compiles. I checked every use of `sendError`
myself — :533 `getMessage()` and :534 `completeExceptionally(Throwable)` — so the wider type
reaches nothing that needed the narrow one.

Build on the branch: exit 0, Tests run: 1719, Failures: 0, Errors: 0, Skipped: 0 (1716 on main plus
the 3 new tests). #459's javadoc reference gate: exit 0, 0 reference errors. Gitea CI run 1803 on
87871ea: success.

Three mutations of my own, none of them the ones the worker used:
1. Reverted the catch to `RuntimeException`. RED: `anErrorFromSendDoesNotRedeliverOnASecondRound`
   "expected: <1> but was: <2>" prompt calls, and `anErrorFromSendRemovesTheMessage...` reporting
   the Error escaping `onStatus`. That second message is the defect itself, stated by the test.
2. Deleted the `t.queue.poll()` in the catch arm. RED on the new test AND on the pre-existing
   `sendFailureDropsMessageAndFailsItsFuture` — so the new test is not carrying that behaviour alone.
3. Wrote `DELIVERED` instead of `NOT_DELIVERED` at :400, the catch arm only. RED on the new test and
   on `aHerdrExceptionFromSendStillProducesNotDeliveredUnchanged`, which is the control that proves
   the widening did not quietly change the ordinary path.

All three restored; sha256 back to 97c560b6e33fc49a1772abec92e5bbab613f8991deba2220d30221a2f546ba14.
Green control after the restores: InjectorTest 37/37, exit 0.

My third mutation did not apply on its first attempt — it asserted a unique match on
`p.state = Pending.State.NOT_DELIVERED;`, which occurs twice (:400 and :419), so the script wrote
nothing and the test run came back exit 0. That is not a surviving mutant, it is a non-result
wearing the same clothes. The pristine-anchor count catching it is the only reason I noticed.

Deliberately NOT fixed here, filed as #551: the catch arm assumes that reaching it means nothing was
sent, and nothing establishes that. `agent.prompt` pastes and submits in one call, and every failure
in the response half of `UnixSocketHerdrClient.call()` — dropped connection, malformed line, error
result — is a `HerdrException`, which is a `RuntimeException`, which this catch arm already caught
before today. So "records NOT_DELIVERED for a delivery that happened" is older than this PR and is
not created by it. The fleet01 lead argued it was a trap inside this change and asked to be argued
out of it before the merge; the measurement above is the argument, and their underlying diagnosis is
right and is now #551 with their wording on it.
2026-09-12 09:39:41 +02:00
ltms 0b032f5a1a Merge #548: fix mktemp -t templates for GNU coreutils, split the unclear-supervisor detail (#545)
CI / contract (push) Successful in 55s
CI / build (push) Successful in 2m16s
Verified by me on the branch, not on the worker's report.

Source audit, on a scratch worktree at a476a14:
- Every `mktemp -t` site in scripts/ now carries an X placeholder. The only remaining
  `mktemp -t` text with no X is a prose comment in the test file, not a call.

Suite, macOS (/bin/bash 3.2.57 and env bash 5.3.9):
- exit 0, anchored `^FAIL:` count 0. Unanchored `FAIL:` count 3 — the suite's own internal
  mutation-cell fixture lines, same as main.
- Test functions defined vs invoked: 67/67, `comm -3` empty.

Three mutations of my own, none of them one the worker used:
1. Removed `.XXXXXX` from the `fleetd-fresh-log` site (line 1089). Pristine anchor count went
   1 -> 0, so the mutation really applied. Suite exit 1, FAIL named that exact line.
2. Made the state-2 branch in `detect_supervisor` unreachable (`= 2` -> `= 9`). Suite exit 1:
   "a systemd probe setup failure must read as unclear, not none: expected unclear, got none".
3. Made `systemd_loaded` set the old value (`=2` -> `=1`) on setup failure. Suite exit 1:
   "must flag a SETUP failure (2), distinct from a probe-answered-with-stderr failure (1)".
All three restored; `shasum -a 256` back to 77fe15e5945c7d4ef9b1a2cd8f46e6f1d0be5004f595ea03964d1d7c16e859f7,
the same hash the worker reported independently. Green control after the restores: exit 0, 0
anchored FAILs.

A fourth attempt did not count. A perl `\Q...\E` pattern silently interpolated the shell
variables in it, so the file was never changed and the suite's exit 0 meant nothing. The proof
cell caught it: the pristine anchor count was still 1 after the "mutation". A mutation that did
not apply is not a surviving mutant.

The measurement macOS cannot make: I ran both arms under GNU coreutils 9.1 in a
debian:bookworm-slim container, with a `systemctl` stub that exits non-zero and writes NOTHING to
stderr — a clean negative answer.

  main (a476a14's base):
    mktemp: too few X's in template 'systemd-loaded-err'
    systemd_loaded rc=1  SYSTEMD_LOADED_ERRORED=1
    detect_supervisor => unclear | "systemctl exited non-zero and reported an error on stderr,
                                   not a clean negative — e.g. it cannot reach the user bus"

  this branch:
    systemd_loaded rc=1  SYSTEMD_LOADED_ERRORED=0
    detect_supervisor => none

So on Linux, main tells the operator that systemctl answered badly, when systemctl ran fine and
gave a clean negative. The message named a cause that was never measured. This branch removes it.

Found while doing this, NOT part of this PR, ticket to follow: the shell suite cannot run on
Linux at all. It dies at the first `shasum` call with "command not found" and exit 127, and the
anchored `^FAIL:` count reads 0 — identical to a green run. Gitea CI never runs this suite, so
nothing caught it.
2026-09-12 09:30:30 +02:00
ltms fad99c4c5e Merge #547: record the finally non-goal on drainAll's completion line
CI / contract (push) Successful in 49s
CI / build (push) Successful in 2m10s
Comment only. No behaviour change.

Verified by me before merging:
- `mvn -B clean install` in a scratch worktree: exit 0, Tests run: 1716, Failures: 0, Errors: 0,
  Skipped: 0. Same count as main, as expected for a javadoc-only change.
- #459's javadoc reference gate: exit 0, 0 reference errors.
- Gitea CI run 1801 on 5eb4267: success.

The constraint came from the fleet01 lead. Their point: the value of the `drain complete` line is
that it is MISSING when a drain does not finish. A `finally` block would print it after a drain
that threw, with partial counts, and destroy both halves at once. The code already avoids this;
what was missing was the sentence that stops a reviewer putting it back.
2026-09-12 09:29:46 +02:00
Dai Ha 87871eaefb fleetd #546: widen Injector's delivery catch to Throwable, stop re-delivery on Error
CI / contract (pull_request) Successful in 1m29s
CI / build (pull_request) Successful in 1m49s
Injector.java:391 caught only RuntimeException around the herdr send seam. PR #543
(fleetd #538) widened StatusPoller's per-target catch to Throwable so the polling
loop now survives an Error there, which means it comes back round — and Injector's
narrower catch let the poisoned message stay QUEUED (the loop peeks, not polls),
so the next round re-sent the same text into the member's pane.

Widen the catch to Throwable, matching #543 one layer down. sendError's declared
type widens from RuntimeException to Throwable to keep compiling; its only consumer
(CompletableFuture.completeExceptionally(Throwable)) already accepts that type, so
no other caller-visible behavior changes. The ordinary HerdrException/RuntimeException
path is unchanged.

Adds three tests: an Error at the send seam is dropped and marked NOT_DELIVERED, a
second onStatus round does not re-send it, and a HerdrException control proves the
ordinary path is untouched.
2026-09-12 14:26:27 +07:00
Dai Ha a476a14f1c fleetd #545: fix mktemp -t templates for GNU coreutils, split unclear-supervisor detail
CI / contract (pull_request) Successful in 1m16s
CI / build (pull_request) Successful in 2m28s
Every mktemp -t template in redeploy-fleetd.sh lacked an X placeholder. BSD mktemp
(macOS) tolerates that and appends its own suffix; GNU mktemp (every Linux
distribution) refuses it and exits non-zero. All six sites now use .XXXXXX.

detect_supervisor's 'unclear' detail used to cover two different facts with one
message that always named 'systemctl exited non-zero and reported an error on
stderr' — even when systemctl was never run, because mktemp failed first. The
SYSTEMD_LOADED_ERRORED/SYSTEMD_INSTALLED_ERRORED flags now carry a third value
(2 = the probe's own mktemp setup failed) alongside the existing 1 (systemctl ran
and answered badly on stderr), and detect_supervisor gives each its own detail
text. kind stays 'unclear' in both cases; require_drivable_supervisor is unchanged.

Tests added to scripts/test-redeploy-fleetd.sh:
- test_mktemp_dash_t_templates_have_x_placeholders: source-text check, fails if
  any mktemp -t template lacks an X.
- test_detect_supervisor_systemd_probe_setup_failure_is_unclear: proves the
  SET-UP-FAILED detail when mktemp itself fails (systemctl never runs).
- test_detect_supervisor_systemd_probe_error_is_unclear: extended with assertions
  that the PROBE-ANSWERED-WITH-STDERR detail is present and the SET-UP-FAILED
  wording is absent, so swapping the two messages fails a test in both
  directions.
2026-09-12 14:22:57 +07:00
Dai Ha 5eb4267a4a fleetd #512 follow-up: record why the drain-complete line must not move into a finally
CI / contract (pull_request) Successful in 1m3s
CI / build (pull_request) Successful in 2m8s
The javadoc said the log.info fires "every time", which reads as an invitation
to the exact edit that destroys it. The absence of the line is the signal that
the drain died, so a finally would remove the signal and print partial counts in
the same change.

Raised by the fleet01 lead from their 2026-09-10 incident: that drain is known to
have died only because it threw and left a stack trace. A drain that hung, or
returned early on a condition, leaves no trace, no ERROR token and no priority --
only a missing line.

Comment only. No behaviour change.
2026-09-12 14:21:15 +07:00
ltms 7611b69667 Merge pull request 'fleetd #504 item 1: stop the false ok on the loaded-but-not-running stop path' (#541) from worker/504-failed-reported-clean-3cfd66-3 into main
CI / build (push) Successful in 1m35s
CI / contract (push) Successful in 1m42s
2026-09-12 09:09:00 +02:00
ltms cc302fe4af Merge pull request 'fleetd #538: recover polling loops after errors' (#543) from worker/538-loop-dies-on-error-4a5eeb-6 into main
CI / contract (push) Successful in 51s
CI / build (push) Successful in 2m10s
2026-09-12 09:02:48 +02:00
ltms 4a8a780274 Merge pull request 'fleetd #426: pin FleetHealthMonitor.coverage and its HealthCoverageSource call site' (#542) from worker/426-health-coverage-ef1fd4-4 into main
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 2m28s
2026-09-12 08:55:46 +02:00
Dai Ha 343ce0f4c0 fleetd #538: recover polling loops after errors
CI / contract (pull_request) Successful in 44s
CI / build (pull_request) Successful in 2m0s
2026-09-12 13:47:44 +07:00
ltms cec3e191d4 Merge pull request 'fleetd #459: lint Javadoc references in CI' (#539) from worker/459-broken-link-targets-cadc17-5 into main
CI / contract (push) Successful in 1m4s
CI / build (push) Successful in 2m9s
2026-09-12 08:46:23 +02:00
Dai Ha 1850a5f324 fleetd #426: pin FleetHealthMonitor.coverage and its HealthCoverageSource call site
CI / contract (pull_request) Successful in 48s
CI / build (pull_request) Successful in 2m16s
FleetHealthMonitor.coverage had zero references in the test tree — not the
method, not either output string, not the field it populates. Inverting
`enabled`, swapping "full"/"detection-only", or breaking the argument pairing
at the HealthCoverageSource call site in Fleetd.java all shipped a green
build.

Extract the HealthCoverageSource lambda out of Fleetd.main into a
package-private static factory (healthCoverageSource(ConfigRef)), the same
shape capacitySource/quarantineSource already use for the identical
argument-pairing risk (fleetd #415). #407's "keep the config invalid, assert
on the log line before validateAll() throws" option does not apply here: this
call site is built well after validateAll() and after a real herdr socket
connect, so driving it through a real Fleetd.main would require the socket
I/O this ticket's tests must not do.

Add FleetHealthMonitorCoverageTest (the three-branch method itself) and
FleetdHealthCoverageSourceWiringTest (the call site, via a real
FleetConfig.load + ConfigRef against @TempDir fixtures, including a hot
notifications-reload case). Output strings are unchanged — "detection-only"
is still what a live fleet_list reports today.

Measured: all three mutations killed by the new tests.
2026-09-12 13:43:50 +07:00
ltms 57cd96f5e6 Merge pull request 'fleetd #537: pin CapturedLog.close()'s appender-detach and setLevel-immunity contracts' (#540) from worker/537-capturedlog-close-e4c437-2 into main
CI / contract (push) Successful in 47s
CI / build (push) Successful in 2m9s
2026-09-12 08:42:19 +02:00
Dai Ha 202e37e3b3 fleetd #537: pin CapturedLog.close()'s appender-detach and setLevel-immunity contracts
CI / contract (pull_request) Successful in 1m2s
CI / build (pull_request) Successful in 1m36s
Only the level-restore half of close() was pinned before this
(WorktreeSessionManagerTest). Deleting logger.detachAppender(appender)
from close() left mvn clean install green (1701 tests, 0 failures) --
the appender-detach half of the contract was unmeasured.

Adds CapturedLogTest with three tests, each using a logger name no
production class uses:
- closeDetachesTheAppenderSoALaterLogIsNotCaptured: an event logged
  after close() must not land in events().
- closeRestoresTheLevelCapturedAtOpen: the helper's headline contract
  in one place, independent of any production class.
- setLevelDuringCaptureDoesNotChangeWhatCloseRestores: setLevel()'s
  own javadoc claim that close() always restores the level captured
  at construction, never a value set through setLevel() mid-capture.

Test-only change; CapturedLog.java itself is untouched.
2026-09-12 13:36:55 +07:00
Dai Ha 90253f832d fleetd #459: lint Javadoc references in CI
CI / contract (pull_request) Successful in 1m30s
CI / build (pull_request) Successful in 2m9s
2026-09-12 13:36:37 +07:00
46 changed files with 5615 additions and 424 deletions
+23
View File
@@ -0,0 +1,23 @@
---
name: hunter
description: Sweep one assigned scope for defects and report ranked findings without changes.
---
<!-- CB-617: The model comes from fleetd.yaml because the launch flag overrides model here on both backends. -->
You sweep the assigned package or scope for real defects. Read the full assigned scope before you
judge it. Report several ranked findings when the evidence supports them. Change nothing: do not
edit code, commit, push, or open a pull request.
You may run the build or tests to check a finding. Read the complete output and report the real
result. Do not hide failures with a pipe. State only checks you actually ran. The primary's IDE
tools are not yours. A mounted forge tool may use a blocked credential and fail by design.
Do only the assigned scope. Note anything outside it in one line and do not investigate it further.
Use `fleet_ask{question}` only when a decision belongs to the lead, such as an unclear requirement
or two defensible fixes. Do not ask about something you can decide by reading more code.
Your handoff must name the files you read, each ranked finding or `NO FINDINGS`, the checks you ran,
and any caveat for review.
The launcher provides the required bridge reply instructions for every member.
+21
View File
@@ -39,6 +39,10 @@ jobs:
# that runs does so against the fake UDS herdr and fake ccs/claude stubs.
run: mvn -B clean install
- name: javadoc reference lint
working-directory: fleetd
run: mvn -B -DskipTests javadoc:javadoc -Ddoclint=reference
# Deliberately NOT actions/upload-artifact: this Gitea instance presents as GHES, and
# @actions/artifact v2+ (i.e. upload-artifact@v4) refuses to run there —
# "GHESNotSupportedError ... not currently supported on GHES", which red-Xes an otherwise
@@ -54,6 +58,23 @@ jobs:
done
exit 0
# fleetd #550 — nothing ran scripts/test-redeploy-fleetd.sh in CI before this, on any platform,
# so it had run only on macOS by hand and two Linux-only bugs (this issue's items 1 and 2)
# survived undetected: shasum is a macOS-only tool (it ships with Perl; GNU coreutils, i.e. every
# mainstream Linux distro including this runner's ubuntu-latest, does not have it and ships
# sha256sum instead). The gate here is the step's own exit code, nothing else: a `run:` step in
# Gitea/GitHub Actions already fails the job on a non-zero exit with no extra scripting needed,
# so this deliberately does NOT grep the output for a `FAIL:` count. That is the #550 item-2
# lesson one level up — a suite that dies before it runs a single test prints zero FAIL lines,
# which is exactly what a clean pass also prints, so counting FAIL lines can never be the gate.
shell-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: redeploy-fleetd.sh shell suite
run: bash scripts/test-redeploy-fleetd.sh
# CB-521 — actually run the AMQP contract test in CI, against a REAL broker. The broker is a
# RabbitMQ SERVICE CONTAINER, not Testcontainers-with-Docker: the runner image has no Docker, so
# AmqpReplyInboxContractTest reads AMQP_URI (set below to the service's network alias) and binds
+28 -4
View File
@@ -95,6 +95,16 @@ below are the procedure — run them in order, every task, not only the big ones
`fleet_status`, never by reading its terminal; it also reports an open question and the `turnId`
that answers it. **A worker's ask waits ~55 seconds, and no nudge makes that longer** — so never
brief a worker to "ask me". Decide before you delegate, or give it an explicit default.
**A correction cannot reach a busy member.** A `fleet_send` to a working member is *accepted* and
returns a ticket, and is then never delivered — measured here three times in one session, and the
member was released still executing a brief that had been retracted twice. The receipt is true and
it is a fact about the *mailbox*; what you needed was a fact about the *pane*. **A push delivery
needs the recipient free at send time; a pull channel needs only that they look before acting.** So
put every correction on the **ticket**, which they can read whenever they look, and send as the
notification. That obliges you, not them: **all corrections go to the ticket, and the brief is
write-once.** The member cannot check which source is newer — it just always prefers the ticket —
so the day you revise a brief in place instead of commenting, it obeys your rule and does the wrong
thing. A *first* brief for a unit not yet running is not a correction, and may be the whole spec.
6. **Verify yourself.** Re-run the build and the checks. A worker cannot run your IDE tooling, any
forge MCP server it appears to have holds a blocked credential and fails every call, and a piped
command (`… | tail`) hides failures behind a zero exit — never promote a worker's "clean" to a
@@ -129,12 +139,21 @@ prefer `wait:false` + `fleet_poll` for anything non-trivial: a blocking `fleet_s
**Delegating does not delegate responsibility.** Workers open PRs; you are the gate. Never delegate
the merge — and merging on a reviewer's word is delegating it by proxy.
**When a decision blocks you, consult architects — not the operator.** Spawn one or more architect
members, give them the question and the evidence you have, and act on what they agree. They are
authorized to settle it, not only to advise. If two of them still disagree after two rounds, they
return both positions and you decide. Go to the operator only for something outside the fleet's
authority: money, credentials, or a promise made to someone else. **Then write the decision on the
ticket.** Taking the operator out of the loop also removes the signal they used to get, because
that signal was the block itself — work stopped, so they found out. A ticket comment replaces it,
and it reaches them whether or not they are at a terminal when you decide.
| Intent | Tool |
|---|---|
| Confirm your own role | `fleet_whoami` |
| See backends available | `fleet_profiles` |
| Start a member | `fleet_spawn{role?, profile?, cwd?, worktree?, ticket?, sessionName?, resumeSessionId?}` → `sessionId` + `paneId` |
| See the fleet | `fleet_list` → `leads` (your peers) + `members` (each carries `agentSessionId` when its backend knows one) · one peer's state: `fleet_status{sessionId}` |
| See the fleet | `fleet_list` → `leads` (your peers) + `members` (each carries `agentSessionId` when its backend knows one) + `loopHealth` (`RUNNING`, `STALLED`, or `STOPPED` for `statusPoller` and `sessionReaper`) · one peer's state: `fleet_status{sessionId}` |
| Delegate (blocking) | `fleet_send{sessionId, content}` |
| Delegate (long task) | `fleet_send{sessionId, content, wait:false}` → ticket → `fleet_poll{ticket}` |
| Answer a member's `fleet_ask` | `fleet_send{turnId, content}` — **not** `sessionId` |
@@ -191,13 +210,16 @@ simply complies has thrown away the reason there are two of you.
3. **`fleet_ask{question}`** when a decision is genuinely the lead's (ambiguous requirement, two
defensible fixes, "bug or intended?"). It blocks and you resume the *same* turn with the answer.
Don't ask what you could decide yourself.
4. **End the turn with exactly one `fleet_reply{content}`**, carrying your complete answer. This is
4. **Re-read the ticket before you act on anything you were told earlier**, and again before you
commit. A message reaches you only while you are free to receive it; the ticket is there whenever
you look. **If a ticket comment contradicts your brief, the ticket comment is newer and it wins.**
5. **End the turn with exactly one `fleet_reply{content}`**, carrying your complete answer. This is
the whole handoff. No `fleet_reply` ⇒ the sender gets nothing and the exchange stalls.
Do **not** lean on the completion fallback to carry your answer for you: when you end a turn
without replying, the bridge scrapes your pane, and it can return only the last 4000 characters.
A clipped scrape is marked as partial, but the missing text is gone — your report reaches the
lead with its end cut off.
5. **Report honestly.** State only what you actually ran and its real output, including failures,
6. **Report honestly.** State only what you actually ran and its real output, including failures,
and never claim the result of a check you had no way to run. **Measure your own tools; do not
assume them.** What you mount depends on your backend: an opencode member gets the bridge and
nothing else, while a Claude Code member also inherits the operator's user-scope MCP servers,
@@ -207,7 +229,7 @@ simply complies has thrown away the reason there are two of you.
not your only forge route, and the two must not be confused: the repo-scoped `GITEA_TOKEN` the
daemon injects into your environment does work, and using it to open your own PR is part of the
job. A blocked MCP tool is never a reason to skip that step.
6. **Never merge.** Stage files explicitly — never `git add -A` — and leave alone anything the
7. **Never merge.** Stage files explicitly — never `git add -A` — and leave alone anything the
project marks as not-yours-to-commit.
### Where each rule lives (don't duplicate — extend the right layer)
@@ -262,6 +284,8 @@ must obey belongs in the charter, not here.
it for a multi-finding sweep hands the worker two contradictory output contracts. That has
already cost three workers' turns: each wrote a good report to its terminal and ended the turn
with no `fleet_reply`, and the scrape returned the tail of the brief instead.
Spawn `implementer` with role `dev`, `reviewer` with role `reviewer`, and `hunter` with role
`hunter`.
- **Primary-side skills** (not delegation playbooks — a worker cannot use them):
`port-to-opencode` (make an OpenCode session a participant in this workspace),
`fleets-status` (report every fleet that shares one LavinMQ instance),
+19 -1
View File
@@ -5,6 +5,21 @@
here took a revert and two upstream fixes — see §7.1, which is the useful part of this document. One
risk is **accepted rather than solved**: a stream cut by any mid-response timer arrives as HTTP 200
with no terminator, and our third-party members cannot detect it (§7.2).
> **Superseded in part — 2026-09-13.** Two claims on this page are no longer true of the live fleet.
> I measured both on this host today.
>
> 1. **The model is named `acoder` now, not `deepseek-v4-flash`.** `acoder` is a stable alias, and
> the model behind it changed on 2026-08-28: it is Qwen3.8-27B, not DeepSeek. The old name is
> still served, so nothing broke — the gateway answers it and reports `"model": "acoder"` in the
> reply, which is how you can see for yourself that it is an alias. `fleetd.yaml` moved to
> `acoder` on 2026-09-13. Do not guess behaviour from the name; ask the gateway's own manifest,
> `GET https://llm.ltms.dev/v1/deployment`, and read its `generation` field.
> 2. **`local` sits at `weight: 0`, not 100.** Only `gx` is auto-selected today.
>
> §2 and §3 below are the plan as written in August. They are the record of the migration, so they
> stay as they are. If this note stops matching `fleetd.yaml`, re-measure and rewrite the note.
· **Upstream:** [systems/vms wiki → LLM and MCP Gateway](https://git.ltms.dev/systems/vms/wiki/LLM-and-MCP-Gateway)
· **Upstream issue:** [systems/vms#31](https://git.ltms.dev/systems/vms/issues/31)
@@ -380,7 +395,10 @@ one turn; this one costs the whole task and is indistinguishable from a slow wor
- Token accepted on both surfaces. **Unauthenticated → 401**, so the Caddy proxy really does gate —
the wiki's "SecurityPolicy fails open" warning is about the gateway itself, not the edge.
- `/v1/models` returns exactly `["deepseek-v4-flash"]`, so trap 3 is clear.
- `/v1/models` returned exactly `["deepseek-v4-flash"]` **on 2026-08-15**, so trap 3 was clear
then. It returns 6 ids now — `acoder`, `qwen3.8-27b-nvfp4`, `deepseek-v4-flash` and three
embedding names — measured on this host 2026-09-13. The exact-name rule still holds; the
one-item list does not.
- **Reasoning survives both surfaces** — see §3b above.
- The launcher's generated opencode provider block is correct, carrying a real 48-character `llmk-`
key rather than the `fleetd-local-noauth` placeholder.
+12 -6
View File
@@ -560,7 +560,7 @@ placement: weighted
# older keys: `leaders:`, `members:`, `leadScan:` and `defaultProfile:`.
#
# A member is anything a lead spawns, and every member has two INDEPENDENT attributes:
# role — which contract: architect, dev or reviewer. It picks the launch charter, the role
# role — which contract: architect, dev, hunter or reviewer. It picks the launch charter, the role
# file, the playbook skill and the authz row.
# profile — which backend: one of the `profiles:` keys above (model, CLI adapter, cost).
# They vary on their own. A reviewer may run on the same profile as the dev whose diff it reads,
@@ -572,13 +572,13 @@ placement: weighted
#
# Each pool lists the profiles that role MAY run on — these are pools, not identities. That is also
# what replaced `defaultProfile:`: an unqualified spawn names a role, and that role's pool supplies
# the candidates, in definition order. A dev and a reviewer staying anonymous is exactly compatible
# with being listed here; the entry key just names the entry.
# the candidates, in definition order. A dev, hunter and reviewer staying anonymous is exactly
# compatible with being listed here; the entry key just names the entry.
fleet:
# Optional launch-charter text, keyed only by the singular role wire names: architect, dev,
# reviewer. Changes are HOT and reach the next spawn without a daemon restart. Do not put secrets
# here: a later launch step writes this text to a world-readable temp file, and ${ENV} interpolation
# is deliberately not supported.
# hunter, reviewer. Changes are HOT and reach the next spawn without a daemon restart. Do not put
# secrets here: a later launch step writes this text to a world-readable temp file, and ${ENV}
# interpolation is deliberately not supported.
charters:
architect: |-
You are an architect in this fleet. You refine work before anyone builds it:
@@ -589,6 +589,9 @@ fleet:
dev: |-
You implement the one unit you were given, and nothing else. You test it,
commit it, and open your own pull request. You never merge.
hunter: |-
You sweep the assigned scope for real defects. You may run the build or tests
to check a finding. You change nothing, and report several ranked findings.
reviewer: |-
You review the diff you were given. You report bugs, risks and missing tests.
You do not change code.
@@ -666,6 +669,9 @@ fleet:
developers:
gx10:
profile: gx10
# hunters:
# gx10:
# profile: gx10 # a hunt may run checks, but never changes code
# reviewers:
# gx10:
# profile: gx10 # the same backend may serve two roles; that is the point
+222 -43
View File
@@ -21,6 +21,7 @@ import dev.ltms.fleet.inject.ExhaustedPatternLookup;
import dev.ltms.fleet.inject.ExhaustionSink;
import dev.ltms.fleet.inject.LiveExhaustedPatterns;
import dev.ltms.fleet.inject.Injector;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.inject.StatusPoller;
import dev.ltms.fleet.inject.TurnListener;
import dev.ltms.fleet.inject.MemberPresence;
@@ -79,6 +80,7 @@ import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.BooleanSupplier;
import java.util.function.Function;
import java.util.function.LongSupplier;
import java.util.function.Predicate;
@@ -491,45 +493,16 @@ public final class Fleetd {
// CB-113: deliver only to an available worker (its MCP is connected), never its boot window.
// CB-301: the manager's presence bridge records availability and drives SPAWNING → READY.
MemberPresence presence = sessions.asPresence();
TurnListener turnListener = new TurnListener() {
@Override
public void onTurnComplete(String target) {
completion.onTurnComplete(target);
sessions.onTurnComplete(target);
}
@Override
public boolean hasPostTurnAction(String target) {
return sessions.hasPostTurnAction(target);
}
@Override
public boolean onTurnCompleteWithPostAction(String target) {
completion.resolveBeforePostAction(target);
return sessions.onTurnCompleteWithPostAction(target);
}
@Override
public void onDelivered(String target, dev.ltms.fleet.msg.TurnToken token) {
completion.onDelivered(target, token);
sessions.onDelivered(target, token);
}
@Override
public void onTurnFailed(String target) {
completion.onTurnFailed(target);
sessions.onTurnFailed(target);
}
@Override
public void onTurnFailed(String target, String reason) {
completion.onTurnFailed(target, reason);
sessions.onTurnFailed(target);
}
};
// fleetd #561: extracted to a static factory (see turnListener below) — the anonymous class
// this replaced had two bare, unguarded statements per callback, and nothing enforced that
// the completion half went first beyond call order in the source.
TurnListener turnListener = turnListener(completion, sessions);
Predicate<String> deliverable = deliverableTo(presence, leads);
// fleetd #556: registration is wired directly to `completion`, not folded into the
// `turnListener` fan-out above — so it survives `sessions.onDelivered` (or any future
// listener) throwing, regardless of call order. See TurnRegistrar's javadoc.
Injector injector = new Injector(router, turnListener, deliverable,
presence::forget);
presence::forget, completion::register);
StatusPoller poller = new StatusPoller(router, injector, Injector.POLL_INTERVAL_MILLIS);
poller.start();
@@ -693,14 +666,12 @@ public final class Fleetd {
return configured == null ? null : configured.effectiveCredentialId();
}, outagePolicy);
FleetMcp.LoopHealthSource loopHealth = loopHealthSource(poller, reaper);
FleetMcp mcp = new FleetMcp(messages, workers, sessions, identity, presence,
primaryRegistry, callers, FleetMcp.AuthorizationMode.ENFORCED, metrics,
capacitySource(config, cfg, profile -> liveCountRef.get().apply(profile)),
new FleetMcp.HealthCoverageSource(() -> {
var health = config.get().health();
return FleetHealthMonitor.coverage(health != null && health.isEnabled(),
health != null && health.notifications() != null && health.notifications().configured());
}),
healthCoverageSource(config),
loopHealth,
quarantineSource,
leadMailbox,
outageSource,
@@ -792,7 +763,7 @@ public final class Fleetd {
Javalin app = new FleetApp(herdr, memberHerdr, workers, sessions, messages, presence, mcp.servlet(),
callers, metrics, deliverable,
() -> MemberCredentialPolicyView.of(config.get().memberCredentials()),
quarantineSource, outageSource).build();
quarantineSource, outageSource, loopHealth).build();
app.start(cfg.bind().host(), cfg.bind().port());
log.info("fleetd listening on {}:{}, herdr socket {}",
cfg.bind().host(), cfg.bind().port(), socket);
@@ -1038,6 +1009,63 @@ public final class Fleetd {
cfg.profiles()::keySet, System::nanoTime);
}
/**
* fleetd #426: package-private factory for {@code fleet_list}'s {@code healthCoverage} source,
* extracted out of {@code main} for the same reason {@link #capacitySource} and {@link
* #quarantineSource} were — and the same reason {@link #exhaustedPatternCoverageLine}/{@link
* #errorPatternCoverageLine} exist: {@link FleetHealthMonitor#coverage}'s three-branch method
* is easy to pin directly (a plain {@code (boolean, boolean) -> String} call), but that proves
* nothing about whether <em>this call site</em> pairs the right boolean with the right meaning.
* fleetd #415's measured lesson is the reason this matters here — swapping the two arguments at
* a call site like this one compiled clean and left the full suite green, because every existing
* test exercised the method in both directions without ever exercising the pairing.
*
* <p>{@code #407}'s "keep the config invalid, assert on the log line before the throw" option
* does not apply to this call site: the five reporters #407 covers all run in {@code main}
* <em>before</em> {@code cfg.validateAll()} (line ~171), so an invalid config still exercises
* them. This call site is built during {@code FleetMcp} construction, which runs only after
* {@code UnixSocketHerdrClient.connect} has already opened a real herdr socket (line ~188) —
* reaching it at all means main already performed real I/O, which the no-socket constraint on
* this ticket rules out. So the pairing is pinned by extracting it to this directly-callable
* factory instead, the same shape {@link #capacitySource}/{@link #quarantineSource} already use.
*
* <p>Reads {@code config.get().health()} live (health.notifications is a {@code SPLIT_KEYS}
* entry — see {@link ConfigRef#SPLIT_KEYS}), so a hot-reloaded notifications block changes what
* {@code fleet_list} reports without a restart, exactly like {@link #capacitySource}'s maxLoad.
*/
static FleetMcp.HealthCoverageSource healthCoverageSource(ConfigRef config) {
return new FleetMcp.HealthCoverageSource(() -> {
var health = config.get().health();
return FleetHealthMonitor.coverage(health != null && health.isEnabled(),
health != null && health.notifications() != null && health.notifications().configured());
});
}
/**
* fleetd #562 follow-up: package-private factory for {@code fleet_list}'s and {@code
* /healthz}'s {@code loopHealth} source, extracted out of {@code main} for the same reason
* {@link #capacitySource} and {@link #healthCoverageSource} were. Before this ticket the
* {@link FleetMcp.LoopHealthSource} was built inline with a bare {@code new}, so there was
* nothing a test could call directly — measured: replacing {@code poller::health} with a
* constant {@code () -> LoopWatchdog.State.RUNNING} at the call site compiled clean and left
* the full suite green, meaning the daemon could report the {@link StatusPoller} as always
* {@code RUNNING} even while it was actually stalled. That is a false negative on the exact
* signal this ticket exists to surface, and is the mirror of a false positive muting a real
* monitoring component — worse, because there is no noise for anyone to notice and then
* silence. {@link FleetdLoopHealthSourceWiringTest} calls this factory directly and pins both
* halves separately, plus the {@code reaper == null} branch below.
*
* <p>{@code reaper} may be {@code null} — a {@link SessionReaper} is only constructed when
* {@code lifecycle.idleTtlSeconds} is configured (see the {@code reaper} local above) — and
* this factory preserves the existing behaviour of reporting {@link LoopWatchdog.State#STOPPED}
* in that case, rather than a {@code NullPointerException} on the first {@code fleet_list} or
* {@code /healthz} call.
*/
static FleetMcp.LoopHealthSource loopHealthSource(StatusPoller poller, SessionReaper reaper) {
return new FleetMcp.LoopHealthSource(poller::health,
() -> reaper == null ? LoopWatchdog.State.STOPPED : reaper.health());
}
/**
* fleetd #248: package-private factory for the member worktree/branch lookup {@link
* CompletionResolver} uses to name a fallback report's worktree and branch (fleetd#241).
@@ -1065,6 +1093,157 @@ public final class Fleetd {
.orElse(null);
}
/**
* fleetd #561: compose the production {@link TurnListener} from its two halves — the
* completion resolver (which resolves a blocked {@code fleet_send}'s waiter) and the session
* manager (which drives the member's lifecycle state) — hardened so a throw from either
* half's callback can never suppress the other half's callback for the same event.
*
* <p>Before this, each method below was two bare, unguarded statements: whichever ran first
* throwing meant the second one never ran at all, and nothing beyond call order in the
* source enforced "the completion half goes first". #556 fixed the identical shape for {@code
* onDelivered}'s registration by moving it off the fan-out entirely (see {@code
* TurnRegistrar}); this fixes the four remaining callbacks — {@code onTurnComplete}, {@code
* onTurnCompleteWithPostAction}, and both {@code onTurnFailed} overloads — by hardening the
* fan-out itself instead, since none of them can be pulled out of the listener the way
* registration was.
*
* <p>The invariant this composition guarantees: <b>a throwing session-listener must not
* prevent the completion resolver from being told the turn ended.</b> The completion half is
* always attempted first, and — as a bonus the resolver does not depend on — its own throw
* does not stop the session half from running either. Whatever escapes (from one half or
* both) is rethrown once both have been attempted, with a second failure recorded via {@link
* Throwable#addSuppressed} on the first, so it still reaches {@link StatusPoller}'s {@code
* catch (Throwable)} and logs at ERROR. Nothing here swallows a failure to make the two halves
* look safe.
*
* <p>{@code onTurnCompleteWithPostAction} is the one place order is not just fault-tolerance
* but a functional requirement: {@code completion.resolveBeforePostAction} must resolve the
* scrape before {@code sessions.onTurnCompleteWithPostAction}'s adapter housekeeping can erase
* the pane's rendered output (see {@link CompletionResolver#resolveBeforePostAction}). That is
* why this composition does not treat the pair symmetrically the way {@link #bothMustRun}
* does for the other three callbacks: the mirror case (completion half throws, session half's
* return value still observed) is not preserved here — once the completion half's failure
* escapes, the session half's return value is discarded, matching how the {@link Injector}
* already treats any throw from this callback as "the action did not start" (see the {@code
* started} default at its call site, {@code Injector.java} ~line 616).
*
* <p>Package-private so {@code FleetdTurnListenerCompositionTest} can build this listener
* directly from a real {@link CompletionResolver} and a fake {@link TurnListener} standing in
* for {@code sessions}, without booting the rest of {@code main}.
*/
static TurnListener turnListener(CompletionResolver completion, TurnListener sessions) {
return new TurnListener() {
@Override
public void onTurnComplete(String target) {
bothMustRun(() -> completion.onTurnComplete(target), () -> sessions.onTurnComplete(target));
}
@Override
public boolean hasPostTurnAction(String target) {
return sessions.hasPostTurnAction(target);
}
@Override
public boolean onTurnCompleteWithPostAction(String target) {
return bothMustRunKeepingSecondResult(() -> completion.resolveBeforePostAction(target),
() -> sessions.onTurnCompleteWithPostAction(target));
}
@Override
public void onDelivered(String target, dev.ltms.fleet.msg.TurnToken token) {
// fleetd #561: left unguarded on purpose, not because registration survives a
// throw elsewhere. completion.onDelivered runs CompletionResolver.captureBaseline,
// which already wraps its scrape read in its own catch (RuntimeException) and
// fails open (baseline = null) — so this call does not realistically throw, and
// there is nothing here for bothMustRun to protect.
completion.onDelivered(target, token);
sessions.onDelivered(target, token);
}
@Override
public void onTurnFailed(String target) {
bothMustRun(() -> completion.onTurnFailed(target), () -> sessions.onTurnFailed(target));
}
@Override
public void onTurnFailed(String target, String reason) {
bothMustRun(() -> completion.onTurnFailed(target, reason), () -> sessions.onTurnFailed(target));
}
};
}
/**
* fleetd #561: run two listener-callback halves for one lifecycle event, guaranteeing the
* SECOND always runs even when the FIRST throws. Whatever is thrown is rethrown once both
* halves have been attempted — a second failure is attached to the first via {@link
* Throwable#addSuppressed} rather than dropped. Never swallows.
*/
private static void bothMustRun(Runnable completionHalf, Runnable sessionsHalf) {
Throwable failure = null;
try {
completionHalf.run();
} catch (Throwable t) {
failure = t;
}
try {
sessionsHalf.run();
} catch (Throwable t) {
if (failure == null) {
failure = t;
} else {
failure.addSuppressed(t);
}
}
if (failure != null) {
throwUnchecked(failure);
}
}
/**
* fleetd #561: like {@link #bothMustRun}, but for {@code onTurnCompleteWithPostAction}, whose
* session half returns the value the {@link Injector} needs. The second (session) half's
* result is what this method returns; if the first (completion) half throws, the second half
* still runs and its result is still computed here, but the throw is rethrown afterward
* regardless — so that result is discarded at the {@link Injector} call site exactly as it
* already is today when this callback throws (see {@link #turnListener}'s javadoc).
*/
private static boolean bothMustRunKeepingSecondResult(Runnable completionHalf,
BooleanSupplier sessionsHalf) {
Throwable failure = null;
try {
completionHalf.run();
} catch (Throwable t) {
failure = t;
}
boolean result = false;
try {
result = sessionsHalf.getAsBoolean();
} catch (Throwable t) {
if (failure == null) {
failure = t;
} else {
failure.addSuppressed(t);
}
}
if (failure != null) {
throwUnchecked(failure);
}
return result;
}
/**
* fleetd #561: rethrow a captured {@link Throwable} without a checked-exception wrapper. The
* two callback halves above never declare a checked exception (both existing production
* halves — {@code CompletionResolver} and {@code SessionManager} — only ever throw unchecked),
* so this only ever actually rethrows a {@link RuntimeException} or {@link Error}; the generic
* cast is the standard "sneaky throw" idiom, not a claim that a checked exception is expected.
*/
@SuppressWarnings("unchecked")
private static <T extends Throwable> void throwUnchecked(Throwable t) throws T {
throw (T) t;
}
/**
* fleetd #480: construct the {@link LeadRollover} executor only when {@code leadRollover:} is
* present at startup — the same presence gate {@code leadHeartbeat:} uses just above this
@@ -221,7 +221,7 @@ public final class CallerResolver {
// The config/live binding names this pane as an architect slot's own. Same
// unforgeable pane mapping; the live binding, never a request argument, decides.
// Check the slot role too: this defence in depth prevents a bad lifecycle bind from
// escalating a dev or reviewer into an architect. Checked before the worker fallback.
// escalating a dev, hunter, or reviewer into an architect. Checked before the worker fallback.
return Principal.architect(memberSlotNames.apply(slot), c.terminal(), c.pid());
}
return Principal.worker(c.terminal(), c.pid()); // unforgeable; never token-gated
@@ -42,7 +42,7 @@ public interface MemberLifecycle {
* Try to bind a newly spawned {@code terminal} into the role it was granted.
*
* @return the role this session actually holds: {@code role} unchanged for a role with no
* live slot-binding semantics (dev, reviewer), or when the bind succeeded; a fallback
* live slot-binding semantics (dev, hunter, reviewer), or when the bind succeeded; a fallback
* role — never {@code role} — when a slot-bound role (architect) could not be bound.
* Callers must record THIS value on the session, never the requested {@code role}, so
* a later roster read never reports a role the session does not hold (CB-619). In
@@ -20,7 +20,8 @@ import java.util.function.Supplier;
*
* <p>Two halves, split by who owns each:
* <ul>
* <li><b>slots</b> — read from {@code fleet.architects}/{@code developers}/{@code reviewers}
* <li><b>slots</b> — read from {@code fleet.architects}/{@code developers}/{@code hunters}/
* {@code reviewers}
* (see {@link #slots()}), each carrying the {@code profile} reference the spawn lifecycle
* reads when it stands the slot up. <strong>Live, since fleetd #424</strong>: {@link #live}
* re-reads {@code fleet:} on every call, through a supplier the same shape as
@@ -322,7 +323,7 @@ public final class MemberRegistry implements MemberLifecycle {
* CB-619 / fleetd #123: refuse an architect acquire before anything spawns when no configured
* slot carries {@code profile} — the config-gap case from the original defect report (a spawn
* asked for {@code role=architect, profile=sonnet}, and {@code fleet.architects} carried only
* {@code opus} and {@code sol}). A dev/reviewer acquire is always a no-op: those pools are
* {@code opus} and {@code sol}). A dev/hunter/reviewer acquire is always a no-op: those pools are
* placement candidates only (see {@code CompositePeerLauncher}), never a live identity binding,
* so there is nothing here to refuse — an explicit profile outside the pool for those roles is a
* documented operator override, not a defect.
@@ -31,7 +31,7 @@ import java.util.function.Supplier;
* {@code placement:}, and an existing profile's {@code weight} / {@code maxLoad}. Both are
* read through a supplier on {@code CompositePeerLauncher}, which is what makes them hot —
* not the fact that they are config. Most of {@code fleet:} — every role pool
* ({@code architects}/{@code developers}/{@code reviewers}), {@code charters}, and
* ({@code architects}/{@code developers}/{@code hunters}/{@code reviewers}), {@code charters}, and
* {@code tabLabel} — is read the same live way, through the same supplier
* ({@code () -> config.get().fleet()}). {@code architects} in particular is hot for
* <strong>two independent consumers</strong> (fleetd #424): {@code CompositePeerLauncher}
@@ -208,7 +208,7 @@ import java.util.function.Supplier;
* that is correctly hot and for a key nobody triaged. Three times now — {@code worktreeGroup} (#323),
* {@code primary}/{@code configReload} (#326), and {@code fleet.leaders} sitting in the escape hatch
* (#333) — the second kind hid among the first. A top-level coverage checker in the
* {@link ConfigRefProfileCoverageTest} shape (one level up, over {@code FleetConfig} itself rather
* {@code ConfigRefProfileCoverageTest} shape (one level up, over {@code FleetConfig} itself rather
* than {@code FleetConfig.Profile}) proves this file's four classes exhaust the record's components
* — see {@code ConfigRefTopLevelCoverageTest}. That test proves the record's <em>shape</em> is fully
* triaged; it does NOT prove a {@code SPLIT_KEYS}/{@code COLD_KEYS}/{@code DEFERRED_KEYS} member has
@@ -605,7 +605,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
+ "opened once and needs a restart; the broker URI env-var name kept out of a "
+ "member's environment is read live on every spawn and already applied");
}
// fleetd #333: unlike health/coordinator above, most of `fleet:` (developers, reviewers,
// fleetd #333: unlike health/coordinator above, most of `fleet:` (developers, hunters, reviewers,
// charters, tabLabel) is genuinely hot — ConfigRefTest.aHotChangeIsAppliedAndRead-
// ThroughGet and aCharterChangeIsHotAndReachesTheLiveConfig prove it reaches the live config
// with no restart note. `architects` is hot too, and — since fleetd #424 — hot for BOTH of
@@ -630,7 +630,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
+ "identity map and to auto-launch leads, and neither is rebuilt on reload, so a "
+ "lead added, removed, or given a new tab: label needs a restart — until then it "
+ "stays unrecognised, and a caller from its new tab resolves as a worker, not a "
+ "lead; the rest of fleet: (developers, reviewers, charters, tabLabel) is read "
+ "lead; the rest of fleet: (developers, hunters, reviewers, charters, tabLabel) is read "
+ "live through the supplier on CompositePeerLauncher, and architects is read "
+ "live through that same supplier for placement AND through a separate supplier "
+ "on MemberRegistry for spawn-time identity — both already applied");
@@ -62,7 +62,7 @@ import java.util.regex.PatternSyntaxException;
* @param fleet who the daemon may run and under which role (CB-557). One block replacing
* the former {@code leaders:}, {@code members:}, {@code leadScan:} and
* {@code defaultProfile:}. Role is the containing key — {@code leaders},
* {@code architects}, {@code developers}, {@code reviewers} — and each entry
* {@code architects}, {@code developers}, {@code hunters}, {@code reviewers} — and each entry
* names the {@code profiles:} backend it runs on. See {@link Fleet}
* @param leadHeartbeat opt-in idle-lead heartbeat (CB-551); {@code null} ⇒ off, and an upgraded
* daemon never nudges an idle lead on its own initiative
@@ -1200,6 +1200,7 @@ public record FleetConfig(
* @param leaders panes that orchestrate rather than are orchestrated, keyed by lead name
* @param architects profiles the {@code architect} role may run on
* @param developers profiles the {@code dev} role may run on
* @param hunters profiles the {@code hunter} role may run on
* @param reviewers profiles the {@code reviewer} role may run on
* @param charters optional launch-charter text keyed by singular role wire name
* @param tabLabel template for a member tab's label; {@code {role}}, {@code {profile}},
@@ -1210,6 +1211,7 @@ public record FleetConfig(
public record Fleet(Map<String, Leader> leaders,
Map<String, Slot> architects,
Map<String, Slot> developers,
Map<String, Slot> hunters,
Map<String, Slot> reviewers,
Map<String, String> charters,
String tabLabel) {
@@ -1226,6 +1228,7 @@ public record FleetConfig(
leaders = unmodifiableOrEmpty(leaders);
architects = unmodifiableOrEmpty(architects);
developers = unmodifiableOrEmpty(developers);
hunters = unmodifiableOrEmpty(hunters);
reviewers = unmodifiableOrEmpty(reviewers);
charters = unmodifiableOrEmpty(charters);
tabLabel = (tabLabel == null || tabLabel.isBlank()) ? DEFAULT_TAB_LABEL : tabLabel;
@@ -1240,9 +1243,15 @@ public record FleetConfig(
* constructor: the launcher reads {@code fleet.charters()} from the live config. Jackson
* binds the canonical constructor, so this one cannot swallow an operator's YAML.
*/
public Fleet(Map<String, Leader> leaders, Map<String, Slot> architects,
Map<String, Slot> developers, Map<String, Slot> reviewers,
Map<String, String> charters, String tabLabel) {
this(leaders, architects, developers, null, reviewers, charters, tabLabel);
}
public Fleet(Map<String, Leader> leaders, Map<String, Slot> architects,
Map<String, Slot> developers, Map<String, Slot> reviewers, String tabLabel) {
this(leaders, architects, developers, reviewers, null, tabLabel);
this(leaders, architects, developers, null, reviewers, null, tabLabel);
}
/**
@@ -1264,6 +1273,7 @@ public record FleetConfig(
return switch (role) {
case ARCHITECT -> architects;
case DEV -> developers;
case HUNTER -> hunters;
case REVIEWER -> reviewers;
};
}
@@ -1670,7 +1680,7 @@ public record FleetConfig(
* <p>A herdr pane runs a login shell that re-sources the operator's own secret store, so a
* member inherits every credential the operator's shell holds — measured at 31 names on this
* host, of which only one ({@code GITEA_ACCESS_TOKEN}) used to be blocked, and that block was a
* single name hardcoded in {@link HerdrPeerLauncher} rather than driven by config (gitea issue
* single name hardcoded in {@link dev.ltms.fleet.member.HerdrPeerLauncher} rather than driven by config (gitea issue
* #82). This record replaces that hardcoded shadow with a config-driven one.
*
* <p><b>deny-by-default, not a deny-list.</b> A deny-list (block these specific names, let
@@ -1872,7 +1882,7 @@ public record FleetConfig(
/** The {@code fleet:} child blocks whose direct children are slot names. */
private static final Set<String> FLEET_POOL_KEYS =
Set.of("leaders", "architects", "developers", "reviewers");
Set.of("leaders", "architects", "developers", "hunters", "reviewers");
/**
* Reject a {@code fleet:} role pool whose slot names repeat (CB-548, re-homed by CB-557).
@@ -1882,7 +1892,7 @@ public record FleetConfig(
* daemon would never know. Jackson's YAML parser does not fail on duplicate mapping keys by
* default, so duplicates are caught here, at parse time, before the map is built.
*
* <p>Only the four pools <em>directly under the top-level {@code fleet:}</em> are considered,
* <p>Only the five pools <em>directly under the top-level {@code fleet:}</em> are considered,
* and only their direct child keys (the slot names). A nested field elsewhere, even one also
* named {@code developers:}, is ignored, so parsing of the rest of the config is unaffected.
*
@@ -2050,7 +2060,7 @@ public record FleetConfig(
+ " and that role's pool supplies the candidate profiles",
"architects", "'fleet.architects'",
"members", "a role pool under 'fleet:' — 'fleet.architects', 'fleet.developers' or"
+ " 'fleet.reviewers'; the role is the containing key, not a 'role:' field",
+ " 'fleet.hunters' or 'fleet.reviewers'; the role is the containing key, not a 'role:' field",
"leaders", "'fleet.leaders'",
"leadScan", "'fleet.leaders.<name>.tabPrefix' and '.scanIntervalSeconds' — lead"
+ " discovery is now configured on the lead it discovers");
@@ -50,7 +50,7 @@ import java.util.regex.Pattern;
* so the scrape's herdr round-trip never stalls the status poller. The captured waiter is read on the
* poller thread (before any next-turn delivery can overwrite it) and passed into the virtual thread.
*/
public final class CompletionResolver implements TurnListener {
public final class CompletionResolver implements TurnListener, TurnRegistrar {
private static final Logger log = LoggerFactory.getLogger(CompletionResolver.class);
@@ -235,6 +235,26 @@ public final class CompletionResolver implements TurnListener {
this.worktreeBranches = Objects.requireNonNull(worktreeBranches, "worktreeBranches");
}
/**
* fleetd #556: {@link TurnRegistrar}'s structural half of what {@link #captureBaseline} used to
* do alone — record the waiter, no I/O. Called directly and unconditionally by the {@link
* Injector} as part of delivering a turn, before {@link #onDelivered} ever runs, so this
* registration cannot be skipped by a {@link TurnListener} throwing (from {@code onDelivered} or
* any other callback). {@link #captureBaseline} still performs this exact check-and-put itself
* as well — harmless and idempotent when it runs right after this — so a caller that only wires
* the {@link TurnListener} path (e.g. an existing test that never mentions {@link TurnRegistrar})
* keeps working unchanged.
*/
@Override
public void register(String target, TurnToken token) {
CompletableFuture<Rendezvous.Resolution> waiter = token.waiter();
if (waiter == null) {
inFlight.remove(target); // no send is waiting on this delivery — nothing to resolve later
return;
}
inFlight.put(target, new InFlight(waiter, null, nowNanos.getAsLong(), token.injectedText()));
}
@Override
public void onDelivered(String target, TurnToken token) {
// Capture the exact waiter this turn belongs to (CB-116) and snapshot the pane's pre-turn
@@ -244,7 +264,14 @@ public final class CompletionResolver implements TurnListener {
captureBaseline(target, token);
}
/** Capture the in-flight turn: its waiter and pre-turn baseline (the testable core of {@link #onDelivered}). */
/**
* Capture the in-flight turn: its waiter and pre-turn baseline (the testable core of {@link
* #onDelivered}). fleetd #556: this still does its own full waiter check and {@code inFlight.put}
* — the same registration {@link #register} performs — so it keeps working standalone (as every
* test calling it directly already does) even though the {@link Injector} now also calls {@link
* #register} on its own, earlier and unconditionally. The two writes are idempotent with each
* other; only the second (this one) carries a real baseline, since only this one pays for a scrape.
*/
void captureBaseline(String target, TurnToken token) {
CompletableFuture<Rendezvous.Resolution> waiter = token.waiter();
if (waiter == null) {
@@ -2,6 +2,7 @@ package dev.ltms.fleet.inject;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.herdr.HerdrRouter;
import dev.ltms.fleet.msg.TurnToken;
import org.slf4j.Logger;
@@ -11,6 +12,7 @@ import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
@@ -45,6 +47,17 @@ import java.util.stream.Collectors;
* turn — the pickup-grace path (a turn too fast to sample) unwedges the queue but does not fire
* completion, since without a sampled {@code working} there is no trustworthy "the worker just
* finished the task" signal to act on.
*
* <p><strong>Delivery honesty (fleetd #551).</strong> The one external send call
* ({@link AgentControl#send}) is irreversible, and its own response can still fail after the text
* has already reached the worker's pane — herdr replied (or may have), so the request was already
* processed, but the reply itself then failed to parse or carried an error. A queued entry is
* polled off the queue and marked {@code ATTEMPTED} <em>before</em> that call is made, not after,
* so a failure from the call itself can never be recorded as a confident {@code NOT_DELIVERED} for
* text that may already be sitting in the pane. {@code NOT_DELIVERED} stays reserved for the cases
* where nothing was ever attempted — the readiness grace expiring, {@link #drop}, or a herdr
* {@code *_not_found} error, which the rest of this codebase already treats as a confirmed absence
* rather than a merely inconclusive failure (see {@code StatusPoller}, {@code AgentControl}).
*/
public final class Injector {
@@ -93,6 +106,11 @@ public final class Injector {
private final AgentControl agents;
private final HerdrRouter router;
private final TurnListener turnListener;
/**
* fleetd #556: the Injector's own registration invariant, called directly and unconditionally —
* never folded into {@link #turnListener}'s fan-out. See {@link TurnRegistrar}.
*/
private final TurnRegistrar registrar;
private final Predicate<String> ready; // CB-113: a target is deliverable only when available
private final Consumer<String> forget; // CB-114: clear a gone worker's readiness/presence
/**
@@ -134,15 +152,43 @@ public final class Injector {
*/
public Injector(AgentControl agents, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget) {
this(agents, turnListener, ready, forget, System::currentTimeMillis);
// fleetd #556: no explicit registrar named here, so fall back to turnListener itself when it
// happens to also implement TurnRegistrar (true for CompletionResolver, the only production
// TurnListener that needs CB-106 registration) — every existing caller of this overload keeps
// working unchanged. A turnListener that does NOT implement TurnRegistrar (a fan-out object,
// or a bare test lambda) gets TurnRegistrar.NOOP here, same as before this ticket.
this(agents, turnListener, ready, forget,
turnListener instanceof TurnRegistrar r ? r : TurnRegistrar.NOOP);
}
/** Full constructor — for tests: an injectable wall-clock supplier (fleetd #501). */
/**
* fleetd #556: explicit-registrar overload. Use this whenever {@code turnListener} does not
* itself implement {@link TurnRegistrar} — e.g. a fan-out object that also notifies unrelated
* observers — so registration is wired directly to the one component that owns it, rather than
* relying on {@code turnListener} happening to implement both interfaces.
*/
public Injector(AgentControl agents, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget, TurnRegistrar registrar) {
this(agents, turnListener, ready, forget, registrar, System::currentTimeMillis);
}
/**
* Package-private constructor — for tests: an injectable wall-clock supplier (fleetd #501),
* with no explicit registrar (fleetd #556 fallback, same as the public 4-arg overload above).
*/
Injector(AgentControl agents, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget, LongSupplier nowMillis) {
this(agents, turnListener, ready, forget,
turnListener instanceof TurnRegistrar r ? r : TurnRegistrar.NOOP, nowMillis);
}
/** Full constructor — for tests: an explicit registrar and an injectable wall-clock supplier. */
Injector(AgentControl agents, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget, TurnRegistrar registrar, LongSupplier nowMillis) {
this.agents = agents;
this.router = null;
this.turnListener = turnListener;
this.registrar = Objects.requireNonNull(registrar, "registrar");
this.ready = ready;
this.forget = forget;
this.nowMillis = nowMillis;
@@ -150,15 +196,28 @@ public final class Injector {
public Injector(HerdrRouter router, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget) {
this(router, turnListener, ready, forget, System::currentTimeMillis);
// fleetd #556: see the AgentControl overload above for the fallback rationale.
this(router, turnListener, ready, forget,
turnListener instanceof TurnRegistrar r ? r : TurnRegistrar.NOOP);
}
/**
* fleetd #556: explicit-registrar overload — production wiring ({@code Fleetd}) uses this, since
* its {@code turnListener} is a fan-out object that also notifies {@code SessionManager} and does
* not itself implement {@link TurnRegistrar}.
*/
public Injector(HerdrRouter router, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget, TurnRegistrar registrar) {
this(router, turnListener, ready, forget, registrar, System::currentTimeMillis);
}
/** Full constructor — for tests: an injectable wall-clock supplier (fleetd #501). */
Injector(HerdrRouter router, TurnListener turnListener, Predicate<String> ready,
Consumer<String> forget, LongSupplier nowMillis) {
Consumer<String> forget, TurnRegistrar registrar, LongSupplier nowMillis) {
this.agents = null;
this.router = router;
this.turnListener = turnListener;
this.registrar = Objects.requireNonNull(registrar, "registrar");
this.ready = ready;
this.forget = forget;
this.nowMillis = nowMillis;
@@ -170,8 +229,23 @@ public final class Injector {
/** The result of trying to remove an undelivered message from the injector. */
public enum Cancellation {
/** The message was still queued and this call removed it; the target saw nothing. */
CANCELLED,
/** The message reached the worker's pane and is confirmed delivered. */
DELIVERED,
/**
* fleetd #551: the injector called {@link AgentControl#send} for this message and that call
* threw before its outcome was known — the text may or may not have reached the pane. A
* caller reporting this to an operator must say "uncertain", not "definitely not
* delivered": reading it as a confident negative invites a resend of text that may already
* be sitting in the pane (a double delivery), which is worse than the ambiguity itself.
*/
ATTEMPTED,
/**
* The message never reached the worker's pane — nothing was ever attempted for it (the
* readiness grace expired, the target was dropped, or send failed with a herdr
* {@code *_not_found} error, which is a confirmed absence, not merely inconclusive).
*/
NOT_DELIVERED
}
@@ -194,7 +268,29 @@ public final class Injector {
/** A pending message and the future that completes when it has been delivered. */
private static final class Pending {
enum State { QUEUED, DELIVERED, NOT_DELIVERED, CANCELLED }
enum State {
/** Still in the target's queue, not yet attempted. */
QUEUED,
/**
* fleetd #551: {@link AgentControl#send} has been called for this entry and its outcome
* is not yet known — recorded BEFORE the call (see {@link #onStatus}), so a Throwable
* from send() can never leave the entry either still QUEUED or falsely marked
* NOT_DELIVERED. Upgraded to DELIVERED on success; left as ATTEMPTED on an ordinary
* failure, since reaching the catch does not prove the text never reached the pane.
*/
ATTEMPTED,
/** {@code send()} returned normally: the text is confirmed to have reached the pane. */
DELIVERED,
/**
* Confirmed — not merely inconclusive — that nothing was ever sent for this entry: the
* worker never became ready ({@link #onStatus}'s readiness-grace expiry), the target was
* dropped ({@link #drop}), or send failed with a herdr {@code *_not_found} error. Never
* written for an entry whose send() outcome is unknown; see ATTEMPTED.
*/
NOT_DELIVERED,
/** Removed from the queue by {@link #cancel} before it was ever attempted. */
CANCELLED
}
final String target;
final String text;
@@ -286,7 +382,14 @@ public final class Injector {
}
private static Cancellation cancellationOf(Pending p) {
return p.state == Pending.State.DELIVERED ? Cancellation.DELIVERED : Cancellation.NOT_DELIVERED;
// fleetd #551 (comment 17037): a two-way split on a now-three-way question folded ATTEMPTED
// into NOT_DELIVERED with no compiler error and no failing test — the exact defect this
// ticket exists to fix, one layer up. ATTEMPTED gets its own answer instead.
return switch (p.state) {
case DELIVERED -> Cancellation.DELIVERED;
case ATTEMPTED -> Cancellation.ATTEMPTED;
case QUEUED, NOT_DELIVERED, CANCELLED -> Cancellation.NOT_DELIVERED;
};
}
private static boolean isQuiescent(Target t) {
@@ -306,7 +409,7 @@ public final class Injector {
if (t == null) return;
Pending sent = null;
RuntimeException sendError = null;
Throwable sendError = null;
boolean turnCompleted = false;
boolean turnFailed = false;
boolean resubmit = false;
@@ -379,20 +482,43 @@ public final class Injector {
if (p != null && ready.test(target)) {
t.notReadySincePoll = 0;
t.notReadySinceMillis = 0;
// fleetd #551: poll and record BEFORE the irreversible send, not after.
// The entry comes off the queue and its state is set to ATTEMPTED here,
// unconditionally — so a Throwable escaping the send call below (caught or
// not) can never leave the entry QUEUED at the head of t.queue (the fleetd
// #546 hazard, since peek() alone would let the next onStatus round re-enter
// this block and send the same text again), and no path can write a
// confident DELIVERED or NOT_DELIVERED before we actually know which one
// happened.
t.queue.poll();
p.state = Pending.State.ATTEMPTED;
try {
agentsFor(target).send(target, p.text());
t.queue.poll();
p.state = Pending.State.DELIVERED;
t.awaitingPickup = true;
t.awaitingCompletion = true;
t.turnObserved = false;
t.injectableSincePickup = 0;
sent = p;
} catch (RuntimeException e) {
// Delivery failed at herdr; drop the poisoned message and surface it
// rather than blocking the queue behind it.
t.queue.poll();
p.state = Pending.State.NOT_DELIVERED;
} catch (Throwable e) {
// fleetd #551: leave p.state == ATTEMPTED (recorded above, before the
// call) rather than downgrading it to NOT_DELIVERED here — reaching this
// catch does not prove the text never reached the pane. Three of the
// four HerdrException throw sites in HerdrCodec fire only after herdr
// has already replied (so it processed the request), and the fourth (a
// transport IOException) leaves it genuinely unknown whether herdr even
// received the bytes — see #551 comment 16867. The one exception is a
// herdr `*_not_found` error: that family is already read as "definitely
// absent, not merely inconclusive" everywhere else in this codebase
// (StatusPoller, AgentControl's own retry, WorkspaceControl,
// HerdrPeerLauncher, FleetApp, ReplyPushLoop) because it means the
// target pane/agent does not exist at all, so nothing could have been
// pasted anywhere — #551 keeps the new state consistent with that
// existing vocabulary rather than inventing a second one.
if (e instanceof HerdrException he && he.code() != null
&& he.code().endsWith("_not_found")) {
p.state = Pending.State.NOT_DELIVERED;
}
sent = p;
sendError = e;
}
@@ -489,54 +615,182 @@ public final class Injector {
// Fire listeners / herdr calls after releasing the monitor so nothing runs on the poller
// thread while it holds the target lock.
if (resubmit) {
try {
agentsFor(target).submit(target); // nudge a raced Enter so the pending paste submits
} catch (RuntimeException e) {
log.debug("resubmit to {} failed (will retry next poll): {}", target, e.getMessage());
//
// fleetd #553: wrapped in try/finally. By this point, if `sent != null`, the delivery has
// already happened inside the monitor above — off the queue, p.state == DELIVERED, text
// typed into the target's pane — so `sent`'s future MUST be completed one way or another,
// in every path out of this region, or the caller (a blocking fleet_send, or an async
// ticket) waits forever on a message it actually received. But the finally must not
// swallow whatever escaped: a listener that throws is a defect in THAT listener, and it
// must still reach StatusPoller's catch (Throwable) so log.error fires — converting a
// loud listener bug into a silently orphaned future would be worse than the bug itself.
//
// There are TWO futures at stake here, not one: `sent.delivered()` (the delivery future)
// and `sent.token().waiter()` (the rendezvous waiter a blocking fleet_send actually waits
// on for the worker's ANSWER). fleetd #556: the waiter is registered by `registrar.register`
// now — called directly and unconditionally, structural rather than delegated to a listener
// callback — never by turnListener.onDelivered() alone (before this ticket, that WAS the
// only path, and any TurnListener wired ahead of the one that mattered could throw and skip
// it; #553 could only make the reachable listener behave, not remove that dependency).
// Completing `sent.delivered()` without also calling `registrar.register` leaves the waiter
// unregistered forever — a hang with a success receipt, which is worse than the plain hang
// #553 was about. So the `finally` below does not just complete the future: on the path
// where the `if (sent != null)` block never ran, it does that block's whole job —
// register, then onDelivered (notification only, may throw), then complete.
//
// `sentHandled` is NOT allowed to move earlier than this: an earlier comment on #553
// proposed running `if (sent != null)` first, before turnCompleted/turnFailed, and that
// was withdrawn — `registrar.register` WRITES CompletionResolver's inFlight record for this
// turn, onTurnComplete READS it for the PREVIOUS turn, and running register first makes
// onTurnComplete resolve the NEW turn's waiter with the PREVIOUS turn's stale output
// (CB-116). The `if (sent != null)` block stays last; the `finally` is a backstop for it,
// not a replacement. fleetd #556 keeps this ordering unchanged — it only splits what used
// to be one listener call (register + notify) into two calls in the same position.
boolean sentHandled = false;
try {
if (resubmit) {
try {
agentsFor(target).submit(target); // nudge a raced Enter so the pending paste submits
} catch (Throwable e) {
// fleetd #553: widened from RuntimeException (same reasoning as #549 at :382) —
// this is the FIRST block after the monitor, so an Error escaping it used to skip
// every block below it, including the `sent` completion.
log.debug("resubmit to {} failed (will retry next poll): {}", target, e.getMessage());
}
}
}
if (notReady != null) {
// Worker never became available: forget its (never-set) readiness, unblock every queued
// caller, and route the awaiting send through the same failure path as a stalled turn so
// a blocking or async waiter resolves WORKER_FAILED rather than riding out the timeout.
forget.accept(target);
RuntimeException cause = new IllegalStateException(
target + " never became available (no bridge MCP connection within the boot window)");
for (Pending p : notReady) {
p.delivered().completeExceptionally(cause);
if (notReady != null) {
// Worker never became available: unblock every queued caller FIRST — these messages'
// fate (NOT_DELIVERED, off the queue) was already decided inside the monitor above, so
// fleetd #553 completes every one of them before calling forget.accept or
// turnListener.onTurnFailed below. Either of those is a listener/consumer callback and
// can throw (an ordinary RuntimeException is enough — the same reasoning as the rest of
// this ticket): completing the futures first means such a throw can no longer leave any
// of them permanently pending, regardless of which one throws or in which order.
RuntimeException cause = new IllegalStateException(
target + " never became available (no bridge MCP connection within the boot window)");
for (Pending p : notReady) {
p.delivered().completeExceptionally(cause);
}
// route the awaiting send through the same failure path as a stalled turn so a blocking
// or async waiter resolves WORKER_FAILED rather than riding out the timeout.
forget.accept(target);
turnListener.onTurnFailed(target);
}
turnListener.onTurnFailed(target);
}
if (turnCompleted) {
if (startPostTurn) {
boolean started = turnListener.onTurnCompleteWithPostAction(target);
synchronized (t) {
t.postTurnPending = false;
if (started) {
t.awaitingPostTurnPickup = true;
t.injectableSincePostTurnPickup = 0;
if (turnCompleted) {
if (startPostTurn) {
// fleetd #553: the listener call is wrapped so `t.postTurnPending` (set true inside
// the monitor above, before this call) is always reset. Before this wrapping, a
// RuntimeException from onTurnCompleteWithPostAction skipped the reset below,
// permanently wedging the target: postTurnPending stayed true forever, so this
// target's delivery guard (:376) would never again pass and no further message to it
// would ever be delivered — a target-level lockup, not just one skipped future.
// `started` defaults to false so an exception is treated as "the post-turn action did
// not start" rather than falsely arming the post-turn pickup latch for an action that
// never ran.
boolean started = false;
try {
started = turnListener.onTurnCompleteWithPostAction(target);
} finally {
synchronized (t) {
t.postTurnPending = false;
if (started) {
t.awaitingPostTurnPickup = true;
t.injectableSincePostTurnPickup = 0;
}
if (t.queue.isEmpty() && !t.awaitingPostTurnPickup) {
targets.remove(target, t);
}
}
}
if (t.queue.isEmpty() && !t.awaitingPostTurnPickup) {
targets.remove(target, t);
} else {
turnListener.onTurnComplete(target);
}
}
if (turnFailed) {
turnListener.onTurnFailed(target);
}
if (sent != null) {
// fleetd #553: record the INTENT to handle before any side effect, not the fact of
// having completed it. If register()/onDelivered() throws part way through — e.g.
// after CompletionResolver's captureBaseline has already done its inFlight.put — a
// flag set only after this block would still read false, and the `finally` below
// would re-run this block's job a SECOND time. A second captureBaseline runs later,
// once onTurnComplete has already thrown and possibly scraped the pane, and can
// snapshot a pane that already absorbed this turn's output — which means the pane's
// tail never differs from that baseline again and CompletionResolver.resolve's own
// suppression at :364-369 (`return` keeping the in-flight record) drops every future
// completion for this turn, permanently. So this flag must go true FIRST.
sentHandled = true;
if (sendError != null) {
log.warn("inject to {} failed, dropped message: {}", target, sendError.getMessage());
sent.delivered().completeExceptionally(sendError);
} else {
// fleetd #556: register the waiter FIRST and unconditionally — this is the
// Injector's own invariant, kept structural rather than delegated. Even if
// turnListener.onDelivered below throws (a bug in an unrelated observer, e.g.
// SessionManager's bookkeeping, or a fan-out wired in a different order), the
// waiter is already registered and resolvable — a throwing TurnListener can no
// longer take this invariant down with it.
registrar.register(target, sent.token());
// Notification only, from here down: baseline the pane's pre-turn content so a
// misattributed completion (no new output) can't resolve this send with the
// previous turn's stale answer (CB-115). Allowed to throw and to fail (the herdr
// scrape inside already fails open with baseline == null on its own).
turnListener.onDelivered(target, sent.token());
sent.delivered().complete(null);
}
}
} finally {
// Backstop (fleetd #553, extended #556): if an earlier block in this try threw before the
// `if (sent != null)` block above ran, `sentHandled` is still false here, and this does
// that block's WHOLE job — not just the future completion. Skipping registrar.register()
// here would leave sent.token().waiter() never registered with CompletionResolver, so a
// blocking fleet_send would be told its message was delivered and then wait out its full
// timeout for an answer that can never resolve — worse than the plain hang, because it now
// looks like success. This block runs only when sendError == null: nothing was delivered
// on the error path, so there is nothing to register.
//
// fleetd #553 (lead review, comment 16916): `sentHandled` guards ONLY the register/notify
// re-call below, never the future completion outside this inner try. One boolean cannot
// carry both meanings — "the turn was handled" and "the future has been dealt with" —
// because they come apart exactly when onDelivered throws PART WAY THROUGH: sentHandled
// is already true (set before the call, correctly — see the `if (sent != null)` block
// above), so a guard on the outer `if` here would skip this whole recovery, including the
// completion, and leave sent.delivered() pending forever even though the message really
// was typed into the pane and taken off the queue. So `sent != null` alone gates whether
// this target has anything to finish; `!sentHandled` gates only the register/notify
// re-call inside. CompletableFuture.complete/completeExceptionally are idempotent — on the
// ordinary path (sentHandled == true, no throw) the `if (sent != null)` block above has
// already completed this future, so the calls below are a no-op returning false.
//
// This recovery is wrapped in its own try/catch(Throwable) that swallows only ITS OWN
// throwable and logs at WARN — the future is still completed either way — while the
// ORIGINAL throwable from the try block above is left alone to keep unwinding out of
// this method to StatusPoller's catch (Throwable), so a listener bug stays loud.
if (sent != null) {
try {
if (!sentHandled && sendError == null) {
// fleetd #556: register first here too, same as the ordinary path above — this
// recovery path is reached only when an EARLIER block (resubmit/turnCompleted/
// turnFailed) threw before the ordinary path ever ran, so nothing has
// registered this turn yet.
registrar.register(target, sent.token());
turnListener.onDelivered(target, sent.token());
}
} catch (Throwable recoveryError) {
log.warn("fleetd #553 backstop: onDelivered failed for {} while recovering from "
+ "an earlier listener failure; completing its delivery future "
+ "anyway: {}",
target, recoveryError.getMessage());
} finally {
// No-op (returns false) on the ordinary path, where the `if (sent != null)` block
// above already completed this future — see the comment above this block.
if (sendError != null) {
sent.delivered().completeExceptionally(sendError);
} else {
sent.delivered().complete(null);
}
}
} else {
turnListener.onTurnComplete(target);
}
}
if (turnFailed) {
turnListener.onTurnFailed(target);
}
if (sent != null) {
if (sendError != null) {
log.warn("inject to {} failed, dropped message: {}", target, sendError.getMessage());
sent.delivered().completeExceptionally(sendError);
} else {
// Baseline the pane's pre-turn content so a misattributed completion (no new output)
// can't resolve this send with the previous turn's stale answer (CB-115).
turnListener.onDelivered(target, sent.token());
sent.delivered().complete(null);
}
}
}
@@ -0,0 +1,91 @@
package dev.ltms.fleet.inject;
import java.util.function.LongSupplier;
/**
* A progress watchdog for a singleton background loop (fleetd #544): {@link StatusPoller} and
* {@code dev.ltms.fleet.session.SessionReaper} each hold one. It tracks the monotonic timestamp
* of the loop's last <em>completed</em> round and classifies health from that timestamp alone —
* never from thread liveness. That choice is deliberate: measured on {@code main} at
* {@code 7611b69}, {@code UnixSocketHerdrClient.call()} reads with no deadline anywhere in
* {@code herdr/}, so a herdrd that accepts a connection and never answers parks the calling
* loop's thread forever — the thread stays alive and {@code Thread.isAlive()} keeps reporting
* {@code true} the whole time. A last-round timestamp that stops advancing catches that parked
* case exactly the same way it catches a thread that died outright: either way, nothing marks a
* new round complete.
*
* <p>Lives in {@code inject} (rather than {@code health}, which would be the more obvious home
* for a health-reporting fact) because {@code health} already depends on {@code session}
* ({@code HealthSnapshot}/{@code FleetHealthMonitor} read {@code MemberSession}), and
* {@code session} already depends on {@code inject} ({@code SessionManager} implements
* {@code TurnListener} and extends {@code MemberPresence}). Putting this class in {@code health}
* would have made {@code SessionReaper} (in {@code session}) depend back on {@code health},
* closing a cycle through {@code health -> session -> health} — {@code PackageCyclesTest} catches
* exactly this. {@code inject} has no dependency on {@code health}, so {@code session} depending
* on {@code inject} for this stays a one-way edge, same direction it already depends in.
*
* <p>Three states, not two (the fleetd #512 shape — one flag carrying two conditions that need
* opposite handling). {@code stop()} also makes the loop go quiet, exactly like a crash or a
* hang does, so a caller must say which one happened: {@link #markStoppedByCaller()} records
* "this halt was on purpose," and {@link #state()} reports {@link State#STOPPED} for it
* regardless of how stale the last round looks — an intentional stop is never an alarm.
*/
public final class LoopWatchdog {
/**
* {@code RUNNING} — the loop is going and its last round finished recently.
* {@code STALLED} — the loop was never stopped on purpose, and its last-completed-round
* timestamp is older than the threshold. Covers both a dead loop and a parked one.
* {@code STOPPED} — {@link #markStoppedByCaller()} was called. Not an alarm.
*/
public enum State { RUNNING, STALLED, STOPPED }
private final LongSupplier nowNanos;
private final long staleAfterNanos;
private volatile long lastRoundNanos;
private volatile boolean stoppedByCaller;
/**
* @param nowNanos a monotonic elapsed-time clock, e.g. {@code System::nanoTime} live, a
* controllable stub in tests.
* @param staleAfterNanos how long a last-completed-round timestamp may age before {@link #state()}
* reports {@link State#STALLED}. Derive this from the loop's own poll
* interval with generous headroom — see the caller's own derivation comment.
*/
public LoopWatchdog(LongSupplier nowNanos, long staleAfterNanos) {
this.nowNanos = nowNanos;
this.staleAfterNanos = staleAfterNanos;
this.lastRoundNanos = nowNanos.getAsLong();
}
/** Call once per completed round, from inside the loop. */
public void recordRoundComplete() {
lastRoundNanos = nowNanos.getAsLong();
}
/**
* Call from {@code start()} (and any future restart): clears a previous stop's mark and
* resets the clock, so the loop gets a fresh grace period before its first round completes
* rather than being judged against a timestamp left over from before it was (re)started.
*/
public void reset() {
stoppedByCaller = false;
lastRoundNanos = nowNanos.getAsLong();
}
/**
* Call from {@code stop()}: marks the halt as intentional, so {@link #state()} reports
* {@link State#STOPPED} rather than {@link State#STALLED} no matter how stale the last round is.
*/
public void markStoppedByCaller() {
stoppedByCaller = true;
}
/** The current fact about this loop, per the three-state contract above. */
public State state() {
if (stoppedByCaller) {
return State.STOPPED;
}
return (nowNanos.getAsLong() - lastRoundNanos >= staleAfterNanos) ? State.STALLED : State.RUNNING;
}
}
@@ -8,6 +8,8 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.function.LongSupplier;
/**
* Drives the {@link Injector} by sampling each active worker's {@code agent_status} and
@@ -22,11 +24,25 @@ public final class StatusPoller {
private static final Logger log = LoggerFactory.getLogger(StatusPoller.class);
/**
* How many multiples of {@code intervalMillis} the last-completed round may age before
* {@link #health()} reports {@link LoopWatchdog.State#STALLED} (fleetd #544). A round this
* loop runs is normally a small fraction of one interval — it only takes longer when a herdr
* call is genuinely wedged (herdr's {@code UnixSocketHerdrClient} has no read deadline, so a
* herdrd that accepts and never answers parks the thread forever) — so a wide multiplier
* avoids false alarms from an ordinarily slow round. At the production interval
* ({@link Injector#POLL_INTERVAL_MILLIS} = 250ms) this puts the threshold at 10s: long enough
* that a transient slow poll never trips it, short enough that a stuck poller is visible well
* before anything else downstream (delivery, `/healthz` consumers) would show a symptom.
*/
private static final long STALE_AFTER_INTERVAL_MULTIPLIER = 40;
private final AgentControl agents;
private final HerdrRouter router;
private final Injector injector;
private final StatusRefiner refiner;
private final long intervalMillis;
private final LoopWatchdog watchdog;
private volatile boolean running;
private Thread thread;
@@ -36,14 +52,26 @@ public final class StatusPoller {
public StatusPoller(AgentControl agents, Injector injector, StatusRefiner refiner,
long intervalMillis) {
this(agents, injector, refiner, intervalMillis, System::nanoTime);
}
/** Full constructor — for tests: an injectable monotonic clock (fleetd #544). */
StatusPoller(AgentControl agents, Injector injector, StatusRefiner refiner,
long intervalMillis, LongSupplier nowNanos) {
this.agents = agents;
this.router = null;
this.injector = injector;
this.refiner = refiner;
this.intervalMillis = intervalMillis;
this.watchdog = new LoopWatchdog(nowNanos, staleAfterNanos(intervalMillis));
}
public StatusPoller(HerdrRouter router, Injector injector, long intervalMillis) {
this(router, injector, intervalMillis, System::nanoTime);
}
/** Full constructor — for tests: an injectable monotonic clock (fleetd #544). */
StatusPoller(HerdrRouter router, Injector injector, long intervalMillis, LongSupplier nowNanos) {
this.agents = null;
this.router = router;
this.injector = injector;
@@ -53,45 +81,70 @@ public final class StatusPoller {
// against the LEAD daemon even though this field points at the member one.
this.refiner = new StatusRefiner(router.memberAgents());
this.intervalMillis = intervalMillis;
this.watchdog = new LoopWatchdog(nowNanos, staleAfterNanos(intervalMillis));
}
private static long staleAfterNanos(long intervalMillis) {
return TimeUnit.MILLISECONDS.toNanos(Math.max(intervalMillis, 1)) * STALE_AFTER_INTERVAL_MULTIPLIER;
}
/**
* This loop's progress fact (fleetd #544): {@code RUNNING}, {@code STALLED} (dead or parked —
* indistinguishable from outside, and never observed on purpose), or {@code STOPPED}
* ({@link #stop()} was called). See {@link LoopWatchdog} for why staleness — not thread
* liveness — is the signal.
*/
public LoopWatchdog.State health() {
return watchdog.state();
}
/** Start the polling loop on a virtual thread. Idempotent. */
public synchronized void start() {
if (running) return;
running = true;
watchdog.reset(); // fleetd #544: a fresh grace period, not last run's stale mark/timestamp.
thread = Thread.ofVirtual().name("status-poller").start(this::loop);
log.info("status poller started (interval {}ms)", intervalMillis);
}
private void loop() {
while (running) {
Set<String> active = injector.activeTargets();
for (String target : active) {
if (!running) return;
try {
// herdr's agent_status can misreport a settled worker as `unknown`; refine it
// against the pane content before it drives delivery/completion (CB-115).
// CB-185: refine THROUGH the same control the raw status came from — a router
// splits lead/member targets across two herdr daemons, and reading a lead's pane
// through the (fixed) member refiner never finds it, wedging that lead at UNKNOWN.
AgentControl control = router != null ? router.agentsFor(target) : agents;
AgentStatus status = refiner.refine(target, control.status(target), control);
injector.onStatus(target, status);
} catch (HerdrException e) {
// The worker's agent is gone — stop trying and unblock its waiters.
if (e.code() != null && e.code().endsWith("_not_found")) {
log.debug("target {} gone; dropping its queue", target);
injector.drop(target, e);
} else {
log.debug("status poll for {} failed (will retry): {}", target, e.getMessage());
try {
while (running) {
Set<String> active = injector.activeTargets();
for (String target : active) {
if (!running) return;
try {
// herdr's agent_status can misreport a settled worker as `unknown`; refine it
// against the pane content before it drives delivery/completion (CB-115).
// CB-185: refine THROUGH the same control the raw status came from — a router
// splits lead/member targets across two herdr daemons, and reading a lead's pane
// through the (fixed) member refiner never finds it, wedging that lead at UNKNOWN.
AgentControl control = router != null ? router.agentsFor(target) : agents;
AgentStatus status = refiner.refine(target, control.status(target), control);
injector.onStatus(target, status);
} catch (HerdrException e) {
// The worker's agent is gone — stop trying and unblock its waiters.
if (e.code() != null && e.code().endsWith("_not_found")) {
log.debug("target {} gone; dropping its queue", target);
injector.drop(target, e);
} else {
log.debug("status poll for {} failed (will retry): {}", target, e.getMessage());
}
} catch (Throwable e) {
log.error("unexpected failure polling {}; skipping this round", target, e);
}
} catch (RuntimeException e) {
// Never let one target's unexpected error (e.g. an odd agent.get shape) kill
// the single poller thread and stall injection for every worker.
log.warn("unexpected error polling {}; skipping this round", target, e);
}
// fleetd #544: a round is "complete" only once every active target has been polled —
// a target parked mid-for-loop (control.status(target) never returning) means this
// line is never reached, so the watchdog goes stale exactly like a dead loop would.
watchdog.recordRoundComplete();
sleep();
}
sleep();
} finally {
if (running) {
log.error("status poller loop exited unexpectedly; it can be restarted");
}
running = false;
}
}
@@ -107,6 +160,7 @@ public final class StatusPoller {
/** Stop the polling loop. Idempotent. */
public synchronized void stop() {
running = false;
watchdog.markStoppedByCaller(); // fleetd #544: this halt is on purpose — health() must say STOPPED, not STALLED.
if (thread != null) thread.interrupt();
}
}
@@ -0,0 +1,29 @@
package dev.ltms.fleet.inject;
import dev.ltms.fleet.msg.TurnToken;
/**
* fleetd #556: the {@link Injector}'s own invariant — every delivered turn has a registered
* waiter — kept structural rather than delegated to a {@link TurnListener} callback that can
* throw. The {@link Injector} calls {@link #register} directly and unconditionally as part of
* delivering a turn, before it ever calls {@code turnListener.onDelivered}. A {@link TurnListener}
* that throws from every callback (a bug in an unrelated observer, e.g. session bookkeeping, or a
* fan-out object wired in the wrong order) can therefore never leave a delivered turn unregistered
* — the shape #553 could not close, since that fix could only make the reachable listener behave,
* never remove the Injector's dependency on a listener behaving at all.
*
* <p>Deliberately narrow: registration only, no I/O, nothing that scrapes a pane or can reasonably
* fail. The CB-115 staleness baseline (a herdr round-trip, allowed to fail) stays a
* {@link TurnListener#onDelivered} concern — fired by the {@link Injector} only after this call has
* already run, so its own failure cannot un-register anything.
*/
@FunctionalInterface
public interface TurnRegistrar {
/** Record {@code token}'s waiter as the turn currently in flight for {@code target}. */
void register(String target, TurnToken token);
/** No-op registrar for callers that don't need CB-106 rendezvous tracking. */
TurnRegistrar NOOP = (_, _) -> {
};
}
@@ -20,6 +20,7 @@ import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.peer.PeerUnreachableException;
import dev.ltms.fleet.placement.BackendOutagePolicy;
import dev.ltms.fleet.placement.BackendQuarantine;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.placement.PlacementException;
import dev.ltms.fleet.session.SessionManager;
import dev.ltms.fleet.session.MemberSession;
@@ -108,6 +109,7 @@ public final class FleetMcp {
private final Metrics metrics; // CB-502: null → auth failures not counted
private final CapacitySource capacity;
private final HealthCoverageSource healthCoverage;
private final LoopHealthSource loopHealth;
private final QuarantineSource quarantine;
/** fleetd #201 Unit 5: SEPARATE from {@link #quarantine} — see {@link OutageSource}'s doc. */
private final OutageSource outage;
@@ -136,6 +138,15 @@ public final class FleetMcp {
/** Coverage is supplied by the health wiring, not inferred from a missing dependency. */
public record HealthCoverageSource(Supplier<String> value) { }
/** Progress states for fleetd's singleton background loops, read by {@code fleet_list} and {@code /healthz}. */
public record LoopHealthSource(Supplier<LoopWatchdog.State> statusPoller,
Supplier<LoopWatchdog.State> sessionReaper) {
/** Inert source for callers that do not wire the background loops. */
public static LoopHealthSource none() {
return new LoopHealthSource(() -> LoopWatchdog.State.STOPPED, () -> LoopWatchdog.State.STOPPED);
}
}
/**
* CB-578 stage B quarantine facts used by {@code fleet_profiles}: a profile → credential id
* lookup, plus the shared {@link BackendQuarantine} to read remaining cooldowns off.
@@ -328,11 +339,22 @@ public final class FleetMcp {
* instead of throwing. See {@link #handover}.
*/
public FleetMcp(MessageService messages, PeerLauncher workers, SessionManager sessions,
ConnectionIdentity identity, MemberPresence presence, PrimaryRegistry primaryRegistry,
CallerResolver callers, AuthorizationMode authorizationMode, Metrics metrics,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, LeadChannel leadChannel, OutageSource outage,
LeadSeatSource leadSeats, List<String> peers, LeadRollover leadRollover) {
ConnectionIdentity identity, MemberPresence presence, PrimaryRegistry primaryRegistry,
CallerResolver callers, AuthorizationMode authorizationMode, Metrics metrics,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, LeadChannel leadChannel, OutageSource outage,
LeadSeatSource leadSeats, List<String> peers, LeadRollover leadRollover) {
this(messages, workers, sessions, identity, presence, primaryRegistry, callers, authorizationMode, metrics,
capacity, healthCoverage, LoopHealthSource.none(), quarantine, leadChannel, outage, leadSeats,
peers, leadRollover);
}
public FleetMcp(MessageService messages, PeerLauncher workers, SessionManager sessions,
ConnectionIdentity identity, MemberPresence presence, PrimaryRegistry primaryRegistry,
CallerResolver callers, AuthorizationMode authorizationMode, Metrics metrics,
CapacitySource capacity, HealthCoverageSource healthCoverage, LoopHealthSource loopHealth,
QuarantineSource quarantine, LeadChannel leadChannel, OutageSource outage,
LeadSeatSource leadSeats, List<String> peers, LeadRollover leadRollover) {
Objects.requireNonNull(callers, "callers");
this.authorizationEnforced = Objects.requireNonNull(authorizationMode, "authorizationMode")
== AuthorizationMode.ENFORCED;
@@ -343,6 +365,7 @@ public final class FleetMcp {
this.outage = Objects.requireNonNull(outage, "outage");
this.leadSeats = Objects.requireNonNull(leadSeats, "leadSeats");
this.healthCoverage = healthCoverage;
this.loopHealth = Objects.requireNonNull(loopHealth, "loopHealth");
this.leadRollover = leadRollover;
McpJsonMapper json = new JacksonMcpJsonMapperSupplier().get();
this.transport = HttpServletStreamableServerTransportProvider.builder()
@@ -474,7 +497,7 @@ public final class FleetMcp {
(exchange, _) -> {
McpSchema.CallToolResult denied = deny(exchange, toolAction("fleet_list", Map.of()), null);
if (denied != null) return denied;
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, outage,
return listFleet(workers, sessions, messages, capacity, healthCoverage, loopHealth, quarantine, outage,
leadSeats, callers.leads(),
callerTerminal(exchange),
new CoordinationSource(leadChannel, peers),
@@ -805,6 +828,12 @@ public final class FleetMcp {
+ "answered (turnId stale)");
case TIMED_OUT_WORKING, TIMED_OUT_QUEUED, BUSY -> text("[no reply within " + timeout + "ms — worker "
+ r.outcome().name().toLowerCase().replace("timed_out_", "") + "; retry or poll status]");
// fleetd #571: delivery is unknown here — agent.prompt pastes and submits in one call,
// so the message may already be sitting in the pane. Do not invite a blind retry the way
// the case above does; a resend on this route can double-deliver the same brief.
case TIMED_OUT_UNCONFIRMED -> text("[no reply within " + timeout + "ms — delivery unconfirmed; "
+ "the message may already have reached the worker, so a retry risks sending it "
+ "twice — poll status before resending]");
};
}
@@ -1546,25 +1575,42 @@ public final class FleetMcp {
* @param selfTerm the calling pane's terminal id, or blank for a caller with no pane
*/
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions,
Map<String, String> leads, String selfTerm) {
Map<String, String> leads, String selfTerm) {
return listFleet(workers, sessions, null, CapacitySource.none(), new HealthCoverageSource(() -> "off"),
QuarantineSource.none(), leads, selfTerm);
LoopHealthSource.none(), QuarantineSource.none(), leads, selfTerm);
}
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, Map<String, String> leads, String selfTerm) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine, leads, selfTerm,
CoordinationSource.none());
}
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, Map<String, String> leads, String selfTerm) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, leads, selfTerm,
LoopHealthSource loopHealth, QuarantineSource quarantine,
Map<String, String> leads, String selfTerm) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, loopHealth, quarantine, leads, selfTerm,
CoordinationSource.none());
}
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
LoopHealthSource loopHealth, QuarantineSource quarantine,
Map<String, String> leads, String selfTerm,
CoordinationSource coordination) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, loopHealth, quarantine,
OutageSource.none(), LeadSeatSource.none(), leads, selfTerm, coordination, false);
}
/** As above, plus fleetd #201 Unit 5 cool-off facts (see {@link OutageSource}). */
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, OutageSource outage,
Map<String, String> leads, String selfTerm) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, outage,
LeadSeatSource.none(), leads, selfTerm, CoordinationSource.none());
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine, outage,
LeadSeatSource.none(), leads, selfTerm, CoordinationSource.none(), false);
}
/**
@@ -1580,8 +1626,8 @@ public final class FleetMcp {
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, Map<String, String> leads, String selfTerm,
CoordinationSource coordination) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, OutageSource.none(),
LeadSeatSource.none(), leads, selfTerm, coordination);
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine, OutageSource.none(),
LeadSeatSource.none(), leads, selfTerm, coordination, false);
}
/** As above, plus fleetd #201 Unit 5 cool-off facts (see {@link OutageSource}). */
@@ -1589,8 +1635,8 @@ public final class FleetMcp {
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, OutageSource outage,
Map<String, String> leads, String selfTerm, CoordinationSource coordination) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, outage,
LeadSeatSource.none(), leads, selfTerm, coordination);
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine, outage,
LeadSeatSource.none(), leads, selfTerm, coordination, false);
}
/**
@@ -1611,7 +1657,7 @@ public final class FleetMcp {
QuarantineSource quarantine, OutageSource outage,
LeadSeatSource leadSeats, Map<String, String> leads, String selfTerm,
CoordinationSource coordination) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, quarantine, outage,
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine, outage,
leadSeats, leads, selfTerm, coordination, false);
}
@@ -1631,8 +1677,18 @@ public final class FleetMcp {
* an explicit {@code true}
*/
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, OutageSource outage,
CapacitySource capacity, HealthCoverageSource healthCoverage,
QuarantineSource quarantine, OutageSource outage,
LeadSeatSource leadSeats, Map<String, String> leads, String selfTerm,
CoordinationSource coordination, boolean callerIsPrimary) {
return listFleet(workers, sessions, messages, capacity, healthCoverage, LoopHealthSource.none(), quarantine,
outage, leadSeats, leads, selfTerm, coordination, callerIsPrimary);
}
static McpSchema.CallToolResult listFleet(PeerLauncher workers, SessionManager sessions, MessageService messages,
CapacitySource capacity, HealthCoverageSource healthCoverage,
LoopHealthSource loopHealth,
QuarantineSource quarantine, OutageSource outage,
LeadSeatSource leadSeats, Map<String, String> leads, String selfTerm,
CoordinationSource coordination, boolean callerIsPrimary) {
try {
@@ -1656,6 +1712,9 @@ public final class FleetMcp {
Map<String, Object> result = new LinkedHashMap<>();
result.put("leads", leadRows); result.put("members", out);
result.put("healthCoverage", healthCoverage.value().get());
result.put("loopHealth", Map.of(
"statusPoller", loopHealth.statusPoller().get().name(),
"sessionReaper", loopHealth.sessionReaper().get().name()));
// fleetd #439: coordinator/coordinatorView is lead-to-lead coordination state and must
// never reach a worker or an architect -- gate BEFORE assembling it, not after, so the
// key is absent rather than present-and-empty.
@@ -2075,8 +2134,9 @@ public final class FleetMcp {
return tool(FleetTool.SPAWN.wireName(),
"Spawn a new off-subscription member session. A member has two independent attributes: "
+ "role (what it is for) and profile (which backend it runs on). Pass role to pick "
+ "the contract — 'dev' implements a unit and opens its own PR, 'reviewer' reviews a "
+ "diff it did not write, 'architect' refines a ticket before anyone builds it; omit "
+ "the contract — 'dev' implements a unit and opens its own PR, 'hunter' sweeps a "
+ "scope without changing it, 'reviewer' reviews a diff it did not write, 'architect' "
+ "refines a ticket before anyone builds it; omit "
+ "it for 'dev'. Pass profile (from fleet_profiles) to pick the backend, or omit it "
+ "for the default. The two are independent: a reviewer may run on the same profile "
+ "as the dev it reviews. The member opens your current directory by default; pass "
@@ -2093,7 +2153,7 @@ public final class FleetMcp {
+ "one. Returns the member's sessionId (use with fleet_send) and paneId (use with "
+ "fleet_stop).",
objectSchema(Map.of(
"role", stringProp("What the member is for: architect, dev or reviewer (default dev)"),
"role", stringProp("What the member is for: architect, dev, hunter, or reviewer (default dev)"),
"profile", stringProp("Which backend to run it on (omit for the default profile)"),
"cwd", stringProp("Working directory for the member (omit to inherit yours)"),
"worktree", Map.of("type", "string", "description", "'true' or a ticket slug — requests an isolated git worktree"),
@@ -2135,9 +2195,10 @@ public final class FleetMcp {
+ "cannot reliably re-identify: some backends (e.g. opencode) resolve it from the "
+ "member's working directory, which only uniquely identifies a member when it "
+ "was spawned into its own fleetd-provisioned worktree (worktree:true/<slug>); a "
+ "member spawned without one shares its directory with others and never reports "
+ "an id, however long it runs (fleetd #249). An empty 'members' "
+ "means no members are spawned; it says nothing about peers. When capacity "
+ "member spawned without one shares its directory with others and never reports "
+ "an id, however long it runs (fleetd #249). An empty 'members' "
+ "means no members are spawned; it says nothing about peers. 'loopHealth' reports "
+ "the RUNNING, STALLED, or STOPPED state of statusPoller and sessionReaper. When capacity "
+ "facts are configured, a 'capacity' row per profile reports 'free' — the "
+ "slots a fresh fleet_spawn on that profile will actually be granted right "
+ "now (max(0, maxLoad - live)), the same check the spawn gate itself runs. A "
@@ -246,7 +246,7 @@ public final class AmqpReplyInbox implements ReplyInbox, AutoCloseable {
* neither. So before this method existed with a requeue step, it dropped {@link #held}'s entries
* for {@code target} while the broker still considered them outstanding: never acked, never
* nacked, never requeued, and no longer reachable by {@link #peek} — permanently invisible. This
* is unlike {@link #handleRecovery} and {@link #close()}, whose bare {@code held.clear()} is
* is unlike {@link RecoveryListener#handleRecovery(Recoverable)} and {@link #close()}, whose bare {@code held.clear()} is
* correct because each has already made the broker requeue (a real connection drop, or
* {@code channel.close()} respectively) before clearing local state.
*
@@ -92,8 +92,31 @@ public final class MessageService {
QUESTION,
/** Timed out after the message was delivered — the worker is still working. */
TIMED_OUT_WORKING,
/** Timed out before delivery — the message is still queued for the worker. */
/**
* Timed out with no confirmed delivery, and the target saw nothing — the message will not
* arrive later, so a caller may resend. {@link #send} reaches this outcome through {@link
* Injector#cancel} reporting one of two routes: {@link Injector.Cancellation#CANCELLED}
* means the message was still queued and this call removed it; {@link
* Injector.Cancellation#NOT_DELIVERED} means nothing was ever sent — the queue was cleared
* because the target never became ready or was abandoned, or the injector's call to the
* target's terminal ({@link dev.ltms.fleet.herdr.AgentControl#send}) failed with a herdr
* error that this codebase already treats as a confirmed absence. A third route,
* {@link Injector.Cancellation#ATTEMPTED}, used to be folded into this same outcome
* (fleetd #571) — it no longer is; see {@link #TIMED_OUT_UNCONFIRMED}.
*/
TIMED_OUT_QUEUED,
/**
* Timed out with delivery unknown. {@link #send} reaches this outcome when {@link
* Injector#cancel} reports {@link Injector.Cancellation#ATTEMPTED} (fleetd #551): the call
* to the target's terminal ({@link dev.ltms.fleet.herdr.AgentControl#send}) was made, but
* this caller never observed whether it reached the pane. {@code agent.prompt} pastes
* <em>and submits</em> in one call, so the target may already hold a complete, submitted
* turn and be working on it right now — the same reality as {@link #TIMED_OUT_WORKING},
* just not confirmed. The message may or may not have arrived. Treat this as neither a
* confirmed delivery nor a confirmed absence: a caller that resends on this outcome risks a
* double delivery — the same brief typed into the pane twice (fleetd #571).
*/
TIMED_OUT_UNCONFIRMED,
/** Another send to this session was in flight for the whole window. */
BUSY,
/**
@@ -299,12 +322,21 @@ public final class MessageService {
*/
private final ConcurrentHashMap<String, Boolean> strandedReplies = new ConcurrentHashMap<>();
/**
* Targets whose last send timed out with {@link Outcome#TIMED_OUT_QUEUED} (CB-640) — the
* message never reached the {@link Injector} delivery window before the caller's deadline, so
* it is still sitting in the injector's own per-target queue. Set where {@link #send} already
* computes {@code wasDelivered} for that outcome; no new queue is kept here, only the fact.
* Cleared the same way as {@link #strandedReplies}: the next accepted delivery for the target
* ({@link #send} opening a fresh waiter) or a teardown ({@link #abandon}).
* Targets whose last send timed out with no confirmed delivery (CB-640) — {@link #send} called
* {@link Injector#cancel} and got back something other than {@code DELIVERED}. That covers
* three histories, not one: {@link Injector.Cancellation#CANCELLED} — the message was still
* queued and {@code cancel} removed it right there; {@link Injector.Cancellation#NOT_DELIVERED}
* — nothing was ever sent, because the target never became ready, was torn down, or the call to
* its terminal failed with a herdr error this codebase already treats as a confirmed absence; or
* {@link Injector.Cancellation#ATTEMPTED} (fleetd #551) — the call to the target's terminal was
* made and its outcome is unknown, so the target may already hold a complete, submitted turn.
* Only the first two mean the message will not arrive later and the target saw nothing; on the
* third it may already have arrived in full — and the caller sees a different outcome for it
* ({@link Outcome#TIMED_OUT_UNCONFIRMED}, fleetd #571) than for the first two ({@link
* Outcome#TIMED_OUT_QUEUED}). Set where {@link #send} already computes {@code wasDelivered} for
* that outcome; no queue is kept here, only the fact that the send ended with no confirmed
* delivery. Cleared the same way as {@link #strandedReplies}: the next accepted delivery for the
* target ({@link #send} opening a fresh waiter) or a teardown ({@link #abandon}).
*/
private final ConcurrentHashMap<String, Boolean> queuedDeliveries = new ConcurrentHashMap<>();
private final AtomicLong ticketSeq = new AtomicLong();
@@ -391,12 +423,22 @@ public final class MessageService {
/**
* Read-only delegation fact for fleet views (CB-640): {@code target}'s last send timed out
* before the {@link Injector} ever delivered it — the caller saw
* {@link Outcome#TIMED_OUT_QUEUED} (see the {@code TimeoutException} branch of {@link #send}),
* and the message is still sitting in the injector's per-target queue waiting for the worker
* to go idle. Distinct from {@link Outcome#TIMED_OUT_WORKING}, where delivery already happened
* and only the reply is outstanding. Cleared the next time this target's delivery is accepted
* or the target is abandoned — see {@link #queuedDeliveries}.
* with no confirmed delivery — the caller saw {@link Outcome#TIMED_OUT_QUEUED} or {@link
* Outcome#TIMED_OUT_UNCONFIRMED} (fleetd #571; see the {@code TimeoutException} branch of
* {@link #send}). Despite the method's name, this is not proof that a message is sitting in a
* queue: {@link Injector#cancel} reports this outcome through three routes. {@link
* Injector.Cancellation#CANCELLED} means the message was still queued and got removed right
* there. {@link Injector.Cancellation#NOT_DELIVERED} means nothing was ever sent — the target
* never became ready, was torn down, or the call to its terminal failed with a herdr error this
* codebase already treats as a confirmed absence. Only these two routes mean the message will
* not arrive later, and both report {@code TIMED_OUT_QUEUED}. {@link
* Injector.Cancellation#ATTEMPTED} (fleetd #551) means the call to the target's terminal was
* made and its outcome is unknown: {@code agent.prompt} pastes <em>and submits</em> in one
* call, so on this route the target may already hold a complete, submitted turn and be
* working on it right now — it does NOT follow that the target saw nothing, and this route
* reports {@code TIMED_OUT_UNCONFIRMED} instead. Distinct from {@link Outcome#TIMED_OUT_WORKING},
* where delivery already happened and only the reply is outstanding. Cleared the next time this
* target's delivery is accepted or the target is abandoned — see {@link #queuedDeliveries}.
*/
public boolean hasQueuedDelivery(String target) {
return target != null && queuedDeliveries.containsKey(target);
@@ -416,15 +458,15 @@ public final class MessageService {
/**
* Read-only delegation fact for fleet views (CB-640): an async ticket is still
* {@link Phase#PENDING} against {@code target}, yet nothing is actually in flight for it — no
* open rendezvous waiter ({@link #hasAcceptedDelivery}) and no message still sitting in the
* injector's queue ({@link #hasQueuedDelivery}). A healthy PENDING ticket can briefly look this
* way while its virtual thread has not yet been scheduled or is blocked on the session lock
* behind another send to the same target, so this is a snapshot fact for the health classifier
* to weigh across ticks, not proof on its own that the ticket is stuck. It also genuinely
* persists — not just as a passing race — once an async {@code fleet_ask} lapses unanswered:
* {@link #ask} clears the ticket's question and returns it to {@code PENDING}, but {@link #send}
* already closed the forward waiter the instant the question surfaced, so the target has
* neither an accepted nor a queued delivery left to show for it.
* open rendezvous waiter ({@link #hasAcceptedDelivery}) and no record of a send that ended with
* no confirmed delivery ({@link #hasQueuedDelivery}). A healthy PENDING ticket can briefly look
* this way while its virtual thread has not yet been scheduled or is blocked on the session
* lock behind another send to the same target, so this is a snapshot fact for the health
* classifier to weigh across ticks, not proof on its own that the ticket is stuck. It also
* genuinely persists — not just as a passing race — once an async {@code fleet_ask} lapses
* unanswered: {@link #ask} clears the ticket's question and returns it to {@code PENDING}, but
* {@link #send} already closed the forward waiter the instant the question surfaced, so the
* target has neither an accepted nor a queued delivery left to show for it.
*
* <p><strong>Deliberately still {@code question == null} only (fleetd #275).</strong> This
* method must not also report a still-{@link Phase#ASKING} task as orphaned: the worker may
@@ -611,7 +653,7 @@ public final class MessageService {
return switch (o) {
case REPLIED -> "replied";
case COMPLETED_UNREPLIED -> "completion_fallback";
case TIMED_OUT_WORKING, TIMED_OUT_QUEUED, BUSY -> "timeout";
case TIMED_OUT_WORKING, TIMED_OUT_QUEUED, TIMED_OUT_UNCONFIRMED, BUSY -> "timeout";
case WORKER_FAILED -> "failed";
case BACKEND_EXHAUSTED -> "backend_exhausted";
case STALE_TURN, QUESTION -> null; // not a completed delegation
@@ -713,7 +755,7 @@ public final class MessageService {
* failure.
*
* <p><strong>Without {@code sweepAsking} on the release path, a target torn down while
* genuinely {@code ASKING} was unrecoverable.</strong> {@link #resolveQuestion} had already
* genuinely {@code ASKING} was unrecoverable.</strong> {@link Rendezvous#resolveQuestion(String, String, String)} had already
* closed the forward waiter the instant the question surfaced (so the {@code waiter} branch
* below finds nothing to fail), the {@code question == null} guard excluded the task from
* {@code matching} (so the loop below skipped it too), and the worker's own {@code fleet_ask}
@@ -940,6 +982,7 @@ public final class MessageService {
} catch (TimeoutException e) {
boolean wasDelivered = delivery.completion().isDone()
&& !delivery.completion().isCompletedExceptionally();
Injector.Cancellation cancellation = null;
if (!wasDelivered) {
if (timeoutCancellationRaceHookForTest != null) {
// Test-only (fleetd #345): see the field's own javadoc.
@@ -947,16 +990,27 @@ public final class MessageService {
}
// The target monitor makes cancellation atomic with onStatus picking this
// Pending up. If pickup won, report TIMED_OUT_WORKING because the text landed.
wasDelivered = injector.cancel(delivery) == Injector.Cancellation.DELIVERED;
cancellation = injector.cancel(delivery);
wasDelivered = cancellation == Injector.Cancellation.DELIVERED;
}
log.debug("send to {} timed out (delivered={})", target, wasDelivered);
if (!wasDelivered) {
// CB-640: record that delivery did not happen for fleet health (see
// queuedDeliveries). The exact Pending was cancelled, so it cannot arrive later.
Outcome outcome;
if (wasDelivered) {
outcome = Outcome.TIMED_OUT_WORKING;
} else if (cancellation == Injector.Cancellation.ATTEMPTED) {
// fleetd #571: the call to the target's terminal was made and its outcome is
// unknown — the message may already have arrived in full, so this must not
// be reported as TIMED_OUT_QUEUED, which promises it never will.
outcome = Outcome.TIMED_OUT_UNCONFIRMED;
} else {
// CB-640: record that delivery is not confirmed, for fleet health (see
// queuedDeliveries). cancellation is CANCELLED (this call removed a
// still-queued Pending) or NOT_DELIVERED (an earlier attempt already failed
// with a confirmed absence) — both mean the target saw nothing.
queuedDeliveries.put(target, Boolean.TRUE);
outcome = Outcome.TIMED_OUT_QUEUED;
}
return recorded(new Reply(
wasDelivered ? Outcome.TIMED_OUT_WORKING : Outcome.TIMED_OUT_QUEUED, null));
return recorded(new Reply(outcome, null));
} catch (ExecutionException e) {
Throwable cause = e.getCause();
throw cause instanceof RuntimeException re ? re : new IllegalStateException(cause);
@@ -1096,21 +1150,33 @@ public final class MessageService {
}
try {
CompletableFuture<Rendezvous.Resolution> reply = rendezvous.open(workerSession);
// #282: mirror send()'s registration (:802) so a SECOND fleet_ask inside this same
// resumed turn can re-associate the async ticket with its new turnId via
// markAsyncQuestion — without this, that second ask has no Task to attach to, and
// markAsyncQuestion silently returns null.
Task task = asyncTasksByTurn.get(turnId);
if (task != null) {
asyncTasksByWaiter.put(reply, task);
}
if (!rendezvous.answerAsk(turnId, content)) {
asyncTasksByWaiter.remove(reply);
rendezvous.close(workerSession, reply);
return new Reply(Outcome.STALE_TURN, null); // lapsed between the lookup and the unblock
}
clearAsyncQuestion(turnId, false);
// fleetd #575: this try used to open below, AFTER the Task lookup/registration and the
// STALE_TURN early return that follows it — so that return was covered only by a
// hand-rolled copy of the finally's own cleanup pair, not the finally itself. Widening the
// try up to wrap the registration closes the gap structurally: every exit from here on,
// STALE_TURN included, now runs through the one finally below exactly once, and the
// duplicated pair is gone. This did not fix a live leak — see the ticket: neither
// rendezvous.answerAsk nor clearAsyncQuestion(turnId, false) can throw, so nothing ever
// actually left through the old gap uncovered — but #572 found this exact drift (one
// finally asserted, an identical sibling not) on this same file, and the hand-rolled copy
// was the wrong shape to keep regardless of whether it was ever exercised.
try {
// #282: mirror send()'s registration (:802) so a SECOND fleet_ask inside this same
// resumed turn can re-associate the async ticket with its new turnId via
// markAsyncQuestion — without this, that second ask has no Task to attach to, and
// markAsyncQuestion silently returns null.
Task task = asyncTasksByTurn.get(turnId);
if (task != null) {
asyncTasksByWaiter.put(reply, task);
}
if (answerAskLapseRaceHookForTest != null) {
// Test-only (fleetd #575): see the field's own javadoc.
answerAskLapseRaceHookForTest.run();
}
if (!rendezvous.answerAsk(turnId, content)) {
return new Reply(Outcome.STALE_TURN, null); // lapsed between the lookup and the unblock
}
clearAsyncQuestion(turnId, false);
Rendezvous.Resolution r = reply.get(remainingMillis(deadlineNanos), TimeUnit.MILLISECONDS);
Reply result = new Reply(outcomeOf(r.kind()), r.text(), r.turnId());
// #282: this waiter can resolve with a FRESH question rather than a terminal reply —
@@ -1610,6 +1676,30 @@ public final class MessageService {
this.askTimeoutRaceHookForTest = hook;
}
/**
* Null in production; test seam for fleetd #575 — invoked from {@link #answer}, right after this
* call's own {@code Task} registration and right before its {@code rendezvous.answerAsk(turnId,
* content)} call. A test installs this to complete the SAME turnId's ask directly via {@link
* Rendezvous#answerAsk} from inside that exact window, deterministically reproducing what a
* second, concurrent {@code answer()} call racing to unblock the same ask can otherwise only
* win by timing luck: this call's own {@code askSession(turnId)} lookup at the top already saw
* the ask as open, but by the time it reaches {@code rendezvous.answerAsk} here, the other call
* already completed it (or the worker's own {@code ask()} teardown already closed it) — so this
* call must see {@code false} and return {@link Outcome#STALE_TURN}, exactly the "lapsed between
* the lookup and the unblock" case named at that call site. Proves the #575 fix (widening this
* method's try so a single finally covers this exit) does not change that outcome and still
* cleans this call's own {@code reply} up exactly once.
*/
private volatile Runnable answerAskLapseRaceHookForTest;
/**
* Test-only (fleetd #575): install {@link #answerAskLapseRaceHookForTest}. Package-private so the
* test, in the same package, can reach it without widening any production API.
*/
void setAnswerAskLapseRaceHookForTest(Runnable hook) {
this.answerAskLapseRaceHookForTest = hook;
}
/** A new send must not open a waiter while an async ticket owns this worker's paused turn. */
private boolean hasAsyncQuestion(String target) {
return asyncTasksByTurn.values().stream().anyMatch(task -> target.equals(task.target));
@@ -44,7 +44,7 @@ import java.util.stream.Collectors;
* still holds an unacked message ({@link #pendingReplies}), tickets not yet collected
* ({@link #pendingTickets}), and open questions not yet answered or lapsed
* ({@link #pendingQuestions}) — and sends at most one combined nudge per tick
* ({@link #injectNudge(String, int, int, int)}). Work that arrives while the lead is busy is
* ({@link #injectNudge(String, int, int, int, int, int)}). Work that arrives while the lead is busy is
* never lost: it is re-read fresh on every tick until the lead is injectable or its own reminder
* cap ({@link #maxReminders}) is reached — each source spends from its own budget, so one source
* exhausting its cap does not stop nudges about the others (post-CB-590 regression fix; see
@@ -29,8 +29,8 @@ public enum MemberRole {
* <p>Reads the repo and writes analysis. Never commits code and never opens a pull request —
* an architect that starts implementing has stopped doing the job that makes it useful.
*
* <p>Architects are the one member kind declared in config, because a lead addresses the same
* slots across many tickets and needs a stable name for them.
* <p>Architects are the one member kind with live slot binding, because a lead addresses the
* same slots across many tickets and needs a stable name for them.
*/
ARCHITECT,
@@ -43,6 +43,14 @@ public enum MemberRole {
*/
DEV,
/**
* Sweeps an assigned package for defects and reports several ranked findings.
*
* <p>Never changes code, commits, or opens a pull request. A hunt gathers evidence, which can
* include running the build, but leaves every fix to a later implementation unit.
*/
HUNTER,
/**
* Reviews a diff it did not write and reports one structured finding.
*
@@ -59,7 +67,7 @@ public enum MemberRole {
/**
* The {@code fleet:} block that holds this role's pool — {@code architects},
* {@code developers}, {@code reviewers}.
* {@code developers}, {@code hunters}, {@code reviewers}.
*
* <p>Plural, and not always the wire name: the pool of things a {@code dev} may run on reads
* naturally as {@code developers:}. The wire name stays the singular {@code dev}, because that
@@ -69,6 +77,7 @@ public enum MemberRole {
return switch (this) {
case ARCHITECT -> "architects";
case DEV -> "developers";
case HUNTER -> "hunters";
case REVIEWER -> "reviewers";
};
}
@@ -94,6 +94,7 @@ public final class FleetApp {
// .none() (the honest "feature not wired" view) for every constructor that does not pass one.
private final FleetMcp.QuarantineSource quarantine;
private final FleetMcp.OutageSource outage;
private final FleetMcp.LoopHealthSource loopHealth;
private final ObjectMapper mapper = new ObjectMapper();
/**
@@ -155,7 +156,8 @@ public final class FleetApp {
HttpServlet mcpServlet, CallerResolver auth, Metrics metrics,
Predicate<String> deliverable, Supplier<MemberCredentialPolicyView> memberCredentials) {
this(herdr, memberHerdr, workers, sessions, messages, presence, mcpServlet, auth, metrics,
deliverable, memberCredentials, FleetMcp.QuarantineSource.none(), FleetMcp.OutageSource.none());
deliverable, memberCredentials, FleetMcp.QuarantineSource.none(), FleetMcp.OutageSource.none(),
FleetMcp.LoopHealthSource.none());
}
/**
@@ -169,8 +171,18 @@ public final class FleetApp {
public FleetApp(HerdrClient herdr, HerdrClient memberHerdr, PeerLauncher workers, SessionManager sessions,
MessageService messages, MemberPresence presence,
HttpServlet mcpServlet, CallerResolver auth, Metrics metrics,
Predicate<String> deliverable, Supplier<MemberCredentialPolicyView> memberCredentials,
FleetMcp.QuarantineSource quarantine, FleetMcp.OutageSource outage) {
Predicate<String> deliverable, Supplier<MemberCredentialPolicyView> memberCredentials,
FleetMcp.QuarantineSource quarantine, FleetMcp.OutageSource outage) {
this(herdr, memberHerdr, workers, sessions, messages, presence, mcpServlet, auth, metrics, deliverable,
memberCredentials, quarantine, outage, FleetMcp.LoopHealthSource.none());
}
public FleetApp(HerdrClient herdr, HerdrClient memberHerdr, PeerLauncher workers, SessionManager sessions,
MessageService messages, MemberPresence presence,
HttpServlet mcpServlet, CallerResolver auth, Metrics metrics,
Predicate<String> deliverable, Supplier<MemberCredentialPolicyView> memberCredentials,
FleetMcp.QuarantineSource quarantine, FleetMcp.OutageSource outage,
FleetMcp.LoopHealthSource loopHealth) {
this.herdr = herdr;
this.memberHerdr = memberHerdr != null ? memberHerdr : herdr;
this.workers = workers;
@@ -183,6 +195,7 @@ public final class FleetApp {
this.memberCredentials = memberCredentials != null ? memberCredentials : MemberCredentialPolicyView::absent;
this.quarantine = quarantine != null ? quarantine : FleetMcp.QuarantineSource.none();
this.outage = outage != null ? outage : FleetMcp.OutageSource.none();
this.loopHealth = loopHealth != null ? loopHealth : FleetMcp.LoopHealthSource.none();
}
/** Wire routes onto a fresh, unstarted Javalin instance. Caller starts it. */
@@ -291,15 +304,19 @@ public final class FleetApp {
* spotted by comparing two numbers by eye.
*/
private void healthz(Context ctx) {
HealthzResponse response = healthzResponse(herdr, memberHerdr, loopHealth);
ctx.status(response.status()).json(response.body());
}
record HealthzResponse(int status, Map<String, Object> body) { }
static HealthzResponse healthzResponse(HerdrClient herdr, HerdrClient memberHerdr,
FleetMcp.LoopHealthSource loopHealth) {
JsonNode pong;
try {
pong = herdr.call("ping");
} catch (HerdrException e) {
ctx.status(503).json(Map.of(
"status", "degraded",
"herdr", "unreachable",
"detail", e.getMessage()));
return;
return degradedResponse("unreachable", e.getMessage(), loopHealth);
}
Map<String, Object> body = new LinkedHashMap<>();
body.put("status", "ok");
@@ -311,11 +328,11 @@ public final class FleetApp {
try {
memberPong = memberHerdr.call("ping");
} catch (HerdrException e) {
ctx.status(503).json(Map.of(
return new HealthzResponse(503, Map.of(
"status", "degraded",
"herdr", "member unreachable",
"detail", e.getMessage()));
return;
"detail", e.getMessage(),
"loopHealth", loopHealthView(loopHealth)));
}
int leadProtocol = pong.path("protocol").asInt();
int memberProtocol = memberPong.path("protocol").asInt();
@@ -326,7 +343,20 @@ public final class FleetApp {
body.put("protocolMismatch", true);
}
}
ctx.status(200).json(body);
body.put("loopHealth", loopHealthView(loopHealth));
return new HealthzResponse(200, body);
}
private static Map<String, String> loopHealthView(FleetMcp.LoopHealthSource loopHealth) {
return Map.of("statusPoller", loopHealth.statusPoller().get().name(),
"sessionReaper", loopHealth.sessionReaper().get().name());
}
private static HealthzResponse degradedResponse(String herdr, String detail,
FleetMcp.LoopHealthSource loopHealth) {
return new HealthzResponse(503, Map.of(
"status", "degraded", "herdr", herdr, "detail", detail,
"loopHealth", loopHealthView(loopHealth)));
}
/**
@@ -640,15 +670,31 @@ public final class FleetApp {
}
default -> ctx.status(202).json(Map.of(
"sessionId", id,
// fleetd #571 (ticket comment 17126): no `default` here on purpose. This switch
// is an expression, so the compiler already demands every Outcome constant have
// an arm — adding an 11th constant to Outcome is a compile error here, not a
// silent fall-through. That is exactly the bug this ticket exists to fix:
// `default -> "done"` used to sit here and would have told a REST caller the
// delegation completed for TIMED_OUT_UNCONFIRMED, the one outcome where delivery
// is unknown. REPLIED, COMPLETED_UNREPLIED, QUESTION and STALE_TURN can never
// actually reach this inner switch — the outer switch above always dispatches
// them first — but they still need an arm to keep this switch exhaustive.
"status", switch (reply.outcome()) {
case TIMED_OUT_WORKING -> "working";
case TIMED_OUT_QUEUED -> "queued";
// Delivery here is unknown, not merely still queued — see
// Outcome#TIMED_OUT_UNCONFIRMED's own javadoc.
case TIMED_OUT_UNCONFIRMED -> "unconfirmed";
case BUSY -> "busy";
case WORKER_FAILED -> "failed";
case BACKEND_EXHAUSTED -> "backend_exhausted";
default -> "done"; // unreachable (terminal outcomes handled above)
case REPLIED, COMPLETED_UNREPLIED, QUESTION, STALE_TURN -> "done"; // unreachable
},
"detail", (reply.outcome() == MessageService.Outcome.WORKER_FAILED
"detail", reply.outcome() == MessageService.Outcome.TIMED_OUT_UNCONFIRMED
? "no reply within " + timeout + "ms; delivery is unconfirmed — the "
+ "message may already have reached the worker, so a resend "
+ "risks sending it twice; poll status first"
: (reply.outcome() == MessageService.Outcome.WORKER_FAILED
|| reply.outcome() == MessageService.Outcome.BACKEND_EXHAUSTED)
&& reply.text() != null
? reply.text()
@@ -1074,6 +1074,29 @@ public final class SessionManager implements TurnListener {
* members were live) and must still produce the line. Both {@link #drainSnapshot} passes (the
* main snapshot and the straggler sweep) are folded into the one line: a caller reading two
* lines could not tell a two-pass drain from two separate drains.
*
* <p><strong>Non-goal: this line must never move into a {@code finally} block, and this method
* must never grow one around it.</strong> "Every time" above means every time the drain
* <em>finishes</em>, not every time this method exits. The absence of the line is the signal
* that the drain died, so a {@code finally} would destroy the signal and print confident
* partial counts in the same edit — the line would appear after a drain that threw, carrying
* whatever {@code tally} it had reached. Both halves of the value are lost at once. The line
* has to be the last statement of the successful path and reachable only from it.
*
* <p>This is written down because it is the obvious review comment ("shouldn't we always log
* the drain result?"), it sounds like thoroughness, and the paragraph above reads as an
* invitation to it. Raised by the fleet01 lead on 2026-09-12, from their 2026-09-10 incident:
* we only know that drain died on that host because it <em>threw</em>, and a
* {@code NoClassDefFoundError} reached the JVM's uncaught handler. A drain that hung on one
* session, or returned early on a condition rather than an exception, would leave no stack
* trace, no {@code ERROR} token and no priority — only a missing line. That makes the loud
* variant the one we have seen and the quiet variants the ones this line exists to catch.
*
* <p>Related: {@code released} and {@code abandoned} are counted incrementally inside {@link
* #drainSnapshot}'s loop and folded with {@link DrainTally#plus}, rather than derived from a
* collection read at the end, for the same reason. If a partial report is ever wanted it must
* be a different line with a different verb. One line must not serve both, or a reader cannot
* tell a finished drain from an interrupted one by its wording.
*/
void drainAll(long timeoutNanos) {
long deadline = System.nanoTime() + timeoutNanos;
@@ -1,9 +1,11 @@
package dev.ltms.fleet.session;
import dev.ltms.fleet.inject.LoopWatchdog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.concurrent.TimeUnit;
import java.util.function.LongSupplier;
/**
* Periodic virtual-thread reaper that tears down {@code READY}/{@code DONE} sessions which have
@@ -14,6 +16,16 @@ public final class SessionReaper {
private static final Logger log = LoggerFactory.getLogger(SessionReaper.class);
private static final long DEFAULT_INTERVAL_MILLIS = 5000;
/**
* How many multiples of {@code intervalMillis} the last-completed round may age before
* {@link #health()} reports {@link LoopWatchdog.State#STALLED} (fleetd #544). One round here
* is {@code sessions.reapIdle} plus the (rare, best-effort) WIP sweep — both normally finish
* in a small fraction of one interval. At the default 5s interval this puts the threshold at
* 60s: generous enough that an occasional slow git call in the WIP sweep never trips it, short
* enough that a genuinely wedged reap round (mirroring the herdr-read-with-no-deadline hang
* {@code StatusPoller} guards against — see fleetd #544) is caught inside about a minute.
*/
private static final long STALE_AFTER_INTERVAL_MULTIPLIER = 12;
/** CB-586: the refs/wip age floor — never sweep a snapshot younger than 24h (the CB-586 rule). */
private static final long WIP_MIN_AGE_MILLIS = TimeUnit.HOURS.toMillis(24);
/**
@@ -25,6 +37,7 @@ public final class SessionReaper {
private final SessionManager sessions;
private final long idleTtlNanos;
private final long intervalMillis;
private final LoopWatchdog watchdog;
private volatile boolean running;
private Thread thread;
/**
@@ -45,29 +58,61 @@ public final class SessionReaper {
/** Construct a reaper with an explicit polling interval (useful for tests). */
public SessionReaper(SessionManager sessions, long idleTtlSeconds, long intervalMillis) {
this(sessions, idleTtlSeconds, intervalMillis, System::nanoTime);
}
/** Full constructor — for tests: an injectable monotonic clock (fleetd #544). */
SessionReaper(SessionManager sessions, long idleTtlSeconds, long intervalMillis, LongSupplier nowNanos) {
this.sessions = sessions;
this.idleTtlNanos = TimeUnit.SECONDS.toNanos(idleTtlSeconds);
this.intervalMillis = intervalMillis;
this.watchdog = new LoopWatchdog(nowNanos, staleAfterNanos(intervalMillis));
}
private static long staleAfterNanos(long intervalMillis) {
return TimeUnit.MILLISECONDS.toNanos(Math.max(intervalMillis, 1)) * STALE_AFTER_INTERVAL_MULTIPLIER;
}
/**
* This loop's progress fact (fleetd #544): {@code RUNNING}, {@code STALLED} (dead or parked —
* indistinguishable from outside, and never observed on purpose), or {@code STOPPED}
* ({@link #stop()} was called). See {@link LoopWatchdog} for why staleness — not thread
* liveness — is the signal.
*/
public LoopWatchdog.State health() {
return watchdog.state();
}
/** Start the reaper loop on a virtual thread. Idempotent. */
public synchronized void start() {
if (running) return;
running = true;
watchdog.reset(); // fleetd #544: a fresh grace period, not last run's stale mark/timestamp.
thread = Thread.ofVirtual().name("session-reaper").start(this::loop);
log.info("session reaper started (idle ttl {}s, interval {}ms)",
TimeUnit.NANOSECONDS.toSeconds(idleTtlNanos), intervalMillis);
}
private void loop() {
while (running) {
try {
sessions.reapIdle(idleTtlNanos);
} catch (RuntimeException e) {
log.warn("session reaper iteration failed; continuing", e);
try {
while (running) {
try {
sessions.reapIdle(idleTtlNanos);
} catch (Throwable e) {
log.error("session reaper iteration failed; continuing", e);
}
maybeSweepWipRefs();
// fleetd #544: a round is "complete" only after reapIdle AND the WIP-sweep gate have
// both returned — a hang in either (e.g. a wedged git call) means this line is never
// reached, so the watchdog goes stale exactly like a dead loop would.
watchdog.recordRoundComplete();
sleep();
}
maybeSweepWipRefs();
sleep();
} finally {
if (running) {
log.error("session reaper loop exited unexpectedly; it can be restarted");
}
running = false;
}
}
@@ -90,8 +135,8 @@ public final class SessionReaper {
log.info("refs/wip retention sweep deleted {} snapshot ref(s) older than 24h whose "
+ "content was already reachable from main", deleted);
}
} catch (RuntimeException e) {
log.warn("refs/wip retention sweep failed; continuing", e);
} catch (Throwable e) {
log.error("refs/wip retention sweep failed; continuing", e);
}
// Set even when the sweep threw, so a broken repo is retried on the slow cadence rather
// than hammering git on every 5-second iteration.
@@ -111,6 +156,7 @@ public final class SessionReaper {
/** Stop the reaper loop. Idempotent. */
public synchronized void stop() {
running = false;
watchdog.markStoppedByCaller(); // fleetd #544: this halt is on purpose — health() must say STOPPED, not STALLED.
if (thread != null) thread.interrupt();
}
}
@@ -0,0 +1,160 @@
package dev.ltms.fleet;
import dev.ltms.fleet.config.ConfigRef;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.mcp.FleetMcp;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #426: {@code FleetHealthMonitor.coverage} had zero references anywhere in the test
* tree — not the method, not either output string, not the field it populates. {@code
* FleetHealthMonitorCoverageTest} (package {@code dev.ltms.fleet.health}) pins the three-branch
* method itself; that is the easy half.
*
* <p>The half that actually matters is this one: {@link Fleetd#healthCoverageSource} is the exact
* call site {@code Fleetd.main} wires into {@code FleetMcp}'s constructor, and it is what feeds
* {@code fleet_list}'s {@code healthCoverage} field (see {@code FleetMcp#listFleet}'s {@code
* result.put("healthCoverage", healthCoverage.value().get())}). Measured precedent on fleetd #423
* (for #415): swapping two arguments at a call site like this one — recreating #415's defect with
* the keys exchanged — compiled with 0 errors and ran the ENTIRE suite (1506 tests) green. A
* thoroughly-tested method proves nothing about whether the call site pairs its arguments correctly;
* only a test that drives the call site itself can catch that.
*
* <p><strong>Why this is not driven through a real {@code Fleetd.main} the way #407 drives its five
* reporters</strong> (keep the config invalid, assert on the log line emitted before {@code
* cfg.validateAll()} throws): all five of #407's reporters run in {@code main} before {@code
* validateAll()} (line ~171). {@link Fleetd#healthCoverageSource} is built during {@code FleetMcp}
* construction, which happens only after {@code UnixSocketHerdrClient.connect} has already opened a
* real herdr socket (line ~188) and after {@code sessions}/{@code workers} are constructed. Reaching
* this call site by actually running {@code main} would require a real socket connect — banned by
* this ticket's hard constraints — so #407's option 1 does not apply here. Instead {@link
* Fleetd#healthCoverageSource} is extracted to a directly-callable package-private factory, the same
* shape {@link Fleetd#capacitySource} and {@link Fleetd#quarantineSource} already use for the same
* reason (see {@code FleetdCapacitySourceWiringTest}, the direct precedent this test follows).
*
* <p>Uses a real {@link FleetConfig#load} + {@link ConfigRef} (no socket, no port bind, no spawn,
* nothing written outside {@code @TempDir}) so the fixture goes through the actual YAML parser and
* {@code FleetConfig.Health}/{@code Notifications} records, not a hand-built stand-in that could
* silently drift from what the parser actually produces.
*/
class FleetdHealthCoverageSourceWiringTest {
private static final String BASE = """
bind:
host: 127.0.0.1
port: 8765
herdrSocket: ~/.config/herdr/herdr.sock
""";
private static final String HEALTH_DISABLED_WITH_WEBHOOK = BASE + """
health:
enabled: false
notifications:
mode: webhook
""";
private static final String HEALTH_DETECTION_ONLY = BASE + """
health:
enabled: true
""";
private static final String HEALTH_FULL = BASE + """
health:
enabled: true
notifications:
mode: webhook
""";
private static final String HEALTH_ABSENT = BASE;
@Test
@DisplayName("enabled: false reports off, even with a webhook configured")
void disabledHealthReportsOff(@TempDir Path dir) throws Exception {
FleetMcp.HealthCoverageSource source = sourceFor(dir, HEALTH_DISABLED_WITH_WEBHOOK);
assertEquals("off", source.value().get(),
"health.enabled: false must report 'off' regardless of notifications — flipping "
+ "the 'enabled' argument at the HealthCoverageSource call site would report "
+ "'full' here instead");
}
@Test
@DisplayName("enabled with no notifications reports detection-only")
void enabledWithoutNotificationsReportsDetectionOnly(@TempDir Path dir) throws Exception {
FleetMcp.HealthCoverageSource source = sourceFor(dir, HEALTH_DETECTION_ONLY);
assertEquals("detection-only", source.value().get());
}
@Test
@DisplayName("enabled with a webhook configured reports full")
void enabledWithNotificationsReportsFull(@TempDir Path dir) throws Exception {
FleetMcp.HealthCoverageSource source = sourceFor(dir, HEALTH_FULL);
assertEquals("full", source.value().get(),
"health.enabled: true with notifications.mode: webhook must report 'full' — "
+ "swapping 'full' and 'detection-only' at the call site, or breaking the "
+ "enabled/notificationConfigured argument pairing, would report "
+ "'detection-only' here instead");
}
@Test
@DisplayName("an absent health: block reports off")
void absentHealthBlockReportsOff(@TempDir Path dir) throws Exception {
FleetMcp.HealthCoverageSource source = sourceFor(dir, HEALTH_ABSENT);
assertEquals("off", source.value().get());
}
/**
* fleetd #426, the live-wiring half: {@code health.notifications} is a {@code
* ConfigRef.SPLIT_KEYS} entry, and {@link Fleetd#healthCoverageSource} reads {@code
* config.get().health()} live (not the frozen startup {@code cfg}) — exactly like {@link
* Fleetd#capacitySource}'s {@code maxLoad} ({@code FleetdCapacitySourceWiringTest}'s {@code
* reloadedMaxLoadStillChangesWhatFleetListReports}). A hot-reloaded notifications block must
* change what {@code fleet_list} reports without a restart; a fix that froze the whole source
* against the startup snapshot would silently break that and every other test above would stay
* green, since none of them reload.
*/
@Test
@DisplayName("a hot notifications reload still changes what fleet_list reports")
void reloadedNotificationsStillChangeWhatFleetListReports(@TempDir Path dir) throws Exception {
Path file = dir.resolve("fleetd.yaml");
Files.writeString(file, HEALTH_DETECTION_ONLY);
FleetConfig cfg = FleetConfig.load(file);
ConfigRef config = new ConfigRef(file, cfg);
FleetMcp.HealthCoverageSource source = Fleetd.healthCoverageSource(config);
assertEquals("detection-only", source.value().get(),
"sanity: detection-only before any reload");
Files.writeString(file, HEALTH_FULL);
assertTrue(config.reload().applied());
// The live snapshot now carries the webhook — proves the reload really happened and this
// test is not accidentally passing because nothing changed.
assertTrue(config.get().health().notifications() != null
&& config.get().health().notifications().configured(),
"sanity: the reloaded config really carries a configured webhook");
assertEquals("full", source.value().get(),
"the SAME HealthCoverageSource instance must reflect a reloaded notifications "
+ "block without a restart — health.notifications is read live off "
+ "config.get(), exactly like capacitySource's maxLoad");
}
private static FleetMcp.HealthCoverageSource sourceFor(Path dir, String yaml) throws Exception {
Path file = dir.resolve("fleetd.yaml");
Files.writeString(file, yaml);
FleetConfig cfg = FleetConfig.load(file);
ConfigRef config = new ConfigRef(file, cfg);
return Fleetd.healthCoverageSource(config);
}
}
@@ -0,0 +1,174 @@
package dev.ltms.fleet;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.inject.Injector;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.inject.StatusPoller;
import dev.ltms.fleet.mcp.FleetMcp;
import dev.ltms.fleet.peer.Capability;
import dev.ltms.fleet.peer.PeerHandle;
import dev.ltms.fleet.peer.PeerLauncher;
import dev.ltms.fleet.peer.SpawnRequest;
import dev.ltms.fleet.placement.PlacementDecision;
import dev.ltms.fleet.session.SessionManager;
import dev.ltms.fleet.session.SessionReaper;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import java.util.List;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* fleetd #562 follow-up (issue comment "HOLD on PR #579"): {@code Fleetd.main}'s {@code loopHealth}
* local used to be a bare {@code new FleetMcp.LoopHealthSource(poller::health, ...)} built inline,
* with nothing a test could call directly. Measured on that shape: replacing {@code
* poller::health} with a constant {@code () -> LoopWatchdog.State.RUNNING} at the call site
* compiled with 0 errors and left all 1771 existing tests green — the daemon could be changed to
* always report the {@link StatusPoller} as {@code RUNNING}, so the watchdog could never fire and
* a stalled poller would be invisible, while every test stayed green. That is exactly the false
* negative this ticket exists to prevent.
*
* <p>The five tests PR #579 added ({@code FleetMcpTest}, {@code FleetAppTest}) all build their own
* {@link FleetMcp.LoopHealthSource} directly with fixed lambdas — they prove the seam ({@code
* LoopHealthSource} reports what it is given) and nothing about what {@code Fleetd.main} actually
* gives it. This is the same hand-built-vs-config-wired shape as fleetd #561/#248/#426.
*
* <p>The fix extracts the inline {@code new} into {@link Fleetd#loopHealthSource}, a package-private
* factory in the same style as {@link Fleetd#capacitySource} and {@link Fleetd#healthCoverageSource}
* — which is exactly what makes it directly callable here. This test calls that factory with real
* {@link StatusPoller}/{@link SessionReaper} instances (never started, so no herdr or git I/O
* happens) and pins each half separately, plus the {@code reaper == null} branch: one invariant
* wired at three places needs three assertions, not one combined check whose non-zero total could
* hide a gap at any single place.
*/
class FleetdLoopHealthSourceWiringTest {
@Test
@DisplayName("the statusPoller half reports the real poller's health, not a hardcoded state")
void statusPollerHalfReflectsThePollersRealHealth() {
// Stopped without ever being started — stop() still marks the watchdog STOPPED. A poller
// that has never reported RUNNING is the discriminating case: if Fleetd.loopHealthSource
// ever hardcoded RUNNING (the exact mutation this test exists to catch), this would fail.
StatusPoller stoppedPoller = freshPoller();
stoppedPoller.stop();
SessionReaper unusedReaper = freshReaper(); // present only to satisfy the signature
FleetMcp.LoopHealthSource source = Fleetd.loopHealthSource(stoppedPoller, unusedReaper);
assertEquals(LoopWatchdog.State.STOPPED, source.statusPoller().get(),
"the statusPoller supplier must delegate to the real poller's health() — "
+ "replacing poller::health with a constant () -> RUNNING at the "
+ "Fleetd.loopHealthSource call site must fail this assertion");
}
@Test
@DisplayName("the sessionReaper half reports the real reaper's health, not a hardcoded state")
void sessionReaperHalfReflectsTheReapersRealHealth() {
StatusPoller unusedPoller = freshPoller(); // present only to satisfy the signature
SessionReaper stoppedReaper = freshReaper();
stoppedReaper.stop();
FleetMcp.LoopHealthSource source = Fleetd.loopHealthSource(unusedPoller, stoppedReaper);
assertEquals(LoopWatchdog.State.STOPPED, source.sessionReaper().get(),
"the sessionReaper supplier must delegate to the real reaper's health() — "
+ "replacing reaper.health() with a constant at the "
+ "Fleetd.loopHealthSource call site must fail this assertion");
}
@Test
@DisplayName("a null reaper (idle ttl not configured) still reports STOPPED, not a crash")
void nullReaperStillReportsStopped() {
// SessionReaper is only constructed when lifecycle.idleTtlSeconds is configured (see the
// `reaper` local in Fleetd.main) — a real deployment routinely passes null here. That null
// check is real behaviour, not a simplification to delete: it must keep reporting STOPPED
// rather than throwing a NullPointerException on the first fleet_list/healthz call.
StatusPoller runningPoller = freshPoller();
FleetMcp.LoopHealthSource source = Fleetd.loopHealthSource(runningPoller, null);
assertEquals(LoopWatchdog.State.STOPPED, source.sessionReaper().get(),
"reaper == null must still report STOPPED, exactly like an intentionally-stopped "
+ "reaper would — do not delete this null check to simplify the wiring");
}
/** Never started, so no herdr call is ever made; freshly constructed reports RUNNING. */
private static StatusPoller freshPoller() {
AgentControl agents = new AgentControl(new FakeHerdr());
return new StatusPoller(agents, new Injector(agents), 1000);
}
/** Never started, so no git/session I/O is ever made; freshly constructed reports RUNNING. */
private static SessionReaper freshReaper() {
return new SessionReaper(new SessionManager(new NeverSpawnsLauncher()), 60, 1000);
}
/**
* Same minimal shape as {@code FleetdBackendErrorSinkTest.NeverSpawnsLauncher} — every method
* throws or returns an empty/no-op value, since a {@link SessionReaper} that is only ever
* constructed and then stopped (never started) never calls any of them.
*/
private static final class NeverSpawnsLauncher implements PeerLauncher {
@Override
public Set<Capability> capabilities() {
return Set.of();
}
@Override
public Set<Capability> capabilitiesFor(String profileName) {
return Set.of();
}
@Override
public PeerHandle spawn(SpawnRequest req) {
throw new UnsupportedOperationException("not reachable — this test never acquires a session");
}
@Override
public PeerHandle spawn(SpawnRequest req, PlacementDecision decision) {
throw new UnsupportedOperationException("not reachable — this test never acquires a session");
}
@Override
public Set<String> profiles() {
return Set.of();
}
@Override
public String defaultProfile() {
return null;
}
@Override
public String effectiveCwd(SpawnRequest req) {
throw new UnsupportedOperationException("not reachable — this test never acquires a session");
}
@Override
public List<String> parityOverlay(String profileName) {
return List.of();
}
@Override
public List<?> list() {
return List.of();
}
@Override
public int reapOrphanWorkers() {
return 0;
}
@Override
public void stop(String id) {
}
@Override
public boolean clearContext(String id) {
return false;
}
}
}
@@ -0,0 +1,286 @@
package dev.ltms.fleet;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.inject.CompletionResolver;
import dev.ltms.fleet.inject.ExhaustedPatternLookup;
import dev.ltms.fleet.inject.ExhaustionSink;
import dev.ltms.fleet.inject.TurnListener;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.TurnToken;
import org.junit.jupiter.api.Test;
import java.util.LinkedHashSet;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.LongSupplier;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #561: {@code Fleetd.turnListener} composes the completion resolver and the session
* manager into one {@link TurnListener}. Each of the four callbacks below used to be two bare,
* unguarded statements (completion first, then sessions) — a throw from the session half used to
* skip nothing <em>after</em> it (there was nothing after it), but nothing enforced that the
* completion half had to come first either, beyond call order in the source. {@code onDelivered}
* had the identical shape and was fixed by #556 (moving its registration off the fan-out
* entirely); these four callbacks cannot be fixed that way, so the fan-out itself is hardened
* instead (see {@link Fleetd#turnListener} and {@code Fleetd.bothMustRun}/{@code
* bothMustRunKeepingSecondResult}).
*
* <p>The invariant under test: <b>a throwing session-listener must not prevent the completion
* resolver from being told the turn ended.</b> Each test below builds the exact production
* composition ({@link Fleetd#turnListener}) from a real {@link CompletionResolver} and a fake
* {@code sessions} half that throws, then asserts the completion half's effect (the captured
* {@link Rendezvous} waiter resolving) happened anyway — never by inspecting call order directly.
*/
class FleetdTurnListenerCompositionTest {
/** Records which callbacks ran and can be told to throw from a chosen one. */
private static final class RecordingSessions implements TurnListener {
final Set<String> called = new LinkedHashSet<>();
private final Set<String> throwing;
RecordingSessions(String... throwingMethods) {
this.throwing = Set.of(throwingMethods);
}
private void maybeThrow(String method) {
called.add(method);
if (throwing.contains(method)) {
throw new IllegalStateException("boom: sessions." + method);
}
}
@Override
public void onTurnComplete(String target) {
maybeThrow("onTurnComplete");
}
@Override
public boolean onTurnCompleteWithPostAction(String target) {
maybeThrow("onTurnCompleteWithPostAction");
return true;
}
@Override
public void onTurnFailed(String target) {
maybeThrow("onTurnFailed");
}
@Override
public void onTurnFailed(String target, String reason) {
maybeThrow("onTurnFailedWithReason");
}
}
private static CompletionResolver newResolver(FakeHerdr herdr, Rendezvous rendezvous) {
return new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
}
/**
* A resolver with a controllable clock, and the clock itself, so a test can register a turn
* "delivered" at time 0 and then jump the clock past {@link CompletionResolver#MIN_TURN_NANOS}
* before resolving it — otherwise {@code onTurnComplete}/{@code onTurnCompleteWithPostAction}
* resolve within microseconds of registering in-test, well inside the fleetd#164 floor, and
* get classified as a too-fast crash rather than a real completion. Mirrors the {@code
* LongSupplier} clock-injection pattern fleetd#164's own tests use.
*/
private static CompletionResolver newResolverPastTheFloor(FakeHerdr herdr, Rendezvous rendezvous,
AtomicLong clock) {
LongSupplier nowNanos = clock::get;
return new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none(), nowNanos);
}
@Test
void onTurnCompleteResolvesTheWaiterEvenWhenTheSessionHalfThrows() throws Exception {
FakeHerdr herdr = new FakeHerdr().readText("⏺ real answer\n❯ ");
Rendezvous rendezvous = new Rendezvous();
AtomicLong clock = new AtomicLong(0);
CompletionResolver completion = newResolverPastTheFloor(herdr, rendezvous, clock);
var waiter = rendezvous.open("term_a");
completion.register("term_a", new TurnToken("term_a", waiter)); // delivered at clock=0
clock.set(CompletionResolver.MIN_TURN_NANOS + 1_000_000); // past the too-fast floor
RecordingSessions sessions = new RecordingSessions("onTurnComplete");
TurnListener composed = Fleetd.turnListener(completion, sessions);
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> composed.onTurnComplete("term_a"),
"the session half's throw must still escape the composed listener");
assertEquals("boom: sessions.onTurnComplete", thrown.getMessage());
assertTrue(sessions.called.contains("onTurnComplete"), "the session half must have run");
// onTurnComplete resolves off-thread (a virtual thread) — wait on the waiter itself,
// exactly like MessageServiceTest.completionFallbackIsNeverQueued does.
Rendezvous.Resolution resolution = waiter.get(5, TimeUnit.SECONDS);
assertEquals(Rendezvous.Kind.COMPLETION, resolution.kind(),
"the completion resolver must still have resolved the send, despite the session "
+ "half throwing");
assertEquals("real answer", resolution.text());
}
@Test
void onTurnFailedResolvesTheWaiterAsFailedEvenWhenTheSessionHalfThrows() throws Exception {
FakeHerdr herdr = new FakeHerdr().readText("⏺ crash context\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = newResolver(herdr, rendezvous);
var waiter = rendezvous.open("term_b");
completion.register("term_b", new TurnToken("term_b", waiter));
RecordingSessions sessions = new RecordingSessions("onTurnFailed");
TurnListener composed = Fleetd.turnListener(completion, sessions);
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> composed.onTurnFailed("term_b"),
"the session half's throw must still escape the composed listener");
assertEquals("boom: sessions.onTurnFailed", thrown.getMessage());
assertTrue(sessions.called.contains("onTurnFailed"), "the session half must have run");
Rendezvous.Resolution resolution = waiter.get(5, TimeUnit.SECONDS);
assertEquals(Rendezvous.Kind.FAILED, resolution.kind(),
"the completion resolver must still have failed the send, despite the session "
+ "half throwing");
}
@Test
void onTurnFailedWithReasonResolvesTheWaiterEvenWhenTheSessionHalfThrows() throws Exception {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = newResolver(herdr, rendezvous);
var waiter = rendezvous.open("term_c");
completion.register("term_c", new TurnToken("term_c", waiter));
// fleetd #561: the composed listener calls sessions.onTurnFailed(target) — the ONE-arg
// overload — for this reason-carrying event too (matching the pre-existing production
// behaviour: SessionManager never overrides the two-arg overload either), so the
// throwing key here is "onTurnFailed", not a distinct "...WithReason" one.
RecordingSessions sessions = new RecordingSessions("onTurnFailed");
TurnListener composed = Fleetd.turnListener(completion, sessions);
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> composed.onTurnFailed("term_c", "worker unreachable"),
"the session half's throw must still escape the composed listener");
assertEquals("boom: sessions.onTurnFailed", thrown.getMessage());
assertTrue(sessions.called.contains("onTurnFailed"), "the session half must have run");
Rendezvous.Resolution resolution = waiter.get(5, TimeUnit.SECONDS);
assertEquals(Rendezvous.Kind.FAILED, resolution.kind(),
"the completion resolver must still have failed the send, despite the session "
+ "half throwing");
assertEquals("worker unreachable", resolution.text(),
"the explicit reason must still reach the resolved send");
}
@Test
void onTurnCompleteWithPostActionResolvesTheWaiterEvenWhenTheSessionHalfThrows() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ answer before reset\n❯ ");
Rendezvous rendezvous = new Rendezvous();
AtomicLong clock = new AtomicLong(0);
CompletionResolver completion = newResolverPastTheFloor(herdr, rendezvous, clock);
var waiter = rendezvous.open("term_d");
completion.register("term_d", new TurnToken("term_d", waiter)); // delivered at clock=0
clock.set(CompletionResolver.MIN_TURN_NANOS + 1_000_000); // past the too-fast floor
RecordingSessions sessions = new RecordingSessions("onTurnCompleteWithPostAction");
TurnListener composed = Fleetd.turnListener(completion, sessions);
IllegalStateException thrown = assertThrows(IllegalStateException.class,
() -> composed.onTurnCompleteWithPostAction("term_d"),
"the session half's throw must still escape the composed listener");
assertEquals("boom: sessions.onTurnCompleteWithPostAction", thrown.getMessage());
assertTrue(sessions.called.contains("onTurnCompleteWithPostAction"), "the session half must have run");
// resolveBeforePostAction is synchronous by design (it must run before the context reset
// can erase the pane) — the waiter is already resolved by the time the throw propagates.
assertTrue(waiter.isDone(), "resolveBeforePostAction is synchronous — the send must "
+ "already be resolved once the composed call returns (by throwing)");
assertEquals(Rendezvous.Kind.COMPLETION, waiter.getNow(null).kind());
assertEquals("answer before reset", waiter.getNow(null).text());
}
/**
* The mirror case (comment 17041's acceptance item 2): the completion half throws, and the
* session half still ran. The production {@link CompletionResolver} is deliberately defensive
* (its scrape reads are wrapped in {@code catch (RuntimeException)}, by the same fail-open
* design {@link CompletionResolver#captureBaseline} documents) so it essentially never throws
* synchronously in normal operation — forcing it to do so needs a genuinely-real trigger, not
* a fabricated one. {@code ConcurrentHashMap.get(null)} is that trigger: passing a {@code null}
* target makes {@code onTurnComplete}'s {@code inFlight.get(target)} throw a
* {@link NullPointerException} before it ever starts its resolving thread — a real code path,
* not a contrived one.
*/
@Test
void sessionHalfStillRunsWhenTheCompletionHalfThrowsSynchronously() {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = newResolver(herdr, rendezvous);
RecordingSessions sessions = new RecordingSessions(); // throws from nothing
TurnListener composed = Fleetd.turnListener(completion, sessions);
assertThrows(NullPointerException.class, () -> composed.onTurnComplete(null),
"ConcurrentHashMap.get(null) inside CompletionResolver.onTurnComplete must still "
+ "escape the composed listener");
assertTrue(sessions.called.contains("onTurnComplete"),
"the session half must still have run even though the completion half threw first");
}
/**
* The mirror case above only exercises {@code onTurnComplete}/{@code bothMustRun}. {@code
* onTurnCompleteWithPostAction} is composed through the OTHER helper,
* {@code bothMustRunKeepingSecondResult}, and nothing previously asserted that its session half
* still runs when its completion half throws — that helper could be reverted to the pre-#561
* broken shape (run the session half only if the completion half did not throw) and the suite
* would still stay green. Uses the same real, non-fabricated trigger as the test above: a
* {@code null} target makes {@code resolveBeforePostAction}'s {@code inFlight.get(target)}
* throw a {@link NullPointerException} before {@code resolve} is ever entered.
*/
@Test
void sessionHalfStillRunsWhenTheCompletionHalfThrowsSynchronouslyForPostAction() {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = newResolver(herdr, rendezvous);
RecordingSessions sessions = new RecordingSessions(); // throws from nothing
TurnListener composed = Fleetd.turnListener(completion, sessions);
assertThrows(NullPointerException.class, () -> composed.onTurnCompleteWithPostAction(null),
"ConcurrentHashMap.get(null) inside CompletionResolver.resolveBeforePostAction must "
+ "still escape the composed listener");
assertTrue(sessions.called.contains("onTurnCompleteWithPostAction"),
"the session half must still have run even though the completion half threw first");
}
/**
* {@code bothMustRun} must not just run both halves — it must not DROP a second failure when
* both halves throw. Forces the completion half to throw (the same real {@code
* inFlight.get(null)} NullPointerException trigger used above) while the session half throws a
* distinct {@link IllegalStateException}, and asserts the completion half's throwable is what
* escapes while the session half's throwable survives as a suppressed exception rather than
* being silently discarded.
*/
@Test
void bothFailuresEscapeWhenBothHalvesThrowDistinctExceptions() {
FakeHerdr herdr = new FakeHerdr();
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = newResolver(herdr, rendezvous);
RecordingSessions sessions = new RecordingSessions("onTurnComplete");
TurnListener composed = Fleetd.turnListener(completion, sessions);
NullPointerException thrown = assertThrows(NullPointerException.class,
() -> composed.onTurnComplete(null),
"the completion half's throw (NPE from inFlight.get(null)) must be what escapes");
assertTrue(sessions.called.contains("onTurnComplete"), "the session half must still have run");
assertEquals(1, thrown.getSuppressed().length,
"the session half's distinct failure must be recorded as suppressed, not dropped");
assertEquals(IllegalStateException.class, thrown.getSuppressed()[0].getClass());
assertEquals("boom: sessions.onTurnComplete", thrown.getSuppressed()[0].getMessage());
}
}
@@ -345,7 +345,7 @@ class FleetConfigTest {
IllegalStateException unknownError = assertThrows(IllegalStateException.class,
() -> FleetConfig.load(unknown).validateCharters());
assertTrue(unknownError.getMessage().contains("architetc"));
assertTrue(unknownError.getMessage().contains("[architect, dev, reviewer]"));
assertTrue(unknownError.getMessage().contains("[architect, dev, hunter, reviewer]"));
}
/**
@@ -812,13 +812,17 @@ class FleetConfigTest {
reviewers:
b:
profile: sonnet
hunters:
c:
profile: sonnet
""");
FleetConfig cfg = FleetConfig.load(f);
assertEquals(List.of("sonnet"), cfg.fleet().profilesFor(MemberRole.DEV));
assertEquals(List.of("sonnet"), cfg.fleet().profilesFor(MemberRole.HUNTER));
assertEquals(List.of("sonnet"), cfg.fleet().profilesFor(MemberRole.REVIEWER));
assertTrue(cfg.fleet().profilesFor(MemberRole.ARCHITECT).isEmpty());
assertEquals(List.of(MemberRole.DEV, MemberRole.REVIEWER), cfg.fleet().rolesConfigured());
assertEquals(List.of(MemberRole.DEV, MemberRole.HUNTER, MemberRole.REVIEWER), cfg.fleet().rolesConfigured());
}
/** The case the two axes exist for: one backend, two roles, and neither is a duplicate. */
@@ -0,0 +1,40 @@
package dev.ltms.fleet.health;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* fleetd #426: {@link FleetHealthMonitor#coverage} had zero references anywhere in the test tree —
* not the method, not either output string, not the field it populates. This pins the method's own
* three branches directly.
*
* <p>This is the easy half. It proves the method words each combination correctly, but it proves
* nothing about whether {@code Fleetd.java}'s two call sites pass the right argument in the right
* position — see {@code FleetdHealthCoverageSourceWiringTest} (package {@code dev.ltms.fleet}) for
* the half that actually guards the call site, following the measured fleetd #415 lesson that a
* thoroughly-tested method and an untested argument pairing at its call site are different risks.
*
* <p><strong>The three output strings are load-bearing and must not change here.</strong> {@code
* "detection-only"} is read live off a running daemon's {@code fleet_list} today (measured
* 2026-09-12) — this test intentionally asserts the exact literal strings so a future edit to the
* wording trips it here first.
*/
class FleetHealthMonitorCoverageTest {
@Test
void disabledIsOffRegardlessOfNotificationConfig() {
assertEquals("off", FleetHealthMonitor.coverage(false, false));
assertEquals("off", FleetHealthMonitor.coverage(false, true));
}
@Test
void enabledWithoutNotificationsIsDetectionOnly() {
assertEquals("detection-only", FleetHealthMonitor.coverage(true, false));
}
@Test
void enabledWithNotificationsIsFull() {
assertEquals("full", FleetHealthMonitor.coverage(true, true));
}
}
@@ -19,6 +19,7 @@ import java.util.regex.Pattern;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Unit behaviour of the CB-106 completion resolver in isolation from the injector. */
@@ -519,6 +520,254 @@ class CompletionResolverTest {
assertTrue(rendezvous.isWaiting("term_a"), "turn N+1 is still awaiting its own resolution");
}
// --- fleetd #556 (comment 16908): name the invariant the two-arg remove(target, turn) idiom
// exists to maintain — a superseded turn's terminal handling must not evict its successor's
// registration — and test THAT, not the idiom's literal spelling. Three entry points reach a
// conditional remove: resolve()'s plain completion path, resolve()'s echoed-brief/noReportMessage
// sub-path, and fail(). This is a non-goal-to-break for fleetd #556's redesign: flattening any of
// these removes to the one-arg form must turn one of these three red. -------------------------
@Test
void aSupersededTurnsPlainCompletionMustNotEvictItsSuccessorsRegistration() {
// Turn A is overtaken by turn B on the same target (a rapid back-to-back send): B's own
// delivery overwrites the map entry A's delivery installed. A's late completion fallback
// then finally fires and reaches resolve()'s ordinary (non-echoed) completion branch — the
// two-arg remove(target, turnA) there must be a no-op once the map holds B, not a blind
// evict.
FakeHerdr herdr = new FakeHerdr().readText("⏺ A's pre-turn pane\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null); // baseline null: fails open, never suppressed
rendezvous.close("term_a", waiterA); // A's own send's finally, freeing the session for B
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB)); // the injector registering B
herdr.readText("⏺ A's late, genuine answer\n❯ "); // what the pane shows when A's fallback fires
resolver.resolve("term_a", turnA);
assertTrue(waiterA.isDone(), "A's own resolve must still complete A's waiter normally");
assertEquals(Rendezvous.Kind.COMPLETION, waiterA.getNow(null).kind());
CompletionResolver.InFlight afterA = resolver.inFlight("term_a");
assertNotNull(afterA, "B's registration must still be present — a one-arg remove(target) "
+ "here would evict it even though the map no longer holds turnA");
assertEquals(waiterB, afterA.waiter(), "the surviving entry must still be B's, untouched by A's "
+ "terminal handling");
assertTrue(rendezvous.resolve("term_a", "B replied"),
"B must still resolve normally through the ordinary reply path");
assertEquals(Rendezvous.Kind.REPLY, waiterB.getNow(null).kind());
}
@Test
void aSupersededTurnsEchoedNoReportSubPathMustNotEvictItsSuccessorsRegistration() {
// Same invariant, driven down resolve()'s OTHER branch: a pane that merely echoes the
// injected brief back (no real report) resolves through noReportMessage(), not the plain
// tail — comment 16908 calls this "a sub-path of the first, not a separate method", reaching
// the same terminal remove() call site from different logic above it.
String echoedBrief = "y".repeat(450); // >= CompletionResolver.ECHO_MIN_CHARS normalised chars
FakeHerdr herdr = new FakeHerdr().readText("⏺ " + echoedBrief + "\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
// Back-dated deliveredAtNanos (like the 2-arg InFlight convenience ctor) so MIN_TURN_NANOS
// never fires; injectedText == the pane's own content so echoesInjectedBrief() is true.
var turnA = new CompletionResolver.InFlight(waiterA, null, Long.MIN_VALUE / 2, echoedBrief);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertTrue(waiterA.isDone(), "A's own resolve must still complete A's waiter via the echoed "
+ "no-report branch");
assertTrue(waiterA.getNow(null).text().contains(CompletionResolver.NO_REPORT_PREFIX),
"sanity: A must actually have gone down the noReportMessage sub-path, not the plain one");
CompletionResolver.InFlight afterA = resolver.inFlight("term_a");
assertNotNull(afterA, "B's registration must still be present after A's echoed-brief resolve");
assertEquals(waiterB, afterA.waiter());
assertTrue(rendezvous.resolve("term_a", "B replied"),
"B must still resolve normally through the ordinary reply path");
assertEquals(Rendezvous.Kind.REPLY, waiterB.getNow(null).kind());
}
@Test
void aSupersededTurnsFailMustNotEvictItsSuccessorsRegistration() {
// The fail() entry point (CB-109 wedge / CB-110 drop): turn A is failed while turn B has
// already superseded it in the map. fail()'s own two-arg remove(target, turnA) must be a
// no-op once the map holds B.
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(new FakeHerdr()), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.fail("term_a", turnA, "A wedged in an unknown state");
assertTrue(waiterA.isDone(), "A's own fail must still complete A's waiter");
assertEquals(Rendezvous.Kind.FAILED, waiterA.getNow(null).kind());
CompletionResolver.InFlight afterA = resolver.inFlight("term_a");
assertNotNull(afterA, "B's registration must still be present — a one-arg remove(target) in "
+ "fail() would evict it even though the map no longer holds turnA");
assertEquals(waiterB, afterA.waiter());
assertTrue(rendezvous.resolve("term_a", "B replied"),
"B must still resolve normally through the ordinary reply path");
assertEquals(Rendezvous.Kind.REPLY, waiterB.getNow(null).kind());
}
@Test
void aSupersededDoneTurnMustNotEvictItsSuccessorsRegistration() {
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(new FakeHerdr()), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
assertTrue(rendezvous.resolve("term_a", "A replied"));
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a done turn must not evict B from resolve()'s early return");
}
@Test
void aSupersededExhaustedTurnMustNotEvictItsSuccessorsRegistration() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ usage limit has been reached\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
target -> Pattern.compile("usage limit has been reached"), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"an exhausted turn must not evict B from resolve()'s exhausted branch");
}
@Test
void aSupersededBackendErrorTurnMustNotEvictItsSuccessorsRegistration() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ API Error: 400 invalid request body\n❯ ");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a backend-error turn must not evict B from resolve()'s error branch");
}
@Test
void aSupersededRawExhaustedTurnMustNotEvictItsSuccessorsRegistration() {
FakeHerdr herdr = new FakeHerdr().readText("╭────\nusage limit has been reached");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
target -> Pattern.compile("usage limit has been reached"), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a raw exhausted turn must not evict B from the raw-scrape exhausted branch");
}
@Test
void aSupersededRawBackendErrorTurnMustNotEvictItsSuccessorsRegistration() {
FakeHerdr herdr = new FakeHerdr().readText("╭────\nAPI Error: 400 invalid request body");
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a raw backend-error turn must not evict B from the raw-scrape error branch");
}
@Test
void aSupersededDoneFailedTurnMustNotEvictItsSuccessorsRegistration() {
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(new FakeHerdr()), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null);
assertTrue(rendezvous.resolve("term_a", "A replied"));
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
resolver.fail("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a done failed turn must not evict B from fail()'s early return");
}
@Test
void aSupersededTooFastBackendErrorTurnMustNotEvictItsSuccessorsRegistration() {
FakeHerdr herdr = new FakeHerdr().readText("⏺ API Error: 400 invalid request body\n❯ ");
Rendezvous rendezvous = new Rendezvous();
long[] clock = {10_000_000_000L};
CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none(), () -> clock[0]);
var waiterA = rendezvous.open("term_a");
var turnA = new CompletionResolver.InFlight(waiterA, null, clock[0]);
rendezvous.close("term_a", waiterA);
var waiterB = rendezvous.open("term_a");
resolver.captureBaseline("term_a", new TurnToken("term_a", waiterB));
clock[0] += CompletionResolver.MIN_TURN_NANOS - 1;
resolver.resolve("term_a", turnA);
assertSuccessorRegistrationSurvives(resolver, rendezvous, waiterB,
"a too-fast backend-error turn must not evict B from failTooFast()");
}
private static void assertSuccessorRegistrationSurvives(CompletionResolver resolver, Rendezvous rendezvous,
Object waiterB, String message) {
CompletionResolver.InFlight afterA = resolver.inFlight("term_a");
assertNotNull(afterA, message + " — a one-arg remove(target) would remove B");
assertEquals(waiterB, afterA.waiter(), message + " — the surviving record must belong to B");
assertTrue(rendezvous.resolve("term_a", "B replied"), message + " — B must still resolve normally");
}
// --- CB-578 stage A: backend-exhausted classification ---------------------------------
@Test
@@ -2,11 +2,15 @@ package dev.ltms.fleet.inject;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.databind.JsonNode;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.msg.Rendezvous;
import dev.ltms.fleet.msg.TestTurnTokens;
import dev.ltms.fleet.msg.TurnToken;
import dev.ltms.fleet.testing.CapturedLog;
import org.junit.jupiter.api.Test;
@@ -664,4 +668,783 @@ class InjectorTest {
ExecutionException ex = assertThrows(ExecutionException.class, f::get);
assertInstanceOf(HerdrException.class, ex.getCause());
}
/**
* A {@link HerdrClient} that throws a non-{@link RuntimeException} {@link Error} from {@code
* agent.prompt} instead of delegating — the fleetd #546 case: a stray non-RuntimeException
* throwable (e.g. a {@code NoClassDefFoundError}, fleetd #413) escaping the send seam at
* {@code Injector.java:383}. Records every call it sees itself, including the ones it throws
* for, since the delegate's own recording is never reached for {@code agent.prompt} — so a test
* can assert on exactly what this fake actually received.
*/
private static final class ErrorOnPrompt implements HerdrClient {
private final FakeHerdr delegate;
private final List<FakeHerdr.Call> calls = new java.util.concurrent.CopyOnWriteArrayList<>();
private ErrorOnPrompt(FakeHerdr delegate) {
this.delegate = delegate;
}
List<FakeHerdr.Call> calls() {
return calls;
}
@Override
public JsonNode call(String method, Object params) {
calls.add(new FakeHerdr.Call(method, params));
if (method.equals("agent.prompt")) {
throw new AssertionError("simulated non-RuntimeException send failure (fleetd #546)");
}
return delegate.call(method, params);
}
@Override
public void close() {
delegate.close();
}
}
/**
* Mirrors StatusPoller's own per-target {@code catch (Throwable)} (fleetd #538 / PR #543): the
* production polling loop already swallows whatever escapes one target's round and comes back
* for the next one. A unit test that calls {@code onStatus} directly (bypassing StatusPoller)
* needs the same survival so it can observe what a SECOND round does, regardless of whether
* fleetd #546's fix is present.
*/
private static void pollOnceSurviving(Injector inj, String target, AgentStatus status) {
try {
inj.onStatus(target, status);
} catch (Throwable ignored) {
// matches StatusPoller.loop's own catch (Throwable) added by PR #543
}
}
@Test
void anErrorFromSendRemovesTheMessageAndMarksItAttempted() {
// fleetd #546, acceptance test 1: an Error (not a RuntimeException) escaping the send seam
// must still be caught and the message dropped from the queue, never left QUEUED at the
// head. fleetd #551 updates what it is marked: an AssertionError from the send call itself
// proves nothing about whether the text reached the pane, so it is recorded as ATTEMPTED
// (uncertain), not a confident NOT_DELIVERED — see anErrorFromSendMarksItNotDeliveredOnlyForANotFoundCode
// below for the one case that still gets NOT_DELIVERED.
ErrorOnPrompt throwing = new ErrorOnPrompt(new FakeHerdr());
Injector inj = new Injector(new AgentControl(throwing));
Injector.Delivery delivery = inj.enqueue(T, "brief", TestTurnTokens.inert(T));
assertDoesNotThrow(() -> inj.onStatus(T, AgentStatus.IDLE),
"fleetd #546: an Error from the send seam must be caught inside onStatus, not "
+ "escape it");
assertTrue(delivery.completion().isCompletedExceptionally(),
"the delivery's future must surface the send failure");
assertEquals(Injector.Cancellation.ATTEMPTED, inj.cancel(delivery),
"fleetd #551: the message must be dropped and marked ATTEMPTED (not a confident "
+ "NOT_DELIVERED), and never left QUEUED at the head of the queue");
}
@Test
void anErrorFromSendDoesNotRedeliverOnASecondRound() {
// fleetd #546, acceptance test 2 — the test this ticket exists for. Before the fix, an
// Error at the send seam left the message QUEUED (Injector.java:378 peeks, not polls), so a
// second onStatus round re-entered the same try and sent the same text again: the member's
// pane got the same brief typed into it twice.
ErrorOnPrompt throwing = new ErrorOnPrompt(new FakeHerdr());
Injector inj = new Injector(new AgentControl(throwing));
inj.enqueue(T, "brief", TestTurnTokens.inert(T));
pollOnceSurviving(inj, T, AgentStatus.IDLE);
pollOnceSurviving(inj, T, AgentStatus.IDLE);
long promptCalls = throwing.calls().stream().filter(c -> c.method().equals("agent.prompt")).count();
assertEquals(1, promptCalls, "fleetd #546: the poisoned text must be sent exactly once — a "
+ "second onStatus round must not re-enter send for the same message");
}
@Test
void aHerdrExceptionFromSendStillSurfacesButNowReportsAttempted() {
// fleetd #546, acceptance test 3 (control): HerdrException extends RuntimeException, so it
// was already caught before #546's widening. The send-failure/surface behaviour is
// unchanged by #551 — still dropped, still surfaced to the caller — but fleetd #551
// deliberately changes WHAT it is marked: "send_failed" is not a herdr `*_not_found` code,
// so this is exactly the response-half case #551 exists to fix. See
// aHerdrExceptionAfterThePasteIsNeverRecordedAsConfidentlyNotDelivered for the acceptance
// test this ticket was filed for.
FakeHerdr failing = new FakeHerdr().agentSendFailsWith("send_failed");
Injector inj = new Injector(new AgentControl(failing));
Injector.Delivery delivery = inj.enqueue(T, "boom", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE);
assertTrue(delivery.completion().isCompletedExceptionally(),
"a HerdrException at the send seam must still surface to the caller, unchanged by "
+ "fleetd #546's widening");
assertEquals(Injector.Cancellation.ATTEMPTED, inj.cancel(delivery),
"fleetd #551: a non-*_not_found HerdrException must now report ATTEMPTED, not a "
+ "confident NOT_DELIVERED");
}
// --- fleetd #551: the injector records the delivery attempt BEFORE the irreversible send, not
// after, so a failure in the send's response window can never be written down as a confident
// NOT_DELIVERED for text that may already be sitting in the worker's pane. ---
@Test
void aHerdrExceptionAfterThePasteIsNeverRecordedAsConfidentlyNotDelivered() {
// fleetd #551, the ticket's acceptance test 1, and comment 17037's added test ("a test
// pinning that a caller cannot receive a confident NOT_DELIVERED for a message that reached
// the paste"): a HerdrException thrown from the response half of the send seam — herdr
// replied, with an error, so it definitely processed the request — must not leave a record
// claiming the text was never delivered. Red before the fix: on main at ba2f4d1 this
// asserted (and got) Injector.Cancellation.NOT_DELIVERED.
FakeHerdr failing = new FakeHerdr().agentSendFailsWith("send_failed");
Injector inj = new Injector(new AgentControl(failing));
Injector.Delivery delivery = inj.enqueue(T, "brief", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE);
assertTrue(delivery.completion().isCompletedExceptionally(),
"the caller must still see the send failure");
assertEquals(Injector.Cancellation.ATTEMPTED, inj.cancel(delivery),
"fleetd #551: a HerdrException from the response half of the send seam must not be "
+ "recorded as a confident NOT_DELIVERED — the text may already be sitting "
+ "in the pane");
}
@Test
void ordinarySuccessStillReportsDeliveredExactlyOnce() {
// fleetd #551, the ticket's acceptance test 2: the record-before-send reorder must not
// change the ordinary success path — exactly one send, and the caller-facing Cancellation
// for it is still DELIVERED, never left at the new ATTEMPTED value.
Injector.Delivery delivery = injector.enqueue(T, "hello", TestTurnTokens.inert(T));
injector.onStatus(T, AgentStatus.IDLE);
assertEquals(List.of("hello"), sent(), "the text must be sent exactly once");
assertTrue(delivery.completion().isDone() && !delivery.completion().isCompletedExceptionally(),
"the ordinary success path must still complete normally");
assertEquals(Injector.Cancellation.DELIVERED, injector.cancel(delivery),
"fleetd #551: the ordinary success path must still report DELIVERED, unaffected by "
+ "the record-before-send reorder");
}
@Test
void transportDownStillSurfacesTheErrorAndDropsTheEntry() {
// fleetd #551, the ticket's acceptance test 3: the ordinary transport-down path (herdr
// unreachable — a plain HerdrException with no error code) must be unaffected by the #551
// reorder — the failure still surfaces to the caller, and the entry is not left sitting in
// the queue for a second onStatus round to resend.
FakeHerdr unreachable = new FakeHerdr().healthy(false);
Injector inj = new Injector(new AgentControl(unreachable));
Injector.Delivery delivery = inj.enqueue(T, "brief", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE);
assertTrue(delivery.completion().isCompletedExceptionally(),
"a transport-down failure must still surface to the caller");
assertTrue(inj.activeTargets().isEmpty(),
"the entry must not be left queued for a second round to resend");
}
@Test
void aNotFoundCodeFromSendStillReportsAConfidentNotDelivered() {
// fleetd #551 (comment 17037, point 3): a herdr `*_not_found` error means the target
// pane/agent does not exist at all, so nothing could have been pasted anywhere — the
// codebase already treats this family as a confirmed absence everywhere else (StatusPoller,
// AgentControl's own retry). #551's new ATTEMPTED state must not swallow this case: it stays
// a confident NOT_DELIVERED.
FakeHerdr failing = new FakeHerdr().agentSendFailsWith("agent_not_found");
Injector inj = new Injector(new AgentControl(failing));
Injector.Delivery delivery = inj.enqueue(T, "brief", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE);
assertTrue(delivery.completion().isCompletedExceptionally(),
"an agent_not_found failure must still surface to the caller");
assertEquals(Injector.Cancellation.NOT_DELIVERED, inj.cancel(delivery),
"fleetd #551: a herdr *_not_found error is a confirmed absence, not merely "
+ "inconclusive — it must stay NOT_DELIVERED, not the new ATTEMPTED");
}
// --- fleetd #553: a throwable from any listener callback in onStatus must not skip the
// delivered-future completion. The whole post-monitor region is now wrapped in try/finally. ---
@Test
void theOriginalThrowableFromAListenerStillEscapesOnStatus() {
// fleetd #553's control: the entire point of the finally backstop is that it completes a
// future WITHOUT swallowing whatever escaped. A fix built the wrong way (e.g. catching
// Throwable in the finally, or wrapping the region in try/catch instead of try/finally)
// could make every other test in this group pass while still converting a loud listener
// bug into a silent one — which the ticket calls a worse outcome than the bug itself. This
// is deliberately its own test, not folded into another one's assertion.
RuntimeException boom = new RuntimeException("fleetd #553 control");
TurnListener throwing = new TurnListener() {
@Override
public void onTurnComplete(String target) {
throw boom;
}
};
Injector inj = new Injector(new AgentControl(herdr), throwing);
inj.enqueue(T, "only", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE); // deliver
inj.onStatus(T, AgentStatus.WORKING); // pickup
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE),
"onStatus must still propagate the listener's own throwable, unmodified");
assertSame(boom, thrown, "must be the EXACT throwable, not a wrapper or a different instance");
}
@Test
void aRuntimeExceptionFromOnTurnCompleteStillCompletesTheNextDelivery() {
// fleetd #553, acceptance test 1: onTurnComplete fires for "first"'s completed turn INSIDE
// the same onStatus call that then peeks and delivers "second" — the turnCompleted check
// (Injector.java, inside the released-pickup branch) clears awaitingCompletion just before
// the delivery guard right below it is evaluated, so both happen in one round when there is
// no post-turn action. Before fleetd #553, a RuntimeException thrown here unwound straight
// out of onStatus and skipped the `if (sent != null)` block, leaving "second"'s future
// pending forever even though it was already off the queue, marked DELIVERED, and typed
// into the pane.
RuntimeException boom = new RuntimeException("boom from onTurnComplete");
TurnListener throwing = new TurnListener() {
@Override
public void onTurnComplete(String target) {
throw boom;
}
};
Injector inj = new Injector(new AgentControl(herdr), throwing);
inj.enqueue(T, "first", TestTurnTokens.inert(T));
Injector.Delivery second = inj.enqueue(T, "second", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE); // delivers "first"
inj.onStatus(T, AgentStatus.WORKING); // picked up
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE),
"the throwable from onTurnComplete must still escape onStatus");
assertSame(boom, thrown);
assertEquals(List.of("first", "second"), sent(),
"fleetd #553: \"second\" must still be sent even though onTurnComplete threw for "
+ "\"first\"'s completion");
assertTrue(second.completion().isDone() && !second.completion().isCompletedExceptionally(),
"fleetd #553: \"second\"'s delivered future must still complete normally despite "
+ "the throw");
}
private static final int READINESS_GRACE_SAMPLES = 240; // Injector.READINESS_GRACE_POLLS is private
@Test
void aRuntimeExceptionFromForgetStillCompletesTheReadinessFailureFutures() {
// fleetd #553: forget.accept fires as part of the CB-114 readiness-grace path (production
// wires it to presence::forget). Before fleetd #553, that block called forget.accept
// BEFORE completing the queued messages' futures, so a RuntimeException from forget left
// every one of them pending forever even though they were already marked NOT_DELIVERED and
// dropped from the queue. fleetd #553 completes those futures first, so forget.accept
// throwing afterward can no longer un-complete them.
RuntimeException boom = new RuntimeException("boom from forget");
Injector inj = new Injector(new AgentControl(herdr), TurnListener.NOOP, _ -> false, _ -> {
throw boom;
});
Injector.Delivery delivery = inj.enqueue(T, "task", TestTurnTokens.inert(T));
// Build the not-ready streak up to (but not past) the grace threshold.
for (int i = 0; i < READINESS_GRACE_SAMPLES - 1; i++) inj.onStatus(T, AgentStatus.IDLE);
assertFalse(delivery.completion().isDone(), "must not be resolved before the grace expires");
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE),
"the throwable from forget.accept must still escape onStatus");
assertSame(boom, thrown);
assertTrue(delivery.completion().isCompletedExceptionally(),
"fleetd #553: the queued message's future must still be completed even though "
+ "forget.accept threw");
}
@Test
void aRuntimeExceptionFromOnTurnFailedStillLeavesTheReadinessFailureFuturesCompleted() {
// fleetd #553, acceptance test: same readiness-grace path as the forget test above, but the
// throw comes from onTurnFailed instead. NOTE: onTurnFailed is the LAST statement in this
// block (both before and after this ticket), so the queued messages' futures are already
// completed by the time it runs, regardless of this fix — this test cannot be made to fail
// against the pre-fleetd-553 code the way the onTurnComplete/forget/postAction tests can; I
// could not find a construction where onTurnFailed's own throw is what stands between a
// future and its completion (flagged to the lead via fleet_ask; no reply arrived before the
// ~55s window closed, so recorded here instead). It still pins a real invariant this ticket
// cares about — an ordinary RuntimeException from this callback must not un-complete a
// future that was already decided — so it stays as a regression lock, not a bug-fix proof.
RuntimeException boom = new RuntimeException("boom from onTurnFailed");
TurnListener throwing = new TurnListener() {
@Override
public void onTurnComplete(String target) {
}
@Override
public void onTurnFailed(String target) {
throw boom;
}
};
Injector inj = new Injector(new AgentControl(herdr), throwing, _ -> false, _ -> {
});
Injector.Delivery delivery = inj.enqueue(T, "task", TestTurnTokens.inert(T));
for (int i = 0; i < READINESS_GRACE_SAMPLES - 1; i++) inj.onStatus(T, AgentStatus.IDLE);
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE),
"the throwable from onTurnFailed must still escape onStatus");
assertSame(boom, thrown);
assertTrue(delivery.completion().isCompletedExceptionally(),
"the queued message's future must remain completed despite onTurnFailed throwing");
}
@Test
void aRuntimeExceptionFromOnTurnCompleteWithPostActionStillUnwedgesTheTarget() {
// fleetd #553, acceptance test: before this ticket, a RuntimeException from
// onTurnCompleteWithPostAction skipped the `t.postTurnPending = false` reset that used to
// sit right after it (with nothing guarding it), permanently wedging the target —
// postTurnPending stayed true forever, so the delivery guard never passed again and
// "second" (already queued) was never delivered, no matter how many further onStatus
// rounds ran. fleetd #553 wraps that call so the reset always runs.
RuntimeException boom = new RuntimeException("boom from onTurnCompleteWithPostAction");
class ThrowingPostTurn implements TurnListener {
@Override
public void onTurnComplete(String target) {
}
@Override
public boolean hasPostTurnAction(String target) {
return true;
}
@Override
public boolean onTurnCompleteWithPostAction(String target) {
throw boom;
}
}
Injector inj = new Injector(new AgentControl(herdr), new ThrowingPostTurn());
inj.enqueue(T, "first", TestTurnTokens.inert(T));
Injector.Delivery second = inj.enqueue(T, "second", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE); // delivers "first"
inj.onStatus(T, AgentStatus.WORKING); // picked up
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE), // "first"'s turn completes; post-action throws
"the throwable from onTurnCompleteWithPostAction must still escape onStatus");
assertSame(boom, thrown);
assertEquals(List.of("first"), sent(), "\"second\" must not be sent in the SAME round as the throw");
inj.onStatus(T, AgentStatus.IDLE); // a later, unrelated round
assertEquals(List.of("first", "second"), sent(),
"fleetd #553: the target must not stay wedged — \"second\" must still be delivered "
+ "once postTurnPending is reset despite the earlier throw");
assertTrue(second.completion().isDone() && !second.completion().isCompletedExceptionally());
}
/**
* A {@link HerdrClient} that throws a non-{@link RuntimeException} {@link Error} from {@code
* agent.send_keys} instead of delegating — the fleetd #553 case for the resubmit nudge: its own
* catch (Injector.java, the first block after the monitor) was {@code RuntimeException}-only,
* the same one-class-too-narrow shape #546 fixed at the send seam. Records every call it sees
* itself, mirroring {@code ErrorOnPrompt} above, since the delegate's own recording is never
* reached for {@code agent.send_keys}.
*/
private static final class ErrorOnSendKeys implements HerdrClient {
private final FakeHerdr delegate;
private final List<FakeHerdr.Call> calls = new java.util.concurrent.CopyOnWriteArrayList<>();
private ErrorOnSendKeys(FakeHerdr delegate) {
this.delegate = delegate;
}
List<FakeHerdr.Call> calls() {
return calls;
}
@Override
public JsonNode call(String method, Object params) {
calls.add(new FakeHerdr.Call(method, params));
if (method.equals("agent.send_keys")) {
throw new AssertionError("simulated non-RuntimeException resubmit failure (fleetd #553)");
}
return delegate.call(method, params);
}
@Override
public void close() {
delegate.close();
}
}
@Test
void anErrorFromTheResubmitNudgeDoesNotPreventTheSentCompletion() {
// fleetd #553, acceptance test: the resubmit nudge's own catch was RuntimeException-only —
// the same one-class-too-narrow shape #546 fixed at the send seam (now :382). It sits FIRST
// after the monitor, so before this ticket an Error escaping it skipped every block below in
// THAT round, including any `sent` completion that round might otherwise produce. Widened to
// Throwable (matching #549's own widening) so it can no longer escape onStatus at all — the
// delivery that already completed at send time, and the worker's later pickup, are both
// unaffected by the Error in between.
ErrorOnSendKeys throwing = new ErrorOnSendKeys(new FakeHerdr());
Injector inj = new Injector(new AgentControl(throwing));
Injector.Delivery delivery = inj.enqueue(T, "task", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE); // delivers "task"; awaiting pickup
assertTrue(delivery.completion().isDone() && !delivery.completion().isCompletedExceptionally(),
"the delivery completes at send time, before any resubmit nudge is attempted");
assertDoesNotThrow(() -> inj.onStatus(T, AgentStatus.IDLE), // still idle -> resubmit; Error thrown+caught
"fleetd #553: an Error from the resubmit nudge must be caught inside onStatus, not "
+ "escape it");
inj.onStatus(T, AgentStatus.WORKING); // the worker's real pickup must still resolve cleanly
long promptCalls = throwing.calls().stream().filter(c -> c.method().equals("agent.prompt")).count();
assertEquals(1, promptCalls,
"sent exactly once, unaffected by the resubmit Error in between");
}
@Test
void ordinarySuccessStillCompletesExactlyOnceAfterOnDelivered() {
// Acceptance: the ordinary success path is unchanged by fleetd #553 — the future completes
// normally exactly once, and onDelivered still runs before it (CB-115's pane baseline).
List<String> events = new ArrayList<>();
TurnListener listener = new TurnListener() {
@Override
public void onTurnComplete(String target) {
}
@Override
public void onDelivered(String target, dev.ltms.fleet.msg.TurnToken token) {
events.add("onDelivered");
}
};
Injector inj = new Injector(new AgentControl(herdr), listener);
CompletableFuture<Void> f = inj.enqueue(T, "hello", TestTurnTokens.inert(T)).completion();
f.whenComplete((v, ex) -> events.add("completed"));
inj.onStatus(T, AgentStatus.IDLE);
assertEquals(List.of("onDelivered", "completed"), events,
"onDelivered must still run before the future completes, unchanged by fleetd #553");
assertTrue(f.isDone() && !f.isCompletedExceptionally());
}
// Acceptance: the ordinary failure path (a HerdrException at the send seam still completes the
// future exceptionally with that exception) is already covered, unchanged, by the existing
// aHerdrExceptionFromSendStillProducesNotDeliveredUnchanged test above (fleetd #546) — its
// codepath is untouched by fleetd #553's try/finally, since sendError != null is set before the
// try block begins and that branch never threw to begin with.
// --- fleetd #553, sharpened acceptance (ticket comments 16870/16884/16890/16903): completing
// sent.delivered() is only HALF the job. See the two tests below. ---
@Test
void aRuntimeExceptionFromOnTurnCompleteStillRegistersTheRendezvousWaiterForTheNextDelivery() {
// fleetd #553's real invariant, not just the completion of sent.delivered(). There are TWO
// futures at stake: sent.delivered() (the delivery future) and sent.token().waiter() (the
// rendezvous waiter a blocking fleet_send actually waits on for the worker's ANSWER). The
// waiter is registered only by turnListener.onDelivered() — production wires this to
// CompletionResolver.captureBaseline, which does the inFlight.put() — and that call lives
// inside the very `if (sent != null)` block a plain "complete the future" backstop does not
// reach. A finally that only completes sent.delivered() converts a hang into a HANG WITH A
// SUCCESS RECEIPT: the caller is told the send landed, then waits out its full timeout for
// an answer that can never resolve, because CompletionResolver.resolve (:301-308) finds no
// inFlight entry and returns silently. This test is red on that half-fix, and green only
// once the finally also runs onDelivered() when the normal block never got the chance.
//
// The scenario: onTurnComplete throws while completing "first"'s turn, in the SAME onStatus
// round that (per Injector.java :361-391) then peeks and delivers "second" — the normal
// case, not a corner, since the :365 assignment is what lets the :376 delivery guard pass.
// turnListener.onTurnComplete runs BEFORE the `if (sent != null)` block, so the throw here
// means onDelivered for "second" is never reached on the normal path — only the finally
// backstop can register it.
Rendezvous rendezvous = new Rendezvous();
CompletionResolver resolver = new CompletionResolver(new AgentControl(new FakeHerdr()), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
RuntimeException boom = new RuntimeException("boom from onTurnComplete");
TurnListener throwingOnTurnCompleteOnly = new TurnListener() {
@Override
public void onDelivered(String target, TurnToken token) {
resolver.onDelivered(target, token); // the real registration, exactly as production wires it
}
@Override
public void onTurnComplete(String target) {
throw boom; // "first"'s completion callback, thrown before "second" is delivered
}
};
Injector inj = new Injector(new AgentControl(herdr), throwingOnTurnCompleteOnly);
CompletableFuture<Rendezvous.Resolution> waiter = rendezvous.open("session-a");
TurnToken secondToken = new TurnToken(T, waiter);
inj.enqueue(T, "first", TestTurnTokens.inert(T));
Injector.Delivery second = inj.enqueue(T, "second", secondToken);
inj.onStatus(T, AgentStatus.IDLE); // delivers "first"
inj.onStatus(T, AgentStatus.WORKING); // picked up
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE), // "first" completes (throws); "second" delivered
"the original throwable from onTurnComplete must still escape onStatus");
assertSame(boom, thrown, "must be the EXACT throwable, not a wrapper or a different instance");
assertEquals(List.of("first", "second"), sent(),
"\"second\" must still be sent even though onTurnComplete threw for \"first\"");
assertTrue(second.completion().isDone() && !second.completion().isCompletedExceptionally(),
"\"second\"'s delivery future must still complete despite the throw");
CompletionResolver.InFlight inFlight = resolver.inFlight(T);
assertNotNull(inFlight,
"fleetd #553: the finally must register the new turn's waiter (via onDelivered), not "
+ "just complete its delivery future — otherwise a blocking fleet_send is told "
+ "its message landed and then waits out the full timeout for an answer that "
+ "can never resolve");
assertSame(waiter, inFlight.waiter(),
"the registered waiter must be exactly \"second\"'s waiter, not some other one");
}
@Test
void anOnDeliveredThrowAfterItsOwnRegistrationDoesNotRunASecondTime() {
// fleetd #553 comment 16903: `sentHandled` must be set to true BEFORE onDelivered() runs,
// not after. Setting it after would mean a throw from onDelivered() PART WAY THROUGH — e.g.
// after CompletionResolver's own captureBaseline has already done its inFlight.put() — still
// leaves the flag false, so the finally backstop (seeing "not handled") calls onDelivered() a
// SECOND time. That second captureBaseline runs later, over a pane that may already have
// absorbed this turn's output, and CompletionResolver.resolve's own baseline-equals-tail
// suppression (:364-369, which deliberately keeps the in-flight record on a match) can then
// drop every future completion for the turn, permanently. This assertion is red on a
// `sentHandled = true` placed AFTER the onDelivered() call (onDelivered runs twice) and green
// on the correct placement (runs exactly once) — regardless of what onDelivered itself did.
AtomicInteger onDeliveredCalls = new AtomicInteger();
RuntimeException boom = new RuntimeException("boom from onDelivered, after its own registration ran");
TurnListener listener = new TurnListener() {
@Override
public void onTurnComplete(String target) {
}
@Override
public void onDelivered(String target, TurnToken token) {
onDeliveredCalls.incrementAndGet(); // stand-in for CompletionResolver's inFlight.put
throw boom; // then fail, as if a LATER step inside onDelivered blew up
}
};
Injector inj = new Injector(new AgentControl(herdr), listener);
inj.enqueue(T, "task", TestTurnTokens.inert(T));
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE), // delivers "task"; onDelivered throws
"the original throwable from onDelivered must still escape onStatus");
assertSame(boom, thrown);
assertEquals(1, onDeliveredCalls.get(),
"fleetd #553: onDelivered must be called exactly once — a `sentHandled` flag set "
+ "AFTER the call (rather than before) would leave it false here and the "
+ "finally backstop would call onDelivered a second time");
}
@Test
void anOnDeliveredThrowOnTheNormalPathStillCompletesTheDeliveryFuture() {
// fleetd #553, lead review of PR #557 (ticket comment 16916): `sentHandled` must guard ONLY
// the onDelivered RE-CALL in the finally, never the future completion alongside it. Gating
// BOTH behind `!sentHandled` (the shape this test is red against) misses the one path the
// flag's own correct placement creates: `sentHandled` is set to true FIRST, before the
// onDelivered() call, inside the `if (sent != null)` block above (see the previous test —
// that placement is right and must not change). So when onDelivered() itself throws on that
// NORMAL path, `sentHandled` already reads true by the time control reaches the finally, and
// a `sent != null && !sentHandled` guard around the WHOLE recovery — completion included —
// skips it entirely. The message was typed into the target's pane and taken off the queue
// inside the monitor, same as any other delivery, so its future is left pending forever: the
// exact defect this ticket exists to close, just reached from a different throwing call.
//
// The fix splits the one flag's two jobs: `!sentHandled` keeps gating only the onDelivered
// call (so the exactly-once guarantee in the test above still holds — CompletableFuture.
// complete/completeExceptionally are idempotent, so completing unconditionally here is a
// no-op on the ordinary path, where the `if (sent != null)` block already completed it.
RuntimeException boom = new RuntimeException("boom from onDelivered on the normal path");
TurnListener listener = new TurnListener() {
@Override
public void onTurnComplete(String target) {
}
@Override
public void onDelivered(String target, TurnToken token) {
throw boom;
}
};
Injector inj = new Injector(new AgentControl(herdr), listener);
CompletableFuture<Void> delivered = inj.enqueue(T, "task", TestTurnTokens.inert(T)).completion();
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE), // delivers "task"; onDelivered throws
"the original throwable from onDelivered must still escape onStatus");
assertSame(boom, thrown);
assertTrue(delivered.isDone(),
"fleetd #553: \"task\" was actually delivered — typed into the pane and taken off "
+ "the queue inside the monitor — so its delivery future must be completed on "
+ "every path out of onStatus, including the one where onDelivered itself is "
+ "what threw. Leaving it pending here is a hang, not a fix");
}
// --- fleetd #556: registration is the Injector's own invariant now, structural rather than
// delegated to a TurnListener callback that can throw. #553 could only make the ONE reachable
// listener behave; this closes the shape itself. ---
@Test
void aTurnListenerThatThrowsFromEveryCallbackStillLeavesTheDeliveredTurnRegisteredAndResolvable() {
// fleetd #556 acceptance criterion 1 (the ticket's own, restated by comment 16966 as the
// replacement for #561's order-dependent test): a TurnListener that throws from EVERY
// callback it implements — nothing about it is safe to lean on — must still leave a
// delivered turn registered, with its waiter still resolvable through the ordinary path.
// Before this ticket, registration lived INSIDE onDelivered, one of the callbacks that
// throws here — this test is red against that shape, because the only listener callback
// this specific delivery scenario exercises (a first delivery, no previous turn to
// complete) is exactly the one that throws.
RuntimeException boom = new RuntimeException("fleetd #556: throws from every callback");
TurnListener throwsEverywhere = new TurnListener() {
@Override
public void onTurnComplete(String target) {
throw boom;
}
@Override
public boolean hasPostTurnAction(String target) {
throw boom;
}
@Override
public boolean onTurnCompleteWithPostAction(String target) {
throw boom;
}
@Override
public void onTurnFailed(String target) {
throw boom;
}
@Override
public void onTurnFailed(String target, String reason) {
throw boom;
}
@Override
public void onDelivered(String target, TurnToken token) {
throw boom;
}
};
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = new CompletionResolver(new AgentControl(new FakeHerdr()), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
Injector inj = new Injector(new AgentControl(herdr), throwsEverywhere, _ -> true, _ -> {
}, completion::register);
CompletableFuture<Rendezvous.Resolution> waiter = rendezvous.open(T);
inj.enqueue(T, "brief", new TurnToken(T, waiter));
RuntimeException thrown = assertThrows(RuntimeException.class,
() -> inj.onStatus(T, AgentStatus.IDLE), // delivers "brief"; onDelivered throws
"the throwing listener's own bug must still escape onStatus — a listener bug stays "
+ "loud (fleetd #553's own guarantee, unchanged by this ticket)");
assertSame(boom, thrown);
CompletionResolver.InFlight inFlight = completion.inFlight(T);
assertNotNull(inFlight,
"fleetd #556: the delivered turn must be registered even though the ONLY "
+ "TurnListener wired throws from every single callback, including "
+ "onDelivered — registration must not depend on that call succeeding");
assertSame(waiter, inFlight.waiter(),
"the registered entry must carry the exact waiter this turn's send opened");
assertFalse(waiter.isDone(), "sanity: nothing has resolved this waiter yet");
assertTrue(rendezvous.resolve(T, "hello"),
"the waiter registrar.register captured must be the real, live rendezvous waiter — "
+ "still resolvable through the ordinary fleet_reply path, unaffected by the "
+ "throwing listener");
assertEquals(Rendezvous.Kind.REPLY, waiter.getNow(null).kind());
}
@Test
void theNewTurnsRegistrationRunsAfterThePreviousTurnsCompletionIsRead() {
// fleetd #556 acceptance criterion 2 (CB-116 guard, ticket body + comment 16966): the
// ordering constraint from fleetd #553 must survive this redesign — onTurnComplete reads
// CompletionResolver's inFlight entry for the PREVIOUS turn, so the new turn's
// registrar.register() call must not run before that read, or a redesign trades this
// ticket's defect for the CB-116 cross-turn stale reply. Pinned here as a call-ORDER
// assertion so a future redesign that hoists registrar.register() earlier (e.g. "for
// simplicity", ahead of the turnCompleted block) goes red immediately, before it can ever
// reach a real cross-turn scrape race.
List<String> events = new ArrayList<>();
TurnListener listener = new TurnListener() {
@Override
public void onTurnComplete(String target) {
events.add("onTurnComplete:" + target);
}
};
TurnRegistrar registrar = (target, token) -> events.add("register:" + target);
Injector inj = new Injector(new AgentControl(herdr), listener, _ -> true, _ -> {
}, registrar);
inj.enqueue(T, "first", TestTurnTokens.inert(T));
inj.enqueue(T, "second", TestTurnTokens.inert(T));
inj.onStatus(T, AgentStatus.IDLE); // delivers "first" -> register:term_a (not asserted below)
inj.onStatus(T, AgentStatus.WORKING); // picked up
events.clear();
inj.onStatus(T, AgentStatus.IDLE); // "first" completes AND "second" delivers, same cycle
assertEquals(List.of("onTurnComplete:" + T, "register:" + T), events,
"onTurnComplete for the previous turn (\"first\") must run before register for the "
+ "new turn (\"second\") within the same onStatus cycle — reversing this order "
+ "is the CB-116 cross-turn stale reply fleetd #553 fixed, and this redesign "
+ "must not reopen it");
}
@Test
void aRuntimeExceptionFromOnTurnCompleteStillLeavesTheNextDeliveryRegisteredOnTheRecoveryPath() {
// fleetd #556 rework (PR #566, comment 17009): there are TWO registrar.register call sites
// in the delivery method — the ordinary path inside `if (sent != null)`, and the fleetd
// #553 `finally` backstop, reached only when an earlier block (here, onTurnComplete for the
// PREVIOUS turn) throws before the ordinary path ever runs. The existing #553 regression
// test for this exact scenario
// (aRuntimeExceptionFromOnTurnCompleteStillCompletesTheNextDelivery, above) asserts only
// that "second"'s DELIVERED FUTURE completes — never that "second" is REGISTERED with
// CompletionResolver — so a redesign that dropped registrar.register() from the backstop
// passed every existing test while reopening this ticket's own defect on the one path
// fleetd #553 exists for. This test closes that gap: on the recovery path, the backstop is
// the ONLY thing that registers "second", so its waiter must still be resolvable afterward.
RuntimeException boom = new RuntimeException("fleetd #556 rework: boom from onTurnComplete");
TurnListener throwing = new TurnListener() {
@Override
public void onTurnComplete(String target) {
throw boom;
}
};
Rendezvous rendezvous = new Rendezvous();
CompletionResolver completion = new CompletionResolver(new AgentControl(herdr), rendezvous,
ExhaustedPatternLookup.none(), ExhaustionSink.none());
Injector inj = new Injector(new AgentControl(herdr), throwing, _ -> true, _ -> {
}, completion::register);
inj.enqueue(T, "first", TestTurnTokens.inert(T));
CompletableFuture<Rendezvous.Resolution> secondWaiter = rendezvous.open(T);
inj.enqueue(T, "second", new TurnToken(T, secondWaiter));
inj.onStatus(T, AgentStatus.IDLE); // delivers "first" (inert token — nothing to register)
inj.onStatus(T, AgentStatus.WORKING); // picked up
RuntimeException thrown = assertThrows(RuntimeException.class,
// "first" completes (onTurnComplete throws) before "second"'s ordinary delivery
// path ever runs; only the finally backstop is left to register "second".
() -> inj.onStatus(T, AgentStatus.IDLE),
"the throwable from onTurnComplete must still escape onStatus");
assertSame(boom, thrown);
CompletionResolver.InFlight inFlight = completion.inFlight(T);
assertNotNull(inFlight,
"fleetd #556: \"second\" must be registered by the fleetd #553 finally backstop "
+ "even though onTurnComplete threw for \"first\"'s completion before the "
+ "ordinary registration path ever ran for \"second\"");
assertSame(secondWaiter, inFlight.waiter(),
"the registered entry must carry \"second\"'s own waiter, not some other value");
}
}
@@ -0,0 +1,61 @@
package dev.ltms.fleet.inject;
import org.junit.jupiter.api.Test;
import java.util.concurrent.atomic.AtomicLong;
import static org.junit.jupiter.api.Assertions.assertEquals;
/** fleetd #544: the three-state contract in isolation from either loop that composes it. */
class LoopWatchdogTest {
private static final long STALE_AFTER_NANOS = 1000;
@Test
void reportsRunningBeforeTheThresholdElapses() {
AtomicLong clock = new AtomicLong(0);
LoopWatchdog watchdog = new LoopWatchdog(clock::get, STALE_AFTER_NANOS);
clock.set(STALE_AFTER_NANOS - 1);
assertEquals(LoopWatchdog.State.RUNNING, watchdog.state());
}
@Test
void reportsStalledOnceTheLastRoundAgesPastTheThreshold() {
AtomicLong clock = new AtomicLong(0);
LoopWatchdog watchdog = new LoopWatchdog(clock::get, STALE_AFTER_NANOS);
clock.set(STALE_AFTER_NANOS);
assertEquals(LoopWatchdog.State.STALLED, watchdog.state());
}
@Test
void recordRoundCompleteResetsTheStaleClock() {
AtomicLong clock = new AtomicLong(0);
LoopWatchdog watchdog = new LoopWatchdog(clock::get, STALE_AFTER_NANOS);
clock.set(STALE_AFTER_NANOS - 1);
watchdog.recordRoundComplete(); // last round is now "now" again
clock.set(2 * STALE_AFTER_NANOS - 2);
assertEquals(LoopWatchdog.State.RUNNING, watchdog.state(),
"a fresh round completion must push the staleness deadline forward");
}
@Test
void markStoppedByCallerReportsStoppedRegardlessOfStaleness() {
AtomicLong clock = new AtomicLong(0);
LoopWatchdog watchdog = new LoopWatchdog(clock::get, STALE_AFTER_NANOS);
watchdog.markStoppedByCaller();
clock.set(STALE_AFTER_NANOS * 1000);
assertEquals(LoopWatchdog.State.STOPPED, watchdog.state(),
"an intentional stop must never be reported as STALLED");
}
@Test
void resetClearsAPreviousStopAndTheStaleClock() {
AtomicLong clock = new AtomicLong(0);
LoopWatchdog watchdog = new LoopWatchdog(clock::get, STALE_AFTER_NANOS);
watchdog.markStoppedByCaller();
clock.set(STALE_AFTER_NANOS * 1000);
watchdog.reset();
assertEquals(LoopWatchdog.State.RUNNING, watchdog.state(),
"reset() must clear both the stop mark and the stale timestamp");
}
}
@@ -0,0 +1,99 @@
package dev.ltms.fleet.inject;
import com.fasterxml.jackson.databind.JsonNode;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.msg.TestTurnTokens;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
class StatusPollerResilienceTest {
@Test
void anErrorForOneTargetDoesNotStopPollingTheNextTarget() throws Exception {
FakeHerdr fake = new FakeHerdr().withAgent("worker", "term_b", "w2:p8", "w2:t8");
CountDownLatch errorThrown = new CountDownLatch(1);
AgentControl agents = new AgentControl(new ErrorOnceForFirstTarget(fake, errorThrown));
Injector injector = new Injector(agents);
StatusPoller poller = new StatusPoller(agents, injector, 1);
poller.start();
try {
injector.enqueue("term_a", "first", TestTurnTokens.inert("term_a"));
assertTrue(errorThrown.await(2, TimeUnit.SECONDS),
"the first target must throw its test Error");
CompletableFuture<Void> delivered =
injector.enqueue("term_b", "second", TestTurnTokens.inert("term_b")).completion();
delivered.get(2, TimeUnit.SECONDS);
} finally {
poller.stop();
}
}
@Test
void anAbnormalExitClearsRunningSoStartCreatesANewLoop() throws Exception {
StatusPoller poller = new StatusPoller(new AgentControl(new FakeHerdr()), new Injector(new AgentControl(new FakeHerdr())), -1);
poller.start();
Thread first = threadOf(poller);
first.join(2000);
assertFalse(runningOf(poller), "an abnormal loop exit must clear running");
poller.start();
Thread restarted = threadOf(poller);
try {
assertNotSame(first, restarted, "start() must create a new loop after an abnormal exit");
restarted.join(2000);
} finally {
poller.stop();
}
}
private static Thread threadOf(StatusPoller poller) throws ReflectiveOperationException {
Field field = StatusPoller.class.getDeclaredField("thread");
field.setAccessible(true);
return (Thread) field.get(poller);
}
private static boolean runningOf(StatusPoller poller) throws ReflectiveOperationException {
Field field = StatusPoller.class.getDeclaredField("running");
field.setAccessible(true);
return field.getBoolean(poller);
}
private static final class ErrorOnceForFirstTarget implements HerdrClient {
private final FakeHerdr delegate;
private final CountDownLatch errorThrown;
private final AtomicBoolean first = new AtomicBoolean(true);
private ErrorOnceForFirstTarget(FakeHerdr delegate, CountDownLatch errorThrown) {
this.delegate = delegate;
this.errorThrown = errorThrown;
}
@Override
public JsonNode call(String method, Object params) {
if (method.equals("agent.get") && params instanceof Map<?, ?> map
&& "w2:p7".equals(map.get("target")) && first.compareAndSet(true, false)) {
errorThrown.countDown();
throw new AssertionError("test Error from the first poll target");
}
return delegate.call(method, params);
}
@Override
public void close() {
delegate.close();
}
}
}
@@ -0,0 +1,172 @@
package dev.ltms.fleet.inject;
import com.fasterxml.jackson.databind.JsonNode;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.msg.TestTurnTokens;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #544: {@link StatusPoller#health()} — the progress watchdog, not a liveness check. Every
* test drives {@link LoopWatchdog}'s clock explicitly (never real elapsed time), so the threshold
* crossing is deterministic rather than a race against the poll interval.
*/
class StatusPollerWatchdogTest {
private static final long INTERVAL_MILLIS = 50;
// Mirrors StatusPoller.staleAfterNanos(50): 50ms * the 40x multiplier = 2000ms.
private static final long STALE_AFTER_NANOS = TimeUnit.MILLISECONDS.toNanos(INTERVAL_MILLIS) * 40;
@Test
void aLoopParkedInAHerdrCallThatNeverReturnsReportsStalled() throws Exception {
CountDownLatch entered = new CountDownLatch(1);
AgentControl agents = new AgentControl(new BlocksOnAgentGet(entered));
Injector injector = new Injector(agents);
AtomicLong clock = new AtomicLong(0);
StatusPoller poller = new StatusPoller(agents, injector, new StatusRefiner(agents),
INTERVAL_MILLIS, clock::get);
poller.start();
try {
injector.enqueue("w1:p1", "hello", TestTurnTokens.inert("w1:p1"));
assertTrue(entered.await(2, TimeUnit.SECONDS),
"the poller must have entered the blocking herdr call");
assertEquals(LoopWatchdog.State.RUNNING, poller.health(),
"freshly parked, before the threshold elapses, must still read RUNNING");
clock.set(STALE_AFTER_NANOS + 1);
assertEquals(LoopWatchdog.State.STALLED, poller.health(),
"a loop parked mid-round past the staleness threshold must report STALLED");
} finally {
poller.stop();
}
}
@Test
void aLoopThatDiedReportsStalled() throws Exception {
AtomicLong clock = new AtomicLong(0);
// intervalMillis=-1: Thread.sleep(-1) throws IllegalArgumentException, uncaught by loop(),
// which kills the carrier thread — the same "died" shape StatusPollerResilienceTest uses.
StatusPoller poller = new StatusPoller(new AgentControl(new IdleHerdr()), new Injector(new AgentControl(new IdleHerdr())),
new StatusRefiner(new AgentControl(new IdleHerdr())), -1, clock::get);
poller.start();
Thread thread = threadOf(poller);
thread.join(2000);
assertFalse(thread.isAlive(), "the loop must have died from Thread.sleep(-1)");
// staleAfterNanos(-1) = TimeUnit.MILLISECONDS.toNanos(max(-1,1)) * 40 = 40ms.
clock.set(TimeUnit.MILLISECONDS.toNanos(1) * 40 + 1);
assertEquals(LoopWatchdog.State.STALLED, poller.health(),
"a dead loop's last-completed round goes stale and must report STALLED");
}
@Test
void aLoopStoppedOnPurposeReportsStoppedNotStalled() throws Exception {
AtomicLong clock = new AtomicLong(0);
StatusPoller poller = new StatusPoller(new AgentControl(new IdleHerdr()), new Injector(new AgentControl(new IdleHerdr())),
new StatusRefiner(new AgentControl(new IdleHerdr())), INTERVAL_MILLIS, clock::get);
poller.start();
poller.stop();
// Advance the clock far past any staleness threshold — an intentional stop must never
// be reported as STALLED no matter how old the last round looks (fleetd #512 shape).
clock.set(STALE_AFTER_NANOS * 100);
assertEquals(LoopWatchdog.State.STOPPED, poller.health(),
"stop() must report STOPPED, never STALLED, however stale the last round looks");
}
@Test
void aRestartedLoopReportsRunningAgainNotStoppedForever() throws Exception {
// fleetd #544 review (issue comment #16944): stoppedByCaller is sticky, and reset() —
// called only from start() — is the sole thing that clears it. start() is documented
// idempotent and loop()'s own error line says "it can be restarted", so stop() followed
// by start() is an anticipated path. Without the reset() call in start(), health() would
// report STOPPED forever after a restart even though the loop is genuinely running again.
AtomicLong clock = new AtomicLong(0);
StatusPoller poller = new StatusPoller(new AgentControl(new IdleHerdr()), new Injector(new AgentControl(new IdleHerdr())),
new StatusRefiner(new AgentControl(new IdleHerdr())), INTERVAL_MILLIS, clock::get);
poller.start();
poller.stop();
poller.start();
assertEquals(LoopWatchdog.State.RUNNING, poller.health(),
"an intentional stop must not outlive the restart that follows it");
}
@Test
void aHealthyLoopReportsRunning() throws Exception {
// Real elapsed time on purpose, unlike the other tests here: a clock frozen at 0 would read
// RUNNING even if recordRoundComplete() were never wired into loop() at all, since the
// constructor's own initial timestamp already satisfies "not stale yet". Running for several
// multiples of this instance's own threshold (5ms interval * 40 = 200ms) and still reading
// RUNNING instead proves the loop's repeated round completions are the thing keeping the
// staleness deadline pushed forward.
long fastIntervalMillis = 5;
StatusPoller poller = new StatusPoller(new AgentControl(new IdleHerdr()), new Injector(new AgentControl(new IdleHerdr())),
new StatusRefiner(new AgentControl(new IdleHerdr())), fastIntervalMillis, System::nanoTime);
poller.start();
try {
Thread.sleep(800);
assertEquals(LoopWatchdog.State.RUNNING, poller.health(),
"a healthy loop's repeated round completions must keep pushing the staleness deadline forward");
} finally {
poller.stop();
}
}
private static Thread threadOf(StatusPoller poller) throws ReflectiveOperationException {
Field field = StatusPoller.class.getDeclaredField("thread");
field.setAccessible(true);
return (Thread) field.get(poller);
}
/** Never has any active target, so the loop just idles and completes empty rounds. */
private static final class IdleHerdr implements HerdrClient {
@Override
public JsonNode call(String method, Object params) {
throw new AssertionError("unexpected herdr call: " + method);
}
@Override
public void close() {
}
}
private static final class BlocksOnAgentGet implements HerdrClient {
private final CountDownLatch entered;
private BlocksOnAgentGet(CountDownLatch entered) {
this.entered = entered;
}
@Override
public JsonNode call(String method, Object params) {
if ("agent.get".equals(method)) {
entered.countDown();
try {
// Blocks forever, mirroring UnixSocketHerdrClient.readLine()'s deadline-free
// read (fleetd #544) — a test double standing in for a parked socket, not a
// real one.
new CountDownLatch(1).await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
throw new AssertionError("unreachable: only stop()'s interrupt reaches here");
}
throw new AssertionError("unexpected herdr call: " + method);
}
@Override
public void close() {
}
}
}
@@ -7,7 +7,9 @@ import dev.ltms.fleet.auth.Role;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.herdr.PaneLocator;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.inject.Injector;
@@ -57,9 +59,10 @@ class FleetMcpTest {
private final FakeHerdr herdr = new FakeHerdr();
private final AgentControl agents = new AgentControl(herdr);
private final Injector injector = new Injector(agents);
private final Rendezvous rendezvous = new Rendezvous();
private final InMemoryReplyInbox inbox = new InMemoryReplyInbox();
private final MessageService messages = new MessageService(agents, new Injector(agents), rendezvous, inbox);
private final MessageService messages = new MessageService(agents, injector, rendezvous, inbox);
@BeforeEach
void setUp() {
@@ -298,6 +301,35 @@ class FleetMcpTest {
assertTrue(textOf(res).contains("no reply"), "got: " + textOf(res));
}
/**
* fleetd #571 (ticket CORRECTION 5): {@code formatReply}'s {@code TIMED_OUT_UNCONFIRMED} arm is
* the one message whose whole job is to stop a caller retrying a delivery that may already have
* arrived. Pin that its wording is actually distinct from the queued/working arm's retry
* invitation — a mutation that swapped this arm's text for that one still passed every other
* test in this suite, because nothing asserted the specific wording.
*/
@Test
void sendTimesOutWithAnUnconfirmedNoteNotARetryInvitation() throws Exception {
herdr.agentSendFailsWith("send_failed");
CompletableFuture<McpSchema.CallToolResult> send = CompletableFuture.supplyAsync(
() -> FleetMcp.send(messages, T, "hi", 150L, null, Set.of()));
long deadline = System.currentTimeMillis() + 2000;
while (!rendezvous.isWaiting(T) && System.currentTimeMillis() < deadline) {
//noinspection BusyWait
Thread.sleep(5);
}
assertTrue(rendezvous.isWaiting(T), "send should have opened its rendezvous waiter");
injector.onStatus(T, AgentStatus.IDLE); // triggers the failing delivery attempt -> ATTEMPTED
McpSchema.CallToolResult res = send.get(5, TimeUnit.SECONDS);
String text = textOf(res);
assertNotEquals(Boolean.TRUE, res.isError(), "a timeout is informational, not a tool error");
assertTrue(text.contains("delivery unconfirmed"), "got: " + text);
assertFalse(text.contains("retry or poll status"),
"an unconfirmed delivery must not carry the queued/working arm's retry invitation — "
+ "a resend here can double-deliver the same brief: got " + text);
}
@Test
void sendRejectsMissingArgs() {
assertTrue(FleetMcp.send(messages, null, "hi", null, null, Set.of()).isError());
@@ -1053,6 +1085,36 @@ class FleetMcpTest {
assertFalse(out.contains("quarantinedForSeconds"), out);
}
@Test
void loopHealthReportsStalledStatusPoller() {
String out = loopHealth(LoopWatchdog.State.STALLED, LoopWatchdog.State.RUNNING);
assertTrue(out.contains("\"statusPoller\":\"STALLED\""),
"fleet_list must report a stalled StatusPoller: " + out);
}
@Test
void loopHealthReportsStoppedSessionReaperAsStopped() {
String out = loopHealth(LoopWatchdog.State.RUNNING, LoopWatchdog.State.STOPPED);
assertTrue(out.contains("\"sessionReaper\":\"STOPPED\""),
"fleet_list must report a deliberately stopped SessionReaper as STOPPED, not an alarm: " + out);
}
@Test
void loopHealthReportsRunningStatusPoller() {
String out = loopHealth(LoopWatchdog.State.RUNNING, LoopWatchdog.State.STOPPED);
assertTrue(out.contains("\"statusPoller\":\"RUNNING\""),
"fleet_list must report a running StatusPoller: " + out);
}
private static String loopHealth(LoopWatchdog.State statusPoller, LoopWatchdog.State sessionReaper) {
FakeHerdr herdr = new FakeHerdr();
return textOf(FleetMcp.listFleet(workerService(herdr, "http://gx00.gw:8000", Set.of("gx00.gw")),
new SessionManager(workerService(herdr, "http://gx00.gw:8000", Set.of("gx00.gw"))), null,
FleetMcp.CapacitySource.none(), new FleetMcp.HealthCoverageSource(() -> "off"),
new FleetMcp.LoopHealthSource(() -> statusPoller, () -> sessionReaper),
FleetMcp.QuarantineSource.none(), Map.of(), ""));
}
@Test
void capacityIncludesConfiguredProfileWithoutMembers() {
FakeHerdr h = new FakeHerdr();
@@ -1875,7 +1937,7 @@ class FleetMcpTest {
null, null, null, null, null, null);
assertEquals(Boolean.TRUE, res.isError());
assertTrue(textOf(res).contains("architect, dev, reviewer"), textOf(res));
assertTrue(textOf(res).contains("architect, dev, hunter, reviewer"), textOf(res));
}
// ── CB-619 / fleetd #123: a spawn asking for a role its profile has no slot for must be
@@ -556,6 +556,31 @@ class ClaudeCodeLauncherTest {
"no --agent flag when the role has no agent-definition file");
}
@Test
void hunterRoleUsesItsAgentFileAndStopsUsingItWhenRemoved(@TempDir Path cwd) throws Exception {
FakeHerdr herdr = new FakeHerdr();
Path agentFile = Files.createDirectories(cwd.resolve(".claude/agents")).resolve("hunter.md");
Files.writeString(agentFile, "---\nname: hunter\n---\nSweep for defects.");
FleetConfig.Profile cfg = new FleetConfig.Profile(
"sonnet", "http://gx00.gw:8000", null, null, "FLEETD_WORKER_TOKEN",
List.of("claude"), "tab", "fleetd-workers", "w #{n}", null, null, null);
ClaudeCodeLauncher svc = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
svc.spawn(new SpawnRequest("sonnet", cwd.toString(), null, null, null, MemberRole.HUNTER));
List<String> args = spawnedArgs(herdr);
int flag = args.indexOf("--agent");
assertTrue(flag >= 0, "the hunter role reaches its agent-definition file: " + args);
assertEquals("hunter", args.get(flag + 1));
Files.delete(agentFile);
svc.spawn(new SpawnRequest("sonnet", cwd.toString(), null, null, null, MemberRole.HUNTER));
assertFalse(spawnedArgs(herdr).contains("--agent"),
"the hunter role no longer gets an agent when its file is removed");
}
private ClaudeCodeLauncher multiProfile(FakeHerdr herdr) {
FleetConfig.Profile gx10 = new FleetConfig.Profile("gx10", "http://gx10.gw:8000", "coder",
null, "FLEETD_WORKER_TOKEN", List.of("claude"), "tab", "fleetd-workers", "w #{n}", null, null, null);
@@ -8,7 +8,10 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.io.IOException;
import java.util.Map;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
@@ -20,6 +23,7 @@ import java.util.concurrent.atomic.AtomicReference;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* CB-528 follow-up: {@link AmqpReplyInbox#failPendingPublishesOnRecovery()} must not fail a publish
@@ -197,11 +201,78 @@ class AmqpReplyInboxRecoveryRaceTest {
+ elapsedMillis.get() + "ms");
}
@Test
void publishIOExceptionRemovesThePendingMessageId() throws Exception {
AtomicLong seqCounter = new AtomicLong();
Channel failing = fakeChannel(seqCounter, new CopyOnWriteArrayList<>(), new CopyOnWriteArrayList<>(),
new AtomicReference<>(), new AtomicReference<>(), true);
AmqpReplyInbox inbox = new AmqpReplyInbox(fakeConnection(failing, failing), AmqpReplyInbox.DEFAULT_PREFETCH);
try {
org.junit.jupiter.api.Assertions.assertThrows(IllegalStateException.class,
() -> inbox.publish("worker", "catch", "body"));
assertEquals(0, pendingByMsgId(inbox).size(), "publish IOException must remove its msgId entry");
} finally {
inbox.close();
}
}
@Test
void interruptedPublishRemovesThePendingMessageIdInFinally() throws Exception {
InboxFixture fixture = new InboxFixture();
Thread publish = fixture.startPublish("finally");
fixture.awaitPublished("finally");
publish.interrupt();
publish.join(5_000);
assertEquals(0, pendingByMsgId(fixture.inbox).size(), "publish finally must remove its msgId entry");
fixture.inbox.close();
}
@Test
void confirmResolutionRemovesThePendingMessageId() throws Exception {
AmqpReplyInbox inbox = new InboxFixture().inbox;
try {
seedPending(inbox, 1, "confirm");
invoke(inbox, "resolveConfirm", new Class<?>[] {long.class, boolean.class, boolean.class}, 1L, false, true);
assertEquals(0, pendingByMsgId(inbox).size(), "confirm resolution must remove its msgId entry");
} finally {
inbox.close();
}
}
@Test
void recoverySweepRemovesThePendingMessageId() throws Exception {
AmqpReplyInbox inbox = new InboxFixture().inbox;
try {
seedPending(inbox, 1, "recovery");
inbox.failPendingPublishesOnRecovery();
assertEquals(0, pendingByMsgId(inbox).size(), "recovery sweep must remove its msgId entry");
} finally {
inbox.close();
}
}
@Test
void closeRemovesThePendingMessageId() throws Exception {
AmqpReplyInbox inbox = new InboxFixture().inbox;
seedPending(inbox, 1, "close");
inbox.close();
assertEquals(0, pendingByMsgId(inbox).size(), "close must remove its msgId entry");
}
/** A {@link Proxy}-backed {@link Channel}: only the calls {@link AmqpReplyInbox} actually makes
* are meaningfully implemented; everything else returns a harmless default. */
private static Channel fakeChannel(AtomicLong seqCounter, List<Long> seqOrder, List<String> msgIdOrder,
AtomicReference<ConfirmCallback> ackCallback,
AtomicReference<ConfirmCallback> nackCallback) {
AtomicReference<ConfirmCallback> ackCallback,
AtomicReference<ConfirmCallback> nackCallback) {
return fakeChannel(seqCounter, seqOrder, msgIdOrder, ackCallback, nackCallback, false);
}
private static Channel fakeChannel(AtomicLong seqCounter, List<Long> seqOrder, List<String> msgIdOrder,
AtomicReference<ConfirmCallback> ackCallback,
AtomicReference<ConfirmCallback> nackCallback, boolean failPublish) {
InvocationHandler handler = (proxy, method, args) -> {
String name = method.getName();
if (name.equals("getNextPublishSeqNo")) {
@@ -210,6 +281,9 @@ class AmqpReplyInboxRecoveryRaceTest {
return value;
}
if (name.equals("basicPublish")) {
if (failPublish) {
throw new IOException("test publish failure");
}
AMQP.BasicProperties props = (AMQP.BasicProperties) args[3];
msgIdOrder.add(props.getMessageId());
return null;
@@ -285,4 +359,60 @@ class AmqpReplyInboxRecoveryRaceTest {
}
return 0;
}
@SuppressWarnings("unchecked")
private static Map<String, Object> pendingByMsgId(AmqpReplyInbox inbox) throws Exception {
var field = AmqpReplyInbox.class.getDeclaredField("pendingByMsgId");
field.setAccessible(true);
return (Map<String, Object>) field.get(inbox);
}
@SuppressWarnings("unchecked")
private static void seedPending(AmqpReplyInbox inbox, long seq, String msgId) throws Exception {
Class<?> pendingType = Class.forName(AmqpReplyInbox.class.getName() + "$Pending");
var constructor = pendingType.getDeclaredConstructor(String.class);
constructor.setAccessible(true);
Object pending = constructor.newInstance(msgId);
var seqField = AmqpReplyInbox.class.getDeclaredField("pendingBySeq");
seqField.setAccessible(true);
((Map<Long, Object>) seqField.get(inbox)).put(seq, pending);
pendingByMsgId(inbox).put(msgId, pending);
}
private static void invoke(AmqpReplyInbox inbox, String name, Class<?>[] types, Object... args) throws Exception {
Method method = AmqpReplyInbox.class.getDeclaredMethod(name, types);
method.setAccessible(true);
method.invoke(inbox, args);
}
private static final class InboxFixture {
final AtomicLong seqCounter = new AtomicLong();
final List<Long> seqOrder = new CopyOnWriteArrayList<>();
final List<String> msgIdOrder = new CopyOnWriteArrayList<>();
final AtomicReference<ConfirmCallback> ackCallback = new AtomicReference<>();
final AtomicReference<ConfirmCallback> nackCallback = new AtomicReference<>();
final AmqpReplyInbox inbox = new AmqpReplyInbox(
fakeConnection(fakeChannel(seqCounter, seqOrder, msgIdOrder, ackCallback, nackCallback),
fakeChannel(seqCounter, seqOrder, msgIdOrder, ackCallback, nackCallback)),
AmqpReplyInbox.DEFAULT_PREFETCH);
Thread startPublish(String msgId) {
Thread thread = Thread.ofVirtual().start(() -> {
try {
inbox.publish("worker", msgId, "body");
} catch (IllegalStateException ignored) {
// Interrupting the confirm wait is the path under test.
}
});
return thread;
}
void awaitPublished(String msgId) throws InterruptedException {
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
while (!msgIdOrder.contains(msgId) && System.nanoTime() < deadline) {
Thread.sleep(10);
}
assertTrue(msgIdOrder.contains(msgId), "publish did not register " + msgId);
}
}
}
@@ -12,13 +12,18 @@ import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import java.io.IOException;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -207,6 +212,31 @@ class LeadMailboxTest {
}
}
/**
* {@link LeadMailbox#inspect} opens a third channel after the mailbox's consume and publish
* channels. Limit this connection to three channels, then require a replacement channel after
* the successful inspect. If inspect leaves its probe open, the broker refuses that replacement.
*/
@Test
void inspectClosesItsSuccessfulProbeChannel() throws Exception {
var factory = LeadMailbox.connectionFactory(uri());
factory.setRequestedChannelMax(3);
Connection connection = factory.newConnection();
try (LeadMailbox mailbox = new LeadMailbox(connection, coordId("lead-inspect-probe-close"))) {
LeadChannel.MailboxState state = mailbox.inspect(mailbox.selfCoordId());
assertTrue(state.exists(), "the owned mailbox must be found before checking the probe channel");
Channel replacement = connection.createChannel();
assertNotNull(replacement,
"inspect must close its successful probe channel; the replacement channel was null");
try {
assertTrue(replacement.isOpen(), "the replacement channel must be open after inspect returns");
} finally {
replacement.close();
}
}
}
/**
* fleetd #440: {@code heldDurable()} must be derived from what {@link LeadMailbox#own} actually
* did against the real broker — a durable queue declare plus a manual-ack consumer — not a
@@ -347,6 +377,72 @@ class LeadMailboxTest {
() -> "expected AlreadyClosedException, got: " + thrown);
}
@Test
void publishIOExceptionRemovesThePendingMessageId() throws Exception {
LeadMailbox mailbox = newMailbox(true);
try {
assertThrows(IllegalStateException.class,
() -> mailbox.publish("target", new LeadMessage("catch", "from", "target", "body")));
assertEquals(0, pendingByMsgId(mailbox).size(), "publish IOException must remove its msgId entry");
} finally {
mailbox.close();
}
}
@Test
void interruptedPublishRemovesThePendingMessageIdInFinally() throws Exception {
LeadMailbox mailbox = newMailbox(false);
Thread publish = Thread.ofVirtual().start(() -> {
try {
mailbox.publish("target", new LeadMessage("finally", "from", "target", "body"));
} catch (IllegalStateException ignored) {
// Interrupting the confirm wait is the path under test.
}
});
awaitPending(mailbox, "finally");
publish.interrupt();
publish.join(5_000);
try {
assertEquals(0, pendingByMsgId(mailbox).size(), "publish finally must remove its msgId entry");
} finally {
mailbox.close();
}
}
@Test
void confirmResolutionRemovesThePendingMessageId() throws Exception {
LeadMailbox mailbox = newMailbox(false);
try {
seedPending(mailbox, 1, "confirm");
invoke(mailbox, "resolveConfirm", new Class<?>[] {long.class, boolean.class, boolean.class}, 1L, false, true);
assertEquals(0, pendingByMsgId(mailbox).size(), "confirm resolution must remove its msgId entry");
} finally {
mailbox.close();
}
}
@Test
void recoverySweepRemovesThePendingMessageId() throws Exception {
LeadMailbox mailbox = newMailbox(false);
try {
seedPending(mailbox, 1, "recovery");
mailbox.failPendingPublishesOnRecovery();
assertEquals(0, pendingByMsgId(mailbox).size(), "recovery sweep must remove its msgId entry");
} finally {
mailbox.close();
}
}
@Test
void closeRemovesThePendingMessageId() throws Exception {
LeadMailbox mailbox = newMailbox(false);
seedPending(mailbox, 1, "close");
mailbox.close();
assertEquals(0, pendingByMsgId(mailbox).size(), "close must remove its msgId entry");
}
/** Poll peek until at least one message is held, or ~10s elapse (broker delivery is async). */
@SuppressWarnings("BusyWait")
private static List<LeadMessage> awaitPeek(LeadMailbox inbox) throws InterruptedException {
@@ -372,4 +468,108 @@ class LeadMailboxTest {
}
return state;
}
private static LeadMailbox newMailbox(boolean failPublish) {
AtomicLong sequence = new AtomicLong();
Channel consume = fakeChannel(sequence, false);
Channel publish = fakeChannel(sequence, failPublish);
return new LeadMailbox(fakeConnection(consume, publish), "self");
}
private static Channel fakeChannel(AtomicLong sequence, boolean failPublish) {
InvocationHandler handler = (proxy, method, args) -> {
if (method.getName().equals("getNextPublishSeqNo")) {
return sequence.incrementAndGet();
}
if (method.getName().equals("basicPublish") && failPublish) {
throw new IOException("test publish failure");
}
if (method.getName().equals("equals")) {
return proxy == args[0];
}
if (method.getName().equals("hashCode")) {
return System.identityHashCode(proxy);
}
return defaultValue(method.getReturnType());
};
return (Channel) Proxy.newProxyInstance(LeadMailboxTest.class.getClassLoader(), new Class<?>[] {Channel.class}, handler);
}
private static Connection fakeConnection(Channel first, Channel second) {
AtomicLong calls = new AtomicLong();
InvocationHandler handler = (proxy, method, args) -> {
if (method.getName().equals("createChannel") && (args == null || args.length == 0)) {
return calls.getAndIncrement() == 0 ? first : second;
}
if (method.getName().equals("equals")) {
return proxy == args[0];
}
if (method.getName().equals("hashCode")) {
return System.identityHashCode(proxy);
}
return defaultValue(method.getReturnType());
};
return (Connection) Proxy.newProxyInstance(LeadMailboxTest.class.getClassLoader(), new Class<?>[] {Connection.class}, handler);
}
@SuppressWarnings("unchecked")
private static Map<String, Object> pendingByMsgId(LeadMailbox mailbox) throws Exception {
var field = LeadMailbox.class.getDeclaredField("pendingByMsgId");
field.setAccessible(true);
return (Map<String, Object>) field.get(mailbox);
}
@SuppressWarnings("unchecked")
private static void seedPending(LeadMailbox mailbox, long seq, String msgId) throws Exception {
Class<?> pendingType = Class.forName(LeadMailbox.class.getName() + "$Pending");
var constructor = pendingType.getDeclaredConstructor(String.class);
constructor.setAccessible(true);
Object pending = constructor.newInstance(msgId);
var seqField = LeadMailbox.class.getDeclaredField("pendingBySeq");
seqField.setAccessible(true);
((Map<Long, Object>) seqField.get(mailbox)).put(seq, pending);
pendingByMsgId(mailbox).put(msgId, pending);
}
private static void invoke(LeadMailbox mailbox, String name, Class<?>[] types, Object... args) throws Exception {
Method method = LeadMailbox.class.getDeclaredMethod(name, types);
method.setAccessible(true);
method.invoke(mailbox, args);
}
private static void awaitPending(LeadMailbox mailbox, String msgId) throws Exception {
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
while (!pendingByMsgId(mailbox).containsKey(msgId) && System.nanoTime() < deadline) {
Thread.sleep(10);
}
assertTrue(pendingByMsgId(mailbox).containsKey(msgId), "publish did not register " + msgId);
}
private static Object defaultValue(Class<?> type) {
if (!type.isPrimitive() || type == void.class) {
return null;
}
if (type == boolean.class) {
return Boolean.FALSE;
}
if (type == long.class) {
return 0L;
}
if (type == short.class) {
return (short) 0;
}
if (type == byte.class) {
return (byte) 0;
}
if (type == char.class) {
return (char) 0;
}
if (type == double.class) {
return 0.0d;
}
if (type == float.class) {
return 0.0f;
}
return 0;
}
}
@@ -22,6 +22,7 @@ import java.util.concurrent.TimeUnit;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -505,6 +506,52 @@ class MessageServiceTest {
"an answer to a turn that never existed (or already lapsed) is stale, not a hang");
}
/**
* fleetd #575. {@code answer()}'s STALE_TURN return for "the ask lapsed between the lookup and
* the unblock" ({@code rendezvous.answerAsk(turnId, content)} returning {@code false} even though
* this call's own {@code rendezvous.askSession(turnId)} check at the top saw the ask as open) used
* to be covered only by a hand-rolled copy of the cleanup pair its own finally already runs, not
* the finally itself — a structural gap, closed by widening the try up to cover the registration
* above it. This test pins the exact race with {@link
* MessageService#setAnswerAskLapseRaceHookForTest}, which fires right before this call's own
* {@code rendezvous.answerAsk} call and completes the same turnId's ask directly — reproducing
* what a second, concurrent {@code answer()} winning that race could otherwise only do by timing
* luck. Proves the fix changed no behaviour on this path: it still returns {@code STALE_TURN},
* and this call's own forward waiter is still closed exactly once (not left open, and not closed
* twice — there is now only one cleanup site left to run).
*/
@Test
void answerLosingTheRaceToAnAlreadyAnsweredAskStillReturnsStaleTurnAndCleansUpOnce() throws Exception {
String ticket = messages.sendAsync(T, "long task");
awaitWaiting();
injectDelivery();
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config?", 5000));
MessageService.TaskView asking = awaitTicketPhase(ticket, MessageService.Phase.ASKING);
String turnId = asking.turnId();
assertNotNull(turnId, "an ASKING view carries the turnId to answer on");
messages.setAnswerAskLapseRaceHookForTest(() -> rendezvous.answerAsk(turnId, "raced in first"));
try {
MessageService.Reply r = messages.answer(turnId, "too late", 500);
assertEquals(MessageService.Outcome.STALE_TURN, r.outcome(),
"an ask already answered by the race must be seen as lapsed, not double-delivered");
} finally {
messages.setAnswerAskLapseRaceHookForTest(null);
}
// Cleanup ran exactly once: the forward waiter THIS call opened is closed, not leaked.
assertNull(rendezvous.currentWaiter(T),
"the forward waiter this answer() call opened must be closed after a STALE_TURN return");
// The worker's own ask() call, unblocked by the hook's direct answerAsk, still completes
// normally — the race this test simulates does not strand it.
MessageService.AskResult a = ask.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.AskOutcome.ANSWERED, a.outcome());
assertEquals("raced in first", a.answer());
}
// --- timeout, answer, poll, and lock-contention edges ----------------------------------
@Test
@@ -554,6 +601,30 @@ class MessageServiceTest {
}
}
/**
* fleetd #571 (the acceptance test the ticket was filed for). The worker is idle so the injector
* attempts delivery, but the {@code agent.prompt} call itself fails with a herdr error that is
* not a confirmed absence (not a {@code *_not_found} code) — {@link Injector} marks the Pending
* {@code ATTEMPTED} (fleetd #551), meaning the call was made and whether it reached the pane is
* unknown. Before this fix, {@code send}'s {@code TimeoutException} branch collapsed
* {@code ATTEMPTED} into {@code TIMED_OUT_QUEUED} — a promise that the message will never arrive,
* which may already be false: {@code agent.prompt} pastes and submits in one call.
*/
@Test
void sendTimesOutWithAttemptedDeliveryReportsUnconfirmedNotQueued() throws Exception {
herdr.agentSendFailsWith("send_failed");
CompletableFuture<MessageService.Reply> send =
CompletableFuture.supplyAsync(() -> messages.send(T, "brief", 150));
awaitWaiting();
injector.onStatus(T, AgentStatus.IDLE); // triggers the failing delivery attempt → ATTEMPTED
MessageService.Reply r = send.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.TIMED_OUT_UNCONFIRMED, r.outcome(),
"an ATTEMPTED delivery must not collapse into TIMED_OUT_QUEUED — the message may "
+ "already have arrived in full, and TIMED_OUT_QUEUED promises it never will");
assertNull(r.text());
}
@Test
void answerTimesOutWhenTheResumedWorkerNeverReplies() throws Exception {
CompletableFuture<MessageService.Reply> send = sendAsync();
@@ -579,6 +650,183 @@ class MessageServiceTest {
assertEquals("config.yaml", a.answer());
}
// --- fleetd #572: answer() must release the session lock on EVERY exit, not just return it -----
//
// answer()'s session lock is released in an outer `finally` (MessageService.java:1217-1219) that
// wraps its whole body. Removing that one line survives the entire suite: coverage is not the
// gap, assertion is — every existing answer() test checks the RETURN VALUE, never that the lock
// it took is reacquirable afterward. If it is not, the session is wedged forever with no
// exception and no log line. Each test below drives answer() through one of its four exits
// (normal reply, TIMED_OUT_WORKING, ExecutionException, InterruptedException) and then proves
// reacquisition the only way that actually proves it: a bounded follow-up `send` on the SAME
// session must not come back BUSY. A `send` only reports BUSY when `tryLock` itself timed out
// (MessageService.java:926-928) — every other early return in `send` still passes through its own
// lock-acquired `try`, so a non-BUSY probe result is specifically evidence the lock was free.
/** The REPLIED exit (the happy path) — the resumed worker's real {@code fleet_reply} arrives. */
@Test
void answerReleasesTheSessionLockAfterANormalReply() throws Exception {
CompletableFuture<MessageService.Reply> send = sendAsync();
awaitWaiting();
injector.onStatus(T, AgentStatus.IDLE);
injector.onStatus(T, AgentStatus.WORKING);
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config file?", 5000));
MessageService.Reply q = send.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.QUESTION, q.outcome());
CompletableFuture<MessageService.Reply> answer =
CompletableFuture.supplyAsync(() -> messages.answer(q.turnId(), "config.yaml", 5000));
ask.get(5, TimeUnit.SECONDS); // worker resumed with the answer
awaitWaiting(); // the answering call has (re)opened its own forward waiter
assertTrue(rendezvous.resolve(T, "done"), "the worker's final reply resolves the answering send");
MessageService.Reply done = answer.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.REPLIED, done.outcome());
MessageService.Reply probe = messages.send(T, "probe after normal reply", 300);
assertNotEquals(MessageService.Outcome.BUSY, probe.outcome(),
"the session lock must be released after a normal REPLIED answer(), or this bounded "
+ "follow-up send would come back BUSY instead of timing out on its own work");
}
/**
* The {@code TIMED_OUT_WORKING} exit. Same setup as {@link #answerTimesOutWhenTheResumedWorkerNeverReplies}
* (which only checks {@code answer()}'s return value — exactly the assertion the fleetd #572
* mutation survives), plus the reacquisition proof that test does not make.
*
* <p>{@code answer()} MUST run on its own thread here, not the test's main thread: {@code lock}
* is a {@link java.util.concurrent.locks.ReentrantLock}, so a probe {@code send} issued by the
* SAME thread that (under the mutation) still "holds" it would reenter for free and report a
* false pass — reentrancy, not release. Measured while writing this test: with {@code answer()}
* called inline, this test stayed green under the mutation while its three siblings correctly
* went red.
*/
@Test
void answerReleasesTheSessionLockAfterATimedOutWorkingReturn() throws Exception {
CompletableFuture<MessageService.Reply> send = sendAsync();
awaitWaiting();
injector.onStatus(T, AgentStatus.IDLE);
injector.onStatus(T, AgentStatus.WORKING);
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config file?", 5000));
MessageService.Reply q = send.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.QUESTION, q.outcome());
// The primary answers, unblocking the worker; the worker never sends its follow-up
// fleet_reply, so the answering call rides out its short window as still-working.
CompletableFuture<MessageService.Reply> answer =
CompletableFuture.supplyAsync(() -> messages.answer(q.turnId(), "config.yaml", 200));
MessageService.Reply answered = answer.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.TIMED_OUT_WORKING, answered.outcome(),
"an answered worker that never replies times out as still working");
ask.get(5, TimeUnit.SECONDS); // drain: the worker resumed with the answer
MessageService.Reply probe = messages.send(T, "probe after timed-out-working", 300);
assertNotEquals(MessageService.Outcome.BUSY, probe.outcome(),
"the session lock must be released after a TIMED_OUT_WORKING answer(), or this bounded "
+ "follow-up send would come back BUSY instead of timing out on its own work");
}
/**
* The {@code ExecutionException} exit. Forces it directly on answer()'s own reopened forward
* waiter — {@code rendezvous.currentWaiter(T)} is the exact {@code CompletableFuture} its
* {@code reply.get(...)} is blocked on — rather than trying to make a real worker fail, since the
* failure mode under test is in answer()'s own wait, not in how it got triggered.
*/
@Test
void answerReleasesTheSessionLockWhenTheReplyFutureFailsExceptionally() throws Exception {
CompletableFuture<MessageService.Reply> send = sendAsync();
awaitWaiting();
injector.onStatus(T, AgentStatus.IDLE);
injector.onStatus(T, AgentStatus.WORKING);
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config file?", 5000));
MessageService.Reply q = send.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.QUESTION, q.outcome());
String turnId = q.turnId();
java.util.concurrent.atomic.AtomicReference<Throwable> caught =
new java.util.concurrent.atomic.AtomicReference<>();
Thread answerer = new Thread(() -> {
try {
messages.answer(turnId, "config.yaml", 5000);
caught.set(new AssertionError("expected answer() to throw"));
} catch (Throwable t) {
caught.set(t);
}
});
answerer.start();
awaitWaiting(); // answer() re-opened its forward waiter and is about to block on it
CompletableFuture<Rendezvous.Resolution> waiter = rendezvous.currentWaiter(T);
assertNotNull(waiter, "answer() should have registered a forward waiter for the worker session");
waiter.completeExceptionally(new RuntimeException("boom"));
answerer.join(5000);
assertFalse(answerer.isAlive(), "answer() should have left by throwing once its reply future failed");
Throwable thrown = caught.get();
assertNotNull(thrown, "answer() should have thrown");
assertTrue(thrown instanceof RuntimeException, "a RuntimeException cause is rethrown as-is: " + thrown);
assertEquals("boom", thrown.getMessage());
ask.get(5, TimeUnit.SECONDS); // drain: the worker resumed with the answer
MessageService.Reply probe = messages.send(T, "probe after exceptional failure", 300);
assertNotEquals(MessageService.Outcome.BUSY, probe.outcome(),
"the session lock must be released when answer()'s reply future fails exceptionally, "
+ "or this bounded follow-up send would come back BUSY instead of timing out on its own work");
}
/**
* The {@code InterruptedException} exit. Same shape as the {@code ExecutionException} case above,
* but the thread blocked in {@code reply.get(...)} is interrupted instead of the future failing.
*/
@Test
void answerReleasesTheSessionLockWhenInterrupted() throws Exception {
CompletableFuture<MessageService.Reply> send = sendAsync();
awaitWaiting();
injector.onStatus(T, AgentStatus.IDLE);
injector.onStatus(T, AgentStatus.WORKING);
CompletableFuture<MessageService.AskResult> ask =
CompletableFuture.supplyAsync(() -> messages.ask(T, "which config file?", 5000));
MessageService.Reply q = send.get(5, TimeUnit.SECONDS);
assertEquals(MessageService.Outcome.QUESTION, q.outcome());
String turnId = q.turnId();
java.util.concurrent.atomic.AtomicReference<Throwable> caught =
new java.util.concurrent.atomic.AtomicReference<>();
Thread answerer = new Thread(() -> {
try {
messages.answer(turnId, "config.yaml", 5000);
caught.set(new AssertionError("expected answer() to throw"));
} catch (Throwable t) {
caught.set(t);
}
});
answerer.start();
awaitWaiting(); // answer() re-opened its forward waiter and is about to block on it
answerer.interrupt();
answerer.join(5000);
assertFalse(answerer.isAlive(), "answer() should have left by throwing once its thread was interrupted");
Throwable thrown = caught.get();
assertNotNull(thrown, "answer() should have thrown");
assertTrue(thrown instanceof IllegalStateException,
"an interrupted wait is wrapped in IllegalStateException: " + thrown);
ask.get(5, TimeUnit.SECONDS); // drain: the worker resumed with the answer
MessageService.Reply probe = messages.send(T, "probe after interruption", 300);
assertNotEquals(MessageService.Outcome.BUSY, probe.outcome(),
"the session lock must be released when answer()'s wait is interrupted, or this bounded "
+ "follow-up send would come back BUSY instead of timing out on its own work");
}
@Test
void pollReturnsNullForAnUnknownTicket() {
assertNull(messages.poll("task-999999"), "a ticket that was never minted is unknown");
@@ -10,12 +10,13 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
class MemberRoleTest {
@Test
void theThreeRolesAreArchitectDevAndReviewer() {
assertEquals(3, MemberRole.values().length,
void theFourRolesAreArchitectDevHunterAndReviewer() {
assertEquals(4, MemberRole.values().length,
"a new role changes the charter, the role file, the skill and the authz row — "
+ "adding one is a deliberate act, so this count is meant to fail first");
assertEquals("architect", MemberRole.ARCHITECT.wireName());
assertEquals("dev", MemberRole.DEV.wireName());
assertEquals("hunter", MemberRole.HUNTER.wireName());
assertEquals("reviewer", MemberRole.REVIEWER.wireName());
}
@@ -30,6 +31,7 @@ class MemberRoleTest {
void parseIsCaseInsensitiveAndTrimsSurroundingSpace() {
assertSame(MemberRole.ARCHITECT, MemberRole.parse("Architect"));
assertSame(MemberRole.DEV, MemberRole.parse(" DEV "));
assertSame(MemberRole.HUNTER, MemberRole.parse("HuNtEr"));
assertSame(MemberRole.REVIEWER, MemberRole.parse("ReViEwEr"));
}
@@ -38,7 +40,7 @@ class MemberRoleTest {
IllegalArgumentException e =
assertThrows(IllegalArgumentException.class, () -> MemberRole.parse("archtiect"));
assertTrue(e.getMessage().contains("archtiect"), e.getMessage());
assertTrue(e.getMessage().contains("architect, dev, reviewer"),
assertTrue(e.getMessage().contains("architect, dev, hunter, reviewer"),
"a typo in config should be fixable from the message alone: " + e.getMessage());
}
@@ -8,6 +8,7 @@ import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.inject.Injector;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.inject.StatusPoller;
import dev.ltms.fleet.inject.MemberPresence;
import dev.ltms.fleet.msg.InMemoryReplyInbox;
@@ -165,6 +166,29 @@ class FleetAppTest {
assertEquals("degraded", mapper.readTree(res.body()).get("status").asText());
}
@Test
void healthzKeepsOkStatusAndReportsLoopHealthInItsBody() {
FleetMcp.LoopHealthSource loops = new FleetMcp.LoopHealthSource(
() -> LoopWatchdog.State.RUNNING, () -> LoopWatchdog.State.STOPPED);
FleetApp.HealthzResponse ok = FleetApp.healthzResponse(new FakeHerdr(), new FakeHerdr(), loops);
assertEquals(200, ok.status(), "a healthy herdr must keep /healthz at 200 regardless of loop states");
assertEquals(Map.of("statusPoller", "RUNNING", "sessionReaper", "STOPPED"), ok.body().get("loopHealth"),
"the /healthz body must report each loop state without making STOPPED an alarm");
}
@Test
void healthzKeepsDegradedStatusAndReportsLoopHealthInItsBody() {
FleetMcp.LoopHealthSource loops = new FleetMcp.LoopHealthSource(
() -> LoopWatchdog.State.RUNNING, () -> LoopWatchdog.State.STOPPED);
FleetApp.HealthzResponse degraded = FleetApp.healthzResponse(new FakeHerdr().healthy(false),
new FakeHerdr(), loops);
assertEquals(503, degraded.status(),
"an unreachable herdr must keep /healthz at 503 regardless of loop states");
assertEquals(Map.of("statusPoller", "RUNNING", "sessionReaper", "STOPPED"),
degraded.body().get("loopHealth"), "the degraded /healthz body must retain loop states");
}
@Test
void sessionsMapsWorkspaceList() throws Exception {
int port = startHealthy();
@@ -623,6 +647,28 @@ class FleetAppTest {
assertTrue(herdr.called("agent.prompt"), "message was injected");
}
/**
* fleetd #571: the worker is idle, so the poller attempts delivery, but the {@code agent.prompt}
* call itself fails with a herdr error that is not a confirmed absence — {@link
* dev.ltms.fleet.inject.Injector} marks this {@code ATTEMPTED}, meaning the call was made and
* whether it reached the pane is unknown. {@code writeReply}'s default arm must map this to its
* own {@code "unconfirmed"} status, not silently fall through to {@code "done"} (which would
* claim the delegation completed) nor collapse into {@code "queued"} (which would claim the
* message will never arrive, when it may already be sitting in the pane).
*/
@Test
void messageTimesOutUnconfirmedWhenDeliveryAttemptFails() throws Exception {
FakeHerdr herdr = new FakeHerdr().agentStatus("idle").agentSendFailsWith("send_failed");
int port = start(herdr, "http://gx00.gw:8000", Set.of("gx00.gw"));
HttpResponse<String> res = postMessage(port, "{\"content\":\"hi\",\"timeoutMs\":250}");
assertEquals(202, res.statusCode());
JsonNode body = mapper.readTree(res.body());
assertEquals("unconfirmed", body.get("status").asText(),
"an ATTEMPTED delivery must report its own status, not \"queued\" or \"done\"");
assertTrue(herdr.called("agent.prompt"), "delivery must have been attempted");
}
@Test
void messageRejectsBlankContent() throws Exception {
int port = startHealthy();
@@ -0,0 +1,91 @@
package dev.ltms.fleet.session;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.LongSupplier;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
class SessionReaperResilienceTest {
@Test
void anErrorInOneIterationDoesNotStopTheNextIteration() throws Exception {
CountDownLatch errorThrown = new CountDownLatch(1);
CountDownLatch nextIteration = new CountDownLatch(1);
AtomicBoolean first = new AtomicBoolean(true);
LongSupplier clock = () -> {
if (first.compareAndSet(true, false)) {
errorThrown.countDown();
throw new AssertionError("test Error from the first reap iteration");
}
nextIteration.countDown();
return System.nanoTime();
};
SessionReaper reaper = new SessionReaper(sessionManager(clock), 60, 1);
reaper.start();
try {
assertTrue(errorThrown.await(2, TimeUnit.SECONDS),
"the first reap iteration must throw its test Error");
assertTrue(nextIteration.await(2, TimeUnit.SECONDS),
"the reaper must continue to the next iteration after an Error");
} finally {
reaper.stop();
}
}
@Test
void anAbnormalExitClearsRunningSoStartCreatesANewLoop() throws Exception {
SessionReaper reaper = new SessionReaper(sessionManager(System::nanoTime), 60, -1);
reaper.start();
Thread first = threadOf(reaper);
first.join(2000);
assertFalse(runningOf(reaper), "an abnormal loop exit must clear running");
reaper.start();
Thread restarted = threadOf(reaper);
try {
assertNotSame(first, restarted, "start() must create a new loop after an abnormal exit");
restarted.join(2000);
} finally {
reaper.stop();
}
}
private static SessionManager sessionManager(LongSupplier clock) {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = new FleetConfig.Profile(
"ltms-local", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN",
List.of("ccs", "ltms-local"), "tab", "fleetd-workers",
"worker: {profile} #{n}", null, null, null);
ClaudeCodeLauncher launcher = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
return new SessionManager(launcher, new FakeWorktrees(), clock);
}
private static Thread threadOf(SessionReaper reaper) throws ReflectiveOperationException {
Field field = SessionReaper.class.getDeclaredField("thread");
field.setAccessible(true);
return (Thread) field.get(reaper);
}
private static boolean runningOf(SessionReaper reaper) throws ReflectiveOperationException {
Field field = SessionReaper.class.getDeclaredField("running");
field.setAccessible(true);
return field.getBoolean(reaper);
}
}
@@ -0,0 +1,170 @@
package dev.ltms.fleet.session;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.inject.LoopWatchdog;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Field;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.LongSupplier;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #544: {@link SessionReaper#health()} — the progress watchdog, not a liveness check. Every
* test drives {@link LoopWatchdog}'s clock explicitly (never real elapsed time), so the threshold
* crossing is deterministic rather than a race against the poll interval.
*/
class SessionReaperWatchdogTest {
private static final long INTERVAL_MILLIS = 50;
// Mirrors SessionReaper.staleAfterNanos(50): 50ms * the 12x multiplier = 600ms.
private static final long STALE_AFTER_NANOS = TimeUnit.MILLISECONDS.toNanos(INTERVAL_MILLIS) * 12;
@Test
void aLoopParkedInAReapCallThatNeverReturnsReportsStalled() throws Exception {
CountDownLatch entered = new CountDownLatch(1);
// reapIdle's very first statement reads sessions' own injected clock — blocking there
// parks the reaper thread mid-round, exactly as a wedged call would (fleetd #544).
SessionManager sessions = sessionManager(new BlocksOnFirstCall(entered));
AtomicLong watchdogClock = new AtomicLong(0);
SessionReaper reaper = new SessionReaper(sessions, 60, INTERVAL_MILLIS, watchdogClock::get);
reaper.start();
try {
assertTrue(entered.await(2, TimeUnit.SECONDS),
"the reaper must have entered the blocking reapIdle call");
assertEquals(LoopWatchdog.State.RUNNING, reaper.health(),
"freshly parked, before the threshold elapses, must still read RUNNING");
watchdogClock.set(STALE_AFTER_NANOS + 1);
assertEquals(LoopWatchdog.State.STALLED, reaper.health(),
"a loop parked mid-round past the staleness threshold must report STALLED");
} finally {
reaper.stop();
}
}
@Test
void aLoopThatDiedReportsStalled() throws Exception {
AtomicLong watchdogClock = new AtomicLong(0);
// intervalMillis=-1: Thread.sleep(-1) throws IllegalArgumentException, uncaught by loop(),
// which kills the carrier thread — the same "died" shape SessionReaperResilienceTest uses.
SessionReaper reaper = new SessionReaper(sessionManager(System::nanoTime), 60, -1, watchdogClock::get);
reaper.start();
Thread thread = threadOf(reaper);
thread.join(2000);
assertFalse(thread.isAlive(), "the loop must have died from Thread.sleep(-1)");
// staleAfterNanos(-1) = TimeUnit.MILLISECONDS.toNanos(max(-1,1)) * 12 = 12ms.
watchdogClock.set(TimeUnit.MILLISECONDS.toNanos(1) * 12 + 1);
assertEquals(LoopWatchdog.State.STALLED, reaper.health(),
"a dead loop's last-completed round goes stale and must report STALLED");
}
@Test
void aLoopStoppedOnPurposeReportsStoppedNotStalled() throws Exception {
AtomicLong watchdogClock = new AtomicLong(0);
SessionReaper reaper = new SessionReaper(sessionManager(System::nanoTime), 60, INTERVAL_MILLIS, watchdogClock::get);
reaper.start();
reaper.stop();
// Advance the clock far past any staleness threshold — an intentional stop must never
// be reported as STALLED no matter how old the last round looks (fleetd #512 shape).
watchdogClock.set(STALE_AFTER_NANOS * 100);
assertEquals(LoopWatchdog.State.STOPPED, reaper.health(),
"stop() must report STOPPED, never STALLED, however stale the last round looks");
}
@Test
void aRestartedLoopReportsRunningAgainNotStoppedForever() throws Exception {
// fleetd #544 review (issue comment #16944): stoppedByCaller is sticky, and reset() —
// called only from start() — is the sole thing that clears it. start() is documented
// idempotent and loop()'s own error line says "it can be restarted", so stop() followed
// by start() is an anticipated path. Without the reset() call in start(), health() would
// report STOPPED forever after a restart even though the loop is genuinely running again.
AtomicLong watchdogClock = new AtomicLong(0);
SessionReaper reaper = new SessionReaper(sessionManager(System::nanoTime), 60, INTERVAL_MILLIS, watchdogClock::get);
reaper.start();
reaper.stop();
reaper.start();
assertEquals(LoopWatchdog.State.RUNNING, reaper.health(),
"an intentional stop must not outlive the restart that follows it");
}
@Test
void aHealthyLoopReportsRunning() throws Exception {
// Real elapsed time on purpose, unlike the other tests here: a clock frozen at 0 would read
// RUNNING even if recordRoundComplete() were never wired into loop() at all, since the
// constructor's own initial timestamp already satisfies "not stale yet". Running for several
// multiples of this instance's own threshold (5ms interval * 12 = 60ms) and still reading
// RUNNING instead proves the loop's repeated round completions are the thing keeping the
// staleness deadline pushed forward.
long fastIntervalMillis = 5;
SessionReaper reaper = new SessionReaper(sessionManager(System::nanoTime), 60, fastIntervalMillis, System::nanoTime);
reaper.start();
try {
Thread.sleep(400);
assertEquals(LoopWatchdog.State.RUNNING, reaper.health(),
"a healthy loop's repeated round completions must keep pushing the staleness deadline forward");
} finally {
reaper.stop();
}
}
private static SessionManager sessionManager(LongSupplier clock) {
FakeHerdr herdr = new FakeHerdr();
FleetConfig.Profile cfg = new FleetConfig.Profile(
"ltms-local", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN",
List.of("ccs", "ltms-local"), "tab", "fleetd-workers",
"worker: {profile} #{n}", null, null, null);
ClaudeCodeLauncher launcher = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
return new SessionManager(launcher, new FakeWorktrees(), clock);
}
private static Thread threadOf(SessionReaper reaper) throws ReflectiveOperationException {
Field field = SessionReaper.class.getDeclaredField("thread");
field.setAccessible(true);
return (Thread) field.get(reaper);
}
/** Blocks forever on its first call, then falls back to real time if ever unblocked. */
private static final class BlocksOnFirstCall implements LongSupplier {
private final CountDownLatch entered;
private volatile boolean first = true;
private BlocksOnFirstCall(CountDownLatch entered) {
this.entered = entered;
}
@Override
public synchronized long getAsLong() {
if (first) {
first = false;
entered.countDown();
try {
// Blocks forever, mirroring a genuinely wedged call (fleetd #544) — a test
// double standing in for a stuck dependency, not a real socket.
new CountDownLatch(1).await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
return System.nanoTime();
}
}
}
@@ -0,0 +1,96 @@
package dev.ltms.fleet.testing;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import org.junit.jupiter.api.Test;
import org.slf4j.LoggerFactory;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* fleetd #537: pins {@link CapturedLog#close}'s own contract — the appender detach half, the level
* restore half, and that {@link CapturedLog#setLevel} does not change what {@code close} restores.
* Before this, only the level-restore half was pinned (by {@code
* WorktreeSessionManagerTest.sharedSessionManagerLoggerLevelIsRestoredAfterDirtyWorktreeReleasePinsWarn}).
* Measured: deleting {@code logger.detachAppender(appender);} from {@code close()} still left
* {@code mvn clean install} green — 1701 tests, 0 failures — before this file existed.
*
* <p>Every test below uses a logger name no production class uses, and unique per test, so this
* file cannot become the next entry in fleetd #525's leak family: {@link CapturedLog}'s own
* javadoc re-measure commands (top of that file) would otherwise need to start naming this class.
*/
class CapturedLogTest {
/**
* The appender must be detached on close: an event logged through the raw logger after close
* must not land in {@link CapturedLog#events()}. Asserting on the observable list (rather than
* {@code logger.iteratorForAppenders()}) is what the ticket asked for, and it is also what a
* real leak would actually break — a later test's own {@code ListAppender} silently gaining
* events emitted by code under test that has nothing to do with it.
*/
@Test
void closeDetachesTheAppenderSoALaterLogIsNotCaptured() {
String loggerName = "capturedlog-test-only.appender-detach";
Logger rawLogger = (Logger) LoggerFactory.getLogger(loggerName);
CapturedLog log = CapturedLog.of(loggerName);
rawLogger.info("while open");
int eventsWhileOpen = log.events().size();
assertEquals(1, eventsWhileOpen, "the event logged while open must be captured");
log.close();
rawLogger.info("after close");
assertEquals(eventsWhileOpen, log.events().size(),
"close() must detach the appender: an event logged after close must not be "
+ "captured, but the captured list grew from " + eventsWhileOpen + " to "
+ log.events().size());
}
/**
* The helper's own headline contract, pinned in one place independent of any production
* class's behaviour: {@code close()} restores the level the logger had before {@link
* CapturedLog#at} pinned it.
*/
@Test
void closeRestoresTheLevelCapturedAtOpen() {
String loggerName = "capturedlog-test-only.level-restore";
Logger rawLogger = (Logger) LoggerFactory.getLogger(loggerName);
rawLogger.setLevel(Level.DEBUG);
CapturedLog log = CapturedLog.at(loggerName, Level.ERROR);
assertEquals(Level.ERROR, rawLogger.getLevel(), "the pinned level took effect while open");
log.close();
assertEquals(Level.DEBUG, rawLogger.getLevel(),
"close() must restore the level captured when at() was called (DEBUG), not leave "
+ "the pinned level (ERROR) in place");
}
/**
* {@link CapturedLog#setLevel}'s javadoc claims that re-pinning the level mid-capture does not
* change what {@code close()} restores — that restore always uses the level captured when the
* instance was created, never a value set through {@code setLevel}. Nothing checked this
* before: open with a pinned WARN, call {@code setLevel(TRACE)}, close, and the result must be
* the level from BEFORE {@code at} — neither WARN nor TRACE.
*/
@Test
void setLevelDuringCaptureDoesNotChangeWhatCloseRestores() {
String loggerName = "capturedlog-test-only.setlevel-no-effect";
Logger rawLogger = (Logger) LoggerFactory.getLogger(loggerName);
rawLogger.setLevel(Level.DEBUG);
CapturedLog log = CapturedLog.at(loggerName, Level.WARN);
log.setLevel(Level.TRACE);
assertEquals(Level.TRACE, rawLogger.getLevel(), "setLevel took effect immediately");
log.close();
assertEquals(Level.DEBUG, rawLogger.getLevel(),
"close() must restore the level captured at open (DEBUG) regardless of any "
+ "later setLevel() call: it must be neither WARN (the level pinned by "
+ "at()) nor TRACE (the level set via setLevel() mid-capture), but got "
+ rawLogger.getLevel());
}
}
+395 -158
View File
@@ -96,13 +96,23 @@ SYSTEMD_UNIT='fleetd'
# any of the prose detail strings and needs no escaping in a `case`/glob pattern.
SUPERVISOR_DETAIL_SEP=$'\x1f'
# fleetd #492 follow-up: set by systemd_loaded/systemd_installed when the underlying `systemctl`
# call could not answer cleanly — it exited non-zero AND wrote something to stderr, which is a real
# tool failure (e.g. it cannot reach the user bus over a non-lingering ssh session), never the same
# fact as a clean negative answer ("not active", no stderr). Initialized here, not just inside the
# probes, so detect_supervisor can read them under `set -u` even before either probe has ever run,
# and so a test that stubs a probe with a plain `return 0`/`return 1` body (leaving these untouched)
# reads a deterministic 0 rather than whatever a previous probe call left behind.
# fleetd #492 follow-up, refined by fleetd #545: three states, not two, set by
# systemd_loaded/systemd_installed —
# 0 = no error, the probe ran and gave a clean answer.
# 1 = the probe RAN and answered badly: `systemctl` exited non-zero AND wrote something to
# stderr, a real tool failure (e.g. it cannot reach the user bus over a non-lingering ssh
# session), never the same fact as a clean negative answer ("not active", no stderr).
# 2 = the probe could not even be SET UP: the `mktemp` call that makes a place to capture
# `systemctl`'s stderr failed before `systemctl` ever ran. This is a fleetd #545 fix: on GNU
# coreutils (every Linux distribution) a template with no `X`s made `mktemp` fail every
# single time, and the two states were folded into one flag and one message that named
# cause 1 ("systemctl exited non-zero and reported an error on stderr") for a failure that
# was actually cause 2 — systemctl was never executed at all. One flag with two meanings
# needing different messages was the defect; a third value is the fix, not a second flag.
# Initialized here, not just inside the probes, so detect_supervisor can read them under `set -u`
# even before either probe has ever run, and so a test that stubs a probe with a plain
# `return 0`/`return 1` body (leaving these untouched) reads a deterministic 0 rather than whatever
# a previous probe call left behind.
SYSTEMD_LOADED_ERRORED=0
SYSTEMD_INSTALLED_ERRORED=0
# fleetd #492 follow-up: SUPERVISOR_UNCLEAR_DETAIL is the specific supervisor/reason that
@@ -130,11 +140,35 @@ ok() { printf ' ok %s\n' "$*"; }
warn() { printf ' WARN %s\n' "$*"; }
die() { printf '\n FAIL %s\n\n' "$*" >&2; exit 1; }
# fleetd #550 — shasum is macOS-only (it ships with Perl, which Debian/Ubuntu/etc. do not install
# by default); GNU coreutils (every mainstream Linux distro) ships sha256sum instead and has no
# shasum at all. Prefer sha256sum, fall back to shasum -a 256 — same idiom as
# probe-member-credentials.sh's `hasher` selection — and when NEITHER is on PATH, say so plainly.
# That third answer matters: without it, a missing hasher makes `cut` succeed on empty input, and
# under `set -o pipefail` the pipeline as a whole still fails, so a caller's own `|| echo "absent"`
# then reports a file that is right there as though it were gone. `hash256` never does that — it
# only ever hashes or says it could not.
hash256() {
local f="$1"
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$f" | cut -c1-12
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 "$f" | cut -c1-12
else
echo "unhashable"
fi
}
# Reports the hash of $JAR by default, or of whatever path is passed — used to report the STAGED
# jar right after a build (before it has been swapped in) without ever changing what a bare
# `jar_id` (no args) means: the live path, $JAR. --check and the final "pid ..., jar ..." line
# both call it with no args on purpose, so neither can ever be fooled by a leftover staged file.
jar_id() { local f="${1:-$JAR}"; [ -f "$f" ] && shasum -a 256 "$f" | cut -c1-12 || echo "absent"; }
# fleetd #550 — THREE distinct answers now, not two: `[ -f "$f" ]` already separates "the jar is
# not there" (-> "absent") from "the jar is there"; for the second case, hash256 itself separates
# "hashed it" (a 12-char hex string) from "could not hash it" (-> "unhashable", when no hasher is
# on PATH). "absent" must never be the answer for a file that exists — that conflation, on Linux,
# was the whole defect this ticket fixes.
jar_id() { local f="${1:-$JAR}"; [ -f "$f" ] && hash256 "$f" || echo "absent"; }
running_pid() { pgrep -f "$PATTERN" || true; }
# fleetd #493 — three small, independently testable pieces of "never build into the path a
@@ -238,12 +272,17 @@ launchd_loaded() { launchctl list "$LAUNCHD_LABEL" >/dev/null 2>&1; }
# fleetd #492 follow-up: both functions used to throw `systemctl`'s stderr straight into
# /dev/null, which meant "systemctl answered no" and "systemctl could not answer at all" (e.g. it
# cannot reach the user bus over a non-lingering ssh session) looked identical — both a plain
# nonzero exit. They now capture stderr separately and set their own *_ERRORED flag ONLY when the
# nonzero exit. They now capture stderr separately and set their own *_ERRORED flag to 1 when the
# call exited non-zero AND wrote something to stderr — a real tool failure, never a clean "not
# installed"/"not active" answer (which exits non-zero with empty stderr). detect_supervisor reads
# the flag right after calling the probe, so a probe that could not answer routes to "unclear",
# never silently becomes "none".
#
# fleetd #545: the flag has a third value, 2, set when the `mktemp` call that sets up the probe's
# own stderr capture fails, before `systemctl` ever runs — see the SYSTEMD_LOADED_ERRORED /
# SYSTEMD_INSTALLED_ERRORED comment above their initialization for why this is a third value on the
# same flag, not a second flag.
#
# "installed": a unit FILE by this name exists, regardless of its current state — the systemd
# analogue of the plist file existing on disk. `list-unit-files` reads unit definitions without
# depending on runtime state, so this stays read-only and safe under --check.
@@ -251,8 +290,8 @@ systemd_installed() {
SYSTEMD_INSTALLED_ERRORED=0
command -v systemctl >/dev/null 2>&1 || return 1
local err_file out rc=0
if ! err_file="$(mktemp -t systemd-installed-err)"; then
SYSTEMD_INSTALLED_ERRORED=1
if ! err_file="$(mktemp -t systemd-installed-err.XXXXXX)"; then
SYSTEMD_INSTALLED_ERRORED=2
return 1
fi
out="$(systemctl --user list-unit-files "$SYSTEMD_UNIT.service" --no-legend 2>"$err_file")" || rc=$?
@@ -275,8 +314,8 @@ systemd_loaded() {
SYSTEMD_LOADED_ERRORED=0
command -v systemctl >/dev/null 2>&1 || return 1
local err_file rc=0
if ! err_file="$(mktemp -t systemd-loaded-err)"; then
SYSTEMD_LOADED_ERRORED=1
if ! err_file="$(mktemp -t systemd-loaded-err.XXXXXX)"; then
SYSTEMD_LOADED_ERRORED=2
return 1
fi
systemctl --user is-active "$SYSTEMD_UNIT" >/dev/null 2>"$err_file" || rc=$?
@@ -302,7 +341,7 @@ systemd_loaded() {
# lost value instead of surfacing it (fleetd #497's defect class).
unload_launchd_if_loaded() {
local err_file rc=0
if ! err_file="$(mktemp -t launchd-unload-err)"; then
if ! err_file="$(mktemp -t launchd-unload-err.XXXXXX)"; then
die "could not create a temp file to capture 'launchctl unload' stderr — cannot tell a real
failure from a clean already-unloaded answer, so refusing to guess. The daemon's
supervision state was NOT touched."
@@ -317,7 +356,7 @@ unload_launchd_if_loaded() {
stop_systemd_if_loaded() {
local err_file rc=0
if ! err_file="$(mktemp -t systemd-stop-err)"; then
if ! err_file="$(mktemp -t systemd-stop-err.XXXXXX)"; then
die "could not create a temp file to capture 'systemctl --user stop' stderr — cannot tell a
real failure from a clean already-stopped answer, so refusing to guess. The daemon's
supervision state was NOT touched."
@@ -349,6 +388,14 @@ stop_systemd_if_loaded() {
# "none" now means only: neither supervisor is installed, neither is loaded, and neither probe
# errored.
#
# fleetd #545: *_ERRORED carries a THIRD state (2 = the probe's own mktemp setup failed, before
# `systemctl` ever ran — see the flag's own comment above its initialization), and it must never be
# reported with the same detail text as state 1 (`systemctl` ran and answered badly on stderr). The
# two are different facts about different failures, and conflating them makes the "unclear" message
# assert a cause ("systemctl exited non-zero and reported an error on stderr") that was never
# measured when the real cause was state 2. detect_supervisor below picks the detail text off the
# flag's value, not off a single "errored at all" boolean.
#
# fleetd #492 follow-up — constraints every caller of this function depends on (learned the hard
# way: an earlier version of this fix set a SUPERVISOR_UNCLEAR_DETAIL global from inside here and
# it was silently lost, because every real call site invokes this as `$(detect_supervisor)`):
@@ -382,7 +429,10 @@ detect_supervisor() {
launchd_installed && li=1
systemd_installed && si=1
if [ "$SYSTEMD_LOADED_ERRORED" = 1 ] || [ "$SYSTEMD_INSTALLED_ERRORED" = 1 ]; then
if [ "$SYSTEMD_LOADED_ERRORED" = 2 ] || [ "$SYSTEMD_INSTALLED_ERRORED" = 2 ]; then
detail="the systemd --user probe for '$SYSTEMD_UNIT' could not even be set up (a temp file to capture systemctl's stderr could not be created) — systemctl was never run, so this says nothing about systemd, the user bus, or the unit itself"
kind="unclear"
elif [ "$SYSTEMD_LOADED_ERRORED" = 1 ] || [ "$SYSTEMD_INSTALLED_ERRORED" = 1 ]; then
detail="the systemd --user probe for '$SYSTEMD_UNIT' could not answer cleanly (systemctl exited non-zero and reported an error on stderr, not a clean negative — e.g. it cannot reach the user bus)"
kind="unclear"
elif [ "$ld" = 1 ] && [ "$sd" = 1 ]; then
@@ -499,14 +549,51 @@ check_log_path_matches_plist() {
ok "log path check: script and plist agree ($resolved_out)"
}
# fleetd #552: the post-restart fresh-log capture, pulled out of the main flow so it is testable by
# sourcing (the same reason systemd_installed/systemd_loaded above guard their OWN mktemp inline
# instead of leaving it bare) even though its only caller sits below the SOURCED guard. By the time
# the caller reaches this, the daemon has already been stopped, the jar swapped, and the new daemon
# started — fleetd #552's whole point is that a `mktemp` failure here must never undo that success
# by aborting the script, the way the old unguarded `FRESH_LOG="$(mktemp ...)"` assignment did.
#
# So this function never dies, and — like detect_supervisor above — never reports failure through a
# global either: it is meant to be called as `VAR="$(capture_fresh_log_region ...)"`, which runs it
# in a subshell, and any global it set there would die with that subshell. Failure crosses back the
# only way that survives: a non-zero exit and empty stdout. The caller decides what a failure means
# (warn, and leave FRESH_LOG at its already-declared "" sentinel) and owns the EXIT trap, because
# that trap has to outlive this function's own return — the file must survive until every reader
# below (classify_amqp_connection_errors, report_shutdown_drain, the result section) has looked.
capture_fresh_log_region() {
local out_file="$1" restart_mark="$2" log_file
if ! log_file="$(mktemp -t fleetd-fresh-log.XXXXXX)"; then
return 1
fi
tail -n "+$((restart_mark + 1))" "$out_file" > "$log_file" 2>/dev/null || true
printf '%s' "$log_file"
}
# Classify ERROR lines in one fresh log region. AMQP failure messages now include the connection
# name, so a recovery can clear only errors for its own connection. A candidate with neither name
# remains unexplained: it must never be quieted by a recovery on the other connection.
#
# fleetd #552: log_file can now arrive as "" — capture_fresh_log_region's own sentinel for "the
# post-restart region could never be captured". That is a DIFFERENT fact from "captured a region
# with nothing in it", which is exactly what every count below staying at its initialized 0 already
# means; conflating the two would make "no errors found" and "could not look for errors" print
# identically; the exact defect class this ticket exists to fix. REDEPLOY_AMQP_CHECK_SKIPPED carries
# the distinction to the result section below, and this reader says so in its own output too.
classify_amqp_connection_errors() {
local log_file="$1" line pending_inbox=0 pending_lead_mailbox=0
REDEPLOY_ERROR_COUNT=0
REDEPLOY_RECOVERED_AMQP_ERRORS=0
REDEPLOY_UNEXPLAINED_ERRORS=0
REDEPLOY_AMQP_CHECK_SKIPPED=0
if [ -z "$log_file" ]; then
REDEPLOY_AMQP_CHECK_SKIPPED=1
warn "cannot check for AMQP/ERROR lines since restart — the post-restart log region could not be captured"
return 0
fi
while IFS= read -r line || [ -n "$line" ]; do
case "$line" in
@@ -630,6 +717,19 @@ report_shutdown_drain() {
return 0
fi
# fleetd #552: checked AFTER the n/a case above, not before it — a cold start has nothing to check
# regardless of whether the log region could be captured, and that fact must not be overridden by
# an unrelated mktemp failure. "skipped" is a fifth state on REDEPLOY_DRAIN_STATE, distinct from
# "unknown" (the log WAS read, and neither a drain-complete line nor an exception shape was found
# in it): here there was no log to read at all, and that is a different fact that must say so in
# its own output, never silently reported as "unknown" or, worse, as "complete"/"n/a".
if [ -z "$log_file" ]; then
REDEPLOY_DRAIN_STATE="skipped"
warn "cannot tell whether the previous daemon's shutdown drain finished — the post-restart log"
warn "region could not be captured"
return 0
fi
find_drain_complete_line "$log_file"
scan_uncaught_exceptions "$log_file"
@@ -706,6 +806,229 @@ refuse_drain_gate() {
die "$(drain_gate_refusal "$do_build" "$staged_path")"
}
# fleetd #555 — the main flow itself was almost entirely untestable: sourcing this file (the SOURCED
# guard below) stops before a single line of the report/build/drain/stop/swap/start/verify sequence
# ever runs, and every decision in that sequence used to live as a bare `if`/`case` written directly
# into the main flow rather than inside a function. Three tests reached that flow at all
# (test_swap_ordered_after_wait_and_before_start, test_refuse_drain_gate_call_site_present,
# test_report_shutdown_drain_call_site_present), and all three do it by grepping this file's own
# source for a call site — which proves the call site exists, never that the guard around it still
# reaches it. Every function below follows the swap_if_built/refuse_drain_gate shape those two
# tickets (#521/#528) already established: the decision (a small, separately-tested predicate) and
# the action it gates live together in ONE function, and the main flow calls that function
# unconditionally — so there is no bare guard left in the main flow for a future edit to invert
# silently. See test_no_untested_main_flow_conditionals in test-redeploy-fleetd.sh for the structural
# guard that keeps a ninth bare conditional from arriving the same way these eight did.
# Ticket item 1 — `--check` must stay genuinely read-only. Inverting the guard used to mean --check
# performs a real redeploy; now the guard is should_stop_for_check, and stop_if_check_only is the
# only thing the main flow calls.
should_stop_for_check() {
local check_only="$1"
[ "$check_only" = 1 ]
}
stop_if_check_only() {
local check_only="$1"
should_stop_for_check "$check_only" || return 0
say "--check: nothing changed"
exit 0
}
# Ticket items 2 and 3 — the drain-gate entry (`-n "$OLD_PID" && "$ASSUME_YES" = 0`) and the reply
# comparison (`"$reply" != "yes"`). drain_gate_required and drain_confirmed are the two predicates;
# run_drain_gate is the only thing the main flow calls, and it is the one place that reads $reply at
# all, so a test can drive it end-to-end over stdin.
drain_gate_required() {
local old_pid="$1" assume_yes="$2"
[ -n "$old_pid" ] && [ "$assume_yes" = 0 ]
}
drain_confirmed() {
[ "$1" = "yes" ]
}
run_drain_gate() {
local old_pid="$1" assume_yes="$2" do_build="$3" staged_path="$4" reply
drain_gate_required "$old_pid" "$assume_yes" || return 0
say "drain check"
echo " A restart drops every in-flight ticket and rendezvous. A member's report"
echo " is NOT recoverable once its ticket is gone."
echo
echo " Confirm with fleet_list that no members are live, and fleet_poll anything"
echo " you still want, BEFORE continuing."
echo
read -r -p " Fleet drained? type yes to restart: " reply
drain_confirmed "$reply" || refuse_drain_gate "$do_build" "$staged_path"
}
# Ticket item 4 — the report-state dispatch on $SUPERVISOR_KIND. Inverting this used to report the
# wrong supervisor and, for the launchd arm specifically, skip check_log_path_matches_plist — whose
# own comment already says every check after it is worthless if it never runs.
report_supervisor_state() {
local kind="$1"
SUPERVISED=0
case "$kind" in
launchd)
SUPERVISED=1
ok "launchd agent loaded ($LAUNCHD_LABEL) — launchd supervises this daemon"
check_log_path_matches_plist "$OUT" "$LAUNCHD_PLIST"
;;
systemd)
SUPERVISED=1
ok "systemd --user unit active ($SYSTEMD_UNIT) — systemd supervises this daemon"
;;
none)
warn "no supervisor loaded — this script is the only thing that will restart the daemon."
;;
*)
warn "unrecognised supervisor kind: '$kind' — detect_supervisor returned a value this block does not know; continuing to report the rest of the state."
;;
esac
}
# Ticket item 5 — the stop dispatch on $SUPERVISOR_KIND. Inverting this kills a supervised daemon
# with a raw `kill` instead of `launchctl unload`/`systemctl --user stop`, so the supervisor revives
# the OLD jar — the exact CB-594/#492 failure both branches exist to prevent. Each mechanism is its
# own thin, overridable function (same seam launchd_installed/launchd_loaded already use) so a test
# can prove dispatch_stop picks the right one without ever calling real launchctl/systemctl/kill.
stop_via_launchd() {
echo " supervision is ON (launchd): using 'launchctl unload' (not kill) so launchd's own"
echo " KeepAlive cannot restart the OLD jar out from under this script — see the CB-594"
echo " comment above."
launchctl unload -w "$LAUNCHD_PLIST" \
|| die "launchctl unload failed — the daemon may still be under supervision; investigate before retrying"
}
stop_via_systemd() {
echo " supervision is ON (systemd --user): using 'systemctl --user stop' (not kill) so"
echo " systemd's own Restart=on-failure cannot restart the OLD jar out from under this"
echo " script — see the fleetd #492 comment above."
systemctl --user stop "$SYSTEMD_UNIT" \
|| die "'systemctl --user stop $SYSTEMD_UNIT' failed — the daemon may still be under supervision; investigate before retrying"
}
stop_via_kill() {
local pid="$1"
kill "$pid"
}
dispatch_stop() {
local kind="$1" pid="$2"
case "$kind" in
launchd) stop_via_launchd ;;
systemd) stop_via_systemd ;;
none) stop_via_kill "$pid" ;;
*)
die "detect_supervisor returned an unrecognized value '$kind' at the stop step —
refusing to guess how to stop a daemon under an unknown supervisor. The daemon was NOT
stopped." ;;
esac
}
# Ticket item 6 — the start dispatch on $SUPERVISOR_KIND. Inverting this starts the daemon by the
# wrong mechanism; for launchd the branch also carries the retry that keeps a failed load from
# leaving the agent stopped-and-disabled. Same shape as the stop dispatch above.
start_via_launchd() {
echo " supervision is ON (launchd): using 'launchctl load' so launchd starts and keeps"
echo " supervising this process, instead of a manual nohup that launchd would know nothing"
echo " about."
# CB-600: 'launchctl unload -w' above already persisted Disabled=true for this label. A load -w
# that succeeds clears it; a load -w that FAILS leaves the agent both stopped and disabled — worse
# than before this script ran, because a later reboot or login will not bring it back either. One
# retry covers a transient race (e.g. launchd not yet fully done deregistering); if it still fails,
# die with the exact recovery command rather than a bare "failed".
if ! launchctl load -w "$LAUNCHD_PLIST" 2>/dev/null; then
warn "launchctl load failed on the first attempt — retrying once after a short pause"
sleep 2
launchctl load -w "$LAUNCHD_PLIST" || die "launchctl load failed twice.
The agent is now STOPPED and DISABLED — it will NOT come back on its own, not even after a
reboot or login, because 'launchctl unload -w' above persisted Disabled=true and load -w
never got the chance to clear it. Recover with:
launchctl load -w \"$LAUNCHD_PLIST\"
If that still fails, check 'launchctl list $LAUNCHD_LABEL', validate the plist with
'plutil -lint \"$LAUNCHD_PLIST\"', and check $OUT before assuming a retry will succeed."
fi
}
start_via_systemd() {
echo " supervision is ON (systemd --user): using 'systemctl --user start' so systemd starts"
echo " and keeps supervising this process, instead of a manual nohup it would know nothing"
echo " about."
systemctl --user start "$SYSTEMD_UNIT" || die "'systemctl --user start $SYSTEMD_UNIT' failed.
Check 'systemctl --user status $SYSTEMD_UNIT' and $OUT before assuming a retry will succeed."
}
start_via_none() {
# Absolute jar path so `ps` names which checkout is running.
( cd "$MODULE" && zsh -lc "nohup java -jar '$JAR' >> fleetd.out 2>&1 &" )
}
dispatch_start() {
local kind="$1"
case "$kind" in
launchd) start_via_launchd ;;
systemd) start_via_systemd ;;
none) start_via_none ;;
*)
die "detect_supervisor returned an unrecognized value '$kind' at the start step —
refusing to guess how to start a daemon under an unknown supervisor. The daemon was NOT
started." ;;
esac
}
# Ticket item 7 — the HEALTH_BODY poll loop and the `[ -z "$HEALTH_BODY" ]` branch. Inverting the
# latter (health_is_up) makes a dead daemon report /healthz 200 or a live one report failure.
# poll_health_body is pulled out too so the loop's own `if ...; then break; fi` is inside a function
# rather than sitting bare in the main flow.
poll_health_body() {
local url="$1" wait_s="$2" body="" _i
for _i in $(seq "$wait_s"); do
if body="$(curl -fsS --max-time 2 "$url" 2>/dev/null)"; then
printf '%s' "$body"
return 0
fi
sleep 1
done
return 1
}
health_is_up() {
[ -n "$1" ]
}
report_health() {
local body="$1" code="$2" out_file="$3" wait_s="$4"
if health_is_up "$body"; then
ok "/healthz 200 — $body"
warn "healthz green only proves herdr ANSWERS. If its protocol number changed, spawns can still"
warn "fail — prove a real spawn before trusting the fleet."
return 0
fi
# 503 still means the daemon is up — it means herdr is unreachable. Say which.
if [ "$code" = "503" ]; then
warn "/healthz answers 503 degraded — the daemon is up but herdr is unreachable."
warn "Spawns will fail. Check herdr before delegating anything."
curl -s --max-time 2 "$HEALTH" 2>/dev/null | head -3 || true
else
die "/healthz never answered within ${wait_s}s (last code: $code). Last lines of $out_file:
$(tail -30 "$out_file" 2>/dev/null)"
fi
}
# Ticket item 8 — `HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1`. Measured safe under `set -e`
# at both bash 3.2.57 and 5.x (see the header comment trap 9 discussion in the ticket) — not a `set
# -e` hazard, but still an untested computation feeding report_shutdown_drain's own four-way
# decision. compute_had_old_pid makes the mapping itself directly testable.
compute_had_old_pid() {
local old_pid="$1"
if [ -n "$old_pid" ]; then
echo 1
else
echo 0
fi
}
# CB-600: sourceable for testing. When this file is SOURCED (not executed) it stops here — nothing
# below runs — so a test harness can `source` it to call check_log_path_matches_plist (or the
# other pure helpers above) against a throwaway plist fixture without ever reaching the mutating
@@ -754,30 +1077,11 @@ SUPERVISOR_KIND="${SUPERVISOR_RAW%%"$SUPERVISOR_DETAIL_SEP"*}"
SUPERVISOR_UNCLEAR_DETAIL="${SUPERVISOR_RAW#*"$SUPERVISOR_DETAIL_SEP"}"
require_drivable_supervisor "$SUPERVISOR_KIND"
ok "supervisor detected: $SUPERVISOR_KIND"
SUPERVISED=0
case "$SUPERVISOR_KIND" in
launchd)
SUPERVISED=1
ok "launchd agent loaded ($LAUNCHD_LABEL) — launchd supervises this daemon"
# CB-600: fail loudly here, before ANY other check runs, if this script and the loaded plist
# would read different log files — every check after this point is worthless otherwise.
check_log_path_matches_plist "$OUT" "$LAUNCHD_PLIST"
;;
systemd)
SUPERVISED=1
ok "systemd --user unit active ($SYSTEMD_UNIT) — systemd supervises this daemon"
;;
none)
warn "no supervisor loaded — this script is the only thing that will restart the daemon."
;;
*)
# fleetd #492 follow-up: this block only DISPLAYS state, it changes nothing yet — so a value
# it doesn't recognise gets reported, not an abort that goes silent on exactly the state most
# worth seeing. (Unreachable today: require_drivable_supervisor above already died on
# "ambiguous"/"unclear" before this case runs. Guards the value nobody has invented yet.)
warn "unrecognised supervisor kind: '$SUPERVISOR_KIND' — detect_supervisor returned a value this block does not know; continuing to report the rest of the state."
;;
esac
# fleetd #555: the dispatch below used to be a bare `case` written directly here — see
# report_supervisor_state above for why it is now the only thing this line calls. (Unreachable
# today for the `*)` arm: require_drivable_supervisor above already died on "ambiguous"/"unclear"
# before this call runs. Guards the value nobody has invented yet.)
report_supervisor_state "$SUPERVISOR_KIND"
# The trap with no log line. Checked in a LOGIN shell, because that is how the daemon is started
# below. Never prints the value — only whether it resolved.
@@ -817,10 +1121,9 @@ else
warn "Replies stop surviving a restart — a held report is lost, not delayed."
fi
if [ "$CHECK_ONLY" = 1 ]; then
say "--check: nothing changed"
exit 0
fi
# fleetd #555: should_stop_for_check/stop_if_check_only above — --check must stay genuinely
# read-only, and this is the only place that decides whether to stop here.
stop_if_check_only "$CHECK_ONLY"
# ---------------------------------------------------------------------- build
# Deliberately before the stop: a failed build must never leave the fleet down.
@@ -830,7 +1133,7 @@ if [ "$DO_BUILD" = 1 ]; then
# fleetd #493: wipe a leftover staged jar from a previous failed/interrupted run BEFORE doing
# anything else, so that run's leftovers can never be mistaken for this run's output.
rm -f "$JAR_STAGED"
BUILD_LOG="$(mktemp -t fleetd-build)"
BUILD_LOG="$(mktemp -t fleetd-build.XXXXXX)"
echo " log: $BUILD_LOG"
if ! mvn -f "$MODULE/pom.xml" clean install > "$BUILD_LOG" 2>&1; then
grep -E 'ERROR|BUILD FAILURE|Tests run:.*Failures: [1-9]|Tests run:.*Errors: [1-9]' "$BUILD_LOG" \
@@ -854,23 +1157,11 @@ fi
# ----------------------------------------------------------------- drain gate
if [ -n "$OLD_PID" ] && [ "$ASSUME_YES" = 0 ]; then
say "drain check"
echo " A restart drops every in-flight ticket and rendezvous. A member's report"
echo " is NOT recoverable once its ticket is gone."
echo
echo " Confirm with fleet_list that no members are live, and fleet_poll anything"
echo " you still want, BEFORE continuing."
echo
read -r -p " Fleet drained? type yes to restart: " reply
if [ "$reply" != "yes" ]; then
# fleetd #493 / #517 / #528: "nothing changed" would be a lie once a build has run and staged a
# jar — see drain_gate_refusal above for the full decision and why each of its four cases reads
# the way it does. refuse_drain_gate composes that message AND calls die itself, so this guard
# has nothing left of its own to get wrong beyond whether it calls refuse_drain_gate at all.
refuse_drain_gate "$DO_BUILD" "$JAR_STAGED"
fi
fi
# fleetd #555: run_drain_gate above is drain_gate_required + the prompt + drain_confirmed, called
# unconditionally — it returns immediately when the gate is not required, and composes/dies through
# refuse_drain_gate itself when the reply does not confirm. See #493/#517/#528 for why "nothing
# changed" would be a lie once a build has staged a jar.
run_drain_gate "$OLD_PID" "$ASSUME_YES" "$DO_BUILD" "$JAR_STAGED"
# ------------------------------------------------------------------ stop
#
@@ -888,33 +1179,9 @@ fi
if [ -n "$OLD_PID" ]; then
say "stop"
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)" # verify a FRESH line appears later
case "$SUPERVISOR_KIND" in
launchd)
echo " supervision is ON (launchd): using 'launchctl unload' (not kill) so launchd's own"
echo " KeepAlive cannot restart the OLD jar out from under this script — see the CB-594"
echo " comment above."
launchctl unload -w "$LAUNCHD_PLIST" \
|| die "launchctl unload failed — the daemon may still be under supervision; investigate before retrying"
;;
systemd)
echo " supervision is ON (systemd --user): using 'systemctl --user stop' (not kill) so"
echo " systemd's own Restart=on-failure cannot restart the OLD jar out from under this"
echo " script — see the fleetd #492 comment above."
systemctl --user stop "$SYSTEMD_UNIT" \
|| die "'systemctl --user stop $SYSTEMD_UNIT' failed — the daemon may still be under supervision; investigate before retrying"
;;
none)
kill "$OLD_PID"
;;
*)
# fleetd #492 follow-up: this block ACTS (stops the daemon one specific way per kind) — an
# unrecognised value must never fall through to a default action, silently picking the wrong
# one (or none at all) while reporting success. (Unreachable today: require_drivable_
# supervisor already died before this runs. Guards the value nobody has invented yet.)
die "detect_supervisor returned an unrecognized value '$SUPERVISOR_KIND' at the stop step —
refusing to guess how to stop a daemon under an unknown supervisor. The daemon was NOT
stopped." ;;
esac
# fleetd #555: dispatch_stop above — the launchd/systemd/none/* case lives there now, so this
# line is the only thing left in the main flow to get wrong.
dispatch_stop "$SUPERVISOR_KIND" "$OLD_PID"
if ! wait_for_daemon_exit "$STOP_WAIT"; then
die "pid $OLD_PID still alive after ${STOP_WAIT}s. Not escalating to kill -9 automatically:
the shutdown hook releases sessions and worktrees in order, and killing it hard can
@@ -963,48 +1230,9 @@ swap_if_built "$DO_BUILD"
# above) and WorkingDirectory is already pinned to fleetd/.
say "start"
case "$SUPERVISOR_KIND" in
launchd)
echo " supervision is ON (launchd): using 'launchctl load' so launchd starts and keeps"
echo " supervising this process, instead of a manual nohup that launchd would know nothing"
echo " about."
# CB-600: 'launchctl unload -w' above already persisted Disabled=true for this label. A load -w
# that succeeds clears it; a load -w that FAILS leaves the agent both stopped and disabled — worse
# than before this script ran, because a later reboot or login will not bring it back either. One
# retry covers a transient race (e.g. launchd not yet fully done deregistering); if it still fails,
# die with the exact recovery command rather than a bare "failed".
if ! launchctl load -w "$LAUNCHD_PLIST" 2>/dev/null; then
warn "launchctl load failed on the first attempt — retrying once after a short pause"
sleep 2
launchctl load -w "$LAUNCHD_PLIST" || die "launchctl load failed twice.
The agent is now STOPPED and DISABLED — it will NOT come back on its own, not even after a
reboot or login, because 'launchctl unload -w' above persisted Disabled=true and load -w
never got the chance to clear it. Recover with:
launchctl load -w \"$LAUNCHD_PLIST\"
If that still fails, check 'launchctl list $LAUNCHD_LABEL', validate the plist with
'plutil -lint \"$LAUNCHD_PLIST\"', and check $OUT before assuming a retry will succeed."
fi
;;
systemd)
echo " supervision is ON (systemd --user): using 'systemctl --user start' so systemd starts"
echo " and keeps supervising this process, instead of a manual nohup it would know nothing"
echo " about."
systemctl --user start "$SYSTEMD_UNIT" || die "'systemctl --user start $SYSTEMD_UNIT' failed.
Check 'systemctl --user status $SYSTEMD_UNIT' and $OUT before assuming a retry will succeed."
;;
none)
# Absolute jar path so `ps` names which checkout is running.
( cd "$MODULE" && zsh -lc "nohup java -jar '$JAR' >> fleetd.out 2>&1 &" )
;;
*)
# fleetd #492 follow-up: this block ACTS (starts the daemon one specific way per kind) — an
# unrecognised value must never fall through to a default action, silently picking the wrong
# one (or none at all) while reporting success. (Unreachable today: require_drivable_
# supervisor already died before this runs. Guards the value nobody has invented yet.)
die "detect_supervisor returned an unrecognized value '$SUPERVISOR_KIND' at the start step —
refusing to guess how to start a daemon under an unknown supervisor. The daemon was NOT
started." ;;
esac
# fleetd #555: dispatch_start above — the launchd/systemd/none/* case (and the launchd retry) live
# there now, so this line is the only thing left in the main flow to get wrong.
dispatch_start "$SUPERVISOR_KIND"
for _ in $(seq 10); do
NEW_PID="$(running_pid)"
@@ -1020,29 +1248,14 @@ ok "started, pid $NEW_PID"
say "verify"
HEALTH_BODY=""
for _ in $(seq "$HEALTH_WAIT"); do
if HEALTH_BODY="$(curl -fsS --max-time 2 "$HEALTH" 2>/dev/null)"; then break; fi
HEALTH_BODY=""
sleep 1
done
if [ -z "$HEALTH_BODY" ]; then
# 503 still means the daemon is up — it means herdr is unreachable. Say which.
CODE="$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 "$HEALTH" 2>/dev/null || echo 000)"
if [ "$CODE" = "503" ]; then
warn "/healthz answers 503 degraded — the daemon is up but herdr is unreachable."
warn "Spawns will fail. Check herdr before delegating anything."
curl -s --max-time 2 "$HEALTH" 2>/dev/null | head -3 || true
else
die "/healthz never answered within ${HEALTH_WAIT}s (last code: $CODE). Last lines of $OUT:
$(tail -30 "$OUT" 2>/dev/null)"
fi
else
ok "/healthz 200 — $HEALTH_BODY"
warn "healthz green only proves herdr ANSWERS. If its protocol number changed, spawns can still"
warn "fail — prove a real spawn before trusting the fleet."
fi
# fleetd #555: poll_health_body/health_is_up/report_health above. HEALTH_CODE is only ever
# consulted by report_health when the body came back empty; `|| true` on both assignments is the
# same "an absent/failing command substitution must not kill the script under set -e" idiom the
# swap/drain helpers already rely on (see poll_health_body's own comment).
HEALTH_BODY="$(poll_health_body "$HEALTH" "$HEALTH_WAIT")" || true
HEALTH_CODE="000"
[ -n "$HEALTH_BODY" ] || HEALTH_CODE="$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 "$HEALTH" 2>/dev/null || echo 000)"
report_health "$HEALTH_BODY" "$HEALTH_CODE" "$OUT" "$HEALTH_WAIT"
# A fresh listening line, strictly after the restart mark. An old daemon that never died would
# otherwise let an old line pass for a new one.
@@ -1060,9 +1273,24 @@ tail -n "+$((RESTART_MARK + 1))" "$OUT" 2>/dev/null \
# Errors since the restart, anchored to the marker so old noise cannot leak in. Keep the fresh
# region in a file because the classifier must preserve the order of errors and recoveries.
FRESH_LOG="$(mktemp -t fleetd-fresh-log)"
#
# fleetd #552: by this point the daemon has already been stopped, the jar swapped, and the new
# daemon started. The old unguarded `FRESH_LOG="$(mktemp ...)"` here aborted the whole script on any
# mktemp failure — AFTER all of that had already succeeded — so a caller read the resulting non-zero
# exit as "the redeploy failed" and the likely next action was to restart an already-correctly-
# restarted daemon. The mktemp itself now lives in capture_fresh_log_region (guarded the same way
# :293/:317/:344/:359 guard their own), and a failure there only warns: there is nothing left to
# protect by refusing after a successful restart. FRESH_LOG="" is declared and the trap installed
# BEFORE the call that may fail to fill it in, not after — an empty FRESH_LOG makes `rm -f ""` a
# harmless no-op, so there is no ordering hazard left between "the file might already exist" and
# "the trap that cleans it up".
FRESH_LOG=""
trap 'rm -f "$FRESH_LOG"' EXIT
tail -n "+$((RESTART_MARK + 1))" "$OUT" > "$FRESH_LOG" 2>/dev/null || true
if ! FRESH_LOG="$(capture_fresh_log_region "$OUT" "$RESTART_MARK")"; then
warn "could not create a temp file for the post-restart log region — the restart itself already"
warn "succeeded (pid $NEW_PID); only the post-restart ERROR/drain checks below are skipped."
warn "Check $OUT yourself for anything since the restart."
fi
classify_amqp_connection_errors "$FRESH_LOG"
# fleetd #512 part 2: the previous daemon's shutdown drain, checked in the same fresh-log region —
@@ -1070,7 +1298,9 @@ classify_amqp_connection_errors "$FRESH_LOG"
# "cannot tell"). HAD_OLD_PID crosses in whether a previous daemon was actually stopped this run;
# see the function's own comment for why that matters.
say "previous daemon's shutdown drain"
HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1
# fleetd #555: compute_had_old_pid above — the mapping itself is now directly testable rather than
# an inline `&&` idiom sitting bare in the main flow.
HAD_OLD_PID="$(compute_had_old_pid "$OLD_PID")"
report_shutdown_drain "$FRESH_LOG" "$HAD_OLD_PID"
# fleetd #492: checked here, after healthz and the fresh-log check have both had time to run, so a
@@ -1081,7 +1311,14 @@ assert_single_daemon "$(running_pid)"
say "result"
ok "pid $NEW_PID, jar $(jar_id)"
if [ "$REDEPLOY_ERROR_COUNT" -eq 0 ]; then
if [ "$REDEPLOY_AMQP_CHECK_SKIPPED" -eq 1 ]; then
# fleetd #552: the fourth reader of the fresh-log region. Without this branch REDEPLOY_ERROR_COUNT
# stays at its untouched 0 (classify_amqp_connection_errors never got a file to read) and
# REDEPLOY_DRAIN_STATE is "skipped" (never "complete"/"n/a"), so the branches below would print
# NOTHING about ERROR lines at all — a silence that reads exactly like the clean bill of health
# this ticket exists to prevent. Say the unknown result out loud instead.
warn "ERROR-line count since restart is UNKNOWN — the post-restart log region could not be captured (see above)"
elif [ "$REDEPLOY_ERROR_COUNT" -eq 0 ]; then
# fleetd #512 item 4: this line must not print when the shutdown-drain check above found the
# previous daemon's drain died, or could not tell — either would make "no ERROR lines" read as a
# clean bill of health it is not (an uncaught exception never carries an ERROR token to begin
+853 -9
View File
@@ -128,8 +128,235 @@ STUB
launchd_loaded() { return 1; }
result="$(PATH="$bin_dir:$PATH" detect_supervisor)"
assert_equals "unclear" "$(supervisor_kind_of "$result")" "a systemd probe error must read as unclear, not none"
printf '%s' "$(supervisor_detail_of "$result")" | grep -qF "$SYSTEMD_UNIT" \
local detail
detail="$(supervisor_detail_of "$result")"
printf '%s' "$detail" | grep -qF "$SYSTEMD_UNIT" \
|| fail "detail does not name the systemd unit whose probe errored"
# fleetd #545: this is the PROBE-RAN-AND-ANSWERED-BADLY case (systemctl actually executed and
# wrote to stderr) — it must carry that story and never the SET-UP-FAILED story (mktemp never
# even ran here), or the two "unclear" causes have collapsed back into one message that asserts a
# cause it did not measure, which is the exact defect this ticket exists to fix.
printf '%s' "$detail" | grep -qF "systemctl exited non-zero and reported an error on stderr" \
|| fail "detail does not say systemctl ran and answered with stderr: $detail"
printf '%s' "$detail" | grep -qF "could not even be set up" \
&& fail "detail wrongly claims the probe could not be set up, but systemctl actually ran and answered on stderr: $detail"
return 0
}
# fleetd #545 — the companion case to the probe-error test above: here `mktemp` itself fails
# (whatever the reason — the historical bug was a GNU-mktemp-rejects-a-template-with-no-Xs case,
# but this stub simulates ANY reason the probe's own stderr-capture temp file cannot be created,
# e.g. a full or unwritable temp dir) and `systemctl` is never invoked at all. Before this ticket,
# this collapsed into the SAME "systemctl exited non-zero and reported an error on stderr" detail
# as the sibling test above, which asserts a cause (systemctl ran and answered badly) that was
# never measured, because systemctl never ran. This proves the SET-UP-FAILED detail is distinct and
# does not claim systemctl said anything.
test_detect_supervisor_systemd_probe_setup_failure_is_unclear() {
# Re-source first for the same reason test_detect_supervisor_systemd_probe_error_is_unclear does:
# restore the REAL probe bodies before driving them through a stub PATH.
source "$ROOT/scripts/redeploy-fleetd.sh"
local bin_dir result rc=0
bin_dir="$TMP/stub-bin-mktemp-fails"
mkdir -p "$bin_dir"
# A systemctl stub that would fail loudly if it were ever actually invoked — proves the mktemp
# failure short-circuits the probe before systemctl runs, not merely that this test forgot to
# supply a working systemctl.
cat > "$bin_dir/systemctl" <<'STUB'
#!/usr/bin/env bash
echo "systemctl must never run when mktemp already failed" >&2
exit 1
STUB
chmod +x "$bin_dir/systemctl"
cat > "$bin_dir/mktemp" <<'STUB'
#!/usr/bin/env bash
echo "mktemp: cannot create temp file" >&2
exit 1
STUB
chmod +x "$bin_dir/mktemp"
PATH="$bin_dir:$PATH" systemd_loaded && rc=0 || rc=$?
[ "$rc" -ne 0 ] \
|| fail "systemd_loaded must not report loaded=true when its own mktemp setup failed"
assert_equals "2" "$SYSTEMD_LOADED_ERRORED" \
"systemd_loaded must flag a SETUP failure (2), distinct from a probe-answered-with-stderr failure (1)"
launchd_installed() { return 1; }
launchd_loaded() { return 1; }
result="$(PATH="$bin_dir:$PATH" detect_supervisor)"
assert_equals "unclear" "$(supervisor_kind_of "$result")" "a systemd probe setup failure must read as unclear, not none"
local detail
detail="$(supervisor_detail_of "$result")"
printf '%s' "$detail" | grep -qF "could not even be set up" \
|| fail "detail does not say the probe could not be SET UP: $detail"
printf '%s' "$detail" | grep -qF "systemctl exited non-zero and reported an error on stderr" \
&& fail "detail wrongly asserts systemctl exited non-zero and reported an error on stderr, but systemctl was never run: $detail"
return 0
}
# fleetd #545 — source-text check: every `mktemp -t` template in redeploy-fleetd.sh must contain an
# `X` placeholder. BSD mktemp (macOS) tolerates a bare template with no `X`s and just appends its
# own random suffix, which is exactly why six such sites survived undetected here — GNU mktemp
# (every Linux distribution) refuses a template with fewer than three `X`s and exits non-zero. There
# is no BSD-vs-GNU seam to stub on this Mac, so this is a source-text check rather than a
# behavioural one, the same shape as test_run_drain_gate_call_site_present above. Anchored on
# `mktemp -t ` (with the trailing space) so it inspects only the `-t`-style templates this ticket is
# about, never the `mktemp -d` calls this file and test-probe-member-credentials.sh already use
# (both already carry their own `XXXXXX` and are a different mktemp mode entirely).
test_mktemp_dash_t_templates_have_x_placeholders() {
local src="$ROOT/scripts/redeploy-fleetd.sh" bad
bad="$(grep -n 'mktemp -t ' "$src" | grep -v 'XXX' || true)"
[ -z "$bad" ] \
|| fail "mktemp -t template(s) with no X placeholder (fails under GNU coreutils): $bad"
}
# fleetd #550 — the shape, not the named lines: #545 showed the exact same failure mode (a
# macOS-only idiom used with no portable fallback) spread from two sites to six across 91 commits
# before anyone tested the SHAPE rather than specific lines. This is the shasum sibling: any script
# under scripts/ that actually INVOKES the macOS-only hasher to compute a hash (as opposed to
# merely probing whether it exists with `command -v`, or mentioning it in prose) must also check
# for the portable one first in that same file — the prefer-portable-fall-back-to-macOS-only idiom
# probe-member-credentials.sh:273-279 and this ticket's own hash256 helper both follow.
#
# The needle is built from two concatenated pieces, deliberately never written as one literal
# string in this file: written whole, it would match THIS CHECK'S OWN source line once the loop
# below reaches this very file, and the check would then "pass" by matching itself rather than any
# real invocation elsewhere — a zero-findings result indistinguishable from a clean file.
test_no_unguarded_macos_only_hasher_calls() {
local needle f bad="" usage
needle='shasum'; needle="$needle -a"
for f in "$ROOT"/scripts/*.sh; do
[ -f "$f" ] || continue
usage="$(grep -Fn "$needle" "$f" || true)"
if [ -n "$usage" ]; then
grep -q 'command -v sha256sum' "$f" \
|| bad="$bad$(basename "$f") "
fi
done
[ -z "$bad" ] \
|| fail "script(s) invoke the macOS-only hasher with no portable-hasher-first fallback guard in the same file: $bad"
}
# fleetd #552 — the shape, not the one line just fixed: a `mktemp` failure AFTER the daemon has
# already been restarted must never be a bare, unguarded assignment again, anywhere in the script,
# so the next one added is caught too — not just line 1113 as it stood at 26f380a. Anchored on the
# real "start" section (the restart call itself), the same anchor test_swap_ordered_after_wait_and_
# before_start above already uses for "before the start section". Needle built from two concatenated
# pieces, the same trick test_no_unguarded_macos_only_hasher_calls uses above, so this check's own
# description of the pattern it looks for can never become a match for itself.
test_no_unguarded_mktemp_assignments_after_restart() {
local src="$ROOT/scripts/redeploy-fleetd.sh" start_line needle bad
start_line="$(grep -Fn 'say "start"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$start_line" ] || fail "could not find the restart call's start section in redeploy-fleetd.sh"
needle='^[[:space:]]*[A-Za-z_][A-Za-z0-9_]*="'
needle="$needle"'\$\(mktemp'
bad="$(grep -nE "$needle" "$src" | awk -F: -v start="$start_line" '$1+0>start')"
[ -z "$bad" ] \
|| fail "unguarded 'VAR=\"\$(mktemp ...)\"' assignment(s) after the restart call (fleetd #552): $bad"
}
# fleetd #552 — capture_fresh_log_region itself. Sourcing stops before the main flow can be driven
# directly (the SOURCED guard below), so this exercises the extracted function the real call site
# now uses, the same stub-mktemp-on-PATH technique
# test_detect_supervisor_systemd_probe_setup_failure_is_unclear uses above.
test_capture_fresh_log_region_control() {
printf 'line one\nline two\nline three\n' > "$TMP/fresh-log-src.out"
local result
result="$(capture_fresh_log_region "$TMP/fresh-log-src.out" 0)"
[ -n "$result" ] || fail "capture_fresh_log_region returned nothing on a working mktemp"
[ -f "$result" ] || fail "capture_fresh_log_region did not create the file it named"
assert_equals "$(printf 'line one\nline two\nline three')" "$(cat "$result")" \
"captured region did not carry the whole source file (restart_mark=0)"
rm -f "$result"
}
test_capture_fresh_log_region_mktemp_failure_returns_nonzero_and_prints_nothing() {
local bin_dir rc=0 out
bin_dir="$TMP/stub-bin-fresh-log-mktemp-fails"; mkdir -p "$bin_dir"
cat > "$bin_dir/mktemp" <<'STUB'
#!/usr/bin/env bash
echo "mktemp: cannot create temp file" >&2
exit 1
STUB
chmod +x "$bin_dir/mktemp"
printf 'line one\n' > "$TMP/fresh-log-src2.out"
out="$(PATH="$bin_dir:$PATH" capture_fresh_log_region "$TMP/fresh-log-src2.out" 0)" && rc=0 || rc=$?
[ "$rc" -ne 0 ] \
|| fail "capture_fresh_log_region must return non-zero when its own mktemp fails"
[ -z "$out" ] \
|| fail "capture_fresh_log_region printed a path even though mktemp failed: $out"
}
# fleetd #552 acceptance: "the exit status after a stubbed mktemp failure must still report the
# restart's own outcome". Sourcing stops before the main flow runs, so this drives the REAL
# call-site shape (the `if ! FRESH_LOG="$(capture_fresh_log_region ...)"; then warn; fi` guard that
# now stands where the old unguarded assignment did) in a subshell, under the SAME `set -euo
# pipefail` the real script runs under, with `mktemp` stubbed to fail on PATH. A regression back to
# the old unguarded `FRESH_LOG="$(mktemp ...)"` shape would abort this subshell before it reaches
# SUBSHELL_REACHED_END, and this test would go red with a non-zero rc.
test_fresh_log_capture_failure_does_not_abort_under_sete() {
local bin_dir rc=0 out
bin_dir="$TMP/stub-bin-call-site-mktemp-fails"; mkdir -p "$bin_dir"
cat > "$bin_dir/mktemp" <<'STUB'
#!/usr/bin/env bash
echo "mktemp: cannot create temp file" >&2
exit 1
STUB
chmod +x "$bin_dir/mktemp"
printf 'line one\n' > "$TMP/fresh-log-callsite.out"
out="$(
PATH="$bin_dir:$PATH"
set -euo pipefail
FRESH_LOG=""
trap 'rm -f "$FRESH_LOG"' EXIT
if ! FRESH_LOG="$(capture_fresh_log_region "$TMP/fresh-log-callsite.out" 0)"; then
echo "WARN skipped"
fi
echo "SUBSHELL_REACHED_END"
)" && rc=0 || rc=$?
[ "$rc" -eq 0 ] \
|| fail "the guarded call-site shape aborted under set -e when mktemp failed (rc=$rc) — a successful restart must not be reported as a failure"
printf '%s' "$out" | grep -qF 'SUBSHELL_REACHED_END' \
|| fail "the subshell aborted before reaching the end — a post-restart mktemp failure must not abort the script"
printf '%s' "$out" | grep -qF 'WARN skipped' \
|| fail "the guarded call site did not report the post-restart check as skipped when mktemp failed"
}
# fleetd #552 item 4 (the fourth reader): the result section must check REDEPLOY_AMQP_CHECK_SKIPPED
# and say so BEFORE it ever consults REDEPLOY_ERROR_COUNT — otherwise a skipped capture (which
# leaves REDEPLOY_ERROR_COUNT at its untouched 0) falls through the same "no ERROR lines" gate a
# genuinely clean restart uses, and (because REDEPLOY_DRAIN_STATE is "skipped", not "complete"/"n/a")
# prints nothing at all: a silence indistinguishable from the clean bill of health this ticket exists
# to prevent. Sourcing stops before the main flow runs, so this is a source-text ordering check, the
# same shape as test_report_shutdown_drain_ordered_after_classify_and_before_result above.
# fleetd #552 — closes the gap test_no_unguarded_mktemp_assignments_after_restart cannot: after
# extracting the mktemp into capture_fresh_log_region, the SAME class of hazard (an unguarded
# command-substitution assignment that can abort the script under set -e, because capture_fresh_
# log_region still returns non-zero on its own mktemp failure) would reappear if the main flow's
# call to it ever lost its own "if !" guard — and that line no longer contains the literal text
# "mktemp", so the sweep above cannot see it. Pin the real call site directly instead, the same
# shape test_swap_ordered_after_wait_and_before_start above uses for its own call site.
test_capture_fresh_log_region_call_site_is_guarded() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'if ! FRESH_LOG="$(capture_fresh_log_region "$OUT" "$RESTART_MARK")"; then' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's guarded capture_fresh_log_region call site in redeploy-fleetd.sh"
}
test_result_section_checks_amqp_skip_before_no_error_lines() {
local src="$ROOT/scripts/redeploy-fleetd.sh" result_line skip_line ok_line
result_line="$(grep -Fn 'say "result"' "$src" | head -1 | cut -d: -f1 || true)"
skip_line="$(grep -Fn '"$REDEPLOY_AMQP_CHECK_SKIPPED" -eq 1' "$src" | tail -1 | cut -d: -f1 || true)"
ok_line="$(grep -Fn 'ok "no ERROR lines since restart"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$result_line" ] || fail "could not find the result section in redeploy-fleetd.sh"
[ -n "$skip_line" ] || fail "could not find a REDEPLOY_AMQP_CHECK_SKIPPED check in redeploy-fleetd.sh"
[ -n "$ok_line" ] || fail "could not find the 'no ERROR lines since restart' line in redeploy-fleetd.sh"
[ "$skip_line" -gt "$result_line" ] \
|| fail "the REDEPLOY_AMQP_CHECK_SKIPPED check (line $skip_line) is not inside the result section (starts line $result_line)"
[ "$skip_line" -lt "$ok_line" ] \
|| fail "the REDEPLOY_AMQP_CHECK_SKIPPED check (line $skip_line) is not checked before 'no ERROR lines since restart' (line $ok_line)"
}
# fleetd #492 follow-up (Item 1): this must go through the REAL call-site shape at :437-440, not a
@@ -218,8 +445,11 @@ test_jar_id_defaults_to_live_and_reports_explicit_path() {
JAR="$dir/fleetd.jar"; JAR_STAGED="$dir/fleetd-new.jar"
printf 'live jar bytes' > "$JAR"
printf 'staged jar bytes, not the same content' > "$JAR_STAGED"
live_hash="$(shasum -a 256 "$JAR" | cut -c1-12)"
staged_hash="$(shasum -a 256 "$JAR_STAGED" | cut -c1-12)"
# fleetd #550: this reference hash must be computed the same portable way jar_id() itself now
# computes one — a bare, unguarded call to the macOS-only hasher here was exactly the item-2
# defect, dying with "command not found" on any Linux runner that has no such hasher at all.
live_hash="$(hash256 "$JAR")"
staged_hash="$(hash256 "$JAR_STAGED")"
default_result="$(jar_id)"
explicit_result="$(jar_id "$JAR_STAGED")"
JAR="$saved_jar"; JAR_STAGED="$saved_staged"
@@ -228,6 +458,26 @@ test_jar_id_defaults_to_live_and_reports_explicit_path() {
assert_equals "$staged_hash" "$explicit_result" "jar_id \"\$JAR_STAGED\" must report the hash of the staged jar, not fall back to \$JAR"
}
# fleetd #550 — closes a gap the test above leaves open. That test's own reference hash is now ALSO
# computed by calling hash256 (needed for item 2: the old bare macOS-only-hasher call there was the
# Linux crash), so its subject (jar_id, via hash256) and its reference (also hash256) share one
# instrument — they agree no matter which algorithm hash256 actually runs, so a mutation that swaps
# BOTH of hash256's arms for the wrong algorithm is invisible to it. This test's expected value
# comes from neither hasher: it is the published SHA-256 test vector for the 3-byte input "abc"
# (no trailing newline), written here as a literal constant, so it can still tell "hashed
# correctly" from "hashed, just with the wrong algorithm" — which is what this whole ticket is
# about.
test_hash256_computes_a_real_sha256() {
local dir f result
dir="$TMP/hash256-known-vector"; mkdir -p "$dir"
f="$dir/abc.txt"
printf 'abc' > "$f"
result="$(hash256 "$f")"
# SHA-256("abc") = ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad, the standard
# FIPS 180 test vector — first 12 hex chars, matching hash256's own `cut -c1-12`.
assert_equals "ba7816bf8f01" "$result" "hash256 of the literal 3-byte input 'abc' must be the known SHA-256 prefix, not some other algorithm's"
}
# fleetd #517 — jar_id()'s "absent" branch was unpinned by any test: the existing test above (#511)
# proves both halves of the present-file contract but never exercises the missing-file path. This
# word matters more than a string usually would: "absent" is the #413 signal that a `mvn clean`
@@ -246,6 +496,37 @@ test_jar_id_reports_absent_for_missing_file() {
assert_equals "absent" "$explicit_result" "jar_id with an explicit missing path must report absent"
}
# fleetd #550 — the whole point of this ticket: a jar that IS there but could not be hashed must
# never read the same as a jar that is not there at all. Drives the REAL hash256/jar_id bodies
# (never stubbed) through a stub PATH that contains neither of the two hashers this script knows —
# same technique test_detect_supervisor_systemd_probe_setup_failure_is_unclear uses for `mktemp`,
# except here the stub directory is used to REPLACE PATH rather than prepend to it, because the
# point is to make BOTH hashers unreachable, not to intercept one specific command while leaving
# everything else on the real PATH reachable. `[ -f ... ]` and the shell's own `command`/`echo`
# builtins need no PATH at all, so this is safe even with PATH reduced to an empty directory.
test_jar_id_reports_unhashable_when_no_hasher_on_path() {
local bin_dir dir saved_jar="$JAR" default_result explicit_result
bin_dir="$TMP/stub-bin-no-hasher"; mkdir -p "$bin_dir"
dir="$TMP/jar-id-no-hasher"; mkdir -p "$dir"
JAR="$dir/fleetd.jar"
printf 'a real jar that exists but nothing here can hash' > "$JAR"
[ -f "$JAR" ] || fail "test fixture error: \$JAR does not exist at $JAR"
default_result="$(PATH="$bin_dir" jar_id)"
explicit_result="$(PATH="$bin_dir" jar_id "$JAR")"
JAR="$saved_jar"
[ "$default_result" != "absent" ] \
|| fail "jar_id reported absent for a file that exists, only because no hasher was on PATH"
[ "$explicit_result" != "absent" ] \
|| fail "jar_id (explicit path) reported absent for a file that exists, only because no hasher was on PATH"
# A 12-char hex hash is the OTHER wrong answer here: with no hasher at all, nothing could have
# produced one, so a value that merely happens to look like one would mean the stub failed to
# hide the real hashers rather than that jar_id degraded correctly.
printf '%s' "$default_result" | grep -Eq '^[0-9a-f]{12}$' \
&& fail "test fixture error: PATH stub did not actually hide the real hasher(s) — got what looks like a real hash"
assert_equals "unhashable" "$default_result" "jar_id with no hasher on PATH must report a third, distinct state — never absent, never a hash"
assert_equals "unhashable" "$explicit_result" "jar_id (explicit path) with no hasher on PATH must report the same third state"
}
# fleetd #493 — never build into the path a running process holds. stage_built_jar/swap_staged_jar
# are exercised directly against real files on disk (not stubs), because the whole point is file
# behavior (does the content move, does the source disappear, does a failure leave both sides
@@ -611,14 +892,489 @@ test_refuse_drain_gate_no_build_staged_absent() {
# flat `die "aborted — nothing changed"` removes this exact needle, where none of the behavioural
# tests above would even notice.
#
# fleetd #555 — the main flow's own call site moved: it used to read
# `refuse_drain_gate "$DO_BUILD" "$JAR_STAGED"` directly; it now reads
# `run_drain_gate "$OLD_PID" "$ASSUME_YES" "$DO_BUILD" "$JAR_STAGED"`, and run_drain_gate (tested
# directly below by test_run_drain_gate_*) is what calls refuse_drain_gate with its own local names.
# This grep now pins THAT call site — the thing that would go missing if a future edit deleted the
# main flow's call to run_drain_gate altogether, the same residual gap #521/#528 already accepted for
# swap_if_built/refuse_drain_gate (sourcing stops before the main flow runs, so no test in this file
# can do better than reading the source for this one specific gap).
#
# The grep ends `|| true`: this file runs under `set -euo pipefail`, so an ABSENT needle would fail
# the assignment and `set -e` would kill the whole suite before the `[ -n ... ] || fail` guard below
# ever ran — the exact dead-check shape fleetd #528 also flags as a sweep finding (see the PR body).
test_refuse_drain_gate_call_site_present() {
test_run_drain_gate_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'refuse_drain_gate "$DO_BUILD" "$JAR_STAGED"' "$src" | head -1 | cut -d: -f1 || true)"
call_line="$(grep -Fn 'run_drain_gate "$OLD_PID" "$ASSUME_YES" "$DO_BUILD" "$JAR_STAGED"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's refuse_drain_gate call site in redeploy-fleetd.sh"
|| fail "could not find the main flow's run_drain_gate call site in redeploy-fleetd.sh"
}
# fleetd #555 item 1 — `--check` must stay genuinely read-only: should_stop_for_check is the
# predicate, stop_if_check_only is the only thing the main flow calls.
test_should_stop_for_check_true_when_check_only() {
should_stop_for_check 1 || fail "should_stop_for_check must be true when CHECK_ONLY=1"
}
test_should_stop_for_check_false_when_not_check_only() {
should_stop_for_check 0 && fail "should_stop_for_check must be false when CHECK_ONLY=0"
return 0
}
# Captured via $(...): stop_if_check_only's own exit only ends this subshell (same reason the
# unload/stop "tolerates clean negative" tests above capture this way), so this test's own message
# is what reaches the report if the exit ever stops happening.
test_stop_if_check_only_exits_zero_and_says_nothing_changed() {
local output rc=0
output="$(stop_if_check_only 1)" || rc=$?
[ "$rc" -eq 0 ] || fail "stop_if_check_only 1 must exit 0, got $rc"
printf '%s' "$output" | grep -qF -- '--check: nothing changed' \
|| fail "stop_if_check_only 1 did not print the --check message"
}
test_stop_if_check_only_returns_without_exiting_when_not_check_only() {
local output
output="$(stop_if_check_only 0; echo REACHED)"
printf '%s' "$output" | grep -qF 'REACHED' \
|| fail "stop_if_check_only 0 must return, not exit — the caller's next line never ran"
}
test_stop_if_check_only_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'stop_if_check_only "$CHECK_ONLY"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's stop_if_check_only call site in redeploy-fleetd.sh"
}
# fleetd #555 items 2 and 3 — the drain-gate entry (`-n "$OLD_PID" && "$ASSUME_YES" = 0`) and the
# reply comparison (`"$reply" != "yes"`).
test_drain_gate_required_true_when_pid_and_not_assume_yes() {
drain_gate_required "123" 0 || fail "drain_gate_required must be true with an old pid and ASSUME_YES=0"
}
test_drain_gate_required_false_when_no_pid() {
drain_gate_required "" 0 && fail "drain_gate_required must be false with no old pid"
return 0
}
test_drain_gate_required_false_when_assume_yes() {
drain_gate_required "123" 1 && fail "drain_gate_required must be false when ASSUME_YES=1"
return 0
}
test_drain_confirmed_true_on_yes() {
drain_confirmed "yes" || fail "drain_confirmed must be true on exactly 'yes'"
}
test_drain_confirmed_false_on_anything_else() {
drain_confirmed "no" && fail "drain_confirmed must be false on 'no'"
drain_confirmed "" && fail "drain_confirmed must be false on empty input"
return 0
}
# run_drain_gate end-to-end, driving real stdin through `read` the same way the real prompt does.
# Redirected from a FILE, never a pipe: `printf ... | run_drain_gate ...` would run the function as
# the last stage of a pipeline, which bash runs in a subshell by default (no `lastpipe`), so any
# global this function set (DIED_CALLED/DIED_MESSAGE below) would be lost the instant the pipe
# closed. `< file` redirection on a plain function call carries no such subshell.
test_run_drain_gate_skips_prompt_when_not_required() {
local output
output="$(run_drain_gate "" 0 1 "$TMP/nonexistent-staged.jar" < /dev/null 2>&1)"
if printf '%s' "$output" | grep -qF 'Fleet drained?'; then
fail "run_drain_gate prompted even though drain_gate_required should have been false (no old pid)"
fi
output="$(run_drain_gate "123" 1 1 "$TMP/nonexistent-staged.jar" < /dev/null 2>&1)"
if printf '%s' "$output" | grep -qF 'Fleet drained?'; then
fail "run_drain_gate prompted even though ASSUME_YES=1 should have skipped the gate"
fi
}
test_run_drain_gate_confirmed_reply_does_not_refuse() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_die_recorder
printf 'yes\n' > "$TMP/drain-reply-yes.txt"
run_drain_gate "123" 0 1 "$TMP/nonexistent-staged.jar" < "$TMP/drain-reply-yes.txt" \
> "$TMP/drain-gate-yes-output" 2>&1
[ "$DIED_CALLED" = 0 ] \
|| fail "run_drain_gate must not refuse when the reply confirms ('yes')"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_run_drain_gate_declined_reply_refuses() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_die_recorder
printf 'no\n' > "$TMP/drain-reply-no.txt"
run_drain_gate "123" 0 1 "$TMP/nonexistent-staged.jar" < "$TMP/drain-reply-no.txt" \
> "$TMP/drain-gate-no-output" 2>&1
[ "$DIED_CALLED" = 1 ] \
|| fail "run_drain_gate must refuse (call die via refuse_drain_gate) when the reply does not confirm"
printf '%s' "$DIED_MESSAGE" | grep -qF 'aborted' \
|| fail "run_drain_gate's refusal message does not read as an abort: $DIED_MESSAGE"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
# fleetd #555 item 4 — the report-state dispatch on $SUPERVISOR_KIND. Inverting this used to report
# the wrong supervisor and, for the launchd arm specifically, skip check_log_path_matches_plist.
CHECK_LOG_PATH_CALLED=0
stub_check_log_path_recorder() {
CHECK_LOG_PATH_CALLED=0
check_log_path_matches_plist() { CHECK_LOG_PATH_CALLED=1; }
}
test_report_supervisor_state_launchd_sets_supervised_and_checks_log_path() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_check_log_path_recorder
SUPERVISED=0
report_supervisor_state "launchd" > "$TMP/report-supervisor-launchd-output" 2>&1
[ "$SUPERVISED" = 1 ] || fail "report_supervisor_state launchd must set SUPERVISED=1"
[ "$CHECK_LOG_PATH_CALLED" = 1 ] \
|| fail "report_supervisor_state launchd must call check_log_path_matches_plist"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_report_supervisor_state_systemd_sets_supervised_without_log_path_check() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_check_log_path_recorder
SUPERVISED=0
report_supervisor_state "systemd" > "$TMP/report-supervisor-systemd-output" 2>&1
[ "$SUPERVISED" = 1 ] || fail "report_supervisor_state systemd must set SUPERVISED=1"
[ "$CHECK_LOG_PATH_CALLED" = 0 ] \
|| fail "report_supervisor_state systemd must NOT call check_log_path_matches_plist"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_report_supervisor_state_none_leaves_supervised_zero() {
SUPERVISED=1
report_supervisor_state "none" > "$TMP/report-supervisor-none-output" 2>&1
[ "$SUPERVISED" = 0 ] || fail "report_supervisor_state none must leave SUPERVISED=0"
grep -qF 'no supervisor loaded' "$TMP/report-supervisor-none-output" \
|| fail "report_supervisor_state none did not print the unsupervised message"
}
test_report_supervisor_state_unrecognized_kind_warns_and_continues() {
SUPERVISED=1
report_supervisor_state "bogus-kind" > "$TMP/report-supervisor-bogus-output" 2>&1
[ "$SUPERVISED" = 0 ] || fail "report_supervisor_state must reset SUPERVISED for an unrecognized kind"
grep -qF "unrecognised supervisor kind: 'bogus-kind'" "$TMP/report-supervisor-bogus-output" \
|| fail "report_supervisor_state did not warn about the unrecognized kind"
}
test_report_supervisor_state_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'report_supervisor_state "$SUPERVISOR_KIND"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's report_supervisor_state call site in redeploy-fleetd.sh"
}
# fleetd #555 item 5 — the stop dispatch on $SUPERVISOR_KIND. Inverting this kills a supervised
# daemon with a raw kill instead of launchctl/systemctl, reviving the OLD jar (CB-594/#492). Each
# mechanism is stubbed to record which one ran, so a swapped or dropped arm shows up directly.
STOP_DISPATCH_CALLED=""
stub_stop_dispatch_recorders() {
STOP_DISPATCH_CALLED=""
stop_via_launchd() { STOP_DISPATCH_CALLED="launchd"; }
stop_via_systemd() { STOP_DISPATCH_CALLED="systemd"; }
stop_via_kill() { STOP_DISPATCH_CALLED="kill:$1"; }
}
test_dispatch_stop_launchd_calls_stop_via_launchd() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_stop_dispatch_recorders
dispatch_stop "launchd" "123"
assert_equals "launchd" "$STOP_DISPATCH_CALLED" "dispatch_stop launchd"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_stop_systemd_calls_stop_via_systemd() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_stop_dispatch_recorders
dispatch_stop "systemd" "123"
assert_equals "systemd" "$STOP_DISPATCH_CALLED" "dispatch_stop systemd"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_stop_none_calls_stop_via_kill_with_pid() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_stop_dispatch_recorders
dispatch_stop "none" "456"
assert_equals "kill:456" "$STOP_DISPATCH_CALLED" "dispatch_stop none"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_stop_unrecognized_kind_dies() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_stop_dispatch_recorders
stub_die_recorder
dispatch_stop "bogus-kind" "789"
[ "$DIED_CALLED" = 1 ] || fail "dispatch_stop must die on an unrecognized kind"
[ -z "$STOP_DISPATCH_CALLED" ] \
|| fail "dispatch_stop must not call any stop mechanism on an unrecognized kind"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_stop_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'dispatch_stop "$SUPERVISOR_KIND" "$OLD_PID"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's dispatch_stop call site in redeploy-fleetd.sh"
}
# fleetd #555 item 6 — the start dispatch on $SUPERVISOR_KIND. Same shape as the stop dispatch.
START_DISPATCH_CALLED=""
stub_start_dispatch_recorders() {
START_DISPATCH_CALLED=""
start_via_launchd() { START_DISPATCH_CALLED="launchd"; }
start_via_systemd() { START_DISPATCH_CALLED="systemd"; }
start_via_none() { START_DISPATCH_CALLED="none"; }
}
test_dispatch_start_launchd_calls_start_via_launchd() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_start_dispatch_recorders
dispatch_start "launchd"
assert_equals "launchd" "$START_DISPATCH_CALLED" "dispatch_start launchd"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_start_systemd_calls_start_via_systemd() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_start_dispatch_recorders
dispatch_start "systemd"
assert_equals "systemd" "$START_DISPATCH_CALLED" "dispatch_start systemd"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_start_none_calls_start_via_none() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_start_dispatch_recorders
dispatch_start "none"
assert_equals "none" "$START_DISPATCH_CALLED" "dispatch_start none"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_start_unrecognized_kind_dies() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_start_dispatch_recorders
stub_die_recorder
dispatch_start "bogus-kind"
[ "$DIED_CALLED" = 1 ] || fail "dispatch_start must die on an unrecognized kind"
[ -z "$START_DISPATCH_CALLED" ] \
|| fail "dispatch_start must not call any start mechanism on an unrecognized kind"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_dispatch_start_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'dispatch_start "$SUPERVISOR_KIND"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's dispatch_start call site in redeploy-fleetd.sh"
}
# fleetd #555 item 7 — the HEALTH_BODY poll loop and the `[ -z "$HEALTH_BODY" ]` branch. Inverting
# health_is_up makes a dead daemon report /healthz 200 or a live one report failure.
test_health_is_up_true_with_body() {
health_is_up "some body" || fail "health_is_up must be true with a non-empty body"
}
test_health_is_up_false_with_empty_body() {
health_is_up "" && fail "health_is_up must be false with an empty body"
return 0
}
# fleetd #555: stub_die_recorder replaces die() globally for the rest of the process, the same as
# every other user of it in this file — re-source before AND after so a later test (e.g.
# test_unload_launchd_if_loaded_dies_on_real_failure, which needs the REAL die() to actually exit)
# never inherits a stub left over from here.
test_report_health_up_reports_ok_and_never_dies() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_die_recorder
local output
output="$(report_health '{"status":"ok"}' "200" "$TMP/fake.out" 60)"
[ "$DIED_CALLED" = 0 ] || fail "report_health must not die when the body is non-empty"
printf '%s' "$output" | grep -qF '/healthz 200' \
|| fail "report_health did not report the healthy body"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_report_health_degraded_503_warns_and_never_dies() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_die_recorder
local output
output="$(report_health "" "503" "$TMP/fake.out" 60 2>&1)"
[ "$DIED_CALLED" = 0 ] || fail "report_health must not die on a 503 (degraded, not dead)"
printf '%s' "$output" | grep -qF '503 degraded' \
|| fail "report_health did not report the 503-degraded case"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_report_health_dead_dies() {
source "$ROOT/scripts/redeploy-fleetd.sh"
stub_die_recorder
report_health "" "000" "$TMP/fake.out" 60 > "$TMP/report-health-dead-output" 2>&1
[ "$DIED_CALLED" = 1 ] || fail "report_health must die when the body is empty and the code is not 503"
printf '%s' "$DIED_MESSAGE" | grep -qF 'never answered' \
|| fail "report_health die message does not say healthz never answered"
source "$ROOT/scripts/redeploy-fleetd.sh"
}
test_poll_health_body_returns_nonzero_when_unreachable() {
local rc=0
poll_health_body "http://127.0.0.1:1/healthz" 1 > /dev/null || rc=$?
[ "$rc" -ne 0 ] || fail "poll_health_body must return non-zero when the health endpoint is unreachable"
}
test_report_health_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'report_health "$HEALTH_BODY" "$HEALTH_CODE" "$OUT" "$HEALTH_WAIT"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's report_health call site in redeploy-fleetd.sh"
}
# fleetd #555 item 8 — `HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1`. Measured safe under
# `set -e` at both bash 3.2.57 and 5.x (the ticket's own header discussion); still an untested
# computation feeding report_shutdown_drain's four-way decision until now.
test_compute_had_old_pid_zero_when_empty() {
assert_equals "0" "$(compute_had_old_pid "")" "compute_had_old_pid empty pid"
}
test_compute_had_old_pid_one_when_present() {
assert_equals "1" "$(compute_had_old_pid "12345")" "compute_had_old_pid present pid"
}
test_compute_had_old_pid_call_site_present() {
local src="$ROOT/scripts/redeploy-fleetd.sh" call_line
call_line="$(grep -Fn 'HAD_OLD_PID="$(compute_had_old_pid "$OLD_PID")"' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$call_line" ] \
|| fail "could not find the main flow's compute_had_old_pid call site in redeploy-fleetd.sh"
}
# fleetd #555 — the structural guard the ticket actually asks for: "pick the shape, not the eight
# sites." A future ninth bare main-flow conditional must fail THIS test on sight, not slip through
# with every function-level test (67, now many more) still green.
#
# Scope: from the SOURCED guard onward — the ticket's own "main flow" (report state / build / drain
# / stop / swap / start / verify). The `for arg in "$@"; do case "$arg" in ... esac; done` option
# parser above the guard is out of scope, the same way it sits outside the ticket's own eight-item
# table: it runs even when this file is sourced for tests, is not part of the mutating procedure,
# and none of the eight decisions the ticket names live there.
#
# For every line whose first non-blank token is `if`, `elif`, or `case`, and which sits OUTSIDE any
# function body, this demands an EXACT match — full trimmed line, never a regex anchor (a regex can
# match inside unrelated text this same mutation could introduce, e.g. a comment) — against
# MAIN_FLOW_ALLOWED_CONDITIONALS below. That allowlist is a closed set of the report-only/display
# conditionals already in the file (bare state prints: is the daemon running, is X installed, does
# this secret resolve — none of them gate a build/stop/start/swap decision) plus the two
# loaded-but-not-running elif branches that are the SAME shape as fleetd #504 items 2/3/4, one
# function over, and explicitly out of THIS ticket's scope per its own "Related" section.
#
# Every one of the eight decisions the ticket names has been lifted into its own predicate+action
# function above this point in the file (should_stop_for_check, drain_gate_required/run_drain_gate,
# report_supervisor_state, dispatch_stop, dispatch_start, health_is_up/report_health,
# compute_had_old_pid) — none of their guards appear in this list any more, which is what makes
# their specific inversions untestable-by-construction now: there is no bare guard left in the main
# flow for a future edit to invert silently. A NEW bare conditional (the ninth) is, by construction,
# not in this list, so this test goes red the instant one is added, naming the exact line — passing
# requires either lifting the decision into a function (with its own predicate test, same shape as
# above) or explicitly extending the allowlist, which is a diff a reviewer sees and can question.
#
# Function-body detection relies on this file's one consistent convention: a function opens as
# `name() {` alone on its own line and closes as a bare `}` alone on its own line — verified: every
# multi-line function in this file follows it, and the handful of one-liners like `say() { ...; }`
# sit above the SOURCED guard, outside the region this scans. Indentation is NOT used to decide
# "inside a function": a bare `if` inside a top-level `for`/`while` loop body (indented, but not
# inside any function) would be exactly as reportable as one at column 0 — this file currently has
# none, but a future one that hid inside a loop must not get a free pass for being indented.
MAIN_FLOW_ALLOWED_CONDITIONALS=(
'if [ -n "$OLD_PID" ]; then'
'if launchd_installed; then'
'if systemd_installed; then'
'if zsh -lc '\''[ -n "${WORKER_GITEA_TOKEN:-}" ]'\'' 2>/dev/null; then'
'if zsh -lc '\''[ -n "${AI_GATEWAY_TOKEN:-}" ]'\'' 2>/dev/null; then'
'if [ -z "$BROKER_URI_ENV" ]; then'
'elif zsh -lc "[ -n \"\${$BROKER_URI_ENV:-}\" ]" 2>/dev/null; then'
'if [ "$DO_BUILD" = 1 ]; then'
'if ! mvn -f "$MODULE/pom.xml" clean install > "$BUILD_LOG" 2>&1; then'
'if ! wait_for_daemon_exit "$STOP_WAIT"; then'
'elif [ "$SUPERVISOR_KIND" = "launchd" ]; then'
'elif [ "$SUPERVISOR_KIND" = "systemd" ]; then'
'if tail -n "+$((RESTART_MARK + 1))" "$OUT" 2>/dev/null | grep -q '\''fleetd listening'\''; then'
'if ! FRESH_LOG="$(capture_fresh_log_region "$OUT" "$RESTART_MARK")"; then'
'if [ "$REDEPLOY_AMQP_CHECK_SKIPPED" -eq 1 ]; then'
'elif [ "$REDEPLOY_ERROR_COUNT" -eq 0 ]; then'
'if [ "$REDEPLOY_DRAIN_STATE" = "complete" ] || [ "$REDEPLOY_DRAIN_STATE" = "n/a" ]; then'
'elif [ "$REDEPLOY_UNEXPLAINED_ERRORS" -eq 0 ]; then'
)
# Emits "<lineno>\tCOND\t<trimmed line>" for every if/elif/case found outside a function, and
# "<lineno>\tFUNC\t<line>" for every function DEFINITION found after guard_line — from guard_line
# onward. A separate function (rather than inlined into the test) so the deliberate-mutation proof
# in the PR description can call it directly against a scratch copy of the script.
#
# fleetd #555 rework (comment 17012): a function defined after the SOURCED guard can never be
# reached by `source`-ing this script (sourcing returns before the main flow, and before any code
# below the guard runs), so anything inside such a function is untestable by construction — the
# depth tracker below would otherwise read it as "inside a function, therefore fine" and wave every
# conditional in it through unexamined. Emitting FUNC records lets the caller flag the function
# itself as the violation, independently of whether its body happens to contain a conditional.
mainflow_bare_conditionals() {
local src="$1" guard_line="$2" in_func=0 lineno=0 line trimmed
while IFS= read -r line || [ -n "$line" ]; do
lineno=$((lineno + 1))
[ "$lineno" -le "$guard_line" ] && continue
if [[ "$line" =~ ^[A-Za-z_][A-Za-z0-9_]*\(\)[[:space:]]*\{[[:space:]]*$ ]]; then
in_func=1
printf '%d\tFUNC\t%s\n' "$lineno" "$line"
continue
fi
if [ "$in_func" = 1 ] && [[ "$line" =~ ^\}[[:space:]]*$ ]]; then
in_func=0
continue
fi
if [ "$in_func" = 0 ]; then
trimmed="$(printf '%s' "$line" | sed -E 's/^[[:space:]]+//')"
if [[ "$trimmed" =~ ^(if|elif|case)[[:space:]] ]]; then
printf '%d\tCOND\t%s\n' "$lineno" "$trimmed"
fi
fi
done < "$src"
}
test_no_untested_main_flow_conditionals() {
local src="$ROOT/scripts/redeploy-fleetd.sh" guard_line
guard_line="$(grep -Fn 'if (return 0 2>/dev/null); then' "$src" | head -1 | cut -d: -f1 || true)"
[ -n "$guard_line" ] || { fail "could not find the SOURCED guard in redeploy-fleetd.sh"; return 1; }
local violations=0 report="" found_line found_kind found_text allowed candidate
while IFS=$'\t' read -r found_line found_kind found_text; do
[ -n "$found_line" ] || continue
if [ "$found_kind" = "FUNC" ]; then
# fleetd #555 rework: a function defined after the SOURCED guard line can never be sourced
# by this suite, so it can never be tested — that is a violation on its own, regardless of
# what its body contains or whether the allowlist would otherwise excuse a bare conditional
# inside it.
violations=$((violations + 1))
report="$report
line $found_line: function defined after the SOURCED guard (line $guard_line) — it cannot be sourced, so it cannot be tested: $found_text"
continue
fi
allowed=0
for candidate in "${MAIN_FLOW_ALLOWED_CONDITIONALS[@]}"; do
if [ "$found_text" = "$candidate" ]; then
allowed=1
break
fi
done
if [ "$allowed" = 0 ]; then
violations=$((violations + 1))
report="$report
line $found_line: $found_text"
fi
done < <(mainflow_bare_conditionals "$src" "$guard_line")
if [ "$violations" -gt 0 ]; then
fail "found $violations untested main-flow if/elif/case line(s) or function definition(s) after the SOURCED guard, not lifted into a tested predicate function and not in MAIN_FLOW_ALLOWED_CONDITIONALS:$report"
fi
}
# fleetd #504 — the "loaded but not currently running" branches for launchd/systemd used to run
@@ -702,7 +1458,7 @@ STUB
|| fail "stop_systemd_if_loaded must tolerate a clean already-stopped answer (non-zero exit, empty stderr): $output"
}
# Closes the same gap test_refuse_drain_gate_call_site_present closes for the drain gate: the four
# Closes the same gap test_run_drain_gate_call_site_present closes for the drain gate: the four
# tests above call unload_launchd_if_loaded/stop_systemd_if_loaded directly, and sourcing stops
# before the main flow ever runs (the SOURCED guard), so none of them can prove the main flow still
# CALLS these two functions instead of the original bare `2>/dev/null || true`. A source-text check,
@@ -732,6 +1488,20 @@ LOG
classify_fixture no-errors.log
assert_equals 0 "$REDEPLOY_ERROR_COUNT" "no-errors total"
assert_equals 0 "$REDEPLOY_UNEXPLAINED_ERRORS" "no-errors unexplained"
# fleetd #552 control: a REAL, readable log region with nothing in it must never look like a
# region that could not be captured at all.
assert_equals 0 "$REDEPLOY_AMQP_CHECK_SKIPPED" "no-errors must not report the check as skipped"
}
# fleetd #552 — the third state: "" (capture_fresh_log_region's own sentinel for "the post-restart
# region could never be captured") must never look like "read a file with nothing in it", which is
# exactly what REDEPLOY_ERROR_COUNT staying at 0 already means on its own (see test_no_errors above).
test_classify_amqp_connection_errors_reports_skipped_when_log_missing() {
classify_amqp_connection_errors "" > "$TMP/classify-skipped-output" 2>&1
assert_equals 1 "$REDEPLOY_AMQP_CHECK_SKIPPED" "an empty log_file must set REDEPLOY_AMQP_CHECK_SKIPPED=1"
assert_equals 0 "$REDEPLOY_ERROR_COUNT" "a skipped check must still leave REDEPLOY_ERROR_COUNT at its initialized 0 (the flag, not the count, carries the distinction)"
grep -qF 'could not be captured' "$TMP/classify-skipped-output" \
|| fail "classify_amqp_connection_errors did not say the post-restart log region could not be captured"
}
test_recovery_patterns_match_source() {
@@ -1045,9 +1815,30 @@ LOG
|| fail "report_shutdown_drain did not report that there was nothing to check"
}
# fleetd #552 — a fifth outcome: log_file="" (capture_fresh_log_region's own sentinel) with a
# previous daemon that WAS stopped this run. Distinct from "unknown" (a log was read and neither
# signal was found in it) — here there was no log to read at all.
test_report_shutdown_drain_skipped_when_log_missing() {
report_shutdown_drain "" 1 > "$TMP/drain-skipped-output" 2>&1
assert_equals "skipped" "$REDEPLOY_DRAIN_STATE" "an empty log_file with a previous daemon must set REDEPLOY_DRAIN_STATE=skipped"
grep -qF 'could not be captured' "$TMP/drain-skipped-output" \
|| fail "report_shutdown_drain did not say the post-restart log region could not be captured"
if grep -qF ' ok' "$TMP/drain-skipped-output"; then
fail "the skipped outcome must not be printed via ok() — it is neither a pass nor a failure"
fi
}
# fleetd #552 — proves the had_previous_daemon=0 check really does run BEFORE the missing-log check:
# with BOTH conditions true at once, n/a must win, because a cold start has nothing to check
# regardless of whether the log region could be captured.
test_report_shutdown_drain_na_wins_over_missing_log() {
report_shutdown_drain "" 0 > "$TMP/drain-na-and-missing-output" 2>&1
assert_equals "n/a" "$REDEPLOY_DRAIN_STATE" "no-previous-daemon must win over a missing log region"
}
# fleetd #512 — closes the gap none of the seven tests above can: they call report_shutdown_drain
# directly, and sourcing stops before the main flow ever runs (the SOURCED guard), so none of them
# can prove the main flow still calls it at all. Same shape as test_refuse_drain_gate_call_site_present
# can prove the main flow still calls it at all. Same shape as test_run_drain_gate_call_site_present
# and test_swap_ordered_after_wait_and_before_start: a source-text grep for the real call site, plus
# an ordering check against its neighbours in the verify/result flow.
test_report_shutdown_drain_call_site_present() {
@@ -1088,6 +1879,15 @@ test_detect_supervisor_none
test_detect_supervisor_systemd_installed_not_loaded_is_unclear
test_detect_supervisor_launchd_installed_not_loaded_is_unclear
test_detect_supervisor_systemd_probe_error_is_unclear
test_detect_supervisor_systemd_probe_setup_failure_is_unclear
test_mktemp_dash_t_templates_have_x_placeholders
test_no_unguarded_macos_only_hasher_calls
test_no_unguarded_mktemp_assignments_after_restart
test_capture_fresh_log_region_control
test_capture_fresh_log_region_mktemp_failure_returns_nonzero_and_prints_nothing
test_fresh_log_capture_failure_does_not_abort_under_sete
test_capture_fresh_log_region_call_site_is_guarded
test_result_section_checks_amqp_skip_before_no_error_lines
test_require_drivable_supervisor_refuses_ambiguous
test_require_drivable_supervisor_refuses_unclear
test_require_drivable_supervisor_accepts_known_kinds
@@ -1095,7 +1895,9 @@ test_count_daemon_pids
test_assert_single_daemon_accepts_one_pid
test_assert_single_daemon_rejects_two_pids
test_jar_id_defaults_to_live_and_reports_explicit_path
test_hash256_computes_a_real_sha256
test_jar_id_reports_absent_for_missing_file
test_jar_id_reports_unhashable_when_no_hasher_on_path
test_stage_built_jar_moves_off_live_path
test_stage_built_jar_dies_when_build_produced_nothing
test_swap_staged_jar_moves_staged_onto_live
@@ -1119,13 +1921,53 @@ test_refuse_drain_gate_build_ran_staged_present
test_refuse_drain_gate_build_ran_staged_absent
test_refuse_drain_gate_no_build_staged_present
test_refuse_drain_gate_no_build_staged_absent
test_refuse_drain_gate_call_site_present
test_run_drain_gate_call_site_present
test_should_stop_for_check_true_when_check_only
test_should_stop_for_check_false_when_not_check_only
test_stop_if_check_only_exits_zero_and_says_nothing_changed
test_stop_if_check_only_returns_without_exiting_when_not_check_only
test_stop_if_check_only_call_site_present
test_drain_gate_required_true_when_pid_and_not_assume_yes
test_drain_gate_required_false_when_no_pid
test_drain_gate_required_false_when_assume_yes
test_drain_confirmed_true_on_yes
test_drain_confirmed_false_on_anything_else
test_run_drain_gate_skips_prompt_when_not_required
test_run_drain_gate_confirmed_reply_does_not_refuse
test_run_drain_gate_declined_reply_refuses
test_report_supervisor_state_launchd_sets_supervised_and_checks_log_path
test_report_supervisor_state_systemd_sets_supervised_without_log_path_check
test_report_supervisor_state_none_leaves_supervised_zero
test_report_supervisor_state_unrecognized_kind_warns_and_continues
test_report_supervisor_state_call_site_present
test_dispatch_stop_launchd_calls_stop_via_launchd
test_dispatch_stop_systemd_calls_stop_via_systemd
test_dispatch_stop_none_calls_stop_via_kill_with_pid
test_dispatch_stop_unrecognized_kind_dies
test_dispatch_stop_call_site_present
test_dispatch_start_launchd_calls_start_via_launchd
test_dispatch_start_systemd_calls_start_via_systemd
test_dispatch_start_none_calls_start_via_none
test_dispatch_start_unrecognized_kind_dies
test_dispatch_start_call_site_present
test_health_is_up_true_with_body
test_health_is_up_false_with_empty_body
test_report_health_up_reports_ok_and_never_dies
test_report_health_degraded_503_warns_and_never_dies
test_report_health_dead_dies
test_poll_health_body_returns_nonzero_when_unreachable
test_report_health_call_site_present
test_compute_had_old_pid_zero_when_empty
test_compute_had_old_pid_one_when_present
test_compute_had_old_pid_call_site_present
test_no_untested_main_flow_conditionals
test_unload_launchd_if_loaded_dies_on_real_failure
test_unload_launchd_if_loaded_tolerates_clean_negative
test_stop_systemd_if_loaded_dies_on_real_failure
test_stop_systemd_if_loaded_tolerates_clean_negative
test_stop_branches_call_tolerant_helpers_not_bare_or_true
test_no_errors
test_classify_amqp_connection_errors_reports_skipped_when_log_missing
test_recovery_patterns_match_source
test_attributed_recovered_connection_error
test_source_derived_error_shapes_recover_by_connection
@@ -1144,6 +1986,8 @@ test_report_shutdown_drain_died_without_error_token
test_report_shutdown_drain_complete_control
test_report_shutdown_drain_unknown_cannot_tell
test_report_shutdown_drain_no_previous_daemon_is_na
test_report_shutdown_drain_skipped_when_log_missing
test_report_shutdown_drain_na_wins_over_missing_log
test_report_shutdown_drain_call_site_present
test_report_shutdown_drain_ordered_after_classify_and_before_result
test_no_error_lines_message_gated_by_drain_state