The deferred roll waits for the calling lead's own turn to end. The old 20s bound
was shorter than one ordinary closing message (measured elapsed=20394ms), so a
lead that followed the handover skill's instruction to say its goodbye in the
same turn landed in the refusal branch. 300 matches idleAfterSeconds' existing
written reason: five minutes absorbs a normal pause without stalling.
The javadoc said both waits are for the pane to report an injectable state.
AgentStatus.injectable() accepts BLOCKED; the settle check requires IDLE or DONE.
The wording named the wrong predicate.
A lead that runs the documented handover procedure writes its goodbye
message in the same turn as fleet_handover confirm. The deferred roll
then waits turnSettleSeconds for that same pane to reach IDLE or DONE.
An ordinary goodbye turn took 20394ms, so a 20s budget was too small
and the roll refused itself with TURN_NEVER_SETTLED. The refusal
branch is correct; only the budget was wrong.
300 is not derived from that single 20394ms observation. It matches
leadHeartbeat.idleAfterSeconds (also default 300), the only other
constant in this codebase answering "how long may a lead legitimately
be mid-turn", whose own javadoc reasons that 5 minutes absorbs normal
pauses without stalling. The two constants answer the same question
and should not disagree by a factor of fifteen. The wait stays bounded
on purpose: an unbounded wait would let a lead whose turn never ends
park a continuation and hold a pending token forever, which is harder
to notice than a logged refusal.
Also corrects FleetConfig's javadoc for turnSettleSeconds and
clearSettleSeconds, which described both waits as waiting for the
pane to report an "injectable" state again. The code requires IDLE or
DONE specifically and excludes BLOCKED (a live turn merely paused),
which is the exact state where sending /clear would destroy context.
LeadRollover.java's javadoc already states this correctly; only
FleetConfig.java's was wrong.
Adds FleetConfigTest coverage for the default's resolution: unset,
positive, and <= 0 all resolve as expected. Coverage for the two
turn-boundary properties (settles-in-time vs. never-settles, and
BLOCKED is not treated as settled) already existed in
LeadRolloverTest and needed no change — confirmed by mutating the
default back to 20 and the IDLE||DONE check to also accept BLOCKED;
both mutations were caught by existing or new tests, then reverted.
Verified by the lead before merge, each mutation re-run independently:
- three-dot diff: 1 commit, 3 files, nothing dragged in; clean trial merge
- suite exit 0, 134 test functions (5 new); the 3 FAIL and 5 mktemp lines are
the documented deliberate self-test output, unchanged from baseline
- JAR moved under target/ -> RED (the core invariant is pinned now)
- JAR moved elsewhere outside it -> PASS (not a refuse-everything test)
- PATTERN narrowed back to run/ -> RED
- plist-jar mismatch die removed -> RED
- comm=java allowlist removed -> RED (widening PATTERN did not weaken #593)
- mvn clean install from fleetd/: Tests run: 1929, Failures: 0, 172 reports
Measured separately: during a full Maven build no extra java process matches
the widened 'fleetd.jar' pattern, so assert_single_daemon does not false-
positive while a worker builds.
A lead calls fleet_handover confirm from inside its own turn, and the skill
tells it to write its goodbye in that same turn. The deferred roll then waits
turnSettleSeconds for that pane to reach IDLE or DONE. Measured on 2026-10-02,
an ordinary goodbye turn took 20394ms against a 20s budget, so the roll wrote
TURN_NEVER_SETTLED and sent no /clear. confirm() had already returned accepted,
so there was no caller left to tell, and the lead carried on believing it had
been replaced.
The refusal is the correct branch — clearing a live turn would destroy context.
The gap is that nobody is told to look afterwards.
fleet_handover{action:"status", token} already reports the outcome: FleetMcp
dispatches it to LeadRollover.status, which reads the same outcomes map the
refusal writes into. Nothing new is needed in the daemon for this half.
The signal costs nothing, because a successful roll clears the lead. A lead
that is still running after its goodbye already knows the roll failed. That is
more reliable than warning it to keep the goodbye short, which would make the
feature worst exactly when it matters most.
The live budget on this host was raised to 300s separately, in fleetd.yaml,
which is hot: the config-watcher reloaded at 21:28:00 with no "needs a restart"
clause, four seconds after the edit, with no daemon restart. The code default
is a separate change.
Part 1: asserts JAR and BUILD_JAR as the script sources them (no test assigns
them first), so reverting JAR to a path under target/ now fails the suite.
Part 2: check_jar_path_matches_plist reads the installed launchd plist's
ProgramArguments and refuses when its jar path does not resolve to $JAR,
mirroring check_log_path_matches_plist. Wired into report_supervisor_state's
launchd branch; unaffected when no plist is installed.
Part 3 (added to the ticket after the brief, by comment): PATTERN narrowed to
'fleetd.jar' so running_pid()/assert_single_daemon see a daemon regardless of
which build layout (run/ or target/) its jar sits under. The comm=java
allowlist still excludes a self-matching shell. Brought
.claude/skills/fleets-status/SKILL.md's pgrep pattern into agreement.
Verified: bash scripts/test-redeploy-fleetd.sh exits 0. Mutation both
directions for part 1 (JAR under target/ -> suite fails; JAR elsewhere ->
suite passes), a positive control for part 2 (neutralizing the mismatch
check makes the new test fail), and a positive control for part 3 (narrowing
PATTERN back to run/fleetd.jar makes the new target/-dir test fail). mvn -o
clean install: Tests run: 1929, Failures: 0, Errors: 0, Skipped: 0, BUILD
SUCCESS, 172 surefire report files.
Verified by the lead before merge:
- three-dot diff: 1 commit, 9 files, nothing dragged in
- trial merge in a throwaway worktree: no conflicts
- mvn clean install from fleetd/: Tests run: 1929, Failures: 0, 172 report files
- scripts/test-redeploy-fleetd.sh: exit 0; its 3 FAIL and 5 mktemp lines are
deliberate self-test output, confirmed identical on unmerged origin/main
- run/ is correctly ignored by the new fleetd/.gitignore rule
- the build writes target/fleetd.jar and does not create run/
Separate Maven's output path (target/fleetd.jar) from the daemon's runtime
path (run/fleetd.jar), so a clean/install in the main clone can no longer
reach the jar a running daemon holds open. redeploy-fleetd.sh now swaps the
built jar onto the runtime path with a same-filesystem mv, only after the
old daemon is confirmed gone; --check reports the built and running jars
as two separately labelled hash+mtime facts. Updates every process-locator
pattern and runtime-path reference found by git grep, with a positive
control added for both running_pid()'s PATTERN and fleets-status's pgrep
pattern. fleetd/run/ is gitignored.
The claim-2 banner said validateAll reaches seven of eight validators. With
the validateLeadRollover case added it reaches all of them, so the banner was
false. Two more places said 'today's six' and were already stale at eight.
Removed the '1491 tests, 0 failures' parenthetical: a measurement in a comment
goes stale on its own, and the suite is far past that number. The count that
readers must keep correct still lives in the Set.of, which is where the
surrounding javadoc already points them.
The class javadoc carried a ticket key, a file:line reference, a comparison
with FleetMcpAuthzTest and pointers at two other assembly tests. That is
history and review justification, which belong in the commit message and the
PR, not in the code. The javadoc now names the behaviour the test protects.
The reflection note keeps the constraint a maintainer needs (the package
boundary, and that no catch can hide a renamed denyFor) and drops the rest.
The assertion message no longer names a line number that will drift.
validateAllReachesEveryOneOfTodaysRealValidators now exercises all
eight FleetConfig validators through validateAll(), not seven -
adding a minimal leadRollover: block with no handoverPath as the
eighth fixture. The canary's failure message in
fleetConfigDeclaresExactlyTheseValidatorsToday now also points the
reader at the reachability enumeration, since updating the expected
set alone does not prove validateAll() reaches a newly added
validator.
Adds FleetdAssemblyAuthorizationModeTest: drives FleetMcp#denyFor (via reflection, since it is package-private to dev.ltms.fleet.mcp) on the real FleetMcp FleetdAssembly#assembleAndStart builds, asserting an unauthorized worker is refused SPAWN and the primary is still allowed. Mutating line 481 to UNENFORCED turns this test red and no other test.
Adds a test that reaches the real LeadTabScanner FleetdAssembly's
production boot path builds (via the LeadCoordLoop field that stores
the same leads supplier instance) and asserts, by reflection, that
the excludedWorkspaceLabels field is empty. Mutating line 265 to any
non-empty set now turns this test red.
FleetConfig declares eight public no-arg void validate* methods, at lines
2671, 2708, 2755, 2790, 2827, 2856, 2896 and 2958. The sweep's javadoc still
named a count. The wording is now count-free, so it cannot drift again.
validateAllReachesEveryOneOfTodaysSixValidators only covered six of
the eight real validators; the new validator was reachability-tested
only from FleetConfigTest, in a different file from the one whose job
is to enumerate every validateAll-reachability case.
Add the pane-placement case to the enumeration, rename the method to
drop the hardcoded count (validateAllReachesEveryOneOfTodaysRealValidators),
and correct the surrounding claims to say seven of eight, naming
validateLeadRollover as the one case still missing (fleetd #668, not
fixed here).
LeadLauncher.LeadCount's javadoc carried the same false member-space-
exclusion claim as the three comments fixed earlier in this ticket;
its neighbouring body comment in countLeads was already correct and
is unchanged.
FleetConfigValidateAllTest's canary test is renamed to drop the
number from its name (the count now lives only in the Set.of literal
and the javadoc, so the two cannot drift), which also fixes the
dangling {@link} to the old name and the stale 'seventh' wording.
The old text warned only that a 'mvn clean' deletes the running daemon's jar.
Replacement is enough: the shutdown drain loads its classes lazily, at
shutdown, from the jar file the JVM opened at boot. So any build that writes
fleetd/target/fleetd.jar under a live daemon breaks its drain, nothing warns
at the time, and the damage surfaces at the next restart where it looks like
the restart's fault.
Both files now state the real rule and the positive one: verify a merge by
building in a throwaway git worktree, and let only
scripts/redeploy-fleetd.sh touch the main clone's jar.
The canonical block is untouched (20938 bytes, identical to main) and the
wiki sync check passes against wiki/7-Use-Cases.md.
The 3-arg form delegated to the 4-arg one with a null window, so any caller
reaching for it silently got the fixed HIGH_THRESHOLD_TOKENS back instead of
the profile's effective auto-compact window. It had no production callers.
All 15 test call sites move to the 4-arg form.
Verified at 60fa86a in a throwaway worktree: Tests run: 1923, Failures: 0,
Errors: 0, Skipped: 0, BUILD SUCCESS, 170 surefire report files. Mutating
HIGH_THRESHOLD_TOKENS to 200_000 * 2 turns
noEffectiveWindowFallsBackToTheFixed200000Default red at line 81.
noEffectiveWindowFallsBackToTheFixed200000Default's third assertion
(legacyConfigDir/legacyGauge) used to call the 3-arg read() to prove
it behaved like a null window. With the 3-arg form gone, it is the
same call, same input (200_000) and same expectation as the atGauge
assertion above it, so it cannot fail unless that one already failed,
and its message named a method that no longer exists. Deleted it; the
first two assertions (199_999 -> OK, 200_000 -> HIGH) are unchanged.
A pane-placed member lands inside the focused tab rather than its own,
so it can land inside a lead's labelled tab and be read back as that
lead by LeadTabScanner, which does not exclude the member space in
production. Add FleetConfig.validatePanePlacementAgainstLeadTabs(),
wired automatically into validateAll() by the existing reflective
sweep, to refuse that combination at startup.
Also correct three stale comments that claimed a member-space
exclusion already blocked this path, in LeadTabScanner, FleetConfig's
validateLeadTabPrefixes javadoc, and LeadTabScannerTest.
The 3-arg read(configDir, sessionId, agentType) delegated to the 4-arg
form with a null window, silently restoring the fixed 200_000 fallback
that #637 moved away from. It had zero production callers; both
production call sites already use the 4-arg form.
Migrate all test call sites to the 4-arg form. For the generic property
tests in LeadContextGaugeTest, the window is irrelevant and null is
filler. In LeadContextGaugeHighThresholdTest's
noEffectiveWindowFallsBackToTheFixed200000Default, null is the
meaningful value under test, not filler; the assertion is unchanged.
FleetMcp.LeadConfigDirSource's 1-arg constructor, Fleetd.leadContextLookup's
4-arg overload, and Fleetd.leadContextSource's 4-arg overload each existed only
to keep old call sites compiling, and each silently resolved no auto-compact
window — reverting any caller that picked one up to LeadContextGauge's fixed
200,000 HIGH threshold, the exact defect #637 fixed. Removed all three and
updated the 8 call sites across 4 test files to pass the window lookup
explicitly.
Added FleetdLeadContextSourceWindowAssemblyTest: no existing test called the
real assembled LeadHeartbeatLoop far enough to prove FleetdAssembly's
window-lookup argument into Fleetd.leadContextSource actually reaches the
gauge. Mutating that argument to `_ -> null` compiled clean and left the whole
suite green; the new test fails against that mutation. Needed a small FakeHerdr
addition (agentSessionId(..)) since its default agent.get response carries no
session id.
Fleetd.leadConfigDirSource built its window argument with nothing calling the
factory itself to prove it, so a mutation to a no-op lookup left the whole
suite green. Add a wiring test that calls the factory directly, modeled on
FleetdLeadConfigDirSourceWiringTest's shape for the configDir half of the
same factory.
LeadContextGauge's result cache keyed only on (configDir, sessionId), but the
cached Reading's state now depends on the caller's resolved effective window.
Fold the derived threshold into the cache key so two reads of the same
session with different windows inside the TTL each report against their own
window.
HIGH_THRESHOLD_TOKENS was a hardcoded 200_000, while the event it warns about
(auto-compaction) is configured per profile via autoCompactWindow and can legally go
as low as 100_000 — making HIGH unreachable before a compaction on such a profile.
LeadContextGauge.read now takes an optional effective window and fires HIGH at 2/3 of
it, falling back to the fixed 200_000 when no window is resolvable (unresolved callers,
including the pre-existing 3-arg read(), keep today's behaviour exactly).
FleetConfig.Profile.effectiveAutoCompactWindow() resolves that window the way a
launched Claude Code session actually reads it: env.CLAUDE_CODE_AUTO_COMPACT_WINDOW
wins over the autoCompactWindow launch flag when both are set.
Wired into both real consumers: fleet_list's context row (FleetMcp.LeadConfigDirSource,
widened with a back-compat constructor so no unrelated call site changes) and the lead
heartbeat's context-high nudge (Fleetd.leadContextLookup/leadContextSource, widened the
same way).
Criterion 19 covers a block-scalar body whose key line falls outside
diff -u's default 3-line context (an 8-line body with only the 6th
line changed). Criterion 20 covers a blank line inside the value,
which used to reset the old indentation-anchored mask.
Both are RED against the pre-#639 redact() (git show 28ea0de) and
GREEN against the current one; each asserts both the secret's
absence and a non-secret control line's presence.
Add a positive anchor per watched file (Fleetd.java and FleetdAssembly.java),
matching FleetdConfigRefWiringTest's [SOURCE TEXT] style, so a broken read
fails loudly instead of passing the negative check vacuously. Fix dangling
javadoc @link references in FleetdConfigRefWiringTest to the *AssemblyTest
names those classes were renamed to.
FleetdAssemblyFleetAppTest's class javadoc stated that /sessions'
Authz.Action.READ gate is always refused, and that READ always needs
Caller.resolved(). That is true only under auth.mode: loopback-trust,
the mode this test runs under because it configures no auth: block.
Under auth.mode: token, CallerResolver.resolve() returns before ever
consulting Caller.resolved()/scanComplete(), so a valid bearer token
resolves to PRIMARY with no pid lookup on that path. Names the two
tests that already exercise that path against a real assembly.
Test-only, 243 lines, one new file. ZERO production change — this is the point
of the PR, and it replaces PR #647, which added three public accessors to
FleetMcp on a false premise.
#647 argued a same-JVM test caller can never reach fleet_list's READ gate,
because FleetdAssembly hardcodes a real LsofPeerPidLookup, so the caller
resolves ANONYMOUS. The premise about lsof is true; the conclusion is not. In
CallerResolver.resolve, once c.terminal() == null the next branch is:
if (tokenMode) {
return presentedTokenMatches(authorizationHeader)
? Principal.primary(c.pid()) : Principal.anonymous();
}
c.resolved() and c.scanComplete() guard only the LATER loopback-trust branch,
which token mode returns before reaching. So under auth.mode: token a bearer
token resolves to PRIMARY with no pid lookup on the path. This PR does exactly
that: real McpSyncClient callTool("fleet_list") over a real transport with an
Authorization: Bearer header, a faked leadMailboxOpener so no broker is touched,
and a real coordinator: block with a non-empty peers list. No reflection.
Lead verification, measured myself (not taken from the worker's report):
git diff --stat origin/main -- fleetd/src/main/java/ -> EMPTY
mutate :481 Fleetd.capacitySource(...) -> CapacitySource.none()
-> Tests run: 3, Failures: 1
fleetListReportsTheAssembledCapacitySource:222
the other two tests stayed GREEN
The failure message carries the live fleet_list JSON body, showing
healthCoverage, loopHealth and coordinator.peers present with capacity absent —
so the round-trip, the PRIMARY resolution and the coordinator visibility are all
real, and the mutation removed exactly one thing.
Worker also reported, each with a grep -c anchor of 1 restored: health mutation
-> 1 failure named fleetListReportsTheAssembledHealthCoverageSource; peers
mutation -> 1 failure named fleetListReportsTheAssembledCoordinatorPeers; final
mvn clean install Tests run: 1899, Failures: 0 / BUILD SUCCESS. I reproduced the
capacity cycle only; the other two are the worker's measurement, not mine.
Carries forward #647's genuine find: the peers ternary at :489 is
inert-equals-absent, so the test configures a real coordinator: block with peers.
Detail on #612 and #647.
Test-only, 368 lines, one new file. No production change.
Lead verification, measured myself in a throwaway detached worktree (not taken
from the worker's report):
starve FleetApp pass site :529 -> Tests run: 4, Failures: 2
(...ByTheRealAssembledFleetApp:333, :363)
both ...FleetMcp tests stayed GREEN
starve FleetMcp pass sites :484/:486 -> Tests run: 4, Failures: 2
(...ByTheRealAssembledFleetMcp:319, :349)
both ...FleetApp tests stayed GREEN
So the two windows are pinned independently. Failure messages carry the live
JSON response body from a real round-trip, not a source-text match.
Why this is worth merging — the REST window was completely unpinned. With this
PR's test parked and :529 starved, the FULL suite reported:
Tests run: 1892, Failures: 0, Errors: 0, Skipped: 0 / BUILD SUCCESS
Zero pre-existing tests notice the REST window losing its sources. An operator
reads GET /profiles exactly when the MCP mount is down. Arithmetic control:
1892 + this PR's 4 = 1896 = main at 6539efe.
Known caveat, recorded not fixed: the MCP-side quarantine assertion (:313) is
DUPLICATE coverage. A reviewer measured on clean origin/main that starving the
MCP quarantine arg already fails three pre-existing tests
(FleetdBackendQuarantineAssemblyTest:167, FleetdExhaustedPatternAssemblyTest:202,
FleetdOpenCodeExhaustionForwardingAssemblyTest:182), so that site was already
pinned and the PR's claim otherwise is wrong. Low severity, left in place as a
valid fleet_profiles output assertion. Detail on #648 and #612.
Reviewed by two reviewers against the diff (soundness: no issue; coupling: the
duplicate-coverage finding above).
FleetdAssembly.java's quarantineSource (:471-472) and outageSource (:473-476)
each feed two consumers: FleetMcp (fleet_profiles, :484/:486) and FleetApp
(GET /profiles, :529). No existing test distinguished the two windows for
either source.
New test drives the real FleetdAssembly.assembleAndStart, classifies a real
exhaustion/outage through the real CompletionResolver, and reads the result
back through a real McpSyncClient (fleet_profiles) and a real HttpClient
(GET /profiles), both authenticated via token-mode auth (sidesteps the
in-JVM pid-resolution dead end). No source text is read; no production code
changed.
Verified with six mutation cycles (3 per site x 2 sites: FleetMcp starved,
FleetApp starved, mis-wire with a disconnected collaborator), each run
against the full unfiltered suite and reverted after confirming the
expected test(s) alone went red. The quarantineSource FleetMcp-starve and
mis-wire cycles also trip three pre-existing tests that read the live
BackendQuarantine via FleetMcp#quarantineSource() for their own unrelated
assertions - a pre-existing incidental coupling, not newly introduced here.
Out of scope, noted per the ticket's dispatch comment: loopHealthSource
(FleetdAssembly.java:478) shares this same two-consumer shape and is
already assigned to a separate unit, r10.
Pins FleetdAssembly's loopHealth local at both of its pass sites: FleetMcp (:483, the fleet_list source) and FleetApp (:529, the real /healthz body). Two independent assertions, so starving one site leaves the other green.
Lead verification, independent of the implementer's own proof, in a throwaway detached worktree at 68397f7:
- :483 starved only -> Tests run: 2, Failures: 1. RED: fleetListUsesTheRunningLoopsInTheRealAssembledMcp, expected: <RUNNING> but was: <STOPPED>. REST test stayed GREEN.
- :529 starved only -> Tests run: 2, Failures: 1. RED: healthzUsesTheRunningLoopsInTheRealAssembledApp, with the real body {"loopHealth":{"sessionReaper":"STOPPED","statusPoller":"STOPPED"}}. MCP test stayed GREEN.
- full suite with the PR: Tests run: 1896, Failures: 0, Errors: 0 — BUILD SUCCESS (1894 baseline + 2).
The REST half binds port 0 (ephemeral) via runtime.app().start("127.0.0.1", 0), not the configured 8765, so it cannot clash with the live daemon. Teardown stops the bound app, closes the runtime, and asserts the shutdown hook was registered and the herdr client actually closed.
I chased the implementer's honestly-reported anomaly (an unrelated ClaudeCodeLauncherTest.noFixtureSeededTheDefaultClaudeJson failure in one cycle, and a suite total that moved between cycles). It is NOT caused by this PR: a baseline run at 68397f7 with this PR absent added the same 3 temp-dir project entries to the real ~/.claude.json as the run with it applied (65->68 without, 68->71 with), and ClaudeCodeLauncherTest passed in both. Filed separately.
Test-only diff, no production code touched.
Pins FleetdAssembly.java:349 behaviourally. The test installs a throwing wrapper on the real assembled Injector's listener to construct the narrow between-delivery-and-completion window, then asserts runtime.completion() still resolves the registered waiter. No sleep: it advances a controllable nano clock. Teardown runs the captured shutdown hook and asserts herdr actually closed.
Lead verification, independent of the implementer's own proof, in a throwaway detached worktree at dac5f88:
- unmutated: Tests run: 1, Failures: 0 — BUILD SUCCESS
- :349 registrar -> (_, _) -> { }: Failures: 1 — "must wire the Injector registrar to this runtime's real CompletionResolver" expected: <true> but was: <false>
- mis-wire I built myself (differs from the implementer's): Fleetd.turnRegistrar(new CompletionResolver(agents, new Rendezvous(), ...)) — a fresh Rendezvous so the registry genuinely differs: Failures: 1, same assertion
- reverted, full suite: Tests run: 1894, Failures: 0, Errors: 0 — BUILD SUCCESS, 58s (1892 baseline + r5 + r12)
Test-only diff, no production code touched. Reflection is used only to install the throwing listener; that is how the failure window is constructed, not how the assertion is made. No reviewer fan-out: member capacity is committed to the remaining Shape A implementers.
Pins the #602/#606 call site behaviourally: drives the real FleetdAssembly.assembleAndStart and asserts the assembled LeadConfigDirSource resolves a real configured configDir, which none() cannot produce.
Lead verification, run independently of the implementer's own proof, in a throwaway detached worktree at 141ae3b:
- unmutated: Tests run: 1, Failures: 0 — BUILD SUCCESS
- FleetdAssembly.java:488 -> FleetMcp.LeadConfigDirSource.none(): Tests run: 1, Failures: 1 — expected: </mnt/fake-lead-configdir> but was: <null>
- reverted, full suite: Tests run: 1893, Failures: 0, Errors: 0 — BUILD SUCCESS, 59s (baseline 1892)
Test-only diff, no production code touched. No reviewer fan-out was run: all member capacity is committed to the five Shape A implementers.
Dai Ha
Dai Ha
ltms