Compare commits

...

17 Commits

Author SHA1 Message Date
Dai Ha 94ec77a1bc t358: guard MemberSession's 5 rebuild sites and Profile.withProfile() against the back-compat-arity trap
CI / contract (pull_request) Successful in 1m23s
CI / build (pull_request) Successful in 1m53s
Follow-up to #357 (FleetConfig.withDefaults()). Reflectively enumerate each
record's own components, resolve the canonical constructor by exact
component types, build a real non-null value per component, run each
rebuild site, and assert every component survives (except the one it is
documented to change). Exclusion lists pinned at 0 for both.

Re-counted the Profile back-compat ladder directly against the source:
8 constructors (arities 25, 24, 22, 20, 18, 15, 14, 12) against a
canonical arity of 26 — the ticket's own number was explicitly untrusted.
2026-09-09 07:19:12 +07:00
Dai Ha 49f285cfda charter: a measured fact in an addendum must carry its own deletion trigger
CI / contract (push) Successful in 49s
CI / build (push) Successful in 1m36s
The operator chose this and its size; the reasoning below is the fleet01 lead's.

Placed in the canonical block's boundary paragraph rather than the orchestration
body. That paragraph already talks about the addendum layer instead of protocol,
every project that mounts the bridge inherits it, and it sits about 3800 characters
before the primary's step list, so it does not dilute the steps a lead reads while
working. Perishability is structurally an addendum problem: the block is
byte-identical across projects by construction, so a dated local measurement in the
block body would already be a layering violation.

What happened. The fleet01 lead's kb addendum held a dated merge-refusal section
that carried an instruction to delete itself once it stopped reproducing. On
2026-09-08 UTC the operator granted merge rights on akb/kb, the lead re-ran the
probe, got 409 'head out of date' where the identical request had returned 405
'User not allowed to merge PR' on 2026-09-06, and deleted the section as instructed.

Why four parts and not one. The lead's finding is that the banner did not work
because it was emphatic. It worked because the falsification condition was
executable: it carried the exact probe, the reason for the all-zeroes
head_commit_id, and what each response code meant. The lead did not have to
reconstruct the experiment or decide what would count as refutation, and just ran
it. A banner saying 'this may be out of date, verify before relying on it' costs
the same space and does nothing, because deciding what would falsify a claim is the
expensive step and a reader in the middle of another task will not pay it. So: the
date, the command, what each outcome means, and the instruction to delete. The
fourth without the second is decoration.

The closing clause is the justification for the machinery. Most stale notes are
merely wrong. This one went stale in the dangerous direction: it would have told a
future lead it could not merge at the exact moment merging became its job, silently
and with confidence. A note that goes harmlessly stale does not need this.

Note what is NOT centralized here. The banner text itself cannot be. What fired for
the lead was a specific instruction sitting on top of the specific stale fact, which
it could not read past on its way to acting. A rule elsewhere saying 'date your
measurements' would not have fired, because nobody reads that rule at the moment
they re-measure. This sentence sets the convention; the trigger still has to live
next to the fact it governs.

Propagated to the wiki template in the same turn, wiki 8c4f152 on main; the sync
check in this file's addendum reports 'in sync: True'.
2026-09-09 04:24:17 +07:00
Dai Ha 5a12ae7930 charter: test a refusal, and do not count a transport failure as one
CI / contract (push) Successful in 1m11s
CI / build (push) Successful in 1m38s
Step 8 gained a refusal paragraph in 2f71a30, which said what a lead does once the
forge refuses a merge. It did not say how a lead establishes that it was refused.
Both halves of this amendment come from the fleet01 lead, measured on akb/kb on
2026-09-08 UTC, and both are ways to be wrong about a permission you never tested.

Do not read a refusal off a permissions field. After the operator granted merge
rights, the lead re-ran its probe: POST .../pulls/53/merge with an all-zeroes
head_commit_id, chosen so the request cannot succeed on its merits and a rejection
can only mean the refusal. It returned 409 'head out of date' where the identical
request returned 405 'User not allowed to merge PR' on 2026-09-06. A 409 is payload
validation and sits after the permission gate, so the grant took. The lead reports
the repository permissions object did not change across that flip -- still
admin:false, push:true, pull:true. I did not read that object myself; my forge token
is a different identity and would return a different one, so this stays the lead's
measurement and not mine. Merge rights on a protected branch live in branch
protection, so a permissions field can be wrong in both directions.

Do not count a transport failure as a refusal. The lead's first attempt returned
HTTP 000, because GITEA_HOST already carries a scheme and a trailing slash and the
URL came out as https://https://git.ltms.dev//api/... Under a 'not 200' test that is
indistinguishable from being refused. A probe exists to separate a refusal from
everything else, so an error that never reached the gate has to be a third answer
that concludes nothing.

Propagated to the wiki template in the same turn, wiki 8c2ef96 on main; the sync
check in this file's addendum reports 'in sync: True'.
2026-09-09 04:09:08 +07:00
Dai Ha 127e6832a9 Merge #374: fleetd holds off idle sleep while any member is live
CI / contract (push) Successful in 1m18s
CI / build (push) Successful in 2m3s
Lands PR #355 (fleetd #354's sibling), rebased onto current main by a worker
after 39 commits of drift left it unmergeable.

The problem, measured on the original branch: a fleetd host idle-slept after as
little as one minute (pmset -g custom reported 'sleep 1' on battery). Overnight
the daemon's AMQP link dropped 13 times, and every drop minute had a sleep or
wake event in pmset -g log in the same minute or the one before. The AMQP churn
is the visible symptom; the real cost is a member mid-turn freezing with the
host, and a long turn with nobody typing is exactly the case that goes idle.

IdleSleepGuard holds an OS-level assertion for as long as at least one member is
live. It is driven by SessionManager's existing onAcquire/onRelease hooks rather
than a second member count kept in parallel, so it reads the same registry
fleet_list's numbers come from, and only a real 0->1 or 1->0 crossing touches the
OS. It fails safe: a mechanism that cannot acquire means nothing is ever held,
and it never throws, never blocks a spawn, a release, or shutdown.

Conflict resolution was the whole job, and all three were in config plumbing:
ConfigRef, FleetConfig and ConfigRefTopLevelReportingCoverageTest. The power
package is byte-identical to the original branch commit.

Verified on this merge, not taken from the worker's report:
  mvn clean install -> Tests run: 1439, Failures: 0, Errors: 0, BUILD SUCCESS
  (1425 on main + 14 new: 4 caffeinate, 5 guard, 1 wiring, 4 config)

The denominator recount, which the worker flagged as its own weakest number
because this file's count has drifted three times before (#330/#333/#337). I
counted it mechanically rather than reading it: FleetConfig has 24 canonical
record components; COLD_KEYS 5, DEFERRED_KEYS 13, SPLIT_KEYS 3, plus the 3 the
javadoc names as hot-excluded (placement, memberCredentials, memberLoginShell).
5+13+3+3 = 24. The javadoc's '24 components: 5 cold, 13 deferred, 3 split, 3
hot-excluded' is correct. The worker's prose called idleSleepGuard the 25th
constructor argument; it is the 24th. The code is right, the report was off by
one.

Mutation run on merge, on the half the worker verified by READING rather than by
proving -- it said it had checked that withDefaults()'s final call binds the true
canonical constructor. I dropped the trailing idleSleepGuard argument so the call
silently binds the 23-arg back-compat overload. It compiles, which is the whole
hazard. Caught: 1 failure, 3 errors, BUILD FAILURE, and
FleetConfigWithDefaultsPreservesEveryComponentTest names the dropped component
and prints its own denominator -- '24 components, 24 checked, 0 excluded, 23
survived'. That test was added on main after this exact defect happened live when
idleSleepGuard was added on a sibling branch; the worker had to add the missing
entry to it, and doing so is what makes the guard cover this component at all.
2026-09-07 20:31:41 +07:00
Dai Ha 6442a583ae t355: give the withDefaults() coverage test a value for idleSleepGuard
CI / contract (pull_request) Successful in 53s
CI / build (pull_request) Successful in 2m6s
FleetConfigWithDefaultsPreservesEveryComponentTest was added on main after
the sleep-guard commit was cherry-picked, specifically to catch this exact
rebase hazard (a withDefaults() call silently rebinding to a stale-arity
back-compat constructor). Its baseValues() map didn't know about the new
idleSleepGuard component yet, so the coverage test itself failed the
name-drift check. Add a real, non-null value for it, consistent with how
the sibling ConfigRefTopLevelReportingCoverageTest already covers it.
2026-09-07 20:25:33 +07:00
Dai Ha 24b96d29ae fleetd must not let the host idle-sleep while members are live
Adds a small IdleSleepGuard (dev.ltms.fleet.power) that holds a macOS
caffeinate -i child while at least one fleet member is live, and
releases it once none are. It hangs off SessionManager's existing
onAcquire/onRelease hooks and SessionManager#size() rather than
tracking members a second way. New idleSleepGuard: config block,
on by default, following the FleetConfig.Health/ConfigReload pattern.
2026-09-07 20:22:11 +07:00
Dai Ha 4721771052 charter: a peer reads your project addendum
CI / contract (push) Successful in 47s
CI / build (push) Successful in 1m47s
Fourth item under Lead <-> lead. The addendum is instruction surface -- every
future session on that host obeys it, and a wrong one is obeyed as faithfully as
a right one -- but nothing in the block said to have anyone check it.

Evidence is one addendum, written on fleet01 this week, and it carried two
defects that its author did not see and a non-author did. First: it described a
forge permission wall as if it were policy, so a token regrade would have made
it tell a session NOT to merge at the moment merging became its job. Second,
found only because the first was raised: it DID carry the 'this is a dated
measurement' caveat, at the bottom, after the prohibition -- so a session that
read the instruction and stopped had already taken it as policy. The caveat sat
downstream of the thing it qualified.

Neither is a writing slip. Both are the author being unable to see their own
qualifier placement, which is what a second reader is for.

Deliberately conditional: not every operator runs two leads, so the rule ends
with what a lone lead can still do -- ask which sentence goes false first, and
whether a reader reaches the caveat before acting.

Weaker evidence than the step 8 amendment (n=1 addendum, 2 defects, versus a
measured 405). Recorded as such so it can be dropped if it does not earn the
context it costs.
2026-09-07 05:03:58 +07:00
Dai Ha 2f71a30bd7 charter: say what step 8 means when the forge refuses the merge
CI / contract (push) Successful in 1m16s
CI / build (push) Successful in 1m35s
Step 8 said 'then merge' and nothing about a refusal. fleet01 is the first host
to hit that: on akb/kb its lead gets 405 'User not allowed to merge PR' from the
API, and main is protected, so merging locally and pushing is refused too. Both
routes shut. The charter was telling a lead to do something the forge would not
let it do, and that gap was latent in every copy of the block.

The amendment keeps the rule that the merge decision is never delegated, while
admitting the mechanical merge may not be the lead's to make.

The second sentence is the one that earns its keep, and it came from the fleet01
lead rather than from me: never call a PR 'ready to merge' without having read
the diff. A refusal is exactly when that shortcut is tempting, because no action
is left that forces the lead to look. Without it, a refused merge quietly turns
step 8 from 'read it yourself, then merge' into 'forward the reviewer's verdict'
— the proxy-delegation the same step forbids two lines earlier.

Propagated to the wiki template in the same turn; the sync check in this file's
addendum reports 'in sync: True'.
2026-09-07 05:01:31 +07:00
Dai Ha 22cdebbdbe fleetd #369: state hermeticGitEnv's real scope, measured
CI / contract (push) Successful in 1m18s
CI / build (push) Successful in 1m32s
The comment claimed no test in this class can reach the real machine's home
directory. Measured on the merge: 53 of the 58 new GitWorktrees(...)
constructions here pass no env override, and stripping the override from
seedingGitWorktrees leaves the class green under the poison command that the
same comment cites as proof. Say what it covers and what it does not.
2026-09-06 20:32:21 +07:00
Dai Ha 154971c2b8 Merge #372: GitWorktreesTest no longer reads the operator's real git config
fleetd #369. Every git subprocess the test class starts now goes through one
gitProcessBuilder factory that applies the hermetic environment. Before this,
gitOutput set GIT_CONFIG_GLOBAL/SYSTEM/TERMINAL_PROMPT but not XDG_CONFIG_HOME,
and status/fullStatus set nothing at all -- so the tests inherited the JVM's
whole real environment, including the operator's default excludes file. That
file applies with no core.excludesFile configured at all, and /dev/null for the
global config does not stop it.

Round 1 pinned this with a call-site count: exactly 2 literal
new ProcessBuilder( occurrences. That catches a NEW helper built the old way,
but it is a proxy, not the property. I measured the gap -- deleting
pb.environment().putAll(hermeticEnv()) from inside the factory left every call
site unchanged, the count stayed 2, and 1413 tests stayed green. Round 2 added
gitProcessBuilderCarriesTheFullHermeticEnvironment, which asserts on what the
factory actually hands to ProcessBuilder#start(). Both checks are kept: they
catch different regressions.

Verified on this merge, not taken from the worker's report:
  mvn clean install -> Tests run: 1414, Failures: 0, Errors: 0, BUILD SUCCESS
  poison control on the PRE-FIX file:
    XDG_CONFIG_HOME=<dir with a '*' git/ignore> mvn test -Dtest=GitWorktreesTest
    -> tests=59 failures=56, so the poison genuinely reaches these tests

Mutation run on merge, on a half neither the worker nor the reviewer touched --
made seedingGitWorktrees pass null instead of hermeticGitEnv(tmp), which strips
the hermetic environment from the PRODUCTION GitWorktrees instances rather than
from the test's own subprocesses: tests=61 failures=0 unpoisoned AND poisoned.
That half is unpinned. It is fleetd #362's protection, not this ticket's, and it
guards a different path -- the Java-side XDG read in
previouslyEffectiveExcludesFileContent, which no assertion observes. Out of
scope here; filed as a follow-up rather than held against this PR.

One javadoc sentence corrected in the merge: hermeticGitEnv claimed "no test in
this class can reach the real machine's home directory". 53 of the 58
new GitWorktrees(...) constructions in this file pass no env override at all, so
the claim is true of the 5 seeding sites and of every test-started subprocess,
not of the class.
2026-09-06 20:31:01 +07:00
Dai Ha fef287c346 Merge #371: a stale lead binding no longer swallows a worker's reply nudge
CI / contract (push) Successful in 47s
CI / build (push) Successful in 1m55s
fleetd #368. PrimaryRegistry.forgetDelegation fires only when a WORKER is
released, never when the delegating LEAD goes away. The map is keyed by the
worker, so a closed, crashed or relaunched lead left its bindings behind. A
stale non-null entry then beat nudgeTargetFor's single-primary fallback every
time -- and that fallback's own javadoc argues it is correct precisely in the
case the stale entry was hiding.

ReplyPushLoop now probes the recorded lead with agents.status before trusting
it, at all five entry points, and a lead that is really gone is forgotten so
resolution reaches the fallback.

Round 1 caught any RuntimeException and treated it as death, and death here
calls forgetDelegation -- destructive and permanent on ONE reading. That is the
#359 mistake repeated two days later: a socket blip or a decode error on a live
lead would silently unbind it forever. I measured the breadth was unpinned
(narrowing it left 1414 green), and round 2 narrowed it to the one affirmative
signal AgentControl.agentCall itself uses, agent_not_found. Every other failure
is treated as live, because guessing wrong costs one extra retry next tick while
guessing wrong the other way is unrecoverable.

The fallback it lands on is live, not stale: PrimaryRegistry.record overwrites
the single slot on every orchestration-side MCP call, and neither host pins
primary.terminal, so a relaunched lead re-registers on its first tool call.

Verified on this merge, not taken from the worker's report:
  mvn clean install -> Tests run: 1415, Failures: 0, Errors: 0, BUILD SUCCESS

Mutation run on merge, on a half the worker did not touch -- dropped the
forgetDelegation call while still returning the fallback, so behaviour on the
first nudge is identical and only the self-healing is lost: 2 failures,
BUILD FAILURE. The cleanup is pinned, not just the fallback.
2026-09-06 20:13:07 +07:00
Dai Ha a37acd5ee3 fleetd #369 review round 2: pin the factory's behaviour, not its call count
CI / contract (pull_request) Successful in 41s
CI / build (pull_request) Successful in 1m26s
everyGitSubprocessGoesThroughTheHermeticFactory counts ProcessBuilder("git",
...) call sites, so it catches a new helper built the old way, but a
reviewer proved it does not catch gitProcessBuilder itself being gutted:
removing pb.environment().putAll(hermeticEnv()) from inside the factory
leaves every call site unchanged, the count stays 2, and the whole
unpoisoned suite stays green.

Add gitProcessBuilderCarriesTheFullHermeticEnvironment, which inspects what
the factory actually hands to ProcessBuilder#start(): every hermetic key
present with the isolating value, and XDG_CONFIG_HOME pointed inside the
class's own throwaway directory rather than left unset or pointing at the
operator's real one. This fails the moment the hermetic environment stops
being applied, on any machine, with no poison needed. Keep the call-site
count check too — the two catch different regressions.
2026-09-06 20:09:32 +07:00
Dai Ha d6ef0c8013 fleetd #368 review: only agent_not_found may forget a lead binding
CI / build (pull_request) Successful in 1m22s
CI / contract (pull_request) Successful in 1m22s
isLive treated any RuntimeException from the liveness probe as "the lead is
gone", which forgetDelegation then acted on destructively and permanently.
That made a transient herdr hiccup (socket blip, decode error) on a perfectly
live lead indistinguishable from the lead actually being dead — the same
one-bad-reading mistake #359 shipped a guard against for lead-tab liveness.

Narrow isLive to match AgentControl.agentCall's own rule: only an affirmative
HerdrException("agent_not_found") counts as gone. Every other failure is
treated as still live and the binding is left alone.

Adds aTransientLivenessFailureMustNotForgetABindingToAStillLiveLead, which
fails with the bare RuntimeException catch and passes with the narrowed one.
2026-09-06 20:08:18 +07:00
Dai Ha dfb70871b4 fleetd #360 follow-up: the install block must say loginctl enable-linger
CI / contract (push) Successful in 43s
CI / build (push) Successful in 1m54s
PR #370 shipped both units but its install block stopped at
`systemctl --user enable --now`. Without lingering a user manager starts at
your first login and stops at your last logout, so the units do not come back
after a reboot -- which is the whole reason this ticket moved fleet01 off the
setsid scripts.

It is easy to miss because leaving it out looks like success: `systemctl --user
enable` reports "enabled" and both units run while you stay logged in. The
issue named this and the PR did not carry it over.

fleet01 itself is fine -- measured `Linger=yes`, both units `enabled`. This is
about the next host that follows these instructions.

Comment only; SystemdUnitSafetyTest still 8 green (a commented line is not an
active directive).
2026-09-06 20:08:05 +07:00
Dai Ha 4ee7b16929 Merge #370: ship systemd units that do not silently disable the daemon
CI / contract (push) Successful in 48s
CI / build (push) Successful in 2m1s
fleetd #360. deploy/fleetd.service shipped four mount-namespacing directives
(ProtectSystem, ProtectHome, ProtectKernelTunables, ProtectControlGroups),
PrivateTmp=true, and an ExecStart that ran java directly. Each one starts green
and breaks the daemon in a way nothing logs: lsof goes blind so every caller is
resolved ANONYMOUS and refused; the member ZDOTDIR scrub becomes a no-op; every
credential is empty. deploy/herdr.service did not exist at all, though
fleetd.service's After=/Wants= already named it.

Both units are now the ones running on fleet01, comments included -- the
bisected lsof counts and the reasons live in the files, because the next person
to 'harden' this needs the reason, not the rule.

A unit file has no compile step, so SystemdUnitSafetyTest reads both units plus
deploy/herdr-inner.sh and fails on an active forbidden directive, on
PrivateTmp=true, on an ExecStart that skips the login shell, on a herdr-inner.sh
that does not exec a login shell, and on one that does not set a non-zero pty
size. Each message names the consequence. A vacuity guard pins that all three
files exist and that the DO-NOT-add comment block still mentions every forbidden
directive, so 'comment survives, directive does not' is actually exercised.

Verified on this merge, not taken from the worker's report:
  mvn clean install -> Tests run: 1420, Failures: 0, Errors: 0, BUILD SUCCESS

Round 1 shipped herdr-inner.sh untested; I mutated it (dropped both the login
shell and the stty sizing) and got 6 green. Round 2 added those two checks.

Mutation run on merge, on a half the worker never touched -- ProtectHome=read-only
added to herdr.service, the unit it only ever mutated fleetd.service for:
Tests run: 8, Failures: 1, BUILD FAILURE. The parameterisation really covers
both files.
2026-09-06 20:04:48 +07:00
Dai Ha dd2efd8541 fleetd #369: make GitWorktreesTest hermetic against the machine's real git config
CI / contract (pull_request) Successful in 47s
CI / build (pull_request) Failing after 1m28s
gitOutput set GIT_CONFIG_GLOBAL/SYSTEM/TERMINAL_PROMPT but not XDG_CONFIG_HOME,
and status/fullStatus (plus every other raw git subprocess in this class) set no
isolation at all — inheriting the JVM's real environment, including the
operator's real ~/.gitconfig and default excludes file. Measured: with a
poisoned XDG_CONFIG_HOME, 56 of 59 tests failed.

Centralize every git subprocess this test starts through one factory,
gitProcessBuilder, which always applies the existing hermeticGitEnv isolation
(extended with XDG_CONFIG_HOME, the same fix #366 already applied to the
production-instance seam). Add a self-check test that counts direct
ProcessBuilder("git", ...) constructions in this file's own source and fails
if a future helper bypasses the factory, so the omission that caused this
ticket is caught by name instead of rediscovered on a poisoned machine.
2026-09-06 19:54:35 +07:00
Dai Ha 5af786d135 fleetd #368: a stale lead delegation binding must not shadow the primary fallback
CI / contract (pull_request) Successful in 51s
CI / build (pull_request) Successful in 1m55s
PrimaryRegistry.forgetDelegation only fires when a WORKER is released, never when
the delegating LEAD terminal itself disappears (closed, crashed, or relaunched).
A stale, non-null leadByTarget entry always beat nudgeTargetFor's single-primary
fallback, so a dead lead silently swallowed every reply nudge for its workers.

Fix: ReplyPushLoop now verifies (via the same agents.status check decide() already
uses every tick) that a recorded delegating lead is actually live before trusting
it. A dead lead is treated as if never recorded — self-healing the binding
(mirroring AgentControl.paneByTerminal's self-heal on agent_not_found) and falling
through to PrimaryRegistry's existing fallback.
2026-09-06 19:54:26 +07:00
22 changed files with 1513 additions and 51 deletions
+25
View File
@@ -7,6 +7,14 @@
> wiki ([Use Cases](https://git.ltms.dev/fleet/fleetd/wiki/7-Use-Cases) → *The portable
> CLAUDE.md block*); improvements go to the template first, then out to each project. Anything
> specific to *this* repo lives under §Project addendum below, never inline above it.
>
> **Anything you measure in an addendum is perishable.** Date it, give the command that
> re-measures it and what each outcome means, and tell the reader to delete the section once
> it stops reproducing. The four parts work together: deciding what would falsify a claim is
> the expensive step, and a reader in the middle of another task will not pay it, so a bare
> "verify before relying on this" costs the same space and does nothing. The case this is for
> is a note that goes stale as a live restriction — it will tell a future session it cannot do
> the thing at the moment doing it becomes the job.
If no `fleet_*` MCP tools are mounted in this session, this section does not apply — skip it.
@@ -94,6 +102,18 @@ below are the procedure — run them in order, every task, not only the big ones
8. **Adjudicate, merge, tear down — yours alone.** Read the diff yourself: fully if it is small,
targeted at the reported findings and the risky paths if it is large. Reviewer findings direct
your attention; they never substitute for it. Then merge, then `fleet_stop{paneId}`.
**If the forge refuses you the merge** — a protected branch, a token without the grant — the
adjudication is still yours. Read the diff, decide, and hand the operator a merge-ready queue
with the refusal quoted. Never report a PR as merged, and never call one "ready to merge"
without having read the diff yourself. A refusal is exactly when that shortcut is tempting,
because no action is left that forces you to look, and taking it turns this step into
forwarding a reviewer's verdict — which is delegating the merge by proxy, two lines above.
**Test a refusal; do not read it off a permissions field.** A protected branch holds its merge
rights separately from the repository permissions, so that field can say yes while the merge is
refused, and still say no after a grant makes it work. Probe instead, with a request that cannot
succeed on its merits, so a rejection can only mean the refusal. Treat a transport failure as a
third answer that proves nothing: a timeout, a DNS error or a bad URL is not a refusal, and
counting it as one makes you sure of something you never measured.
**Steps 3 and 4 are separate on purpose** — spawning and sending in one loop is how parallel work
silently becomes serial, and it is the most common way this layer is wasted. For the same reason,
@@ -140,6 +160,11 @@ The traffic between leads is coordination and nothing else:
3. **Verify a peer exactly as you verify yourself.** Peer status buys nothing: check the claim
against the code, and re-run the build. A peer's correction gets the same treatment — right or
wrong on the evidence, not on who said it. Neither of you merges the other's work unreviewed.
4. **Ask a peer to read your project addendum.** Your addendum is instruction surface: every future
session on your host obeys it, and a wrong one is obeyed just as faithfully as a right one. The
author is the worst reader of their own qualifier placement — measured here, one addendum carried
two defects and a non-author found both. If you have no peer, at least re-read it asking "which
sentence goes false first, and would a reader reach the caveat before acting?"
Being messaged by a peer does not make you its worker: answer with `fleet_reply`, and push back on
the substance if it is wrong. A peer that simply complies has thrown away the reason there are two of
+8
View File
@@ -12,8 +12,16 @@
# # edit WorkingDirectory / ExecStart below for your host's paths and java location
# systemctl --user daemon-reload
# systemctl --user enable --now herdr fleetd
# loginctl enable-linger $USER # REQUIRED -- see below
# journalctl --user -u fleetd -f
#
# `loginctl enable-linger` is not optional and is easy to miss, because leaving it out looks like
# success: `systemctl --user enable` reports "enabled" and both units run for as long as you stay
# logged in. A user manager without lingering starts at your first login and stops at your last
# logout, so the fleet simply does not come back after a reboot -- which is the whole reason to
# use systemd here rather than the setsid scripts these units replaced. Check it with
# `loginctl show-user $USER -p Linger`; the answer must be `Linger=yes`.
#
# Secrets (AI_GATEWAY_TOKEN, WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI, ...) are not set
# here and need no systemd drop-in: ExecStart runs a login shell, so they come from wherever your
# login shell already sources them (this host: ~/.fleet/secrets.sh via ~/.zprofile). If a token is
+8
View File
@@ -110,6 +110,14 @@ bind:
# notifications:
# mode: disabled
# Idle-sleep guard: while at least one member is live, hold an OS-level assertion against idle
# sleep (macOS only — a `caffeinate -i` child; a no-op elsewhere or if caffeinate is missing), so
# an unattended host does not idle-sleep out from under a member's long turn. Unlike health/
# configReload above, this is ON BY DEFAULT — omitting the block entirely leaves it enabled, the
# same as `enabled: true`. Uncomment only to turn it off:
# idleSleepGuard:
# enabled: false
# herdr Unix socket. Omit to use the client default
# (${HERDR_SOCKET_PATH:-~/.config/herdr/herdr.sock}).
herdrSocket: ~/.config/herdr/herdr.sock
@@ -55,6 +55,8 @@ import dev.ltms.fleet.member.MemberCredentialPolicyView;
import dev.ltms.fleet.member.OpenCodeLauncher;
import dev.ltms.fleet.placement.BackendOutagePolicy;
import dev.ltms.fleet.placement.BackendQuarantine;
import dev.ltms.fleet.power.CaffeinateSleepAssertionMechanism;
import dev.ltms.fleet.power.IdleSleepGuard;
import io.javalin.Javalin;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -253,6 +255,26 @@ public final class Fleetd {
System::nanoTime, contextCap, clearAfterTurn);
liveCountRef.set(profileName -> liveSessionCount(sessions.roster(), profileName));
// Idle-sleep guard: hold an OS-level assertion against idle sleep while at least one
// member is live, so an unattended host does not idle-sleep out from under a member's
// long turn (see FleetConfig.IdleSleepGuard / dev.ltms.fleet.power.IdleSleepGuard for the
// measurement that motivated this). Opt-out via idleSleepGuard.enabled: false; on by
// default. Hangs off SessionManager's own onAcquire/onRelease hooks (CB-520/CB-516,
// previously wired only to the reply inbox) and SessionManager#size() — the exact registry
// fleet_list's live/capacity numbers are themselves computed from — rather than tracking
// members a second way. No-op (never constructed) off macOS or when idleSleepGuard.enabled
// is explicitly false; the mechanism itself is additionally a no-op if 'caffeinate' cannot
// be started, so this can never fail a spawn, a release, or startup.
boolean idleSleepGuardEnabled = cfg.idleSleepGuard() == null || cfg.idleSleepGuard().isEnabled();
final IdleSleepGuard idleSleepGuard;
if (idleSleepGuardEnabled) {
idleSleepGuard = new IdleSleepGuard(new CaffeinateSleepAssertionMechanism(), sessions::size);
sessions.onAcquire(_ -> idleSleepGuard.recheck());
sessions.onRelease(_ -> idleSleepGuard.recheck());
} else {
idleSleepGuard = null;
}
// CB-303 part 1: idle-ttl reaper — only when configured, defaults to disabled.
final SessionReaper reaper;
if (cfg.lifecycle() != null
@@ -704,6 +726,11 @@ public final class Fleetd {
if (configWatcher != null) configWatcher.stop(); // CB-559: stop polling the config file
mcp.close();
if (reaper != null) reaper.stop();
// Idle-sleep guard: release unconditionally, even though sessions.close() above already
// drained every session (and each release already drove the live count to 0, which
// releases the guard's assertion on its own) — this is the backstop for a drain that was
// itself interrupted or threw, so no caffeinate child ever outlives the daemon.
if (idleSleepGuard != null) idleSleepGuard.close();
// Release the broker connection last among message resources (no-op for the in-memory inbox).
if (replyInbox instanceof AutoCloseable closeable) {
try {
@@ -37,6 +37,10 @@ import java.util.function.Supplier;
* makes {@code fleet:} split rather than hot — see below.</li>
* <li><strong>Deferred</strong> — accepted into the new snapshot, but the wiring built at startup
* keeps the old value until a restart: {@code lifecycle:}, {@code leadHeartbeat:},
* {@code idleSleepGuard:} ({@code Fleetd.java} reads it once, at startup, to decide whether
* to construct an {@code IdleSleepGuard} and wire {@code SessionManager}'s
* {@code onAcquire}/{@code onRelease} hooks to it — neither is rebuilt on reload, so a
* running daemon keeps whatever this was at startup regardless of a later edit),
* {@code spawnReadyTimeoutMs} / {@code spawnReadyPollMs}, {@code quarantineCooldownSeconds}
* (CB-578 stage B — baked once into the {@code BackendQuarantine} built at startup),
* {@code guard:}, {@code worktreeRoot:}, {@code worktreeGroup:} and {@code memberSkills:}
@@ -131,9 +135,9 @@ import java.util.function.Supplier;
* </ul>
*
* <p><strong>The denominator, measured on 2026-09-04 (fleetd #330; recounted for fleetd #333);
* recounted again for fleetd #362.</strong> {@code FleetConfig} has 23 top-level record components:
* 5 cold, 12 deferred, 3 split, 3 hot-excluded. Three of them are named nowhere in this file, and
* the reason is the same for all
* recounted again for fleetd #362, and again after {@code idleSleepGuard:} was added.</strong>
* {@code FleetConfig} has 24 top-level record components: 5 cold, 13 deferred, 3 split, 3
* hot-excluded. Three of them are named nowhere in this file, and the reason is the same for all
* three: {@code placement}, {@code memberCredentials} and {@code memberLoginShell} are
* <strong>hot</strong> and correctly absent — all three are read live off {@code config.get()}
* (placement through the {@code CompositePeerLauncher} supplier the Hot bullet names;
@@ -215,7 +219,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
static final Set<String> DEFERRED_KEYS = Set.of(
"guard", "worktreeRoot", "worktreeGroup", "memberSkills", "primary", "configReload",
"leadHeartbeat", "lifecycle", "spawnReadyTimeoutMs", "spawnReadyPollMs",
"quarantineCooldownSeconds", "profiles");
"quarantineCooldownSeconds", "profiles", "idleSleepGuard");
private final Path path;
private final AtomicReference<FleetConfig> current;
@@ -427,6 +431,14 @@ public final class ConfigRef implements Supplier<FleetConfig> {
if (!Objects.equals(old.configReload(), fresh.configReload())) {
changed.add("configReload");
}
// Fleetd.java reads cfg.idleSleepGuard() once, at startup, to decide whether to construct
// an IdleSleepGuard at all and wire SessionManager's onAcquire/onRelease hooks to it —
// neither is rebuilt on reload, so a running daemon keeps whatever this was at startup
// (armed or not) regardless of a later edit here. Not cold: nothing already-open goes
// inconsistent with the new value, an armed-or-not guard just keeps its original answer.
if (!Objects.equals(old.idleSleepGuard(), fresh.idleSleepGuard())) {
changed.add("idleSleepGuard");
}
if (!Objects.equals(old.spawnReadyTimeoutMs(), fresh.spawnReadyTimeoutMs())
|| !Objects.equals(old.spawnReadyPollMs(), fresh.spawnReadyPollMs())) {
changed.add("spawnReady*");
@@ -119,6 +119,11 @@ import java.util.regex.PatternSyntaxException;
* subdirectory of this directory is copied wholesale, with no per-file
* allowlist — do not park scratch files or drafts alongside the real skill
* folders, they will be copied into every provisioned worktree too.
* @param idleSleepGuard opt-in-by-default: hold an OS-level assertion against idle sleep while at
* least one member is live, so an unattended host does not sleep out from
* under a member's long turn. {@code null} (the block omitted) behaves the
* same as an explicit {@code enabled: true}; set {@code enabled: false} to
* turn it off. See {@link dev.ltms.fleet.power.IdleSleepGuard}.
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record FleetConfig(
@@ -144,7 +149,22 @@ public record FleetConfig(
Coordinator coordinator,
String worktreeGroup,
String memberLoginShell,
String memberSkills) {
String memberSkills,
IdleSleepGuard idleSleepGuard) {
/** Back-compat form before the {@code idleSleepGuard:} block was added. */
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
Guard guard, String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
ConfigReload configReload, Integer quarantineCooldownSeconds,
MemberCredentials memberCredentials, Coordinator coordinator, String worktreeGroup,
String memberLoginShell, String memberSkills) {
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup,
memberLoginShell, memberSkills, null);
}
/** Back-compat form before the {@code memberSkills} key was added. */
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
@@ -157,7 +177,7 @@ public record FleetConfig(
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup,
memberLoginShell, null);
memberLoginShell, null, null);
}
/** Back-compat form before the {@code memberLoginShell} key was added. */
@@ -169,7 +189,7 @@ public record FleetConfig(
MemberCredentials memberCredentials, Coordinator coordinator, String worktreeGroup) {
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup, null);
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup, null, null);
}
/** Back-compat form before the {@code worktreeGroup} key was added. */
@@ -1279,6 +1299,25 @@ public record FleetConfig(
}
}
/**
* Hold an OS-level assertion against idle sleep while at least one member is live (see
* {@link dev.ltms.fleet.power.IdleSleepGuard}).
*
* <p>Unlike most opt-in blocks in this file, this one defaults to <em>on</em>: an unattended
* host idle-sleeping mid-turn is a correctness problem (a dropped AMQP link, a frozen member),
* not a convenience, so the safer default is armed. An operator who wants the previous
* behaviour (no assertion held, ever) sets {@code enabled: false} explicitly.
*
* @param enabled {@code false} turns the guard off; {@code null} (the block omitted
* entirely) or {@code true} leaves it on
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public record IdleSleepGuard(Boolean enabled) {
public boolean isEnabled() {
return !Boolean.FALSE.equals(enabled);
}
}
/**
* The terminal → lead-name map seeded from the legacy singular {@code primary:} pin (CB-530).
*
@@ -1529,7 +1568,8 @@ public record FleetConfig(
"bind", "herdrSocket", "memberHerdrSocket", "profiles", "guard", "worktreeRoot",
"lifecycle", "spawnReadyTimeoutMs", "spawnReadyPollMs", "broker", "primary", "fleet",
"leadHeartbeat", "health", "placement", "auth", "configReload", "quarantineCooldownSeconds",
"memberCredentials", "coordinator", "worktreeGroup", "memberLoginShell", "memberSkills");
"memberCredentials", "coordinator", "worktreeGroup", "memberLoginShell", "memberSkills",
"idleSleepGuard");
/** Load and validate config from {@code path}. */
public static FleetConfig load(Path path) {
@@ -2209,9 +2249,15 @@ public record FleetConfig(
// memberSkills is left as-is (fleetd #362), like worktreeGroup/memberLoginShell: null/blank
// is "off", and there is no sane non-null default — the daemon may not even run from a
// checkout that ships its own .claude/skills/.
// idleSleepGuard is left as-is, like leadHeartbeat/configReload above, but for the opposite
// reason: it is on by default already (its own isEnabled() treats null the same as
// enabled: true — see its javadoc), so defaulting the block here would change nothing a
// reader observes and would only obscure that "block omitted" and "block present and
// enabled" are deliberately the same outcome.
return new FleetConfig(b, herdrSocket, memberHerdrSocket, profiles, g, worktreeRoot, l, timeout, pollMs,
broker, primary, f, leadHeartbeat, health, placementOrDefault, a, configReload,
quarantineCooldown, mc, coordinator, worktreeGroup, memberLoginShell, memberSkills);
quarantineCooldown, mc, coordinator, worktreeGroup, memberLoginShell, memberSkills,
idleSleepGuard);
}
/**
@@ -2,6 +2,7 @@ package dev.ltms.fleet.msg;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.mcp.PrimaryRegistry;
import dev.ltms.fleet.metrics.FleetMetrics;
import dev.ltms.fleet.metrics.Metrics;
@@ -13,6 +14,7 @@ import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledExecutorService;
@@ -375,6 +377,80 @@ public final class ReplyPushLoop {
return Action.WAIT_BUSY;
}
/**
* Resolve who to nudge about {@code target}, the way every public entry point below wants it:
* {@link PrimaryRegistry#nudgeTargetFor}, but only after checking the delegating lead it names
* is still actually there (fleetd #368).
*
* <p><strong>The bug this closes.</strong> {@code PrimaryRegistry.forgetDelegation} is wired to
* exactly one event — a worker's release — because that is the only teardown the daemon already
* observes for a session in this map. Nothing removes a binding when the LEAD half goes away: a
* lead that is closed, crashes, or is relaunched leaves {@code leadByTarget} entries pointing at
* a terminal that no longer exists. {@code nudgeTargetFor} falls back to the single known
* primary only when the map holds nothing for {@code target} — a stale non-null entry beats the
* fallback every time, which is exactly backwards: the fallback's own javadoc argues it is safe
* precisely in the case a stale entry now hides.
*
* <p><strong>The fix.</strong> Before trusting a recorded delegation, probe the lead the same
* way {@link #decide} already does every tick ({@code agents.status}) — cheap, since it is a
* local herdr round-trip, and it is the same signal {@code AgentControl.paneByTerminal} already
* trusts to tell a genuinely dead target from a live one. A lead that fails the probe is treated
* as if it had never been recorded: the stale entry is forgotten (self-healing, exactly like
* {@code AgentControl.paneByTerminal} already does on {@code agent_not_found}) and resolution is
* retried, which now reaches the fallback {@code nudgeTargetFor} was built to reach — the same
* empty-map state its javadoc already argues is correct.
*
* <p><strong>fleetd #368 review — only a positive "gone" reading forgets the binding.</strong>
* The first version of this method treated <em>any</em> {@code RuntimeException} from the probe
* as death, which is the #359 mistake repeated: a transient socket blip or a codec error on a
* perfectly live lead would silently and permanently unbind it, with no re-record ever coming.
* That is destructive on one bad reading, exactly what #359 shipped a two-reading guard to avoid
* for the analogous lead-tab-liveness question. {@link #isLive} now matches
* {@code AgentControl.agentCall}'s own narrower rule (see its {@code agent_not_found} check): only
* that specific, affirmative "herdr has no such agent" signal counts as gone. Every other failure
* — timeout, transport error, a decode error — is treated as still live and the binding is left
* alone, because guessing wrong here is unrecoverable while guessing "live" merely costs one more
* retry on the next tick, which {@link #decide} already tolerates.
*/
private Optional<String> resolveLiveLead(String target) {
Optional<String> lead = primaryRegistry.nudgeTargetFor(target);
if (lead.isEmpty() || isLive(lead.get())) {
return lead;
}
log.debug("push: lead {} delegated to for {} is no longer live, forgetting the stale binding "
+ "and falling back", lead.get(), target);
primaryRegistry.forgetDelegation(target);
return primaryRegistry.nudgeTargetFor(target);
}
/**
* Whether {@code lead} should still be trusted: {@code false} only when herdr affirmatively
* reports the terminal gone ({@code agent_not_found}), never on a merely inconclusive failure.
*
* <p>fleetd #368 review: an earlier version returned {@code false} for any {@code RuntimeException},
* which made a transient herdr hiccup on a live lead indistinguishable from the lead actually
* being dead — and the caller's response to {@code false} ({@code forgetDelegation}) is
* destructive and permanent. Narrowed to the one code {@code AgentControl.agentCall} itself
* already treats as a genuine, resolvable absence (see its {@code agent_not_found} handling) —
* every other {@code RuntimeException} is treated as "still live" and the binding survives to be
* probed again next time, which costs nothing worse than one more retry.
*/
private boolean isLive(String lead) {
try {
agents.status(lead);
return true;
} catch (RuntimeException e) {
boolean gone = e instanceof HerdrException he && "agent_not_found".equals(he.code());
if (gone) {
log.debug("push: lead {} no longer exists ({})", lead, e.toString());
} else {
log.debug("push: liveness check for lead {} was inconclusive ({}); treating as live "
+ "rather than risk destroying a live binding", lead, e.toString());
}
return !gone;
}
}
// --- public entrypoints ----------------------------------------------------------------------
/**
@@ -385,7 +461,7 @@ public final class ReplyPushLoop {
* backstop until a lead is recorded.
*/
public void onReplyQueued(String target) {
var lead = primaryRegistry.nudgeTargetFor(target);
var lead = resolveLiveLead(target);
if (lead.isEmpty()) {
log.debug("push: no lead is known to be waiting on {}, skipping reminder", target);
return;
@@ -413,7 +489,7 @@ public final class ReplyPushLoop {
* @param failed whether the ticket ended in a failure phase rather than {@code DONE}
*/
public void onTicketTerminal(String ticket, String target, boolean failed) {
var lead = primaryRegistry.nudgeTargetFor(target);
var lead = resolveLiveLead(target);
if (lead.isEmpty()) {
log.debug("push: no lead is known to be waiting on ticket {} (target {}), skipping nudge",
ticket, target);
@@ -448,7 +524,7 @@ public final class ReplyPushLoop {
* @param question the question text
*/
public void onQuestionOpened(String ticket, String target, String turnId, String question) {
var lead = primaryRegistry.nudgeTargetFor(target);
var lead = resolveLiveLead(target);
if (lead.isEmpty()) {
log.debug("push: no lead is known to be waiting on {}'s question (turnId {}), skipping nudge",
target, turnId);
@@ -476,7 +552,7 @@ public final class ReplyPushLoop {
Collection<String> profiles, int remainingCoolOffSeconds) {
Map<String, List<String>> targetsByLead = new ConcurrentHashMap<>();
for (String target : targets) {
var lead = primaryRegistry.nudgeTargetFor(target);
var lead = resolveLiveLead(target);
if (lead.isEmpty()) {
log.warn("push: backend incident {} has no known lead for target {}", incidentId, target);
continue;
@@ -498,7 +574,7 @@ public final class ReplyPushLoop {
* Without an owning lead, emit a warning because no control can act on the target.
*/
public void onBackendTargetUnmapped(String target, String reason) {
var lead = primaryRegistry.nudgeTargetFor(target);
var lead = resolveLiveLead(target);
if (lead.isEmpty()) {
log.warn("push: backend target {} could not map to a credential: {}", target, reason);
return;
@@ -0,0 +1,93 @@
package dev.ltms.fleet.power;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Locale;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
/**
* Holds macOS idle sleep off by keeping a {@code caffeinate -i} child process alive for the life
* of the returned {@link SleepAssertion}.
*
* <p>{@code -i} asserts only against <em>idle</em> sleep — it does not stop the lid closing or an
* operator-requested sleep from taking effect. That is deliberate: this class exists to stop an
* unattended host from sleeping out from under a member's long turn, never to override the
* operator. {@code -s}/{@code -d} (which also block system/display sleep on demand) are
* intentionally not used here.
*
* <p>{@link #acquire()} never throws. It returns {@code null} — a no-op — off macOS, and again if
* starting the {@code caffeinate} child fails for any reason (binary missing, process table full,
* …); either case is logged once at INFO, not on every occurrence, so a daemon that runs for
* weeks with the tool unavailable does not fill its log.
*/
public final class CaffeinateSleepAssertionMechanism implements SleepAssertionMechanism {
private static final Logger log = LoggerFactory.getLogger(CaffeinateSleepAssertionMechanism.class);
private final AtomicBoolean loggedOnce = new AtomicBoolean(false);
/** {@code true} when running on macOS, the only platform {@code caffeinate} ships on. */
public static boolean isSupportedPlatform() {
return isSupportedPlatform(System.getProperty("os.name"));
}
/** Package-visible so a test can drive the platform check without touching a real property. */
static boolean isSupportedPlatform(String osName) {
return osName != null && osName.toLowerCase(Locale.ROOT).contains("mac");
}
@Override
public SleepAssertion acquire() {
if (!isSupportedPlatform()) {
logOnce("not running on macOS (os.name={}); the idle-sleep guard is a no-op on this platform",
System.getProperty("os.name"));
return null;
}
try {
Process process = new ProcessBuilder("caffeinate", "-i")
.redirectOutput(ProcessBuilder.Redirect.DISCARD)
.redirectError(ProcessBuilder.Redirect.DISCARD)
.start();
return new CaffeinateAssertion(process);
} catch (IOException | RuntimeException e) {
logOnce("could not start 'caffeinate -i' ({}); the host may idle-sleep while members are live",
e.toString());
return null;
}
}
private void logOnce(String format, Object arg) {
if (loggedOnce.compareAndSet(false, true)) {
log.info("idle-sleep guard: " + format, arg);
}
}
/** Wraps the live {@code caffeinate} child; {@link #close} force-destroys it, idempotently. */
private static final class CaffeinateAssertion implements SleepAssertion {
private final Process process;
CaffeinateAssertion(Process process) {
this.process = process;
}
@Override
public void close() {
if (!process.isAlive()) {
return;
}
process.destroy();
try {
if (!process.waitFor(2, TimeUnit.SECONDS)) {
process.destroyForcibly();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
process.destroyForcibly();
}
}
}
}
@@ -0,0 +1,105 @@
package dev.ltms.fleet.power;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.function.IntSupplier;
/**
* Holds an OS-level assertion against idle sleep for exactly as long as at least one fleet
* member is live.
*
* <p><strong>Why this exists:</strong> a fleetd host was measured idle-sleeping after as little
* as one minute of inactivity (its {@code pmset -g custom} reports {@code sleep 1} on battery).
* Overnight the daemon's AMQP link to the broker dropped 13 times, and cross-checking every drop
* minute against {@code pmset -g log} found a sleep or wake event in the same minute or the one
* before, every time. The AMQP churn is only the visible symptom — the real problem is that a
* member mid-turn freezes with the host, and a long turn with nobody typing is exactly the case
* that goes idle.
*
* <p><strong>How it tracks "live":</strong> this is driven by {@code SessionManager}'s existing
* {@code onAcquire}/{@code onRelease} lifecycle hooks (added for CB-520/CB-516, previously wired
* to nothing but the reply inbox) rather than a second member count kept in parallel. Wire it as:
* <pre>{@code
* IdleSleepGuard guard = new IdleSleepGuard(mechanism, sessions::size);
* sessions.onAcquire(_ -> guard.recheck());
* sessions.onRelease(_ -> guard.recheck());
* }</pre>
* Every acquire/release event re-reads {@code SessionManager#size()} — the same registry {@code
* fleet_list}'s live/capacity numbers are themselves computed from — and only an actual 0→1 or
* 1→0 crossing touches the OS. A listener exception is already caught and logged by {@code
* SessionManager} itself (it must never let a listener failure block the acquire/release it is
* reacting to), so {@link #recheck()} does not need its own top-level try/catch to honor that.
*
* <p><strong>Failure posture:</strong> every method here is safe to call whether or not {@link
* SleepAssertionMechanism#acquire()} actually works. A mechanism that returns {@code null} (wrong
* platform, missing tool, spawn failure) simply means this guard never holds anything — it never
* throws and never blocks a spawn, a release, or shutdown.
*/
public final class IdleSleepGuard implements AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(IdleSleepGuard.class);
private final SleepAssertionMechanism mechanism;
private final IntSupplier liveCount;
private final Object lock = new Object();
private SleepAssertion held;
public IdleSleepGuard(SleepAssertionMechanism mechanism, IntSupplier liveCount) {
this.mechanism = mechanism;
this.liveCount = liveCount;
}
/**
* Re-read the live count and acquire or release the held assertion to match: nothing held and
* at least one member live ⇒ acquire; something held and no member live ⇒ release. A steady
* count (still zero, still positive) is a no-op either way, so a single spawn or release only
* ever touches the OS on the crossing, not on every call.
*/
public void recheck() {
synchronized (lock) {
int live = liveCount.getAsInt();
if (live > 0 && held == null) {
held = mechanism.acquire();
if (held != null) {
log.debug("idle-sleep guard armed: {} live member(s)", live);
}
} else if (live == 0 && held != null) {
releaseHeldLocked();
}
}
}
/** {@code true} while an assertion is actually held. Exposed for tests. */
boolean isHeld() {
synchronized (lock) {
return held != null;
}
}
/**
* Release whatever is held, if anything. Idempotent and safe to call at any time, including
* repeatedly — a daemon shutdown hook calls this unconditionally so no assertion (and no
* {@code caffeinate} child) survives the process, even if the drain that would otherwise have
* driven the live count to zero was itself interrupted or threw.
*/
@Override
public void close() {
synchronized (lock) {
if (held != null) {
releaseHeldLocked();
}
}
}
/** Caller must hold {@link #lock}. */
private void releaseHeldLocked() {
try {
held.close();
} catch (RuntimeException e) {
log.warn("idle-sleep guard: failed to release its assertion cleanly: {}", e.toString());
} finally {
held = null;
}
}
}
@@ -0,0 +1,11 @@
package dev.ltms.fleet.power;
/**
* A held OS-level assertion against idle sleep. {@link #close} must be idempotent — safe to call
* more than once — and must never throw, matching {@link IdleSleepGuard}'s "never break the
* fleet" contract.
*/
public interface SleepAssertion extends AutoCloseable {
@Override
void close();
}
@@ -0,0 +1,21 @@
package dev.ltms.fleet.power;
/**
* The OS mechanism {@link IdleSleepGuard} uses to hold and release an idle-sleep assertion. This
* is the seam a test exercises instead of the real effect (a live {@code caffeinate} child) — see
* {@code IdleSleepGuardTest}.
*
* <p>Implementations must never throw. Every failure — wrong platform, missing tool, a spawn
* error — must show up as {@link #acquire()} returning {@code null}, so a caller can treat "no
* assertion held" and "the mechanism could not be used" identically and the fleet keeps running
* either way.
*/
public interface SleepAssertionMechanism {
/**
* Acquire a fresh assertion against idle sleep, or {@code null} when this mechanism is not
* usable right now (wrong platform, the tool is missing, the child process could not start).
* Never throws.
*/
SleepAssertion acquire();
}
@@ -108,6 +108,7 @@ class ConfigRefTopLevelReportingCoverageTest {
v.put("worktreeGroup", "group-a");
v.put("memberLoginShell", null);
v.put("memberSkills", "/skills/a");
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(true));
assertNamesMatchComponents(v);
return v;
}
@@ -149,6 +150,7 @@ class ConfigRefTopLevelReportingCoverageTest {
v.put("worktreeGroup", "group-b");
v.put("memberLoginShell", null);
v.put("memberSkills", "/skills/b");
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(false));
assertNamesMatchComponents(v);
return v;
}
@@ -0,0 +1,172 @@
package dev.ltms.fleet.config;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Constructor;
import java.lang.reflect.RecordComponent;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* Fleetd #358, the same "defect factory" #357 guarded on {@code FleetConfig.withDefaults()}
* (see {@code FleetConfigWithDefaultsPreservesEveryComponentTest}), reproduced here on
* {@link FleetConfig.Profile}. {@code Profile} carries a long back-compat constructor ladder — 8
* constructors, re-counted directly against the source rather than trusted from the ticket, at
* arities 25, 24, 22, 20, 18, 15, 14 and 12, against a canonical arity of 26 — and exactly ONE
* rebuild site, {@link FleetConfig.Profile#withProfile(String)}, whose own
* {@code return new Profile(...)} call is written at a literal 26-arg count. Add a 27th component
* and its established back-compat constructor at the old (26-arg) arity, and {@code withProfile}'s
* own call becomes a legal match for that new overload — silently dropping the new component every
* time a profile's name is defaulted from its {@code workers:} key.
*
* <p>Builds one {@link FleetConfig.Profile} through the TRUE canonical constructor — resolved by
* the record's own component types via {@code getDeclaredConstructor}, never by argument count —
* with a real, distinctive, non-null value in every component, calls {@link
* FleetConfig.Profile#withProfile(String)}, and asserts every component except {@code profile}
* itself survives unchanged, while {@code profile} comes back as the new name it was given.
*
* <p>Every value here is chosen so {@code Profile}'s own compact constructor (which normalizes
* several components — defaults {@code argv}/{@code kind}/{@code placement}/{@code workspace}/
* {@code gitHostEnv}, nulls a handful of blank-checked strings, clamps {@code weight}, coerces
* {@code subscription}) leaves it unchanged: every String is non-blank and already in the shape the
* compact constructor would otherwise coerce it to (e.g. {@code placement} is already lowercase),
* and every collection is non-empty. That is what makes "must survive unchanged" a valid assertion
* for every component below, the same reasoning {@code FleetConfigWithDefaultsPreservesEveryComponentTest}
* documents for {@code withDefaults()}.
*
* <p>{@link #EXCLUDED_FROM_SURVIVAL_CHECK} is kept deliberately empty and size-pinned by
* {@link #exclusionListSizeIsPinned()} — a checker whose escape hatch can grow to silence a failure
* is not a checker. Every one of {@code Profile}'s 26 current components has a real, non-null,
* non-blank value here and none is excluded.
*/
class FleetConfigProfileWithProfilePreservesEveryComponentTest {
private static final RecordComponent[] COMPONENTS = FleetConfig.Profile.class.getRecordComponents();
/** Deliberately empty today; grow it only with a matching justification, and re-pin the size. */
private static final Set<String> EXCLUDED_FROM_SURVIVAL_CHECK = Set.of();
/** One real, distinctive, non-null value per component, chosen to survive the compact ctor. */
private static Map<String, Object> baseValues() {
Map<String, Object> v = new LinkedHashMap<>();
v.put("profile", "profile-guard");
v.put("baseUrl", "https://guard.example/base");
v.put("model", "model-guard");
v.put("configDir", "/config/guard");
v.put("tokenEnv", "GUARD_TOKEN");
v.put("argv", List.of("guard-cmd"));
v.put("placement", "guard-placement");
v.put("workspace", "workspace-guard");
v.put("tabLabel", "tab-guard");
v.put("mcpUrl", "https://mcp.guard/");
v.put("cwd", "/cwd/guard");
v.put("parityOverlay", List.of(".guardrc"));
v.put("gitTokenEnv", "GUARD_GIT_TOKEN");
v.put("gitHostEnv", "GUARD_GIT_HOST");
v.put("kind", "claude-code");
v.put("env", Map.of("GUARD_ENV", "1"));
v.put("weight", 2.5f);
v.put("maxLoad", 4);
v.put("subscription", Boolean.TRUE);
v.put("exhaustedPattern", "pattern-guard");
v.put("credentialId", "cred-guard");
v.put("ideMcpUrl", "https://ide.guard/");
v.put("ideProjectDir", "ide-project-guard");
v.put("ideOpenCommand", "open-guard {dir}");
v.put("autoCompactWindow", 150_000);
v.put("errorPattern", "error-pattern-guard");
assertNamesMatchComponents(v);
return v;
}
/**
* Guards {@link #baseValues()} itself against drifting from the record's real shape — forgetting
* to add a new component here fails this assertion by name, rather than silently checking one
* component fewer than the record has.
*/
private static void assertNamesMatchComponents(Map<String, Object> values) {
Set<String> names = new TreeSet<>();
for (RecordComponent rc : COMPONENTS) {
names.add(rc.getName());
}
assertEquals(names, new TreeSet<>(values.keySet()),
"this test's value map has drifted from FleetConfig.Profile's actual components — "
+ "update baseValues() alongside the record");
}
/**
* Builds a {@link FleetConfig.Profile} through the TRUE canonical constructor — resolved by the
* record's own component types, not by argument count — so this never accidentally exercises a
* back-compat overload the way a literal {@code new Profile(...)} call risks doing.
*/
private static FleetConfig.Profile profileOf(Map<String, Object> values) throws ReflectiveOperationException {
Class<?>[] types = Arrays.stream(COMPONENTS).map(RecordComponent::getType).toArray(Class<?>[]::new);
Object[] args = Arrays.stream(COMPONENTS).map(rc -> values.get(rc.getName())).toArray();
Constructor<FleetConfig.Profile> ctor = FleetConfig.Profile.class.getDeclaredConstructor(types);
return ctor.newInstance(args);
}
@Test
void exclusionListSizeIsPinned() {
assertEquals(0, EXCLUDED_FROM_SURVIVAL_CHECK.size(),
"EXCLUDED_FROM_SURVIVAL_CHECK grew from 0 — every entry needs a justification in "
+ "this test class's javadoc AND this assertion re-pinned to the new size; a "
+ "growing exclusion list that silences failures on its own is not a guard");
}
/**
* The mutation this is built to catch: make {@code withProfile(String)}'s final constructor call
* literal at some arg count, add one more component to the record with a new back-compat
* constructor at the old arity, and the stale call silently rebinds. Every component here is real
* and non-null/non-blank, so none of it should be replaced by {@code withProfile}, except
* {@code profile} itself, which the method is documented to replace.
*/
@Test
void withProfilePreservesEveryOtherComponent() throws ReflectiveOperationException {
Map<String, Object> base = baseValues();
FleetConfig.Profile profile = profileOf(base);
FleetConfig.Profile renamed = profile.withProfile("renamed-profile-guard");
List<String> dropped = new ArrayList<>();
int checked = 0;
for (RecordComponent rc : COMPONENTS) {
String name = rc.getName();
if (EXCLUDED_FROM_SURVIVAL_CHECK.contains(name)) {
continue;
}
checked++;
Object expected = "profile".equals(name) ? "renamed-profile-guard" : base.get(name);
Object actual;
try {
actual = rc.getAccessor().invoke(renamed);
} catch (ReflectiveOperationException e) {
throw new RuntimeException("failed to read FleetConfig.Profile." + name + "()", e);
}
if (!Objects.equals(expected, actual)) {
dropped.add(String.format(Locale.ROOT,
"%s: withProfile() was expected to carry (%s) for '%s' but returned %s — a "
+ "component silently dropped by withProfile(), the shape of the "
+ "defect this test exists to catch (its final \"return new "
+ "Profile(...)\" call binding to a back-compat constructor instead "
+ "of the true canonical one)",
name, expected, name, actual));
}
}
System.out.printf(Locale.ROOT,
"FleetConfig.Profile.withProfile() component-survival coverage — %d components, %d "
+ "checked, %d excluded, %d survived%n",
COMPONENTS.length, checked, EXCLUDED_FROM_SURVIVAL_CHECK.size(), checked - dropped.size());
assertEquals(List.of(), dropped,
"withProfile() silently dropped these components: " + dropped);
}
}
@@ -2629,4 +2629,58 @@ class FleetConfigTest {
"with no pool to choose from, every configured profile is a candidate and the "
+ "first one wins");
}
// ── idle-sleep guard: default-on config block ───────────────────────────────────────────────
@Test
void idleSleepGuardIsOnByDefaultWhenTheBlockIsEntirelyAbsent(@TempDir Path dir) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, """
bind:
host: 127.0.0.1
port: 8080
""");
FleetConfig cfg = FleetConfig.load(f);
assertNull(cfg.idleSleepGuard(), "an absent block parses to null, unlike most other blocks here");
// The block itself is absent, but the FEATURE stays on: Fleetd treats a null block the
// same as enabled: true (see FleetConfig.idleSleepGuard's javadoc) — this test only pins
// the parse result, the on-by-default behaviour is Fleetd's own null check.
}
@Test
void idleSleepGuardExplicitlyEnabledIsOn(@TempDir Path dir) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, """
idleSleepGuard:
enabled: true
""");
FleetConfig cfg = FleetConfig.load(f);
assertTrue(cfg.idleSleepGuard().isEnabled());
}
@Test
void idleSleepGuardExplicitlyDisabledIsOff(@TempDir Path dir) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, """
idleSleepGuard:
enabled: false
""");
FleetConfig cfg = FleetConfig.load(f);
assertFalse(cfg.idleSleepGuard().isEnabled());
}
@Test
void idleSleepGuardBlockPresentButEmptyDefaultsToEnabled(@TempDir Path dir) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, """
idleSleepGuard: {}
""");
FleetConfig cfg = FleetConfig.load(f);
assertTrue(cfg.idleSleepGuard().isEnabled(),
"unlike ConfigReload/Health, this block defaults to ON even when present but empty");
}
}
@@ -46,7 +46,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
* comments document that it only ever REPLACES a component when the incoming value is {@code null}
* (or blank, for {@code placement}) — {@code broker}/{@code primary}/{@code leadHeartbeat}/
* {@code configReload}/{@code coordinator}/{@code worktreeGroup}/{@code memberLoginShell}/
* {@code memberSkills} are left as-is unconditionally, and {@code bind}/{@code guard}/{@code lifecycle}/{@code auth}/
* {@code memberSkills}/{@code idleSleepGuard} are left as-is unconditionally, and {@code bind}/{@code guard}/{@code lifecycle}/{@code auth}/
* {@code fleet}/{@code quarantineCooldownSeconds}/{@code memberCredentials}/{@code placement} are
* replaced only on null/blank input. A value that is never null or blank going in must therefore
* never change coming out, for every current component. No exclusion is needed today.
@@ -96,6 +96,7 @@ class FleetConfigWithDefaultsPreservesEveryComponentTest {
v.put("worktreeGroup", "group-guard");
v.put("memberLoginShell", "/bin/zsh");
v.put("memberSkills", "/skills/guard");
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(true));
assertNamesMatchComponents(v);
return v;
}
@@ -4,6 +4,7 @@ import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.mcp.PrimaryRegistry;
import dev.ltms.fleet.metrics.FleetMetrics;
import dev.ltms.fleet.metrics.Metrics;
@@ -44,6 +45,7 @@ class ReplyPushLoopTest {
private static final String WORKER = "term_worker";
private static final String WORKER2 = "term_worker2";
private static final String OTHER_PRIMARY = "term_other_primary";
private static final String DEAD_LEAD = "term_dead_lead";
private static final ObjectMapper MAPPER = new ObjectMapper();
private PrimaryRegistry registry;
@@ -207,6 +209,101 @@ class ReplyPushLoopTest {
"exactly " + cap + " agent.prompt calls (cap=" + cap + ")");
}
// --- fleetd #368: a lead's delegation binding must not outlive the lead ---------------------
/**
* The bug: {@code PrimaryRegistry.forgetDelegation} is wired to a worker's release, never to
* the delegating lead's own disappearance, so a lead that closed, crashed, or was relaunched
* leaves {@code leadByTarget} pointing at a terminal herdr no longer knows. Before the fix,
* {@code onReplyQueued} took that stale, non-null entry at face value — {@code nudgeTargetFor}
* only ever falls back to the pinned primary when the map holds nothing for the target — so
* the nudge's only schedule ran against the dead terminal forever and the live primary never
* heard about the reply through this path.
*
* <p>This drives {@link ReplyPushLoop#onReplyQueued(String)} itself (not {@code PrimaryRegistry}
* directly), because the registry lookup was never the defect — the caller trusting it without
* checking liveness was. A test that only asserted on {@code PrimaryRegistry.nudgeTargetFor}
* would pass whether or not {@code ReplyPushLoop} ever adopted the fix.
*/
@Test
void aStaleLeadBindingFallsBackToTheLiveLeadInsteadOfNudgingADeadTerminal() throws Exception {
// PRIMARY is the single known (pinned) lead — set up in @BeforeEach via `registry`.
// DEAD_LEAD is a second lead that once delegated to WORKER and is now gone: herdr reports
// agent_not_found for it, exactly as it would for a closed/crashed/relaunched terminal.
registry.recordDelegation(WORKER, DEAD_LEAD);
var rec = new DeadLeadHerdrClient(DEAD_LEAD);
agents = new AgentControl(rec);
inbox.publish(WORKER, "m1", "hello");
loop(1, 50).onReplyQueued(WORKER);
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS),
"the nudge should still reach the live primary, not silently vanish with the dead lead");
assertEquals(List.of(PRIMARY), rec.promptTargets(),
"the nudge must be sent to the live primary, never to the dead lead's terminal");
assertEquals(PRIMARY, registry.nudgeTargetFor(WORKER).orElseThrow(),
"the stale binding must be forgotten (self-healed) once found dead, exactly like "
+ "AgentControl.paneByTerminal already does on agent_not_found");
}
/**
* Same dead binding, but with no pinned primary to fall back to (the multi-lead, no-fallback
* case {@code PrimaryRegistry.nudgeTargetFor}'s own javadoc already covers): the loop must
* never nudge the dead terminal, and must not spin — no schedule starts at all once the stale
* binding resolves to empty, same as if the map had never held an entry for this target.
*/
@Test
void aStaleLeadBindingWithNoFallbackNeverNudgesTheDeadTerminal() throws Exception {
var unpinned = new PrimaryRegistry(null);
unpinned.recordDelegation(WORKER, DEAD_LEAD);
var rec = new DeadLeadHerdrClient(DEAD_LEAD);
agents = new AgentControl(rec);
inbox.publish(WORKER, "m1", "hello");
var loop = new ReplyPushLoop(unpinned, agents, inbox, scheduler, 1, 50);
loop.onReplyQueued(WORKER);
Thread.sleep(200);
assertEquals(0, rec.sendCount(), "no lead is live to nudge, so nothing should ever be sent");
assertTrue(unpinned.nudgeTargetFor(WORKER).isEmpty(),
"the stale binding must be forgotten even when there is no fallback to hand back");
}
/**
* fleetd #368 review, must-fix: the first version of {@code isLive} treated <em>any</em>
* {@code RuntimeException} from the liveness probe as "the lead is gone" — indistinguishable
* from a transient herdr hiccup (a socket blip, a decode error) on a lead that is actually
* still live. The consequence of that misdiagnosis is destructive and permanent
* ({@code forgetDelegation}), which is the exact #359 mistake repeated two days later: a single
* bad reading must never destroy a live binding. This pins the narrower rule — only an
* affirmative {@code agent_not_found} may forget a binding; a merely inconclusive failure must
* leave the binding alone, and the lead must still be nudged once the probe recovers.
*/
@Test
void aTransientLivenessFailureMustNotForgetABindingToAStillLiveLead() throws Exception {
// OTHER_PRIMARY is delegated to and genuinely live — its FIRST agent.get call fails with a
// transient, non-agent_not_found HerdrException (a transport-level failure, code null,
// exactly what a socket blip looks like), then succeeds on every call after.
registry.recordDelegation(WORKER, OTHER_PRIMARY);
var rec = new FlakyThenLiveHerdrClient(OTHER_PRIMARY);
agents = new AgentControl(rec);
inbox.publish(WORKER, "m1", "hello");
loop(2, 50).onReplyQueued(WORKER);
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS),
"the nudge must still reach the live lead once the transient failure clears");
assertEquals(List.of(OTHER_PRIMARY), rec.promptTargets(),
"the nudge must go to the lead that was only transiently unreachable, not the "
+ "unrelated pinned primary");
assertEquals(OTHER_PRIMARY, registry.nudgeTargetFor(WORKER).orElseThrow(),
"a merely transient failure must not forget the binding to a lead that is actually "
+ "still live");
}
// --- nudge format --------------------------------------------------------------------------
@Test
@@ -1098,4 +1195,101 @@ class ReplyPushLoopTest {
public void close() {
}
}
/**
* Fake herdr client for fleetd #368: {@code deadTarget} is a terminal herdr genuinely no
* longer knows about — {@code agent.get} fails with {@code agent_not_found} exactly as
* {@code AgentControl.agentCall} expects for a real dead/closed pane (see its javadoc). Every
* other target reports {@code idle} (injectable). Records the {@code target} named by every
* {@code agent.prompt} call, so a test can prove which terminal actually got nudged.
*/
private static final class DeadLeadHerdrClient implements HerdrClient {
private final String deadTarget;
private final List<String> promptTargets = Collections.synchronizedList(new ArrayList<>());
volatile CountDownLatch sendLatch = new CountDownLatch(1);
DeadLeadHerdrClient(String deadTarget) {
this.deadTarget = deadTarget;
}
@Override
@SuppressWarnings("unchecked")
public JsonNode call(String method, Object params) {
Map<String, Object> p = params instanceof Map ? (Map<String, Object>) params : Map.of();
if ("agent.get".equals(method)) {
String target = String.valueOf(p.get("target"));
if (deadTarget.equals(target)) {
throw new HerdrException("no such agent: " + target, "agent_not_found", null);
}
return MAPPER.createObjectNode()
.set("agent", MAPPER.createObjectNode()
.put("terminal_id", target)
.put("agent_status", "idle"));
}
if ("agent.prompt".equals(method)) {
promptTargets.add(String.valueOf(p.get("target")));
sendLatch.countDown();
}
return MAPPER.createObjectNode();
}
List<String> promptTargets() {
return List.copyOf(promptTargets);
}
long sendCount() {
return promptTargets.size();
}
@Override
public void close() {
}
}
/**
* Fake herdr client for fleetd #368 review: {@code flakyTarget}'s FIRST {@code agent.get} call
* fails with a transient, non-{@code agent_not_found} {@code HerdrException} — a transport-level
* failure (code {@code null}), exactly what a socket blip or a decode error on a perfectly live
* lead looks like — then succeeds ({@code idle}) on every call after. Used to prove a merely
* inconclusive failure must not be treated as the lead being gone.
*/
private static final class FlakyThenLiveHerdrClient implements HerdrClient {
private final String flakyTarget;
private final AtomicInteger getCalls = new AtomicInteger();
private final List<String> promptTargets = Collections.synchronizedList(new ArrayList<>());
volatile CountDownLatch sendLatch = new CountDownLatch(1);
FlakyThenLiveHerdrClient(String flakyTarget) {
this.flakyTarget = flakyTarget;
}
@Override
@SuppressWarnings("unchecked")
public JsonNode call(String method, Object params) {
Map<String, Object> p = params instanceof Map ? (Map<String, Object>) params : Map.of();
if ("agent.get".equals(method)) {
String target = String.valueOf(p.get("target"));
if (flakyTarget.equals(target) && getCalls.getAndIncrement() == 0) {
throw new HerdrException("herdr socket read timed out"); // transport failure, code == null
}
return MAPPER.createObjectNode()
.set("agent", MAPPER.createObjectNode()
.put("terminal_id", target)
.put("agent_status", "idle"));
}
if ("agent.prompt".equals(method)) {
promptTargets.add(String.valueOf(p.get("target")));
sendLatch.countDown();
}
return MAPPER.createObjectNode();
}
List<String> promptTargets() {
return List.copyOf(promptTargets);
}
@Override
public void close() {
}
}
}
@@ -0,0 +1,54 @@
package dev.ltms.fleet.power;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* Platform-detection unit tests for {@link CaffeinateSleepAssertionMechanism}.
*
* <p>This deliberately never calls {@link CaffeinateSleepAssertionMechanism#acquire()} itself —
* doing so on a real macOS machine would actually start a live {@code caffeinate} child and hold
* a real idle-sleep assertion, which the ticket this class exists for explicitly forbids testing
* with. Instead this exercises the pure {@code isSupportedPlatform(String)} predicate that
* {@code acquire()} consults before ever touching {@link ProcessBuilder} — so it proves the
* platform check itself is correct on any CI OS, but it does <strong>not</strong> prove that a
* real {@code caffeinate -i} spawn succeeds or that its child is torn down correctly; that half is
* exercised indirectly by {@link IdleSleepGuardTest} against a {@link FakeSleepAssertionMechanism}
* instead, which is the seam invariant 2/3 in the ticket call for.
*/
class CaffeinateSleepAssertionMechanismTest {
@Test
void macOsNamesAreSupported() {
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Mac OS X"));
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("macOS"));
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("MAC OS X"));
}
@Test
void nonMacNamesAreNotSupported() {
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Linux"));
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Windows 11"));
}
@Test
void nullOsNameIsNotSupported() {
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform(null));
}
/**
* The overload {@code isSupportedPlatform()} (no args) reads the JVM's real {@code os.name} —
* proves the wiring is live, without asserting a specific answer (this suite itself must pass
* on both macOS and Linux CI).
*/
@Test
void noArgOverloadReadsRealSystemProperty() {
boolean expected = CaffeinateSleepAssertionMechanism
.isSupportedPlatform(System.getProperty("os.name"));
boolean actual = CaffeinateSleepAssertionMechanism.isSupportedPlatform();
assertEquals(expected, actual);
}
}
@@ -0,0 +1,56 @@
package dev.ltms.fleet.power;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
/**
* Recording fake {@link SleepAssertionMechanism} — the seam behind the real OS effect (a live
* {@code caffeinate} child process). No test in this package ever spawns that real process; every
* assertion here is against this fake's own call log instead.
*
* <p>Each acquired {@link FakeAssertion} records its own {@code close()} calls, and every
* acquired instance is kept in {@link #acquired} so a test can inspect all of them, including
* ones {@link IdleSleepGuard} has already released.
*/
final class FakeSleepAssertionMechanism implements SleepAssertionMechanism {
/** Every {@link FakeAssertion} this mechanism has ever handed out, in order. */
final CopyOnWriteArrayList<FakeAssertion> acquired = new CopyOnWriteArrayList<>();
private final AtomicInteger acquireCalls = new AtomicInteger();
private volatile boolean unavailable = false;
/** Make the next (and every subsequent) {@link #acquire()} return {@code null}, like a missing tool. */
void makeUnavailable() {
unavailable = true;
}
int acquireCallCount() {
return acquireCalls.get();
}
@Override
public SleepAssertion acquire() {
acquireCalls.incrementAndGet();
if (unavailable) {
return null;
}
FakeAssertion a = new FakeAssertion();
acquired.add(a);
return a;
}
/** A held fake assertion; records how many times {@code close()} was actually called. */
static final class FakeAssertion implements SleepAssertion {
private final AtomicInteger closeCalls = new AtomicInteger();
int closeCallCount() {
return closeCalls.get();
}
@Override
public void close() {
closeCalls.incrementAndGet();
}
}
}
@@ -0,0 +1,118 @@
package dev.ltms.fleet.power;
import org.junit.jupiter.api.Test;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* {@link IdleSleepGuard} against a {@link FakeSleepAssertionMechanism} — the seam that stands in
* for a real {@code caffeinate} child process. No test in this class ever spawns a real OS
* process or asserts against real idle sleep; every assertion is against the fake's call log
* (how many times {@code acquire()}/{@code close()} were actually called). That proves the
* <em>orchestration</em> — when the guard decides to hold or release an assertion, and that it
* never throws — but it does <strong>not</strong> prove that {@code caffeinate -i} itself
* actually stops macOS from idle-sleeping; that half is outside what a unit test can safely
* exercise (see {@link CaffeinateSleepAssertionMechanismTest}'s class doc).
*/
class IdleSleepGuardTest {
@Test
void acquiresOnZeroToOneAndReleasesOnOneToZero() {
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
AtomicInteger liveCount = new AtomicInteger(0);
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
assertFalse(guard.isHeld(), "nothing held before any member is live");
liveCount.set(1);
guard.recheck();
assertTrue(guard.isHeld(), "an assertion must be held once a member is live");
assertEquals(1, mechanism.acquired.size());
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
liveCount.set(0);
guard.recheck();
assertFalse(guard.isHeld(), "the assertion must be released once the last member goes");
assertEquals(1, mechanism.acquired.get(0).closeCallCount(), "the SAME held assertion must be closed");
}
@Test
void steadyLiveCountDoesNotReacquireOrRerelease() {
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
AtomicInteger liveCount = new AtomicInteger(2);
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
guard.recheck(); // 0 -> 2 crossing: acquires
guard.recheck(); // still 2: must be a no-op
guard.recheck(); // still 2: must be a no-op
assertEquals(1, mechanism.acquireCallCount(), "only the crossing touches the mechanism");
liveCount.set(1); // 2 -> 1: still > 0, still a no-op
guard.recheck();
assertTrue(guard.isHeld());
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
assertEquals(1, mechanism.acquireCallCount());
}
/**
* Invariant 2: a missing/unavailable mechanism must never throw, and the guard must simply
* hold nothing. {@link FakeSleepAssertionMechanism#makeUnavailable()} makes {@code acquire()}
* return {@code null}, exactly like {@link CaffeinateSleepAssertionMechanism} does off macOS
* or when the {@code caffeinate} binary is missing.
*/
@Test
void unavailableMechanismNeverThrowsAndHoldsNothing() {
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
mechanism.makeUnavailable();
AtomicInteger liveCount = new AtomicInteger(1);
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
guard.recheck(); // must not throw
assertFalse(guard.isHeld(), "acquire() returned null, so nothing is held");
assertEquals(1, mechanism.acquireCallCount());
// still must not throw or leak on release, even though nothing was ever actually held
liveCount.set(0);
guard.recheck();
assertFalse(guard.isHeld());
guard.close(); // teardown with nothing held must also be a safe no-op
}
/**
* Invariant 3 (teardown). This is the test the mutation testing step removes the production
* release call to fail: with {@code releaseHeldLocked()} not invoked from {@link
* IdleSleepGuard#close()}, the held fake assertion's {@code close()} would never be called and
* this assertion would fail.
*/
@Test
void closeReleasesAHeldAssertionEvenWithoutAZeroCrossing() {
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
AtomicInteger liveCount = new AtomicInteger(1);
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
guard.recheck();
assertTrue(guard.isHeld());
guard.close();
assertFalse(guard.isHeld(), "close() must release whatever is held, independent of live count");
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
}
@Test
void closeIsIdempotent() {
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
AtomicInteger liveCount = new AtomicInteger(1);
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
guard.recheck();
guard.close();
guard.close(); // must not throw, must not double-release
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
}
}
@@ -0,0 +1,72 @@
package dev.ltms.fleet.power;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import dev.ltms.fleet.session.MemberSession;
import dev.ltms.fleet.session.SessionManager;
import org.junit.jupiter.api.Test;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* Proves the wiring {@code Fleetd.main} actually performs — {@code
* sessions.onAcquire(_ -> guard.recheck())} / {@code sessions.onRelease(_ -> guard.recheck())} —
* not just {@link IdleSleepGuard}'s own orchestration logic in isolation
* ({@link IdleSleepGuardTest} already covers that in isolation, which on its own would not catch
* a wiring gap — e.g. an {@code onAcquire} call typo'd to a no-op lambda, or the listener wired to
* the wrong SessionManager instance — see fleetd's own "a test on the seam does not prove the
* caller" lesson). This test builds a real {@link SessionManager} exactly as
* {@code SessionManagerTest} does (a {@link FakeHerdr}-backed {@link ClaudeCodeLauncher}, no live
* herdr process), wires it to an {@link IdleSleepGuard} the same two lines {@code Fleetd.main}
* uses, and drives real {@link SessionManager#acquire} / {@link SessionManager#release} calls.
*/
class IdleSleepGuardWiringTest {
private SessionManager sessionManager(FakeHerdr herdr) {
FleetConfig.Profile cfg = new FleetConfig.Profile(
"ltms-local", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN",
List.of("ccs", "ltms-local"), "tab", "fleetd-workers",
"worker: {profile} #{n}", null, null, null);
ClaudeCodeLauncher workers = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
return new SessionManager(workers);
}
@Test
void acquiringAndReleasingRealSessionsDrivesTheGuardThroughTheSameWiringFleetdUses() {
FakeHerdr herdr = new FakeHerdr();
SessionManager sessions = sessionManager(herdr);
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
IdleSleepGuard guard = new IdleSleepGuard(mechanism, sessions::size);
// The exact two lines Fleetd.main wires up.
sessions.onAcquire(_ -> guard.recheck());
sessions.onRelease(_ -> guard.recheck());
assertFalse(guard.isHeld(), "no member yet: nothing held");
MemberSession a = sessions.acquire("ltms-local", "/a", "/caller", "ownerA");
assertTrue(guard.isHeld(), "0 -> 1: the first live member must arm the guard");
MemberSession b = sessions.acquire("ltms-local", "/b", "/caller", "ownerB");
assertEquals(1, mechanism.acquireCallCount(), "2nd member: still just 1 live-to-2 step, no new acquire");
sessions.release(a.paneId());
assertTrue(guard.isHeld(), "one member still live: the guard must stay armed");
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
sessions.release(b.paneId());
assertFalse(guard.isHeld(), "1 -> 0: the last member releasing must disarm the guard");
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
}
}
@@ -54,6 +54,56 @@ class GitWorktreesTest {
/** A non-empty autoenv file — the form that would prompt for authorization in a worktree. */
private static final String AUTOENV_WITH_DIRECTIVE = "export HELLO=world\n";
/**
* fleetd #369. A throwaway directory that lives for the whole class (JUnit 5.4+ supports a
* static {@code @TempDir} field, created once and removed once every test in this class has
* run) — backing every raw {@code git} subprocess's {@code XDG_CONFIG_HOME} below. It only
* ever needs to exist and be guaranteed free of a {@code git/ignore} file; nothing writes
* inside it.
*/
@TempDir
private static Path CLASS_TMP;
/**
* fleetd #369 — the leak measured: {@code XDG_CONFIG_HOME=<dir with a `*` git/ignore> mvn test
* -Dtest=GitWorktreesTest} failed 56 of 59 tests on an unpatched checkout, because {@link
* #gitOutput} set {@code GIT_CONFIG_GLOBAL}/{@code GIT_CONFIG_SYSTEM}/{@code
* GIT_TERMINAL_PROMPT} but not {@code XDG_CONFIG_HOME}, and {@link #status}/{@link
* #fullStatus} (plus every other raw {@code git} subprocess this class started) set NOTHING at
* all — inheriting the JVM's whole real environment, including the operator's real {@code
* ~/.gitconfig} and real default excludes file ({@code $XDG_CONFIG_HOME/git/ignore} or {@code
* $HOME/.config/git/ignore}, applied by git with no {@code core.excludesFile} configured at
* all — see {@code gitignore(5)}). {@code GIT_CONFIG_GLOBAL=/dev/null} does not stop that
* default from applying; only setting {@code XDG_CONFIG_HOME} to a directory that provably
* carries no {@code git/ignore} does.
*
* <p>This is the same isolation {@link #hermeticGitEnv} already gives {@link
* #seedingGitWorktrees}'s production {@link GitWorktrees} instances (fleetd #362 review fix,
* finding 2), reused here for every subprocess the TEST ITSELF starts to drive and inspect
* those fixture repos.
*/
private static Map<String, String> hermeticEnv() {
return hermeticGitEnv(CLASS_TMP);
}
/**
* The one seam every git subprocess in this class is built through — see criterion 4's
* self-check, {@link #everyGitSubprocessGoesThroughTheHermeticFactory}, which fails the moment
* a future helper builds its own {@code git} subprocess directly instead of calling this, so
* the omission that caused fleetd #369 gets caught by name rather than rediscovered by a
* poisoned machine. The one deliberate exception is {@link
* #worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper}, which needs a
* non-hermetic, test-controlled global config to prove the credential helper ignores it — see
* the comment on that test.
*/
private static ProcessBuilder gitProcessBuilder(Path cwd, String... args) {
List<String> cmd = new java.util.ArrayList<>(List.of("git"));
cmd.addAll(List.of(args));
ProcessBuilder pb = new ProcessBuilder(cmd).directory(cwd.toFile()).redirectErrorStream(true);
pb.environment().putAll(hermeticEnv());
return pb;
}
private static Path initRepo(Path dir) throws Exception {
Files.createDirectories(dir);
git(dir, "init", "-q", "-b", "main");
@@ -71,23 +121,16 @@ class GitWorktreesTest {
}
private static String gitOutput(Path cwd, String... args) throws Exception {
List<String> cmd = new java.util.ArrayList<>(List.of("git"));
cmd.addAll(List.of(args));
ProcessBuilder pb = new ProcessBuilder(cmd).directory(cwd.toFile()).redirectErrorStream(true);
pb.environment().put("GIT_CONFIG_GLOBAL", "/dev/null");
pb.environment().put("GIT_CONFIG_SYSTEM", "/dev/null");
pb.environment().put("GIT_TERMINAL_PROMPT", "0");
Process p = pb.start();
Process p = gitProcessBuilder(cwd, args).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git timed out: " + String.join(" ", cmd));
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git timed out: git " + String.join(" ", args));
assertEquals(0, p.exitValue(), "git " + String.join(" ", args) + " failed:\n" + out);
return out;
}
/** Pending changes to {@code file} in {@code cwd}, empty when git considers it unmodified. */
private static String status(Path cwd, String file) throws Exception {
Process p = new ProcessBuilder("git", "status", "--porcelain", "--", file)
.directory(cwd.toFile()).redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "status", "--porcelain", "--", file).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
return out;
@@ -95,16 +138,14 @@ class GitWorktreesTest {
/** Every pending change in {@code cwd} — the whole-tree porcelain status, unlike {@link #status}. */
private static String fullStatus(Path cwd) throws Exception {
Process p = new ProcessBuilder("git", "status", "--porcelain")
.directory(cwd.toFile()).redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "status", "--porcelain").start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
return out;
}
private static String revParse(Path cwd, String ref) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "rev-parse", ref)
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "rev-parse", ref).start();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git rev-parse timed out");
assertEquals(0, p.exitValue(), "git rev-parse " + ref + " failed:\n" + out);
@@ -113,8 +154,7 @@ class GitWorktreesTest {
/** The recursive file list of a commit's tree — used to check what a snapshot actually committed. */
private static String lsTree(Path cwd, String ref) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "ls-tree", "-r", "--name-only", ref)
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "ls-tree", "-r", "--name-only", ref).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git ls-tree timed out");
assertEquals(0, p.exitValue(), "git ls-tree " + ref + " failed:\n" + out);
@@ -125,8 +165,7 @@ class GitWorktreesTest {
* snapshot's tree changed relative to its parent, the same shape {@code git status --porcelain}
* reports for the worktree it was taken from. */
private static Set<String> diffNameOnly(Path cwd, String from, String to) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "diff", "--name-only", from, to)
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "diff", "--name-only", from, to).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git diff timed out");
assertEquals(0, p.exitValue(), "git diff " + from + ".." + to + " failed:\n" + out);
@@ -152,8 +191,7 @@ class GitWorktreesTest {
}
private static String forEachRef(Path cwd, String pattern) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "for-each-ref", pattern)
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "for-each-ref", pattern).start();
String out = new String(p.getInputStream().readAllBytes());
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git for-each-ref timed out");
assertEquals(0, p.exitValue(), "git for-each-ref " + pattern + " failed:\n" + out);
@@ -162,8 +200,7 @@ class GitWorktreesTest {
/** Write {@code content} as a blob into the object database; returns its sha. */
private static String blobOf(Path cwd, String content) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "hash-object", "-w", "--stdin")
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "hash-object", "-w", "--stdin").start();
p.getOutputStream().write(content.getBytes(StandardCharsets.UTF_8));
p.getOutputStream().close();
String out = new String(p.getInputStream().readAllBytes()).trim();
@@ -174,8 +211,7 @@ class GitWorktreesTest {
/** Build a single-file tree object from {@code blob}; returns the tree's sha. */
private static String treeOf(Path cwd, String path, String blob) throws Exception {
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "mktree")
.redirectErrorStream(true).start();
Process p = gitProcessBuilder(cwd, "mktree").start();
p.getOutputStream().write(("100644 blob " + blob + "\t" + path + "\n").getBytes(StandardCharsets.UTF_8));
p.getOutputStream().close();
String out = new String(p.getInputStream().readAllBytes()).trim();
@@ -187,10 +223,9 @@ class GitWorktreesTest {
/** {@code git commit-tree} rooted at {@code tree} with a chosen committer date; returns the sha. */
private static String commitTree(Path cwd, String tree, String parent, String committerDate,
String message) throws Exception {
ProcessBuilder pb = new ProcessBuilder("git", "-C", cwd.toString(), "commit-tree",
tree, "-p", parent, "-m", message);
ProcessBuilder pb = gitProcessBuilder(cwd, "commit-tree", tree, "-p", parent, "-m", message);
pb.environment().put("GIT_COMMITTER_DATE", committerDate);
Process p = pb.redirectErrorStream(true).start();
Process p = pb.start();
String out = new String(p.getInputStream().readAllBytes()).trim();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git commit-tree timed out");
assertEquals(0, p.exitValue(), "git commit-tree failed:\n" + out);
@@ -263,6 +298,11 @@ class GitWorktreesTest {
helper = !f() { printf 'username=%s\\npassword=%s\\n\\n' operator operator-secret; }; f
""");
// fleetd #369: the one deliberate exception to gitProcessBuilder. This test's whole point is
// that git must resolve `globalConfig` (a synthetic "operator's global config", never the
// real machine's) and then IGNORE it — so it cannot use the shared hermetic env, which would
// point GIT_CONFIG_GLOBAL at /dev/null and defeat the very thing under test. It never runs
// `git status`, so it does not need XDG_CONFIG_HOME isolation either.
ProcessBuilder pb = new ProcessBuilder("git", "credential", "fill")
.directory(Path.of(wt).toFile()).redirectErrorStream(true);
pb.environment().put("GIT_CONFIG_GLOBAL", globalConfig.toString());
@@ -329,20 +369,16 @@ class GitWorktreesTest {
Path worktree = Path.of(wt);
assertEquals("https://git.ltms.dev/akb/kb.git",
gitOutput(worktree, "remote", "get-url", "origin").trim());
assertEquals(1, exitCode("git", "-C", wt, "config", "--worktree", "--get-regexp", "^url\\."),
assertEquals(1, gitExitCode(worktree, "config", "--worktree", "--get-regexp", "^url\\."),
"no url.*.insteadOf rewrite should be added for an already-HTTPS origin");
}
/** Test-local exit-code probe, mirroring {@link GitWorktrees#exitCode} for an assertion the
* production class does not expose. */
private static int exitCode(String... command) throws Exception {
ProcessBuilder pb = new ProcessBuilder(command).redirectErrorStream(true);
pb.environment().put("GIT_CONFIG_GLOBAL", "/dev/null");
pb.environment().put("GIT_CONFIG_SYSTEM", "/dev/null");
pb.environment().put("GIT_TERMINAL_PROMPT", "0");
Process p = pb.start();
private static int gitExitCode(Path cwd, String... args) throws Exception {
Process p = gitProcessBuilder(cwd, args).start();
p.getInputStream().readAllBytes();
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "command timed out: " + String.join(" ", command));
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "command timed out: git " + String.join(" ", args));
return p.exitValue();
}
@@ -1493,7 +1529,15 @@ class GitWorktreesTest {
* an empty, machine-independent {@code XDG_CONFIG_HOME} (so the fallback resolves to a file that
* provably does not exist) plus the same {@code GIT_CONFIG_GLOBAL}/{@code GIT_CONFIG_SYSTEM}/
* {@code GIT_TERMINAL_PROMPT} isolation the {@link #git}/{@link #gitOutput} helpers already use
* for repo setup — so no test in this class can reach the real machine's home directory.
* for repo setup.
*
* <p>Scope, measured on the fleetd #369 merge and narrower than an earlier version of this
* comment claimed: this protects the 5 {@link #seedingGitWorktrees} sites plus — through
* {@link #gitProcessBuilder} — every {@code git} subprocess the TEST itself starts. It does
* NOT cover the other 53 {@code new GitWorktrees(...)} constructions in this file, which pass
* no env override, so a production instance built that way still inherits the JVM's real
* environment. Stripping this override from {@code seedingGitWorktrees} leaves the class green
* both with and without the poison command above, so that half is currently unpinned.
*/
private static Map<String, String> hermeticGitEnv(Path tmp) {
return Map.of(
@@ -1739,4 +1783,77 @@ class GitWorktreesTest {
"the XDG default excludesFile pattern ('xdg-fallback-marker') must still apply "
+ "after skill seeding ran — got:\n" + porcelain);
}
/**
* fleetd #369, acceptance criterion 4 — make the fix hard to undo by accident. Every git
* subprocess this class starts is required to go through {@link #gitProcessBuilder}, the one
* place {@link #hermeticEnv} is applied; a helper built directly, the way the original leak in
* {@link #status}/{@link #fullStatus} was, is now a source-level fact this test can catch by
* name instead of a machine-dependent failure someone has to rediscover.
*
* <p>This counts a literal marker in this very file's own source, split into three
* concatenated pieces below so the count is not thrown off by this method's own text — a
* plain, unsplit occurrence of the marker anywhere in this file (a helper's construction, or a
* comment that happens to spell it out contiguously) adds to the count the same way. Today
* there are exactly two: the factory itself, and the one documented exception in {@link
* #worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper}, which needs a
* non-hermetic, test-controlled global config to prove the credential helper ignores it. A
* third means a new helper was added the old, leak-prone way — route it through {@link
* #gitProcessBuilder} instead, or explain the new exception here and bump this number.
*/
@Test
void everyGitSubprocessGoesThroughTheHermeticFactory() throws Exception {
Path source = Path.of("src/test/java/dev/ltms/fleet/session/GitWorktreesTest.java");
String text = Files.readString(source);
String marker = "new " + "ProcessBuilder" + "(";
int count = 0;
for (int from = text.indexOf(marker); from >= 0; from = text.indexOf(marker, from + marker.length())) {
count++;
}
assertEquals(2, count,
"expected exactly 2 direct git-subprocess constructions in this file (the "
+ "gitProcessBuilder factory itself, plus the one documented exception in "
+ "worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper) — a "
+ "different count means a helper now bypasses the hermetic factory; route "
+ "it through gitProcessBuilder or document the new exception here");
}
/**
* fleetd #369 review round 2. {@link #everyGitSubprocessGoesThroughTheHermeticFactory} counts
* call sites, not behaviour — it catches a NEW helper built the old, leak-prone way, but it
* cannot catch {@link #gitProcessBuilder} itself being gutted: deleting {@code
* pb.environment().putAll(hermeticEnv())} from inside the factory leaves every call site
* unchanged, the count stays 2, and the whole unpoisoned suite stays green — the exact leak
* this ticket fixed would come back silently, with nothing but a human remembering to re-run
* the poison command to catch it. This test instead inspects what the factory actually hands
* to {@link ProcessBuilder#start()}, so it fails the moment the hermetic environment stops
* being applied, on any machine, with no poison needed.
*
* <p>The property under test: every git subprocess this class starts must run with an
* environment that cannot see the operator's real git configuration. A call-site count is a
* proxy for that; this is the thing itself.
*/
@Test
void gitProcessBuilderCarriesTheFullHermeticEnvironment(@TempDir Path tmp) {
Map<String, String> env = gitProcessBuilder(tmp, "status", "--porcelain").environment();
assertEquals("/dev/null", env.get("GIT_CONFIG_GLOBAL"),
"GIT_CONFIG_GLOBAL must be neutralized, or the operator's real ~/.gitconfig applies");
assertEquals("/dev/null", env.get("GIT_CONFIG_SYSTEM"),
"GIT_CONFIG_SYSTEM must be neutralized, or the machine's real /etc/gitconfig applies");
assertEquals("0", env.get("GIT_TERMINAL_PROMPT"),
"GIT_TERMINAL_PROMPT must be disabled, or a credential prompt can hang the subprocess");
String xdg = env.get("XDG_CONFIG_HOME");
assertNotNull(xdg,
"XDG_CONFIG_HOME must be set — left unset, git falls back to the operator's real "
+ "$HOME/.config/git/ignore (gitignore(5)), exactly fleetd #369's leak");
assertFalse(xdg.isBlank(), "XDG_CONFIG_HOME must not be blank — blank behaves like unset");
assertTrue(Path.of(xdg).startsWith(CLASS_TMP),
"XDG_CONFIG_HOME must point inside this test class's own throwaway directory, "
+ "never the operator's real one or the JVM's inherited value — got: " + xdg);
assertFalse(Files.exists(Path.of(xdg, "git", "ignore")),
"the resolved XDG default excludes file must provably not exist, or its contents "
+ "would silently apply to every git status this test class runs");
}
}
@@ -0,0 +1,190 @@
package dev.ltms.fleet.session;
import dev.ltms.fleet.peer.CharterReceipt;
import dev.ltms.fleet.peer.MemberRole;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Constructor;
import java.lang.reflect.RecordComponent;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.function.Function;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* Fleetd #358, the same "defect factory" #357 guarded on {@code FleetConfig.withDefaults()}
* (see {@code FleetConfigWithDefaultsPreservesEveryComponentTest}), reproduced here on
* {@link MemberSession} — the worse of the two sibling cases named in #358, because this record
* has FIVE independent rebuild sites instead of one: {@link MemberSession#withState},
* {@link MemberSession#withActivity}, {@link MemberSession#bumpTurn},
* {@link MemberSession#withAgentSessionId} and {@link MemberSession#withFailureReason} each end in
* their own literal {@code new MemberSession(...)} call. Add a 16th component and add the
* established back-compat constructor at the old (15-arg) arity, and every one of those five
* literal calls becomes a legal match for that new overload — silently dropping the new component,
* independently, on whichever of the five paths a missed update leaves behind. That is harder to
* spot than #357's single call site: the field would survive through some transitions and vanish
* through others.
*
* <p>Each check below builds one {@link MemberSession} through the TRUE canonical constructor —
* resolved by the record's own component types via {@code getDeclaredConstructor}, never by
* argument count, so it can never itself land on a back-compat overload — with a real, distinctive,
* non-null value in every component, calls the real rebuild method under test, and asserts every
* component the method is not documented to change survives unchanged, while the component(s) it IS
* documented to change come back as the new value it was given. A component that comes back
* anything else was silently dropped or lost — the shape of the defect this test exists to catch.
*
* <p>{@link #EXCLUDED_FROM_SURVIVAL_CHECK} is kept deliberately empty and size-pinned by
* {@link #exclusionListSizeIsPinned()}, for the same reason {@code FleetConfig}'s guard pins its own
* exclusion list at zero: a checker whose escape hatch can grow to silence a failure is not a
* checker. Every one of {@link MemberSession}'s 15 current components has a real, non-null,
* non-blank value here and none is excluded.
*/
class MemberSessionRebuildPreservesEveryComponentTest {
private static final RecordComponent[] COMPONENTS = MemberSession.class.getRecordComponents();
/** Deliberately empty today; grow it only with a matching justification, and re-pin the size. */
private static final Set<String> EXCLUDED_FROM_SURVIVAL_CHECK = Set.of();
/** One real, distinctive, non-null value per component — none of the 15 is excluded. */
private static Map<String, Object> baseValues() {
Map<String, Object> v = new LinkedHashMap<>();
v.put("paneId", "pane-guard");
v.put("terminalId", "term-guard");
v.put("profile", "profile-guard");
v.put("role", MemberRole.REVIEWER);
v.put("cwd", "/wt/guard");
v.put("ownerTerminal", "owner-guard");
v.put("spawnedAtNanos", 111_111L);
v.put("lastActivityAtNanos", 222_222L);
v.put("turnCount", 7);
v.put("state", MemberSession.State.BUSY);
v.put("worktree", "/wt/guard-tree");
v.put("branch", "worker/guard-branch");
v.put("charterReceipt", new CharterReceipt(
MemberRole.DEV, "profile-guard", "fleet.charters.dev", "deadbeefguard", 42));
v.put("agentSessionId", "agent-guard");
v.put("failureReason", "reason-guard");
assertNamesMatchComponents(v);
return v;
}
/**
* Guards {@link #baseValues()} itself against drifting from the record's real shape — forgetting
* to add a new component here fails this assertion by name, rather than silently checking one
* component fewer than the record has.
*/
private static void assertNamesMatchComponents(Map<String, Object> values) {
Set<String> names = new TreeSet<>();
for (RecordComponent rc : COMPONENTS) {
names.add(rc.getName());
}
assertEquals(names, new TreeSet<>(values.keySet()),
"this test's value map has drifted from MemberSession's actual components — "
+ "update baseValues() alongside the record");
}
/**
* Builds a {@link MemberSession} through the TRUE canonical constructor — resolved by the
* record's own component types, not by argument count — so this never accidentally exercises a
* back-compat overload the way a literal {@code new MemberSession(...)} call risks doing.
*/
private static MemberSession sessionOf(Map<String, Object> values) throws ReflectiveOperationException {
Class<?>[] types = Arrays.stream(COMPONENTS).map(RecordComponent::getType).toArray(Class<?>[]::new);
Object[] args = Arrays.stream(COMPONENTS).map(rc -> values.get(rc.getName())).toArray();
Constructor<MemberSession> ctor = MemberSession.class.getDeclaredConstructor(types);
return ctor.newInstance(args);
}
@Test
void exclusionListSizeIsPinned() {
assertEquals(0, EXCLUDED_FROM_SURVIVAL_CHECK.size(),
"EXCLUDED_FROM_SURVIVAL_CHECK grew from 0 — every entry needs a justification in "
+ "this test class's javadoc AND this assertion re-pinned to the new size; a "
+ "growing exclusion list that silences failures on its own is not a guard");
}
/**
* Shared check for one rebuild site: build a base session with a real value in every component,
* call {@code rebuild}, and assert every component comes back equal to {@code expectedOverrides}
* when named there, or equal to the base value otherwise. Prints the same denominator style as
* {@code FleetConfigWithDefaultsPreservesEveryComponentTest}.
*/
private void checkRebuildSite(String siteName, Function<MemberSession, MemberSession> rebuild,
Map<String, Object> expectedOverrides) throws ReflectiveOperationException {
Map<String, Object> base = baseValues();
MemberSession session = sessionOf(base);
MemberSession result = rebuild.apply(session);
List<String> dropped = new ArrayList<>();
int checked = 0;
for (RecordComponent rc : COMPONENTS) {
String name = rc.getName();
if (EXCLUDED_FROM_SURVIVAL_CHECK.contains(name)) {
continue;
}
checked++;
Object expected = expectedOverrides.containsKey(name) ? expectedOverrides.get(name) : base.get(name);
Object actual;
try {
actual = rc.getAccessor().invoke(result);
} catch (ReflectiveOperationException e) {
throw new RuntimeException("failed to read MemberSession." + name + "()", e);
}
if (!Objects.equals(expected, actual)) {
dropped.add(String.format(Locale.ROOT,
"%s: %s() was expected to carry (%s) for '%s' but returned %s — a component "
+ "silently dropped by %s(), the shape of the defect this test exists "
+ "to catch (its final \"return new MemberSession(...)\" call binding "
+ "to a back-compat constructor instead of the true canonical one)",
name, siteName, expected, name, actual, siteName));
}
}
System.out.printf(Locale.ROOT,
"MemberSession.%s() component-survival coverage — %d components, %d checked, %d "
+ "excluded, %d survived%n",
siteName, COMPONENTS.length, checked, EXCLUDED_FROM_SURVIVAL_CHECK.size(),
checked - dropped.size());
assertEquals(List.of(), dropped,
siteName + "() silently dropped these components: " + dropped);
}
@Test
void withStatePreservesEveryOtherComponent() throws ReflectiveOperationException {
checkRebuildSite("withState", s -> s.withState(MemberSession.State.FAILED),
Map.of("state", MemberSession.State.FAILED));
}
@Test
void withActivityPreservesEveryOtherComponent() throws ReflectiveOperationException {
checkRebuildSite("withActivity", s -> s.withActivity(999_999L),
Map.of("lastActivityAtNanos", 999_999L));
}
@Test
void bumpTurnPreservesEveryOtherComponent() throws ReflectiveOperationException {
checkRebuildSite("bumpTurn", s -> s.bumpTurn(999_999L),
Map.of("lastActivityAtNanos", 999_999L, "turnCount", 8));
}
@Test
void withAgentSessionIdPreservesEveryOtherComponent() throws ReflectiveOperationException {
checkRebuildSite("withAgentSessionId", s -> s.withAgentSessionId("agent-updated"),
Map.of("agentSessionId", "agent-updated"));
}
@Test
void withFailureReasonPreservesEveryOtherComponent() throws ReflectiveOperationException {
checkRebuildSite("withFailureReason", s -> s.withFailureReason("reason-updated"),
Map.of("failureReason", "reason-updated"));
}
}