Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a9a3c12232 | |||
| 49f285cfda | |||
| 5a12ae7930 | |||
| 127e6832a9 | |||
| 6442a583ae | |||
| 24b96d29ae | |||
| 4721771052 | |||
| 2f71a30bd7 | |||
| 22cdebbdbe | |||
| 154971c2b8 | |||
| fef287c346 | |||
| a37acd5ee3 | |||
| d6ef0c8013 | |||
| dfb70871b4 | |||
| 4ee7b16929 | |||
| dd2efd8541 | |||
| 5af786d135 |
@@ -7,6 +7,14 @@
|
||||
> wiki ([Use Cases](https://git.ltms.dev/fleet/fleetd/wiki/7-Use-Cases) → *The portable
|
||||
> CLAUDE.md block*); improvements go to the template first, then out to each project. Anything
|
||||
> specific to *this* repo lives under §Project addendum below, never inline above it.
|
||||
>
|
||||
> **Anything you measure in an addendum is perishable.** Date it, give the command that
|
||||
> re-measures it and what each outcome means, and tell the reader to delete the section once
|
||||
> it stops reproducing. The four parts work together: deciding what would falsify a claim is
|
||||
> the expensive step, and a reader in the middle of another task will not pay it, so a bare
|
||||
> "verify before relying on this" costs the same space and does nothing. The case this is for
|
||||
> is a note that goes stale as a live restriction — it will tell a future session it cannot do
|
||||
> the thing at the moment doing it becomes the job.
|
||||
|
||||
If no `fleet_*` MCP tools are mounted in this session, this section does not apply — skip it.
|
||||
|
||||
@@ -94,6 +102,18 @@ below are the procedure — run them in order, every task, not only the big ones
|
||||
8. **Adjudicate, merge, tear down — yours alone.** Read the diff yourself: fully if it is small,
|
||||
targeted at the reported findings and the risky paths if it is large. Reviewer findings direct
|
||||
your attention; they never substitute for it. Then merge, then `fleet_stop{paneId}`.
|
||||
**If the forge refuses you the merge** — a protected branch, a token without the grant — the
|
||||
adjudication is still yours. Read the diff, decide, and hand the operator a merge-ready queue
|
||||
with the refusal quoted. Never report a PR as merged, and never call one "ready to merge"
|
||||
without having read the diff yourself. A refusal is exactly when that shortcut is tempting,
|
||||
because no action is left that forces you to look, and taking it turns this step into
|
||||
forwarding a reviewer's verdict — which is delegating the merge by proxy, two lines above.
|
||||
**Test a refusal; do not read it off a permissions field.** A protected branch holds its merge
|
||||
rights separately from the repository permissions, so that field can say yes while the merge is
|
||||
refused, and still say no after a grant makes it work. Probe instead, with a request that cannot
|
||||
succeed on its merits, so a rejection can only mean the refusal. Treat a transport failure as a
|
||||
third answer that proves nothing: a timeout, a DNS error or a bad URL is not a refusal, and
|
||||
counting it as one makes you sure of something you never measured.
|
||||
|
||||
**Steps 3 and 4 are separate on purpose** — spawning and sending in one loop is how parallel work
|
||||
silently becomes serial, and it is the most common way this layer is wasted. For the same reason,
|
||||
@@ -140,6 +160,11 @@ The traffic between leads is coordination and nothing else:
|
||||
3. **Verify a peer exactly as you verify yourself.** Peer status buys nothing: check the claim
|
||||
against the code, and re-run the build. A peer's correction gets the same treatment — right or
|
||||
wrong on the evidence, not on who said it. Neither of you merges the other's work unreviewed.
|
||||
4. **Ask a peer to read your project addendum.** Your addendum is instruction surface: every future
|
||||
session on your host obeys it, and a wrong one is obeyed just as faithfully as a right one. The
|
||||
author is the worst reader of their own qualifier placement — measured here, one addendum carried
|
||||
two defects and a non-author found both. If you have no peer, at least re-read it asking "which
|
||||
sentence goes false first, and would a reader reach the caveat before acting?"
|
||||
|
||||
Being messaged by a peer does not make you its worker: answer with `fleet_reply`, and push back on
|
||||
the substance if it is wrong. A peer that simply complies has thrown away the reason there are two of
|
||||
|
||||
@@ -12,8 +12,16 @@
|
||||
# # edit WorkingDirectory / ExecStart below for your host's paths and java location
|
||||
# systemctl --user daemon-reload
|
||||
# systemctl --user enable --now herdr fleetd
|
||||
# loginctl enable-linger $USER # REQUIRED -- see below
|
||||
# journalctl --user -u fleetd -f
|
||||
#
|
||||
# `loginctl enable-linger` is not optional and is easy to miss, because leaving it out looks like
|
||||
# success: `systemctl --user enable` reports "enabled" and both units run for as long as you stay
|
||||
# logged in. A user manager without lingering starts at your first login and stops at your last
|
||||
# logout, so the fleet simply does not come back after a reboot -- which is the whole reason to
|
||||
# use systemd here rather than the setsid scripts these units replaced. Check it with
|
||||
# `loginctl show-user $USER -p Linger`; the answer must be `Linger=yes`.
|
||||
#
|
||||
# Secrets (AI_GATEWAY_TOKEN, WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI, ...) are not set
|
||||
# here and need no systemd drop-in: ExecStart runs a login shell, so they come from wherever your
|
||||
# login shell already sources them (this host: ~/.fleet/secrets.sh via ~/.zprofile). If a token is
|
||||
|
||||
@@ -110,6 +110,14 @@ bind:
|
||||
# notifications:
|
||||
# mode: disabled
|
||||
|
||||
# Idle-sleep guard: while at least one member is live, hold an OS-level assertion against idle
|
||||
# sleep (macOS only — a `caffeinate -i` child; a no-op elsewhere or if caffeinate is missing), so
|
||||
# an unattended host does not idle-sleep out from under a member's long turn. Unlike health/
|
||||
# configReload above, this is ON BY DEFAULT — omitting the block entirely leaves it enabled, the
|
||||
# same as `enabled: true`. Uncomment only to turn it off:
|
||||
# idleSleepGuard:
|
||||
# enabled: false
|
||||
|
||||
# herdr Unix socket. Omit to use the client default
|
||||
# (${HERDR_SOCKET_PATH:-~/.config/herdr/herdr.sock}).
|
||||
herdrSocket: ~/.config/herdr/herdr.sock
|
||||
|
||||
@@ -55,6 +55,8 @@ import dev.ltms.fleet.member.MemberCredentialPolicyView;
|
||||
import dev.ltms.fleet.member.OpenCodeLauncher;
|
||||
import dev.ltms.fleet.placement.BackendOutagePolicy;
|
||||
import dev.ltms.fleet.placement.BackendQuarantine;
|
||||
import dev.ltms.fleet.power.CaffeinateSleepAssertionMechanism;
|
||||
import dev.ltms.fleet.power.IdleSleepGuard;
|
||||
import io.javalin.Javalin;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
@@ -253,6 +255,26 @@ public final class Fleetd {
|
||||
System::nanoTime, contextCap, clearAfterTurn);
|
||||
liveCountRef.set(profileName -> liveSessionCount(sessions.roster(), profileName));
|
||||
|
||||
// Idle-sleep guard: hold an OS-level assertion against idle sleep while at least one
|
||||
// member is live, so an unattended host does not idle-sleep out from under a member's
|
||||
// long turn (see FleetConfig.IdleSleepGuard / dev.ltms.fleet.power.IdleSleepGuard for the
|
||||
// measurement that motivated this). Opt-out via idleSleepGuard.enabled: false; on by
|
||||
// default. Hangs off SessionManager's own onAcquire/onRelease hooks (CB-520/CB-516,
|
||||
// previously wired only to the reply inbox) and SessionManager#size() — the exact registry
|
||||
// fleet_list's live/capacity numbers are themselves computed from — rather than tracking
|
||||
// members a second way. No-op (never constructed) off macOS or when idleSleepGuard.enabled
|
||||
// is explicitly false; the mechanism itself is additionally a no-op if 'caffeinate' cannot
|
||||
// be started, so this can never fail a spawn, a release, or startup.
|
||||
boolean idleSleepGuardEnabled = cfg.idleSleepGuard() == null || cfg.idleSleepGuard().isEnabled();
|
||||
final IdleSleepGuard idleSleepGuard;
|
||||
if (idleSleepGuardEnabled) {
|
||||
idleSleepGuard = new IdleSleepGuard(new CaffeinateSleepAssertionMechanism(), sessions::size);
|
||||
sessions.onAcquire(_ -> idleSleepGuard.recheck());
|
||||
sessions.onRelease(_ -> idleSleepGuard.recheck());
|
||||
} else {
|
||||
idleSleepGuard = null;
|
||||
}
|
||||
|
||||
// CB-303 part 1: idle-ttl reaper — only when configured, defaults to disabled.
|
||||
final SessionReaper reaper;
|
||||
if (cfg.lifecycle() != null
|
||||
@@ -704,6 +726,11 @@ public final class Fleetd {
|
||||
if (configWatcher != null) configWatcher.stop(); // CB-559: stop polling the config file
|
||||
mcp.close();
|
||||
if (reaper != null) reaper.stop();
|
||||
// Idle-sleep guard: release unconditionally, even though sessions.close() above already
|
||||
// drained every session (and each release already drove the live count to 0, which
|
||||
// releases the guard's assertion on its own) — this is the backstop for a drain that was
|
||||
// itself interrupted or threw, so no caffeinate child ever outlives the daemon.
|
||||
if (idleSleepGuard != null) idleSleepGuard.close();
|
||||
// Release the broker connection last among message resources (no-op for the in-memory inbox).
|
||||
if (replyInbox instanceof AutoCloseable closeable) {
|
||||
try {
|
||||
|
||||
@@ -37,6 +37,10 @@ import java.util.function.Supplier;
|
||||
* makes {@code fleet:} split rather than hot — see below.</li>
|
||||
* <li><strong>Deferred</strong> — accepted into the new snapshot, but the wiring built at startup
|
||||
* keeps the old value until a restart: {@code lifecycle:}, {@code leadHeartbeat:},
|
||||
* {@code idleSleepGuard:} ({@code Fleetd.java} reads it once, at startup, to decide whether
|
||||
* to construct an {@code IdleSleepGuard} and wire {@code SessionManager}'s
|
||||
* {@code onAcquire}/{@code onRelease} hooks to it — neither is rebuilt on reload, so a
|
||||
* running daemon keeps whatever this was at startup regardless of a later edit),
|
||||
* {@code spawnReadyTimeoutMs} / {@code spawnReadyPollMs}, {@code quarantineCooldownSeconds}
|
||||
* (CB-578 stage B — baked once into the {@code BackendQuarantine} built at startup),
|
||||
* {@code guard:}, {@code worktreeRoot:}, {@code worktreeGroup:} and {@code memberSkills:}
|
||||
@@ -131,9 +135,9 @@ import java.util.function.Supplier;
|
||||
* </ul>
|
||||
*
|
||||
* <p><strong>The denominator, measured on 2026-09-04 (fleetd #330; recounted for fleetd #333);
|
||||
* recounted again for fleetd #362.</strong> {@code FleetConfig} has 23 top-level record components:
|
||||
* 5 cold, 12 deferred, 3 split, 3 hot-excluded. Three of them are named nowhere in this file, and
|
||||
* the reason is the same for all
|
||||
* recounted again for fleetd #362, and again after {@code idleSleepGuard:} was added.</strong>
|
||||
* {@code FleetConfig} has 24 top-level record components: 5 cold, 13 deferred, 3 split, 3
|
||||
* hot-excluded. Three of them are named nowhere in this file, and the reason is the same for all
|
||||
* three: {@code placement}, {@code memberCredentials} and {@code memberLoginShell} are
|
||||
* <strong>hot</strong> and correctly absent — all three are read live off {@code config.get()}
|
||||
* (placement through the {@code CompositePeerLauncher} supplier the Hot bullet names;
|
||||
@@ -215,7 +219,7 @@ public final class ConfigRef implements Supplier<FleetConfig> {
|
||||
static final Set<String> DEFERRED_KEYS = Set.of(
|
||||
"guard", "worktreeRoot", "worktreeGroup", "memberSkills", "primary", "configReload",
|
||||
"leadHeartbeat", "lifecycle", "spawnReadyTimeoutMs", "spawnReadyPollMs",
|
||||
"quarantineCooldownSeconds", "profiles");
|
||||
"quarantineCooldownSeconds", "profiles", "idleSleepGuard");
|
||||
|
||||
private final Path path;
|
||||
private final AtomicReference<FleetConfig> current;
|
||||
@@ -427,6 +431,14 @@ public final class ConfigRef implements Supplier<FleetConfig> {
|
||||
if (!Objects.equals(old.configReload(), fresh.configReload())) {
|
||||
changed.add("configReload");
|
||||
}
|
||||
// Fleetd.java reads cfg.idleSleepGuard() once, at startup, to decide whether to construct
|
||||
// an IdleSleepGuard at all and wire SessionManager's onAcquire/onRelease hooks to it —
|
||||
// neither is rebuilt on reload, so a running daemon keeps whatever this was at startup
|
||||
// (armed or not) regardless of a later edit here. Not cold: nothing already-open goes
|
||||
// inconsistent with the new value, an armed-or-not guard just keeps its original answer.
|
||||
if (!Objects.equals(old.idleSleepGuard(), fresh.idleSleepGuard())) {
|
||||
changed.add("idleSleepGuard");
|
||||
}
|
||||
if (!Objects.equals(old.spawnReadyTimeoutMs(), fresh.spawnReadyTimeoutMs())
|
||||
|| !Objects.equals(old.spawnReadyPollMs(), fresh.spawnReadyPollMs())) {
|
||||
changed.add("spawnReady*");
|
||||
|
||||
@@ -119,6 +119,11 @@ import java.util.regex.PatternSyntaxException;
|
||||
* subdirectory of this directory is copied wholesale, with no per-file
|
||||
* allowlist — do not park scratch files or drafts alongside the real skill
|
||||
* folders, they will be copied into every provisioned worktree too.
|
||||
* @param idleSleepGuard opt-in-by-default: hold an OS-level assertion against idle sleep while at
|
||||
* least one member is live, so an unattended host does not sleep out from
|
||||
* under a member's long turn. {@code null} (the block omitted) behaves the
|
||||
* same as an explicit {@code enabled: true}; set {@code enabled: false} to
|
||||
* turn it off. See {@link dev.ltms.fleet.power.IdleSleepGuard}.
|
||||
*/
|
||||
@JsonIgnoreProperties(ignoreUnknown = true)
|
||||
public record FleetConfig(
|
||||
@@ -144,7 +149,22 @@ public record FleetConfig(
|
||||
Coordinator coordinator,
|
||||
String worktreeGroup,
|
||||
String memberLoginShell,
|
||||
String memberSkills) {
|
||||
String memberSkills,
|
||||
IdleSleepGuard idleSleepGuard) {
|
||||
|
||||
/** Back-compat form before the {@code idleSleepGuard:} block was added. */
|
||||
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
|
||||
Guard guard, String worktreeRoot, Lifecycle lifecycle, Integer spawnReadyTimeoutMs,
|
||||
Integer spawnReadyPollMs, Broker broker, Primary primary, Fleet fleet,
|
||||
LeadHeartbeat leadHeartbeat, Health health, String placement, Auth auth,
|
||||
ConfigReload configReload, Integer quarantineCooldownSeconds,
|
||||
MemberCredentials memberCredentials, Coordinator coordinator, String worktreeGroup,
|
||||
String memberLoginShell, String memberSkills) {
|
||||
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
|
||||
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
|
||||
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup,
|
||||
memberLoginShell, memberSkills, null);
|
||||
}
|
||||
|
||||
/** Back-compat form before the {@code memberSkills} key was added. */
|
||||
public FleetConfig(Bind bind, String herdrSocket, String memberHerdrSocket, Map<String, Profile> profiles,
|
||||
@@ -157,7 +177,7 @@ public record FleetConfig(
|
||||
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
|
||||
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
|
||||
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup,
|
||||
memberLoginShell, null);
|
||||
memberLoginShell, null, null);
|
||||
}
|
||||
|
||||
/** Back-compat form before the {@code memberLoginShell} key was added. */
|
||||
@@ -169,7 +189,7 @@ public record FleetConfig(
|
||||
MemberCredentials memberCredentials, Coordinator coordinator, String worktreeGroup) {
|
||||
this(bind, herdrSocket, memberHerdrSocket, profiles, guard, worktreeRoot, lifecycle, spawnReadyTimeoutMs,
|
||||
spawnReadyPollMs, broker, primary, fleet, leadHeartbeat, health, placement, auth,
|
||||
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup, null);
|
||||
configReload, quarantineCooldownSeconds, memberCredentials, coordinator, worktreeGroup, null, null);
|
||||
}
|
||||
|
||||
/** Back-compat form before the {@code worktreeGroup} key was added. */
|
||||
@@ -1279,6 +1299,25 @@ public record FleetConfig(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Hold an OS-level assertion against idle sleep while at least one member is live (see
|
||||
* {@link dev.ltms.fleet.power.IdleSleepGuard}).
|
||||
*
|
||||
* <p>Unlike most opt-in blocks in this file, this one defaults to <em>on</em>: an unattended
|
||||
* host idle-sleeping mid-turn is a correctness problem (a dropped AMQP link, a frozen member),
|
||||
* not a convenience, so the safer default is armed. An operator who wants the previous
|
||||
* behaviour (no assertion held, ever) sets {@code enabled: false} explicitly.
|
||||
*
|
||||
* @param enabled {@code false} turns the guard off; {@code null} (the block omitted
|
||||
* entirely) or {@code true} leaves it on
|
||||
*/
|
||||
@JsonIgnoreProperties(ignoreUnknown = true)
|
||||
public record IdleSleepGuard(Boolean enabled) {
|
||||
public boolean isEnabled() {
|
||||
return !Boolean.FALSE.equals(enabled);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The terminal → lead-name map seeded from the legacy singular {@code primary:} pin (CB-530).
|
||||
*
|
||||
@@ -1529,7 +1568,8 @@ public record FleetConfig(
|
||||
"bind", "herdrSocket", "memberHerdrSocket", "profiles", "guard", "worktreeRoot",
|
||||
"lifecycle", "spawnReadyTimeoutMs", "spawnReadyPollMs", "broker", "primary", "fleet",
|
||||
"leadHeartbeat", "health", "placement", "auth", "configReload", "quarantineCooldownSeconds",
|
||||
"memberCredentials", "coordinator", "worktreeGroup", "memberLoginShell", "memberSkills");
|
||||
"memberCredentials", "coordinator", "worktreeGroup", "memberLoginShell", "memberSkills",
|
||||
"idleSleepGuard");
|
||||
|
||||
/** Load and validate config from {@code path}. */
|
||||
public static FleetConfig load(Path path) {
|
||||
@@ -2209,9 +2249,15 @@ public record FleetConfig(
|
||||
// memberSkills is left as-is (fleetd #362), like worktreeGroup/memberLoginShell: null/blank
|
||||
// is "off", and there is no sane non-null default — the daemon may not even run from a
|
||||
// checkout that ships its own .claude/skills/.
|
||||
// idleSleepGuard is left as-is, like leadHeartbeat/configReload above, but for the opposite
|
||||
// reason: it is on by default already (its own isEnabled() treats null the same as
|
||||
// enabled: true — see its javadoc), so defaulting the block here would change nothing a
|
||||
// reader observes and would only obscure that "block omitted" and "block present and
|
||||
// enabled" are deliberately the same outcome.
|
||||
return new FleetConfig(b, herdrSocket, memberHerdrSocket, profiles, g, worktreeRoot, l, timeout, pollMs,
|
||||
broker, primary, f, leadHeartbeat, health, placementOrDefault, a, configReload,
|
||||
quarantineCooldown, mc, coordinator, worktreeGroup, memberLoginShell, memberSkills);
|
||||
quarantineCooldown, mc, coordinator, worktreeGroup, memberLoginShell, memberSkills,
|
||||
idleSleepGuard);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -870,6 +870,11 @@ public final class FleetMcp {
|
||||
* or — when no send is open — queueing the reply in the inbox for later drain (CB-307).
|
||||
* {@code callerTerminal} is resolved from the connection (never an argument); a {@code null}
|
||||
* means the caller is not a known worker (e.g. the primary called it by mistake).
|
||||
*
|
||||
* <p>fleetd #365: the result text names which of those actually happened
|
||||
* ({@link MessageService.ReplyOutcome#description()}) instead of the single word "delivered"
|
||||
* for both — a queued reply is a real success, but it is not the same fact as one that resolved
|
||||
* a live waiter, and the caller could not previously tell them apart.
|
||||
*/
|
||||
static McpSchema.CallToolResult reply(MessageService messages, String callerTerminal, String content) {
|
||||
if (callerTerminal == null) {
|
||||
@@ -884,8 +889,8 @@ public final class FleetMcp {
|
||||
if (isBlank(content)) {
|
||||
return error("content is required");
|
||||
}
|
||||
messages.reply(callerTerminal, content);
|
||||
return text("delivered");
|
||||
MessageService.ReplyOutcome outcome = messages.reply(callerTerminal, content);
|
||||
return text(outcome.description());
|
||||
}
|
||||
|
||||
/** {@code fleet_ack}: acknowledge (remove) a specific reply from the inbox. */
|
||||
|
||||
@@ -56,10 +56,13 @@ public final class FleetMetrics {
|
||||
m.describe(REPLIES, "counter",
|
||||
"Worker replies by delivery path (rendezvous=resolved an open send, inbox=stranded and held).");
|
||||
m.describe(PUSH_NUDGES, "counter",
|
||||
"CB-307 push-loop nudges to the primary (delivered|exhausted).");
|
||||
"CB-307 push-loop nudges to the primary (sent|exhausted). fleetd #365: \"sent\" means "
|
||||
+ "the herdr paste-and-submit call succeeded, not that the pane read it — this "
|
||||
+ "layer has no read-receipt concept.");
|
||||
m.describe(HEARTBEAT_NUDGES, "counter",
|
||||
"CB-551 idle-lead heartbeat nudges (delivered|failed|exhausted). Quiet-cap exhaustion "
|
||||
+ "means the lead idled with nothing pending and was told to stand down.");
|
||||
"CB-551 idle-lead heartbeat nudges (sent|failed|exhausted). Quiet-cap exhaustion "
|
||||
+ "means the lead idled with nothing pending and was told to stand down. "
|
||||
+ "fleetd #365: \"sent\" means the herdr call succeeded, not that the lead read it.");
|
||||
m.describe(SPAWNS, "counter",
|
||||
"Worker spawn attempts by peer kind and outcome (ready|timeout|guard_rejected).");
|
||||
m.describe(HERDR_CALLS, "counter",
|
||||
|
||||
@@ -96,7 +96,14 @@ public final class LeadHeartbeatLoop {
|
||||
this.metrics = metrics;
|
||||
}
|
||||
|
||||
/** Count one nudge outcome when a registry is wired; a no-op in unit tests. */
|
||||
/**
|
||||
* Count one nudge outcome when a registry is wired; a no-op in unit tests.
|
||||
*
|
||||
* <p>fleetd #365: the {@code "sent"} outcome (renamed from {@code "delivered"}) records only
|
||||
* that {@link #injectNudge} — a one-way herdr {@code agent.prompt} paste-and-submit — returned
|
||||
* without throwing, not that the lead's pane actually read or acted on the text. This layer has
|
||||
* no read-receipt concept, so "sent" is the honest word for what this call can ever establish.
|
||||
*/
|
||||
private void countNudge(String outcome) {
|
||||
if (metrics != null) {
|
||||
metrics.inc(FleetMetrics.HEARTBEAT_NUDGES, "outcome", outcome);
|
||||
@@ -245,7 +252,7 @@ public final class LeadHeartbeatLoop {
|
||||
agents.send(leadTerminal, fleet.nudgeText());
|
||||
log.debug("idle-heartbeat: nudge sent to lead {} (quiet nudges so far in this stretch: {})",
|
||||
leadTerminal, quietCount);
|
||||
countNudge("delivered");
|
||||
countNudge("sent");
|
||||
} catch (RuntimeException e) {
|
||||
log.warn("idle-heartbeat: failed to nudge lead {}: {}", leadTerminal, e.toString());
|
||||
countNudge("failed");
|
||||
|
||||
@@ -138,6 +138,53 @@ public final class MessageService {
|
||||
public record AskResult(AskOutcome outcome, String answer) {
|
||||
}
|
||||
|
||||
/**
|
||||
* How a worker's {@code fleet_reply} ({@link #reply(String, String)}) actually landed
|
||||
* (fleetd #365) — the two doors that expose it, {@code fleet_reply} and {@code POST
|
||||
* /sessions/{id}/reply}, both used to report the single word "delivered" whichever of these
|
||||
* happened, so a caller could not tell an active handoff from a reply merely held for later
|
||||
* drain. Both are successes; they are not the same fact.
|
||||
*/
|
||||
public enum ReplyOutcome {
|
||||
/** Resolved a {@code fleet_send}/{@code fleet_ask} that was actively waiting on this reply. */
|
||||
RESOLVED_SEND("resolved_send", true,
|
||||
"delivered — resolved the fleet_send that was waiting for it"),
|
||||
/**
|
||||
* No live waiter was open, but the reply completed a parked async ticket directly
|
||||
* ({@link #askAnsweredAsyncTasks}) — a {@code fleet_poll} caller sees it immediately.
|
||||
*/
|
||||
RESOLVED_ASYNC_TICKET("resolved_async_ticket", true,
|
||||
"delivered — resolved a pending async ticket (visible to fleet_poll)"),
|
||||
/** Nothing was waiting; the reply was queued in the inbox for a later drain (CB-307). */
|
||||
QUEUED("queued", false,
|
||||
"queued — no send or ticket was waiting; held in the inbox for a later drain");
|
||||
|
||||
private final String wireName;
|
||||
private final boolean delivered;
|
||||
private final String description;
|
||||
|
||||
ReplyOutcome(String wireName, boolean delivered, String description) {
|
||||
this.wireName = wireName;
|
||||
this.delivered = delivered;
|
||||
this.description = description;
|
||||
}
|
||||
|
||||
/** Stable machine-readable name for a JSON/metrics label (REST's {@code outcome} field). */
|
||||
public String wireName() {
|
||||
return wireName;
|
||||
}
|
||||
|
||||
/** Whether something was actively waiting and received this reply right now. */
|
||||
public boolean delivered() {
|
||||
return delivered;
|
||||
}
|
||||
|
||||
/** Shared human-readable text — the one place both {@code fleet_reply} and REST word this. */
|
||||
public String description() {
|
||||
return description;
|
||||
}
|
||||
}
|
||||
|
||||
/** Lifecycle phase of an async delegation ticket. */
|
||||
public enum Phase {
|
||||
/** Delegated and in flight — queued for the worker or being worked. */
|
||||
@@ -439,16 +486,15 @@ public final class MessageService {
|
||||
* @throws IllegalArgumentException if {@code content} is {@code null} or blank — the caller must
|
||||
* report this as a client error (REST: 400 {@code bad_request}) rather than resolve
|
||||
* anything
|
||||
* @return always {@code true} — the reply resolved a live send, completed a parked ticket, or
|
||||
* was queued
|
||||
* @return which of the three ways (fleetd #365) the reply actually landed — never {@code null}
|
||||
*/
|
||||
public boolean reply(String session, String content) {
|
||||
public ReplyOutcome reply(String session, String content) {
|
||||
if (content == null || content.isBlank()) {
|
||||
throw new IllegalArgumentException("content is required");
|
||||
}
|
||||
if (rendezvous.resolve(session, content)) {
|
||||
count(FleetMetrics.REPLIES, "path", "rendezvous");
|
||||
return true; // a live send took it — unchanged fast path
|
||||
return ReplyOutcome.RESOLVED_SEND; // a live send took it — unchanged fast path
|
||||
}
|
||||
// #137/fleetd #307: no live rendezvous waiter, but this may be the worker's real fleet_reply resuming
|
||||
// a turn that either answer() (#137) or ask() (fleetd #307) already gave up waiting on:
|
||||
@@ -480,7 +526,7 @@ public final class MessageService {
|
||||
asyncTasksByTurn.remove(turnId, orphan);
|
||||
}
|
||||
count(FleetMetrics.REPLIES, "path", "async-recovered");
|
||||
return true; // the ticket itself took it — no inbox stranding at all
|
||||
return ReplyOutcome.RESOLVED_ASYNC_TICKET; // the ticket itself took it — no inbox stranding
|
||||
}
|
||||
} else if (candidates.size() > 1) {
|
||||
List<String> tickets = candidates.stream().map(t -> t.ticket).toList();
|
||||
@@ -498,7 +544,7 @@ public final class MessageService {
|
||||
if (pushLoop != null) {
|
||||
pushLoop.onReplyQueued(session);
|
||||
}
|
||||
return true; // held, not lost
|
||||
return ReplyOutcome.QUEUED; // held, not lost — but not delivered either
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -2,6 +2,7 @@ package dev.ltms.fleet.msg;
|
||||
|
||||
import dev.ltms.fleet.herdr.AgentControl;
|
||||
import dev.ltms.fleet.herdr.AgentStatus;
|
||||
import dev.ltms.fleet.herdr.HerdrException;
|
||||
import dev.ltms.fleet.mcp.PrimaryRegistry;
|
||||
import dev.ltms.fleet.metrics.FleetMetrics;
|
||||
import dev.ltms.fleet.metrics.Metrics;
|
||||
@@ -13,6 +14,7 @@ import java.util.Collection;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ScheduledExecutorService;
|
||||
@@ -130,7 +132,15 @@ public final class ReplyPushLoop {
|
||||
this.metrics = metrics;
|
||||
}
|
||||
|
||||
/** Count one nudge outcome when a registry is wired; a no-op in unit tests. */
|
||||
/**
|
||||
* Count one nudge outcome when a registry is wired; a no-op in unit tests.
|
||||
*
|
||||
* <p>fleetd #365: the {@code "sent"} outcome (renamed from {@code "delivered"}) records only
|
||||
* that {@code agents.send} — a one-way herdr {@code agent.prompt} paste-and-submit — returned
|
||||
* without throwing. Nothing in this loop, or anywhere downstream of it, confirms the pane
|
||||
* actually read or acted on the text; there is no read-receipt concept at this layer. "Sent"
|
||||
* says exactly that; "delivered" claimed more than this call can ever establish.
|
||||
*/
|
||||
private void countNudge(String outcome) {
|
||||
if (metrics != null) {
|
||||
metrics.inc(FleetMetrics.PUSH_NUDGES, "outcome", outcome);
|
||||
@@ -375,6 +385,80 @@ public final class ReplyPushLoop {
|
||||
return Action.WAIT_BUSY;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve who to nudge about {@code target}, the way every public entry point below wants it:
|
||||
* {@link PrimaryRegistry#nudgeTargetFor}, but only after checking the delegating lead it names
|
||||
* is still actually there (fleetd #368).
|
||||
*
|
||||
* <p><strong>The bug this closes.</strong> {@code PrimaryRegistry.forgetDelegation} is wired to
|
||||
* exactly one event — a worker's release — because that is the only teardown the daemon already
|
||||
* observes for a session in this map. Nothing removes a binding when the LEAD half goes away: a
|
||||
* lead that is closed, crashes, or is relaunched leaves {@code leadByTarget} entries pointing at
|
||||
* a terminal that no longer exists. {@code nudgeTargetFor} falls back to the single known
|
||||
* primary only when the map holds nothing for {@code target} — a stale non-null entry beats the
|
||||
* fallback every time, which is exactly backwards: the fallback's own javadoc argues it is safe
|
||||
* precisely in the case a stale entry now hides.
|
||||
*
|
||||
* <p><strong>The fix.</strong> Before trusting a recorded delegation, probe the lead the same
|
||||
* way {@link #decide} already does every tick ({@code agents.status}) — cheap, since it is a
|
||||
* local herdr round-trip, and it is the same signal {@code AgentControl.paneByTerminal} already
|
||||
* trusts to tell a genuinely dead target from a live one. A lead that fails the probe is treated
|
||||
* as if it had never been recorded: the stale entry is forgotten (self-healing, exactly like
|
||||
* {@code AgentControl.paneByTerminal} already does on {@code agent_not_found}) and resolution is
|
||||
* retried, which now reaches the fallback {@code nudgeTargetFor} was built to reach — the same
|
||||
* empty-map state its javadoc already argues is correct.
|
||||
*
|
||||
* <p><strong>fleetd #368 review — only a positive "gone" reading forgets the binding.</strong>
|
||||
* The first version of this method treated <em>any</em> {@code RuntimeException} from the probe
|
||||
* as death, which is the #359 mistake repeated: a transient socket blip or a codec error on a
|
||||
* perfectly live lead would silently and permanently unbind it, with no re-record ever coming.
|
||||
* That is destructive on one bad reading, exactly what #359 shipped a two-reading guard to avoid
|
||||
* for the analogous lead-tab-liveness question. {@link #isLive} now matches
|
||||
* {@code AgentControl.agentCall}'s own narrower rule (see its {@code agent_not_found} check): only
|
||||
* that specific, affirmative "herdr has no such agent" signal counts as gone. Every other failure
|
||||
* — timeout, transport error, a decode error — is treated as still live and the binding is left
|
||||
* alone, because guessing wrong here is unrecoverable while guessing "live" merely costs one more
|
||||
* retry on the next tick, which {@link #decide} already tolerates.
|
||||
*/
|
||||
private Optional<String> resolveLiveLead(String target) {
|
||||
Optional<String> lead = primaryRegistry.nudgeTargetFor(target);
|
||||
if (lead.isEmpty() || isLive(lead.get())) {
|
||||
return lead;
|
||||
}
|
||||
log.debug("push: lead {} delegated to for {} is no longer live, forgetting the stale binding "
|
||||
+ "and falling back", lead.get(), target);
|
||||
primaryRegistry.forgetDelegation(target);
|
||||
return primaryRegistry.nudgeTargetFor(target);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether {@code lead} should still be trusted: {@code false} only when herdr affirmatively
|
||||
* reports the terminal gone ({@code agent_not_found}), never on a merely inconclusive failure.
|
||||
*
|
||||
* <p>fleetd #368 review: an earlier version returned {@code false} for any {@code RuntimeException},
|
||||
* which made a transient herdr hiccup on a live lead indistinguishable from the lead actually
|
||||
* being dead — and the caller's response to {@code false} ({@code forgetDelegation}) is
|
||||
* destructive and permanent. Narrowed to the one code {@code AgentControl.agentCall} itself
|
||||
* already treats as a genuine, resolvable absence (see its {@code agent_not_found} handling) —
|
||||
* every other {@code RuntimeException} is treated as "still live" and the binding survives to be
|
||||
* probed again next time, which costs nothing worse than one more retry.
|
||||
*/
|
||||
private boolean isLive(String lead) {
|
||||
try {
|
||||
agents.status(lead);
|
||||
return true;
|
||||
} catch (RuntimeException e) {
|
||||
boolean gone = e instanceof HerdrException he && "agent_not_found".equals(he.code());
|
||||
if (gone) {
|
||||
log.debug("push: lead {} no longer exists ({})", lead, e.toString());
|
||||
} else {
|
||||
log.debug("push: liveness check for lead {} was inconclusive ({}); treating as live "
|
||||
+ "rather than risk destroying a live binding", lead, e.toString());
|
||||
}
|
||||
return !gone;
|
||||
}
|
||||
}
|
||||
|
||||
// --- public entrypoints ----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
@@ -385,7 +469,7 @@ public final class ReplyPushLoop {
|
||||
* backstop until a lead is recorded.
|
||||
*/
|
||||
public void onReplyQueued(String target) {
|
||||
var lead = primaryRegistry.nudgeTargetFor(target);
|
||||
var lead = resolveLiveLead(target);
|
||||
if (lead.isEmpty()) {
|
||||
log.debug("push: no lead is known to be waiting on {}, skipping reminder", target);
|
||||
return;
|
||||
@@ -413,7 +497,7 @@ public final class ReplyPushLoop {
|
||||
* @param failed whether the ticket ended in a failure phase rather than {@code DONE}
|
||||
*/
|
||||
public void onTicketTerminal(String ticket, String target, boolean failed) {
|
||||
var lead = primaryRegistry.nudgeTargetFor(target);
|
||||
var lead = resolveLiveLead(target);
|
||||
if (lead.isEmpty()) {
|
||||
log.debug("push: no lead is known to be waiting on ticket {} (target {}), skipping nudge",
|
||||
ticket, target);
|
||||
@@ -448,7 +532,7 @@ public final class ReplyPushLoop {
|
||||
* @param question the question text
|
||||
*/
|
||||
public void onQuestionOpened(String ticket, String target, String turnId, String question) {
|
||||
var lead = primaryRegistry.nudgeTargetFor(target);
|
||||
var lead = resolveLiveLead(target);
|
||||
if (lead.isEmpty()) {
|
||||
log.debug("push: no lead is known to be waiting on {}'s question (turnId {}), skipping nudge",
|
||||
target, turnId);
|
||||
@@ -476,7 +560,7 @@ public final class ReplyPushLoop {
|
||||
Collection<String> profiles, int remainingCoolOffSeconds) {
|
||||
Map<String, List<String>> targetsByLead = new ConcurrentHashMap<>();
|
||||
for (String target : targets) {
|
||||
var lead = primaryRegistry.nudgeTargetFor(target);
|
||||
var lead = resolveLiveLead(target);
|
||||
if (lead.isEmpty()) {
|
||||
log.warn("push: backend incident {} has no known lead for target {}", incidentId, target);
|
||||
continue;
|
||||
@@ -498,7 +582,7 @@ public final class ReplyPushLoop {
|
||||
* Without an owning lead, emit a warning because no control can act on the target.
|
||||
*/
|
||||
public void onBackendTargetUnmapped(String target, String reason) {
|
||||
var lead = primaryRegistry.nudgeTargetFor(target);
|
||||
var lead = resolveLiveLead(target);
|
||||
if (lead.isEmpty()) {
|
||||
log.warn("push: backend target {} could not map to a credential: {}", target, reason);
|
||||
return;
|
||||
@@ -654,7 +738,7 @@ public final class ReplyPushLoop {
|
||||
lead, replyReminderCount + 1, maxReminders, ticketReminderCount + 1, maxReminders,
|
||||
questionReminderCount + 1, maxReminders,
|
||||
replyTargets.size(), tickets.size(), questions.size());
|
||||
countNudge("delivered");
|
||||
countNudge("sent");
|
||||
for (PendingIncident incident : incidents) {
|
||||
if (pendingIncidents.remove(incident.key(), incident)) {
|
||||
deliveredIncidents.add(incident.key());
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Locale;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
|
||||
/**
|
||||
* Holds macOS idle sleep off by keeping a {@code caffeinate -i} child process alive for the life
|
||||
* of the returned {@link SleepAssertion}.
|
||||
*
|
||||
* <p>{@code -i} asserts only against <em>idle</em> sleep — it does not stop the lid closing or an
|
||||
* operator-requested sleep from taking effect. That is deliberate: this class exists to stop an
|
||||
* unattended host from sleeping out from under a member's long turn, never to override the
|
||||
* operator. {@code -s}/{@code -d} (which also block system/display sleep on demand) are
|
||||
* intentionally not used here.
|
||||
*
|
||||
* <p>{@link #acquire()} never throws. It returns {@code null} — a no-op — off macOS, and again if
|
||||
* starting the {@code caffeinate} child fails for any reason (binary missing, process table full,
|
||||
* …); either case is logged once at INFO, not on every occurrence, so a daemon that runs for
|
||||
* weeks with the tool unavailable does not fill its log.
|
||||
*/
|
||||
public final class CaffeinateSleepAssertionMechanism implements SleepAssertionMechanism {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CaffeinateSleepAssertionMechanism.class);
|
||||
|
||||
private final AtomicBoolean loggedOnce = new AtomicBoolean(false);
|
||||
|
||||
/** {@code true} when running on macOS, the only platform {@code caffeinate} ships on. */
|
||||
public static boolean isSupportedPlatform() {
|
||||
return isSupportedPlatform(System.getProperty("os.name"));
|
||||
}
|
||||
|
||||
/** Package-visible so a test can drive the platform check without touching a real property. */
|
||||
static boolean isSupportedPlatform(String osName) {
|
||||
return osName != null && osName.toLowerCase(Locale.ROOT).contains("mac");
|
||||
}
|
||||
|
||||
@Override
|
||||
public SleepAssertion acquire() {
|
||||
if (!isSupportedPlatform()) {
|
||||
logOnce("not running on macOS (os.name={}); the idle-sleep guard is a no-op on this platform",
|
||||
System.getProperty("os.name"));
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
Process process = new ProcessBuilder("caffeinate", "-i")
|
||||
.redirectOutput(ProcessBuilder.Redirect.DISCARD)
|
||||
.redirectError(ProcessBuilder.Redirect.DISCARD)
|
||||
.start();
|
||||
return new CaffeinateAssertion(process);
|
||||
} catch (IOException | RuntimeException e) {
|
||||
logOnce("could not start 'caffeinate -i' ({}); the host may idle-sleep while members are live",
|
||||
e.toString());
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private void logOnce(String format, Object arg) {
|
||||
if (loggedOnce.compareAndSet(false, true)) {
|
||||
log.info("idle-sleep guard: " + format, arg);
|
||||
}
|
||||
}
|
||||
|
||||
/** Wraps the live {@code caffeinate} child; {@link #close} force-destroys it, idempotently. */
|
||||
private static final class CaffeinateAssertion implements SleepAssertion {
|
||||
|
||||
private final Process process;
|
||||
|
||||
CaffeinateAssertion(Process process) {
|
||||
this.process = process;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
if (!process.isAlive()) {
|
||||
return;
|
||||
}
|
||||
process.destroy();
|
||||
try {
|
||||
if (!process.waitFor(2, TimeUnit.SECONDS)) {
|
||||
process.destroyForcibly();
|
||||
}
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
process.destroyForcibly();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.function.IntSupplier;
|
||||
|
||||
/**
|
||||
* Holds an OS-level assertion against idle sleep for exactly as long as at least one fleet
|
||||
* member is live.
|
||||
*
|
||||
* <p><strong>Why this exists:</strong> a fleetd host was measured idle-sleeping after as little
|
||||
* as one minute of inactivity (its {@code pmset -g custom} reports {@code sleep 1} on battery).
|
||||
* Overnight the daemon's AMQP link to the broker dropped 13 times, and cross-checking every drop
|
||||
* minute against {@code pmset -g log} found a sleep or wake event in the same minute or the one
|
||||
* before, every time. The AMQP churn is only the visible symptom — the real problem is that a
|
||||
* member mid-turn freezes with the host, and a long turn with nobody typing is exactly the case
|
||||
* that goes idle.
|
||||
*
|
||||
* <p><strong>How it tracks "live":</strong> this is driven by {@code SessionManager}'s existing
|
||||
* {@code onAcquire}/{@code onRelease} lifecycle hooks (added for CB-520/CB-516, previously wired
|
||||
* to nothing but the reply inbox) rather than a second member count kept in parallel. Wire it as:
|
||||
* <pre>{@code
|
||||
* IdleSleepGuard guard = new IdleSleepGuard(mechanism, sessions::size);
|
||||
* sessions.onAcquire(_ -> guard.recheck());
|
||||
* sessions.onRelease(_ -> guard.recheck());
|
||||
* }</pre>
|
||||
* Every acquire/release event re-reads {@code SessionManager#size()} — the same registry {@code
|
||||
* fleet_list}'s live/capacity numbers are themselves computed from — and only an actual 0→1 or
|
||||
* 1→0 crossing touches the OS. A listener exception is already caught and logged by {@code
|
||||
* SessionManager} itself (it must never let a listener failure block the acquire/release it is
|
||||
* reacting to), so {@link #recheck()} does not need its own top-level try/catch to honor that.
|
||||
*
|
||||
* <p><strong>Failure posture:</strong> every method here is safe to call whether or not {@link
|
||||
* SleepAssertionMechanism#acquire()} actually works. A mechanism that returns {@code null} (wrong
|
||||
* platform, missing tool, spawn failure) simply means this guard never holds anything — it never
|
||||
* throws and never blocks a spawn, a release, or shutdown.
|
||||
*/
|
||||
public final class IdleSleepGuard implements AutoCloseable {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(IdleSleepGuard.class);
|
||||
|
||||
private final SleepAssertionMechanism mechanism;
|
||||
private final IntSupplier liveCount;
|
||||
private final Object lock = new Object();
|
||||
private SleepAssertion held;
|
||||
|
||||
public IdleSleepGuard(SleepAssertionMechanism mechanism, IntSupplier liveCount) {
|
||||
this.mechanism = mechanism;
|
||||
this.liveCount = liveCount;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-read the live count and acquire or release the held assertion to match: nothing held and
|
||||
* at least one member live ⇒ acquire; something held and no member live ⇒ release. A steady
|
||||
* count (still zero, still positive) is a no-op either way, so a single spawn or release only
|
||||
* ever touches the OS on the crossing, not on every call.
|
||||
*/
|
||||
public void recheck() {
|
||||
synchronized (lock) {
|
||||
int live = liveCount.getAsInt();
|
||||
if (live > 0 && held == null) {
|
||||
held = mechanism.acquire();
|
||||
if (held != null) {
|
||||
log.debug("idle-sleep guard armed: {} live member(s)", live);
|
||||
}
|
||||
} else if (live == 0 && held != null) {
|
||||
releaseHeldLocked();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** {@code true} while an assertion is actually held. Exposed for tests. */
|
||||
boolean isHeld() {
|
||||
synchronized (lock) {
|
||||
return held != null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Release whatever is held, if anything. Idempotent and safe to call at any time, including
|
||||
* repeatedly — a daemon shutdown hook calls this unconditionally so no assertion (and no
|
||||
* {@code caffeinate} child) survives the process, even if the drain that would otherwise have
|
||||
* driven the live count to zero was itself interrupted or threw.
|
||||
*/
|
||||
@Override
|
||||
public void close() {
|
||||
synchronized (lock) {
|
||||
if (held != null) {
|
||||
releaseHeldLocked();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Caller must hold {@link #lock}. */
|
||||
private void releaseHeldLocked() {
|
||||
try {
|
||||
held.close();
|
||||
} catch (RuntimeException e) {
|
||||
log.warn("idle-sleep guard: failed to release its assertion cleanly: {}", e.toString());
|
||||
} finally {
|
||||
held = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
/**
|
||||
* A held OS-level assertion against idle sleep. {@link #close} must be idempotent — safe to call
|
||||
* more than once — and must never throw, matching {@link IdleSleepGuard}'s "never break the
|
||||
* fleet" contract.
|
||||
*/
|
||||
public interface SleepAssertion extends AutoCloseable {
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
/**
|
||||
* The OS mechanism {@link IdleSleepGuard} uses to hold and release an idle-sleep assertion. This
|
||||
* is the seam a test exercises instead of the real effect (a live {@code caffeinate} child) — see
|
||||
* {@code IdleSleepGuardTest}.
|
||||
*
|
||||
* <p>Implementations must never throw. Every failure — wrong platform, missing tool, a spawn
|
||||
* error — must show up as {@link #acquire()} returning {@code null}, so a caller can treat "no
|
||||
* assertion held" and "the mechanism could not be used" identically and the fleet keeps running
|
||||
* either way.
|
||||
*/
|
||||
public interface SleepAssertionMechanism {
|
||||
|
||||
/**
|
||||
* Acquire a fresh assertion against idle sleep, or {@code null} when this mechanism is not
|
||||
* usable right now (wrong platform, the tool is missing, the child process could not start).
|
||||
* Never throws.
|
||||
*/
|
||||
SleepAssertion acquire();
|
||||
}
|
||||
@@ -696,6 +696,10 @@ public final class FleetApp {
|
||||
/**
|
||||
* The worker's structured reply ({@code fleet_reply}) — resolves the blocking send awaiting
|
||||
* on this session, or queues the reply in the inbox when no send is open (CB-307).
|
||||
*
|
||||
* <p>fleetd #365: the response body's {@code delivered} field used to be unconditionally
|
||||
* {@code true} for either case; it now reports whether a send/ticket was actually resolved,
|
||||
* with {@code outcome} naming which (see {@link MessageService.ReplyOutcome}).
|
||||
*/
|
||||
private void replyMessage(Context ctx) {
|
||||
String id = ctx.pathParam("id");
|
||||
@@ -719,13 +723,19 @@ public final class FleetApp {
|
||||
// a WRONG value instead of failing loudly. The check lives in MessageService.reply so both
|
||||
// this door and FleetMcp.reply inherit the same rule; this catch only translates it into the
|
||||
// {error, detail} envelope this file uses everywhere else.
|
||||
MessageService.ReplyOutcome outcome;
|
||||
try {
|
||||
messages.reply(id, content);
|
||||
outcome = messages.reply(id, content);
|
||||
} catch (IllegalArgumentException e) {
|
||||
ctx.status(400).json(Map.of("error", "bad_request", "detail", e.getMessage()));
|
||||
return;
|
||||
}
|
||||
ctx.status(200).json(Map.of("sessionId", id, "delivered", true));
|
||||
// fleetd #365: "delivered": true used to be unconditional here, whether the reply resolved
|
||||
// a waiting send or was merely queued in the inbox for a later drain — the same gap
|
||||
// FleetMcp.reply had over MCP. `delivered` now reflects which actually happened, and
|
||||
// `outcome` names the specific case (see MessageService.ReplyOutcome).
|
||||
ctx.status(200).json(Map.of("sessionId", id, "delivered", outcome.delivered(),
|
||||
"outcome", outcome.wireName()));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -108,6 +108,7 @@ class ConfigRefTopLevelReportingCoverageTest {
|
||||
v.put("worktreeGroup", "group-a");
|
||||
v.put("memberLoginShell", null);
|
||||
v.put("memberSkills", "/skills/a");
|
||||
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(true));
|
||||
assertNamesMatchComponents(v);
|
||||
return v;
|
||||
}
|
||||
@@ -149,6 +150,7 @@ class ConfigRefTopLevelReportingCoverageTest {
|
||||
v.put("worktreeGroup", "group-b");
|
||||
v.put("memberLoginShell", null);
|
||||
v.put("memberSkills", "/skills/b");
|
||||
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(false));
|
||||
assertNamesMatchComponents(v);
|
||||
return v;
|
||||
}
|
||||
|
||||
@@ -2629,4 +2629,58 @@ class FleetConfigTest {
|
||||
"with no pool to choose from, every configured profile is a candidate and the "
|
||||
+ "first one wins");
|
||||
}
|
||||
|
||||
// ── idle-sleep guard: default-on config block ───────────────────────────────────────────────
|
||||
|
||||
@Test
|
||||
void idleSleepGuardIsOnByDefaultWhenTheBlockIsEntirelyAbsent(@TempDir Path dir) throws Exception {
|
||||
Path f = dir.resolve("fleetd.yaml");
|
||||
Files.writeString(f, """
|
||||
bind:
|
||||
host: 127.0.0.1
|
||||
port: 8080
|
||||
""");
|
||||
|
||||
FleetConfig cfg = FleetConfig.load(f);
|
||||
assertNull(cfg.idleSleepGuard(), "an absent block parses to null, unlike most other blocks here");
|
||||
// The block itself is absent, but the FEATURE stays on: Fleetd treats a null block the
|
||||
// same as enabled: true (see FleetConfig.idleSleepGuard's javadoc) — this test only pins
|
||||
// the parse result, the on-by-default behaviour is Fleetd's own null check.
|
||||
}
|
||||
|
||||
@Test
|
||||
void idleSleepGuardExplicitlyEnabledIsOn(@TempDir Path dir) throws Exception {
|
||||
Path f = dir.resolve("fleetd.yaml");
|
||||
Files.writeString(f, """
|
||||
idleSleepGuard:
|
||||
enabled: true
|
||||
""");
|
||||
|
||||
FleetConfig cfg = FleetConfig.load(f);
|
||||
assertTrue(cfg.idleSleepGuard().isEnabled());
|
||||
}
|
||||
|
||||
@Test
|
||||
void idleSleepGuardExplicitlyDisabledIsOff(@TempDir Path dir) throws Exception {
|
||||
Path f = dir.resolve("fleetd.yaml");
|
||||
Files.writeString(f, """
|
||||
idleSleepGuard:
|
||||
enabled: false
|
||||
""");
|
||||
|
||||
FleetConfig cfg = FleetConfig.load(f);
|
||||
assertFalse(cfg.idleSleepGuard().isEnabled());
|
||||
}
|
||||
|
||||
@Test
|
||||
void idleSleepGuardBlockPresentButEmptyDefaultsToEnabled(@TempDir Path dir) throws Exception {
|
||||
Path f = dir.resolve("fleetd.yaml");
|
||||
Files.writeString(f, """
|
||||
idleSleepGuard: {}
|
||||
""");
|
||||
|
||||
FleetConfig cfg = FleetConfig.load(f);
|
||||
assertTrue(cfg.idleSleepGuard().isEnabled(),
|
||||
"unlike ConfigReload/Health, this block defaults to ON even when present but empty");
|
||||
}
|
||||
}
|
||||
|
||||
+2
-1
@@ -46,7 +46,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
* comments document that it only ever REPLACES a component when the incoming value is {@code null}
|
||||
* (or blank, for {@code placement}) — {@code broker}/{@code primary}/{@code leadHeartbeat}/
|
||||
* {@code configReload}/{@code coordinator}/{@code worktreeGroup}/{@code memberLoginShell}/
|
||||
* {@code memberSkills} are left as-is unconditionally, and {@code bind}/{@code guard}/{@code lifecycle}/{@code auth}/
|
||||
* {@code memberSkills}/{@code idleSleepGuard} are left as-is unconditionally, and {@code bind}/{@code guard}/{@code lifecycle}/{@code auth}/
|
||||
* {@code fleet}/{@code quarantineCooldownSeconds}/{@code memberCredentials}/{@code placement} are
|
||||
* replaced only on null/blank input. A value that is never null or blank going in must therefore
|
||||
* never change coming out, for every current component. No exclusion is needed today.
|
||||
@@ -96,6 +96,7 @@ class FleetConfigWithDefaultsPreservesEveryComponentTest {
|
||||
v.put("worktreeGroup", "group-guard");
|
||||
v.put("memberLoginShell", "/bin/zsh");
|
||||
v.put("memberSkills", "/skills/guard");
|
||||
v.put("idleSleepGuard", new FleetConfig.IdleSleepGuard(true));
|
||||
assertNamesMatchComponents(v);
|
||||
return v;
|
||||
}
|
||||
|
||||
@@ -108,8 +108,10 @@ class FleetMcpTest {
|
||||
}
|
||||
assertTrue(rendezvous.isWaiting("term_a"), "send should have opened its waiter");
|
||||
|
||||
// fleetd #365: a resolved live send must read distinctly from a merely-queued reply —
|
||||
// see replyWithNoPendingSendIsQueuedNotError below for the other case.
|
||||
McpSchema.CallToolResult reply = FleetMcp.reply(messages, "term_a", "LGTM");
|
||||
assertEquals("delivered", textOf(reply));
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND.description(), textOf(reply));
|
||||
|
||||
McpSchema.CallToolResult res = send.get(6, TimeUnit.SECONDS);
|
||||
assertNotEquals(Boolean.TRUE, res.isError());
|
||||
@@ -135,7 +137,7 @@ class FleetMcpTest {
|
||||
assertTrue(rendezvous.isWaiting("term_a"), "send should have opened its waiter");
|
||||
|
||||
McpSchema.CallToolResult reply = FleetMcp.reply(messages, "term_a", "async LGTM");
|
||||
assertEquals("delivered", textOf(reply));
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND.description(), textOf(reply));
|
||||
|
||||
// Poll until the async send completes and reports the reply.
|
||||
McpSchema.CallToolResult polled = FleetMcp.poll(messages, ticket, null);
|
||||
@@ -328,9 +330,10 @@ class FleetMcpTest {
|
||||
@Test
|
||||
void replyWithNoPendingSendIsQueuedNotError() {
|
||||
// CB-307: a reply with no open send is now queued in the inbox, not an error.
|
||||
// fleetd #365: it must also no longer claim "delivered" — nothing was waiting for it.
|
||||
McpSchema.CallToolResult res = FleetMcp.reply(messages, "term_a", "orphan");
|
||||
assertNotEquals(Boolean.TRUE, res.isError(), "a queued reply is not an error");
|
||||
assertEquals("delivered", textOf(res));
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED.description(), textOf(res));
|
||||
|
||||
// The reply is drainable by target.
|
||||
var drained = messages.drainReplies("term_a");
|
||||
@@ -413,7 +416,7 @@ class FleetMcpTest {
|
||||
}
|
||||
assertTrue(rendezvous.isWaiting("term_a"), "the answer should have reopened a waiter");
|
||||
McpSchema.CallToolResult reply = FleetMcp.reply(messages, "term_a", "done");
|
||||
assertEquals("delivered", textOf(reply));
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND.description(), textOf(reply));
|
||||
assertEquals("done", textOf(answer.get(6, TimeUnit.SECONDS)));
|
||||
}
|
||||
|
||||
|
||||
@@ -479,7 +479,8 @@ class MessageServiceTest {
|
||||
|
||||
// The worker resumes on its own (per the ask() contract) and eventually sends its real
|
||||
// fleet_reply; the async ticket must still resolve with it, not strand at PENDING.
|
||||
assertTrue(messages.reply(T, "real result"), "the worker's real reply must still be accepted");
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_ASYNC_TICKET, messages.reply(T, "real result"),
|
||||
"the worker's real reply must still be accepted, resolving the parked async ticket");
|
||||
} finally {
|
||||
messages.setAskTimeoutRaceHookForTest(null);
|
||||
}
|
||||
@@ -767,7 +768,9 @@ class MessageServiceTest {
|
||||
@Test
|
||||
void replyQueuesInInboxWhenNoSendIsOpen() {
|
||||
// No send is open for this session — reply should queue in the inbox.
|
||||
assertTrue(messages.reply(T, "queued-text"), "reply should succeed (queued)");
|
||||
// fleetd #365: this is the case that must read as QUEUED, not "delivered".
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED, messages.reply(T, "queued-text"),
|
||||
"reply should succeed but only as queued — nothing was waiting for it");
|
||||
|
||||
var drained = messages.drainReplies(T);
|
||||
assertEquals(1, drained.size());
|
||||
@@ -780,7 +783,9 @@ class MessageServiceTest {
|
||||
awaitUninterruptibly(T);
|
||||
|
||||
// An explicit reply resolves the open send.
|
||||
assertTrue(messages.reply(T, "send-resolved"), "reply should succeed (resolved live send)");
|
||||
// fleetd #365: this is the other case — RESOLVED_SEND, distinct from QUEUED above.
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND, messages.reply(T, "send-resolved"),
|
||||
"reply should succeed by resolving the live waiting send");
|
||||
|
||||
// The inbox should be empty — the reply went to the send, not the inbox.
|
||||
assertTrue(messages.drainReplies(T).isEmpty(), "no reply in the inbox");
|
||||
@@ -1094,8 +1099,11 @@ class MessageServiceTest {
|
||||
assertEquals(MessageService.Outcome.TIMED_OUT_WORKING, answerReply.outcome(),
|
||||
"the primary's own bounded wait gives up before the worker finishes resuming");
|
||||
|
||||
// The worker keeps working past that window and only now calls fleet_reply.
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// The worker keeps working past that window and only now calls fleet_reply. The forward
|
||||
// waiter answer() opened already timed out, so this resolves via the parked async ticket,
|
||||
// not a live send (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_ASYNC_TICKET,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
MessageService.TaskView done = awaitTicketPhase(ticket, MessageService.Phase.DONE);
|
||||
assertEquals("PR opened: https://example/pulls/42", done.reply(),
|
||||
@@ -1119,7 +1127,10 @@ class MessageServiceTest {
|
||||
assertEquals("config.yaml", ask.get(5, TimeUnit.SECONDS).answer());
|
||||
assertEquals(MessageService.Outcome.TIMED_OUT_WORKING, answerReply.outcome());
|
||||
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// No live waiter (answer()'s own forward wait already timed out) — resolves the parked
|
||||
// async ticket instead (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_ASYNC_TICKET,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
// fleet_stop tears the worker's session down right after the reply landed — this must never
|
||||
// report the misleading "the worker session was released before it replied": a reply is
|
||||
@@ -1170,7 +1181,9 @@ class MessageServiceTest {
|
||||
assertEquals("config.yaml", ask.get(5, TimeUnit.SECONDS).answer());
|
||||
awaitWaiting(); // answer() opened its own forward waiter for the resumed worker turn
|
||||
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// A live waiter is open (the forward wait above) — this resolves it directly (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
assertEquals(MessageService.Outcome.REPLIED, answer.get(5, TimeUnit.SECONDS).outcome(),
|
||||
"the lead's own answer() call must not throw because ask()'s timeout cleanup raced it");
|
||||
@@ -1221,8 +1234,9 @@ class MessageServiceTest {
|
||||
|
||||
// The worker keeps working past the timeout and only now calls fleet_reply — with no live
|
||||
// rendezvous waiter open (ask()'s timeout already closed it) and no new send() having
|
||||
// reopened one for this target.
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// reopened one for this target. So it resolves the parked async ticket (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_ASYNC_TICKET,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
MessageService.TaskView done = awaitTicketPhase(ticket, MessageService.Phase.DONE);
|
||||
assertEquals("PR opened: https://example/pulls/42", done.reply(),
|
||||
@@ -1279,7 +1293,10 @@ class MessageServiceTest {
|
||||
// real reply — reproduce that interleaving directly instead of trying to win a real race.
|
||||
messages.forgetTurnForTest(turnId);
|
||||
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// answer() is still waiting on its own forward waiter for the resumed turn — a live send —
|
||||
// so this resolves it directly, not the async ticket (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
assertEquals(MessageService.Outcome.REPLIED, answer.get(5, TimeUnit.SECONDS).outcome(),
|
||||
"the primary's own answer() call must still see the worker's real reply");
|
||||
@@ -1321,7 +1338,9 @@ class MessageServiceTest {
|
||||
|
||||
messages.setReplyOrphanTurnIdRaceHookForTest(() -> messages.forgetTurnForTest(turnId));
|
||||
try {
|
||||
assertTrue(messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
// No live waiter — resolves the parked async ticket (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_ASYNC_TICKET,
|
||||
messages.reply(T, "PR opened: https://example/pulls/42"));
|
||||
|
||||
MessageService.TaskView done = awaitTicketPhase(ticket, MessageService.Phase.DONE);
|
||||
assertEquals("PR opened: https://example/pulls/42", done.reply(),
|
||||
@@ -1353,7 +1372,8 @@ class MessageServiceTest {
|
||||
injectDelivery();
|
||||
assertEquals(MessageService.AskOutcome.TIMED_OUT, messages.ask(T, "Q2?", 200).outcome());
|
||||
|
||||
assertTrue(messages.reply(T, "which task does this answer?"));
|
||||
// Ambiguous — two candidates, so it must fall back to the inbox rather than guess (fleetd #365).
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED, messages.reply(T, "which task does this answer?"));
|
||||
|
||||
assertEquals(MessageService.Phase.PENDING, messages.poll(ticket1).phase(),
|
||||
"an ambiguous reply must not guess ticket1");
|
||||
@@ -2032,7 +2052,7 @@ class MessageServiceTest {
|
||||
CompletableFuture<MessageService.Reply> send = sendAsync();
|
||||
awaitWaiting();
|
||||
|
||||
assertTrue(messages.reply(T, "resolved-live"));
|
||||
assertEquals(MessageService.ReplyOutcome.RESOLVED_SEND, messages.reply(T, "resolved-live"));
|
||||
assertFalse(messages.hasStrandedReply(T), "a reply that resolved an open send is not stranded");
|
||||
|
||||
MessageService.Reply r = send.get(5, TimeUnit.SECONDS);
|
||||
@@ -2042,14 +2062,14 @@ class MessageServiceTest {
|
||||
@Test
|
||||
void hasStrandedReplyIsTrueWhenNoSendWasWaiting() {
|
||||
// No send is open for T — the reply queues into the inbox and is recorded as stranded.
|
||||
assertTrue(messages.reply(T, "nobody was waiting"));
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED, messages.reply(T, "nobody was waiting"));
|
||||
assertTrue(messages.hasStrandedReply(T),
|
||||
"a reply with no open send strands, even though it is safely queued in the inbox");
|
||||
}
|
||||
|
||||
@Test
|
||||
void hasStrandedReplyClearsOnceTheTargetsNextDeliveryIsAccepted() throws Exception {
|
||||
assertTrue(messages.reply(T, "stray"));
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED, messages.reply(T, "stray"));
|
||||
assertTrue(messages.hasStrandedReply(T));
|
||||
|
||||
// The next accepted delivery for T clears the stale stranding fact — the one case the
|
||||
@@ -2067,7 +2087,7 @@ class MessageServiceTest {
|
||||
|
||||
@Test
|
||||
void hasStrandedReplyClearsOnAbandon() {
|
||||
assertTrue(messages.reply(T, "stray"));
|
||||
assertEquals(MessageService.ReplyOutcome.QUEUED, messages.reply(T, "stray"));
|
||||
assertTrue(messages.hasStrandedReply(T));
|
||||
|
||||
messages.abandon(T, "session released");
|
||||
|
||||
@@ -4,6 +4,7 @@ import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import dev.ltms.fleet.herdr.AgentControl;
|
||||
import dev.ltms.fleet.herdr.HerdrClient;
|
||||
import dev.ltms.fleet.herdr.HerdrException;
|
||||
import dev.ltms.fleet.mcp.PrimaryRegistry;
|
||||
import dev.ltms.fleet.metrics.FleetMetrics;
|
||||
import dev.ltms.fleet.metrics.Metrics;
|
||||
@@ -44,6 +45,7 @@ class ReplyPushLoopTest {
|
||||
private static final String WORKER = "term_worker";
|
||||
private static final String WORKER2 = "term_worker2";
|
||||
private static final String OTHER_PRIMARY = "term_other_primary";
|
||||
private static final String DEAD_LEAD = "term_dead_lead";
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
private PrimaryRegistry registry;
|
||||
@@ -207,6 +209,101 @@ class ReplyPushLoopTest {
|
||||
"exactly " + cap + " agent.prompt calls (cap=" + cap + ")");
|
||||
}
|
||||
|
||||
// --- fleetd #368: a lead's delegation binding must not outlive the lead ---------------------
|
||||
|
||||
/**
|
||||
* The bug: {@code PrimaryRegistry.forgetDelegation} is wired to a worker's release, never to
|
||||
* the delegating lead's own disappearance, so a lead that closed, crashed, or was relaunched
|
||||
* leaves {@code leadByTarget} pointing at a terminal herdr no longer knows. Before the fix,
|
||||
* {@code onReplyQueued} took that stale, non-null entry at face value — {@code nudgeTargetFor}
|
||||
* only ever falls back to the pinned primary when the map holds nothing for the target — so
|
||||
* the nudge's only schedule ran against the dead terminal forever and the live primary never
|
||||
* heard about the reply through this path.
|
||||
*
|
||||
* <p>This drives {@link ReplyPushLoop#onReplyQueued(String)} itself (not {@code PrimaryRegistry}
|
||||
* directly), because the registry lookup was never the defect — the caller trusting it without
|
||||
* checking liveness was. A test that only asserted on {@code PrimaryRegistry.nudgeTargetFor}
|
||||
* would pass whether or not {@code ReplyPushLoop} ever adopted the fix.
|
||||
*/
|
||||
@Test
|
||||
void aStaleLeadBindingFallsBackToTheLiveLeadInsteadOfNudgingADeadTerminal() throws Exception {
|
||||
// PRIMARY is the single known (pinned) lead — set up in @BeforeEach via `registry`.
|
||||
// DEAD_LEAD is a second lead that once delegated to WORKER and is now gone: herdr reports
|
||||
// agent_not_found for it, exactly as it would for a closed/crashed/relaunched terminal.
|
||||
registry.recordDelegation(WORKER, DEAD_LEAD);
|
||||
|
||||
var rec = new DeadLeadHerdrClient(DEAD_LEAD);
|
||||
agents = new AgentControl(rec);
|
||||
inbox.publish(WORKER, "m1", "hello");
|
||||
|
||||
loop(1, 50).onReplyQueued(WORKER);
|
||||
|
||||
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS),
|
||||
"the nudge should still reach the live primary, not silently vanish with the dead lead");
|
||||
assertEquals(List.of(PRIMARY), rec.promptTargets(),
|
||||
"the nudge must be sent to the live primary, never to the dead lead's terminal");
|
||||
assertEquals(PRIMARY, registry.nudgeTargetFor(WORKER).orElseThrow(),
|
||||
"the stale binding must be forgotten (self-healed) once found dead, exactly like "
|
||||
+ "AgentControl.paneByTerminal already does on agent_not_found");
|
||||
}
|
||||
|
||||
/**
|
||||
* Same dead binding, but with no pinned primary to fall back to (the multi-lead, no-fallback
|
||||
* case {@code PrimaryRegistry.nudgeTargetFor}'s own javadoc already covers): the loop must
|
||||
* never nudge the dead terminal, and must not spin — no schedule starts at all once the stale
|
||||
* binding resolves to empty, same as if the map had never held an entry for this target.
|
||||
*/
|
||||
@Test
|
||||
void aStaleLeadBindingWithNoFallbackNeverNudgesTheDeadTerminal() throws Exception {
|
||||
var unpinned = new PrimaryRegistry(null);
|
||||
unpinned.recordDelegation(WORKER, DEAD_LEAD);
|
||||
|
||||
var rec = new DeadLeadHerdrClient(DEAD_LEAD);
|
||||
agents = new AgentControl(rec);
|
||||
inbox.publish(WORKER, "m1", "hello");
|
||||
|
||||
var loop = new ReplyPushLoop(unpinned, agents, inbox, scheduler, 1, 50);
|
||||
loop.onReplyQueued(WORKER);
|
||||
|
||||
Thread.sleep(200);
|
||||
assertEquals(0, rec.sendCount(), "no lead is live to nudge, so nothing should ever be sent");
|
||||
assertTrue(unpinned.nudgeTargetFor(WORKER).isEmpty(),
|
||||
"the stale binding must be forgotten even when there is no fallback to hand back");
|
||||
}
|
||||
|
||||
/**
|
||||
* fleetd #368 review, must-fix: the first version of {@code isLive} treated <em>any</em>
|
||||
* {@code RuntimeException} from the liveness probe as "the lead is gone" — indistinguishable
|
||||
* from a transient herdr hiccup (a socket blip, a decode error) on a lead that is actually
|
||||
* still live. The consequence of that misdiagnosis is destructive and permanent
|
||||
* ({@code forgetDelegation}), which is the exact #359 mistake repeated two days later: a single
|
||||
* bad reading must never destroy a live binding. This pins the narrower rule — only an
|
||||
* affirmative {@code agent_not_found} may forget a binding; a merely inconclusive failure must
|
||||
* leave the binding alone, and the lead must still be nudged once the probe recovers.
|
||||
*/
|
||||
@Test
|
||||
void aTransientLivenessFailureMustNotForgetABindingToAStillLiveLead() throws Exception {
|
||||
// OTHER_PRIMARY is delegated to and genuinely live — its FIRST agent.get call fails with a
|
||||
// transient, non-agent_not_found HerdrException (a transport-level failure, code null,
|
||||
// exactly what a socket blip looks like), then succeeds on every call after.
|
||||
registry.recordDelegation(WORKER, OTHER_PRIMARY);
|
||||
|
||||
var rec = new FlakyThenLiveHerdrClient(OTHER_PRIMARY);
|
||||
agents = new AgentControl(rec);
|
||||
inbox.publish(WORKER, "m1", "hello");
|
||||
|
||||
loop(2, 50).onReplyQueued(WORKER);
|
||||
|
||||
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS),
|
||||
"the nudge must still reach the live lead once the transient failure clears");
|
||||
assertEquals(List.of(OTHER_PRIMARY), rec.promptTargets(),
|
||||
"the nudge must go to the lead that was only transiently unreachable, not the "
|
||||
+ "unrelated pinned primary");
|
||||
assertEquals(OTHER_PRIMARY, registry.nudgeTargetFor(WORKER).orElseThrow(),
|
||||
"a merely transient failure must not forget the binding to a lead that is actually "
|
||||
+ "still live");
|
||||
}
|
||||
|
||||
// --- nudge format --------------------------------------------------------------------------
|
||||
|
||||
@Test
|
||||
@@ -887,7 +984,7 @@ class ReplyPushLoopTest {
|
||||
// --- metrics (CB-512) ----------------------------------------------------------------------
|
||||
|
||||
@Test
|
||||
void successfulNudgeIncrementsDelivered() throws Exception {
|
||||
void successfulNudgeIncrementsSent() throws Exception {
|
||||
var rec = recordingClient();
|
||||
agents = new AgentControl(rec);
|
||||
inbox.publish(WORKER, "m1", "hello");
|
||||
@@ -897,11 +994,13 @@ class ReplyPushLoopTest {
|
||||
|
||||
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS),
|
||||
"one nudge (1 agent.prompt call) should have been sent");
|
||||
// The delivered count is bumped on the scheduler thread right after the send that releases
|
||||
// The sent count is bumped on the scheduler thread right after the send that releases
|
||||
// the latch — settle briefly so the counter is published before we read it.
|
||||
Thread.sleep(200);
|
||||
assertEquals(1, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "delivered"),
|
||||
"a successfully sent nudge must count as delivered");
|
||||
// fleetd #365: "sent", not "delivered" — this only proves the herdr call succeeded, not
|
||||
// that the primary's pane read it.
|
||||
assertEquals(1, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "sent"),
|
||||
"a successfully sent nudge must count as sent");
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -916,11 +1015,11 @@ class ReplyPushLoopTest {
|
||||
|
||||
assertEquals(1, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "exhausted"),
|
||||
"hitting the reminder cap must count as exhausted");
|
||||
assertEquals(0, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "delivered"));
|
||||
assertEquals(0, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "sent"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void successfulTicketNudgeIncrementsDelivered() throws Exception {
|
||||
void successfulTicketNudgeIncrementsSent() throws Exception {
|
||||
var rec = recordingClient();
|
||||
agents = new AgentControl(rec);
|
||||
Metrics metrics = new Metrics();
|
||||
@@ -929,8 +1028,8 @@ class ReplyPushLoopTest {
|
||||
|
||||
assertTrue(rec.sendLatch.await(3, TimeUnit.SECONDS), "one ticket nudge should have been sent");
|
||||
Thread.sleep(200);
|
||||
assertEquals(1, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "delivered"),
|
||||
"a successfully sent ticket nudge must count as delivered, same metric as CB-307");
|
||||
assertEquals(1, metrics.count(FleetMetrics.PUSH_NUDGES, "outcome", "sent"),
|
||||
"a successfully sent ticket nudge must count as sent, same metric as CB-307");
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -1098,4 +1197,101 @@ class ReplyPushLoopTest {
|
||||
public void close() {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fake herdr client for fleetd #368: {@code deadTarget} is a terminal herdr genuinely no
|
||||
* longer knows about — {@code agent.get} fails with {@code agent_not_found} exactly as
|
||||
* {@code AgentControl.agentCall} expects for a real dead/closed pane (see its javadoc). Every
|
||||
* other target reports {@code idle} (injectable). Records the {@code target} named by every
|
||||
* {@code agent.prompt} call, so a test can prove which terminal actually got nudged.
|
||||
*/
|
||||
private static final class DeadLeadHerdrClient implements HerdrClient {
|
||||
private final String deadTarget;
|
||||
private final List<String> promptTargets = Collections.synchronizedList(new ArrayList<>());
|
||||
volatile CountDownLatch sendLatch = new CountDownLatch(1);
|
||||
|
||||
DeadLeadHerdrClient(String deadTarget) {
|
||||
this.deadTarget = deadTarget;
|
||||
}
|
||||
|
||||
@Override
|
||||
@SuppressWarnings("unchecked")
|
||||
public JsonNode call(String method, Object params) {
|
||||
Map<String, Object> p = params instanceof Map ? (Map<String, Object>) params : Map.of();
|
||||
if ("agent.get".equals(method)) {
|
||||
String target = String.valueOf(p.get("target"));
|
||||
if (deadTarget.equals(target)) {
|
||||
throw new HerdrException("no such agent: " + target, "agent_not_found", null);
|
||||
}
|
||||
return MAPPER.createObjectNode()
|
||||
.set("agent", MAPPER.createObjectNode()
|
||||
.put("terminal_id", target)
|
||||
.put("agent_status", "idle"));
|
||||
}
|
||||
if ("agent.prompt".equals(method)) {
|
||||
promptTargets.add(String.valueOf(p.get("target")));
|
||||
sendLatch.countDown();
|
||||
}
|
||||
return MAPPER.createObjectNode();
|
||||
}
|
||||
|
||||
List<String> promptTargets() {
|
||||
return List.copyOf(promptTargets);
|
||||
}
|
||||
|
||||
long sendCount() {
|
||||
return promptTargets.size();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fake herdr client for fleetd #368 review: {@code flakyTarget}'s FIRST {@code agent.get} call
|
||||
* fails with a transient, non-{@code agent_not_found} {@code HerdrException} — a transport-level
|
||||
* failure (code {@code null}), exactly what a socket blip or a decode error on a perfectly live
|
||||
* lead looks like — then succeeds ({@code idle}) on every call after. Used to prove a merely
|
||||
* inconclusive failure must not be treated as the lead being gone.
|
||||
*/
|
||||
private static final class FlakyThenLiveHerdrClient implements HerdrClient {
|
||||
private final String flakyTarget;
|
||||
private final AtomicInteger getCalls = new AtomicInteger();
|
||||
private final List<String> promptTargets = Collections.synchronizedList(new ArrayList<>());
|
||||
volatile CountDownLatch sendLatch = new CountDownLatch(1);
|
||||
|
||||
FlakyThenLiveHerdrClient(String flakyTarget) {
|
||||
this.flakyTarget = flakyTarget;
|
||||
}
|
||||
|
||||
@Override
|
||||
@SuppressWarnings("unchecked")
|
||||
public JsonNode call(String method, Object params) {
|
||||
Map<String, Object> p = params instanceof Map ? (Map<String, Object>) params : Map.of();
|
||||
if ("agent.get".equals(method)) {
|
||||
String target = String.valueOf(p.get("target"));
|
||||
if (flakyTarget.equals(target) && getCalls.getAndIncrement() == 0) {
|
||||
throw new HerdrException("herdr socket read timed out"); // transport failure, code == null
|
||||
}
|
||||
return MAPPER.createObjectNode()
|
||||
.set("agent", MAPPER.createObjectNode()
|
||||
.put("terminal_id", target)
|
||||
.put("agent_status", "idle"));
|
||||
}
|
||||
if ("agent.prompt".equals(method)) {
|
||||
promptTargets.add(String.valueOf(p.get("target")));
|
||||
sendLatch.countDown();
|
||||
}
|
||||
return MAPPER.createObjectNode();
|
||||
}
|
||||
|
||||
List<String> promptTargets() {
|
||||
return List.copyOf(promptTargets);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* Platform-detection unit tests for {@link CaffeinateSleepAssertionMechanism}.
|
||||
*
|
||||
* <p>This deliberately never calls {@link CaffeinateSleepAssertionMechanism#acquire()} itself —
|
||||
* doing so on a real macOS machine would actually start a live {@code caffeinate} child and hold
|
||||
* a real idle-sleep assertion, which the ticket this class exists for explicitly forbids testing
|
||||
* with. Instead this exercises the pure {@code isSupportedPlatform(String)} predicate that
|
||||
* {@code acquire()} consults before ever touching {@link ProcessBuilder} — so it proves the
|
||||
* platform check itself is correct on any CI OS, but it does <strong>not</strong> prove that a
|
||||
* real {@code caffeinate -i} spawn succeeds or that its child is torn down correctly; that half is
|
||||
* exercised indirectly by {@link IdleSleepGuardTest} against a {@link FakeSleepAssertionMechanism}
|
||||
* instead, which is the seam invariant 2/3 in the ticket call for.
|
||||
*/
|
||||
class CaffeinateSleepAssertionMechanismTest {
|
||||
|
||||
@Test
|
||||
void macOsNamesAreSupported() {
|
||||
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Mac OS X"));
|
||||
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("macOS"));
|
||||
assertTrue(CaffeinateSleepAssertionMechanism.isSupportedPlatform("MAC OS X"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void nonMacNamesAreNotSupported() {
|
||||
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Linux"));
|
||||
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform("Windows 11"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void nullOsNameIsNotSupported() {
|
||||
assertFalse(CaffeinateSleepAssertionMechanism.isSupportedPlatform(null));
|
||||
}
|
||||
|
||||
/**
|
||||
* The overload {@code isSupportedPlatform()} (no args) reads the JVM's real {@code os.name} —
|
||||
* proves the wiring is live, without asserting a specific answer (this suite itself must pass
|
||||
* on both macOS and Linux CI).
|
||||
*/
|
||||
@Test
|
||||
void noArgOverloadReadsRealSystemProperty() {
|
||||
boolean expected = CaffeinateSleepAssertionMechanism
|
||||
.isSupportedPlatform(System.getProperty("os.name"));
|
||||
boolean actual = CaffeinateSleepAssertionMechanism.isSupportedPlatform();
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import java.util.concurrent.CopyOnWriteArrayList;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
/**
|
||||
* Recording fake {@link SleepAssertionMechanism} — the seam behind the real OS effect (a live
|
||||
* {@code caffeinate} child process). No test in this package ever spawns that real process; every
|
||||
* assertion here is against this fake's own call log instead.
|
||||
*
|
||||
* <p>Each acquired {@link FakeAssertion} records its own {@code close()} calls, and every
|
||||
* acquired instance is kept in {@link #acquired} so a test can inspect all of them, including
|
||||
* ones {@link IdleSleepGuard} has already released.
|
||||
*/
|
||||
final class FakeSleepAssertionMechanism implements SleepAssertionMechanism {
|
||||
|
||||
/** Every {@link FakeAssertion} this mechanism has ever handed out, in order. */
|
||||
final CopyOnWriteArrayList<FakeAssertion> acquired = new CopyOnWriteArrayList<>();
|
||||
|
||||
private final AtomicInteger acquireCalls = new AtomicInteger();
|
||||
private volatile boolean unavailable = false;
|
||||
|
||||
/** Make the next (and every subsequent) {@link #acquire()} return {@code null}, like a missing tool. */
|
||||
void makeUnavailable() {
|
||||
unavailable = true;
|
||||
}
|
||||
|
||||
int acquireCallCount() {
|
||||
return acquireCalls.get();
|
||||
}
|
||||
|
||||
@Override
|
||||
public SleepAssertion acquire() {
|
||||
acquireCalls.incrementAndGet();
|
||||
if (unavailable) {
|
||||
return null;
|
||||
}
|
||||
FakeAssertion a = new FakeAssertion();
|
||||
acquired.add(a);
|
||||
return a;
|
||||
}
|
||||
|
||||
/** A held fake assertion; records how many times {@code close()} was actually called. */
|
||||
static final class FakeAssertion implements SleepAssertion {
|
||||
private final AtomicInteger closeCalls = new AtomicInteger();
|
||||
|
||||
int closeCallCount() {
|
||||
return closeCalls.get();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
closeCalls.incrementAndGet();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* {@link IdleSleepGuard} against a {@link FakeSleepAssertionMechanism} — the seam that stands in
|
||||
* for a real {@code caffeinate} child process. No test in this class ever spawns a real OS
|
||||
* process or asserts against real idle sleep; every assertion is against the fake's call log
|
||||
* (how many times {@code acquire()}/{@code close()} were actually called). That proves the
|
||||
* <em>orchestration</em> — when the guard decides to hold or release an assertion, and that it
|
||||
* never throws — but it does <strong>not</strong> prove that {@code caffeinate -i} itself
|
||||
* actually stops macOS from idle-sleeping; that half is outside what a unit test can safely
|
||||
* exercise (see {@link CaffeinateSleepAssertionMechanismTest}'s class doc).
|
||||
*/
|
||||
class IdleSleepGuardTest {
|
||||
|
||||
@Test
|
||||
void acquiresOnZeroToOneAndReleasesOnOneToZero() {
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
AtomicInteger liveCount = new AtomicInteger(0);
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
|
||||
|
||||
assertFalse(guard.isHeld(), "nothing held before any member is live");
|
||||
|
||||
liveCount.set(1);
|
||||
guard.recheck();
|
||||
assertTrue(guard.isHeld(), "an assertion must be held once a member is live");
|
||||
assertEquals(1, mechanism.acquired.size());
|
||||
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
|
||||
|
||||
liveCount.set(0);
|
||||
guard.recheck();
|
||||
assertFalse(guard.isHeld(), "the assertion must be released once the last member goes");
|
||||
assertEquals(1, mechanism.acquired.get(0).closeCallCount(), "the SAME held assertion must be closed");
|
||||
}
|
||||
|
||||
@Test
|
||||
void steadyLiveCountDoesNotReacquireOrRerelease() {
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
AtomicInteger liveCount = new AtomicInteger(2);
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
|
||||
|
||||
guard.recheck(); // 0 -> 2 crossing: acquires
|
||||
guard.recheck(); // still 2: must be a no-op
|
||||
guard.recheck(); // still 2: must be a no-op
|
||||
assertEquals(1, mechanism.acquireCallCount(), "only the crossing touches the mechanism");
|
||||
|
||||
liveCount.set(1); // 2 -> 1: still > 0, still a no-op
|
||||
guard.recheck();
|
||||
assertTrue(guard.isHeld());
|
||||
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
|
||||
assertEquals(1, mechanism.acquireCallCount());
|
||||
}
|
||||
|
||||
/**
|
||||
* Invariant 2: a missing/unavailable mechanism must never throw, and the guard must simply
|
||||
* hold nothing. {@link FakeSleepAssertionMechanism#makeUnavailable()} makes {@code acquire()}
|
||||
* return {@code null}, exactly like {@link CaffeinateSleepAssertionMechanism} does off macOS
|
||||
* or when the {@code caffeinate} binary is missing.
|
||||
*/
|
||||
@Test
|
||||
void unavailableMechanismNeverThrowsAndHoldsNothing() {
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
mechanism.makeUnavailable();
|
||||
AtomicInteger liveCount = new AtomicInteger(1);
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
|
||||
|
||||
guard.recheck(); // must not throw
|
||||
assertFalse(guard.isHeld(), "acquire() returned null, so nothing is held");
|
||||
assertEquals(1, mechanism.acquireCallCount());
|
||||
|
||||
// still must not throw or leak on release, even though nothing was ever actually held
|
||||
liveCount.set(0);
|
||||
guard.recheck();
|
||||
assertFalse(guard.isHeld());
|
||||
|
||||
guard.close(); // teardown with nothing held must also be a safe no-op
|
||||
}
|
||||
|
||||
/**
|
||||
* Invariant 3 (teardown). This is the test the mutation testing step removes the production
|
||||
* release call to fail: with {@code releaseHeldLocked()} not invoked from {@link
|
||||
* IdleSleepGuard#close()}, the held fake assertion's {@code close()} would never be called and
|
||||
* this assertion would fail.
|
||||
*/
|
||||
@Test
|
||||
void closeReleasesAHeldAssertionEvenWithoutAZeroCrossing() {
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
AtomicInteger liveCount = new AtomicInteger(1);
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
|
||||
|
||||
guard.recheck();
|
||||
assertTrue(guard.isHeld());
|
||||
|
||||
guard.close();
|
||||
|
||||
assertFalse(guard.isHeld(), "close() must release whatever is held, independent of live count");
|
||||
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
|
||||
}
|
||||
|
||||
@Test
|
||||
void closeIsIdempotent() {
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
AtomicInteger liveCount = new AtomicInteger(1);
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, liveCount::get);
|
||||
|
||||
guard.recheck();
|
||||
guard.close();
|
||||
guard.close(); // must not throw, must not double-release
|
||||
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package dev.ltms.fleet.power;
|
||||
|
||||
import dev.ltms.fleet.config.FleetConfig;
|
||||
import dev.ltms.fleet.guard.SubscriptionGuard;
|
||||
import dev.ltms.fleet.herdr.AgentControl;
|
||||
import dev.ltms.fleet.herdr.FakeHerdr;
|
||||
import dev.ltms.fleet.herdr.WorkspaceControl;
|
||||
import dev.ltms.fleet.member.ClaudeCodeLauncher;
|
||||
import dev.ltms.fleet.session.MemberSession;
|
||||
import dev.ltms.fleet.session.SessionManager;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
/**
|
||||
* Proves the wiring {@code Fleetd.main} actually performs — {@code
|
||||
* sessions.onAcquire(_ -> guard.recheck())} / {@code sessions.onRelease(_ -> guard.recheck())} —
|
||||
* not just {@link IdleSleepGuard}'s own orchestration logic in isolation
|
||||
* ({@link IdleSleepGuardTest} already covers that in isolation, which on its own would not catch
|
||||
* a wiring gap — e.g. an {@code onAcquire} call typo'd to a no-op lambda, or the listener wired to
|
||||
* the wrong SessionManager instance — see fleetd's own "a test on the seam does not prove the
|
||||
* caller" lesson). This test builds a real {@link SessionManager} exactly as
|
||||
* {@code SessionManagerTest} does (a {@link FakeHerdr}-backed {@link ClaudeCodeLauncher}, no live
|
||||
* herdr process), wires it to an {@link IdleSleepGuard} the same two lines {@code Fleetd.main}
|
||||
* uses, and drives real {@link SessionManager#acquire} / {@link SessionManager#release} calls.
|
||||
*/
|
||||
class IdleSleepGuardWiringTest {
|
||||
|
||||
private SessionManager sessionManager(FakeHerdr herdr) {
|
||||
FleetConfig.Profile cfg = new FleetConfig.Profile(
|
||||
"ltms-local", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN",
|
||||
List.of("ccs", "ltms-local"), "tab", "fleetd-workers",
|
||||
"worker: {profile} #{n}", null, null, null);
|
||||
ClaudeCodeLauncher workers = new ClaudeCodeLauncher(new AgentControl(herdr), new WorkspaceControl(herdr),
|
||||
new SubscriptionGuard(Set.of("gx00.gw")), Map.of(cfg.profile(), cfg), cfg.profile(), _ -> null);
|
||||
return new SessionManager(workers);
|
||||
}
|
||||
|
||||
@Test
|
||||
void acquiringAndReleasingRealSessionsDrivesTheGuardThroughTheSameWiringFleetdUses() {
|
||||
FakeHerdr herdr = new FakeHerdr();
|
||||
SessionManager sessions = sessionManager(herdr);
|
||||
FakeSleepAssertionMechanism mechanism = new FakeSleepAssertionMechanism();
|
||||
IdleSleepGuard guard = new IdleSleepGuard(mechanism, sessions::size);
|
||||
|
||||
// The exact two lines Fleetd.main wires up.
|
||||
sessions.onAcquire(_ -> guard.recheck());
|
||||
sessions.onRelease(_ -> guard.recheck());
|
||||
|
||||
assertFalse(guard.isHeld(), "no member yet: nothing held");
|
||||
|
||||
MemberSession a = sessions.acquire("ltms-local", "/a", "/caller", "ownerA");
|
||||
assertTrue(guard.isHeld(), "0 -> 1: the first live member must arm the guard");
|
||||
|
||||
MemberSession b = sessions.acquire("ltms-local", "/b", "/caller", "ownerB");
|
||||
assertEquals(1, mechanism.acquireCallCount(), "2nd member: still just 1 live-to-2 step, no new acquire");
|
||||
|
||||
sessions.release(a.paneId());
|
||||
assertTrue(guard.isHeld(), "one member still live: the guard must stay armed");
|
||||
assertEquals(0, mechanism.acquired.get(0).closeCallCount());
|
||||
|
||||
sessions.release(b.paneId());
|
||||
assertFalse(guard.isHeld(), "1 -> 0: the last member releasing must disarm the guard");
|
||||
assertEquals(1, mechanism.acquired.get(0).closeCallCount());
|
||||
}
|
||||
}
|
||||
@@ -476,6 +476,11 @@ class FleetAppTest {
|
||||
Thread.sleep(200);
|
||||
HttpResponse<String> reply = postJson(port, "/sessions/term_a/reply", "{\"content\":\"LGTM ship it\"}");
|
||||
assertEquals(200, reply.statusCode());
|
||||
// fleetd #365: "delivered" used to be unconditionally true; a live send was actually waiting
|
||||
// here, so this is the case where it must genuinely read true, with outcome naming why.
|
||||
JsonNode replyBody = mapper.readTree(reply.body());
|
||||
assertEquals(true, replyBody.get("delivered").asBoolean());
|
||||
assertEquals("resolved_send", replyBody.get("outcome").asText());
|
||||
|
||||
HttpResponse<String> res = send.get(6, java.util.concurrent.TimeUnit.SECONDS);
|
||||
assertEquals(200, res.statusCode());
|
||||
@@ -527,6 +532,11 @@ class FleetAppTest {
|
||||
int port = startHealthy();
|
||||
HttpResponse<String> res = postJson(port, "/sessions/term_a/reply", "{\"content\":\"orphan\"}");
|
||||
assertEquals(200, res.statusCode());
|
||||
// fleetd #365: nothing was waiting, so "delivered" must now read false, not the old
|
||||
// unconditional true — outcome names this as queued.
|
||||
JsonNode resBody = mapper.readTree(res.body());
|
||||
assertEquals(false, resBody.get("delivered").asBoolean());
|
||||
assertEquals("queued", resBody.get("outcome").asText());
|
||||
|
||||
// The queued reply is drainable.
|
||||
HttpResponse<String> drain = req(port, "GET", "/sessions/term_a/replies");
|
||||
|
||||
@@ -54,6 +54,56 @@ class GitWorktreesTest {
|
||||
/** A non-empty autoenv file — the form that would prompt for authorization in a worktree. */
|
||||
private static final String AUTOENV_WITH_DIRECTIVE = "export HELLO=world\n";
|
||||
|
||||
/**
|
||||
* fleetd #369. A throwaway directory that lives for the whole class (JUnit 5.4+ supports a
|
||||
* static {@code @TempDir} field, created once and removed once every test in this class has
|
||||
* run) — backing every raw {@code git} subprocess's {@code XDG_CONFIG_HOME} below. It only
|
||||
* ever needs to exist and be guaranteed free of a {@code git/ignore} file; nothing writes
|
||||
* inside it.
|
||||
*/
|
||||
@TempDir
|
||||
private static Path CLASS_TMP;
|
||||
|
||||
/**
|
||||
* fleetd #369 — the leak measured: {@code XDG_CONFIG_HOME=<dir with a `*` git/ignore> mvn test
|
||||
* -Dtest=GitWorktreesTest} failed 56 of 59 tests on an unpatched checkout, because {@link
|
||||
* #gitOutput} set {@code GIT_CONFIG_GLOBAL}/{@code GIT_CONFIG_SYSTEM}/{@code
|
||||
* GIT_TERMINAL_PROMPT} but not {@code XDG_CONFIG_HOME}, and {@link #status}/{@link
|
||||
* #fullStatus} (plus every other raw {@code git} subprocess this class started) set NOTHING at
|
||||
* all — inheriting the JVM's whole real environment, including the operator's real {@code
|
||||
* ~/.gitconfig} and real default excludes file ({@code $XDG_CONFIG_HOME/git/ignore} or {@code
|
||||
* $HOME/.config/git/ignore}, applied by git with no {@code core.excludesFile} configured at
|
||||
* all — see {@code gitignore(5)}). {@code GIT_CONFIG_GLOBAL=/dev/null} does not stop that
|
||||
* default from applying; only setting {@code XDG_CONFIG_HOME} to a directory that provably
|
||||
* carries no {@code git/ignore} does.
|
||||
*
|
||||
* <p>This is the same isolation {@link #hermeticGitEnv} already gives {@link
|
||||
* #seedingGitWorktrees}'s production {@link GitWorktrees} instances (fleetd #362 review fix,
|
||||
* finding 2), reused here for every subprocess the TEST ITSELF starts to drive and inspect
|
||||
* those fixture repos.
|
||||
*/
|
||||
private static Map<String, String> hermeticEnv() {
|
||||
return hermeticGitEnv(CLASS_TMP);
|
||||
}
|
||||
|
||||
/**
|
||||
* The one seam every git subprocess in this class is built through — see criterion 4's
|
||||
* self-check, {@link #everyGitSubprocessGoesThroughTheHermeticFactory}, which fails the moment
|
||||
* a future helper builds its own {@code git} subprocess directly instead of calling this, so
|
||||
* the omission that caused fleetd #369 gets caught by name rather than rediscovered by a
|
||||
* poisoned machine. The one deliberate exception is {@link
|
||||
* #worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper}, which needs a
|
||||
* non-hermetic, test-controlled global config to prove the credential helper ignores it — see
|
||||
* the comment on that test.
|
||||
*/
|
||||
private static ProcessBuilder gitProcessBuilder(Path cwd, String... args) {
|
||||
List<String> cmd = new java.util.ArrayList<>(List.of("git"));
|
||||
cmd.addAll(List.of(args));
|
||||
ProcessBuilder pb = new ProcessBuilder(cmd).directory(cwd.toFile()).redirectErrorStream(true);
|
||||
pb.environment().putAll(hermeticEnv());
|
||||
return pb;
|
||||
}
|
||||
|
||||
private static Path initRepo(Path dir) throws Exception {
|
||||
Files.createDirectories(dir);
|
||||
git(dir, "init", "-q", "-b", "main");
|
||||
@@ -71,23 +121,16 @@ class GitWorktreesTest {
|
||||
}
|
||||
|
||||
private static String gitOutput(Path cwd, String... args) throws Exception {
|
||||
List<String> cmd = new java.util.ArrayList<>(List.of("git"));
|
||||
cmd.addAll(List.of(args));
|
||||
ProcessBuilder pb = new ProcessBuilder(cmd).directory(cwd.toFile()).redirectErrorStream(true);
|
||||
pb.environment().put("GIT_CONFIG_GLOBAL", "/dev/null");
|
||||
pb.environment().put("GIT_CONFIG_SYSTEM", "/dev/null");
|
||||
pb.environment().put("GIT_TERMINAL_PROMPT", "0");
|
||||
Process p = pb.start();
|
||||
Process p = gitProcessBuilder(cwd, args).start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git timed out: " + String.join(" ", cmd));
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git timed out: git " + String.join(" ", args));
|
||||
assertEquals(0, p.exitValue(), "git " + String.join(" ", args) + " failed:\n" + out);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Pending changes to {@code file} in {@code cwd}, empty when git considers it unmodified. */
|
||||
private static String status(Path cwd, String file) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "status", "--porcelain", "--", file)
|
||||
.directory(cwd.toFile()).redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "status", "--porcelain", "--", file).start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
|
||||
return out;
|
||||
@@ -95,16 +138,14 @@ class GitWorktreesTest {
|
||||
|
||||
/** Every pending change in {@code cwd} — the whole-tree porcelain status, unlike {@link #status}. */
|
||||
private static String fullStatus(Path cwd) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "status", "--porcelain")
|
||||
.directory(cwd.toFile()).redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "status", "--porcelain").start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git status timed out");
|
||||
return out;
|
||||
}
|
||||
|
||||
private static String revParse(Path cwd, String ref) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "rev-parse", ref)
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "rev-parse", ref).start();
|
||||
String out = new String(p.getInputStream().readAllBytes()).trim();
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git rev-parse timed out");
|
||||
assertEquals(0, p.exitValue(), "git rev-parse " + ref + " failed:\n" + out);
|
||||
@@ -113,8 +154,7 @@ class GitWorktreesTest {
|
||||
|
||||
/** The recursive file list of a commit's tree — used to check what a snapshot actually committed. */
|
||||
private static String lsTree(Path cwd, String ref) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "ls-tree", "-r", "--name-only", ref)
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "ls-tree", "-r", "--name-only", ref).start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git ls-tree timed out");
|
||||
assertEquals(0, p.exitValue(), "git ls-tree " + ref + " failed:\n" + out);
|
||||
@@ -125,8 +165,7 @@ class GitWorktreesTest {
|
||||
* snapshot's tree changed relative to its parent, the same shape {@code git status --porcelain}
|
||||
* reports for the worktree it was taken from. */
|
||||
private static Set<String> diffNameOnly(Path cwd, String from, String to) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "diff", "--name-only", from, to)
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "diff", "--name-only", from, to).start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git diff timed out");
|
||||
assertEquals(0, p.exitValue(), "git diff " + from + ".." + to + " failed:\n" + out);
|
||||
@@ -152,8 +191,7 @@ class GitWorktreesTest {
|
||||
}
|
||||
|
||||
private static String forEachRef(Path cwd, String pattern) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "for-each-ref", pattern)
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "for-each-ref", pattern).start();
|
||||
String out = new String(p.getInputStream().readAllBytes());
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git for-each-ref timed out");
|
||||
assertEquals(0, p.exitValue(), "git for-each-ref " + pattern + " failed:\n" + out);
|
||||
@@ -162,8 +200,7 @@ class GitWorktreesTest {
|
||||
|
||||
/** Write {@code content} as a blob into the object database; returns its sha. */
|
||||
private static String blobOf(Path cwd, String content) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "hash-object", "-w", "--stdin")
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "hash-object", "-w", "--stdin").start();
|
||||
p.getOutputStream().write(content.getBytes(StandardCharsets.UTF_8));
|
||||
p.getOutputStream().close();
|
||||
String out = new String(p.getInputStream().readAllBytes()).trim();
|
||||
@@ -174,8 +211,7 @@ class GitWorktreesTest {
|
||||
|
||||
/** Build a single-file tree object from {@code blob}; returns the tree's sha. */
|
||||
private static String treeOf(Path cwd, String path, String blob) throws Exception {
|
||||
Process p = new ProcessBuilder("git", "-C", cwd.toString(), "mktree")
|
||||
.redirectErrorStream(true).start();
|
||||
Process p = gitProcessBuilder(cwd, "mktree").start();
|
||||
p.getOutputStream().write(("100644 blob " + blob + "\t" + path + "\n").getBytes(StandardCharsets.UTF_8));
|
||||
p.getOutputStream().close();
|
||||
String out = new String(p.getInputStream().readAllBytes()).trim();
|
||||
@@ -187,10 +223,9 @@ class GitWorktreesTest {
|
||||
/** {@code git commit-tree} rooted at {@code tree} with a chosen committer date; returns the sha. */
|
||||
private static String commitTree(Path cwd, String tree, String parent, String committerDate,
|
||||
String message) throws Exception {
|
||||
ProcessBuilder pb = new ProcessBuilder("git", "-C", cwd.toString(), "commit-tree",
|
||||
tree, "-p", parent, "-m", message);
|
||||
ProcessBuilder pb = gitProcessBuilder(cwd, "commit-tree", tree, "-p", parent, "-m", message);
|
||||
pb.environment().put("GIT_COMMITTER_DATE", committerDate);
|
||||
Process p = pb.redirectErrorStream(true).start();
|
||||
Process p = pb.start();
|
||||
String out = new String(p.getInputStream().readAllBytes()).trim();
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "git commit-tree timed out");
|
||||
assertEquals(0, p.exitValue(), "git commit-tree failed:\n" + out);
|
||||
@@ -263,6 +298,11 @@ class GitWorktreesTest {
|
||||
helper = !f() { printf 'username=%s\\npassword=%s\\n\\n' operator operator-secret; }; f
|
||||
""");
|
||||
|
||||
// fleetd #369: the one deliberate exception to gitProcessBuilder. This test's whole point is
|
||||
// that git must resolve `globalConfig` (a synthetic "operator's global config", never the
|
||||
// real machine's) and then IGNORE it — so it cannot use the shared hermetic env, which would
|
||||
// point GIT_CONFIG_GLOBAL at /dev/null and defeat the very thing under test. It never runs
|
||||
// `git status`, so it does not need XDG_CONFIG_HOME isolation either.
|
||||
ProcessBuilder pb = new ProcessBuilder("git", "credential", "fill")
|
||||
.directory(Path.of(wt).toFile()).redirectErrorStream(true);
|
||||
pb.environment().put("GIT_CONFIG_GLOBAL", globalConfig.toString());
|
||||
@@ -329,20 +369,16 @@ class GitWorktreesTest {
|
||||
Path worktree = Path.of(wt);
|
||||
assertEquals("https://git.ltms.dev/akb/kb.git",
|
||||
gitOutput(worktree, "remote", "get-url", "origin").trim());
|
||||
assertEquals(1, exitCode("git", "-C", wt, "config", "--worktree", "--get-regexp", "^url\\."),
|
||||
assertEquals(1, gitExitCode(worktree, "config", "--worktree", "--get-regexp", "^url\\."),
|
||||
"no url.*.insteadOf rewrite should be added for an already-HTTPS origin");
|
||||
}
|
||||
|
||||
/** Test-local exit-code probe, mirroring {@link GitWorktrees#exitCode} for an assertion the
|
||||
* production class does not expose. */
|
||||
private static int exitCode(String... command) throws Exception {
|
||||
ProcessBuilder pb = new ProcessBuilder(command).redirectErrorStream(true);
|
||||
pb.environment().put("GIT_CONFIG_GLOBAL", "/dev/null");
|
||||
pb.environment().put("GIT_CONFIG_SYSTEM", "/dev/null");
|
||||
pb.environment().put("GIT_TERMINAL_PROMPT", "0");
|
||||
Process p = pb.start();
|
||||
private static int gitExitCode(Path cwd, String... args) throws Exception {
|
||||
Process p = gitProcessBuilder(cwd, args).start();
|
||||
p.getInputStream().readAllBytes();
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "command timed out: " + String.join(" ", command));
|
||||
assertTrue(p.waitFor(30, TimeUnit.SECONDS), "command timed out: git " + String.join(" ", args));
|
||||
return p.exitValue();
|
||||
}
|
||||
|
||||
@@ -1493,7 +1529,15 @@ class GitWorktreesTest {
|
||||
* an empty, machine-independent {@code XDG_CONFIG_HOME} (so the fallback resolves to a file that
|
||||
* provably does not exist) plus the same {@code GIT_CONFIG_GLOBAL}/{@code GIT_CONFIG_SYSTEM}/
|
||||
* {@code GIT_TERMINAL_PROMPT} isolation the {@link #git}/{@link #gitOutput} helpers already use
|
||||
* for repo setup — so no test in this class can reach the real machine's home directory.
|
||||
* for repo setup.
|
||||
*
|
||||
* <p>Scope, measured on the fleetd #369 merge and narrower than an earlier version of this
|
||||
* comment claimed: this protects the 5 {@link #seedingGitWorktrees} sites plus — through
|
||||
* {@link #gitProcessBuilder} — every {@code git} subprocess the TEST itself starts. It does
|
||||
* NOT cover the other 53 {@code new GitWorktrees(...)} constructions in this file, which pass
|
||||
* no env override, so a production instance built that way still inherits the JVM's real
|
||||
* environment. Stripping this override from {@code seedingGitWorktrees} leaves the class green
|
||||
* both with and without the poison command above, so that half is currently unpinned.
|
||||
*/
|
||||
private static Map<String, String> hermeticGitEnv(Path tmp) {
|
||||
return Map.of(
|
||||
@@ -1739,4 +1783,77 @@ class GitWorktreesTest {
|
||||
"the XDG default excludesFile pattern ('xdg-fallback-marker') must still apply "
|
||||
+ "after skill seeding ran — got:\n" + porcelain);
|
||||
}
|
||||
|
||||
/**
|
||||
* fleetd #369, acceptance criterion 4 — make the fix hard to undo by accident. Every git
|
||||
* subprocess this class starts is required to go through {@link #gitProcessBuilder}, the one
|
||||
* place {@link #hermeticEnv} is applied; a helper built directly, the way the original leak in
|
||||
* {@link #status}/{@link #fullStatus} was, is now a source-level fact this test can catch by
|
||||
* name instead of a machine-dependent failure someone has to rediscover.
|
||||
*
|
||||
* <p>This counts a literal marker in this very file's own source, split into three
|
||||
* concatenated pieces below so the count is not thrown off by this method's own text — a
|
||||
* plain, unsplit occurrence of the marker anywhere in this file (a helper's construction, or a
|
||||
* comment that happens to spell it out contiguously) adds to the count the same way. Today
|
||||
* there are exactly two: the factory itself, and the one documented exception in {@link
|
||||
* #worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper}, which needs a
|
||||
* non-hermetic, test-controlled global config to prove the credential helper ignores it. A
|
||||
* third means a new helper was added the old, leak-prone way — route it through {@link
|
||||
* #gitProcessBuilder} instead, or explain the new exception here and bump this number.
|
||||
*/
|
||||
@Test
|
||||
void everyGitSubprocessGoesThroughTheHermeticFactory() throws Exception {
|
||||
Path source = Path.of("src/test/java/dev/ltms/fleet/session/GitWorktreesTest.java");
|
||||
String text = Files.readString(source);
|
||||
String marker = "new " + "ProcessBuilder" + "(";
|
||||
int count = 0;
|
||||
for (int from = text.indexOf(marker); from >= 0; from = text.indexOf(marker, from + marker.length())) {
|
||||
count++;
|
||||
}
|
||||
assertEquals(2, count,
|
||||
"expected exactly 2 direct git-subprocess constructions in this file (the "
|
||||
+ "gitProcessBuilder factory itself, plus the one documented exception in "
|
||||
+ "worktreeCredentialHelperCompletesWithoutUsingAnInheritedHelper) — a "
|
||||
+ "different count means a helper now bypasses the hermetic factory; route "
|
||||
+ "it through gitProcessBuilder or document the new exception here");
|
||||
}
|
||||
|
||||
/**
|
||||
* fleetd #369 review round 2. {@link #everyGitSubprocessGoesThroughTheHermeticFactory} counts
|
||||
* call sites, not behaviour — it catches a NEW helper built the old, leak-prone way, but it
|
||||
* cannot catch {@link #gitProcessBuilder} itself being gutted: deleting {@code
|
||||
* pb.environment().putAll(hermeticEnv())} from inside the factory leaves every call site
|
||||
* unchanged, the count stays 2, and the whole unpoisoned suite stays green — the exact leak
|
||||
* this ticket fixed would come back silently, with nothing but a human remembering to re-run
|
||||
* the poison command to catch it. This test instead inspects what the factory actually hands
|
||||
* to {@link ProcessBuilder#start()}, so it fails the moment the hermetic environment stops
|
||||
* being applied, on any machine, with no poison needed.
|
||||
*
|
||||
* <p>The property under test: every git subprocess this class starts must run with an
|
||||
* environment that cannot see the operator's real git configuration. A call-site count is a
|
||||
* proxy for that; this is the thing itself.
|
||||
*/
|
||||
@Test
|
||||
void gitProcessBuilderCarriesTheFullHermeticEnvironment(@TempDir Path tmp) {
|
||||
Map<String, String> env = gitProcessBuilder(tmp, "status", "--porcelain").environment();
|
||||
|
||||
assertEquals("/dev/null", env.get("GIT_CONFIG_GLOBAL"),
|
||||
"GIT_CONFIG_GLOBAL must be neutralized, or the operator's real ~/.gitconfig applies");
|
||||
assertEquals("/dev/null", env.get("GIT_CONFIG_SYSTEM"),
|
||||
"GIT_CONFIG_SYSTEM must be neutralized, or the machine's real /etc/gitconfig applies");
|
||||
assertEquals("0", env.get("GIT_TERMINAL_PROMPT"),
|
||||
"GIT_TERMINAL_PROMPT must be disabled, or a credential prompt can hang the subprocess");
|
||||
|
||||
String xdg = env.get("XDG_CONFIG_HOME");
|
||||
assertNotNull(xdg,
|
||||
"XDG_CONFIG_HOME must be set — left unset, git falls back to the operator's real "
|
||||
+ "$HOME/.config/git/ignore (gitignore(5)), exactly fleetd #369's leak");
|
||||
assertFalse(xdg.isBlank(), "XDG_CONFIG_HOME must not be blank — blank behaves like unset");
|
||||
assertTrue(Path.of(xdg).startsWith(CLASS_TMP),
|
||||
"XDG_CONFIG_HOME must point inside this test class's own throwaway directory, "
|
||||
+ "never the operator's real one or the JVM's inherited value — got: " + xdg);
|
||||
assertFalse(Files.exists(Path.of(xdg, "git", "ignore")),
|
||||
"the resolved XDG default excludes file must provably not exist, or its contents "
|
||||
+ "would silently apply to every git status this test class runs");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user