Compare commits

..

2 Commits

Author SHA1 Message Date
Dai Ha 027d413ce9 fleetd #796: document bootstrap retry readiness bound
CI / shell-tests (pull_request) Failing after 7s
CI / contract (pull_request) Successful in 51s
CI / build (pull_request) Failing after 1m59s
2026-10-06 18:26:01 +02:00
Dai Ha c88f01ecb8 fleetd #796: retry bootstrap delivery after agent readiness refusal
CI / shell-tests (pull_request) Failing after 8s
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Failing after 2m4s
2026-10-06 18:16:47 +02:00
11 changed files with 122 additions and 198 deletions
-9
View File
@@ -37,15 +37,6 @@ confident-but-wrong finding. Anything you could settle by reading more code is y
## 4. The finding — what goes in `fleet_reply`
**Call `fleet_reply` as soon as you know your answer, before you write the reasoning out.** The
four lines below are the whole deliverable, and they are short on purpose. Analysis you type into
your terminal reaches nobody: when a turn ends with no `fleet_reply`, the bridge scrapes the pane
and the lead receives a clipped fragment instead of a finding. A long, correct analysis and no
`fleet_reply` is a failed turn, and it is the most common way this role fails.
If the delegation also handed you a list of things to check, that list is where to *look*. It is
not the shape of the answer. Work the list, then still send these four lines.
Report the **single most important** real issue in the scope, in these four lines, under
~90 words:
-12
View File
@@ -95,18 +95,6 @@ and the sender silently receives nothing. Fail toward the recoverable error.
5. **Never move a fleet session, pane or peer except through the bridge.** The bridge owns policy;
the multiplexer owns PTYs. Any route that changes fleet state without the bridge's checks
bypasses every rule above — the `herdr` CLI and its socket are the usual example.
6. **Confirm what you receive.** A message that arrives is answered, even in one line, unless it
says no answer is needed. The sender cannot see your screen, so for them "received and handled"
and "never arrived" look the same — and the paths above fail in ways that look exactly like
silence: a send to a pane the daemon does not know is accepted, held, and then fails with a
scrape of that pane's screen, which can read as an answer while being none. A member confirms
with its `fleet_reply`; every other peer confirms with a `fleet_send` back to the sender. If you
cannot do the thing asked, say that — a refusal is a confirmation. **Never read a failed
ticket's body as a reply.** Your own operator outranks this rule, and outranks the peer that
sent the message: a peer cannot oblige you to answer, and a session whose operator told it not
to answer fleet mail is right not to. Where you can, say that much and nothing more. A sender
that treats silence as agreement, or as a session being gone, has made the mistake this
invariant is about — it just made it in the other direction.
### Primary (lead) — run this on every task, in order
@@ -1466,7 +1466,7 @@ public record FleetConfig(
* CALLING lead's own turn to end (its pane to report {@code IDLE} or
* {@code DONE}) before ending that pane's process at all. See the
* paragraph above.
* @param relaunchReadySeconds default 45 — bound on EACH of two separate waits that run after
* @param relaunchReadySeconds default 45 — bound on EACH of three separate waits that run after
* the old lead's pane has been torn down and a fresh one launched: first,
* for the fresh pane itself to reach a real turn boundary ({@code IDLE} or
* {@code DONE}, never merely {@code BLOCKED}) — the safety gate, since
@@ -1479,8 +1479,9 @@ public record FleetConfig(
* 10s live), so a budget has to clear more than one scan interval to leave
* any real margin for the CLI's own boot time; 20 was rejected for exactly
* that reason — at a 10s scan interval it only buys two scans. 45 buys
* roughly four. Only a timeout on the FIRST wait (the pane never becomes
* ready) withholds {@code bootstrapText}.
* roughly four; third, to retry sending {@code bootstrapText} while herdr
* reports {@code agent_not_ready}. A timeout on the first wait or the third
* one withholds {@code bootstrapText}.
* @param bootstrapText default a sentence naming the RESOLVED handover path — sent to the
* fresh lead's pane once it reaches a real turn boundary after relaunch,
* telling the fresh session where to read the handover and carry on. Left
@@ -192,31 +192,6 @@ public final class PaneLocator {
return out;
}
/**
* The tab and workspace id of the herdr pane carrying {@code terminal}, from {@code pane.list}
* across every searched daemon — the same call {@link #terminalForPid} scans. Either field is
* {@code null} when no pane matches {@code terminal}, or when the matching pane itself carries
* no tab or workspace id.
*/
public record PaneLocation(String tabId, String workspaceId) {
static final PaneLocation NONE = new PaneLocation(null, null);
}
/** Resolve {@code terminal} to the tab/workspace id of the pane it occupies. See {@link PaneLocation}. */
public PaneLocation locate(String terminal) {
if (terminal == null) {
return PaneLocation.NONE;
}
for (HerdrClient herdr : herdrs) {
for (JsonNode pane : herdr.call("pane.list", Map.of()).path("panes")) {
if (terminal.equals(pane.path("terminal_id").asText(null))) {
return new PaneLocation(pane.path("tab_id").asText(null), pane.path("workspace_id").asText(null));
}
}
}
return PaneLocation.NONE;
}
/** Whether a pane owns one of the scanned pid's ancestors, or the check of it failed outright. */
private enum Ownership { OWNS, DOES_NOT_OWN, UNKNOWN }
@@ -232,7 +232,7 @@ public final class LeadRollover {
/**
* What is known about one token, right now — the answer {@link #status} gives. Distinguishes
* five terminal outcomes an approved roll can finish with, one in-flight outcome for a roll
* terminal outcomes an approved roll can finish with, one in-flight outcome for a roll
* that has been approved but has not finished yet, and two answers for a token that names no
* active work at all: still pending confirmation, or nothing known about this token at all.
*/
@@ -256,7 +256,8 @@ public final class LeadRollover {
* that is, in fact, actively running. This is not sticky: the deferred continuation
* overwrites this same entry with a terminal state ({@link #ROLLED}, {@link
* #TURN_NEVER_SETTLED}, {@link #OLD_PANE_NEVER_DIED}, {@link #RELAUNCH_FAILED}, {@link
* #RELAUNCH_NEVER_READY}, {@link #RELAUNCH_NOT_RECOGNISED}, or {@link #FAILED}) once it
* #RELAUNCH_NEVER_READY}, {@link #RELAUNCH_NOT_RECOGNISED}, {@link #BOOTSTRAP_NEVER_SENT},
* or {@link #FAILED}) once it
* finishes — including by throwing, which {@link #runRollover}'s catch turns into {@link
* #FAILED} instead of leaving this entry stuck forever.
*/
@@ -305,6 +306,12 @@ public final class LeadRollover {
* an operator should check why the tab was not recognised.
*/
RELAUNCH_NOT_RECOGNISED,
/**
* A fresh lead was ready, but herdr kept reporting {@code agent_not_ready} while this class
* retried {@code bootstrapText} for {@code relaunchReadySeconds}. The fresh session did not
* receive its handover instruction.
*/
BOOTSTRAP_NEVER_SENT,
/**
* The deferred continuation threw a {@link RuntimeException} and the continuation thread
* died with it. Without this state, that throw would leave {@link #outcomes} holding {@link
@@ -731,7 +738,19 @@ public final class LeadRollover {
IdentityResult identityResult = waitUntilRecognisedAsLead(newAgent.terminalId(),
cfg.relaunchReadySeconds());
agents.send(newAgent.terminalId(), cfg.bootstrapTextFor(p.handoverPath()));
BootstrapResult bootstrapResult = sendBootstrapWithRetry(newAgent.terminalId(),
cfg.bootstrapTextFor(p.handoverPath()), cfg.relaunchReadySeconds());
if (!bootstrapResult.sent()) {
log.warn("lead-rollover: bootstrapText was never sent to fresh terminal {} for lead '{}' "
+ "after agent_not_ready persisted for {}ms (token={}, configured={}s)",
newAgent.terminalId(), leadName, bootstrapResult.elapsedMillis(), p.token(),
cfg.relaunchReadySeconds());
outcomes.put(p.token(), new RollStatus(RollState.BOOTSTRAP_NEVER_SENT,
"fresh terminal " + newAgent.terminalId() + " kept rejecting bootstrapText with "
+ "agent_not_ready for relaunchReadySeconds=" + cfg.relaunchReadySeconds()
+ "s (measured elapsed=" + bootstrapResult.elapsedMillis() + "ms)"));
return;
}
if (!identityResult.ready()) {
log.warn("lead-rollover: fresh terminal {} for lead '{}' is alive and bootstrapped, but "
+ "was never recognised as a live lead — an operator should check why "
@@ -755,6 +774,30 @@ public final class LeadRollover {
+ newAgent.terminalId()));
}
/**
* Sends {@code bootstrapText}, retrying only the transient herdr {@code agent_not_ready} refusal
* until {@code readySeconds} elapses. All other failures propagate to {@link #runRollover}.
*/
private BootstrapResult sendBootstrapWithRetry(String terminal, String bootstrapText, int readySeconds) {
long startMillis = nowMillis.getAsLong();
long deadline = startMillis + TimeUnit.SECONDS.toMillis(readySeconds);
while (nowMillis.getAsLong() < deadline) {
try {
agents.send(terminal, bootstrapText);
return new BootstrapResult(true, nowMillis.getAsLong() - startMillis);
} catch (HerdrException e) {
if (!"agent_not_ready".equals(e.code())) {
throw e;
}
pollSleeper.run();
}
}
return new BootstrapResult(false, nowMillis.getAsLong() - startMillis);
}
/** The result of {@link #sendBootstrapWithRetry}. */
private record BootstrapResult(boolean sent, long elapsedMillis) {}
/** Attempts {@link #captureAgentWithRetry} makes before letting the failure propagate. */
static final int CAPTURE_RETRIES = 3;
@@ -9,7 +9,6 @@ import dev.ltms.fleet.auth.Role;
import dev.ltms.fleet.guard.GuardException;
import dev.ltms.fleet.herdr.Agent;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.PaneLocator;
import dev.ltms.fleet.metrics.FleetMetrics;
import dev.ltms.fleet.metrics.Metrics;
import dev.ltms.fleet.inject.MemberPresence;
@@ -504,7 +503,7 @@ public final class FleetMcp {
// An observer's SEND reaches a pane that cannot otherwise distinguish this
// from a human paste (see attributeIfObserver); every other caller's content
// passes through unchanged.
String content = attributeIfObserver(caller, str(a, "content"), identity.panes());
String content = attributeIfObserver(caller, str(a, "content"));
String turnId = str(a, "turnId");
String coordId = str(a, "coordId");
if (coordId != null && !coordId.isBlank()) {
@@ -988,49 +987,12 @@ public final class FleetMcp {
/**
* The text an observer's {@code SEND} actually delivers: prefixed with the sender's own
* connection-resolved terminal, which the receiving pane cannot otherwise tell apart from a
* human paste, plus its tab and workspace display label in parentheses when herdr can supply
* either. The terminal id is the only thing in the header a reply can target — a label is
* display-only, is never unique, and is never substituted for it. Every other caller's content
* passes through unchanged. Shared with {@code FleetApp}'s REST entry path so both surfaces
* attribute identically.
* human paste. Every other caller's content passes through unchanged. Shared with {@code
* FleetApp}'s REST entry path so both surfaces attribute identically.
*/
public static String attributeIfObserver(Principal caller, String content, PaneLocator panes) {
if (caller == null || !caller.isObserver()) {
return content;
}
return "[fleet_send from observer " + observerHeader(caller.terminal(), panes) + "]\n" + content;
}
/**
* {@code terminal}, plus {@code (space "…", tab "…")} for whichever of its workspace/tab
* labels herdr reports — omitted entirely when neither is known, so a degraded lookup still
* reads as the bare id and never as {@code null} or an empty parenthetical.
*/
private static String observerHeader(String terminal, PaneLocator panes) {
String tabLabel = null;
String workspaceLabel = null;
try {
PaneLocator.PaneLocation location = panes.locate(terminal);
tabLabel = location.tabId() == null ? null : panes.tabLabelsByTabId().get(location.tabId());
workspaceLabel = location.workspaceId() == null ? null
: panes.workspaceLabelsByWorkspaceId().get(location.workspaceId());
} catch (HerdrException e) {
// degrade to the id alone
}
if (tabLabel == null && workspaceLabel == null) {
return terminal;
}
StringBuilder header = new StringBuilder(terminal).append(" (");
if (workspaceLabel != null) {
header.append("space \"").append(workspaceLabel).append('"');
}
if (tabLabel != null && workspaceLabel != null) {
header.append(", ");
}
if (tabLabel != null) {
header.append("tab \"").append(tabLabel).append('"');
}
return header.append(')').toString();
public static String attributeIfObserver(Principal caller, String content) {
return caller != null && caller.isObserver()
? "[fleet_send from observer " + caller.terminal() + "]\n" + content : content;
}
/**
@@ -714,7 +714,7 @@ public final class FleetApp {
// An observer's SEND reaches a pane that cannot otherwise distinguish this from a human
// paste (see FleetMcp#attributeIfObserver, the same rule on the MCP entry path); every
// other caller's content passes through unchanged.
content = FleetMcp.attributeIfObserver(caller, content, new PaneLocator(herdr, memberHerdr));
content = FleetMcp.attributeIfObserver(caller, content);
timeout = Math.clamp(timeout, 1, MAX_MESSAGE_TIMEOUT_MS);
// Answering a worker's fleet_ask (CB-205): always blocks, and derives the worker from turnId.
@@ -2,10 +2,12 @@ package dev.ltms.fleet.lead;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.databind.JsonNode;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.testing.CapturedLog;
import org.junit.jupiter.api.DisplayName;
@@ -20,6 +22,7 @@ import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Function;
import java.util.function.LongSupplier;
@@ -1323,6 +1326,66 @@ class LeadRolloverTest {
+ "so an operator reading status() has something to act on: " + status.detail());
}
@Test
@DisplayName("[BOOTSTRAP 1] one agent_not_ready bootstrap refusal is retried and sends the "
+ "handover instruction exactly once")
void transientAgentNotReadyRetriesBootstrapAndRolls() throws IOException {
FakeHerdr fake = herdrReadyForAFullRoll();
AtomicInteger refusedSends = new AtomicInteger();
HerdrClient transientRefusal = new HerdrClient() {
@Override
public JsonNode call(String method, Object params) {
if ("agent.prompt".equals(method) && refusedSends.getAndIncrement() == 0) {
throw new HerdrException("herdr error [agent_not_ready]: agent.prompt failed",
"agent_not_ready", null);
}
return fake.call(method, params);
}
@Override
public void close() {
}
};
Path handover = writeHandover("handover contents");
AtomicLong clock = new AtomicLong(1_000);
AgentControl agents = new AgentControl(transientRefusal);
WorkspaceControl spaces = new WorkspaceControl(transientRefusal);
LeadRollover rollover = new LeadRollover(agents, spaces,
new LeadLauncher(agents, spaces, fleetConfigWithRelaunchableLead()),
() -> cfg(handover.toString()), _ -> null, _ -> LEAD_NAME,
() -> Map.of(NEW_TERMINAL, LEAD_NAME), fixedClock(clock), () -> { }, Runnable::run);
LeadRollover.PendingRollover pending = rollover.open(LEAD, "context is full");
assertTrue(rollover.confirm(LEAD, pending.token(), true).accepted());
assertEquals(2, refusedSends.get(), "one agent_not_ready refusal must be followed by one retry");
assertEquals(1, promptCallCount(fake), "only the successful retry reaches herdr delivery");
assertEquals(LeadRollover.RollState.ROLLED, rollover.status(pending.token()).state());
}
@Test
@DisplayName("[BOOTSTRAP 2] persistent agent_not_ready records BOOTSTRAP_NEVER_SENT and "
+ "releases the single-flight claim")
void persistentAgentNotReadyRecordsBootstrapNeverSentAndReleasesClaim() throws IOException {
FakeHerdr fake = herdrReadyForAFullRoll();
fake.agentSendFailsWith("agent_not_ready");
Path handover = writeHandover("handover contents");
AtomicLong clock = new AtomicLong(1_000);
LeadRollover rollover = newRolloverForAFullRoll(fake, cfg(handover.toString()),
() -> clock.addAndGet(1_000), Map.of(NEW_TERMINAL, LEAD_NAME));
LeadRollover.PendingRollover pending = rollover.open(LEAD, "context is full");
assertTrue(rollover.confirm(LEAD, pending.token(), true).accepted());
LeadRollover.RollStatus status = rollover.status(pending.token());
assertEquals(LeadRollover.RollState.BOOTSTRAP_NEVER_SENT, status.state());
assertNotEquals(LeadRollover.RollState.ROLLED, status.state());
assertNotEquals(LeadRollover.RollState.FAILED, status.state());
LeadRollover.PendingRollover retry = rollover.open(LEAD, "retry after bootstrap timeout");
assertTrue(rollover.confirm(LEAD, retry.token(), true).accepted(),
"the terminal outcome must release the single-flight claim");
}
// ---- fleetd #726 unit 3: confirm() single-flights one roll at a time per lead ---------------
@Test
@@ -1,93 +0,0 @@
package dev.ltms.fleet.mcp;
import dev.ltms.fleet.auth.Principal;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.PaneLocator;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* {@link FleetMcp#attributeIfObserver}: the terminal id in an observer's header is always
* present, a tab/workspace label is display-only and additive, and a missing label degrades to
* the id alone rather than rendering {@code null} or an empty parenthetical.
*/
class FleetMcpAttributeIfObserverTest {
@Test
void aNonObserverPassesContentThroughUnchangedRegardlessOfPanes() {
Principal worker = Principal.worker("term_worker", 1);
assertEquals("hello", FleetMcp.attributeIfObserver(worker, "hello", null));
}
@Test
void theTerminalIdIsAlwaysPresentAndNeverReplacedByALabel() {
FakeHerdr herdr = new FakeHerdr()
.withTab("w2", "w2:t8", "shell-tab");
PaneLocator panes = new PaneLocator(herdr);
Principal observer = Principal.observer("term_shell", 2);
String result = FleetMcp.attributeIfObserver(observer, "hi", panes);
assertEquals("[fleet_send from observer term_shell (space \"ltms\", tab \"shell-tab\")]\nhi", result);
}
/**
* Two tabs sharing one label must still read as two different senders. The label is the same
* on both headers; the id is what differs, and is the only part of either header a reply can
* target.
*/
@Test
void aSharedLabelStillYieldsDistinctHeadersByTerminalId() {
FakeHerdr herdr = new FakeHerdr()
.withTab("w2", "w2:t7", "lead")
.withTab("w2", "w2:t8", "lead");
PaneLocator panes = new PaneLocator(herdr);
String fromA = FleetMcp.attributeIfObserver(Principal.observer("term_a", 1), "msg", panes);
String fromShell = FleetMcp.attributeIfObserver(Principal.observer("term_shell", 2), "msg", panes);
assertEquals("[fleet_send from observer term_a (space \"ltms\", tab \"lead\")]\nmsg", fromA);
assertEquals("[fleet_send from observer term_shell (space \"ltms\", tab \"lead\")]\nmsg", fromShell);
assertNotEquals(fromA, fromShell, "identical labels must not collapse two senders into one header");
}
@Test
void anUnknownPaneDegradesToTheIdAloneWithNoLabelAndNoEmptyParens() {
PaneLocator panes = new PaneLocator(new FakeHerdr().withNoPanes());
Principal observer = Principal.observer("term_ghost", 3);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_ghost]\nmsg", result);
assertFalse(result.contains("null"), "a missing label must never render as the literal \"null\"");
assertFalse(result.contains("()"), "a missing label must never leave an empty parenthetical");
}
@Test
void aHerdrFailureDuringTheLabelLookupDegradesToTheIdAlone() {
FakeHerdr herdr = new FakeHerdr().workspaceListFailsWith("unavailable");
PaneLocator panes = new PaneLocator(herdr);
Principal observer = Principal.observer("term_shell", 4);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_shell]\nmsg", result);
}
@Test
void onlyTheKnownLabelAppearsWhenTheOtherIsMissing() {
// term_shell's pane sits in workspace "w2" ("ltms"); its tab "w2:t8" carries no seeded
// label, so only the space label appears.
PaneLocator panes = new PaneLocator(new FakeHerdr());
Principal observer = Principal.observer("term_shell", 5);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\nmsg", result);
assertTrue(result.contains("term_shell"), "the id must still be present");
}
}
@@ -119,10 +119,7 @@ class FleetMcpObserverSendDeliveryTest {
@SuppressWarnings("unchecked")
Map<String, Object> params = (Map<String, Object>) herdr.lastCall("agent.prompt").params();
// term_shell's pane sits in workspace "w2", which FakeHerdr's workspace.list labels
// "ltms"; its tab "w2:t8" carries no label in FakeHerdr's tab.list, so only the space
// label appears.
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\nhi there", params.get("text"),
assertEquals("[fleet_send from observer term_shell]\nhi there", params.get("text"),
"the receiving pane must see the sender's own daemon-resolved terminal, never a raw "
+ "echo of the content and never a client-supplied name");
}
@@ -140,10 +140,7 @@ class FleetMcpObserverSendToLeadDeliveryTest {
@SuppressWarnings("unchecked")
Map<String, Object> params = (Map<String, Object>) herdr.lastCall("agent.prompt").params();
// term_shell's pane sits in workspace "w2", which FakeHerdr's workspace.list labels
// "ltms"; its tab "w2:t8" carries no label in FakeHerdr's tab.list, so only the space
// label appears.
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\ncan we split the review?",
assertEquals("[fleet_send from observer term_shell]\ncan we split the review?",
params.get("text"),
"a lead must see the sender's own daemon-resolved terminal, never a raw echo of "
+ "the content and never a client-supplied name");