Compare commits

..

4 Commits

Author SHA1 Message Date
Dai Ha a3eeace447 handover skill: BOOTSTRAP_NEVER_SENT, and relaunchReadySeconds bounds three waits
CI / shell-tests (push) Failing after 7s
CI / contract (push) Successful in 1m2s
CI / build (push) Failing after 1m55s
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-06 19:28:21 +02:00
Dai Ha b696c31756 fleetd #796: retry bootstrap delivery to a fresh lead pane
CI / shell-tests (push) Failing after 6s
CI / contract (push) Successful in 1m5s
CI / build (push) Failing after 2m17s
A roll killed the old pane, launched the new one, then lost bootstrapText to a
herdr agent_not_ready refusal. The successor woke with no handover and no way to
tell a failed roll from a cold start.

sendBootstrapWithRetry retries only that refusal, bounded by
relaunchReadySeconds. A persistent refusal now ends the roll with the new
BOOTSTRAP_NEVER_SENT outcome, so fleet_handover{action:"status"} can report it.

Verified at 027d413 in a throwaway worktree: 2210 tests, 0 failures, 0 errors,
clean install.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-06 19:27:36 +02:00
Dai Ha 027d413ce9 fleetd #796: document bootstrap retry readiness bound
CI / shell-tests (pull_request) Failing after 7s
CI / contract (pull_request) Successful in 51s
CI / build (pull_request) Failing after 1m59s
2026-10-06 18:26:01 +02:00
Dai Ha c88f01ecb8 fleetd #796: retry bootstrap delivery after agent readiness refusal
CI / shell-tests (pull_request) Failing after 8s
CI / contract (pull_request) Successful in 54s
CI / build (pull_request) Failing after 2m4s
2026-10-06 18:16:47 +02:00
10 changed files with 126 additions and 179 deletions
+4 -2
View File
@@ -189,8 +189,9 @@ If the first number has grown past 20, somebody has rolled under the restart pat
should be replaced with what they measured.
**Three separate timeouts bound a roll.** `leadRollover.relaunchReadySeconds` (default 45,
`FleetConfig.java:1483`) bounds **each** of two waits that run after the relaunch, so the worst case
there is about twice that number, not 45 seconds in total. A third bound gives your old pane 10
`FleetConfig.java:1483`) bounds **each** of three waits that run after the relaunch, so the worst
case there is about three times that number, not 45 seconds in total. The third wait retries
`bootstrapText` while herdr answers `agent_not_ready`. A separate bound gives your old pane 10
seconds to die (`LeadRollover.PANE_DEATH_TIMEOUT_SECONDS`).
`fleet_handover{action: "status", token}` answers with one of these:
@@ -204,6 +205,7 @@ seconds to die (`LeadRollover.PANE_DEATH_TIMEOUT_SECONDS`).
| `RELAUNCH_FAILED` | launching the fresh pane failed |
| `RELAUNCH_NEVER_READY` | the fresh pane never became ready within `relaunchReadySeconds` |
| `RELAUNCH_NOT_RECOGNISED` | the fresh terminal never resolved as a lead |
| `BOOTSTRAP_NEVER_SENT` | the fresh pane was ready, but herdr refused `bootstrapText` with `agent_not_ready` for the whole bound, so the successor never learned where the handover file is |
| `FAILED` | the roll threw; `runRollover`'s catch records this rather than leaving it stuck |
Only `TURN_NEVER_SETTLED` guarantees your context is intact. The other failures can leave you
@@ -1466,7 +1466,7 @@ public record FleetConfig(
* CALLING lead's own turn to end (its pane to report {@code IDLE} or
* {@code DONE}) before ending that pane's process at all. See the
* paragraph above.
* @param relaunchReadySeconds default 45 — bound on EACH of two separate waits that run after
* @param relaunchReadySeconds default 45 — bound on EACH of three separate waits that run after
* the old lead's pane has been torn down and a fresh one launched: first,
* for the fresh pane itself to reach a real turn boundary ({@code IDLE} or
* {@code DONE}, never merely {@code BLOCKED}) — the safety gate, since
@@ -1479,8 +1479,9 @@ public record FleetConfig(
* 10s live), so a budget has to clear more than one scan interval to leave
* any real margin for the CLI's own boot time; 20 was rejected for exactly
* that reason — at a 10s scan interval it only buys two scans. 45 buys
* roughly four. Only a timeout on the FIRST wait (the pane never becomes
* ready) withholds {@code bootstrapText}.
* roughly four; third, to retry sending {@code bootstrapText} while herdr
* reports {@code agent_not_ready}. A timeout on the first wait or the third
* one withholds {@code bootstrapText}.
* @param bootstrapText default a sentence naming the RESOLVED handover path — sent to the
* fresh lead's pane once it reaches a real turn boundary after relaunch,
* telling the fresh session where to read the handover and carry on. Left
@@ -192,31 +192,6 @@ public final class PaneLocator {
return out;
}
/**
* The tab and workspace id of the herdr pane carrying {@code terminal}, from {@code pane.list}
* across every searched daemon — the same call {@link #terminalForPid} scans. Either field is
* {@code null} when no pane matches {@code terminal}, or when the matching pane itself carries
* no tab or workspace id.
*/
public record PaneLocation(String tabId, String workspaceId) {
static final PaneLocation NONE = new PaneLocation(null, null);
}
/** Resolve {@code terminal} to the tab/workspace id of the pane it occupies. See {@link PaneLocation}. */
public PaneLocation locate(String terminal) {
if (terminal == null) {
return PaneLocation.NONE;
}
for (HerdrClient herdr : herdrs) {
for (JsonNode pane : herdr.call("pane.list", Map.of()).path("panes")) {
if (terminal.equals(pane.path("terminal_id").asText(null))) {
return new PaneLocation(pane.path("tab_id").asText(null), pane.path("workspace_id").asText(null));
}
}
}
return PaneLocation.NONE;
}
/** Whether a pane owns one of the scanned pid's ancestors, or the check of it failed outright. */
private enum Ownership { OWNS, DOES_NOT_OWN, UNKNOWN }
@@ -232,7 +232,7 @@ public final class LeadRollover {
/**
* What is known about one token, right now — the answer {@link #status} gives. Distinguishes
* five terminal outcomes an approved roll can finish with, one in-flight outcome for a roll
* terminal outcomes an approved roll can finish with, one in-flight outcome for a roll
* that has been approved but has not finished yet, and two answers for a token that names no
* active work at all: still pending confirmation, or nothing known about this token at all.
*/
@@ -256,7 +256,8 @@ public final class LeadRollover {
* that is, in fact, actively running. This is not sticky: the deferred continuation
* overwrites this same entry with a terminal state ({@link #ROLLED}, {@link
* #TURN_NEVER_SETTLED}, {@link #OLD_PANE_NEVER_DIED}, {@link #RELAUNCH_FAILED}, {@link
* #RELAUNCH_NEVER_READY}, {@link #RELAUNCH_NOT_RECOGNISED}, or {@link #FAILED}) once it
* #RELAUNCH_NEVER_READY}, {@link #RELAUNCH_NOT_RECOGNISED}, {@link #BOOTSTRAP_NEVER_SENT},
* or {@link #FAILED}) once it
* finishes — including by throwing, which {@link #runRollover}'s catch turns into {@link
* #FAILED} instead of leaving this entry stuck forever.
*/
@@ -305,6 +306,12 @@ public final class LeadRollover {
* an operator should check why the tab was not recognised.
*/
RELAUNCH_NOT_RECOGNISED,
/**
* A fresh lead was ready, but herdr kept reporting {@code agent_not_ready} while this class
* retried {@code bootstrapText} for {@code relaunchReadySeconds}. The fresh session did not
* receive its handover instruction.
*/
BOOTSTRAP_NEVER_SENT,
/**
* The deferred continuation threw a {@link RuntimeException} and the continuation thread
* died with it. Without this state, that throw would leave {@link #outcomes} holding {@link
@@ -731,7 +738,19 @@ public final class LeadRollover {
IdentityResult identityResult = waitUntilRecognisedAsLead(newAgent.terminalId(),
cfg.relaunchReadySeconds());
agents.send(newAgent.terminalId(), cfg.bootstrapTextFor(p.handoverPath()));
BootstrapResult bootstrapResult = sendBootstrapWithRetry(newAgent.terminalId(),
cfg.bootstrapTextFor(p.handoverPath()), cfg.relaunchReadySeconds());
if (!bootstrapResult.sent()) {
log.warn("lead-rollover: bootstrapText was never sent to fresh terminal {} for lead '{}' "
+ "after agent_not_ready persisted for {}ms (token={}, configured={}s)",
newAgent.terminalId(), leadName, bootstrapResult.elapsedMillis(), p.token(),
cfg.relaunchReadySeconds());
outcomes.put(p.token(), new RollStatus(RollState.BOOTSTRAP_NEVER_SENT,
"fresh terminal " + newAgent.terminalId() + " kept rejecting bootstrapText with "
+ "agent_not_ready for relaunchReadySeconds=" + cfg.relaunchReadySeconds()
+ "s (measured elapsed=" + bootstrapResult.elapsedMillis() + "ms)"));
return;
}
if (!identityResult.ready()) {
log.warn("lead-rollover: fresh terminal {} for lead '{}' is alive and bootstrapped, but "
+ "was never recognised as a live lead — an operator should check why "
@@ -755,6 +774,30 @@ public final class LeadRollover {
+ newAgent.terminalId()));
}
/**
* Sends {@code bootstrapText}, retrying only the transient herdr {@code agent_not_ready} refusal
* until {@code readySeconds} elapses. All other failures propagate to {@link #runRollover}.
*/
private BootstrapResult sendBootstrapWithRetry(String terminal, String bootstrapText, int readySeconds) {
long startMillis = nowMillis.getAsLong();
long deadline = startMillis + TimeUnit.SECONDS.toMillis(readySeconds);
while (nowMillis.getAsLong() < deadline) {
try {
agents.send(terminal, bootstrapText);
return new BootstrapResult(true, nowMillis.getAsLong() - startMillis);
} catch (HerdrException e) {
if (!"agent_not_ready".equals(e.code())) {
throw e;
}
pollSleeper.run();
}
}
return new BootstrapResult(false, nowMillis.getAsLong() - startMillis);
}
/** The result of {@link #sendBootstrapWithRetry}. */
private record BootstrapResult(boolean sent, long elapsedMillis) {}
/** Attempts {@link #captureAgentWithRetry} makes before letting the failure propagate. */
static final int CAPTURE_RETRIES = 3;
@@ -9,7 +9,6 @@ import dev.ltms.fleet.auth.Role;
import dev.ltms.fleet.guard.GuardException;
import dev.ltms.fleet.herdr.Agent;
import dev.ltms.fleet.herdr.AgentStatus;
import dev.ltms.fleet.herdr.PaneLocator;
import dev.ltms.fleet.metrics.FleetMetrics;
import dev.ltms.fleet.metrics.Metrics;
import dev.ltms.fleet.inject.MemberPresence;
@@ -504,7 +503,7 @@ public final class FleetMcp {
// An observer's SEND reaches a pane that cannot otherwise distinguish this
// from a human paste (see attributeIfObserver); every other caller's content
// passes through unchanged.
String content = attributeIfObserver(caller, str(a, "content"), identity.panes());
String content = attributeIfObserver(caller, str(a, "content"));
String turnId = str(a, "turnId");
String coordId = str(a, "coordId");
if (coordId != null && !coordId.isBlank()) {
@@ -988,49 +987,12 @@ public final class FleetMcp {
/**
* The text an observer's {@code SEND} actually delivers: prefixed with the sender's own
* connection-resolved terminal, which the receiving pane cannot otherwise tell apart from a
* human paste, plus its tab and workspace display label in parentheses when herdr can supply
* either. The terminal id is the only thing in the header a reply can target — a label is
* display-only, is never unique, and is never substituted for it. Every other caller's content
* passes through unchanged. Shared with {@code FleetApp}'s REST entry path so both surfaces
* attribute identically.
* human paste. Every other caller's content passes through unchanged. Shared with {@code
* FleetApp}'s REST entry path so both surfaces attribute identically.
*/
public static String attributeIfObserver(Principal caller, String content, PaneLocator panes) {
if (caller == null || !caller.isObserver()) {
return content;
}
return "[fleet_send from observer " + observerHeader(caller.terminal(), panes) + "]\n" + content;
}
/**
* {@code terminal}, plus {@code (space "…", tab "…")} for whichever of its workspace/tab
* labels herdr reports — omitted entirely when neither is known, so a degraded lookup still
* reads as the bare id and never as {@code null} or an empty parenthetical.
*/
private static String observerHeader(String terminal, PaneLocator panes) {
String tabLabel = null;
String workspaceLabel = null;
try {
PaneLocator.PaneLocation location = panes.locate(terminal);
tabLabel = location.tabId() == null ? null : panes.tabLabelsByTabId().get(location.tabId());
workspaceLabel = location.workspaceId() == null ? null
: panes.workspaceLabelsByWorkspaceId().get(location.workspaceId());
} catch (HerdrException e) {
// degrade to the id alone
}
if (tabLabel == null && workspaceLabel == null) {
return terminal;
}
StringBuilder header = new StringBuilder(terminal).append(" (");
if (workspaceLabel != null) {
header.append("space \"").append(workspaceLabel).append('"');
}
if (tabLabel != null && workspaceLabel != null) {
header.append(", ");
}
if (tabLabel != null) {
header.append("tab \"").append(tabLabel).append('"');
}
return header.append(')').toString();
public static String attributeIfObserver(Principal caller, String content) {
return caller != null && caller.isObserver()
? "[fleet_send from observer " + caller.terminal() + "]\n" + content : content;
}
/**
@@ -714,7 +714,7 @@ public final class FleetApp {
// An observer's SEND reaches a pane that cannot otherwise distinguish this from a human
// paste (see FleetMcp#attributeIfObserver, the same rule on the MCP entry path); every
// other caller's content passes through unchanged.
content = FleetMcp.attributeIfObserver(caller, content, new PaneLocator(herdr, memberHerdr));
content = FleetMcp.attributeIfObserver(caller, content);
timeout = Math.clamp(timeout, 1, MAX_MESSAGE_TIMEOUT_MS);
// Answering a worker's fleet_ask (CB-205): always blocks, and derives the worker from turnId.
@@ -2,10 +2,12 @@ package dev.ltms.fleet.lead;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.databind.JsonNode;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.HerdrClient;
import dev.ltms.fleet.herdr.HerdrException;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.testing.CapturedLog;
import org.junit.jupiter.api.DisplayName;
@@ -20,6 +22,7 @@ import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Function;
import java.util.function.LongSupplier;
@@ -1323,6 +1326,66 @@ class LeadRolloverTest {
+ "so an operator reading status() has something to act on: " + status.detail());
}
@Test
@DisplayName("[BOOTSTRAP 1] one agent_not_ready bootstrap refusal is retried and sends the "
+ "handover instruction exactly once")
void transientAgentNotReadyRetriesBootstrapAndRolls() throws IOException {
FakeHerdr fake = herdrReadyForAFullRoll();
AtomicInteger refusedSends = new AtomicInteger();
HerdrClient transientRefusal = new HerdrClient() {
@Override
public JsonNode call(String method, Object params) {
if ("agent.prompt".equals(method) && refusedSends.getAndIncrement() == 0) {
throw new HerdrException("herdr error [agent_not_ready]: agent.prompt failed",
"agent_not_ready", null);
}
return fake.call(method, params);
}
@Override
public void close() {
}
};
Path handover = writeHandover("handover contents");
AtomicLong clock = new AtomicLong(1_000);
AgentControl agents = new AgentControl(transientRefusal);
WorkspaceControl spaces = new WorkspaceControl(transientRefusal);
LeadRollover rollover = new LeadRollover(agents, spaces,
new LeadLauncher(agents, spaces, fleetConfigWithRelaunchableLead()),
() -> cfg(handover.toString()), _ -> null, _ -> LEAD_NAME,
() -> Map.of(NEW_TERMINAL, LEAD_NAME), fixedClock(clock), () -> { }, Runnable::run);
LeadRollover.PendingRollover pending = rollover.open(LEAD, "context is full");
assertTrue(rollover.confirm(LEAD, pending.token(), true).accepted());
assertEquals(2, refusedSends.get(), "one agent_not_ready refusal must be followed by one retry");
assertEquals(1, promptCallCount(fake), "only the successful retry reaches herdr delivery");
assertEquals(LeadRollover.RollState.ROLLED, rollover.status(pending.token()).state());
}
@Test
@DisplayName("[BOOTSTRAP 2] persistent agent_not_ready records BOOTSTRAP_NEVER_SENT and "
+ "releases the single-flight claim")
void persistentAgentNotReadyRecordsBootstrapNeverSentAndReleasesClaim() throws IOException {
FakeHerdr fake = herdrReadyForAFullRoll();
fake.agentSendFailsWith("agent_not_ready");
Path handover = writeHandover("handover contents");
AtomicLong clock = new AtomicLong(1_000);
LeadRollover rollover = newRolloverForAFullRoll(fake, cfg(handover.toString()),
() -> clock.addAndGet(1_000), Map.of(NEW_TERMINAL, LEAD_NAME));
LeadRollover.PendingRollover pending = rollover.open(LEAD, "context is full");
assertTrue(rollover.confirm(LEAD, pending.token(), true).accepted());
LeadRollover.RollStatus status = rollover.status(pending.token());
assertEquals(LeadRollover.RollState.BOOTSTRAP_NEVER_SENT, status.state());
assertNotEquals(LeadRollover.RollState.ROLLED, status.state());
assertNotEquals(LeadRollover.RollState.FAILED, status.state());
LeadRollover.PendingRollover retry = rollover.open(LEAD, "retry after bootstrap timeout");
assertTrue(rollover.confirm(LEAD, retry.token(), true).accepted(),
"the terminal outcome must release the single-flight claim");
}
// ---- fleetd #726 unit 3: confirm() single-flights one roll at a time per lead ---------------
@Test
@@ -1,93 +0,0 @@
package dev.ltms.fleet.mcp;
import dev.ltms.fleet.auth.Principal;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.PaneLocator;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* {@link FleetMcp#attributeIfObserver}: the terminal id in an observer's header is always
* present, a tab/workspace label is display-only and additive, and a missing label degrades to
* the id alone rather than rendering {@code null} or an empty parenthetical.
*/
class FleetMcpAttributeIfObserverTest {
@Test
void aNonObserverPassesContentThroughUnchangedRegardlessOfPanes() {
Principal worker = Principal.worker("term_worker", 1);
assertEquals("hello", FleetMcp.attributeIfObserver(worker, "hello", null));
}
@Test
void theTerminalIdIsAlwaysPresentAndNeverReplacedByALabel() {
FakeHerdr herdr = new FakeHerdr()
.withTab("w2", "w2:t8", "shell-tab");
PaneLocator panes = new PaneLocator(herdr);
Principal observer = Principal.observer("term_shell", 2);
String result = FleetMcp.attributeIfObserver(observer, "hi", panes);
assertEquals("[fleet_send from observer term_shell (space \"ltms\", tab \"shell-tab\")]\nhi", result);
}
/**
* Two tabs sharing one label must still read as two different senders. The label is the same
* on both headers; the id is what differs, and is the only part of either header a reply can
* target.
*/
@Test
void aSharedLabelStillYieldsDistinctHeadersByTerminalId() {
FakeHerdr herdr = new FakeHerdr()
.withTab("w2", "w2:t7", "lead")
.withTab("w2", "w2:t8", "lead");
PaneLocator panes = new PaneLocator(herdr);
String fromA = FleetMcp.attributeIfObserver(Principal.observer("term_a", 1), "msg", panes);
String fromShell = FleetMcp.attributeIfObserver(Principal.observer("term_shell", 2), "msg", panes);
assertEquals("[fleet_send from observer term_a (space \"ltms\", tab \"lead\")]\nmsg", fromA);
assertEquals("[fleet_send from observer term_shell (space \"ltms\", tab \"lead\")]\nmsg", fromShell);
assertNotEquals(fromA, fromShell, "identical labels must not collapse two senders into one header");
}
@Test
void anUnknownPaneDegradesToTheIdAloneWithNoLabelAndNoEmptyParens() {
PaneLocator panes = new PaneLocator(new FakeHerdr().withNoPanes());
Principal observer = Principal.observer("term_ghost", 3);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_ghost]\nmsg", result);
assertFalse(result.contains("null"), "a missing label must never render as the literal \"null\"");
assertFalse(result.contains("()"), "a missing label must never leave an empty parenthetical");
}
@Test
void aHerdrFailureDuringTheLabelLookupDegradesToTheIdAlone() {
FakeHerdr herdr = new FakeHerdr().workspaceListFailsWith("unavailable");
PaneLocator panes = new PaneLocator(herdr);
Principal observer = Principal.observer("term_shell", 4);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_shell]\nmsg", result);
}
@Test
void onlyTheKnownLabelAppearsWhenTheOtherIsMissing() {
// term_shell's pane sits in workspace "w2" ("ltms"); its tab "w2:t8" carries no seeded
// label, so only the space label appears.
PaneLocator panes = new PaneLocator(new FakeHerdr());
Principal observer = Principal.observer("term_shell", 5);
String result = FleetMcp.attributeIfObserver(observer, "msg", panes);
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\nmsg", result);
assertTrue(result.contains("term_shell"), "the id must still be present");
}
}
@@ -119,10 +119,7 @@ class FleetMcpObserverSendDeliveryTest {
@SuppressWarnings("unchecked")
Map<String, Object> params = (Map<String, Object>) herdr.lastCall("agent.prompt").params();
// term_shell's pane sits in workspace "w2", which FakeHerdr's workspace.list labels
// "ltms"; its tab "w2:t8" carries no label in FakeHerdr's tab.list, so only the space
// label appears.
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\nhi there", params.get("text"),
assertEquals("[fleet_send from observer term_shell]\nhi there", params.get("text"),
"the receiving pane must see the sender's own daemon-resolved terminal, never a raw "
+ "echo of the content and never a client-supplied name");
}
@@ -140,10 +140,7 @@ class FleetMcpObserverSendToLeadDeliveryTest {
@SuppressWarnings("unchecked")
Map<String, Object> params = (Map<String, Object>) herdr.lastCall("agent.prompt").params();
// term_shell's pane sits in workspace "w2", which FakeHerdr's workspace.list labels
// "ltms"; its tab "w2:t8" carries no label in FakeHerdr's tab.list, so only the space
// label appears.
assertEquals("[fleet_send from observer term_shell (space \"ltms\")]\ncan we split the review?",
assertEquals("[fleet_send from observer term_shell]\ncan we split the review?",
params.get("text"),
"a lead must see the sender's own daemon-resolved terminal, never a raw echo of "
+ "the content and never a client-supplied name");