fleetd#341: a per-name guard so a later spawn's different unprotected name still warns

unprotectedGapLogged was one AtomicBoolean guarding two WARN branches in
logCredentialGap that name different env var names (the allow-list
keptByDerivedList branch, and warnGapUnprotected's deny-by-default /
non-zsh-fallback branch). memberCredentials is a live, re-read-per-spawn
supplier, so between two spawns a policy reload can change which names are
in the gap: spawn 1 warns about name A and trips the shared flag, and
spawn 2's gap containing a different name B never gets its WARN.

Replace the AtomicBoolean with unprotectedGapNamesWarned, a
ConcurrentHashMap-backed Set<String> guard keyed per name (same shape as
OpenCodeLauncher.modelCheckSkippedWarned), so each distinct credential-shaped
name is warned about exactly once, ever, regardless of which branch or
which spawn first reports it. allowListGapLogged (the separate INFO guard,
#192) is untouched. Neither WARN's wording changed.

fleetd/src/main/java/dev/ltms/fleet/inject/BackendErrorSink.java

@@ -1,12 +1,10 @@
 package dev.ltms.fleet.inject;
 
 /**
- * Notified when {@link CompletionResolver} has a backend-error match at the start of a pane line,
- * or both a match and its too-fast crash signature, for a waiting send (fleetd#201 / #227). A text
- * match inside ordinary pane prose can be a member's report about an error, so it fails the send
- * without notifying this sink.
- * {@link CompletionResolver} calls this only after {@code Rendezvous.resolveFailure} returns
- * {@code true} for that exact waiter, mirroring the win-only race rule {@link ExhaustionSink} uses.
+ * Notified when {@link CompletionResolver} actually delivers a typed backend-error classification
+ * to a waiting send (fleetd#201 / #227) — never on a race that lost. {@link CompletionResolver}
+ * calls this only after {@code Rendezvous.resolveFailure} returns {@code true} for that exact
+ * waiter, mirroring the win-only race rule {@link ExhaustionSink} already uses.
  *
  * <p>The public send result is unchanged by this classification — it is still a failed send
  * ({@code Rendezvous.Kind#FAILED}); this sink is the internal seam a later stage (fleetd#201 Unit

fleetd/src/main/java/dev/ltms/fleet/inject/CompletionResolver.java

@@ -387,10 +387,9 @@ public final class CompletionResolver implements TurnListener {
         // fleetd#164 (part 2) / fleetd#201: a scrape that read cleanly and produced content still
         // isn't a real reply when that content is the backend's own rejection (e.g. an HTTP 400
         // before the worker did any work). Classify it as a failure naming the member, rather than
-        // handing the caller a scrape that reads like a completed answer. A text match alone is not
-        // enough to notify the typed backend-error sink: this assistant block can be a member's
-        // normal prose about an error. A line that starts with the error match is stronger evidence;
-        // the too-fast path below also has its crash signature before it records a credential failure.
+        // handing the caller a scrape that reads like a completed answer, and — only on the
+        // resolution that actually wins the race, mirroring the exhaustion sink above — notify the
+        // typed backend-error sink so a later stage can act on repeated failures.
         String backendError = firstMatchingLine(assistantBlock, backendErrorPatternOrFallback(target));
         if (backendError != null) {
             // Carry the whole scrape, not just the matched line. The pattern is a heuristic: a member
@@ -402,9 +401,9 @@ public final class CompletionResolver implements TurnListener {
             if (rendezvous.resolveFailure(waiter, reason)) {
                 inFlight.remove(target, turn);
                 log.warn("failing send to {} via turn-stall fallback: {}", target, reason);
-                if (startsWithBackendError(backendError, backendErrorPatternOrFallback(target))) {
-                    backendErrorSink.onBackendError(target, backendError, reason);
-                }
+                // fleetd#201 Unit 1: only on the resolution that actually won the race — a late
+                // duplicate must never double-count one backend failure.
+                backendErrorSink.onBackendError(target, backendError, reason);
             }
             return;
         }
@@ -493,9 +492,8 @@ public final class CompletionResolver implements TurnListener {
             if (rendezvous.resolveFailure(waiter, reason)) {
                 inFlight.remove(target, turn);
                 log.warn("failing send to {} via turn-stall fallback from the raw scrape: {}", target, reason);
-                if (startsWithBackendError(backendError, backendErrorPatternOrFallback(target))) {
-                    backendErrorSink.onBackendError(target, backendError, reason);
-                }
+                // fleetd#201 Unit 1: only on the resolution that actually won the race.
+                backendErrorSink.onBackendError(target, backendError, reason);
             }
             return true;
         }
@@ -542,10 +540,10 @@ public final class CompletionResolver implements TurnListener {
      * {@link #MIN_TURN_NANOS} — a crash signature (e.g. a backend HTTP 400 before the worker did
      * anything) that a bare {@code BUSY -> DONE} transition cannot be told apart from a genuinely
      * fast completion. Runs the same backend-error classification the normal and raw-scrape paths
-     * apply, against whatever is on screen right now: a match together with the too-fast crash
-     * signature notifies {@link #backendErrorSink} (only on the resolution that wins the race). A
-     * non-match stays the original generic too-fast failure, naming the member and both timings,
-     * with whatever the pane shows appended so the caller sees the cause, not just "it failed".
+     * apply, against whatever is on screen right now: a match is a typed failure that notifies
+     * {@link #backendErrorSink} (only on the resolution that wins the race); a non-match stays the
+     * original generic too-fast failure, naming the member and both timings, with whatever the pane
+     * shows appended so the caller sees the cause, not just "it failed".
      */
     private void failTooFast(String target, InFlight turn, CompletableFuture<Rendezvous.Resolution> waiter,
                               long elapsedNanos) {
@@ -613,11 +611,6 @@ public final class CompletionResolver implements TurnListener {
         return null;
     }
 
-    /** True when the error pattern begins the matched pane line, rather than appearing in prose. */
-    private static boolean startsWithBackendError(String line, Pattern pattern) {
-        return pattern.matcher(line).lookingAt();
-    }
-
     /**
      * Coverage summary for the CB-578 stage A exhausted-pattern classification, logged at startup
      * the way {@link dev.ltms.fleet.health.FleetHealthMonitor#coverage} is — so an operator can

fleetd/src/main/java/dev/ltms/fleet/member/HerdrPeerLauncher.java

@@ -1665,30 +1665,59 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
 
     /**
      * Guards the {@code effectiveAllowed == null} branch of {@link #logCredentialGap} — the
-     * genuinely-unprotected report (deny-by-default, and the allow-list non-zsh fallback) — to one
-     * WARN per launcher instance, not one per spawn.
+     * genuinely-unprotected report (deny-by-default, and the allow-list non-zsh fallback) — AND
+     * the {@code effectiveAllowed != null} / {@code keptByDerivedList} branch, the allow-list case
+     * where a name is in the gap but the derived allow-list keeps it anyway. Both branches log the
+     * same severity (WARN) about the same fact — a name genuinely reaching a member pane
+     * unprotected — so they share this one guard, keyed per NAME rather than per launcher instance:
+     * each credential-shaped name that is ever reported unprotected gets exactly one WARN, however
+     * many spawns see it and whichever of the two branches first reports it.
+     *
+     * <p>fleetd #341: {@code memberCredentials} is a live, re-read-per-spawn supplier, so the
+     * policy — and so the gap's actual member names — can change between two spawns on the same
+     * launcher. Before this fix the guard was a single {@code AtomicBoolean} tripped by either
+     * branch: spawn 1 could warn about name A and trip the flag, and a later spawn's gap containing
+     * a different name B would never be reported, even though B is just as unprotected as A was.
+     * {@code AtomicBoolean} could not express "once per distinct name" at all — only "once, ever,
+     * for whichever name got there first" — so this is a {@code Set<String>} guard instead, the
+     * same shape {@link OpenCodeLauncher#modelCheckSkippedWarned} already uses for its own
+     * once-per-distinct-thing WARN. {@link #add}'s return value (true only the first time a name is
+     * added) is what turns "log the whole gap" into "log only the names never warned about before".
+     *
+     * <p>Bounded by construction: every name added here first passed {@link
+     * #CREDENTIAL_SHAPED_NAME}'s filter over {@link #hostEnvNames}, i.e. it is an actual
+     * environment variable name from the daemon's own process — a small, OS-bounded set (the host
+     * environment has, in practice, tens to a few hundred entries), not an attacker- or
+     * request-controlled input. So this set cannot grow past "however many distinct credential-
+     * shaped names this host's environment has ever held across this launcher's lifetime," which is
+     * effectively fixed for the life of one daemon process — no separate cap is needed.
      *
      * <p>CB-633 follow-up (#192): kept SEPARATE from {@link #allowListGapLogged} on purpose.
      * {@code memberCredentials} is a live, re-read-per-spawn supplier, so the policy can change
      * between two spawns on the same launcher. A single shared flag would let a harmless allow-list
      * INFO on spawn 1 permanently suppress the real deny-by-default WARN a later spawn deserves —
      * the report that matters most getting hidden by the report that doesn't. Two flags mean each
-     * report kind fires exactly once, independent of what the other kind already logged.
+     * report kind (WARN vs. INFO) fires independently of what the other kind already logged; within
+     * the WARN kind itself, the set above further separates by name, for the same reason.
      */
-    private final AtomicBoolean unprotectedGapLogged = new AtomicBoolean();
+    private final Set<String> unprotectedGapNamesWarned = ConcurrentHashMap.newKeySet();
 
     /**
      * Guards the {@code effectiveAllowed != null} branch of {@link #logCredentialGap} — the
      * allow-list-scrub-covered report — to one INFO per launcher instance. See {@link
-     * #unprotectedGapLogged}'s javadoc for why this is a separate flag rather than a shared one.
+     * #unprotectedGapNamesWarned}'s javadoc for why this is a separate flag rather than a shared
+     * one; unlike that guard it stays a per-instance {@code AtomicBoolean}, not a per-name set —
+     * fleetd #341 fixed the WARN-vs-WARN suppression, not this INFO's own one-shot shape, which was
+     * not reported as broken and is out of that ticket's scope.
      */
     private final AtomicBoolean allowListGapLogged = new AtomicBoolean();
 
     /**
      * fleetd #185 stage 2: guards {@link #warnUnknownMemberEnvironment} to one WARN per launcher
-     * instance, not one per spawn — the same one-per-instance shape as {@link #unprotectedGapLogged}
-     * and {@link #allowListGapLogged}, kept as its own flag for the same reason those two are split:
-     * this mode is orthogonal to which of the other two branches would otherwise have fired.
+     * instance, not one per spawn — the same one-shot shape {@link #unprotectedGapNamesWarned} and
+     * {@link #allowListGapLogged} guard their own branches with, kept as its own flag for the same
+     * reason those two are split: this mode is orthogonal to which of the other two branches would
+     * otherwise have fired.
      */
     private final AtomicBoolean unknownMemberEnvironmentWarned = new AtomicBoolean();
 
@@ -1816,14 +1845,21 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
         List<String> blankedByScrub = gap.stream()
                 .filter(name -> !MemberEnvAllowList.keeps(effectiveAllowed, name))
                 .toList();
-        if (!keptByDerivedList.isEmpty() && unprotectedGapLogged.compareAndSet(false, true)) {
+        // fleetd #341: filter to names this guard has never warned about before — not just
+        // "isEmpty" on the whole branch — so a name this spawn's gap shares with an EARLIER
+        // spawn's (already-warned) gap does not re-print, while a name unique to THIS gap still
+        // does, whichever of the two WARN branches reported it first.
+        List<String> newlyUnprotected = keptByDerivedList.stream()
+                .filter(unprotectedGapNamesWarned::add)
+                .toList();
+        if (!newlyUnprotected.isEmpty()) {
             log.warn("memberCredentials gap: {} credential-shaped env var name(s) are on neither "
                             + "known: nor allow: — the derived allow-list keeps them anyway (a profile's "
                             + "gitTokenEnv/gitHostEnv/tokenEnv/env: names one, or this spawn injects it), "
                             + "so every member pane inherits them UNBLOCKED — {}. Add each to "
                             + "memberCredentials.known (or .allow if a member legitimately needs it), or "
                             + "remove it from whatever profile setting derives it in.",
-                    keptByDerivedList.size(), keptByDerivedList);
+                    newlyUnprotected.size(), newlyUnprotected);
         }
         if (!blankedByScrub.isEmpty() && allowListGapLogged.compareAndSet(false, true)) {
             log.info("memberCredentials gap: {} credential-shaped env var name(s) are on neither "
@@ -1836,12 +1872,18 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
 
     /** The deny-by-default (and allow-list non-zsh fallback) WARN — unchanged byte-for-byte by #192. */
     private void warnGapUnprotected(List<String> gap) {
-        if (unprotectedGapLogged.compareAndSet(false, true)) {
+        // fleetd #341: same "only the names never warned before" filter as the sibling branch in
+        // logCredentialGap above — see unprotectedGapNamesWarned's javadoc. Both branches share
+        // this one guard because both report the exact same fact (a name reaching a member pane
+        // unprotected) at the exact same severity; keying it by name is what lets a later spawn's
+        // DIFFERENT name still get its own WARN after an earlier spawn's already fired.
+        List<String> newlyUnprotected = gap.stream().filter(unprotectedGapNamesWarned::add).toList();
+        if (!newlyUnprotected.isEmpty()) {
             log.warn("memberCredentials gap: {} credential-shaped env var name(s) are on neither "
                             + "known: nor allow: — every member pane inherits them UNBLOCKED — {}. "
                             + "Add each to memberCredentials.known (blocked by default) or .allow "
                             + "(if a member legitimately needs it).",
-                    gap.size(), gap);
+                    newlyUnprotected.size(), newlyUnprotected);
         }
     }
 

fleetd/src/test/java/dev/ltms/fleet/inject/CompletionResolverTest.java

@@ -804,28 +804,7 @@ class CompletionResolverTest {
     // --- fleetd#201 Unit 1: target-keyed backend-error pattern + typed sink ----------------------
 
     @Test
-    void aNormalMemberReportMentioningTheFallbackErrorPatternFailsButDoesNotNotifyTheSink() {
-        String block = "⏺ I checked the retry path. An API Error: makes it back off.\n❯ ";
-        FakeHerdr herdr = new FakeHerdr().readText(block);
-        Rendezvous rendezvous = new Rendezvous();
-        java.util.List<String> notified = new java.util.ArrayList<>();
-        BackendErrorSink sink = (target, matchedLine, reason) -> notified.add(target + ": " + matchedLine);
-        CompletionResolver resolver = new CompletionResolver(new AgentControl(herdr), rendezvous,
-                ExhaustedPatternLookup.none(), ExhaustionSink.none(), BackendErrorPatternLookup.legacy(), sink);
-
-        var waiter = rendezvous.open("term_a");
-        resolver.resolve("term_a", new CompletionResolver.InFlight(waiter, null));
-
-        assertEquals(Rendezvous.Kind.FAILED, waiter.getNow(null).kind(),
-                "a scrape mentioning the pattern must still fail the send");
-        assertTrue(waiter.getNow(null).text().contains("I checked the retry path. An API Error: makes it back off."),
-                "the failure must keep the whole pane tail");
-        assertTrue(notified.isEmpty(),
-                "a normal report mentioning the fallback pattern must not record a credential failure");
-    }
-
-    @Test
-    void aConfiguredBackendErrorPatternAtTheStartOfALineClassifiesAMatchAndNotifiesTheSinkOnce() {
+    void aConfiguredBackendErrorPatternClassifiesAMatchAsAFailureAndNotifiesTheSinkOnce() {
         String block = "⏺ 503 Service Unavailable: upstream credential rejected\n❯ ";
         FakeHerdr herdr = new FakeHerdr().readText(block);
         Rendezvous rendezvous = new Rendezvous();
@@ -945,7 +924,7 @@ class CompletionResolverTest {
     }
 
     @Test
-    void aConfiguredPatternAtTheStartOfALineAlsoClassifiesTheRawScrapeFallbackAndNotifiesTheSink() {
+    void aConfiguredPatternAlsoClassifiesTheRawScrapeFallbackAndNotifiesTheSink() {
         // No ⏺ marker and leading TUI chrome ⇒ lastAssistantBlock() yields "", so classification must
         // fall back to the raw scrape (fleetd#211) — and it must use the configured pattern too.
         String block = """

fleetd/src/test/java/dev/ltms/fleet/member/HerdrPeerLauncherAllowListWiringTest.java

@@ -23,6 +23,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Function;
 import java.util.function.Supplier;
 
@@ -370,6 +371,69 @@ class HerdrPeerLauncherAllowListWiringTest {
                 "expected the pre-existing 'scrub blanks them' INFO unchanged, got: " + messages);
     }
 
+    /**
+     * fleetd #341: {@code unprotectedGapLogged} guarded TWO WARN branches that name DIFFERENT env
+     * var names — the allow-list branch ({@code keptByDerivedList}, below) and the deny-by-default
+     * / non-zsh-fallback branch ({@link HerdrPeerLauncher#warnGapUnprotected}). {@code
+     * memberCredentials} is a live, re-read-per-spawn supplier, so the policy can change between
+     * two spawns on the same launcher instance — a config reload needs no restart. Spawn 1 runs
+     * under {@code deny-by-default} with a gap of {@code SPAWN_ONE_UNCOVERED_TOKEN}, which trips
+     * the (before this fix) SHARED one-shot flag. The policy is then reloaded to {@code
+     * allow-list}; spawn 2's gap is {@code FLEETD_WORKER_TOKEN} instead — the test profile's own
+     * {@code tokenEnv}, which the derived allow-list keeps even though it is on neither {@code
+     * known:} nor {@code allow:}, so it is genuinely unprotected and deserves its own WARN. Before
+     * this fix that WARN never fires, because the shared flag was already {@code true} — the
+     * operator is never told {@code FLEETD_WORKER_TOKEN} reaches every member pane unblocked. Real
+     * path: two real {@link HerdrPeerLauncher#spawn} calls on ONE launcher instance, with mutable
+     * {@code memberCredentials}/host-env suppliers standing in for a live config reload between
+     * spawns.
+     */
+    @Test
+    void aDifferentUnprotectedGapOnALaterSpawnIsNotSuppressedByAnEarlierSpawnsWarn() {
+        FakeHerdr herdr = new FakeHerdr();
+        AtomicReference<FleetConfig.MemberCredentials> credsState = new AtomicReference<>(
+                new FleetConfig.MemberCredentials(null, List.of(), List.of(), null)); // deny-by-default
+        AtomicReference<Set<String>> hostEnvState =
+                new AtomicReference<>(Set.of("SPAWN_ONE_UNCOVERED_TOKEN"));
+        WiringLauncher launcher = new WiringLauncher(herdr, credsState::get, "/bin/zsh", hostEnvState::get);
+
+        Logger logger = (Logger) LoggerFactory.getLogger(HerdrPeerLauncher.class);
+        Level original = logger.getLevel();
+        logger.setLevel(Level.WARN);
+        ListAppender<ILoggingEvent> appender = new ListAppender<>();
+        appender.start();
+        logger.addAppender(appender);
+        try {
+            // Spawn 1: deny-by-default, gap = {SPAWN_ONE_UNCOVERED_TOKEN} — the effectiveAllowed ==
+            // null branch, via warnGapUnprotected.
+            launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
+
+            // Live policy reload to allow-list, with a DIFFERENT gap name.
+            credsState.set(new FleetConfig.MemberCredentials(
+                    FleetConfig.MemberCredentials.POLICY_ALLOW_LIST, List.of(), List.of(), null));
+            hostEnvState.set(Set.of("FLEETD_WORKER_TOKEN"));
+            // Spawn 2: allow-list, gap = {FLEETD_WORKER_TOKEN} — kept by the derived allow-list
+            // (the profile's own tokenEnv), so it is the effectiveAllowed != null / keptByDerivedList
+            // branch, at the SAME log line HerdrPeerLauncher:1819 guards with the shared flag.
+            launcher.spawn(new SpawnRequest("test", null, null, null, null, MemberRole.DEV));
+        } finally {
+            logger.detachAppender(appender);
+            logger.setLevel(original);
+        }
+
+        List<String> messages = appender.list.stream().map(ILoggingEvent::getFormattedMessage).toList();
+        assertTrue(messages.stream().anyMatch(
+                        m -> m.contains("UNBLOCKED") && m.contains("SPAWN_ONE_UNCOVERED_TOKEN")),
+                "spawn 1's deny-by-default gap must still warn — got: " + messages);
+        assertTrue(messages.stream().anyMatch(
+                        m -> m.contains("UNBLOCKED") && m.contains("FLEETD_WORKER_TOKEN")),
+                "spawn 2's gap names a DIFFERENT env var than spawn 1 (FLEETD_WORKER_TOKEN, not "
+                        + "SPAWN_ONE_UNCOVERED_TOKEN) — it must still be warned about even though a "
+                        + "flag already fired once for spawn 1's unrelated name. Before fleetd #341's "
+                        + "fix this WARN never fires because unprotectedGapLogged was already true. "
+                        + "Got: " + messages);
+    }
+
     /**
      * fleetd #185 stage 2: with {@code memberHerdrSocket:} configured, member panes run under a
      * different OS user — {@link HerdrPeerLauncher#hostEnvNames} describes fleetd's own process, not