Compare commits

..

1 Commits

Author SHA1 Message Date
ltms d678783af7 scrub: stop export UID= aborting the allow-list scrub mid-loop
CI / contract (pull_request) Successful in 1m23s
CI / build (pull_request) Successful in 1m32s
CB-633's generated scrub.zsh blanks every non-allow-listed exported name
in one loop wrapped in `{ ... } 2>/dev/null`. In zsh, `export UID=` is
not a failed command: it is a fatal parameter error ("failed to change
user ID: operation not permitted") that aborts the whole sourced file.
The loop stops at UID, every later name is left unscrubbed, and the
report-writing block never runs -- silently, because of the 2>/dev/null.

The inversion is the severity. `env` lists inherited names first and the
names a startup file exports last, so the loop blanked the harmless
inherited half and died immediately before the operator's own exports --
exactly the credentials this policy exists to remove.

Measured on a fleet01 member pane: UID is name 42 of 57, and a decoy
exported from ~/.zshrc sat at 58, non-empty, on 8 of 8 spawns. xtrace
ends at `+scrub.zsh:28> export 'UID='` with rc=126. Reproduced
independently on macOS/zsh 5.9 (fleetd #394).

CONTAINMENT: `eval`, then VERIFY.

Neither `export "$n=" 2>/dev/null || true` nor a `${(t)n}` type guard
contains it -- both still abort, because it is an assignment error and
not a command failure. Only `eval` does. `eval` is safe at that point
because the loop above already rejected every name that is not
[A-Za-z_][A-Za-z0-9_]*, so nothing but a bare identifier reaches it.

Enumerating the specials (UID|EUID|GID|EGID|PPID|LINENO) also works, but
only for the names enumerated: one that turns up exported on another
host brings the abort straight back. `eval` covers all of them.

`eval` alone, though, contains the error WITHOUT blanking the value, so
the name would be reported as blanked having never been blanked --
measured: attempted=59 with UID still 1000. So each name is verified
after the attempt and only counted when it actually blanked. A name that
could not be blanked currently falls into the "allowed" count, which is
imprecise in the safe direction; the honest third count ("tried and could
not blank") needs a report-format change -- readReport treats every
non-first non-blank line as a blanked name -- and belongs with #394.

Why the suite stayed green: EnvAllowListScrubTest starts zsh from a
CLEARED parent (`pb.environment().clear()`), where UID is not an
exported name at all, so the abort cannot reproduce there. The new test
supplies the production shape -- UID present and exported -- and asserts
a report exists, which is written by the last statement in the file and
so proves the script ran to completion. Verified by mutation in both
shapes: reverting the fix fails exactly that test and no other.

Orthogonal to #388, which adds a fifth call site to a script that still
aborts at the same name.
2026-09-10 01:12:27 +00:00
9 changed files with 74 additions and 622 deletions
+4 -10
View File
@@ -137,7 +137,7 @@ the merge — and merging on a reviewer's word is delegating it by proxy.
| Answer a member's `fleet_ask` | `fleet_send{turnId, content}` — **not** `sessionId` |
| Message a **peer lead** on this host | `fleet_send{sessionId: <their terminal>, content}` — `fleet_list` → `leads` reports it. Coordination only, **never** a task |
| Message a **peer lead** on another daemon or host | `fleet_send{coordId: <their coord-id>, content}` — needs a `coordinator:` block; your own coord-id is in `fleet_list`. Coordination only, **never** a task |
| Answer a peer lead that messaged you | `fleet_send{coordId}` — or `{sessionId}` if they are on this host. **Not** `fleet_reply`: it has no peer route and the publish is refused |
| Answer a peer lead that messaged you | `fleet_reply{content}` — the one case a lead replies |
| Collect a held reply | `fleet_poll{target}` · then `fleet_ack{target, msgId}` |
| Tear down a member | `fleet_stop{paneId}` |
@@ -163,21 +163,15 @@ The traffic between leads is coordination and nothing else:
3. **Verify a peer exactly as you verify yourself.** Peer status buys nothing: check the claim
against the code, and re-run the build. A peer's correction gets the same treatment — right or
wrong on the evidence, not on who said it. Neither of you merges the other's work unreviewed.
**N observations are N data points only if they differ in the axis you are trusting.** This cuts
both ways. N *failures* blamed on one cause are one data point when the cases share what you are
not varying. N *agreeing measurements* are also one data point when they share an instrument —
two hosts, two operators and the same formula is one formula, not two confirmations.
4. **Ask a peer to read your project addendum.** Your addendum is instruction surface: every future
session on your host obeys it, and a wrong one is obeyed just as faithfully as a right one. The
author is the worst reader of their own qualifier placement — measured here, one addendum carried
two defects and a non-author found both. If you have no peer, at least re-read it asking "which
sentence goes false first, and would a reader reach the caveat before acting?"
Being messaged by a peer does not make you its worker: answer the way you would open —
`fleet_send{coordId}` for another daemon, `fleet_send{sessionId}` on this host — and push back on
the substance if it is wrong. `fleet_reply` resolves a member's blocked `fleet_send`; a peer's
coord-id message is durable and non-blocking, so there is nothing for it to resolve. A peer that
simply complies has thrown away the reason there are two of you.
Being messaged by a peer does not make you its worker: answer with `fleet_reply`, and push back on
the substance if it is wrong. A peer that simply complies has thrown away the reason there are two of
you.
### Member (worker or architect) — the turn contract
@@ -134,10 +134,6 @@ public final class Fleetd {
// secret is reported above, so upgrading past this commit never silently drops CB-592's
// protection.
reportMemberCredentialsGap(cfg);
// fleetd #395: an unset exhaustedPattern is a silent opt-out of usage-limit detection for
// that profile — say so loudly, the same way the two reports above do, rather than let an
// operator discover it only when a limit goes undetected.
reportExhaustedPatternGap(cfg);
// CB-559: `cfg` stays the startup snapshot — every validation and every piece of one-time
// wiring below reads it, and must, because those decisions cannot be unmade. `config` is the
// live reference the hot paths read per use. Which keys can actually move is ConfigRef's
@@ -665,8 +661,10 @@ public final class Fleetd {
// built a second time — two independently-constructed sources reading the SAME BackendQuarantine
// / BackendOutagePolicy would still be able to drift (e.g. a future edit to the credentialIdFor
// closure in only one of the two places), exactly the shape #284 was.
FleetMcp.QuarantineSource quarantineSource = quarantineSource(config, quarantine,
exhaustedPatternsByProfile);
FleetMcp.QuarantineSource quarantineSource = new FleetMcp.QuarantineSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
}, quarantine);
FleetMcp.OutageSource outageSource = new FleetMcp.OutageSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
@@ -800,19 +798,6 @@ public final class Fleetd {
return target -> presence.isPresent(target) || leads.get().containsKey(target);
}
/**
* fleetd #404: production source for quarantine reporting. Credential IDs are hot, but
* exhausted patterns are compiled once at startup for {@link CompletionResolver}, so the armed
* field must use that same compiled map until restart.
*/
static FleetMcp.QuarantineSource quarantineSource(ConfigRef config, BackendQuarantine quarantine,
Map<String, Pattern> startupExhaustedPatterns) {
return new FleetMcp.QuarantineSource(profile -> {
var configured = config.get().profiles().get(profile);
return configured == null ? null : configured.effectiveCredentialId();
}, quarantine, profile -> startupExhaustedPatterns.containsKey(profile));
}
/**
* fleetd #248: package-private factory for the member worktree/branch lookup {@link
* CompletionResolver} uses to name a fallback report's worktree and branch (fleetd#241).
@@ -1338,52 +1323,6 @@ public final class Fleetd {
+ "fleetd.yaml — see fleetd.example.yaml — and restart.");
}
/**
* fleetd #395: {@code exhaustedPattern} (see {@link FleetConfig.Profile#exhaustedPattern}) is
* deliberately opt-in — {@code null}/blank means a backend refusal on that profile is never
* classified as {@code BACKEND_EXHAUSTED}, so its credential is never quarantined. That is a
* legitimate choice (guessing the vendor's wording would be worse), but an operator who never
* opted a profile in should not discover the gap only when a usage limit silently goes
* undetected. Warn once at startup, naming every unarmed profile, exactly like {@link
* #reportMemberCredentialsGap} — never refuse to start over it.
*
* <p>A {@code subscription: true} profile that is unarmed gets a SECOND, louder WARN of its
* own: it bills the operator's metered Claude plan, the case where an undetected usage limit
* costs the most.
*
* <p>Package-private so a test can capture the real log via a {@link
* ch.qos.logback.core.read.ListAppender}, the same pattern {@link
* #reportMemberCredentialsGap}'s own test uses.
*/
static void reportExhaustedPatternGap(FleetConfig cfg) {
List<String> unarmedSubscription = new ArrayList<>();
List<String> unarmedOther = new ArrayList<>();
cfg.profiles().forEach((name, profile) -> {
if (!profile.hasExhaustedPattern()) {
(profile.isSubscription() ? unarmedSubscription : unarmedOther).add(name);
}
});
if (unarmedSubscription.isEmpty() && unarmedOther.isEmpty()) {
log.info("exhaustedPattern: every configured profile has usage-limit detection armed");
return;
}
List<String> allUnarmed = new ArrayList<>(unarmedSubscription);
allUnarmed.addAll(unarmedOther);
allUnarmed = allUnarmed.stream().sorted().toList();
log.warn("exhaustedPattern: profile(s) {} have no exhaustedPattern configured — a "
+ "usage-limit refusal on any of them is never detected and never "
+ "quarantines its credential. Set exhaustedPattern (see "
+ "fleetd.example.yaml) to arm detection for a profile.",
allUnarmed);
if (!unarmedSubscription.isEmpty()) {
List<String> sortedSubscription = unarmedSubscription.stream().sorted().toList();
log.warn("exhaustedPattern: subscription profile(s) {} run on the operator's metered "
+ "Claude plan and have NO usage-limit detection armed — this is the "
+ "case where a missed usage limit costs the most.",
sortedSubscription);
}
}
/**
* Poll herdr's {@code ping} until it answers or {@link #HERDR_WAIT_SECONDS} elapses (CB-504).
*
@@ -119,28 +119,9 @@ public final class FleetMcp {
/**
* CB-578 stage B quarantine facts used by {@code fleet_profiles}: a profile → credential id
* lookup, plus the shared {@link BackendQuarantine} to read remaining cooldowns off.
*
* @param exhaustedPatternArmed fleetd #395: profile → whether that profile's {@code
* exhaustedPattern} is configured (see {@code
* FleetConfig.Profile#hasExhaustedPattern}), i.e. whether a backend refusal
* on it can EVER be classified {@code BACKEND_EXHAUSTED} and quarantine its
* credential. Bundled here, not a separate Source, because it answers the
* exact question {@code fleet_profiles}'s quarantine facts already answer
* for a QUARANTINED profile — "can this profile's usage limit ever be
* caught?" — just for every profile, not only one currently caught.
*/
public record QuarantineSource(Function<String, String> credentialIdFor, BackendQuarantine quarantine,
Function<String, Boolean> exhaustedPatternArmed) {
/**
* Backward-compatible 2-arg form, before fleetd #395 added {@code exhaustedPatternArmed} —
* reports every profile unarmed. Keeps every pre-existing call site (production and test)
* compiling and behaving identically for the quarantine facts they actually asked for.
*/
public QuarantineSource(Function<String, String> credentialIdFor, BackendQuarantine quarantine) {
this(credentialIdFor, quarantine, _ -> false);
}
/** Inert source — no profile is ever reported quarantined or armed. Explicit stand-in, not a default. */
public record QuarantineSource(Function<String, String> credentialIdFor, BackendQuarantine quarantine) {
/** Inert source — no profile is ever reported quarantined. Explicit stand-in, not a default. */
public static QuarantineSource none() { return new QuarantineSource(_ -> null, BackendQuarantine.none()); }
}
@@ -1128,14 +1109,6 @@ public final class FleetMcp {
* not the other, and the two then disagree about a live outage. That is exactly what fleetd
* #284 was, where one rule computed in two places was widened in only one and a single response
* contradicted itself. Shared inputs do not make duplicated computation safe.
*
* <p>fleetd #395: also reports {@code exhaustionDetectionArmed}, one boolean per configured
* profile — {@code true} when that profile's {@code exhaustedPattern} is set, {@code false}
* when it is not, so an operator can tell "this profile is healthy" from "nothing can ever
* quarantine this profile" without reading {@code fleetd.yaml}. Unlike {@code quarantined}/
* {@code coolingOff}, this map always names every profile: an unarmed profile never enters a
* transient state to be absent from, so silence here would read as "healthy" rather than "not
* being watched at all".
*/
public static Map<String, Object> profilesView(PeerLauncher workers, QuarantineSource quarantine, OutageSource outage) {
Map<String, Object> result = new LinkedHashMap<>();
@@ -1143,9 +1116,7 @@ public final class FleetMcp {
result.put("default", workers.defaultProfile() == null ? "" : workers.defaultProfile());
Map<String, Object> quarantined = new LinkedHashMap<>();
Map<String, Object> coolingOff = new LinkedHashMap<>();
Map<String, Object> exhaustionDetectionArmed = new LinkedHashMap<>();
for (String profile : workers.profiles()) {
exhaustionDetectionArmed.put(profile, quarantine.exhaustedPatternArmed().apply(profile));
String credentialId = quarantine.credentialIdFor().apply(profile);
if (credentialId != null) {
quarantine.quarantine().remainingSeconds(credentialId).ifPresent(remaining -> {
@@ -1165,7 +1136,6 @@ public final class FleetMcp {
});
}
}
result.put("exhaustionDetectionArmed", exhaustionDetectionArmed);
if (!quarantined.isEmpty()) {
result.put("quarantined", quarantined);
}
@@ -1693,11 +1663,7 @@ public final class FleetMcp {
"List the configured worker profiles (backends) and which one fleet_spawn uses by "
+ "default. A 'quarantined' map is present when a backend-exhausted refusal put "
+ "a profile's credential on cooldown — fleet_spawn onto it is refused until "
+ "quarantinedForSeconds elapses; a profile sharing that credential is listed too. "
+ "'exhaustionDetectionArmed' reports, per profile, whether a usage-limit refusal "
+ "on it can EVER be classified and quarantined (its exhaustedPattern is "
+ "configured) — false means that profile's credential can never be quarantined "
+ "by this mechanism, however many usage-limit refusals it sees.",
+ "quarantinedForSeconds elapses; a profile sharing that credential is listed too.",
objectSchema(Map.of(), List.of()));
}
@@ -75,28 +75,9 @@ import java.util.stream.Stream;
* never to the control.
*
* <p>The scrub also writes {@code scrub-report.txt} into its own directory: one {@code allowed N of
* M failed F} line (N = exports left untouched, M = exports present when the scrub ran, F = names
* the scrub attempted to blank but could not), then the NAMES — blanked ones bare, unblankable ones
* {@code !}-prefixed — never values. The launcher reads this back at teardown and logs it, because a
* blocked count next to an unknown denominator is not a finding.
*
* <p><b>fleetd #394:</b> plain {@code export "$n="} is a FATAL error for a zsh read-only or special
* parameter (for example {@code UID}) — it aborts the whole sourced file, so every name still to
* come is never blanked and the report above is never written at all. The blanking loop instead
* routes each attempt through {@code eval}, which contains that error to the single iteration: the
* loop always finishes, and a name that could not be blanked is counted as {@code failed} and
* listed {@code !}-prefixed rather than silently disappearing. This is deliberately not a skip-list
* of known-bad names — every enumerated name is still attempted, so a name nobody has thought of
* yet still gets tried and, if it fails, still gets counted.
*
* <p>The blanking loop also re-asserts, on its own, the same {@code [A-Za-z_][A-Za-z0-9_]*} shape
* check the enumeration loop already applied. Before {@code eval} was introduced a non-conforming
* name reaching {@code export "$n="} was harmless either way — the quoting made it inert. With
* {@code eval}, the name is spliced into a string and interpreted as shell syntax, so the enumeration
* loop's check is no longer sufficient on its own to keep that call site safe — it is a guard on a
* different loop, and the two must not silently drift apart. Re-checking right before the
* {@code eval} keeps that call site safe by its own reading, independent of whatever the enumeration
* loop does or stops doing in a later change.
* M} line (N = exports left untouched, M = exports present when the scrub ran), then the blanked
* NAMES — never values. The launcher reads this back at teardown and logs it, because a blocked
* count next to an unknown denominator is not a finding.
*/
public final class EnvAllowListScrub {
@@ -151,16 +132,10 @@ public final class EnvAllowListScrub {
}
/**
* A parsed {@code scrub-report.txt}: how many exported variables existed when the scrub ran
* ({@code total}), how many were left untouched ({@code allowed}), how many the scrub attempted
* to blank but could not ({@code failed} — fleetd #394: a zsh read-only/special parameter such
* as {@code UID} fatally errors on plain {@code export NAME=}, so those attempts go through
* {@code eval} instead so the loop keeps going and the failure is counted rather than left
* invisible), and the NAMES in each of the latter two categories. {@code allowed +
* blanked.size() + unblankable.size() == total}, and {@code unblankable.size() == failed}.
* Values never appear.
* A parsed {@code scrub-report.txt}: how many exported variables existed when the scrub ran,
* how many were left untouched (allowed), and the NAMES that were blanked. Values never appear.
*/
record ScrubReport(int allowed, int total, int failed, List<String> blanked, List<String> unblankable) {
record ScrubReport(int allowed, int total, List<String> blanked) {
}
/**
@@ -359,46 +334,38 @@ public final class EnvAllowListScrub {
_cb633_blank+=("$_cb633_n")
done
# fleetd #394: plain `export "$n="` is FATAL for a zsh read-only/special parameter
# (e.g. UID) and aborts this whole sourced file — every name still to come is never
# blanked, and the report below is never written, silently. `eval` contains that
# error to the single iteration instead: it still fails for that one name, but the
# loop continues and we can tell allowed / blanked / unblankable apart afterwards.
# This is not a skip-list of known-bad names (that would miss the next one nobody
# thought of) — every name in _cb633_blank is still attempted, unconditionally.
# Every name reaching this loop already passed the identical identifier check in the
# enumeration loop above — but that guard is 20 lines away in a different loop, and
# this line is about to splice the name into a string handed to `eval`. Before this
# fix the name only ever reached `export` quoted ("$n="), which is inert on a
# non-identifier string either way; `eval` makes THIS line the only thing standing
# between such a string and code execution in the member's pane, so it re-asserts the
# same check on its own rather than trusting a guard it does not own. Under normal
# operation this can never fire (the enumeration guard already filtered everything
# reaching _cb633_blank), so a name caught here is counted as unblankable rather than
# silently dropped — it is real evidence that the upstream guard was bypassed.
typeset -a _cb633_ok _cb633_unblankable
_cb633_ok=()
_cb633_unblankable=()
# `export UID=` is not a failed command: it is a FATAL zsh parameter error
# ("failed to change user ID") that aborts this whole sourced file mid-loop,
# leaving every later name unscrubbed and the report below unwritten — silently,
# because of the 2>/dev/null. Neither `|| true` nor a `${(t)n}` type guard
# contains it; only `eval` does. `eval` is safe here precisely because the loop
# above already rejected every name that is not [A-Za-z_][A-Za-z0-9_]*, so
# nothing but a bare identifier can reach it.
#
# Enumerating the special names instead (UID|EUID|GID|EGID|PPID|LINENO) also
# works, but only for the ones enumerated: a special that turns up exported on
# some other host brings the abort straight back. `eval` contains all of them.
#
# Then VERIFY. A contained failure is still a failure, so a name that did not
# actually blank must not be reported as blanked. It currently falls into the
# "allowed" count, which is imprecise in the safe direction; the honest third
# count ("tried and could not blank") needs a report-format change and belongs
# with fleetd #394, not here.
typeset -a _cb633_done
_cb633_done=()
for _cb633_n in "${_cb633_blank[@]}"; do
if [[ ! "$_cb633_n" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]]; then
_cb633_unblankable+=("$_cb633_n")
continue
fi
if eval "export ${_cb633_n}=" 2>/dev/null; then
_cb633_ok+=("$_cb633_n")
else
_cb633_unblankable+=("$_cb633_n")
fi
eval "export ${_cb633_n}=" 2>/dev/null
[[ -z "${(P)_cb633_n}" ]] && _cb633_done+=("$_cb633_n")
done
_cb633_blank=("${_cb633_done[@]}")
integer _cb633_kept=$(( _cb633_total - ${#_cb633_blank} ))
{
print -r -- "allowed $_cb633_kept of $_cb633_total failed ${#_cb633_unblankable}"
for _cb633_n in "${_cb633_ok[@]}"; do print -r -- "$_cb633_n"; done
for _cb633_n in "${_cb633_unblankable[@]}"; do print -r -- "!$_cb633_n"; done
print -r -- "allowed $_cb633_kept of $_cb633_total"
for _cb633_n in "${_cb633_blank[@]}"; do print -r -- "$_cb633_n"; done
} > "$ZDOTDIR/%s" 2>/dev/null
unset _cb633_allowed _cb633_names _cb633_blank _cb633_ok _cb633_unblankable _cb633_n _cb633_total _cb633_kept
unset _cb633_done _cb633_allowed _cb633_names _cb633_blank _cb633_n _cb633_total _cb633_kept
""".formatted(names, MemberEnvAllowList.zshCasePattern(), REPORT_FILE);
}
@@ -412,11 +379,6 @@ public final class EnvAllowListScrub {
* Read and parse {@link #REPORT_FILE} out of a generated ZDOTDIR directory. Returns {@code null}
* when absent or unreadable (the pane may have been torn down before its login shell ever got to
* the scrub) — callers treat that as "no measurement available", never as success.
*
* <p>First line is {@code "allowed <N> of <M> failed <F>"} (fleetd #394 added the trailing
* {@code failed <F>} — a count of names the scrub attempted to blank but could not, e.g. a zsh
* read-only/special parameter). Every following non-blank line is a name: a bare name was
* blanked, a {@code !}-prefixed name was attempted and failed. Values never appear on either.
*/
static ScrubReport readReport(Path zdotdir) {
Path report = zdotdir.resolve(REPORT_FILE);
@@ -429,24 +391,17 @@ public final class EnvAllowListScrub {
return null;
}
String[] parts = lines.getFirst().substring("allowed ".length()).trim().split("\\s+");
if (parts.length != 5 || !"of".equals(parts[1]) || !"failed".equals(parts[3])) {
if (parts.length != 3 || !"of".equals(parts[1])) {
return null;
}
List<String> blanked = new ArrayList<>();
List<String> unblankable = new ArrayList<>();
for (int i = 1; i < lines.size(); i++) {
String line = lines.get(i);
if (line.isBlank()) {
continue;
}
if (line.startsWith("!")) {
unblankable.add(line.substring(1));
} else {
blanked.add(line);
if (!lines.get(i).isBlank()) {
blanked.add(lines.get(i));
}
}
return new ScrubReport(Integer.parseInt(parts[0]), Integer.parseInt(parts[2]),
Integer.parseInt(parts[4]), List.copyOf(blanked), List.copyOf(unblankable));
List.copyOf(blanked));
} catch (IOException | NumberFormatException e) {
return null;
}
@@ -1643,19 +1643,11 @@ public abstract class HerdrPeerLauncher implements PeerLauncher {
log.warn("memberCredentials allow-list: pane {} left no scrub report in {} — the "
+ "environment scrub cannot be confirmed to have run. Either the pane ended "
+ "before its shell finished starting, or its shell never read our generated "
+ "startup files. Either way, we cannot tell from here whether the scrub ran, "
+ "so we do not know what that member's environment contained.",
+ "startup files, in which case that member saw the full host environment.",
paneId, dir);
} else {
log.info("memberCredentials allow-list: pane {} allowed {} of {} environment variables",
paneId, report.allowed(), report.total());
if (report.failed() > 0) {
log.warn("memberCredentials allow-list: pane {} could not blank {} environment "
+ "variable(s) — {} (likely a zsh read-only/special parameter) — those "
+ "names were left in the member's environment. Confirm none of them is a "
+ "credential.",
paneId, report.failed(), report.unblankable());
}
List<String> shaped = report.blanked().stream()
.filter(name -> CREDENTIAL_SHAPED_NAME.matcher(name).matches())
.toList();
@@ -1,150 +0,0 @@
package dev.ltms.fleet;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.read.ListAppender;
import dev.ltms.fleet.config.FleetConfig;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.slf4j.LoggerFactory;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #395: {@code exhaustedPattern} (see {@link FleetConfig.Profile#exhaustedPattern}) is
* deliberately opt-in — an unset one leaves usage-limit detection silently OFF for that profile,
* and nothing quarantines its credential. {@link Fleetd#reportExhaustedPatternGap} must say so at
* startup, naming every unarmed profile, and must never fire when every profile is armed. Mirrors
* {@link MemberCredentialsGapReportTest}'s pattern, capturing the real log via a
* {@link ListAppender}.
*
* <p>The 8-profile shape in {@link #theLiveEightProfileShapeWarnsExactlyTheSixUnarmedProfiles} is
* the live {@code fleetd.yaml} shape measured 2026-09-10 (fleetd #395's own ticket): 6 of 8
* profiles unarmed, 2 of those 6 ({@code opus}, {@code sonnet}) running on the operator's Claude
* subscription. {@code fleetd.yaml} itself is gitignored and unavailable to this test, so the
* shape is reproduced as a throwaway config in a {@code @TempDir} rather than read off disk.
*/
class ExhaustedPatternGapReportTest {
private static FleetConfig load(Path dir, String yaml) throws Exception {
Path f = dir.resolve("fleetd.yaml");
Files.writeString(f, yaml);
return FleetConfig.load(f);
}
private static ListAppender<ILoggingEvent> attach() {
Logger logger = (Logger) LoggerFactory.getLogger(Fleetd.class);
ListAppender<ILoggingEvent> appender = new ListAppender<>();
appender.start();
logger.addAppender(appender);
return appender;
}
private static void detach(ListAppender<ILoggingEvent> appender) {
((Logger) LoggerFactory.getLogger(Fleetd.class)).detachAppender(appender);
}
/** Every profile name mentioned by a WARN-level log line, across every WARN this call produced. */
private static List<String> warnMessages(ListAppender<ILoggingEvent> appender) {
return appender.list.stream()
.filter(e -> e.getLevel() == Level.WARN)
.map(ILoggingEvent::getFormattedMessage)
.toList();
}
@Test
void theLiveEightProfileShapeWarnsExactlyTheSixUnarmedProfiles(@TempDir Path dir) throws Exception {
// Reproduces the live shape measured 2026-09-10: 8 profiles, 2 armed (sol, terra), 6
// unarmed (local, local-direct, gx, opus, sonnet, xf) — 2 of the unarmed 6 (opus, sonnet)
// are subscription: true.
FleetConfig cfg = load(dir, """
profiles:
local:
baseUrl: http://gx00.gw:8000
local-direct:
baseUrl: http://gx01.gw:8000
gx:
kind: opencode
baseUrl: https://llm.ltms.dev/v1
opus:
subscription: true
model: claude-opus-5
sonnet:
subscription: true
model: claude-sonnet-5
sol:
baseUrl: https://llm.ltms.dev/v1
exhaustedPattern: "The usage limit has been reached"
terra:
baseUrl: https://llm.ltms.dev/v1
exhaustedPattern: "The usage limit has been reached"
xf:
baseUrl: https://llm.ltms.dev/v1
""");
ListAppender<ILoggingEvent> appender = attach();
try {
Fleetd.reportExhaustedPatternGap(cfg);
} finally {
detach(appender);
}
List<String> warns = warnMessages(appender);
assertFalse(warns.isEmpty(), "6 of 8 profiles are unarmed — at least one WARN must fire");
// Exactly one WARN aggregates every unarmed profile, naming all 6 and none of the 2 armed.
String aggregate = warns.stream()
.filter(m -> m.contains("no exhaustedPattern configured"))
.findFirst()
.orElseThrow(() -> new AssertionError("expected an aggregate unarmed-profiles WARN: " + warns));
for (String unarmed : List.of("local", "local-direct", "gx", "opus", "sonnet", "xf")) {
assertTrue(aggregate.contains(unarmed), "aggregate WARN must name '" + unarmed + "': " + aggregate);
}
for (String armed : List.of("sol", "terra")) {
assertFalse(aggregate.contains(armed), "aggregate WARN must NOT name armed profile '" + armed + "': " + aggregate);
}
// A second, louder WARN calls out the subscription profiles specifically.
String subscriptionWarn = warns.stream()
.filter(m -> m.contains("metered Claude plan"))
.findFirst()
.orElseThrow(() -> new AssertionError("expected a subscription-specific WARN: " + warns));
assertTrue(subscriptionWarn.contains("opus"), subscriptionWarn);
assertTrue(subscriptionWarn.contains("sonnet"), subscriptionWarn);
assertFalse(subscriptionWarn.contains("local-direct"),
"the subscription WARN must not name a non-subscription profile: " + subscriptionWarn);
}
@Test
void everyProfileArmedProducesNoWarningAtAll(@TempDir Path dir) throws Exception {
FleetConfig cfg = load(dir, """
profiles:
sol:
baseUrl: https://llm.ltms.dev/v1
exhaustedPattern: "The usage limit has been reached"
terra:
baseUrl: https://llm.ltms.dev/v1
exhaustedPattern: "The usage limit has been reached"
opus:
subscription: true
model: claude-opus-5
exhaustedPattern: "5-hour limit reached"
""");
ListAppender<ILoggingEvent> appender = attach();
try {
Fleetd.reportExhaustedPatternGap(cfg);
} finally {
detach(appender);
}
assertTrue(warnMessages(appender).isEmpty(),
"every profile is armed — a checker that warns anyway always fires: " + warnMessages(appender));
}
}
@@ -1,94 +0,0 @@
package dev.ltms.fleet;
import dev.ltms.fleet.config.ConfigRef;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.mcp.FleetMcp;
import dev.ltms.fleet.placement.BackendQuarantine;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Map;
import java.util.regex.Pattern;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #404: {@code exhaustionDetectionArmed} must describe the startup pattern map, not the
* reloaded config snapshot.
*
* <p>This test needs a reload. At startup the two snapshots agree, so a test of only a newly
* started daemon would not detect a live {@code config.get()} lookup in the report field.
*/
class FleetdExhaustionDetectionArmedWiringTest {
private static final String NO_PATTERN = """
bind:
host: 127.0.0.1
port: 8765
herdrSocket: ~/.config/herdr/herdr.sock
profiles:
terra:
baseUrl: http://gx00.gw:8000
model: terra
guard:
offSubscriptionHosts:
- gx00.gw
""";
private static final String WITH_PATTERN = """
bind:
host: 127.0.0.1
port: 8765
herdrSocket: ~/.config/herdr/herdr.sock
profiles:
terra:
baseUrl: http://gx00.gw:8000
model: terra
exhaustedPattern: "usage limit"
guard:
offSubscriptionHosts:
- gx00.gw
""";
@Test
@DisplayName("reloading an exhaustedPattern does not arm the startup detection source")
void reloadedPatternDoesNotChangeTheArmedFieldUntilRestart(@TempDir Path dir) throws Exception {
Path file = dir.resolve("fleetd.yaml");
Files.writeString(file, NO_PATTERN);
ConfigRef config = new ConfigRef(file, FleetConfig.load(file));
Files.writeString(file, WITH_PATTERN);
assertTrue(config.reload().applied());
assertTrue(config.get().profiles().get("terra").hasExhaustedPattern());
FleetMcp.QuarantineSource source = Fleetd.quarantineSource(config, BackendQuarantine.none(),
Map.of());
assertFalse(source.exhaustedPatternArmed().apply("terra"),
"exhaustionDetectionArmed must use the startup pattern map, not config.get()");
}
@Test
@DisplayName("a profile in the startup pattern map is reported as armed")
void aProfileInTheStartupMapIsArmed(@TempDir Path dir) throws Exception {
// fleetd #404, second direction. The test above only ever passes an EMPTY startup map, so
// it cannot tell a correct lookup from one that is permanently off. Measured: replacing the
// armed lambda with `profile -> false` left the whole suite green at 1475 tests. That
// mutation would make #395's visibility feature dead — an operator fixing a detection gap
// would be told the gap is still open after fixing it, forever. Both directions are needed:
// this test is the only thing that fails when the field stops reporting armed at all.
Path file = dir.resolve("fleetd.yaml");
Files.writeString(file, WITH_PATTERN);
ConfigRef config = new ConfigRef(file, FleetConfig.load(file));
FleetMcp.QuarantineSource source = Fleetd.quarantineSource(config, BackendQuarantine.none(),
Map.of("terra", Pattern.compile("usage limit")));
assertTrue(source.exhaustedPatternArmed().apply("terra"),
"a profile whose pattern was compiled at startup must report armed");
assertFalse(source.exhaustedPatternArmed().apply("sonnet"),
"a profile absent from the startup map must not report armed");
}
}
@@ -1,62 +0,0 @@
package dev.ltms.fleet.mcp;
import dev.ltms.fleet.config.FleetConfig;
import dev.ltms.fleet.guard.SubscriptionGuard;
import dev.ltms.fleet.herdr.AgentControl;
import dev.ltms.fleet.herdr.FakeHerdr;
import dev.ltms.fleet.herdr.WorkspaceControl;
import dev.ltms.fleet.member.ClaudeCodeLauncher;
import dev.ltms.fleet.peer.PeerLauncher;
import io.modelcontextprotocol.spec.McpSchema;
import org.junit.jupiter.api.Test;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Set;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
* fleetd #395: {@code fleet_profiles} must let an operator tell "this profile's usage-limit
* detection is armed" from "nothing can ever quarantine this profile" — see {@link
* FleetMcp.QuarantineSource#exhaustedPatternArmed()} and {@link
* FleetMcp#profilesView(PeerLauncher, FleetMcp.QuarantineSource, FleetMcp.OutageSource)}.
*/
class FleetProfilesArmedFieldTest {
private static PeerLauncher twoProfileLauncher(FakeHerdr h) {
FleetConfig.Profile armed = new FleetConfig.Profile(
"armed-profile", "http://gx00.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN", null,
"tab", "fleetd-workers", "worker: {profile} #{n}", null, null, null);
FleetConfig.Profile unarmed = new FleetConfig.Profile(
"unarmed-profile", "http://gx01.gw:8000", "coder", null, "FLEETD_WORKER_TOKEN", null,
"tab", "fleetd-workers", "worker: {profile} #{n}", null, null, null);
Map<String, FleetConfig.Profile> profiles = new LinkedHashMap<>();
profiles.put(armed.profile(), armed);
profiles.put(unarmed.profile(), unarmed);
return new ClaudeCodeLauncher(new AgentControl(h), new WorkspaceControl(h),
new SubscriptionGuard(Set.of("gx00.gw", "gx01.gw")), profiles, armed.profile(), _ -> "tok");
}
private static String textOf(McpSchema.CallToolResult r) {
return ((McpSchema.TextContent) r.content().getFirst()).text();
}
@Test
void armedProfileReportsArmedAndUnarmedReportsUnarmed() {
FakeHerdr h = new FakeHerdr();
PeerLauncher workers = twoProfileLauncher(h);
FleetMcp.QuarantineSource source = new FleetMcp.QuarantineSource(
_ -> null, dev.ltms.fleet.placement.BackendQuarantine.none(),
profile -> "armed-profile".equals(profile));
McpSchema.CallToolResult res = FleetMcp.profiles(workers, source);
assertNotEquals(Boolean.TRUE, res.isError());
String out = textOf(res);
assertTrue(out.contains("\"exhaustionDetectionArmed\""), out);
assertTrue(out.contains("\"armed-profile\":true"), out);
assertTrue(out.contains("\"unarmed-profile\":false"), out);
}
}
@@ -120,128 +120,40 @@ class EnvAllowListScrubTest {
}
/**
* fleetd #394: the actual defect. Plain {@code export "$n="} is FATAL for a zsh read-only or
* special parameter (e.g. {@code UID}) and aborts the whole sourced file — every name still to
* come is never blanked, and the {@code scrub-report.txt} below is never written at all,
* silently ({@code 2>/dev/null} swallows the error). This plants an unblankable, exported,
* read-only variable in the MIDDLE of the names the scrub attempts to blank, with two more
* names after it, and asserts that both of those later names are STILL blanked and the report
* is STILL written with the failure counted — a test that only checked names BEFORE the failure
* point would pass today and prove nothing.
* A pane inherits {@code UID}; a cleared test parent does not. The scrub must survive it.
*
* <p>The planted name is a made-up one ({@code FLEETD_TEST_UNBLANKABLE}), not {@code UID} or
* any other name a skip-list might already know about — invariant 1 is that the loop survives
* ANY unblankable name, not a known one, so the test must not lean on one either.
* <p>Every other test here starts zsh from a CLEARED environment, so {@code UID} is never an
* exported name and never reaches the blanking loop. In a real member pane it is exported and
* it IS reached — and {@code export UID=} is a fatal zsh parameter error that aborts the whole
* sourced file, leaving every later name unscrubbed and writing no report at all. The abort is
* silent: the loop is wrapped in {@code 2>/dev/null}.
*
* <p>Exercises the real artefact: {@link EnvAllowListScrub#scrubScript} is run verbatim under a
* real {@code /bin/zsh}, not just asserted on as a Java string. The four planted names are
* exported one at a time via {@code typeset -x}/{@code typeset -rx} immediately before the
* script runs, in a fixed order — zsh's {@code export}/{@code typeset -x} appends to the
* process's environment table in call order (verified empirically: a freshly-exported name
* always sorts after every inherited one and after every earlier freshly-exported name in
* {@code command env}'s own output), which is what makes the "middle" position deterministic
* here, unlike relying on the OS's own inherited-environment order.
* <p>The assertion is deliberately "a report exists" rather than "the canary is blanked". The
* report is written by the last statement in the file, so its presence proves the script ran
* to completion; the canary alone would depend on where it happens to sit in {@code env} order.
* Both are checked, but only the first one fails deterministically without the fix.
*/
@Test
void unblankableNameInTheMiddleDoesNotAbortNamesAfterIt(@TempDir Path tmp) throws Exception {
void scrubSurvivesAnInheritedUidTheWayARealPaneHasIt(@TempDir Path tmp) throws Exception {
assumeTrue(Files.isExecutable(ZSH), "/bin/zsh not present — nothing to prove here");
Set<String> allowed = MemberEnvAllowList.derive(List.of());
Path zdotdir = EnvAllowListScrub.generate(tmp, allowed);
// Only ZDOTDIR is allowed — it must survive the scrub itself, since the report is written
// to "$ZDOTDIR/..." AFTER the blanking loop runs; if ZDOTDIR were blanked as a side effect,
// the report write would silently go to the wrong place instead of testing anything.
String script = EnvAllowListScrub.scrubScript(Set.of("ZDOTDIR"));
String setup = """
typeset -x FLEETD_TEST_BEFORE=1
typeset -rx FLEETD_TEST_UNBLANKABLE=1
typeset -x FLEETD_TEST_AFTER_A=1
typeset -x FLEETD_TEST_AFTER_B=1
""";
// The production shape: UID present and exported, as every pane shell inherits it.
Map<String, String> paneLikeParent = Map.of(
"HOME", System.getProperty("user.home"),
"PATH", "/usr/bin:/bin",
"SHELL", "/bin/zsh",
"UID", "1000",
"CB633_CANARY", "must-not-survive-the-scrub");
Set<String> survivors = exportedNamesFromCleanParent(paneLikeParent, zdotdir);
ProcessBuilder pb = new ProcessBuilder("/bin/zsh");
pb.environment().clear();
pb.environment().put("PATH", "/usr/bin:/bin");
pb.environment().put("ZDOTDIR", tmp.toAbsolutePath().toString());
pb.redirectError(ProcessBuilder.Redirect.DISCARD);
Process zsh = pb.start();
zsh.getOutputStream().write((setup + script).getBytes(StandardCharsets.UTF_8));
zsh.getOutputStream().flush();
zsh.getOutputStream().close();
assertTrue(zsh.waitFor(60, java.util.concurrent.TimeUnit.SECONDS),
"the scrub script did not exit within 60s");
assertEquals(0, zsh.exitValue(),
"the scrub script itself must never abort — an unblankable name must not kill the "
+ "sourced file");
EnvAllowListScrub.ScrubReport report = EnvAllowListScrub.readReport(tmp);
assertNotNull(report, "the report must still be written even though one name could not be "
+ "blanked — a report that silently never appears is the #394 bug");
assertTrue(report.blanked().contains("FLEETD_TEST_BEFORE"),
"sanity: the name before the unblankable one must be blanked");
assertTrue(report.blanked().contains("FLEETD_TEST_AFTER_A"),
"the FIRST name AFTER the unblankable one must still be blanked — before the fix, "
+ "the whole loop aborted at the unblankable name and every later name was "
+ "silently left untouched");
assertTrue(report.blanked().contains("FLEETD_TEST_AFTER_B"),
"the SECOND name after the unblankable one must also still be blanked");
assertEquals(1, report.failed(),
"exactly one attempted name could not be blanked, and that count must be visible "
+ "without reading the member's environment");
assertEquals(List.of("FLEETD_TEST_UNBLANKABLE"), report.unblankable(),
"the unblankable name is reported by name, not silently dropped");
}
/**
* fleetd #394 follow-up: the blanking loop's {@code eval "export ${n}="} splices {@code n} into
* a string that zsh then interprets as shell syntax. That is only safe because every name
* reaching {@code _cb633_blank} already passed an identifier check in the ENUMERATION loop
* (20 lines away, in a different loop) — so the fix re-asserts the identical check immediately
* before the {@code eval} call, rather than trusting that distant guard to keep holding.
*
* <p>This test plants a value with an embedded newline, exploiting the exact "junk from
* multi-line values" gap the enumeration loop's own comment already documents: {@code command
* env}'s text output is read line-by-line, so a value's second line becomes a spurious extra
* "name" that was never a real exported variable. The fragment used here ({@code
* junk.fragment}) is merely non-conforming (it contains a dot) — never command-shaped; this
* test must never demonstrate command execution and plants no command-shaped payload.
*
* <p>Exercises the real artefact end-to-end: {@link EnvAllowListScrub#scrubScript} runs
* verbatim under a real {@code /bin/zsh}, exactly as {@code generate()} would produce it — this
* is not a synthetic call into just the blanking loop.
*/
@Test
void nonIdentifierJunkFromAMultilineValueIsSkippedNotBlankedOrUnblankable(@TempDir Path tmp)
throws Exception {
assumeTrue(Files.isExecutable(ZSH), "/bin/zsh not present — nothing to prove here");
String script = EnvAllowListScrub.scrubScript(Set.of("ZDOTDIR"));
ProcessBuilder pb = new ProcessBuilder("/bin/zsh");
pb.environment().clear();
pb.environment().put("PATH", "/usr/bin:/bin");
pb.environment().put("ZDOTDIR", tmp.toAbsolutePath().toString());
// Embedded newline: `command env`'s own text output splits this into two lines, and the
// second ("junk.fragment") has no "=" at all, so `cut -d= -f1` returns it unchanged as a
// spurious candidate "name" — it was never an actual exported variable by that name.
pb.environment().put("FLEETD_TEST_MULTILINE", "keep\njunk.fragment");
pb.redirectError(ProcessBuilder.Redirect.DISCARD);
Process zsh = pb.start();
zsh.getOutputStream().write(script.getBytes(StandardCharsets.UTF_8));
zsh.getOutputStream().flush();
zsh.getOutputStream().close();
assertTrue(zsh.waitFor(60, java.util.concurrent.TimeUnit.SECONDS),
"the scrub script did not exit within 60s");
assertEquals(0, zsh.exitValue(), "the scrub script must reach its end");
EnvAllowListScrub.ScrubReport report = EnvAllowListScrub.readReport(tmp);
assertNotNull(report, "the report must still be written");
assertTrue(report.blanked().contains("FLEETD_TEST_MULTILINE"),
"sanity: the real, identifier-shaped variable must still be blanked normally");
assertFalse(report.blanked().contains("junk.fragment"),
"a non-identifier fragment is not a real variable and must never be blanked");
assertFalse(report.unblankable().contains("junk.fragment"),
"a non-identifier fragment must never even become a candidate the blanking loop "
+ "attempts — it must be filtered before either guard has to catch it, so "
+ "it is neither blanked nor counted as a failed attempt");
EnvAllowListScrub.ScrubReport report = EnvAllowListScrub.readReport(zdotdir);
assertNotNull(report,
"an inherited UID must not abort the scrub — no report means the file died mid-loop "
+ "and every name after UID in `env` order was left unscrubbed");
assertFalse(survivors.contains("CB633_CANARY"),
"a non-allow-listed name must still be blanked when UID is in the environment");
}
/** A group-shared ZDOTDIR still lets the member truncate and write its pre-created receipt. */