fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fleet_list advertises free capacity for a profile the spawn gate refuses — CapacitySource reads the live profile keySet #416

New Issue
Closed
opened 2026-09-10 04:57:47 +02:00 by ltms · 0 comments
ltms commented 2026-09-10 04:57:47 +02:00
Owner
Copy Link
Copy Source

Found by the fleet01 lead. Confirmed in the code on the Mac before filing. This is a live twin of #404, and it is worse than #404.

The defect

profiles is a deferred key — ConfigRef.java:229 lists it in DEFERRED_KEYS. Adding or removing a profile needs a restart, because a new backend needs its own launcher and HerdrPeerLauncher takes Map.copyOf(profiles) at construction.

But Fleetd.java:677 hands CapacitySource the live profile set:

new FleetMcp.CapacitySource(profile -> liveCountRef.get().apply(profile),
    profile -> { var configured = config.get().profiles().get(profile);
                 return configured == null ? null : configured.maxLoad(); },
    () -> config.get().profiles().keySet(), System::nanoTime),

So fleet_list enumerates profiles from a hot-reloaded map, while every path that can actually use a profile reads the startup snapshot.

Evidence — measured on fleet01, not reasoned

A throwaway ghost404 profile added to fleetd.yaml and hot-reloaded, nothing else changed:

reload log     "config reloaded; these changes need a restart to take effect:
                profiles (added/removed: ghost404)"
fleet_profiles ["local","opus","gx","xf"]                            <- frozen map: correct
fleet_list     {"profile":"ghost404","maxLoad":3,"live":0,"free":3}   <- live config: WRONG
fleet_spawn    error: unknown worker profile: ghost404

Three doors, and the loudest one lies.

Why this ranks above #404

fleet_list's own contract says free is "the slots a fresh fleet_spawn on that profile will actually be granted right now — the same check the spawn gate itself runs". It is demonstrably not the same check: it reported 3 free slots for a profile the spawn gate refuses outright.

#404 misreported whether a safety net was armed. This misreports available capacity — the number a lead reads when deciding where to send work. A lead that trusts it routes work to a profile which cannot exist, and the resulting error reads as a bug in the spawn path rather than as stale config.

Scope of the fix — the set, not the fields

Only the profile set is wrong. The per-profile lookups beside it are correct and must stay live: the class javadoc at ConfigRef.java:79-81 says credentialId "is NOT [deferred] ... exactly like weight / maxLoad, so it is hot instead". Changing a live maxLoad is meant to take effect without a restart.

profiles is documented at ConfigRef.java:86-88 as a key read "both ways at different sites, so the key does not fit any class above as a whole". That is the trap: it is legitimately hot for tunables and frozen for identity, so "profiles is deferred" and "this live read is fine" are both true of different halves.

The correct pattern is already written three lines below the defect, for coordinator.peers (Fleetd.java:687-690): read from "the SAME snapshot leadMailbox itself opened from ... not the live config.get() ... so peers follows the same rule rather than half hot-reloading." The rule this ticket needs was stated next door and not applied here.

Acceptance

  1. The profile set given to CapacitySource comes from the startup snapshot, so a hot-added profile never appears in fleet_list.
  2. maxLoad, weight and credentialId stay live — a test must pin that a hot maxLoad edit still changes fleet_list, or the fix trades this bug for the opposite one.
  3. Both directions pinned, the #404 lesson: one test that a profile in the startup set IS listed, and one that a profile added only to the live config is NOT. A one-directional test cannot tell a correct lookup from a permanently empty one — that is exactly how #404's armed lambda survived profile -> false with 1475 tests green.
  4. Add the same shape of check for any other closure that reads config.get().profiles().keySet() — report what you find, do not fix it here.
  5. mvn clean install green, real unpiped Tests run: line.

Credit: fleet01 lead, who found it from the "sitting next to a hot lookup does not make a deferred key hot" generalisation and proved it with a live ghost profile rather than an argument.

Found by the fleet01 lead. Confirmed in the code on the Mac before filing. **This is a live twin of #404, and it is worse than #404.** ## The defect `profiles` is a deferred key — `ConfigRef.java:229` lists it in `DEFERRED_KEYS`. Adding or removing a profile needs a restart, because a new backend needs its own launcher and `HerdrPeerLauncher` takes `Map.copyOf(profiles)` at construction. But `Fleetd.java:677` hands `CapacitySource` the **live** profile set: ```java new FleetMcp.CapacitySource(profile -> liveCountRef.get().apply(profile), profile -> { var configured = config.get().profiles().get(profile); return configured == null ? null : configured.maxLoad(); }, () -> config.get().profiles().keySet(), System::nanoTime), ``` So `fleet_list` enumerates profiles from a hot-reloaded map, while every path that can actually *use* a profile reads the startup snapshot. ## Evidence — measured on fleet01, not reasoned A throwaway `ghost404` profile added to `fleetd.yaml` and hot-reloaded, nothing else changed: ``` reload log "config reloaded; these changes need a restart to take effect: profiles (added/removed: ghost404)" fleet_profiles ["local","opus","gx","xf"] <- frozen map: correct fleet_list {"profile":"ghost404","maxLoad":3,"live":0,"free":3} <- live config: WRONG fleet_spawn error: unknown worker profile: ghost404 ``` Three doors, and the loudest one lies. ## Why this ranks above #404 `fleet_list`'s own contract says `free` is "the slots a fresh `fleet_spawn` on that profile will actually be granted right now — **the same check the spawn gate itself runs**". It is demonstrably not the same check: it reported 3 free slots for a profile the spawn gate refuses outright. #404 misreported whether a safety net was armed. This misreports **available capacity** — the number a lead reads when deciding where to send work. A lead that trusts it routes work to a profile which cannot exist, and the resulting error reads as a bug in the spawn path rather than as stale config. ## Scope of the fix — the set, not the fields Only the profile **set** is wrong. The per-profile lookups beside it are correct and must stay live: the class javadoc at `ConfigRef.java:79-81` says `credentialId` "is NOT [deferred] ... exactly like `weight` / `maxLoad`, so it is hot instead". Changing a live `maxLoad` is meant to take effect without a restart. `profiles` is documented at `ConfigRef.java:86-88` as a key read "**both** ways at different sites, so the key does not fit any class above as a whole". That is the trap: it is legitimately hot for tunables and frozen for identity, so "profiles is deferred" and "this live read is fine" are both true of different halves. **The correct pattern is already written three lines below the defect**, for `coordinator.peers` (`Fleetd.java:687-690`): read from "the SAME snapshot leadMailbox itself opened from ... not the live `config.get()` ... so peers follows the same rule rather than half hot-reloading." The rule this ticket needs was stated next door and not applied here. ## Acceptance 1. The profile **set** given to `CapacitySource` comes from the startup snapshot, so a hot-added profile never appears in `fleet_list`. 2. `maxLoad`, `weight` and `credentialId` stay live — a test must pin that a hot `maxLoad` edit still changes `fleet_list`, or the fix trades this bug for the opposite one. 3. **Both directions pinned**, the #404 lesson: one test that a profile in the startup set IS listed, and one that a profile added only to the live config is NOT. A one-directional test cannot tell a correct lookup from a permanently empty one — that is exactly how #404's `armed` lambda survived `profile -> false` with 1475 tests green. 4. Add the same shape of check for any other closure that reads `config.get().profiles().keySet()` — report what you find, do not fix it here. 5. `mvn clean install` green, real unpiped `Tests run:` line. Credit: fleet01 lead, who found it from the "sitting next to a hot lookup does not make a deferred key hot" generalisation and proved it with a live ghost profile rather than an argument.
ltms closed this issue 2026-09-10 06:27:37 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#416
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?