fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fleet_list's free and the spawn gate now disagree by one on subscription profiles #257

New Issue
Closed
opened 2026-09-03 08:26:14 +02:00 by ltms · 1 comment
ltms commented 2026-09-03 08:26:14 +02:00
Owner
Copy Link
Copy Source

Found while merging #176 stage 2 (0e8bfb7). Not a regression in that change — it is a question that change makes visible and that nobody has answered.

The two numbers

fleet_list computes, in FleetMcp:

row.put("free", cap == null ? null : Math.max(0, cap - live - leadSeatCount));

The spawn gate, CompositePeerLauncher, does not consult leadSeatCount at all. It uses maxLoad and the live count. LeadSeatSource is wired into the fleet_list row and nowhere else.

So on this host, with sonnet.maxLoad: 3, both opus and sonnet on the subscription, and 2 members live:

value
fleet_list reports free: 0
fleet_spawn{profile: "sonnet"} succeeds

Measured, not reasoned: the lead-seat lookup returns 1 for sonnet after #176 stage 2, and the placement path never reads it.

Why this matters

#176 was filed because free overstated capacity — it advertised a slot the account could not really carry. free now understates what the daemon will actually do. A caller that trusts free and stops at 0 leaves a slot unused; a caller that ignores free and spawns gets a session. Neither is wrong given the code, which is the problem.

The lead is the main consumer of this number. If it believes free: 0, the fleet quietly runs one member short of what the operator configured.

The unanswered question

What should free mean?

  1. Slots the spawn gate will grant. Then free must not subtract the lead seat, and #176's subtraction belongs on a separate field (leadSeats already exists in the row and carries the fact without changing free).
  2. Sessions this account can carry. Then the spawn gate should subtract the seat too, and maxLoad: 3 starts meaning "3 sessions including the lead" — which changes what every existing config file means, so it needs the treatment a semantics change gets.

Both are defensible. Having them differ silently is not.

Local evidence that argues against option 2

The live fleetd.yaml records a test from 2026-08-29: three concurrent interactive claude sessions all started fine, and claude -p with the full member flag set returned rc=0. No backend seat limit was ever measured on this host. The config comment ends: "Left at 3 because 2 was never shown to be the ceiling."

If there is no real ceiling at 2, then subtracting the lead's seat from free describes an accounting policy, not a constraint — and dressing a policy as a capacity fact is what makes the two numbers diverge.

That evidence is from one host and one backend, so it is not decisive. It is the only measurement anyone has taken.

Suggested acceptance criteria

  1. One documented meaning for free, stated where an operator reads it (fleetd.example.yaml) and where a caller reads it (the tool's own description).
  2. fleet_list's free and the spawn gate agree, or the row carries both numbers under distinct names with the difference explained.
  3. A test that fails when the two paths disagree — driving the real placement gate, not a copy of its arithmetic. A test that reimplements the subtraction proves the copy.
  4. If option 2 is chosen: maxLoad's meaning changes, so every shipped config comment describing it has to be corrected in the same change.

Milestone

2.0. Nothing is broken on one host today — the worst outcome is one unused member slot. It matters because free is the number the lead uses to decide how wide to fan out.

Related: #176 (the change that surfaced this), #113 (a checker narrower than it looks — same family of "two things that should agree, and nothing says they don't").

Found while merging #176 stage 2 (`0e8bfb7`). Not a regression in that change — it is a question that change makes visible and that nobody has answered. ## The two numbers `fleet_list` computes, in `FleetMcp`: ```java row.put("free", cap == null ? null : Math.max(0, cap - live - leadSeatCount)); ``` The spawn gate, `CompositePeerLauncher`, does not consult `leadSeatCount` at all. It uses `maxLoad` and the live count. `LeadSeatSource` is wired into the `fleet_list` row and nowhere else. So on this host, with `sonnet.maxLoad: 3`, both `opus` and `sonnet` on the subscription, and 2 members live: | | value | |---|---| | `fleet_list` reports | `free: 0` | | `fleet_spawn{profile: "sonnet"}` | **succeeds** | Measured, not reasoned: the lead-seat lookup returns 1 for `sonnet` after #176 stage 2, and the placement path never reads it. ## Why this matters #176 was filed because `free` **overstated** capacity — it advertised a slot the account could not really carry. `free` now **understates** what the daemon will actually do. A caller that trusts `free` and stops at 0 leaves a slot unused; a caller that ignores `free` and spawns gets a session. Neither is wrong given the code, which is the problem. The lead is the main consumer of this number. If it believes `free: 0`, the fleet quietly runs one member short of what the operator configured. ## The unanswered question **What should `free` mean?** 1. *Slots the spawn gate will grant.* Then `free` must not subtract the lead seat, and #176's subtraction belongs on a separate field (`leadSeats` already exists in the row and carries the fact without changing `free`). 2. *Sessions this account can carry.* Then the spawn gate should subtract the seat too, and `maxLoad: 3` starts meaning "3 sessions including the lead" — which changes what every existing config file means, so it needs the treatment a semantics change gets. Both are defensible. Having them differ silently is not. ## Local evidence that argues against option 2 The live `fleetd.yaml` records a test from 2026-08-29: three concurrent interactive `claude` sessions all started fine, and `claude -p` with the full member flag set returned rc=0. No backend seat limit was ever measured on this host. The config comment ends: *"Left at 3 because 2 was never shown to be the ceiling."* If there is no real ceiling at 2, then subtracting the lead's seat from `free` describes an accounting policy, not a constraint — and dressing a policy as a capacity fact is what makes the two numbers diverge. That evidence is from one host and one backend, so it is not decisive. It is the only measurement anyone has taken. ## Suggested acceptance criteria 1. One documented meaning for `free`, stated where an operator reads it (`fleetd.example.yaml`) and where a caller reads it (the tool's own description). 2. `fleet_list`'s `free` and the spawn gate agree, or the row carries both numbers under distinct names with the difference explained. 3. A test that fails when the two paths disagree — driving the real placement gate, not a copy of its arithmetic. A test that reimplements the subtraction proves the copy. 4. If option 2 is chosen: `maxLoad`'s meaning changes, so every shipped config comment describing it has to be corrected in the same change. ## Milestone **2.0.** Nothing is broken on one host today — the worst outcome is one unused member slot. It matters because `free` is the number the lead uses to decide how wide to fan out. Related: #176 (the change that surfaced this), #113 (a checker narrower than it looks — same family of "two things that should agree, and nothing says they don't").
ltms added this to the 2.0 — one operation centre, many hosts milestone 2026-09-03 08:26:14 +02:00
ltms commented 2026-09-03 11:35:31 +02:00
Author
Owner
Copy Link
Copy Source

Fixed. PR #263 merged to main as 96d8191. Build: 1262 tests, 0 failures.

The answer to "what should free mean?"

Option 1: the slots the spawn gate will actually grant. free is now max(0, maxLoad - live), which is exactly the check CompositePeerLauncher#enforceMaxLoad runs. leadSeats stays in the row and carries the lead's seat as a fact, but never changes free.

I decided this rather than waiting, because the ticket's own evidence points one way and option 2 is the expensive, breaking choice:

  • Option 2 makes maxLoad: 3 mean "3 sessions including the lead". That silently changes what every existing config file means, on every host.
  • No backend seat ceiling was ever measured. Which brings me to the thing I nearly got wrong.

I nearly took the wrong option, and the reason is worth recording

I was carrying a note saying real sonnet capacity is 2, not 3, because the lead holds a subscription seat. If that were true, option 1 would make the daemon promise a slot the account cannot carry — the opposite of a fix.

So I went and read the live config instead of trusting the note. It records the test:

"2026-08-28 this said 'only 2 are usable, the lead holds a subscription seat'. That theory was TESTED on 2026-08-29 and NOT reproduced: three concurrent interactive claude sessions all started fine, and claude -p with the full member flag set returned rc=0. So there is no measured seat limit here."

The failed spawns that produced the "only 2" theory had a different cause: a long-running fleetd. A restart fixed them. So the seat subtraction described an accounting policy dressed as a capacity fact — which is precisely what this ticket said made the two numbers diverge.

My note was stale and has been corrected.

Acceptance criteria

  1. One documented meaning, where an operator reads it and where a caller reads it. Done — fleetd.example.yaml (maxLoad GOTCHA 3, and the lead profile: doc) and the fleet_list tool description. Also the LeadSeatSource and capacityView javadoc.
  2. The two paths agree. Done, and leadSeats still carries the fact under its own name.
  3. A test that fails when they disagree, driving the real gate. Done, and this is the part worth reading.
  4. Option 2 not chosen, so maxLoad's meaning is unchanged and criterion 4 does not apply.

On criterion 3

freeMatchesWhatTheRealPlacementGateActuallyGrants builds a real CompositePeerLauncher over a real ClaudeCodeLauncher, reads whatever free the real listFleet reports, drives that many spawns through the real FleetMcp.spawn → SessionManager.acquire → CompositePeerLauncher.spawn → enforceMaxLoad path asserting each succeeds, then asserts the next is refused.

It never recomputes the formula. That was the whole risk: a test that hand-computes cap - live passes even when both sides share the same wrong formula.

It also fails in both directions. Overstate free and a spawn inside the loop is refused; understate it and the overflow spawn succeeds. I checked that separately rather than assuming it.

Putting the subtraction back turns it red, with a message that names the exact bug:

fleet_list reported free:0 but the real placement gate granted at least one more spawn
than that: {"sessionId":"term_new_3",...,"profile":"sonnet","status":"spawning"}

One thing I fixed that no worker could

The live fleetd.yaml carried a comment written this morning describing the subtraction as intentional — "the two numbers therefore disagree by 1 on purpose". That is now false. It is gitignored, so I corrected it directly. The "do NOT raise maxLoad to 4" warning is kept, for a new reason: nothing takes a slot any more, so raising it would grant a real fourth member.

Closing.

Fixed. PR #263 merged to `main` as `96d8191`. Build: 1262 tests, 0 failures. ## The answer to "what should `free` mean?" **Option 1: the slots the spawn gate will actually grant.** `free` is now `max(0, maxLoad - live)`, which is exactly the check `CompositePeerLauncher#enforceMaxLoad` runs. `leadSeats` stays in the row and carries the lead's seat as a fact, but never changes `free`. I decided this rather than waiting, because the ticket's own evidence points one way and option 2 is the expensive, breaking choice: - Option 2 makes `maxLoad: 3` mean "3 sessions including the lead". That silently changes what every existing config file means, on every host. - No backend seat ceiling was ever measured. Which brings me to the thing I nearly got wrong. ## I nearly took the wrong option, and the reason is worth recording I was carrying a note saying real sonnet capacity is 2, not 3, because the lead holds a subscription seat. If that were true, option 1 would make the daemon promise a slot the account cannot carry — the opposite of a fix. So I went and read the live config instead of trusting the note. It records the test: > "2026-08-28 this said 'only 2 are usable, the lead holds a subscription seat'. That theory was TESTED on 2026-08-29 and NOT reproduced: three concurrent interactive claude sessions all started fine, and `claude -p` with the full member flag set returned rc=0. So there is no measured seat limit here." The failed spawns that produced the "only 2" theory had a different cause: a long-running `fleetd`. A restart fixed them. So the seat subtraction described an accounting policy dressed as a capacity fact — which is precisely what this ticket said made the two numbers diverge. My note was stale and has been corrected. ## Acceptance criteria 1. **One documented meaning, where an operator reads it and where a caller reads it.** Done — `fleetd.example.yaml` (`maxLoad` GOTCHA 3, and the lead `profile:` doc) and the `fleet_list` tool description. Also the `LeadSeatSource` and `capacityView` javadoc. 2. **The two paths agree.** Done, and `leadSeats` still carries the fact under its own name. 3. **A test that fails when they disagree, driving the real gate.** Done, and this is the part worth reading. 4. Option 2 not chosen, so `maxLoad`'s meaning is unchanged and criterion 4 does not apply. ## On criterion 3 `freeMatchesWhatTheRealPlacementGateActuallyGrants` builds a real `CompositePeerLauncher` over a real `ClaudeCodeLauncher`, reads whatever `free` the real `listFleet` reports, drives that many spawns through the real `FleetMcp.spawn → SessionManager.acquire → CompositePeerLauncher.spawn → enforceMaxLoad` path asserting each succeeds, then asserts the next is refused. It never recomputes the formula. That was the whole risk: a test that hand-computes `cap - live` passes even when both sides share the same wrong formula. It also fails in **both** directions. Overstate `free` and a spawn inside the loop is refused; understate it and the overflow spawn succeeds. I checked that separately rather than assuming it. Putting the subtraction back turns it red, with a message that names the exact bug: ``` fleet_list reported free:0 but the real placement gate granted at least one more spawn than that: {"sessionId":"term_new_3",...,"profile":"sonnet","status":"spawning"} ``` ## One thing I fixed that no worker could The live `fleetd.yaml` carried a comment written this morning describing the subtraction as intentional — "the two numbers therefore disagree by 1 on purpose". That is now false. It is gitignored, so I corrected it directly. The "do NOT raise `maxLoad` to 4" warning is kept, for a new reason: nothing takes a slot any more, so raising it would grant a real fourth member. Closing.
ltms closed this issue 2026-09-03 11:35:37 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone 2.0 — one operation centre, many hosts
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#257
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?