fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Nothing warns that a profile has no usage-limit detection, so "off when the subscription limit is reached" silently covers only the profiles that opted in #441

New Issue
Closed
opened 2026-09-10 09:25:12 +02:00 by ltms · 1 comment
ltms commented 2026-09-10 09:25:12 +02:00
Owner
Copy Link
Copy Source

This is the missing half of the model gate work (#422 and the models: allow-list). The gate can turn a model off. What is missing is the daemon telling anyone that, for most profiles, nothing will ever turn it off automatically.

Measured on this host, today

fleet_profiles against the live daemon, and the same fact re-derived from fleetd/fleetd.yaml:

profile        exhaustedPattern set?
local          no
local-direct   no
gx             no
opus           no
sonnet         no
sol            YES
terra          YES
xf             no

Live fleet_profiles.exhaustionDetectionArmed agrees exactly: sol and terra true, the
other six false. Two independent reads, same answer.

So 2 of 8 profiles can classify a usage-limit refusal. The other six cannot, ever.

Why this is a defect and not just a config choice

exhaustedPattern is opt-in by design, and that design is right — FleetConfig.java:532
deliberately refuses to default it:

exhaustedPattern stays null when unset/blank (opt-in) — no defaulting, no vendor …

A wrong vendor-guessed regex would quarantine a healthy credential, which is worse than not
detecting. I am not asking for a default.

The defect is that the absence is silent. A profile with no exhaustedPattern:

  • never quarantines on exhaustion, however many times the backend refuses it
  • looks completely healthy in every log line
  • differs from an armed profile only in a field an operator must go and ask for

fleet_profiles reports it per profile (#395), which is why I could measure it — but that is
a pull, and nobody pulls it until something is already wrong. The daemon already knows the
whole picture at boot and says nothing.

The precedent to copy

Fleetd.java:241 already does exactly the right thing for the sibling feature:

log.info("model gate (fleetd #422): {}", modelGateCoverageLine(workers.modelGateState()));

and modelGateCoverageLine (Fleetd.java:857-880) renders three distinct states in words,
including the "nothing is gated, and nothing can be" case. Usage-limit detection has no
equivalent line.

Acceptance criteria

  1. At startup, log one coverage line for usage-limit detection, next to the model gate line.
    It must name the count and the uncovered profiles, e.g.
    usage-limit detection: armed on 2 of 8 profiles; NOT armed: local, local-direct, gx, opus, sonnet, xf.
  2. It must render the three states distinctly, the way modelGateCoverageLine does:
    armed on all, armed on some (name the gaps), armed on none.
  3. Read the fact from the same source the behaviour reads — the per-profile
    exhaustedPatternArmed function that FleetMcp.java:1225 already calls. Do not add a
    second, independently-derived read of cfg.profiles(); a receipt that reads a different
    source than the behaviour is a bug this repo has shipped before, and the javadoc at
    Fleetd.java:860 says so explicitly for the model gate line.
  4. Extract it as a testable pure function, like modelGateCoverageLine, and cover all three
    states. FleetdModelGateCoverageLineTest is the pattern to follow.
  5. The mutation that must fail: make the function report every profile as armed. At least
    one test must go red, and the failing test must be named in the report.

Explicitly out of scope

  • Do not add an exhaustedPattern to any profile in fleetd.yaml. Whether to arm a given
    credential is the operator's call, and a regex guessed by a worker that quarantines a
    healthy paid credential is a worse outcome than the current silence. This ticket makes the
    gap visible; it does not close it.
  • Do not change quarantine, cooldown, or the classification logic.
  • Do not add an automatic probe that retries an exhausted credential to discover whether the
    limit has lifted. That spends the quota it is trying to measure. The "back on" direction
    stays a deliberate act: models: is hot (ConfigRef.java:156-157 moved it from deferred to
    hot-excluded in #422), so turning a model back on already takes effect with no restart.

Note for whoever picks this up

exhaustedPattern itself is not hot — ConfigRef.java:628-634 records that it is
compiled once into Fleetd.main's pattern map and a reload never re-reads it. So the line
this ticket adds is accurate for the life of the process, which is exactly what a startup
line should be. Do not describe it as a live figure.

This is the missing half of the model gate work (#422 and the `models:` allow-list). The gate can turn a model off. What is missing is the daemon telling anyone that, for most profiles, **nothing will ever turn it off automatically.** ## Measured on this host, today `fleet_profiles` against the live daemon, and the same fact re-derived from `fleetd/fleetd.yaml`: ``` profile exhaustedPattern set? local no local-direct no gx no opus no sonnet no sol YES terra YES xf no ``` Live `fleet_profiles.exhaustionDetectionArmed` agrees exactly: `sol` and `terra` true, the other six false. Two independent reads, same answer. So **2 of 8 profiles** can classify a usage-limit refusal. The other six cannot, ever. ## Why this is a defect and not just a config choice `exhaustedPattern` is opt-in by design, and that design is right — `FleetConfig.java:532` deliberately refuses to default it: > `exhaustedPattern` stays null when unset/blank (opt-in) — no defaulting, no vendor … A wrong vendor-guessed regex would quarantine a healthy credential, which is worse than not detecting. I am not asking for a default. The defect is that the *absence* is silent. A profile with no `exhaustedPattern`: - never quarantines on exhaustion, however many times the backend refuses it - looks completely healthy in every log line - differs from an armed profile only in a field an operator must go and ask for `fleet_profiles` reports it per profile (#395), which is why I could measure it — but that is a pull, and nobody pulls it until something is already wrong. The daemon already knows the whole picture at boot and says nothing. ## The precedent to copy `Fleetd.java:241` already does exactly the right thing for the sibling feature: ```java log.info("model gate (fleetd #422): {}", modelGateCoverageLine(workers.modelGateState())); ``` and `modelGateCoverageLine` (`Fleetd.java:857-880`) renders three distinct states in words, including the "nothing is gated, and nothing can be" case. Usage-limit detection has no equivalent line. ## Acceptance criteria 1. At startup, log one coverage line for usage-limit detection, next to the model gate line. It must name the count and the uncovered profiles, e.g. `usage-limit detection: armed on 2 of 8 profiles; NOT armed: local, local-direct, gx, opus, sonnet, xf`. 2. It must render the three states distinctly, the way `modelGateCoverageLine` does: armed on all, armed on some (name the gaps), armed on none. 3. Read the fact from the **same source** the behaviour reads — the per-profile `exhaustedPatternArmed` function that `FleetMcp.java:1225` already calls. Do not add a second, independently-derived read of `cfg.profiles()`; a receipt that reads a different source than the behaviour is a bug this repo has shipped before, and the javadoc at `Fleetd.java:860` says so explicitly for the model gate line. 4. Extract it as a testable pure function, like `modelGateCoverageLine`, and cover all three states. `FleetdModelGateCoverageLineTest` is the pattern to follow. 5. **The mutation that must fail:** make the function report every profile as armed. At least one test must go red, and the failing test must be named in the report. ## Explicitly out of scope - **Do not add an `exhaustedPattern` to any profile in `fleetd.yaml`.** Whether to arm a given credential is the operator's call, and a regex guessed by a worker that quarantines a healthy paid credential is a worse outcome than the current silence. This ticket makes the gap visible; it does not close it. - Do not change quarantine, cooldown, or the classification logic. - Do not add an automatic probe that retries an exhausted credential to discover whether the limit has lifted. That spends the quota it is trying to measure. The "back on" direction stays a deliberate act: `models:` is hot (`ConfigRef.java:156-157` moved it from deferred to hot-excluded in #422), so turning a model back on already takes effect with no restart. ## Note for whoever picks this up `exhaustedPattern` itself is **not** hot — `ConfigRef.java:628-634` records that it is compiled once into `Fleetd.main`'s pattern map and a reload never re-reads it. So the line this ticket adds is accurate for the life of the process, which is exactly what a startup line should be. Do not describe it as a live figure.
ltms commented 2026-09-10 09:28:58 +02:00
Author
Owner
Copy Link
Copy Source

Closing this as a duplicate. The feature already ships, and filing this was my error.

It landed as fleetd #395:

  • Fleetd.java:141 — reportExhaustedPatternGap(cfg);
  • Fleetd.java:1449 — the implementation
  • ExhaustedPatternGapReportTest — its behaviour test, two cases

It already does more than I asked for here: an aggregate WARN naming every profile with no
exhaustedPattern, a second louder WARN when any of them is a subscription: profile, and an
INFO when every profile is armed so the healthy case is on the record too. wiki/11-Features.md
documents it under "Startup says which profiles have usage-limit detection turned off".

How I got it wrong, since the mistake is the reusable part

My measurements were all correct. 2 of 8 profiles armed is right, and exhaustedPattern being
opt-in with no default is right. The conclusion was wrong: "nothing warns about this".

I reached it by grepping for exhaustionDetection|exhaustionArmed — the API field name. That
search found FleetMcp's reporting and nothing else, so I concluded nothing warned at startup.
But the startup warning is not called that. It is reportExhaustedPatternGap, and my pattern
could never have matched it. The search returned a true answer to a question I was not asking.

Then a second filter hid the correction. I grepped wiki/11-Features.md for
exhaust|quarantin and piped it through head -10. The existing feature's heading was further
down the file, so the truncation cut off the one line that would have stopped me.

Both are the same disease, and it is one I have a note about: a search that matches nothing and a
codebase that contains nothing look identical. The rule I already had was "run a control that must
be non-zero in the same pass". This adds a second rule I did not have: a control proves the
search ran, but it does not prove the search addressed the thing you meant. For an "X does not
exist anywhere" claim, grep for the capability in the docs and in the log strings, not only for
the identifier you happen to know.

Cost: one worker turn, stopped a few minutes in.

What survives

One real gap did come out of this, and the Features page named it rather than my grep:

The startup call site is not pinned by a test. Deleting reportExhaustedPatternGap(cfg)
from Fleetd.java leaves the suite green.

I checked, and it holds for all four startup reports. Filed separately as #442. The Features page
also says "fleetd #398 owns closing it", which is now stale — #398 is the closed allow-list PR,
and the validateXxx half it referred to was fixed by collapsing those six calls into one pinned
cfg.validateAll(). I will correct that line on the Features page.

Closing this as a duplicate. **The feature already ships**, and filing this was my error. It landed as fleetd #395: - `Fleetd.java:141` — `reportExhaustedPatternGap(cfg);` - `Fleetd.java:1449` — the implementation - `ExhaustedPatternGapReportTest` — its behaviour test, two cases It already does more than I asked for here: an aggregate `WARN` naming every profile with no `exhaustedPattern`, a second louder `WARN` when any of them is a `subscription:` profile, and an `INFO` when every profile is armed so the healthy case is on the record too. `wiki/11-Features.md` documents it under "Startup says which profiles have usage-limit detection turned off". ## How I got it wrong, since the mistake is the reusable part My measurements were all correct. 2 of 8 profiles armed is right, and `exhaustedPattern` being opt-in with no default is right. The **conclusion** was wrong: "nothing warns about this". I reached it by grepping for `exhaustionDetection|exhaustionArmed` — the *API field* name. That search found `FleetMcp`'s reporting and nothing else, so I concluded nothing warned at startup. But the startup warning is not called that. It is `reportExhaustedPatternGap`, and my pattern could never have matched it. The search returned a true answer to a question I was not asking. Then a second filter hid the correction. I grepped `wiki/11-Features.md` for `exhaust|quarantin` and piped it through `head -10`. The existing feature's heading was further down the file, so the truncation cut off the one line that would have stopped me. Both are the same disease, and it is one I have a note about: a search that matches nothing and a codebase that contains nothing look identical. The rule I already had was "run a control that must be non-zero in the same pass". This adds a second rule I did not have: **a control proves the search ran, but it does not prove the search addressed the thing you meant.** For an "X does not exist anywhere" claim, grep for the *capability* in the docs and in the log strings, not only for the identifier you happen to know. Cost: one worker turn, stopped a few minutes in. ## What survives One real gap did come out of this, and the Features page named it rather than my grep: > **The startup call site is not pinned by a test.** Deleting `reportExhaustedPatternGap(cfg)` > from `Fleetd.java` leaves the suite green. I checked, and it holds for all four startup reports. Filed separately as #442. The Features page also says "fleetd #398 owns closing it", which is now stale — #398 is the closed allow-list PR, and the `validateXxx` half it referred to was fixed by collapsing those six calls into one pinned `cfg.validateAll()`. I will correct that line on the Features page.
ltms closed this issue 2026-09-10 09:29:04 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#441
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?