fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fleetd must not let the host idle-sleep while members are live #355

Closed
agent wants to merge 1 commits from worker/sleepguard-82076d-1 into main
pull from: worker/sleepguard-82076d-1
merge into: fleet:main
Conversation 1 Commits 1 Files Changed 14
+685 -7
agent commented 2026-09-05 00:38:26 +02:00
Member
Copy Link
Copy Source

What

New ticket, no forge issue yet. fleetd's host was measured idle-sleeping after as little as 1
minute of inactivity (pmset -g custom → sleep 1 on battery). Overnight the daemon's AMQP link
dropped 13 times; cross-checking every drop minute against pmset -g log found a sleep/wake event
in the same minute or the one before, all 13 of 13. A long member turn with nobody typing is
exactly the case that goes idle.

This adds a small IdleSleepGuard (new package dev.ltms.fleet.power) that holds an OS-level
assertion against idle sleep for exactly as long as at least one fleet member is live, and a new
idleSleepGuard: config block (on by default) to turn it off.

Which class this hangs off, and why

SessionManager (fleetd/src/main/java/dev/ltms/fleet/session/SessionManager.java) is the single
source fleet_list's live/capacity numbers are themselves computed from (Fleetd.liveSessionCount
walks its roster(); FleetMcp.capacityView reads liveCount/maxLoad off the same registry).
Rather than tracking members a second way, IdleSleepGuard hangs off SessionManager's existing
onAcquire/onRelease listener hooks (added for CB-520/CB-516, previously wired only to the
reply inbox) and SessionManager#size() — the exact registry those hooks fire against. Every
acquire/release event re-reads size(); only an actual 0→1 or 1→0 crossing touches the OS.

Wired in Fleetd.java right after sessions is constructed:

sessions.onAcquire(_ -> idleSleepGuard.recheck());
sessions.onRelease(_ -> idleSleepGuard.recheck());

The mechanism, and why

CaffeinateSleepAssertionMechanism spawns caffeinate -i (idle-sleep assertion only — never
-s/-d, which would also block system/display sleep on operator request, violating invariant 4)
as a child process while armed, and destroys it on release. It is entirely behind a
SleepAssertionMechanism seam so tests never touch a real process or real idle-sleep behavior — a
FakeSleepAssertionMechanism stands in throughout the test suite.

The five invariants

  1. No-op off macOS: CaffeinateSleepAssertionMechanism.isSupportedPlatform() checks os.name
    before ever touching ProcessBuilder; acquire() returns null (never throws) on any other
    platform.
  2. No-op when the mechanism is missing: acquire() catches IOException/RuntimeException
    from ProcessBuilder.start() and returns null, logged once at INFO via an AtomicBoolean
    latch — never repeated log spam.
  3. Released, everywhere: IdleSleepGuard.close() is called unconditionally in Fleetd's
    single ordered shutdown hook (a backstop — the session drain above it already drives every
    session's release, which already drives the live count to 0 and releases the assertion on its
    own via the same recheck() path). SleepAssertion.close()/CaffeinateAssertion.close() are
    idempotent, and close() never throws.
  4. Idle sleep only: caffeinate -i is the one flag combination that never touches system/lid
    sleep — see the class doc on CaffeinateSleepAssertionMechanism.
  5. Operator can turn it off, default on: new FleetConfig.IdleSleepGuard(Boolean enabled)
    record, following the Health/ConfigReload nested-record pattern, but inverted default —
    isEnabled() returns true unless explicitly enabled: false. Documented (commented, since
    it's opt-out) in fleetd.example.yaml.

Config-reporting coverage this change had to satisfy

Adding a FleetConfig top-level component is gated by three existing coverage tests, all now
updated:

  • ConfigRefTopLevelCoverageTest — every component must be triaged into COLD_KEYS/
    DEFERRED_KEYS/SPLIT_KEYS/hot-excluded. idleSleepGuard went into ConfigRef.DEFERRED_KEYS
    (Fleetd reads it once, at startup, to decide whether to construct the guard at all — not
    rebuilt on reload) with a new comparison branch in changedDeferredKeys.
  • ConfigRefTopLevelReportingCoverageTest — BASE/ALT value maps updated with real, distinct
    FleetConfig.IdleSleepGuard values so the reflective mutation check proves the new branch is
    actually reported, not just declared in the set.
  • FleetConfigTest.everyNestedConfigKeyIsDocumentedInTheExample — added a commented
    idleSleepGuard: block to fleetd.example.yaml.

Also fixed a real bug this surfaced: FleetConfig.withDefaults() was rebuilding the config through
the second-newest back-compat constructor, silently dropping the new field on every load() call
— caught by my own FleetConfigTest additions before it ever reached a mutation test.

Tests

New package dev.ltms.fleet.power:

  • IdleSleepGuardTest — orchestration against a FakeSleepAssertionMechanism: 0→1 acquires, 1→0
    releases, a steady live count re-touches the mechanism zero times, an unavailable mechanism
    never throws and holds nothing, close() releases independent of live count and is idempotent.
  • CaffeinateSleepAssertionMechanismTest — pure isSupportedPlatform(String) predicate only;
    deliberately never calls the real acquire(), which would actually assert against idle sleep on
    whatever machine runs the suite (forbidden by the brief). Runs the same on macOS or Linux CI.
  • IdleSleepGuardWiringTest — proves the caller, not just the seam: builds a real
    SessionManager (FakeHerdr-backed ClaudeCodeLauncher, same fixture SessionManagerTest
    uses) wired the same two lines Fleetd.main uses, and drives real acquire()/release() calls.
  • FleetConfigTest — 4 new cases for idleSleepGuard: default-on/explicit-on/explicit-off/
    empty-block parsing.

Mutation proof

Copied IdleSleepGuard.java aside (cp to /tmp, never git checkout --), removed the release
call from close():

public void close() {
    synchronized (lock) {
        // release call removed
    }
}

Ran mvn test -Dtest=IdleSleepGuardTest,IdleSleepGuardWiringTest,CaffeinateSleepAssertionMechanismTest:

[ERROR] IdleSleepGuardTest.closeIsIdempotent:116 expected: <1> but was: <0>
[ERROR] IdleSleepGuardTest.closeReleasesAHeldAssertionEvenWithoutAZeroCrossing:103
        close() must release whatever is held, independent of live count ==> expected: <false> but was: <true>
[ERROR] Tests run: 10, Failures: 2, Errors: 0, Skipped: 0

Restored the original file from the /tmp copy (verified byte-identical via diff) before
committing.

Build

cd fleetd && mvn clean install, run unpiped, full output read:

[INFO] Tests run: 1386, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

(main was green at 1372 before this change; +14 new tests here.)

Scope note

Out of scope, not investigated: nothing else touched — the brief's scope (message layer, launchers'
spawn paths, inject/) was left alone.

## What New ticket, no forge issue yet. fleetd's host was measured idle-sleeping after as little as 1 minute of inactivity (`pmset -g custom` → `sleep 1` on battery). Overnight the daemon's AMQP link dropped 13 times; cross-checking every drop minute against `pmset -g log` found a sleep/wake event in the same minute or the one before, all 13 of 13. A long member turn with nobody typing is exactly the case that goes idle. This adds a small `IdleSleepGuard` (new package `dev.ltms.fleet.power`) that holds an OS-level assertion against idle sleep for exactly as long as at least one fleet member is live, and a new `idleSleepGuard:` config block (on by default) to turn it off. ## Which class this hangs off, and why `SessionManager` (`fleetd/src/main/java/dev/ltms/fleet/session/SessionManager.java`) is the single source `fleet_list`'s live/capacity numbers are themselves computed from (`Fleetd.liveSessionCount` walks its `roster()`; `FleetMcp.capacityView` reads `liveCount`/`maxLoad` off the same registry). Rather than tracking members a second way, `IdleSleepGuard` hangs off `SessionManager`'s existing `onAcquire`/`onRelease` listener hooks (added for CB-520/CB-516, previously wired only to the reply inbox) and `SessionManager#size()` — the exact registry those hooks fire against. Every acquire/release event re-reads `size()`; only an actual 0→1 or 1→0 crossing touches the OS. Wired in `Fleetd.java` right after `sessions` is constructed: ```java sessions.onAcquire(_ -> idleSleepGuard.recheck()); sessions.onRelease(_ -> idleSleepGuard.recheck()); ``` ## The mechanism, and why `CaffeinateSleepAssertionMechanism` spawns `caffeinate -i` (idle-sleep assertion only — never `-s`/`-d`, which would also block system/display sleep on operator request, violating invariant 4) as a child process while armed, and destroys it on release. It is entirely behind a `SleepAssertionMechanism` seam so tests never touch a real process or real idle-sleep behavior — a `FakeSleepAssertionMechanism` stands in throughout the test suite. ## The five invariants 1. **No-op off macOS**: `CaffeinateSleepAssertionMechanism.isSupportedPlatform()` checks `os.name` before ever touching `ProcessBuilder`; `acquire()` returns `null` (never throws) on any other platform. 2. **No-op when the mechanism is missing**: `acquire()` catches `IOException`/`RuntimeException` from `ProcessBuilder.start()` and returns `null`, logged once at INFO via an `AtomicBoolean` latch — never repeated log spam. 3. **Released, everywhere**: `IdleSleepGuard.close()` is called unconditionally in `Fleetd`'s single ordered shutdown hook (a backstop — the session drain above it already drives every session's release, which already drives the live count to 0 and releases the assertion on its own via the same `recheck()` path). `SleepAssertion.close()`/`CaffeinateAssertion.close()` are idempotent, and `close()` never throws. 4. **Idle sleep only**: `caffeinate -i` is the one flag combination that never touches system/lid sleep — see the class doc on `CaffeinateSleepAssertionMechanism`. 5. **Operator can turn it off, default on**: new `FleetConfig.IdleSleepGuard(Boolean enabled)` record, following the `Health`/`ConfigReload` nested-record pattern, but inverted default — `isEnabled()` returns `true` unless explicitly `enabled: false`. Documented (commented, since it's opt-out) in `fleetd.example.yaml`. ## Config-reporting coverage this change had to satisfy Adding a `FleetConfig` top-level component is gated by three existing coverage tests, all now updated: - `ConfigRefTopLevelCoverageTest` — every component must be triaged into `COLD_KEYS`/ `DEFERRED_KEYS`/`SPLIT_KEYS`/hot-excluded. `idleSleepGuard` went into `ConfigRef.DEFERRED_KEYS` (Fleetd reads it once, at startup, to decide whether to construct the guard at all — not rebuilt on reload) with a new comparison branch in `changedDeferredKeys`. - `ConfigRefTopLevelReportingCoverageTest` — `BASE`/`ALT` value maps updated with real, distinct `FleetConfig.IdleSleepGuard` values so the reflective mutation check proves the new branch is actually reported, not just declared in the set. - `FleetConfigTest.everyNestedConfigKeyIsDocumentedInTheExample` — added a commented `idleSleepGuard:` block to `fleetd.example.yaml`. Also fixed a real bug this surfaced: `FleetConfig.withDefaults()` was rebuilding the config through the second-newest back-compat constructor, silently dropping the new field on every `load()` call — caught by my own `FleetConfigTest` additions before it ever reached a mutation test. ## Tests New package `dev.ltms.fleet.power`: - `IdleSleepGuardTest` — orchestration against a `FakeSleepAssertionMechanism`: 0→1 acquires, 1→0 releases, a steady live count re-touches the mechanism zero times, an unavailable mechanism never throws and holds nothing, `close()` releases independent of live count and is idempotent. - `CaffeinateSleepAssertionMechanismTest` — pure `isSupportedPlatform(String)` predicate only; deliberately never calls the real `acquire()`, which would actually assert against idle sleep on whatever machine runs the suite (forbidden by the brief). Runs the same on macOS or Linux CI. - `IdleSleepGuardWiringTest` — proves the *caller*, not just the seam: builds a real `SessionManager` (`FakeHerdr`-backed `ClaudeCodeLauncher`, same fixture `SessionManagerTest` uses) wired the same two lines `Fleetd.main` uses, and drives real `acquire()`/`release()` calls. - `FleetConfigTest` — 4 new cases for `idleSleepGuard:` default-on/explicit-on/explicit-off/ empty-block parsing. ## Mutation proof Copied `IdleSleepGuard.java` aside (`cp` to `/tmp`, never `git checkout --`), removed the release call from `close()`: ```java public void close() { synchronized (lock) { // release call removed } } ``` Ran `mvn test -Dtest=IdleSleepGuardTest,IdleSleepGuardWiringTest,CaffeinateSleepAssertionMechanismTest`: ``` [ERROR] IdleSleepGuardTest.closeIsIdempotent:116 expected: <1> but was: <0> [ERROR] IdleSleepGuardTest.closeReleasesAHeldAssertionEvenWithoutAZeroCrossing:103 close() must release whatever is held, independent of live count ==> expected: <false> but was: <true> [ERROR] Tests run: 10, Failures: 2, Errors: 0, Skipped: 0 ``` Restored the original file from the `/tmp` copy (verified byte-identical via `diff`) before committing. ## Build `cd fleetd && mvn clean install`, run unpiped, full output read: ``` [INFO] Tests run: 1386, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS ``` (main was green at 1372 before this change; +14 new tests here.) ## Scope note Out of scope, not investigated: nothing else touched — the brief's scope (message layer, launchers' spawn paths, `inject/`) was left alone.
agent added 1 commit 2026-09-05 00:38:26 +02:00
fleetd must not let the host idle-sleep while members are live
CI / contract (pull_request) Successful in 54s
Details
CI / build (pull_request) Successful in 1m54s
Details
130962e8e3 
Adds a small IdleSleepGuard (dev.ltms.fleet.power) that holds a macOS
caffeinate -i child while at least one fleet member is live, and
releases it once none are. It hangs off SessionManager's existing
onAcquire/onRelease hooks and SessionManager#size() rather than
tracking members a second way. New idleSleepGuard: config block,
on by default, following the FleetConfig.Health/ConfigReload pattern.
ltms commented 2026-09-07 15:38:36 +02:00
Owner
Copy Link
Copy Source

Landed on main as 127e683, by way of #374.

This branch itself was 39 commits behind and no longer merged: three conflicts, all in config-record plumbing that grew new fields on main (memberSkills, #362) while this branch grew its own (idleSleepGuard). So the work here was rebased rather than merged directly — commit 130962e cherry-picked as 24b96d2, plus one needed fix as 6442a58.

The power package came across byte-for-byte: git diff 130962e 24b96d2 -- .../power/ is empty. Nothing in the guard itself changed.

The extra commit was necessary, and worth recording why. FleetConfigWithDefaultsPreservesEveryComponentTest was added to main after this branch was cut, and it exists to catch one specific defect — withDefaults()'s final new FleetConfig(...) call silently binding a narrower back-compat overload, which compiles and drops the new component to null on every load. Its javadoc records that this already happened live on this very branch. Its baseValues() map did not know about idleSleepGuard, so the new component was invisible to the test built to protect it. Adding the entry is what makes the guard real here.

I proved that on merge rather than taking it on trust: dropping the trailing idleSleepGuard argument from withDefaults() still compiles, and now fails 1 test plus 3 errors, with the message naming the dropped component and printing its own denominator — 24 components, 24 checked, 0 excluded, 23 survived.

mvn clean install after the merge: Tests run: 1439, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS.

Written up on the wiki Features page as The host no longer idle-sleeps while a member is working.

Closing as landed.

Landed on `main` as `127e683`, by way of #374. This branch itself was 39 commits behind and no longer merged: three conflicts, all in config-record plumbing that grew new fields on `main` (`memberSkills`, #362) while this branch grew its own (`idleSleepGuard`). So the work here was rebased rather than merged directly — commit `130962e` cherry-picked as `24b96d2`, plus one needed fix as `6442a58`. The `power` package came across byte-for-byte: `git diff 130962e 24b96d2 -- .../power/` is empty. Nothing in the guard itself changed. The extra commit was necessary, and worth recording why. `FleetConfigWithDefaultsPreservesEveryComponentTest` was added to `main` *after* this branch was cut, and it exists to catch one specific defect — `withDefaults()`'s final `new FleetConfig(...)` call silently binding a narrower back-compat overload, which compiles and drops the new component to `null` on every load. Its javadoc records that this already happened live on this very branch. Its `baseValues()` map did not know about `idleSleepGuard`, so the new component was invisible to the test built to protect it. Adding the entry is what makes the guard real here. I proved that on merge rather than taking it on trust: dropping the trailing `idleSleepGuard` argument from `withDefaults()` still compiles, and now fails 1 test plus 3 errors, with the message naming the dropped component and printing its own denominator — `24 components, 24 checked, 0 excluded, 23 survived`. `mvn clean install` after the merge: `Tests run: 1439, Failures: 0, Errors: 0, Skipped: 0`, BUILD SUCCESS. Written up on the wiki Features page as *The host no longer idle-sleeps while a member is working*. Closing as landed.
ltms closed this pull request 2026-09-07 15:38:41 +02:00
Some checks are pending
CI / contract (pull_request) Successful in 54s
Details
CI / build (pull_request) Successful in 1m54s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#355
Delete Branch "worker/sleepguard-82076d-1"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?