fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

The idle-sleep guard is a no-op on Linux, so fleet01 still sleeps under a live member #375

New Issue
Closed
opened 2026-09-08 11:57:05 +02:00 by ltms · 1 comment
ltms commented 2026-09-08 11:57:05 +02:00
Owner
Copy Link
Copy Source

Follow-up to #355 / #374, merged as 127e683.

What ships today

IdleSleepGuard holds an OS-level assertion against idle sleep while at least one member is live. The wiring is platform-neutral: it hangs off SessionManager's onAcquire/onRelease hooks and only a real 0→1 or 1→0 crossing touches the OS. The mechanism behind it is not neutral. CaffeinateSleepAssertionMechanism.isSupportedPlatform() looks for mac in os.name, and returns null from acquire() on anything else.

So on fleet01 the guard is enabled, logs one INFO line, and holds nothing. There is no bug in that code — it degrades exactly as designed. The gap is that no Linux mechanism exists yet.

Why this matters

The motivation for #355 was measured on the Mac: 13 AMQP drops in one night, every one with a sleep or wake event in pmset -g log in the same minute or the one before. That was the visible symptom of the real problem — a member freezing mid-turn with its host.

fleet01 has the same exposure and does not get the fix. Whether it actually sleeps is not yet measured, and that is the first task here, not an assumption. A headless box on mains power may never idle-sleep; it may also be configured to suspend. Measure before building.

Suggested shape

  1. Measure first. On fleet01, check what the host is actually configured to do — systemctl status sleep.target suspend.target, whether sleep.target is masked, and whether anything in journalctl shows a real suspend/resume. If the host never suspends, close this as not needed and say so, with the command output. That is a good outcome, not a wasted ticket.
  2. If it does suspend, add a SystemdInhibitSleepAssertionMechanism behind the existing SleepAssertionMechanism seam — systemd-inhibit --what=idle:sleep --who=fleetd --why=... --mode=block sleep infinity, held as a child process and destroyed on release, exactly the shape CaffeinateSleepAssertionMechanism already uses.
  3. Pick the mechanism by platform at construction, and keep the no-op fallback for every platform neither one covers. A host with no mechanism must stay a clean no-op that logs once — never a throw, never a blocked spawn.

Two things to get right

  • Do not test against real sleep. The existing tests never call the real acquire(), because that would assert against idle sleep on whatever machine runs the suite. Keep the platform predicate pure and testable on both platforms, the way CaffeinateSleepAssertionMechanismTest does. A test that behaves differently on the CI runner than on a laptop is worse than no test.
  • The failure is silent by design. "The guard is enabled" is not proof the host is awake — a mechanism that cannot start holds nothing and says so once. So any claim that this works on Linux needs a live check: a held inhibitor (systemd-inhibit --list) while a member runs, not a green build.

Out of scope

caffeinate -i and systemd-inhibit --what=idle both cover idle sleep only. A closed lid, or an operator asking the host to sleep, still sleeps it. That is deliberate — the guard stops an unattended host sleeping under a member's turn, it never overrides the operator — and it is not what this ticket is for.

Follow-up to #355 / #374, merged as `127e683`. ## What ships today `IdleSleepGuard` holds an OS-level assertion against idle sleep while at least one member is live. The wiring is platform-neutral: it hangs off `SessionManager`'s `onAcquire`/`onRelease` hooks and only a real 0→1 or 1→0 crossing touches the OS. **The mechanism behind it is not neutral.** `CaffeinateSleepAssertionMechanism.isSupportedPlatform()` looks for `mac` in `os.name`, and returns `null` from `acquire()` on anything else. So on fleet01 the guard is enabled, logs one INFO line, and holds nothing. There is no bug in that code — it degrades exactly as designed. The gap is that no Linux mechanism exists yet. ## Why this matters The motivation for #355 was measured on the Mac: 13 AMQP drops in one night, every one with a sleep or wake event in `pmset -g log` in the same minute or the one before. That was the visible symptom of the real problem — a member freezing mid-turn with its host. **fleet01 has the same exposure and does not get the fix.** Whether it actually sleeps is not yet measured, and that is the first task here, not an assumption. A headless box on mains power may never idle-sleep; it may also be configured to suspend. Measure before building. ## Suggested shape 1. **Measure first.** On fleet01, check what the host is actually configured to do — `systemctl status sleep.target suspend.target`, whether `sleep.target` is masked, and whether anything in `journalctl` shows a real suspend/resume. If the host never suspends, close this as not needed and say so, with the command output. That is a good outcome, not a wasted ticket. 2. **If it does suspend**, add a `SystemdInhibitSleepAssertionMechanism` behind the existing `SleepAssertionMechanism` seam — `systemd-inhibit --what=idle:sleep --who=fleetd --why=... --mode=block sleep infinity`, held as a child process and destroyed on release, exactly the shape `CaffeinateSleepAssertionMechanism` already uses. 3. **Pick the mechanism by platform** at construction, and keep the no-op fallback for every platform neither one covers. A host with no mechanism must stay a clean no-op that logs once — never a throw, never a blocked spawn. ## Two things to get right - **Do not test against real sleep.** The existing tests never call the real `acquire()`, because that would assert against idle sleep on whatever machine runs the suite. Keep the platform predicate pure and testable on both platforms, the way `CaffeinateSleepAssertionMechanismTest` does. A test that behaves differently on the CI runner than on a laptop is worse than no test. - **The failure is silent by design.** "The guard is enabled" is not proof the host is awake — a mechanism that cannot start holds nothing and says so once. So any claim that this works on Linux needs a live check: a held inhibitor (`systemd-inhibit --list`) while a member runs, not a green build. ## Out of scope `caffeinate -i` and `systemd-inhibit --what=idle` both cover *idle* sleep only. A closed lid, or an operator asking the host to sleep, still sleeps it. That is deliberate — the guard stops an unattended host sleeping under a member's turn, it never overrides the operator — and it is not what this ticket is for.
ltms commented 2026-09-09 02:15:25 +02:00
Author
Owner
Copy Link
Copy Source

Measured on fleet01, 2026-09-09 00:14 UTC. This ticket said to measure first and close it as not needed if the host never suspends. It never has.

uptime                       15 days, 7:04
boots in the journal         1  (2026-08-24 17:09:44 UTC -> now)
suspend/resume events        0  (grep over the WHOLE journal for
                                 "Reached target Sleep", "Suspending system",
                                 "PM: suspend entry")
sleep.target                 inactive (dead)
suspend.target               inactive (dead)
hibernate.target             inactive (dead)
/etc/systemd/logind.conf     no setting uncommented, only the [Login] header

The zero is a real zero. A count of 0 can mean "it did not happen" or "I could not look", and those are not the same answer — the same trap as #377. So I checked that I can actually read the system journal before trusting it:

journalctl -k    -> kernel lines from Aug 24 17:09:44, readable
systemd[1] lines -> 11966
id -nG           -> ltms adm cdrom sudo dip plugdev lxd docker   (adm = journal read)

So the host has been up for 15 days without one sleep event, and nothing in logind.conf asks it to sleep.

What I did not establish

Two limits, so nobody reads this as stronger than it is.

  1. sleep.target and suspend.target are static, which means not masked. Nothing stops a systemctl suspend run by hand, or a future config change. "It has not suspended" is not "it cannot".
  2. I could not read IdleAction: systemctl show systemd-logind -p IdleAction returns nothing on this systemd version, so I have no measured value for it. I am not quoting a default I did not observe. The evidence for "no idle suspend is configured" is the empty logind.conf and 15 days of silence, not that property.

Closing

Closing as not needed, on this ticket's own gate. The idle-sleep guard from #355/#374 is macOS-only, and on this host it would protect against something that does not happen.

What would reopen this: any suspend or resume line appearing in fleet01's journal. The command is the one above:

journalctl --no-pager | grep -icE "Reached target Sleep|Suspending system|PM: suspend entry"

Non-zero means the premise for closing is gone and Linux needs a real SystemdInhibitSleepAssertionMechanism behind the existing SleepAssertionMechanism seam. Zero means leave this closed. If that command ever errors instead of printing a number, that is the third answer — it says nothing, and you check journal access before concluding anything.

Measured on fleet01, 2026-09-09 00:14 UTC. This ticket said to measure first and close it as not needed if the host never suspends. It never has. ``` uptime 15 days, 7:04 boots in the journal 1 (2026-08-24 17:09:44 UTC -> now) suspend/resume events 0 (grep over the WHOLE journal for "Reached target Sleep", "Suspending system", "PM: suspend entry") sleep.target inactive (dead) suspend.target inactive (dead) hibernate.target inactive (dead) /etc/systemd/logind.conf no setting uncommented, only the [Login] header ``` **The zero is a real zero.** A count of 0 can mean "it did not happen" or "I could not look", and those are not the same answer — the same trap as #377. So I checked that I can actually read the system journal before trusting it: ``` journalctl -k -> kernel lines from Aug 24 17:09:44, readable systemd[1] lines -> 11966 id -nG -> ltms adm cdrom sudo dip plugdev lxd docker (adm = journal read) ``` So the host has been up for 15 days without one sleep event, and nothing in `logind.conf` asks it to sleep. ## What I did not establish Two limits, so nobody reads this as stronger than it is. 1. `sleep.target` and `suspend.target` are `static`, which means **not masked**. Nothing stops a `systemctl suspend` run by hand, or a future config change. "It has not suspended" is not "it cannot". 2. I could not read `IdleAction`: `systemctl show systemd-logind -p IdleAction` returns nothing on this systemd version, so I have no measured value for it. I am not quoting a default I did not observe. The evidence for "no idle suspend is configured" is the empty `logind.conf` and 15 days of silence, not that property. ## Closing Closing as not needed, on this ticket's own gate. The idle-sleep guard from #355/#374 is macOS-only, and on this host it would protect against something that does not happen. **What would reopen this:** any suspend or resume line appearing in fleet01's journal. The command is the one above: ```bash journalctl --no-pager | grep -icE "Reached target Sleep|Suspending system|PM: suspend entry" ``` Non-zero means the premise for closing is gone and Linux needs a real `SystemdInhibitSleepAssertionMechanism` behind the existing `SleepAssertionMechanism` seam. Zero means leave this closed. If that command ever errors instead of printing a number, that is the third answer — it says nothing, and you check journal access before concluding anything.
ltms closed this issue 2026-09-09 02:15:30 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#375
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?