fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

deploy/fleetd.service silently disables caller identity and the credential scrub #360

New Issue
Closed
opened 2026-09-05 02:50:14 +02:00 by ltms · 1 comment
ltms commented 2026-09-05 02:50:14 +02:00
Owner
Copy Link
Copy Source

deploy/fleetd.service has never been run in anger (CB-504 wrote it ahead of CB-308). I installed
it on fleet01 on 2026-09-05 and it breaks two controls without failing to start.

1. The sandbox directives break caller identity

fleetd resolves a caller's role by running lsof to find the loopback peer PID
(mcp/LsofPeerPidLookup, mcp/ConnectionIdentity). Any directive that gives the unit its own
mount namespace makes lsof return nothing. An unresolved caller is ANONYMOUS (#317), so the
primary is refused every orchestration call:

{"result":{"content":[{"text":"unauthenticated: anonymous may not SPAWN"}],"isError":true}}
DEBUG LsofPeerPidLookup - lsof peer-pid lookup for port 47656 found no matching process

ss found that same port instantly (claude pid=661919), so the socket was there — only lsof
inside the namespace could not see it.

Bisected with systemd-run --user, counting lsof output lines against a live connection:

directive lsof lines
no sandbox 3
ProtectSystem=strict 0
ProtectHome=read-only 0
ProtectKernelTunables=true 0
ProtectControlGroups=true 0
RestrictSUIDSGID=true 3
NoNewPrivileges=true 3

The unit ships the first four. The two safe ones are the two that add no mount namespace.

What makes this bad is that nothing looks wrong: the unit starts, /healthz returns
{"status":"ok"}, and startup secret ...: set reports all three secrets. The only symptom is
that the fleet cannot be driven at all.

2. PrivateTmp=true breaks the member credential scrub

HerdrPeerLauncher creates the member's ZDOTDIR scrub directory under java.io.tmpdir, and
OpenCodeLauncher writes $OPENCODE_CONFIG there too. The member pane is a child of herdr,
a different unit. With PrivateTmp=true on fleetd, the pane cannot see either directory.

This is the same class as the finding above: policy: allow-list would report as configured while
the control it names does nothing.

3. ProtectHome=read-write is not a valid value

systemd accepts yes, no, read-only, tmpfs. systemd-run rejects it outright:

Failed to start transient service unit: Invalid ProtectHome setting: read-write

4. After=herdr.service names a unit the repo does not ship

deploy/ has no herdr.service. herdr also needs a pty with a real window size or every pane
spawn fails with ghostty error -2, so the unit is not a one-liner.

What works (verified live on fleet01)

A user unit with the four namespacing directives removed, PrivateTmp=false, and
ExecStart=/bin/zsh -lc "exec java -jar target/fleetd.jar fleetd.yaml":

  • fleet_whoami → {"role":"primary"}
  • fleet_spawn → worktree provisioned, .mcp.json neutralized, ZDOTDIR allow-list 25 names,
    /tmp/fleetd-zdotdir-3332885004035289899 present, CB-634 auto-open fired, pane injectable
  • all three startup secret ...: set — the login shell reuses ~/.fleet/secrets.sh, so no second
    copy of the secrets in a systemd drop-in

The login shell matters: ExecStart= pointing straight at java starts fine with empty tokens,
and the failure appears hours later as a member that cannot open a pull request.

Notes

  • The Environment=PATH= line also hardcodes /usr/lib/jvm/temurin-25-jdk/bin and
    /usr/share/maven/bin; fleet01 has /usr/bin/java and /usr/bin/mvn. The login shell removes
    the need for that line at all.
  • Lingering (loginctl enable-linger) is required for a user unit to start at boot; it is not
    mentioned in the install comment.
`deploy/fleetd.service` has never been run in anger (CB-504 wrote it ahead of CB-308). I installed it on fleet01 on 2026-09-05 and it breaks two controls without failing to start. ## 1. The sandbox directives break caller identity fleetd resolves a caller's role by running `lsof` to find the loopback peer PID (`mcp/LsofPeerPidLookup`, `mcp/ConnectionIdentity`). Any directive that gives the unit its **own mount namespace** makes `lsof` return nothing. An unresolved caller is `ANONYMOUS` (#317), so the primary is refused every orchestration call: ``` {"result":{"content":[{"text":"unauthenticated: anonymous may not SPAWN"}],"isError":true}} DEBUG LsofPeerPidLookup - lsof peer-pid lookup for port 47656 found no matching process ``` `ss` found that same port instantly (`claude pid=661919`), so the socket was there — only `lsof` inside the namespace could not see it. Bisected with `systemd-run --user`, counting `lsof` output lines against a live connection: | directive | lsof lines | |---|---| | no sandbox | 3 | | `ProtectSystem=strict` | **0** | | `ProtectHome=read-only` | **0** | | `ProtectKernelTunables=true` | **0** | | `ProtectControlGroups=true` | **0** | | `RestrictSUIDSGID=true` | 3 | | `NoNewPrivileges=true` | 3 | The unit ships the first four. The two safe ones are the two that add no mount namespace. What makes this bad is that nothing looks wrong: the unit starts, `/healthz` returns `{"status":"ok"}`, and `startup secret ...: set` reports all three secrets. The only symptom is that the fleet cannot be driven at all. ## 2. `PrivateTmp=true` breaks the member credential scrub `HerdrPeerLauncher` creates the member's ZDOTDIR scrub directory under `java.io.tmpdir`, and `OpenCodeLauncher` writes `$OPENCODE_CONFIG` there too. The member pane is a child of **herdr**, a different unit. With `PrivateTmp=true` on fleetd, the pane cannot see either directory. This is the same class as the finding above: `policy: allow-list` would report as configured while the control it names does nothing. ## 3. `ProtectHome=read-write` is not a valid value systemd accepts `yes`, `no`, `read-only`, `tmpfs`. `systemd-run` rejects it outright: ``` Failed to start transient service unit: Invalid ProtectHome setting: read-write ``` ## 4. `After=herdr.service` names a unit the repo does not ship `deploy/` has no `herdr.service`. herdr also needs a pty with a real window size or every pane spawn fails with `ghostty error -2`, so the unit is not a one-liner. ## What works (verified live on fleet01) A user unit with the four namespacing directives removed, `PrivateTmp=false`, and `ExecStart=/bin/zsh -lc "exec java -jar target/fleetd.jar fleetd.yaml"`: - `fleet_whoami` → `{"role":"primary"}` - `fleet_spawn` → worktree provisioned, `.mcp.json` neutralized, ZDOTDIR allow-list 25 names, `/tmp/fleetd-zdotdir-3332885004035289899` present, `CB-634 auto-open` fired, pane injectable - all three `startup secret ...: set` — the login shell reuses `~/.fleet/secrets.sh`, so no second copy of the secrets in a systemd drop-in The login shell matters: `ExecStart=` pointing straight at `java` starts fine with empty tokens, and the failure appears hours later as a member that cannot open a pull request. ## Notes - The `Environment=PATH=` line also hardcodes `/usr/lib/jvm/temurin-25-jdk/bin` and `/usr/share/maven/bin`; fleet01 has `/usr/bin/java` and `/usr/bin/mvn`. The login shell removes the need for that line at all. - Lingering (`loginctl enable-linger`) is required for a user unit to start at boot; it is not mentioned in the install comment.
ltms commented 2026-09-06 15:07:00 +02:00
Author
Owner
Copy Link
Copy Source

Fixed by PR #370, merged as 4ee7b16 on main.

deploy/fleetd.service is now the unit that has been running on fleet01,
comments included. deploy/herdr.service exists at last — fleetd.service's
After=/Wants= had named it since the beginning while nothing shipped it.
deploy/herdr-inner.sh is added as the login-shell template that
herdr.service wraps in a pty.

All three failures this issue named are closed:

  • no ProtectSystem / ProtectHome / ProtectKernelTunables /
    ProtectControlGroups, so lsof still sees the loopback peer and callers are
    not resolved ANONYMOUS. NoNewPrivileges and RestrictSUIDSGID are kept —
    the bisection showed those two add no mount namespace.
  • PrivateTmp=false on both units, so the member ZDOTDIR scrub directory
    stays readable by the member pane, which is a child of the other unit.
  • ExecStart goes through a login shell, so the daemon's secrets resolve.

The guard is SystemdUnitSafetyTest. A unit file has no compile step, so it
reads all three files and fails on an active forbidden directive, on
PrivateTmp=true, on an ExecStart that skips the login shell, on a
herdr-inner.sh that does not exec a login shell, and on one that does not set
a non-zero pty size. Every message names the consequence, not the rule. A
vacuity guard pins that all three files exist and that the DO-NOT-add comment
block still mentions each forbidden directive, so a commented mention stays
legal while an active directive does not.

Measured on merge: 1420 green. Adding ProtectHome=read-only to
herdr.service fails 1 test; removing the login shell from herdr-inner.sh
fails 1; removing its stty fails 1; deleting that file errors 3 and fails the
build rather than passing vacuously.

Wiki: 11-Features.md -> "The shipped systemd units no longer disable the
daemon they start".

Fixed by PR #370, merged as `4ee7b16` on `main`. `deploy/fleetd.service` is now the unit that has been running on fleet01, comments included. `deploy/herdr.service` exists at last — `fleetd.service`'s `After=`/`Wants=` had named it since the beginning while nothing shipped it. `deploy/herdr-inner.sh` is added as the login-shell template that `herdr.service` wraps in a pty. All three failures this issue named are closed: - no `ProtectSystem` / `ProtectHome` / `ProtectKernelTunables` / `ProtectControlGroups`, so `lsof` still sees the loopback peer and callers are not resolved ANONYMOUS. `NoNewPrivileges` and `RestrictSUIDSGID` are kept — the bisection showed those two add no mount namespace. - `PrivateTmp=false` on **both** units, so the member ZDOTDIR scrub directory stays readable by the member pane, which is a child of the other unit. - `ExecStart` goes through a login shell, so the daemon's secrets resolve. The guard is `SystemdUnitSafetyTest`. A unit file has no compile step, so it reads all three files and fails on an active forbidden directive, on `PrivateTmp=true`, on an `ExecStart` that skips the login shell, on a `herdr-inner.sh` that does not exec a login shell, and on one that does not set a non-zero pty size. Every message names the consequence, not the rule. A vacuity guard pins that all three files exist and that the DO-NOT-add comment block still mentions each forbidden directive, so a commented mention stays legal while an active directive does not. Measured on merge: 1420 green. Adding `ProtectHome=read-only` to `herdr.service` fails 1 test; removing the login shell from `herdr-inner.sh` fails 1; removing its `stty` fails 1; deleting that file errors 3 and fails the build rather than passing vacuously. Wiki: `11-Features.md` -> "The shipped systemd units no longer disable the daemon they start".
ltms closed this issue 2026-09-06 15:07:06 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#360
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?