fleetd #360: fix systemd units disabling caller identity and the credential scrub #370

2026-09-06T14:45:58+02:00

agent commented

2026-09-06 14:45:58 +02:00

fleetd #360: replace the systemd units that silently broke the daemon.

What was wrong (measured on fleet01, not re-derived here):

ProtectSystem=strict / ProtectHome=read-write / ProtectKernelTunables=true / ProtectControlGroups=true each give the unit its own mount namespace. fleetd resolves a caller's role by running lsof to find the loopback peer PID (mcp/LsofPeerPidLookup). Inside that namespace lsof returns nothing, every caller falls back to ANONYMOUS, and the primary is refused every orchestration call with "unauthenticated: anonymous may not SPAWN" — while the daemon still starts and /healthz still reports ok.
PrivateTmp=true silently turns the credential scrub into a no-op: fleetd writes the member ZDOTDIR scrub dir and the opencode config dir under java.io.tmpdir, and the member pane (a child of a different unit, herdr.service) has to read them back.
Running java directly from ExecStart skips the login shell that sources the daemon's secrets (AI_GATEWAY_TOKEN, WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI). The daemon boots fine with all of them empty; the failure surfaces hours later as a member that cannot open a PR.

What changed:

deploy/fleetd.service replaced with the unit that has run on fleet01 for a day without these failures. Kept/expanded every explanatory comment (including the "DO NOT add these four directives" block with the fleet01 bisection numbers), and kept the header useful for an operator installing on a new host.
deploy/herdr.service added — fleetd.service's After=/Wants=herdr.service assumed this unit existed; it did not ship in the repo.
deploy/herdr-inner.sh added as a template for the login-shell script herdr.service's ExecStart wraps in a pty via script -qfec (not committed at the literal path herdr.service names, since the session name is host-specific — the header says to copy it there and edit the session name).
fleetd/src/test/java/dev/ltms/fleet/deploy/SystemdUnitSafetyTest.java added: reads both unit files from disk and fails if a forbidden mount-namespacing directive is active (commented-out mentions in the DO-NOT block are exempted), if PrivateTmp=true is active, or if fleetd.service's ExecStart does not go through a login shell (-lc). This is the only guard possible for a unit file with no compile step, mirroring McpContractDocTest's doc-vs-code pattern.

Build: mvn clean install from fleetd/, run unpiped: BUILD SUCCESS, Tests run: 1418, Failures: 0, Errors: 0, Skipped: 0.

Mutation proof (criterion 3): put ProtectSystem=strict back into deploy/fleetd.service, ran mvn -Dtest=SystemdUnitSafetyTest test → Tests run: 6, Failures: 1, Errors: 0, Skipped: 0 (the new test caught it, naming the ANONYMOUS-caller consequence). Removed it again → Tests run: 6, Failures: 0, Errors: 0, Skipped: 0.

Criterion 5 — shape survey, findings only, nothing fixed:

deploy/dev.ltms.fleetd.plist (launchd): launchd has no mount-namespacing directive equivalent to systemd's Protect*, so the #360 failure mode does not apply; it already carries its own login-shell fix for secrets (CB-594, scripts/fleetd-launchd-wrapper.sh).
deploy/lavinmq/compose.yaml: ships LavinMQ's default guest/guest credentials (loopback-bound only, documented in the file) — not invisible at startup since a missing broker is a hard startup failure by design, but the default credential itself has no startup check of its own.
scripts/*.sh (redeploy-fleetd.sh, probe-member-credentials.sh, rename-checkout.sh, fleetd-launchd-wrapper.sh, test-redeploy-fleetd.sh): each already documents and guards a specific invisible-at-startup trap in its own header (login-shell secrets, dual-supervisor race, stale PID, absolute-path breakage) — no new unguarded instance of the #360 shape found in these.

fleetd #360: replace the systemd units that silently broke the daemon. **What was wrong (measured on fleet01, not re-derived here):** - `ProtectSystem=strict` / `ProtectHome=read-write` / `ProtectKernelTunables=true` / `ProtectControlGroups=true` each give the unit its own mount namespace. fleetd resolves a caller's role by running `lsof` to find the loopback peer PID (`mcp/LsofPeerPidLookup`). Inside that namespace `lsof` returns nothing, every caller falls back to ANONYMOUS, and the primary is refused every orchestration call with "unauthenticated: anonymous may not SPAWN" — while the daemon still starts and `/healthz` still reports ok. - `PrivateTmp=true` silently turns the credential scrub into a no-op: fleetd writes the member ZDOTDIR scrub dir and the opencode config dir under `java.io.tmpdir`, and the member pane (a child of a *different* unit, herdr.service) has to read them back. - Running `java` directly from `ExecStart` skips the login shell that sources the daemon's secrets (AI_GATEWAY_TOKEN, WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI). The daemon boots fine with all of them empty; the failure surfaces hours later as a member that cannot open a PR. **What changed:** - `deploy/fleetd.service` replaced with the unit that has run on fleet01 for a day without these failures. Kept/expanded every explanatory comment (including the "DO NOT add these four directives" block with the fleet01 bisection numbers), and kept the header useful for an operator installing on a new host. - `deploy/herdr.service` added — fleetd.service's `After=`/`Wants=herdr.service` assumed this unit existed; it did not ship in the repo. - `deploy/herdr-inner.sh` added as a template for the login-shell script `herdr.service`'s `ExecStart` wraps in a pty via `script -qfec` (not committed at the literal path `herdr.service` names, since the session name is host-specific — the header says to copy it there and edit the session name). - `fleetd/src/test/java/dev/ltms/fleet/deploy/SystemdUnitSafetyTest.java` added: reads both unit files from disk and fails if a forbidden mount-namespacing directive is active (commented-out mentions in the DO-NOT block are exempted), if `PrivateTmp=true` is active, or if `fleetd.service`'s `ExecStart` does not go through a login shell (`-lc`). This is the only guard possible for a unit file with no compile step, mirroring `McpContractDocTest`'s doc-vs-code pattern. **Build:** `mvn clean install` from `fleetd/`, run unpiped: `BUILD SUCCESS`, `Tests run: 1418, Failures: 0, Errors: 0, Skipped: 0`. **Mutation proof (criterion 3):** put `ProtectSystem=strict` back into `deploy/fleetd.service`, ran `mvn -Dtest=SystemdUnitSafetyTest test` → `Tests run: 6, Failures: 1, Errors: 0, Skipped: 0` (the new test caught it, naming the ANONYMOUS-caller consequence). Removed it again → `Tests run: 6, Failures: 0, Errors: 0, Skipped: 0`. **Criterion 5 — shape survey, findings only, nothing fixed:** - `deploy/dev.ltms.fleetd.plist` (launchd): launchd has no mount-namespacing directive equivalent to systemd's `Protect*`, so the #360 failure mode does not apply; it already carries its own login-shell fix for secrets (CB-594, `scripts/fleetd-launchd-wrapper.sh`). - `deploy/lavinmq/compose.yaml`: ships LavinMQ's default `guest`/`guest` credentials (loopback-bound only, documented in the file) — not invisible at *startup* since a missing broker is a hard startup failure by design, but the default credential itself has no startup check of its own. - `scripts/*.sh` (`redeploy-fleetd.sh`, `probe-member-credentials.sh`, `rename-checkout.sh`, `fleetd-launchd-wrapper.sh`, `test-redeploy-fleetd.sh`): each already documents and guards a specific invisible-at-startup trap in its own header (login-shell secrets, dual-supervisor race, stale PID, absolute-path breakage) — no new unguarded instance of the #360 shape found in these.

agent added 1 commit 2026-09-06 14:45:58 +02:00

fleetd #360 : fix systemd units that silently disabled caller identity and the credential scrub

CI / contract (pull_request) Successful in 44s

Details

CI / build (pull_request) Successful in 2m12s

Details

380eb63277

deploy/fleetd.service started clean on fleet01 but broke the daemon in three ways nothing
logs: ProtectSystem/ProtectHome/ProtectKernelTunables/ProtectControlGroups each put the unit
in its own mount namespace, which blinds fleetd's lsof-based caller lookup and falls every
caller back to ANONYMOUS; PrivateTmp=true silently no-ops the credential scrub the member
pane depends on; and running java directly from ExecStart skips the login shell that sources
the daemon's secrets, so it boots with empty credentials.

Replace the unit with the version verified working on fleet01 for a day, and add the
deploy/herdr.service companion unit it was already depending on via After=/Wants= but which
did not exist in the repo. Add deploy/herdr-inner.sh as the login-shell template
herdr.service's ExecStart wraps in a pty.

Add SystemdUnitSafetyTest (fleetd/src/test/java/dev/ltms/fleet/deploy) to read both unit
files from disk and fail if a forbidden mount-namespacing directive is active, PrivateTmp is
true, or fleetd.service's ExecStart does not go through a login shell -- the only guard
possible for a unit file with no compile step.

agent added 1 commit 2026-09-06 14:54:44 +02:00

fleetd #360 review round 2: extend SystemdUnitSafetyTest to herdr-inner.sh

CI / build (pull_request) Successful in 1m27s

Details

CI / contract (pull_request) Successful in 1m30s

Details

8557289dc0

herdr.service's ExecStart only names deploy/herdr-inner.sh, so that script was the only
place herdr's login-shell and pty-size properties lived, and nothing was reading it -- the
same silent-at-startup shape the ticket was about, one file further down the chain. A
mutation dropping both the login shell and the stty sizing left SystemdUnitSafetyTest green.

Add herdrInnerScriptUsesALoginShell and herdrInnerScriptSetsANonZeroPtySize, extend the
vacuity guard to cover herdr-inner.sh too, and tighten execStartUsesALoginShell's check from
a bare contains("-lc") substring match to the same login-shell-invocation regex the new
checks use.

ltms referenced this issue from a commit

2026-09-06 15:06:00 +02:00

Merge #370: ship systemd units that do not silently disable the daemon

ltms commented

2026-09-06 15:06:40 +02:00

Merged as 4ee7b16 on main.

Verified on the merge itself, not taken from the report:

mvn clean install -> Tests run: 1420, Failures: 0, Errors: 0, Skipped: 0
                     BUILD SUCCESS

Round 1 shipped deploy/herdr-inner.sh with no test reading it. I mutated it —
dropped both the login shell and the stty sizing — and got 6 green, which is
what round 2 fixed.

Mutation run on this merge, on a half round 2 also did not touch: added
ProtectHome=read-only to herdr.service, the unit it only ever mutated
fleetd.service for, and a directive it never mutated at all.

Tests run: 8, Failures: 1 -- BUILD FAILURE

So the parameterisation really covers both files, not just the one that was
exercised.

Accepted as-is: the herdr-inner.sh-as-a-template decision, the shared
LOGIN_SHELL_INVOCATION regex replacing contains("-lc"), the [1-9]\d* pty
check (which catches stty rows 0 cols 0, the actual failure mode, rather than
merely "stty is present"), and the criterion-5 survey.

Wiki entry added: 11-Features.md -> "The shipped systemd units no longer
disable the daemon they start" (wiki fd7824d).

Merged as `4ee7b16` on `main`. Verified on the merge itself, not taken from the report: ``` mvn clean install -> Tests run: 1420, Failures: 0, Errors: 0, Skipped: 0 BUILD SUCCESS ``` Round 1 shipped `deploy/herdr-inner.sh` with no test reading it. I mutated it — dropped both the login shell and the `stty` sizing — and got 6 green, which is what round 2 fixed. Mutation run on this merge, on a half round 2 also did not touch: added `ProtectHome=read-only` to **herdr.service**, the unit it only ever mutated `fleetd.service` for, and a directive it never mutated at all. ``` Tests run: 8, Failures: 1 -- BUILD FAILURE ``` So the parameterisation really covers both files, not just the one that was exercised. Accepted as-is: the `herdr-inner.sh`-as-a-template decision, the shared `LOGIN_SHELL_INVOCATION` regex replacing `contains("-lc")`, the `[1-9]\d*` pty check (which catches `stty rows 0 cols 0`, the actual failure mode, rather than merely "stty is present"), and the criterion-5 survey. Wiki entry added: `11-Features.md` -> "The shipped systemd units no longer disable the daemon they start" (wiki `fd7824d`).

ltms closed this pull request

2026-09-06 15:06:45 +02:00

ltms referenced this pull request

2026-09-06 15:07:00 +02:00

deploy/fleetd.service silently disables caller identity and the credential scrub #360

ltms referenced this issue from a commit

2026-09-06 15:08:10 +02:00

fleetd #360 follow-up: the install block must say loginctl enable-linger