Dai Ha
380eb63277
deploy/fleetd.service started clean on fleet01 but broke the daemon in three ways nothing logs: ProtectSystem/ProtectHome/ProtectKernelTunables/ProtectControlGroups each put the unit in its own mount namespace, which blinds fleetd's lsof-based caller lookup and falls every caller back to ANONYMOUS; PrivateTmp=true silently no-ops the credential scrub the member pane depends on; and running java directly from ExecStart skips the login shell that sources the daemon's secrets, so it boots with empty credentials. Replace the unit with the version verified working on fleet01 for a day, and add the deploy/herdr.service companion unit it was already depending on via After=/Wants= but which did not exist in the repo. Add deploy/herdr-inner.sh as the login-shell template herdr.service's ExecStart wraps in a pty. Add SystemdUnitSafetyTest (fleetd/src/test/java/dev/ltms/fleet/deploy) to read both unit files from disk and fail if a forbidden mount-namespacing directive is active, PrivateTmp is true, or fleetd.service's ExecStart does not go through a login shell -- the only guard possible for a unit file with no compile step.
78 lines
4.1 KiB
Desktop File
78 lines
4.1 KiB
Desktop File
# CB-504 — systemd unit for fleetd (Linux).
|
|
#
|
|
# fleetd #360: the previous version of this file started clean and broke the daemon in three ways
|
|
# that nothing logs (see the DO NOT block and the ExecStart/PrivateTmp comments below for what and
|
|
# why). The unit below, plus its companion deploy/herdr.service, is the version that has actually
|
|
# run on fleet01 without those failures. Do not "improve" it back toward the old shape without
|
|
# re-reading why each line is the way it is.
|
|
#
|
|
# Install (user service — fleetd drives the user's herdr, not a system daemon):
|
|
# mkdir -p ~/.config/systemd/user
|
|
# cp deploy/fleetd.service deploy/herdr.service ~/.config/systemd/user/
|
|
# # edit WorkingDirectory / ExecStart below for your host's paths and java location
|
|
# systemctl --user daemon-reload
|
|
# systemctl --user enable --now herdr fleetd
|
|
# journalctl --user -u fleetd -f
|
|
#
|
|
# Secrets (AI_GATEWAY_TOKEN, WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI, ...) are not set
|
|
# here and need no systemd drop-in: ExecStart runs a login shell, so they come from wherever your
|
|
# login shell already sources them (this host: ~/.fleet/secrets.sh via ~/.zprofile). If a token is
|
|
# missing there, fleetd still starts — the daemon reports every secret a configured profile
|
|
# references, by name, never by value:
|
|
# journalctl --user -u fleetd | grep 'startup secret'
|
|
# A resolved one logs "startup secret NAME: set (profile 'x' tokenEnv)"; a missing one logs
|
|
# "startup secret NAME: MISSING" at WARN and the daemon starts anyway — the first visible symptom
|
|
# is a member that cannot open a pull request, hours later and in a different component.
|
|
|
|
[Unit]
|
|
Description=fleetd — fleet message server
|
|
Documentation=https://git.ltms.dev/fleet/fleetd/wiki
|
|
# Ordering only. fleetd retries the herdr socket rather than exiting, which is what actually makes
|
|
# a late socket survivable. Do NOT add Requires=: a herdr restart must not take fleetd down too.
|
|
After=herdr.service
|
|
Wants=herdr.service
|
|
|
|
[Service]
|
|
Type=simple
|
|
WorkingDirectory=%h/LTMS/fleetd/fleetd
|
|
|
|
# A LOGIN shell, not java directly. Every secret this daemon needs (AI_GATEWAY_TOKEN,
|
|
# WORKER_GITEA_TOKEN, LAVINMQ_URI, COORD_AMQP_URI) lives in ~/.fleet/secrets.sh, which only
|
|
# ~/.zprofile sources. systemd runs no login shell. Started any other way the daemon boots fine
|
|
# and looks healthy, and the failure appears hours later as a member that cannot open a pull
|
|
# request. exec keeps it one process, so systemd tracks the right PID.
|
|
# This also avoids a SECOND copy of the secrets in a systemd drop-in: one source of truth.
|
|
ExecStart=/bin/zsh -lc "exec java -jar target/fleetd.jar fleetd.yaml"
|
|
|
|
# PrivateTmp MUST stay false -- see herdr.service. fleetd creates the member ZDOTDIR scrub dir and
|
|
# the opencode config dir under java.io.tmpdir, and the member pane (a herdr child, a different
|
|
# unit) has to read them. A private /tmp turns the credential scrub into a silent no-op.
|
|
PrivateTmp=false
|
|
|
|
Restart=on-failure
|
|
RestartSec=10s
|
|
# A bad config makes fleetd fail fast by design. Give up rather than restart-loop forever.
|
|
StartLimitBurst=5
|
|
StartLimitIntervalSec=120
|
|
|
|
# DO NOT add ProtectSystem=, ProtectHome=, ProtectKernelTunables= or ProtectControlGroups=.
|
|
# Measured on fleet01 2026-09-05: each of those gives the unit its own mount namespace, and
|
|
# fleetd resolves a caller role by running lsof to find the loopback peer PID
|
|
# (mcp/LsofPeerPidLookup). Inside such a namespace lsof returns nothing, every caller falls back
|
|
# to ANONYMOUS, and the primary is refused every orchestration call with
|
|
# "unauthenticated: anonymous may not SPAWN".
|
|
# The daemon still starts, healthz still returns ok and the secrets still resolve - the only
|
|
# symptom is that the fleet cannot be driven at all. Verified by bisecting the directives:
|
|
# no sandbox 3 lsof lines | ProtectSystem=strict 0 | ProtectHome=read-only 0
|
|
# ProtectKernelTunables 0 | ProtectControlGroups 0 | RestrictSUIDSGID 3 | NoNewPrivileges 3
|
|
# The two below add no mount namespace and are safe.
|
|
NoNewPrivileges=true
|
|
RestrictSUIDSGID=true
|
|
|
|
StandardOutput=journal
|
|
StandardError=journal
|
|
SyslogIdentifier=fleetd
|
|
|
|
[Install]
|
|
WantedBy=default.target
|