fleetd #504 item 1: stop swallowing real launchctl/systemctl failures on the 'loaded but not running' path
CI / contract (pull_request) Successful in 52s
CI / build (pull_request) Successful in 1m55s

The two 'loaded but not currently running' branches in the stop step (launchd/systemd, reached
when $OLD_PID is empty) ran 'launchctl unload'/'systemctl --user stop' with '2>/dev/null || true'
and printed 'ok' unconditionally. That swallowed a real supervisor failure (e.g. launchd or the
systemd user bus unreachable) exactly like a harmless already-stopped answer, and let the script
proceed to start a new daemon believing nothing was loaded -- the two-daemons failure fleetd #492
exists to prevent.

Adds unload_launchd_if_loaded/stop_systemd_if_loaded, applying systemd_loaded's own pattern
(capture stderr separately; a non-zero exit WITH stderr is a real failure, a non-zero exit with
empty stderr is a clean already-stopped answer) to the write side. The two call sites now use
these functions instead of the bare '|| true'.

Adds 5 tests: dies-on-real-failure and tolerates-clean-negative for each function, plus a
source-text check that the main flow calls the new functions instead of the original bare
'2>/dev/null || true'. All 5 verified by mutation (reintroducing the swallow, and separately
over-correcting to die unconditionally) -- each goes red with its own message, restores
byte-identical (full sha256), and passes a green control.
This commit is contained in:
Dai Ha
2026-09-12 13:42:27 +07:00
parent f1640f5dcc
commit e4eb3dbed4
2 changed files with 158 additions and 2 deletions
+49 -2
View File
@@ -287,6 +287,49 @@ systemd_loaded() {
return "$rc"
}
# fleetd #504: the "loaded but not currently running" branches in the main stop step (case
# launchd/systemd, reached when $OLD_PID is empty) used to run `launchctl unload`/`systemctl --user
# stop` with `2>/dev/null || true` and then print `ok` unconditionally — the exact conflation
# systemd_loaded/systemd_installed above already fixed on the READ side (fleetd #492 follow-up): a
# genuine "already stopped" answer (nonzero exit, nothing on stderr) is harmless, but a real tool
# failure (nonzero exit WITH a stderr message — e.g. launchd or the systemd user bus is
# unreachable) is not, and reporting `ok` on THAT means the start step below can register a fresh
# load on top of a supervisor that never actually let go: the exact two-daemons failure fleetd #492
# exists to prevent, reached from the one state (already odd) where a false `ok` is least
# affordable. These two functions apply the same "capture stderr separately, flag only a nonzero
# exit WITH stderr as a real failure" pattern to the WRITE side. No ${VAR:-default} anywhere here —
# see the #492 follow-up constraints comment above detect_supervisor for why a default would hide a
# lost value instead of surfacing it (fleetd #497's defect class).
unload_launchd_if_loaded() {
local err_file rc=0
if ! err_file="$(mktemp -t launchd-unload-err)"; then
die "could not create a temp file to capture 'launchctl unload' stderr — cannot tell a real
failure from a clean already-unloaded answer, so refusing to guess. The daemon's
supervision state was NOT touched."
fi
launchctl unload -w "$LAUNCHD_PLIST" >/dev/null 2>"$err_file" || rc=$?
if [ "$rc" -ne 0 ] && [ -s "$err_file" ]; then
die "'launchctl unload -w $LAUNCHD_PLIST' failed: $(cat "$err_file")
The daemon may still be under supervision; investigate before retrying."
fi
rm -f "$err_file"
}
stop_systemd_if_loaded() {
local err_file rc=0
if ! err_file="$(mktemp -t systemd-stop-err)"; then
die "could not create a temp file to capture 'systemctl --user stop' stderr — cannot tell a
real failure from a clean already-stopped answer, so refusing to guess. The daemon's
supervision state was NOT touched."
fi
systemctl --user stop "$SYSTEMD_UNIT" >/dev/null 2>"$err_file" || rc=$?
if [ "$rc" -ne 0 ] && [ -s "$err_file" ]; then
die "'systemctl --user stop $SYSTEMD_UNIT' failed: $(cat "$err_file")
The daemon may still be under supervision; investigate before retrying."
fi
rm -f "$err_file"
}
# fleetd #492: three real answers, not two — launchd, systemd, or genuinely unsupervised — plus a
# fourth, "ambiguous", for the one case this script cannot tell apart: both signals firing at once.
# That is exactly "I cannot tell who supervises this process", and guessing wrong here is how two
@@ -881,16 +924,20 @@ if [ -n "$OLD_PID" ]; then
elif [ "$SUPERVISOR_KIND" = "launchd" ]; then
# Loaded but not currently running (e.g. throttled after a crash loop). Unload it anyway so the
# start step below does a clean load, never a load stacked on an already-loaded label.
# fleetd #504: unload_launchd_if_loaded (above) tolerates a genuine already-unloaded answer but
# dies on a real `launchctl` failure — never a bare `|| true` that would print `ok` either way.
say "stop"
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"
launchctl unload -w "$LAUNCHD_PLIST" 2>/dev/null || true
unload_launchd_if_loaded
ok "launchd agent unloaded (was already not running)"
elif [ "$SUPERVISOR_KIND" = "systemd" ]; then
# Same case for systemd: the unit is known/active-capable but not currently running. `stop` on an
# already-stopped unit is a harmless no-op — kept for symmetry with the launchd branch above.
# fleetd #504: stop_systemd_if_loaded (above) tolerates that genuine no-op but dies on a real
# `systemctl` failure — never a bare `|| true` that would print `ok` either way.
say "stop"
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"
systemctl --user stop "$SYSTEMD_UNIT" 2>/dev/null || true
stop_systemd_if_loaded
ok "systemd --user unit stopped (was already not running)"
else
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"