Dai Ha
8d79d229ff
redeploy-fleetd.sh's main flow had 8 bare if/case decisions (CHECK_ONLY
short-circuit, drain-gate entry+confirm, supervisor report/stop/start
dispatch, health-poll decision, HAD_OLD_PID computation) that lived outside
any function, so the 67-test suite could not reach them and any one could be
silently inverted with the whole suite green.
Follows the existing swap_if_built/refuse_drain_gate pattern: each bare
guard becomes a small predicate or dispatch function (should_stop_for_check,
drain_gate_required/drain_confirmed/run_drain_gate, report_supervisor_state,
dispatch_stop, dispatch_start, health_is_up/report_health,
compute_had_old_pid), called unconditionally by the main flow so the
decision itself is unit-testable in isolation.
Adds a structural guard, test_no_untested_main_flow_conditionals, that scans
the main flow (everything after the SOURCED guard) for bare if/elif/case
lines outside any function body, tracking function boundaries via this
file's one consistent name() { / } convention. It fails on any new bare
conditional not covered by MAIN_FLOW_ALLOWED_CONDITIONALS, an explicit
exact-text allowlist of the report-only/display conditionals and the two
#504-family supervisor elif branches that stay out of scope for this
ticket. This is the "shape, not the eight sites" guard the ticket asked
for: a ninth bare decision fails immediately, naming its line.
Out of scope, not touched: #504 items 2/3/4 and #528 item 2 (same
untested-main-flow family) — the seam here generalizes to make them
testable too, but lifting them was left for their own tickets.
1340 lines
76 KiB
Bash
Executable File
1340 lines
76 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Rebuild and restart the fleetd daemon.
|
|
#
|
|
# A merge is not a deployment: the running daemon holds the jar it was started with, so code merged
|
|
# to main does nothing until this runs. See CLAUDE.md -> "Redeploying the daemon".
|
|
#
|
|
# This script exists to turn eight remembered traps into one auditable command:
|
|
#
|
|
# 1. A piped `mvn` hides BUILD FAILURE behind a zero exit, so the build here is never piped.
|
|
# 2. The daemon must start from a LOGIN shell, or the tokens it hands to members are empty:
|
|
# WORKER_GITEA_TOKEN (workers cannot open a PR) and AI_GATEWAY_TOKEN (401 at llm.ltms.dev).
|
|
# Both are read from the DAEMON's own environment at spawn time, so a value added to
|
|
# secrets.sh after startup is absent. Nothing logs this here, so the script checks and says
|
|
# so — and since CB-594, fleetd's own startup log says so too, by env var name.
|
|
# 3. An old daemon that never actually died looks identical from the outside, so the script waits
|
|
# for the process to exit and for the port to free before it starts a new one.
|
|
# 4. "It started" is not "it works": the script polls /healthz until it answers, and reports the
|
|
# herdr protocol number, because healthz can be green while every spawn fails on a protocol
|
|
# mismatch.
|
|
# 5. Restarting under live members drops their tickets, so the script refuses unless you confirm
|
|
# the fleet is drained.
|
|
# 6. CB-594 — the launchd agent (deploy/dev.ltms.fleetd.plist), if installed and loaded, is a
|
|
# SECOND supervisor: its KeepAlive.SuccessfulExit=false restarts the daemon on any nonzero
|
|
# exit, and a bare SIGTERM makes this JVM exit 143 even with its shutdown hook running to
|
|
# completion (measured — see the CB-594 report). A plain `kill` here would race launchd's own
|
|
# restart of the OLD jar. So this script detects whether the agent is loaded and, only then,
|
|
# swaps `kill` + manual `nohup` for `launchctl unload`/`load` — the one supervisor in control
|
|
# at any moment is whichever one you asked to act, never both.
|
|
# 7. fleetd #492 — a systemd --user unit is a THIRD possible supervisor (seen on a second host):
|
|
# Restart=on-failure treats this JVM's SIGTERM exit code (143, per CB-594 above) as a failure
|
|
# too, so a bare `kill` there would race systemd's own restart of the OLD jar exactly like
|
|
# launchd would. This script now tells launchd, systemd, and "genuinely unsupervised" apart as
|
|
# three different answers, drives whichever one it finds through its own control plane
|
|
# (`launchctl` / `systemctl --user`), and REFUSES outright — never falls back to `kill` — when
|
|
# it finds a supervision signal it cannot map to exactly one of the two it knows how to drive.
|
|
# A wrong guess here is how two daemons end up running against one herdr session. Follow-up:
|
|
# "not currently loaded" is not the same fact as "unsupervised" — a unit that is installed but
|
|
# activating/failed/pending-restart, or a `systemctl` call that could not answer at all (e.g.
|
|
# no user-bus access), both now read as a fifth answer, "unclear", and REFUSE the same way
|
|
# "ambiguous" does, rather than silently falling through to "none".
|
|
# 8. fleetd #492 — a post-restart check counts running fleetd processes and fails the whole run if
|
|
# more than one is alive. That is the one thing none of the checks above (healthz 200, jar id,
|
|
# the fresh "listening" line) can see: every one of them is satisfied by EITHER daemon.
|
|
# 9. fleetd #512 — the ERROR-line count above is blind by construction to the exact failure #493
|
|
# is about: an uncaught exception in a shutdown thread never passes through the logger, so it
|
|
# never carries an ERROR (or SEVERE) token that any count could see. This script now also
|
|
# greps the previous daemon's shutdown window for that exception's real shape, and separately
|
|
# asserts that SessionManager's drain-complete line (fleetd #522) is present there — its
|
|
# absence is the real signal, because a drain that dies on its first session prints nothing
|
|
# else either. Warns loudly; never fails the redeploy, because by the time this is detectable
|
|
# the new daemon is already up and healthy.
|
|
#
|
|
# Usage:
|
|
# scripts/redeploy-fleetd.sh # build, confirm, restart, verify
|
|
# scripts/redeploy-fleetd.sh --yes # skip the drain confirmation (fleet already checked)
|
|
# scripts/redeploy-fleetd.sh --no-build # restart the jar already on disk
|
|
# scripts/redeploy-fleetd.sh --check # report state and exit; changes nothing
|
|
#
|
|
# Exits non-zero on any failure. A failed build never stops the running daemon.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
MODULE="$REPO/fleetd"
|
|
JAR="$MODULE/target/fleetd.jar"
|
|
# fleetd #493: never build into the path a running process holds. The build writes here first
|
|
# (Maven's shade plugin has finalName=fleetd, so `clean install` still lands its output at
|
|
# target/fleetd.jar — that part is unchanged and out of this script's control), but this script
|
|
# now moves it out to JAR_STAGED immediately, and only swaps it back to JAR (a plain `mv`, so a
|
|
# rename, never a byte-by-byte overwrite) after the OLD daemon has been confirmed exited. See
|
|
# stage_built_jar/swap_staged_jar below.
|
|
JAR_STAGED="$MODULE/target/fleetd-new.jar"
|
|
OUT="$MODULE/fleetd.out"
|
|
# Matches BOTH the absolute form and the relative `java -jar target/fleetd.jar` a hand-start
|
|
# produces from inside fleetd/. Anchoring on the absolute path alone was a real bug: the daemon
|
|
# restarted correctly and the script still reported "no process appeared", because it launched with
|
|
# a relative path and then looked for an absolute one.
|
|
PATTERN='target/fleetd.jar'
|
|
HEALTH='http://127.0.0.1:8765/healthz'
|
|
STOP_WAIT=30 # seconds to wait for a clean exit before reporting failure
|
|
HEALTH_WAIT=60 # seconds to wait for /healthz to answer after start
|
|
|
|
# CB-594: the launchd agent this script must not fight with (see trap 6 above).
|
|
LAUNCHD_LABEL='dev.ltms.fleetd'
|
|
LAUNCHD_PLIST="$HOME/Library/LaunchAgents/$LAUNCHD_LABEL.plist"
|
|
|
|
# fleetd #492: the systemd --user unit this script must not fight with either (see trap 7 above).
|
|
# Measured on the second host: `systemctl --user cat fleetd` names the unit "fleetd" (not
|
|
# "dev.ltms.fleetd" — systemd user units here are not namespaced the way the launchd label is).
|
|
SYSTEMD_UNIT='fleetd'
|
|
|
|
# fleetd #492 follow-up: detect_supervisor packs TWO values (kind, detail) onto the one stdout
|
|
# line that survives its $(...) call — see the constraints comment above that function. This is
|
|
# the separator between them: the ASCII "unit separator" byte, chosen because it never occurs in
|
|
# any of the prose detail strings and needs no escaping in a `case`/glob pattern.
|
|
SUPERVISOR_DETAIL_SEP=$'\x1f'
|
|
|
|
# fleetd #492 follow-up, refined by fleetd #545: three states, not two, set by
|
|
# systemd_loaded/systemd_installed —
|
|
# 0 = no error, the probe ran and gave a clean answer.
|
|
# 1 = the probe RAN and answered badly: `systemctl` exited non-zero AND wrote something to
|
|
# stderr, a real tool failure (e.g. it cannot reach the user bus over a non-lingering ssh
|
|
# session), never the same fact as a clean negative answer ("not active", no stderr).
|
|
# 2 = the probe could not even be SET UP: the `mktemp` call that makes a place to capture
|
|
# `systemctl`'s stderr failed before `systemctl` ever ran. This is a fleetd #545 fix: on GNU
|
|
# coreutils (every Linux distribution) a template with no `X`s made `mktemp` fail every
|
|
# single time, and the two states were folded into one flag and one message that named
|
|
# cause 1 ("systemctl exited non-zero and reported an error on stderr") for a failure that
|
|
# was actually cause 2 — systemctl was never executed at all. One flag with two meanings
|
|
# needing different messages was the defect; a third value is the fix, not a second flag.
|
|
# Initialized here, not just inside the probes, so detect_supervisor can read them under `set -u`
|
|
# even before either probe has ever run, and so a test that stubs a probe with a plain
|
|
# `return 0`/`return 1` body (leaving these untouched) reads a deterministic 0 rather than whatever
|
|
# a previous probe call left behind.
|
|
SYSTEMD_LOADED_ERRORED=0
|
|
SYSTEMD_INSTALLED_ERRORED=0
|
|
# fleetd #492 follow-up: SUPERVISOR_UNCLEAR_DETAIL is the specific supervisor/reason that
|
|
# require_drivable_supervisor's die() names on an "unclear" answer. Deliberately NOT pre-declared
|
|
# here (unlike the two flags above): it is set only by the real call site, right after it unpacks
|
|
# detect_supervisor's stdout (see the constraints comment above detect_supervisor). If that call
|
|
# site is ever skipped or broken, a bare `set -u` reference to this variable in
|
|
# require_drivable_supervisor must fail loudly with "unbound variable" — a pre-declared empty
|
|
# default would instead silently print an empty reason, hiding exactly the value this ticket
|
|
# exists to surface.
|
|
|
|
DO_BUILD=1; ASSUME_YES=0; CHECK_ONLY=0
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--yes|-y) ASSUME_YES=1 ;;
|
|
--no-build) DO_BUILD=0 ;;
|
|
--check) CHECK_ONLY=1 ;;
|
|
-h|--help) sed -n '3,48p' "${BASH_SOURCE[0]}"; exit 0 ;;
|
|
*) echo "unknown option: $arg (try --help)" >&2; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
say() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
|
|
ok() { printf ' ok %s\n' "$*"; }
|
|
warn() { printf ' WARN %s\n' "$*"; }
|
|
die() { printf '\n FAIL %s\n\n' "$*" >&2; exit 1; }
|
|
|
|
# fleetd #550 — shasum is macOS-only (it ships with Perl, which Debian/Ubuntu/etc. do not install
|
|
# by default); GNU coreutils (every mainstream Linux distro) ships sha256sum instead and has no
|
|
# shasum at all. Prefer sha256sum, fall back to shasum -a 256 — same idiom as
|
|
# probe-member-credentials.sh's `hasher` selection — and when NEITHER is on PATH, say so plainly.
|
|
# That third answer matters: without it, a missing hasher makes `cut` succeed on empty input, and
|
|
# under `set -o pipefail` the pipeline as a whole still fails, so a caller's own `|| echo "absent"`
|
|
# then reports a file that is right there as though it were gone. `hash256` never does that — it
|
|
# only ever hashes or says it could not.
|
|
hash256() {
|
|
local f="$1"
|
|
if command -v sha256sum >/dev/null 2>&1; then
|
|
sha256sum "$f" | cut -c1-12
|
|
elif command -v shasum >/dev/null 2>&1; then
|
|
shasum -a 256 "$f" | cut -c1-12
|
|
else
|
|
echo "unhashable"
|
|
fi
|
|
}
|
|
|
|
# Reports the hash of $JAR by default, or of whatever path is passed — used to report the STAGED
|
|
# jar right after a build (before it has been swapped in) without ever changing what a bare
|
|
# `jar_id` (no args) means: the live path, $JAR. --check and the final "pid ..., jar ..." line
|
|
# both call it with no args on purpose, so neither can ever be fooled by a leftover staged file.
|
|
# fleetd #550 — THREE distinct answers now, not two: `[ -f "$f" ]` already separates "the jar is
|
|
# not there" (-> "absent") from "the jar is there"; for the second case, hash256 itself separates
|
|
# "hashed it" (a 12-char hex string) from "could not hash it" (-> "unhashable", when no hasher is
|
|
# on PATH). "absent" must never be the answer for a file that exists — that conflation, on Linux,
|
|
# was the whole defect this ticket fixes.
|
|
jar_id() { local f="${1:-$JAR}"; [ -f "$f" ] && hash256 "$f" || echo "absent"; }
|
|
running_pid() { pgrep -f "$PATTERN" || true; }
|
|
|
|
# fleetd #493 — three small, independently testable pieces of "never build into the path a
|
|
# running process holds":
|
|
#
|
|
# stage_built_jar moves the jar Maven just produced OUT of the live path and onto the staging
|
|
# path, immediately after a successful build. Dies (leaving the OLD daemon
|
|
# untouched — this runs before the stop step) if Maven reported success but
|
|
# left no jar behind, or if the move itself fails.
|
|
# require_no_build_jar the --no-build path never builds or stages anything: it must find a
|
|
# jar already sitting at the live path from an earlier successful run, and
|
|
# die with the same truthful message this script has always used if not.
|
|
# wait_for_daemon_exit polls running_pid() for up to $1 seconds and reports whether the OLD
|
|
# daemon actually exited — extracted to its own function so the main flow
|
|
# can be relied on to call swap_staged_jar only AFTER this returns success,
|
|
# and so a test can prove that ordering by reading the script's own source.
|
|
# swap_staged_jar the actual swap: a plain `mv` of the staged jar onto the live path. Called
|
|
# only once the OLD daemon is confirmed gone (see wait_for_daemon_exit above),
|
|
# so this is never a write into a path a running process holds — by the time
|
|
# it runs, nothing holds that path anymore. If it fails, the caller must not
|
|
# start a new daemon: die() below already refuses that by exiting the script.
|
|
stage_built_jar() {
|
|
[ -f "$JAR" ] || die "build succeeded but produced no jar at $JAR — cannot stage it for restart.
|
|
The running daemon was NOT touched."
|
|
mv -f "$JAR" "$JAR_STAGED" \
|
|
|| die "could not move the freshly built jar from $JAR to the staging path $JAR_STAGED.
|
|
The running daemon was NOT touched."
|
|
}
|
|
|
|
require_no_build_jar() {
|
|
[ -f "$JAR" ] || die "no jar at $JAR — run without --no-build"
|
|
}
|
|
|
|
wait_for_daemon_exit() {
|
|
local timeout="$1" _i
|
|
for _i in $(seq "$timeout"); do
|
|
[ -z "$(running_pid)" ] && return 0
|
|
sleep 1
|
|
done
|
|
[ -z "$(running_pid)" ]
|
|
}
|
|
|
|
swap_staged_jar() {
|
|
local staged="$1" live="$2"
|
|
[ -f "$staged" ] || die "no staged jar at $staged to swap in — the daemon was NOT started."
|
|
mv -f "$staged" "$live" \
|
|
|| die "could not move the staged jar from $staged into place at $live — the daemon was NOT
|
|
started. The built jar is still sitting at $staged; a manual 'mv \"$staged\" \"$live\"'
|
|
may recover this once you find out why the move failed."
|
|
}
|
|
|
|
# fleetd #521 — the swap decision, and the step that acts on it.
|
|
#
|
|
# The defect: the swap step used to be guarded inline by `if [ "$DO_BUILD" = 1 ]` in the main flow.
|
|
# Changing that to `if false` left the suite green and the swap never ran, so a redeploy reported
|
|
# every step succeeding while the daemon started on no jar at all (stage_built_jar has already moved
|
|
# the freshly built one to $JAR_STAGED by then) or on a stale one.
|
|
# test_swap_ordered_after_wait_and_before_start could not catch it: it reads this script's own text
|
|
# and compares line positions, and a same-line edit moves no line.
|
|
#
|
|
# Why these are TWO functions, and why the second one exists at all. Extracting only the predicate
|
|
# — `should_swap`, which is what #521 asked for — is not enough, and this was measured, not guessed:
|
|
# with the main flow calling `if should_swap "$DO_BUILD"; then`, changing THAT to `if false; then`
|
|
# still left the whole suite at exit 0 with no failures. Tests that call a predicate directly prove
|
|
# the predicate is right; nothing makes the code that does the work consult it. Extraction had moved
|
|
# the untested decision one level up rather than removing it.
|
|
#
|
|
# So the decision and the action live together in swap_if_built, and the main flow has no guard of
|
|
# its own to get wrong — it calls one function unconditionally. A test then calls swap_if_built with
|
|
# both values of do_build and checks whether the swap actually happened, which fails if the guard is
|
|
# removed, inverted, or stops being consulted. should_swap stays a separate predicate because it is
|
|
# the decision itself and is worth naming and testing on its own.
|
|
#
|
|
# What this still does not pin: deleting the swap_if_built call from the main flow altogether. That
|
|
# is the ordering test's job — its needle is that call site — and no test in this file can do better,
|
|
# because sourcing stops before the main flow ever runs (see the SOURCED guard below).
|
|
should_swap() {
|
|
local do_build="$1"
|
|
[ "$do_build" = 1 ]
|
|
}
|
|
|
|
swap_if_built() {
|
|
local do_build="$1"
|
|
should_swap "$do_build" || return 0
|
|
say "swap"
|
|
swap_staged_jar "$JAR_STAGED" "$JAR"
|
|
ok "jar in place: $(jar_id)"
|
|
}
|
|
|
|
# `launchctl list <label>` exits 0 iff the label is loaded (registered with launchd) — true whether
|
|
# or not it is currently running, which is exactly "supervision is active" for our purposes. Read-
|
|
# only: neither helper below changes anything, so both are also safe under --check.
|
|
launchd_installed() { [ -f "$LAUNCHD_PLIST" ]; }
|
|
launchd_loaded() { launchctl list "$LAUNCHD_LABEL" >/dev/null 2>&1; }
|
|
|
|
# fleetd #492: same two questions for systemd --user. Kept as separate, overridable functions
|
|
# (never an inline `systemctl` call at each use site) so a test on a box with no systemd at all
|
|
# (this repo is developed on macOS) can substitute each one independently — the same seam
|
|
# launchd_installed/launchd_loaded above already use.
|
|
#
|
|
# fleetd #492 follow-up: both functions used to throw `systemctl`'s stderr straight into
|
|
# /dev/null, which meant "systemctl answered no" and "systemctl could not answer at all" (e.g. it
|
|
# cannot reach the user bus over a non-lingering ssh session) looked identical — both a plain
|
|
# nonzero exit. They now capture stderr separately and set their own *_ERRORED flag to 1 when the
|
|
# call exited non-zero AND wrote something to stderr — a real tool failure, never a clean "not
|
|
# installed"/"not active" answer (which exits non-zero with empty stderr). detect_supervisor reads
|
|
# the flag right after calling the probe, so a probe that could not answer routes to "unclear",
|
|
# never silently becomes "none".
|
|
#
|
|
# fleetd #545: the flag has a third value, 2, set when the `mktemp` call that sets up the probe's
|
|
# own stderr capture fails, before `systemctl` ever runs — see the SYSTEMD_LOADED_ERRORED /
|
|
# SYSTEMD_INSTALLED_ERRORED comment above their initialization for why this is a third value on the
|
|
# same flag, not a second flag.
|
|
#
|
|
# "installed": a unit FILE by this name exists, regardless of its current state — the systemd
|
|
# analogue of the plist file existing on disk. `list-unit-files` reads unit definitions without
|
|
# depending on runtime state, so this stays read-only and safe under --check.
|
|
systemd_installed() {
|
|
SYSTEMD_INSTALLED_ERRORED=0
|
|
command -v systemctl >/dev/null 2>&1 || return 1
|
|
local err_file out rc=0
|
|
if ! err_file="$(mktemp -t systemd-installed-err.XXXXXX)"; then
|
|
SYSTEMD_INSTALLED_ERRORED=2
|
|
return 1
|
|
fi
|
|
out="$(systemctl --user list-unit-files "$SYSTEMD_UNIT.service" --no-legend 2>"$err_file")" || rc=$?
|
|
if [ "$rc" -ne 0 ]; then
|
|
if [ -s "$err_file" ]; then
|
|
SYSTEMD_INSTALLED_ERRORED=1
|
|
fi
|
|
rm -f "$err_file"
|
|
return "$rc"
|
|
fi
|
|
rm -f "$err_file"
|
|
printf '%s' "$out" | grep -q .
|
|
}
|
|
# "loaded": systemd currently supervises this unit as an active job — the systemd analogue of
|
|
# `launchctl list <label>` succeeding. Measured on the second host: `systemctl --user is-active
|
|
# fleetd` -> "active". A clean "no" (inactive/failed/activating/deactivating) exits non-zero with
|
|
# nothing on stderr; a probe that could not reach systemd at all exits non-zero WITH a stderr
|
|
# message — see the fleetd #492 follow-up note above.
|
|
systemd_loaded() {
|
|
SYSTEMD_LOADED_ERRORED=0
|
|
command -v systemctl >/dev/null 2>&1 || return 1
|
|
local err_file rc=0
|
|
if ! err_file="$(mktemp -t systemd-loaded-err.XXXXXX)"; then
|
|
SYSTEMD_LOADED_ERRORED=2
|
|
return 1
|
|
fi
|
|
systemctl --user is-active "$SYSTEMD_UNIT" >/dev/null 2>"$err_file" || rc=$?
|
|
if [ "$rc" -ne 0 ] && [ -s "$err_file" ]; then
|
|
SYSTEMD_LOADED_ERRORED=1
|
|
fi
|
|
rm -f "$err_file"
|
|
return "$rc"
|
|
}
|
|
|
|
# fleetd #504: the "loaded but not currently running" branches in the main stop step (case
|
|
# launchd/systemd, reached when $OLD_PID is empty) used to run `launchctl unload`/`systemctl --user
|
|
# stop` with `2>/dev/null || true` and then print `ok` unconditionally — the exact conflation
|
|
# systemd_loaded/systemd_installed above already fixed on the READ side (fleetd #492 follow-up): a
|
|
# genuine "already stopped" answer (nonzero exit, nothing on stderr) is harmless, but a real tool
|
|
# failure (nonzero exit WITH a stderr message — e.g. launchd or the systemd user bus is
|
|
# unreachable) is not, and reporting `ok` on THAT means the start step below can register a fresh
|
|
# load on top of a supervisor that never actually let go: the exact two-daemons failure fleetd #492
|
|
# exists to prevent, reached from the one state (already odd) where a false `ok` is least
|
|
# affordable. These two functions apply the same "capture stderr separately, flag only a nonzero
|
|
# exit WITH stderr as a real failure" pattern to the WRITE side. No ${VAR:-default} anywhere here —
|
|
# see the #492 follow-up constraints comment above detect_supervisor for why a default would hide a
|
|
# lost value instead of surfacing it (fleetd #497's defect class).
|
|
unload_launchd_if_loaded() {
|
|
local err_file rc=0
|
|
if ! err_file="$(mktemp -t launchd-unload-err.XXXXXX)"; then
|
|
die "could not create a temp file to capture 'launchctl unload' stderr — cannot tell a real
|
|
failure from a clean already-unloaded answer, so refusing to guess. The daemon's
|
|
supervision state was NOT touched."
|
|
fi
|
|
launchctl unload -w "$LAUNCHD_PLIST" >/dev/null 2>"$err_file" || rc=$?
|
|
if [ "$rc" -ne 0 ] && [ -s "$err_file" ]; then
|
|
die "'launchctl unload -w $LAUNCHD_PLIST' failed: $(cat "$err_file")
|
|
The daemon may still be under supervision; investigate before retrying."
|
|
fi
|
|
rm -f "$err_file"
|
|
}
|
|
|
|
stop_systemd_if_loaded() {
|
|
local err_file rc=0
|
|
if ! err_file="$(mktemp -t systemd-stop-err.XXXXXX)"; then
|
|
die "could not create a temp file to capture 'systemctl --user stop' stderr — cannot tell a
|
|
real failure from a clean already-stopped answer, so refusing to guess. The daemon's
|
|
supervision state was NOT touched."
|
|
fi
|
|
systemctl --user stop "$SYSTEMD_UNIT" >/dev/null 2>"$err_file" || rc=$?
|
|
if [ "$rc" -ne 0 ] && [ -s "$err_file" ]; then
|
|
die "'systemctl --user stop $SYSTEMD_UNIT' failed: $(cat "$err_file")
|
|
The daemon may still be under supervision; investigate before retrying."
|
|
fi
|
|
rm -f "$err_file"
|
|
}
|
|
|
|
# fleetd #492: three real answers, not two — launchd, systemd, or genuinely unsupervised — plus a
|
|
# fourth, "ambiguous", for the one case this script cannot tell apart: both signals firing at once.
|
|
# That is exactly "I cannot tell who supervises this process", and guessing wrong here is how two
|
|
# daemons end up running against one herdr session (see trap 7 in the header).
|
|
#
|
|
# fleetd #492 follow-up: a fifth answer, "unclear", for two more situations that must NEVER be read
|
|
# as "none" (measured — see the report this ticket is a follow-up to):
|
|
# - installed-but-not-loaded, on EITHER supervisor. `systemctl --user is-active` answers "no" for
|
|
# `activating`, `deactivating`, `failed`, and while an auto-restart is pending — every one of
|
|
# those is a host that IS under systemd (or launchd) and whose supervisor is about to act again.
|
|
# `*_installed` already knows the unit/agent exists; this is the first place that fact is
|
|
# actually consulted in the decision, not just printed as a warning.
|
|
# - a probe that could not answer at all. systemd_loaded/systemd_installed set their own
|
|
# *_ERRORED flag (see the comment above them) when `systemctl` exits non-zero WITH a stderr
|
|
# message — a real tool failure, e.g. it cannot reach the user bus over a non-lingering ssh
|
|
# session — never conflated with a clean negative answer.
|
|
# "none" now means only: neither supervisor is installed, neither is loaded, and neither probe
|
|
# errored.
|
|
#
|
|
# fleetd #545: *_ERRORED carries a THIRD state (2 = the probe's own mktemp setup failed, before
|
|
# `systemctl` ever ran — see the flag's own comment above its initialization), and it must never be
|
|
# reported with the same detail text as state 1 (`systemctl` ran and answered badly on stderr). The
|
|
# two are different facts about different failures, and conflating them makes the "unclear" message
|
|
# assert a cause ("systemctl exited non-zero and reported an error on stderr") that was never
|
|
# measured when the real cause was state 2. detect_supervisor below picks the detail text off the
|
|
# flag's value, not off a single "errored at all" boolean.
|
|
#
|
|
# fleetd #492 follow-up — constraints every caller of this function depends on (learned the hard
|
|
# way: an earlier version of this fix set a SUPERVISOR_UNCLEAR_DETAIL global from inside here and
|
|
# it was silently lost, because every real call site invokes this as `$(detect_supervisor)`):
|
|
# 1. It is called as `$(detect_supervisor)`, so ONLY STDOUT crosses back to the caller. Anything
|
|
# this function needs to tell its caller — the "unclear" detail included — must be printed,
|
|
# never assigned to a global: a global set inside a `$( )` subshell dies with that subshell.
|
|
# This function packs BOTH values (kind and detail) onto that one stdout line, joined by
|
|
# $SUPERVISOR_DETAIL_SEP, and the caller unpacks them on its own side of the subshell boundary.
|
|
# 2. This script runs under `set -u` (part of the `set -euo pipefail` at the top of the file), so
|
|
# an unset variable is a loud failure. Do not add a `${VAR:-default}` anywhere downstream to
|
|
# paper over a value that should always be there — that hides a lost value instead of
|
|
# surfacing it (fleetd #497's defect class). The mechanism is named rather than cited by line
|
|
# number on purpose: a line number in a comment goes stale on the next insert above it, and
|
|
# this one already had — it said line 50 while the `set` line was at 54.
|
|
# 3. Every `case` on this function's return value needs an explicit final `*)` arm, chosen by
|
|
# whether that caller ACTS on the value (`die` — an unrecognised value must never be silently
|
|
# driven) or only DISPLAYS it (`echo`/`warn` and continue — a diagnostic must not go silent on
|
|
# exactly the value it most needs to report).
|
|
#
|
|
# Pure and side-effect-free besides the two *_ERRORED flags (read back within this same call, never
|
|
# by the caller — see the constraints above): reads the four probes and decides — never mutates
|
|
# anything, so it is safe under --check and testable by overriding
|
|
# launchd_installed/launchd_loaded/systemd_installed/systemd_loaded after sourcing.
|
|
detect_supervisor() {
|
|
local ld=0 sd=0 li=0 si=0 kind detail=""
|
|
SYSTEMD_LOADED_ERRORED=0
|
|
SYSTEMD_INSTALLED_ERRORED=0
|
|
|
|
launchd_loaded && ld=1
|
|
systemd_loaded && sd=1
|
|
launchd_installed && li=1
|
|
systemd_installed && si=1
|
|
|
|
if [ "$SYSTEMD_LOADED_ERRORED" = 2 ] || [ "$SYSTEMD_INSTALLED_ERRORED" = 2 ]; then
|
|
detail="the systemd --user probe for '$SYSTEMD_UNIT' could not even be set up (a temp file to capture systemctl's stderr could not be created) — systemctl was never run, so this says nothing about systemd, the user bus, or the unit itself"
|
|
kind="unclear"
|
|
elif [ "$SYSTEMD_LOADED_ERRORED" = 1 ] || [ "$SYSTEMD_INSTALLED_ERRORED" = 1 ]; then
|
|
detail="the systemd --user probe for '$SYSTEMD_UNIT' could not answer cleanly (systemctl exited non-zero and reported an error on stderr, not a clean negative — e.g. it cannot reach the user bus)"
|
|
kind="unclear"
|
|
elif [ "$ld" = 1 ] && [ "$sd" = 1 ]; then
|
|
kind="ambiguous"
|
|
elif [ "$li" = 1 ] && [ "$ld" = 0 ]; then
|
|
detail="the launchd agent ($LAUNCHD_LABEL) is installed ($LAUNCHD_PLIST exists) but is not currently loaded"
|
|
kind="unclear"
|
|
elif [ "$si" = 1 ] && [ "$sd" = 0 ]; then
|
|
detail="the systemd --user unit ($SYSTEMD_UNIT) is installed but not currently active — it may be activating, deactivating, failed, or waiting on an auto-restart"
|
|
kind="unclear"
|
|
elif [ "$ld" = 1 ]; then
|
|
kind="launchd"
|
|
elif [ "$sd" = 1 ]; then
|
|
kind="systemd"
|
|
else
|
|
kind="none"
|
|
fi
|
|
|
|
printf '%s%s%s' "$kind" "$SUPERVISOR_DETAIL_SEP" "$detail"
|
|
}
|
|
|
|
# fleetd #492: turns anything detect_supervisor returns that is NOT exactly one of the two
|
|
# supervisors this script knows how to drive into a die() — never a fall-through to the `kill`
|
|
# path. Kept as its own function so a test can call it directly (in a subshell, since it die()s)
|
|
# without running the whole report-state flow or needing a real launchd/systemd.
|
|
require_drivable_supervisor() {
|
|
local kind="$1"
|
|
case "$kind" in
|
|
launchd|systemd|none) ;;
|
|
ambiguous)
|
|
die "both launchd ($LAUNCHD_LABEL) and systemd --user ($SYSTEMD_UNIT) report themselves as
|
|
loaded for this daemon at the same time. This script cannot tell which one actually
|
|
supervises the running process, and driving either alone risks the OTHER reviving the
|
|
OLD jar out from under it — the exact failure this ticket (fleetd #492) exists to
|
|
prevent. Stop one of the two supervisors by hand, confirm only one remains loaded, then
|
|
rerun." ;;
|
|
unclear)
|
|
# fleetd #492 follow-up: SUPERVISOR_UNCLEAR_DETAIL crosses back from detect_supervisor's
|
|
# subshell via its stdout, unpacked by the caller BEFORE it calls this function (see the
|
|
# constraints comment above detect_supervisor). No ${VAR:-default} here on purpose: if the
|
|
# detail is somehow missing, `set -u` makes this reference fail loudly instead of silently
|
|
# naming nothing — a default that hides a lost value is the same defect class as fleetd
|
|
# #497.
|
|
die "a supervisor looks present but this script cannot tell whether it actually drives this
|
|
daemon: $SUPERVISOR_UNCLEAR_DETAIL. Guessing wrong here is the same failure 'ambiguous'
|
|
above exists to prevent: driving the daemon while an unseen supervisor revives the OLD
|
|
jar out from under it (fleetd #492). Check 'launchctl list $LAUNCHD_LABEL' and
|
|
'systemctl --user status $SYSTEMD_UNIT' by hand, resolve whichever looks unclear, then
|
|
rerun." ;;
|
|
*)
|
|
die "detect_supervisor returned an unrecognized value '$kind' — refusing to guess which
|
|
supervisor, if any, controls this daemon." ;;
|
|
esac
|
|
}
|
|
|
|
# fleetd #492: the exact symptom a racing supervisor produces — count how many fleetd processes are
|
|
# alive right now. Takes the pid list as a parameter (rather than calling running_pid() itself) so a
|
|
# test can pass a canned two-line string without a real second process running. Pure except for the
|
|
# die() in assert_single_daemon below.
|
|
count_daemon_pids() {
|
|
local pids="$1"
|
|
if [ -z "$pids" ]; then
|
|
echo 0
|
|
else
|
|
printf '%s\n' "$pids" | grep -c .
|
|
fi
|
|
}
|
|
assert_single_daemon() {
|
|
local pids="$1" count
|
|
count="$(count_daemon_pids "$pids")"
|
|
if [ "$count" -gt 1 ]; then
|
|
die "more than one fleetd process is running after this restart (pids: $(printf '%s' "$pids" | tr '\n' ' ')).
|
|
This is the exact failure a racing supervisor produces: the OLD jar was revived by its
|
|
supervisor while this script started a NEW copy. Two daemons on one herdr session kill
|
|
each other's members. Investigate with 'pgrep -f \"$PATTERN\"' and stop the wrong one by
|
|
hand — do not assume either pid is the one you want."
|
|
fi
|
|
}
|
|
|
|
# CB-600: the script computes its own log path from where it sits on disk (REPO, above); the
|
|
# plist hard-codes an absolute StandardOutPath. Nothing forced the two to agree — if this script
|
|
# were ever run from a checkout other than the one the loaded plist names, launchd would start and
|
|
# log the daemon correctly, while every check below (the fresh "fleetd listening" line, the
|
|
# ERROR-count scan) would read a different, empty or stale file and the script would report a
|
|
# clean restart while the daemon crash-loops. Pure and side-effect-free besides `die`/`ok` — reads
|
|
# the two paths, resolves them, compares — so it never touches launchd or the daemon and can be
|
|
# exercised by sourcing this script (see the SOURCED guard below) without installing the agent.
|
|
check_log_path_matches_plist() {
|
|
local script_out="$1" plist_path="$2"
|
|
local plist_out resolved_out resolved_plist_out
|
|
# Checked by exit status, not by emptiness: on a missing file/key PlistBuddy exits nonzero but
|
|
# still writes a message ("File Doesn't Exist, Will Create: ...") that command substitution
|
|
# would happily capture as if it were the real value — testing only `-z` missed that case.
|
|
if ! plist_out="$(/usr/libexec/PlistBuddy -c 'Print :StandardOutPath' "$plist_path" 2>/dev/null)" \
|
|
|| [ -z "$plist_out" ]; then
|
|
die "launchd agent is loaded but PlistBuddy could not read StandardOutPath from
|
|
$plist_path
|
|
— cannot verify the daemon logs where this script is about to look. Fix the plist before
|
|
redeploying supervised."
|
|
fi
|
|
resolved_out="$(cd "$(dirname "$script_out")" 2>/dev/null && pwd -P)/$(basename "$script_out")" || true
|
|
resolved_plist_out="$(cd "$(dirname "$plist_out")" 2>/dev/null && pwd -P)/$(basename "$plist_out")" || true
|
|
if [ -z "$resolved_out" ] || [ -z "$resolved_plist_out" ] || [ "$resolved_out" != "$resolved_plist_out" ]; then
|
|
die "log path mismatch — this script reads
|
|
$script_out (resolved: ${resolved_out:-<directory does not exist>})
|
|
but the loaded plist's StandardOutPath is
|
|
$plist_out (resolved: ${resolved_plist_out:-<directory does not exist>})
|
|
Under supervision the daemon writes to the PLIST's path, not necessarily this script's — every
|
|
post-restart check below (the fresh 'fleetd listening' line, the ERROR-count scan) would read
|
|
the wrong file and could report a clean restart while the daemon crash-loops. Fix the mismatch
|
|
(move this checkout to match the plist, or edit the plist's StandardOutPath/StandardErrorPath)
|
|
before redeploying supervised."
|
|
fi
|
|
ok "log path check: script and plist agree ($resolved_out)"
|
|
}
|
|
|
|
# fleetd #552: the post-restart fresh-log capture, pulled out of the main flow so it is testable by
|
|
# sourcing (the same reason systemd_installed/systemd_loaded above guard their OWN mktemp inline
|
|
# instead of leaving it bare) even though its only caller sits below the SOURCED guard. By the time
|
|
# the caller reaches this, the daemon has already been stopped, the jar swapped, and the new daemon
|
|
# started — fleetd #552's whole point is that a `mktemp` failure here must never undo that success
|
|
# by aborting the script, the way the old unguarded `FRESH_LOG="$(mktemp ...)"` assignment did.
|
|
#
|
|
# So this function never dies, and — like detect_supervisor above — never reports failure through a
|
|
# global either: it is meant to be called as `VAR="$(capture_fresh_log_region ...)"`, which runs it
|
|
# in a subshell, and any global it set there would die with that subshell. Failure crosses back the
|
|
# only way that survives: a non-zero exit and empty stdout. The caller decides what a failure means
|
|
# (warn, and leave FRESH_LOG at its already-declared "" sentinel) and owns the EXIT trap, because
|
|
# that trap has to outlive this function's own return — the file must survive until every reader
|
|
# below (classify_amqp_connection_errors, report_shutdown_drain, the result section) has looked.
|
|
capture_fresh_log_region() {
|
|
local out_file="$1" restart_mark="$2" log_file
|
|
if ! log_file="$(mktemp -t fleetd-fresh-log.XXXXXX)"; then
|
|
return 1
|
|
fi
|
|
tail -n "+$((restart_mark + 1))" "$out_file" > "$log_file" 2>/dev/null || true
|
|
printf '%s' "$log_file"
|
|
}
|
|
|
|
# Classify ERROR lines in one fresh log region. AMQP failure messages now include the connection
|
|
# name, so a recovery can clear only errors for its own connection. A candidate with neither name
|
|
# remains unexplained: it must never be quieted by a recovery on the other connection.
|
|
#
|
|
# fleetd #552: log_file can now arrive as "" — capture_fresh_log_region's own sentinel for "the
|
|
# post-restart region could never be captured". That is a DIFFERENT fact from "captured a region
|
|
# with nothing in it", which is exactly what every count below staying at its initialized 0 already
|
|
# means; conflating the two would make "no errors found" and "could not look for errors" print
|
|
# identically; the exact defect class this ticket exists to fix. REDEPLOY_AMQP_CHECK_SKIPPED carries
|
|
# the distinction to the result section below, and this reader says so in its own output too.
|
|
classify_amqp_connection_errors() {
|
|
local log_file="$1" line pending_inbox=0 pending_lead_mailbox=0
|
|
REDEPLOY_ERROR_COUNT=0
|
|
REDEPLOY_RECOVERED_AMQP_ERRORS=0
|
|
REDEPLOY_UNEXPLAINED_ERRORS=0
|
|
REDEPLOY_AMQP_CHECK_SKIPPED=0
|
|
|
|
if [ -z "$log_file" ]; then
|
|
REDEPLOY_AMQP_CHECK_SKIPPED=1
|
|
warn "cannot check for AMQP/ERROR lines since restart — the post-restart log region could not be captured"
|
|
return 0
|
|
fi
|
|
|
|
while IFS= read -r line || [ -n "$line" ]; do
|
|
case "$line" in
|
|
*' ERROR '*|*' SEVERE '*)
|
|
REDEPLOY_ERROR_COUNT=$((REDEPLOY_ERROR_COUNT + 1))
|
|
case "$line" in
|
|
*'AMQP connection fleetd-reply-inbox: An unexpected connection driver error occurred'*|*'AMQP connection fleetd-reply-inbox: Caught an exception during connection recovery!'*)
|
|
pending_inbox=$((pending_inbox + 1))
|
|
;;
|
|
*'AMQP connection fleetd-lead-mailbox: An unexpected connection driver error occurred'*|*'AMQP connection fleetd-lead-mailbox: Caught an exception during connection recovery!'*)
|
|
pending_lead_mailbox=$((pending_lead_mailbox + 1))
|
|
;;
|
|
*'AMQP connection'*'An unexpected connection driver error occurred'*|*'AMQP connection'*'Caught an exception during connection recovery!'*)
|
|
REDEPLOY_UNEXPLAINED_ERRORS=$((REDEPLOY_UNEXPLAINED_ERRORS + 1))
|
|
;;
|
|
*) REDEPLOY_UNEXPLAINED_ERRORS=$((REDEPLOY_UNEXPLAINED_ERRORS + 1)) ;;
|
|
esac
|
|
;;
|
|
*'AMQP connection recovered; cleared held replies for fresh redelivery'*)
|
|
if [ "$pending_inbox" -gt 0 ]; then
|
|
pending_inbox=$((pending_inbox - 1))
|
|
REDEPLOY_RECOVERED_AMQP_ERRORS=$((REDEPLOY_RECOVERED_AMQP_ERRORS + 1))
|
|
fi
|
|
;;
|
|
*'AMQP lead mailbox connection recovered; cleared held messages for fresh redelivery'*)
|
|
if [ "$pending_lead_mailbox" -gt 0 ]; then
|
|
pending_lead_mailbox=$((pending_lead_mailbox - 1))
|
|
REDEPLOY_RECOVERED_AMQP_ERRORS=$((REDEPLOY_RECOVERED_AMQP_ERRORS + 1))
|
|
fi
|
|
;;
|
|
esac
|
|
done < "$log_file"
|
|
|
|
REDEPLOY_UNEXPLAINED_ERRORS=$((REDEPLOY_UNEXPLAINED_ERRORS + pending_inbox + pending_lead_mailbox))
|
|
}
|
|
|
|
# fleetd #512 part 2 — the negative check. #493's failure (an uncaught exception in a shutdown
|
|
# thread) never passes through the logger: the JVM's default uncaught-exception handler prints
|
|
# straight to stderr, so the line never carries a level, so classify_amqp_connection_errors's
|
|
# ERROR/SEVERE token scan is structurally blind to it — measured on two hosts, including one where
|
|
# even a syslog PRIORITY filter is blind to it too (fd 1 and fd 2 collapse to one socket there, so
|
|
# every uncaught-exception line lands at priority 6/info). The fix is to grep the shape instead of
|
|
# the level: `Exception in thread` at the start of a line (the handler's own banner) or
|
|
# `NoClassDefFoundError` anywhere in it (the one real instance seen so far, but not the only shape
|
|
# this could take). Kept as its own function, never folded into classify_amqp_connection_errors —
|
|
# this is not an AMQP concern, and the two must stay independently readable and independently
|
|
# testable.
|
|
#
|
|
# Sets REDEPLOY_UNCAUGHT_EXCEPTION_COUNT (lines matched) and REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE
|
|
# (the first matching line, "" if none) so a caller can report both a count and a concrete quote
|
|
# without re-reading the file. Pure: reads $1, sets globals, no side effects.
|
|
scan_uncaught_exceptions() {
|
|
local log_file="$1" line
|
|
REDEPLOY_UNCAUGHT_EXCEPTION_COUNT=0
|
|
REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE=""
|
|
while IFS= read -r line || [ -n "$line" ]; do
|
|
case "$line" in
|
|
'Exception in thread'*|*NoClassDefFoundError*)
|
|
REDEPLOY_UNCAUGHT_EXCEPTION_COUNT=$((REDEPLOY_UNCAUGHT_EXCEPTION_COUNT + 1))
|
|
[ -n "$REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE" ] || REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE="$line"
|
|
;;
|
|
esac
|
|
done < "$log_file"
|
|
}
|
|
|
|
# fleetd #512 part 2 — the positive check. fleetd #522 added a `log.info` at the very end of
|
|
# SessionManager.drainAll's normal path (never in a `finally` — see the ticket discussion for why
|
|
# that distinction matters): "drain complete: released=N abandoned=M (still BUSY at the shutdown
|
|
# deadline)", printed once, on every successful drain, including the all-zero case. A drain that
|
|
# dies partway through never reaches that statement, so the line's ABSENCE is a real signal — unlike
|
|
# the ERROR-count check above, this one does not depend on the failure happening to throw.
|
|
#
|
|
# Sets REDEPLOY_DRAIN_COMPLETE_LINE to the matching line (last one, though drainAll runs at most
|
|
# once per shutdown so there should never be more than one) or "" if absent. Pure, same shape as
|
|
# scan_uncaught_exceptions above.
|
|
find_drain_complete_line() {
|
|
local log_file="$1"
|
|
REDEPLOY_DRAIN_COMPLETE_LINE="$(grep -F 'drain complete: released=' "$log_file" | tail -1 || true)"
|
|
}
|
|
|
|
# fleetd #512 part 2 — THE TRAP, and the reason this is one function instead of two independent
|
|
# checks the caller ORs together. Absence of the drain-complete line has TWO causes that need
|
|
# OPPOSITE handling, and a naive "line absent -> the drain died" reading collapses them exactly the
|
|
# way this whole ticket exists to stop: the line is emitted by the daemon being STOPPED, which is
|
|
# running the OLD jar. Until a redeploy has landed fleetd #522 once, every previous daemon predates
|
|
# the line and cannot emit it no matter how cleanly it drained — so on the very first redeploy after
|
|
# #522 merged, "absent" means "too old to know how", not "died". Only once BOTH signals — this
|
|
# line's absence AND scan_uncaught_exceptions' result — have been read together can the three real
|
|
# outcomes be told apart:
|
|
#
|
|
# complete -> the line is present: the drain finished. Name the counts it reported.
|
|
# died -> the line is absent AND an uncaught-exception shape was found: the drain died. Name
|
|
# what was found.
|
|
# unknown -> the line is absent AND no exception shape either: cannot tell. Say so, and say why
|
|
# (predates the line, or failed without throwing) — never worded as a pass or a
|
|
# failure, and never reassuring: "ok, no ERROR lines" one level up is the exact mistake
|
|
# this ticket exists to fix, and this outcome must not reproduce it.
|
|
#
|
|
# A fourth case, n/a, covers a cold start or a "loaded but wasn't running" restart: no previous
|
|
# daemon was actually stopped THIS run, so there is no shutdown window in $log_file to have an
|
|
# opinion about at all — scanning it anyway would read the NEW daemon's own startup lines and could
|
|
# misreport "cannot tell" on every clean cold start. had_previous_daemon carries that fact in from
|
|
# the caller (it already knows $OLD_PID) rather than this function re-deriving it from log content.
|
|
#
|
|
# Same shape as swap_if_built/refuse_drain_gate (fleetd #521/#528): the decision (which of the four
|
|
# outcomes applies) and the action (which ok/warn line to print, and setting REDEPLOY_DRAIN_STATE
|
|
# for the "result" section below to consult) live together in ONE function that the main flow calls
|
|
# unconditionally — there is no guard left in the main flow to remove, invert, or bypass
|
|
# independently of this function. Never calls die(): #512's own decision is to warn loudly and let
|
|
# the redeploy stand, because by the time this is detectable the new daemon is already up and
|
|
# healthy and failing here would give the operator nothing to do differently.
|
|
report_shutdown_drain() {
|
|
local log_file="$1" had_previous_daemon="$2"
|
|
REDEPLOY_UNCAUGHT_EXCEPTION_COUNT=0
|
|
REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE=""
|
|
REDEPLOY_DRAIN_COMPLETE_LINE=""
|
|
|
|
if [ "$had_previous_daemon" != 1 ]; then
|
|
REDEPLOY_DRAIN_STATE="n/a"
|
|
ok "no previous daemon was running before this restart — nothing to check for a died shutdown drain"
|
|
return 0
|
|
fi
|
|
|
|
# fleetd #552: checked AFTER the n/a case above, not before it — a cold start has nothing to check
|
|
# regardless of whether the log region could be captured, and that fact must not be overridden by
|
|
# an unrelated mktemp failure. "skipped" is a fifth state on REDEPLOY_DRAIN_STATE, distinct from
|
|
# "unknown" (the log WAS read, and neither a drain-complete line nor an exception shape was found
|
|
# in it): here there was no log to read at all, and that is a different fact that must say so in
|
|
# its own output, never silently reported as "unknown" or, worse, as "complete"/"n/a".
|
|
if [ -z "$log_file" ]; then
|
|
REDEPLOY_DRAIN_STATE="skipped"
|
|
warn "cannot tell whether the previous daemon's shutdown drain finished — the post-restart log"
|
|
warn "region could not be captured"
|
|
return 0
|
|
fi
|
|
|
|
find_drain_complete_line "$log_file"
|
|
scan_uncaught_exceptions "$log_file"
|
|
|
|
if [ -n "$REDEPLOY_DRAIN_COMPLETE_LINE" ]; then
|
|
REDEPLOY_DRAIN_STATE="complete"
|
|
ok "previous daemon's shutdown drain finished: $REDEPLOY_DRAIN_COMPLETE_LINE"
|
|
elif [ "$REDEPLOY_UNCAUGHT_EXCEPTION_COUNT" -gt 0 ]; then
|
|
REDEPLOY_DRAIN_STATE="died"
|
|
warn "previous daemon's shutdown drain DIED — no drain-complete line, and an uncaught exception"
|
|
warn "was found in its shutdown window ($REDEPLOY_UNCAUGHT_EXCEPTION_COUNT line(s)):"
|
|
warn " $REDEPLOY_UNCAUGHT_EXCEPTION_SAMPLE"
|
|
warn "Some sessions from the PREVIOUS daemon may not have been released."
|
|
else
|
|
REDEPLOY_DRAIN_STATE="unknown"
|
|
warn "cannot tell whether the previous daemon's shutdown drain finished — no drain-complete line"
|
|
warn "and no uncaught-exception shape either. This is NOT a pass and NOT a failure: it means"
|
|
warn "either that daemon predates fleetd #522's drain-complete log line, or its drain failed"
|
|
warn "without throwing (hung, or returned early)."
|
|
fi
|
|
}
|
|
|
|
# fleetd #517: extracted so the suite can call this decision directly, the same way #510 extracted
|
|
# wait_for_daemon_exit so its ordering became checkable. Before this, the only test of the drain-gate
|
|
# abort message was a grep of this script's own source for the wording — so mutating the `if` below
|
|
# to `if false` (making the branch unreachable) left every test green, because the wording was still
|
|
# sitting in the file. Pure: only decides which message applies and prints it, no side effects, so a
|
|
# test can call it directly with an in-memory staged path instead of driving the real drain-gate flow
|
|
# (which needs a live $OLD_PID and an interactive prompt neither test can supply).
|
|
#
|
|
# The four cases:
|
|
# build ran, staged jar present -> names the staged jar and how to finish or discard it
|
|
# build ran, staged jar absent -> "nothing changed" (nothing was staged this run either)
|
|
# --no-build, staged jar present -> ALSO "nothing changed", deliberately: --no-build itself builds
|
|
# and stages nothing (see require_no_build_jar above), so a staged jar found here is a leftover
|
|
# from an earlier, unrelated run. THIS run truly changed nothing, and the next DO_BUILD=1 run
|
|
# wipes that leftover before it builds (`rm -f "$JAR_STAGED"` in the build section above) — so
|
|
# there is nothing here for the operator to lose track of.
|
|
# --no-build, staged jar absent -> "nothing changed"
|
|
drain_gate_refusal() {
|
|
local do_build="$1" staged_path="$2"
|
|
if [ "$do_build" = 1 ] && [ -f "$staged_path" ]; then
|
|
printf 'aborted — the running daemon was NOT touched, but the freshly built jar is sitting at
|
|
%s, not yet swapped into %s. Rerun WITHOUT --no-build to finish the restart —
|
|
the freshly built jar is no longer at the live path that --no-build requires — or
|
|
remove %s by hand if you want to discard this build.' "$staged_path" "$JAR" "$staged_path"
|
|
else
|
|
printf 'aborted — nothing changed'
|
|
fi
|
|
}
|
|
|
|
# fleetd #528 — drain_gate_refusal above is well tested (four cases, all direct), but nothing made
|
|
# the MAIN FLOW's abort actually consult it. Before this, the main flow read
|
|
# `die "$(drain_gate_refusal "$DO_BUILD" "$JAR_STAGED")"` directly, and mutating that one line to a
|
|
# flat `die "aborted — nothing changed"` left the whole suite at exit 0 with zero FAIL lines and
|
|
# byte-identical output to a clean run — every one of drain_gate_refusal's own tests still passed,
|
|
# because they call the predicate directly and never touch this call site. That silently reinstated
|
|
# the exact defect #517 was filed to fix. Same shape as #521/#526's should_swap/swap_if_built: a
|
|
# predicate alone is not enough, because a test proving the predicate is right cannot also prove the
|
|
# main flow consults it. So the decision (drain_gate_refusal) and the action (die) now live together
|
|
# in ONE function, and the main flow calls it unconditionally instead of building the die() call
|
|
# itself — there is no guard left in the main flow to remove, invert, or bypass independently of this
|
|
# function. drain_gate_refusal stays separate and separately tested because the message-selection
|
|
# logic is worth naming and testing on its own; refuse_drain_gate is the only thing that ever dies.
|
|
#
|
|
# What the behavioural tests above still cannot pin on their own: deleting the call to this function
|
|
# from the main flow altogether — they call refuse_drain_gate directly, never through the main flow,
|
|
# because sourcing stops before the main flow ever runs (see the SOURCED guard below). That gap is
|
|
# closed the same way swap_if_built's is: test_refuse_drain_gate_call_site_present greps this script
|
|
# for the real invocation, the same shape test_swap_ordered_after_wait_and_before_start already uses
|
|
# for the swap call. Deliberately NOT written out here as a literal quoted string, so this comment
|
|
# itself can never become a second match for that test's needle.
|
|
refuse_drain_gate() {
|
|
local do_build="$1" staged_path="$2"
|
|
die "$(drain_gate_refusal "$do_build" "$staged_path")"
|
|
}
|
|
|
|
# fleetd #555 — the main flow itself was almost entirely untestable: sourcing this file (the SOURCED
|
|
# guard below) stops before a single line of the report/build/drain/stop/swap/start/verify sequence
|
|
# ever runs, and every decision in that sequence used to live as a bare `if`/`case` written directly
|
|
# into the main flow rather than inside a function. Three tests reached that flow at all
|
|
# (test_swap_ordered_after_wait_and_before_start, test_refuse_drain_gate_call_site_present,
|
|
# test_report_shutdown_drain_call_site_present), and all three do it by grepping this file's own
|
|
# source for a call site — which proves the call site exists, never that the guard around it still
|
|
# reaches it. Every function below follows the swap_if_built/refuse_drain_gate shape those two
|
|
# tickets (#521/#528) already established: the decision (a small, separately-tested predicate) and
|
|
# the action it gates live together in ONE function, and the main flow calls that function
|
|
# unconditionally — so there is no bare guard left in the main flow for a future edit to invert
|
|
# silently. See test_no_untested_main_flow_conditionals in test-redeploy-fleetd.sh for the structural
|
|
# guard that keeps a ninth bare conditional from arriving the same way these eight did.
|
|
|
|
# Ticket item 1 — `--check` must stay genuinely read-only. Inverting the guard used to mean --check
|
|
# performs a real redeploy; now the guard is should_stop_for_check, and stop_if_check_only is the
|
|
# only thing the main flow calls.
|
|
should_stop_for_check() {
|
|
local check_only="$1"
|
|
[ "$check_only" = 1 ]
|
|
}
|
|
|
|
stop_if_check_only() {
|
|
local check_only="$1"
|
|
should_stop_for_check "$check_only" || return 0
|
|
say "--check: nothing changed"
|
|
exit 0
|
|
}
|
|
|
|
# Ticket items 2 and 3 — the drain-gate entry (`-n "$OLD_PID" && "$ASSUME_YES" = 0`) and the reply
|
|
# comparison (`"$reply" != "yes"`). drain_gate_required and drain_confirmed are the two predicates;
|
|
# run_drain_gate is the only thing the main flow calls, and it is the one place that reads $reply at
|
|
# all, so a test can drive it end-to-end over stdin.
|
|
drain_gate_required() {
|
|
local old_pid="$1" assume_yes="$2"
|
|
[ -n "$old_pid" ] && [ "$assume_yes" = 0 ]
|
|
}
|
|
|
|
drain_confirmed() {
|
|
[ "$1" = "yes" ]
|
|
}
|
|
|
|
run_drain_gate() {
|
|
local old_pid="$1" assume_yes="$2" do_build="$3" staged_path="$4" reply
|
|
drain_gate_required "$old_pid" "$assume_yes" || return 0
|
|
say "drain check"
|
|
echo " A restart drops every in-flight ticket and rendezvous. A member's report"
|
|
echo " is NOT recoverable once its ticket is gone."
|
|
echo
|
|
echo " Confirm with fleet_list that no members are live, and fleet_poll anything"
|
|
echo " you still want, BEFORE continuing."
|
|
echo
|
|
read -r -p " Fleet drained? type yes to restart: " reply
|
|
drain_confirmed "$reply" || refuse_drain_gate "$do_build" "$staged_path"
|
|
}
|
|
|
|
# Ticket item 4 — the report-state dispatch on $SUPERVISOR_KIND. Inverting this used to report the
|
|
# wrong supervisor and, for the launchd arm specifically, skip check_log_path_matches_plist — whose
|
|
# own comment already says every check after it is worthless if it never runs.
|
|
report_supervisor_state() {
|
|
local kind="$1"
|
|
SUPERVISED=0
|
|
case "$kind" in
|
|
launchd)
|
|
SUPERVISED=1
|
|
ok "launchd agent loaded ($LAUNCHD_LABEL) — launchd supervises this daemon"
|
|
check_log_path_matches_plist "$OUT" "$LAUNCHD_PLIST"
|
|
;;
|
|
systemd)
|
|
SUPERVISED=1
|
|
ok "systemd --user unit active ($SYSTEMD_UNIT) — systemd supervises this daemon"
|
|
;;
|
|
none)
|
|
warn "no supervisor loaded — this script is the only thing that will restart the daemon."
|
|
;;
|
|
*)
|
|
warn "unrecognised supervisor kind: '$kind' — detect_supervisor returned a value this block does not know; continuing to report the rest of the state."
|
|
;;
|
|
esac
|
|
}
|
|
|
|
# Ticket item 5 — the stop dispatch on $SUPERVISOR_KIND. Inverting this kills a supervised daemon
|
|
# with a raw `kill` instead of `launchctl unload`/`systemctl --user stop`, so the supervisor revives
|
|
# the OLD jar — the exact CB-594/#492 failure both branches exist to prevent. Each mechanism is its
|
|
# own thin, overridable function (same seam launchd_installed/launchd_loaded already use) so a test
|
|
# can prove dispatch_stop picks the right one without ever calling real launchctl/systemctl/kill.
|
|
stop_via_launchd() {
|
|
echo " supervision is ON (launchd): using 'launchctl unload' (not kill) so launchd's own"
|
|
echo " KeepAlive cannot restart the OLD jar out from under this script — see the CB-594"
|
|
echo " comment above."
|
|
launchctl unload -w "$LAUNCHD_PLIST" \
|
|
|| die "launchctl unload failed — the daemon may still be under supervision; investigate before retrying"
|
|
}
|
|
|
|
stop_via_systemd() {
|
|
echo " supervision is ON (systemd --user): using 'systemctl --user stop' (not kill) so"
|
|
echo " systemd's own Restart=on-failure cannot restart the OLD jar out from under this"
|
|
echo " script — see the fleetd #492 comment above."
|
|
systemctl --user stop "$SYSTEMD_UNIT" \
|
|
|| die "'systemctl --user stop $SYSTEMD_UNIT' failed — the daemon may still be under supervision; investigate before retrying"
|
|
}
|
|
|
|
stop_via_kill() {
|
|
local pid="$1"
|
|
kill "$pid"
|
|
}
|
|
|
|
dispatch_stop() {
|
|
local kind="$1" pid="$2"
|
|
case "$kind" in
|
|
launchd) stop_via_launchd ;;
|
|
systemd) stop_via_systemd ;;
|
|
none) stop_via_kill "$pid" ;;
|
|
*)
|
|
die "detect_supervisor returned an unrecognized value '$kind' at the stop step —
|
|
refusing to guess how to stop a daemon under an unknown supervisor. The daemon was NOT
|
|
stopped." ;;
|
|
esac
|
|
}
|
|
|
|
# Ticket item 6 — the start dispatch on $SUPERVISOR_KIND. Inverting this starts the daemon by the
|
|
# wrong mechanism; for launchd the branch also carries the retry that keeps a failed load from
|
|
# leaving the agent stopped-and-disabled. Same shape as the stop dispatch above.
|
|
start_via_launchd() {
|
|
echo " supervision is ON (launchd): using 'launchctl load' so launchd starts and keeps"
|
|
echo " supervising this process, instead of a manual nohup that launchd would know nothing"
|
|
echo " about."
|
|
# CB-600: 'launchctl unload -w' above already persisted Disabled=true for this label. A load -w
|
|
# that succeeds clears it; a load -w that FAILS leaves the agent both stopped and disabled — worse
|
|
# than before this script ran, because a later reboot or login will not bring it back either. One
|
|
# retry covers a transient race (e.g. launchd not yet fully done deregistering); if it still fails,
|
|
# die with the exact recovery command rather than a bare "failed".
|
|
if ! launchctl load -w "$LAUNCHD_PLIST" 2>/dev/null; then
|
|
warn "launchctl load failed on the first attempt — retrying once after a short pause"
|
|
sleep 2
|
|
launchctl load -w "$LAUNCHD_PLIST" || die "launchctl load failed twice.
|
|
The agent is now STOPPED and DISABLED — it will NOT come back on its own, not even after a
|
|
reboot or login, because 'launchctl unload -w' above persisted Disabled=true and load -w
|
|
never got the chance to clear it. Recover with:
|
|
launchctl load -w \"$LAUNCHD_PLIST\"
|
|
If that still fails, check 'launchctl list $LAUNCHD_LABEL', validate the plist with
|
|
'plutil -lint \"$LAUNCHD_PLIST\"', and check $OUT before assuming a retry will succeed."
|
|
fi
|
|
}
|
|
|
|
start_via_systemd() {
|
|
echo " supervision is ON (systemd --user): using 'systemctl --user start' so systemd starts"
|
|
echo " and keeps supervising this process, instead of a manual nohup it would know nothing"
|
|
echo " about."
|
|
systemctl --user start "$SYSTEMD_UNIT" || die "'systemctl --user start $SYSTEMD_UNIT' failed.
|
|
Check 'systemctl --user status $SYSTEMD_UNIT' and $OUT before assuming a retry will succeed."
|
|
}
|
|
|
|
start_via_none() {
|
|
# Absolute jar path so `ps` names which checkout is running.
|
|
( cd "$MODULE" && zsh -lc "nohup java -jar '$JAR' >> fleetd.out 2>&1 &" )
|
|
}
|
|
|
|
dispatch_start() {
|
|
local kind="$1"
|
|
case "$kind" in
|
|
launchd) start_via_launchd ;;
|
|
systemd) start_via_systemd ;;
|
|
none) start_via_none ;;
|
|
*)
|
|
die "detect_supervisor returned an unrecognized value '$kind' at the start step —
|
|
refusing to guess how to start a daemon under an unknown supervisor. The daemon was NOT
|
|
started." ;;
|
|
esac
|
|
}
|
|
|
|
# Ticket item 7 — the HEALTH_BODY poll loop and the `[ -z "$HEALTH_BODY" ]` branch. Inverting the
|
|
# latter (health_is_up) makes a dead daemon report /healthz 200 or a live one report failure.
|
|
# poll_health_body is pulled out too so the loop's own `if ...; then break; fi` is inside a function
|
|
# rather than sitting bare in the main flow.
|
|
poll_health_body() {
|
|
local url="$1" wait_s="$2" body="" _i
|
|
for _i in $(seq "$wait_s"); do
|
|
if body="$(curl -fsS --max-time 2 "$url" 2>/dev/null)"; then
|
|
printf '%s' "$body"
|
|
return 0
|
|
fi
|
|
sleep 1
|
|
done
|
|
return 1
|
|
}
|
|
|
|
health_is_up() {
|
|
[ -n "$1" ]
|
|
}
|
|
|
|
report_health() {
|
|
local body="$1" code="$2" out_file="$3" wait_s="$4"
|
|
if health_is_up "$body"; then
|
|
ok "/healthz 200 — $body"
|
|
warn "healthz green only proves herdr ANSWERS. If its protocol number changed, spawns can still"
|
|
warn "fail — prove a real spawn before trusting the fleet."
|
|
return 0
|
|
fi
|
|
# 503 still means the daemon is up — it means herdr is unreachable. Say which.
|
|
if [ "$code" = "503" ]; then
|
|
warn "/healthz answers 503 degraded — the daemon is up but herdr is unreachable."
|
|
warn "Spawns will fail. Check herdr before delegating anything."
|
|
curl -s --max-time 2 "$HEALTH" 2>/dev/null | head -3 || true
|
|
else
|
|
die "/healthz never answered within ${wait_s}s (last code: $code). Last lines of $out_file:
|
|
$(tail -30 "$out_file" 2>/dev/null)"
|
|
fi
|
|
}
|
|
|
|
# Ticket item 8 — `HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1`. Measured safe under `set -e`
|
|
# at both bash 3.2.57 and 5.x (see the header comment trap 9 discussion in the ticket) — not a `set
|
|
# -e` hazard, but still an untested computation feeding report_shutdown_drain's own four-way
|
|
# decision. compute_had_old_pid makes the mapping itself directly testable.
|
|
compute_had_old_pid() {
|
|
local old_pid="$1"
|
|
if [ -n "$old_pid" ]; then
|
|
echo 1
|
|
else
|
|
echo 0
|
|
fi
|
|
}
|
|
|
|
# CB-600: sourceable for testing. When this file is SOURCED (not executed) it stops here — nothing
|
|
# below runs — so a test harness can `source` it to call check_log_path_matches_plist (or the
|
|
# other pure helpers above) against a throwaway plist fixture without ever reaching the mutating
|
|
# flow (build/stop/start) or touching the real daemon or launchd. On a normal `./redeploy-fleetd.sh`
|
|
# invocation `(return 0 2>/dev/null)` fails (return is illegal at top level of an executed script),
|
|
# so this whole block is a no-op and every line below still runs exactly as before.
|
|
if (return 0 2>/dev/null); then
|
|
return 0
|
|
fi
|
|
|
|
# ---------------------------------------------------------------- report state
|
|
|
|
say "current state"
|
|
OLD_PID="$(running_pid)"
|
|
if [ -n "$OLD_PID" ]; then
|
|
ok "daemon running, pid $OLD_PID"
|
|
else
|
|
warn "no daemon running — this will be a cold start"
|
|
fi
|
|
ok "jar on disk: $(jar_id) ($([ -f "$JAR" ] && date -r "$JAR" '+%Y-%m-%d %H:%M:%S' || echo 'none'))"
|
|
ok "HEAD: $(git -C "$REPO" log --oneline -1)"
|
|
|
|
# CB-594 / fleetd #492: supervision state. Installed and loaded are different facts — a
|
|
# copied-but-never-loaded plist (or an unloaded systemd unit) supervises nothing, and a loaded
|
|
# label/unit with no file backing it is still what its supervisor will act on.
|
|
if launchd_installed; then
|
|
ok "launchd agent installed: $LAUNCHD_PLIST"
|
|
else
|
|
warn "launchd agent NOT installed."
|
|
fi
|
|
if systemd_installed; then
|
|
ok "systemd --user unit installed: $SYSTEMD_UNIT"
|
|
else
|
|
warn "systemd --user unit NOT installed ($SYSTEMD_UNIT)."
|
|
fi
|
|
|
|
# fleetd #492: decide which of the two (if either) actually supervises this daemon, and refuse
|
|
# outright — before touching anything — if that cannot be told apart (see require_drivable_
|
|
# supervisor above). --check reaches this same line, so a host with an undrivable supervisor is
|
|
# reported as a failure even in --check, without ever reaching the build/stop/start steps.
|
|
# fleetd #492 follow-up: detect_supervisor runs as $(...), so only the printed line survives —
|
|
# unpack kind and detail from it HERE, in this shell, before calling anything downstream. See the
|
|
# constraints comment above detect_supervisor for why this cannot be done any other way.
|
|
SUPERVISOR_RAW="$(detect_supervisor)"
|
|
SUPERVISOR_KIND="${SUPERVISOR_RAW%%"$SUPERVISOR_DETAIL_SEP"*}"
|
|
SUPERVISOR_UNCLEAR_DETAIL="${SUPERVISOR_RAW#*"$SUPERVISOR_DETAIL_SEP"}"
|
|
require_drivable_supervisor "$SUPERVISOR_KIND"
|
|
ok "supervisor detected: $SUPERVISOR_KIND"
|
|
# fleetd #555: the dispatch below used to be a bare `case` written directly here — see
|
|
# report_supervisor_state above for why it is now the only thing this line calls. (Unreachable
|
|
# today for the `*)` arm: require_drivable_supervisor above already died on "ambiguous"/"unclear"
|
|
# before this call runs. Guards the value nobody has invented yet.)
|
|
report_supervisor_state "$SUPERVISOR_KIND"
|
|
|
|
# The trap with no log line. Checked in a LOGIN shell, because that is how the daemon is started
|
|
# below. Never prints the value — only whether it resolved.
|
|
if zsh -lc '[ -n "${WORKER_GITEA_TOKEN:-}" ]' 2>/dev/null; then
|
|
ok "WORKER_GITEA_TOKEN resolves in a login shell"
|
|
else
|
|
warn "WORKER_GITEA_TOKEN is EMPTY in a login shell."
|
|
warn "The daemon will start fine and workers will silently fail to open PRs."
|
|
warn "Fix \${SHARED_ENV}/tools/secrets.sh before relying on worker checkpoints."
|
|
fi
|
|
|
|
# Same trap, second variable (CB-591). A profile's `tokenEnv:` is resolved from the DAEMON's own
|
|
# process environment by HerdrPeerLauncher.resolveEnv, so a token added to secrets.sh after the
|
|
# daemon started is simply absent. The launcher then injects an empty token and llm.ltms.dev answers
|
|
# 401 — long after the restart, and with nothing tying the two together.
|
|
if zsh -lc '[ -n "${AI_GATEWAY_TOKEN:-}" ]' 2>/dev/null; then
|
|
ok "AI_GATEWAY_TOKEN resolves in a login shell"
|
|
else
|
|
warn "AI_GATEWAY_TOKEN is EMPTY in a login shell."
|
|
warn "Any profile whose tokenEnv is AI_GATEWAY_TOKEN will get an empty token and 401 at the gateway."
|
|
warn "This only matters once a profile points at llm.ltms.dev — harmless before that."
|
|
fi
|
|
|
|
# Third variable, same trap (CB-635). broker.uriEnv names the env var holding the AMQP URI, so the
|
|
# password stays out of fleetd.yaml — but that moves the failure into the environment. If the
|
|
# variable is empty the daemon still starts: since #152 it warns and falls back to the in-memory
|
|
# reply inbox, so nothing crashes and replies simply stop surviving a restart. Only this check says
|
|
# so before the fact. Read the name out of the config so a renamed key cannot make the check lie.
|
|
BROKER_URI_ENV=$(sed -n 's/^[[:space:]]*uriEnv:[[:space:]]*\([A-Za-z_][A-Za-z0-9_]*\).*/\1/p' "$MODULE/fleetd.yaml" | head -1)
|
|
if [ -z "$BROKER_URI_ENV" ]; then
|
|
ok "no broker.uriEnv configured — reply inbox is in-memory by design"
|
|
elif zsh -lc "[ -n \"\${$BROKER_URI_ENV:-}\" ]" 2>/dev/null; then
|
|
ok "$BROKER_URI_ENV (broker.uriEnv) resolves in a login shell"
|
|
else
|
|
warn "$BROKER_URI_ENV (broker.uriEnv) is EMPTY in a login shell."
|
|
warn "The daemon will start and fall back to the IN-MEMORY reply inbox."
|
|
warn "Replies stop surviving a restart — a held report is lost, not delayed."
|
|
fi
|
|
|
|
# fleetd #555: should_stop_for_check/stop_if_check_only above — --check must stay genuinely
|
|
# read-only, and this is the only place that decides whether to stop here.
|
|
stop_if_check_only "$CHECK_ONLY"
|
|
|
|
# ---------------------------------------------------------------------- build
|
|
# Deliberately before the stop: a failed build must never leave the fleet down.
|
|
|
|
if [ "$DO_BUILD" = 1 ]; then
|
|
say "build"
|
|
# fleetd #493: wipe a leftover staged jar from a previous failed/interrupted run BEFORE doing
|
|
# anything else, so that run's leftovers can never be mistaken for this run's output.
|
|
rm -f "$JAR_STAGED"
|
|
BUILD_LOG="$(mktemp -t fleetd-build.XXXXXX)"
|
|
echo " log: $BUILD_LOG"
|
|
if ! mvn -f "$MODULE/pom.xml" clean install > "$BUILD_LOG" 2>&1; then
|
|
grep -E 'ERROR|BUILD FAILURE|Tests run:.*Failures: [1-9]|Tests run:.*Errors: [1-9]' "$BUILD_LOG" \
|
|
| head -20 || true
|
|
die "build failed — the running daemon was NOT touched. Full log: $BUILD_LOG"
|
|
fi
|
|
grep -E '^\[INFO\] Tests run:.*Failures' "$BUILD_LOG" | tail -1 | sed 's/^\[INFO\] / /' || true
|
|
ok "BUILD SUCCESS"
|
|
# fleetd #493: move the freshly built jar off the live path immediately — the running (OLD)
|
|
# daemon, if any, is still up at this point (build always runs before stop). From here until the
|
|
# swap step below (after the OLD daemon is confirmed gone), $JAR_STAGED is the only artefact this
|
|
# script treats as "the new jar" — $JAR itself is not touched again until the swap.
|
|
stage_built_jar
|
|
ok "jar now: $(jar_id "$JAR_STAGED")"
|
|
else
|
|
say "build skipped (--no-build)"
|
|
# fleetd #493: --no-build never builds or stages anything — it restarts whatever jar is already
|
|
# sitting at the live path from an earlier successful run. Same check, same message as before.
|
|
require_no_build_jar
|
|
fi
|
|
|
|
# ----------------------------------------------------------------- drain gate
|
|
|
|
# fleetd #555: run_drain_gate above is drain_gate_required + the prompt + drain_confirmed, called
|
|
# unconditionally — it returns immediately when the gate is not required, and composes/dies through
|
|
# refuse_drain_gate itself when the reply does not confirm. See #493/#517/#528 for why "nothing
|
|
# changed" would be a lie once a build has staged a jar.
|
|
run_drain_gate "$OLD_PID" "$ASSUME_YES" "$DO_BUILD" "$JAR_STAGED"
|
|
|
|
# ------------------------------------------------------------------ stop
|
|
#
|
|
# CB-594 / fleetd #492: when SUPERVISED, the supervisor owns the stop — never a raw `kill` here. A
|
|
# bare SIGTERM makes this JVM exit 143 even with its shutdown hook running to completion (verified
|
|
# separately: a throwaway Java process with an equivalent shutdown hook, sent SIGTERM from a login
|
|
# shell that could `wait` on it directly, reported exit code 143 every time — never 0). launchd's
|
|
# KeepAlive.SuccessfulExit=false and systemd's Restart=on-failure both treat any nonzero exit as a
|
|
# crash and restart the OLD jar, which would race this script's own restart of the NEW one.
|
|
# `launchctl unload` avoids that race by deregistering the job first, so no KeepAlive is left
|
|
# armed when the process actually stops. `systemctl --user stop` needs no such dance: unlike
|
|
# KeepAlive, systemd's Restart= does not fire on a deliberate stop, only on an unexpected exit of
|
|
# an active unit.
|
|
|
|
if [ -n "$OLD_PID" ]; then
|
|
say "stop"
|
|
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)" # verify a FRESH line appears later
|
|
# fleetd #555: dispatch_stop above — the launchd/systemd/none/* case lives there now, so this
|
|
# line is the only thing left in the main flow to get wrong.
|
|
dispatch_stop "$SUPERVISOR_KIND" "$OLD_PID"
|
|
if ! wait_for_daemon_exit "$STOP_WAIT"; then
|
|
die "pid $OLD_PID still alive after ${STOP_WAIT}s. Not escalating to kill -9 automatically:
|
|
the shutdown hook releases sessions and worktrees in order, and killing it hard can
|
|
leave worktrees and panes behind. Investigate, then kill -9 by hand if you accept that."
|
|
fi
|
|
ok "pid $OLD_PID exited"
|
|
elif [ "$SUPERVISOR_KIND" = "launchd" ]; then
|
|
# Loaded but not currently running (e.g. throttled after a crash loop). Unload it anyway so the
|
|
# start step below does a clean load, never a load stacked on an already-loaded label.
|
|
# fleetd #504: unload_launchd_if_loaded (above) tolerates a genuine already-unloaded answer but
|
|
# dies on a real `launchctl` failure — never a bare `|| true` that would print `ok` either way.
|
|
say "stop"
|
|
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"
|
|
unload_launchd_if_loaded
|
|
ok "launchd agent unloaded (was already not running)"
|
|
elif [ "$SUPERVISOR_KIND" = "systemd" ]; then
|
|
# Same case for systemd: the unit is known/active-capable but not currently running. `stop` on an
|
|
# already-stopped unit is a harmless no-op — kept for symmetry with the launchd branch above.
|
|
# fleetd #504: stop_systemd_if_loaded (above) tolerates that genuine no-op but dies on a real
|
|
# `systemctl` failure — never a bare `|| true` that would print `ok` either way.
|
|
say "stop"
|
|
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"
|
|
stop_systemd_if_loaded
|
|
ok "systemd --user unit stopped (was already not running)"
|
|
else
|
|
RESTART_MARK="$(wc -l < "$OUT" 2>/dev/null || echo 0)"
|
|
fi
|
|
|
|
# ------------------------------------------------------------------ swap
|
|
#
|
|
# fleetd #493: every branch above has now either confirmed the OLD daemon actually exited
|
|
# (wait_for_daemon_exit, above) or established there was never one running to begin with. Only
|
|
# NOW is it safe to put the freshly built jar at the path the NEXT `java -jar` (direct, or via
|
|
# launchd/systemd's ExecStart) will read from — this mv is the one and only write to $JAR anywhere
|
|
# in this script's mutating flow. If it fails, do not start: die() below exits before "start" runs.
|
|
swap_if_built "$DO_BUILD"
|
|
|
|
# ------------------------------------------------------------------ start
|
|
# Unsupervised: login shell (zsh -l) is what puts the secrets on the daemon's environment, and cwd
|
|
# must be fleetd/ because the daemon resolves fleetd.yaml, logs/ and target/ relative to it.
|
|
# Supervised (launchd): launchd does both — deploy/dev.ltms.fleetd.plist points ProgramArguments at
|
|
# scripts/fleetd-launchd-wrapper.sh (CB-594), which is what execs the login shell in launchd's
|
|
# place, and WorkingDirectory in the plist already pins fleetd/.
|
|
# Supervised (systemd --user): the unit does both too — measured on the second host, ExecStart is
|
|
# `/bin/zsh -lc "exec java -jar target/fleetd.jar fleetd.yaml"` (a login shell, same reason as
|
|
# above) and WorkingDirectory is already pinned to fleetd/.
|
|
|
|
say "start"
|
|
# fleetd #555: dispatch_start above — the launchd/systemd/none/* case (and the launchd retry) live
|
|
# there now, so this line is the only thing left in the main flow to get wrong.
|
|
dispatch_start "$SUPERVISOR_KIND"
|
|
|
|
for _ in $(seq 10); do
|
|
NEW_PID="$(running_pid)"
|
|
[ -n "$NEW_PID" ] && break
|
|
sleep 1
|
|
done
|
|
[ -n "${NEW_PID:-}" ] || die "no process appeared. Last lines of $OUT:
|
|
$(tail -20 "$OUT" 2>/dev/null)"
|
|
[ "$NEW_PID" != "${OLD_PID:-}" ] || die "pid unchanged ($NEW_PID) — the old daemon never died"
|
|
ok "started, pid $NEW_PID"
|
|
|
|
# ------------------------------------------------------------------ verify
|
|
|
|
say "verify"
|
|
|
|
# fleetd #555: poll_health_body/health_is_up/report_health above. HEALTH_CODE is only ever
|
|
# consulted by report_health when the body came back empty; `|| true` on both assignments is the
|
|
# same "an absent/failing command substitution must not kill the script under set -e" idiom the
|
|
# swap/drain helpers already rely on (see poll_health_body's own comment).
|
|
HEALTH_BODY="$(poll_health_body "$HEALTH" "$HEALTH_WAIT")" || true
|
|
HEALTH_CODE="000"
|
|
[ -n "$HEALTH_BODY" ] || HEALTH_CODE="$(curl -s -o /dev/null -w '%{http_code}' --max-time 2 "$HEALTH" 2>/dev/null || echo 000)"
|
|
report_health "$HEALTH_BODY" "$HEALTH_CODE" "$OUT" "$HEALTH_WAIT"
|
|
|
|
# A fresh listening line, strictly after the restart mark. An old daemon that never died would
|
|
# otherwise let an old line pass for a new one.
|
|
if tail -n "+$((RESTART_MARK + 1))" "$OUT" 2>/dev/null | grep -q 'fleetd listening'; then
|
|
ok "$(tail -n "+$((RESTART_MARK + 1))" "$OUT" | grep 'fleetd listening' | tail -1)"
|
|
else
|
|
warn "no fresh 'fleetd listening' line after the restart — check $OUT yourself"
|
|
fi
|
|
|
|
# Config keys the daemon accepted or deferred at boot. This is usually WHY you restarted.
|
|
say "config at boot"
|
|
tail -n "+$((RESTART_MARK + 1))" "$OUT" 2>/dev/null \
|
|
| grep -iE 'deferred|classification:|fleet health:|coverage' | tail -8 | sed 's/^/ /' \
|
|
|| echo " (nothing reported)"
|
|
|
|
# Errors since the restart, anchored to the marker so old noise cannot leak in. Keep the fresh
|
|
# region in a file because the classifier must preserve the order of errors and recoveries.
|
|
#
|
|
# fleetd #552: by this point the daemon has already been stopped, the jar swapped, and the new
|
|
# daemon started. The old unguarded `FRESH_LOG="$(mktemp ...)"` here aborted the whole script on any
|
|
# mktemp failure — AFTER all of that had already succeeded — so a caller read the resulting non-zero
|
|
# exit as "the redeploy failed" and the likely next action was to restart an already-correctly-
|
|
# restarted daemon. The mktemp itself now lives in capture_fresh_log_region (guarded the same way
|
|
# :293/:317/:344/:359 guard their own), and a failure there only warns: there is nothing left to
|
|
# protect by refusing after a successful restart. FRESH_LOG="" is declared and the trap installed
|
|
# BEFORE the call that may fail to fill it in, not after — an empty FRESH_LOG makes `rm -f ""` a
|
|
# harmless no-op, so there is no ordering hazard left between "the file might already exist" and
|
|
# "the trap that cleans it up".
|
|
FRESH_LOG=""
|
|
trap 'rm -f "$FRESH_LOG"' EXIT
|
|
if ! FRESH_LOG="$(capture_fresh_log_region "$OUT" "$RESTART_MARK")"; then
|
|
warn "could not create a temp file for the post-restart log region — the restart itself already"
|
|
warn "succeeded (pid $NEW_PID); only the post-restart ERROR/drain checks below are skipped."
|
|
warn "Check $OUT yourself for anything since the restart."
|
|
fi
|
|
classify_amqp_connection_errors "$FRESH_LOG"
|
|
|
|
# fleetd #512 part 2: the previous daemon's shutdown drain, checked in the same fresh-log region —
|
|
# see report_shutdown_drain above for the full decision (four outcomes, one of them a deliberate
|
|
# "cannot tell"). HAD_OLD_PID crosses in whether a previous daemon was actually stopped this run;
|
|
# see the function's own comment for why that matters.
|
|
say "previous daemon's shutdown drain"
|
|
# fleetd #555: compute_had_old_pid above — the mapping itself is now directly testable rather than
|
|
# an inline `&&` idiom sitting bare in the main flow.
|
|
HAD_OLD_PID="$(compute_had_old_pid "$OLD_PID")"
|
|
report_shutdown_drain "$FRESH_LOG" "$HAD_OLD_PID"
|
|
|
|
# fleetd #492: checked here, after healthz and the fresh-log check have both had time to run, so a
|
|
# supervisor that revives the OLD jar a few seconds late is caught too. Every check above (healthz
|
|
# 200, jar id, the fresh 'listening' line) is satisfied by EITHER daemon if two are alive — this is
|
|
# the only one that can tell.
|
|
assert_single_daemon "$(running_pid)"
|
|
|
|
say "result"
|
|
ok "pid $NEW_PID, jar $(jar_id)"
|
|
if [ "$REDEPLOY_AMQP_CHECK_SKIPPED" -eq 1 ]; then
|
|
# fleetd #552: the fourth reader of the fresh-log region. Without this branch REDEPLOY_ERROR_COUNT
|
|
# stays at its untouched 0 (classify_amqp_connection_errors never got a file to read) and
|
|
# REDEPLOY_DRAIN_STATE is "skipped" (never "complete"/"n/a"), so the branches below would print
|
|
# NOTHING about ERROR lines at all — a silence that reads exactly like the clean bill of health
|
|
# this ticket exists to prevent. Say the unknown result out loud instead.
|
|
warn "ERROR-line count since restart is UNKNOWN — the post-restart log region could not be captured (see above)"
|
|
elif [ "$REDEPLOY_ERROR_COUNT" -eq 0 ]; then
|
|
# fleetd #512 item 4: this line must not print when the shutdown-drain check above found the
|
|
# previous daemon's drain died, or could not tell — either would make "no ERROR lines" read as a
|
|
# clean bill of health it is not (an uncaught exception never carries an ERROR token to begin
|
|
# with, so this count alone cannot see that failure). "complete" and "n/a" are the only two
|
|
# outcomes report_shutdown_drain sets that mean nothing is wrong there.
|
|
if [ "$REDEPLOY_DRAIN_STATE" = "complete" ] || [ "$REDEPLOY_DRAIN_STATE" = "n/a" ]; then
|
|
ok "no ERROR lines since restart"
|
|
fi
|
|
elif [ "$REDEPLOY_UNEXPLAINED_ERRORS" -eq 0 ]; then
|
|
ok "$REDEPLOY_RECOVERED_AMQP_ERRORS AMQP connection reset ERROR lines recovered since restart"
|
|
else
|
|
warn "$REDEPLOY_ERROR_COUNT ERROR lines since restart:"
|
|
grep -E ' (ERROR|SEVERE) ' "$FRESH_LOG" | tail -5 | sed 's/^/ /'
|
|
fi
|
|
echo
|
|
echo " Next: call fleet_whoami and confirm it still answers 'primary'. A lead whose tab label"
|
|
echo " no longer matches fleet.leaders.*.tab is demoted to worker and refuses orchestration."
|
|
echo
|