fleetd #493: never build into the path a running daemon holds #510

2026-09-12T05:36:13+02:00

agent commented

2026-09-12 05:36:13 +02:00

Fixes fleetd #493.

The defect

redeploy-fleetd.sh ran mvn -f fleetd/pom.xml clean install directly into
fleetd/target/fleetd.jar — the exact path the still-running daemon was started
from — before ever stopping it. A JVM loads classes lazily, so a class the
daemon had not needed yet could be read from a jar that had already been
replaced or removed while the OLD daemon was still alive. The failure landed on
the shutdown drain (NoClassDefFoundError on an unloaded failure-path class,
process still exits 143, looks clean). See fleetd #413.

The fix

stage_built_jar() — right after a successful build, moves the jar Maven
produced (still target/fleetd.jar, since pom.xml is unchanged and out of
this ticket's scope) to a staging path, target/fleetd-new.jar. Dies (daemon
untouched — this runs before stop) if the build reported success but left no
jar, or if the move itself fails.
wait_for_daemon_exit(timeout) — the existing wait-for-exit loop, extracted
to its own function so the ordering guarantee below is checkable.
swap_staged_jar(staged, live) — the actual swap: a plain mv of the staged
jar onto the live path. Called only after wait_for_daemon_exit confirms the
OLD daemon (if any) has exited, right before the "start" section. If it
fails, the script dies and never starts a new daemon.
require_no_build_jar() — --no-build restarts whatever jar is already at
the live path; same check, same truthful message as before
(no jar at $JAR — run without --no-build).
A leftover fleetd-new.jar from an interrupted prior run is wiped before the
next build starts, so it can never poison the next run.
The build still runs before anything is stopped (unchanged), so a failed
build still never takes the fleet down.
jar_id() now takes an optional path argument (defaults to $JAR, the live
path) so the post-build "jar now:" line can report the staged jar's hash
without ever changing what a bare jar_id means — --check and the final
"pid …, jar …" line both still call it with no args, so neither can be fooled
by a leftover staged file.
Fixed as a direct consequence: the drain-gate abort message
("aborted — nothing changed") now names the staged jar when a build already
ran, since staging moves the freshly built jar off the live path before that
prompt runs — "nothing changed" was no longer quite true otherwise.

Testing

Baseline (measured on the unmodified tree, HEAD): bash scripts/test-redeploy-fleetd.sh
exits 0, invokes 23 test functions, and prints 3 literal FAIL: lines from
its own internal mutation-test cells (not real suite failures) plus a final
PASS: redeploy log classifier.

After this change: same command exits 0, invokes 33 test functions (10 new:
stage_built_jar x2, swap_staged_jar x3, require_no_build_jar x2,
wait_for_daemon_exit x2, plus one source-order test), and prints the same 3
literal FAIL: lines plus the same final PASS: line.

bash -n passes on both scripts/redeploy-fleetd.sh and
scripts/test-redeploy-fleetd.sh.

Mutation proof (5 guards, each: mutant applied with two different-string

greps, suite run, restored, shasum -a 256 byte-identical, green control run)

stage_built_jar disabled (return 0 inserted before its body) →
FAIL: stage_built_jar left the jar behind at the live path …/fleetd.jar,
exit 1.
swap_staged_jar disabled → FAIL: swap_staged_jar left the staged file behind at …/fleetd-new.jar, exit 1.
require_no_build_jar disabled → FAIL: require_no_build_jar accepted a missing jar, exit 1.
wait_for_daemon_exit disabled → FAIL: wait_for_daemon_exit returned before actually re-checking running_pid, exit 1.
Ordering violation — moved the swap block from right before start to
right before the drain gate (ahead of wait_for_daemon_exit) →
FAIL: swap_staged_jar (line 604) is not after wait_for_daemon_exit (line 665), exit 1.

Each mutant was restored from a saved copy and confirmed byte-identical via
shasum -a 256 before the corresponding green control run (exit 0, same 3
literal FAIL: lines, same final PASS: line).

What I could NOT verify

The real swap end-to-end, against a live daemon. I was explicitly
forbidden from running the script against the live fleetd (it's the channel
this session talks through), so I never exercised the actual
build → stage → stop → wait → swap → start sequence against a real process.
The unit tests exercise the pure functions and the source-order guarantee in
isolation only.
Whether Maven's shade-plugin package step (which still targets
target/fleetd.jar directly per pom.xml's finalName=fleetd, unchanged —
out of this ticket's scope) does an atomic rename-over-existing-file or an
in-place truncate-and-rewrite of the live inode. I did not run a real
mvn clean install to inspect this, and the brief's own text explicitly
accepts either outcome ("if Maven still writes target/fleetd.jar as part of
package, copy or move it to the staging name immediately").
No CI/build system was run for the wider fleetd module (out of scope: only
the two scripts/*.sh files changed; "Nothing under fleetd/src. No Java.").

Per the brief: the primary should run the real redeploy as the acceptance
test.

Out of scope, spotted but not investigated

None spotted beyond what's already noted above.

Fixes fleetd #493. ## The defect `redeploy-fleetd.sh` ran `mvn -f fleetd/pom.xml clean install` directly into `fleetd/target/fleetd.jar` — the exact path the still-running daemon was started from — before ever stopping it. A JVM loads classes lazily, so a class the daemon had not needed yet could be read from a jar that had already been replaced or removed while the OLD daemon was still alive. The failure landed on the shutdown drain (`NoClassDefFoundError` on an unloaded failure-path class, process still exits 143, looks clean). See fleetd #413. ## The fix - `stage_built_jar()` — right after a successful build, moves the jar Maven produced (still `target/fleetd.jar`, since `pom.xml` is unchanged and out of this ticket's scope) to a staging path, `target/fleetd-new.jar`. Dies (daemon untouched — this runs before stop) if the build reported success but left no jar, or if the move itself fails. - `wait_for_daemon_exit(timeout)` — the existing wait-for-exit loop, extracted to its own function so the ordering guarantee below is checkable. - `swap_staged_jar(staged, live)` — the actual swap: a plain `mv` of the staged jar onto the live path. Called only after `wait_for_daemon_exit` confirms the OLD daemon (if any) has exited, right before the "start" section. If it fails, the script dies and never starts a new daemon. - `require_no_build_jar()` — `--no-build` restarts whatever jar is already at the live path; same check, same truthful message as before (`no jar at $JAR — run without --no-build`). - A leftover `fleetd-new.jar` from an interrupted prior run is wiped before the next build starts, so it can never poison the next run. - The build still runs before anything is stopped (unchanged), so a failed build still never takes the fleet down. - `jar_id()` now takes an optional path argument (defaults to `$JAR`, the live path) so the post-build "jar now:" line can report the *staged* jar's hash without ever changing what a bare `jar_id` means — `--check` and the final "pid …, jar …" line both still call it with no args, so neither can be fooled by a leftover staged file. - Fixed as a direct consequence: the drain-gate abort message ("aborted — nothing changed") now names the staged jar when a build already ran, since staging moves the freshly built jar off the live path before that prompt runs — "nothing changed" was no longer quite true otherwise. ## Testing Baseline (measured on the unmodified tree, `HEAD`): `bash scripts/test-redeploy-fleetd.sh` exits `0`, invokes 23 test functions, and prints 3 literal `FAIL:` lines from its own internal mutation-test cells (not real suite failures) plus a final `PASS: redeploy log classifier`. After this change: same command exits `0`, invokes 33 test functions (10 new: `stage_built_jar` x2, `swap_staged_jar` x3, `require_no_build_jar` x2, `wait_for_daemon_exit` x2, plus one source-order test), and prints the same 3 literal `FAIL:` lines plus the same final `PASS:` line. `bash -n` passes on both `scripts/redeploy-fleetd.sh` and `scripts/test-redeploy-fleetd.sh`. ### Mutation proof (5 guards, each: mutant applied with two different-string greps, suite run, restored, `shasum -a 256` byte-identical, green control run) 1. `stage_built_jar` disabled (`return 0` inserted before its body) → `FAIL: stage_built_jar left the jar behind at the live path …/fleetd.jar`, exit 1. 2. `swap_staged_jar` disabled → `FAIL: swap_staged_jar left the staged file behind at …/fleetd-new.jar`, exit 1. 3. `require_no_build_jar` disabled → `FAIL: require_no_build_jar accepted a missing jar`, exit 1. 4. `wait_for_daemon_exit` disabled → `FAIL: wait_for_daemon_exit returned before actually re-checking running_pid`, exit 1. 5. Ordering violation — moved the swap block from right before `start` to right before the drain gate (ahead of `wait_for_daemon_exit`) → `FAIL: swap_staged_jar (line 604) is not after wait_for_daemon_exit (line 665)`, exit 1. Each mutant was restored from a saved copy and confirmed byte-identical via `shasum -a 256` before the corresponding green control run (exit 0, same 3 literal `FAIL:` lines, same final `PASS:` line). ## What I could NOT verify - **The real swap end-to-end, against a live daemon.** I was explicitly forbidden from running the script against the live fleetd (it's the channel this session talks through), so I never exercised the actual build → stage → stop → wait → swap → start sequence against a real process. The unit tests exercise the pure functions and the source-order guarantee in isolation only. - Whether Maven's shade-plugin `package` step (which still targets `target/fleetd.jar` directly per `pom.xml`'s `finalName=fleetd`, unchanged — out of this ticket's scope) does an atomic rename-over-existing-file or an in-place truncate-and-rewrite of the live inode. I did not run a real `mvn clean install` to inspect this, and the brief's own text explicitly accepts either outcome ("if Maven still writes target/fleetd.jar as part of package, copy or move it to the staging name immediately"). - No CI/build system was run for the wider `fleetd` module (out of scope: only the two `scripts/*.sh` files changed; "Nothing under fleetd/src. No Java."). Per the brief: the primary should run the real redeploy as the acceptance test. ## Out of scope, spotted but not investigated None spotted beyond what's already noted above.

agent added 1 commit 2026-09-12 05:36:13 +02:00

fleetd #493 : never build into the path a running daemon holds

CI / contract (pull_request) Successful in 1m30s

Details

CI / build (pull_request) Successful in 1m37s

Details

979adf82eb

redeploy-fleetd.sh's build step wrote straight into fleetd/target/fleetd.jar
via `mvn clean install` while the OLD daemon was still running from that
exact path. A JVM loads classes lazily, so a class the daemon had not
touched yet could be read from a jar already replaced or removed -- the
failure landed on the shutdown drain (NoClassDefFoundError, exit 143,
looks clean).

Stage the freshly built jar at target/fleetd-new.jar (stage_built_jar),
confirm the OLD pid has actually exited (wait_for_daemon_exit, extracted
from the existing wait loop), and only then swap it into the live path
(swap_staged_jar) -- strictly after the wait, strictly before start. A
failed swap dies without starting. --no-build and --check keep their
existing, truthful behavior (require_no_build_jar; jar_id still reads
the live path by default). A leftover staged jar from an interrupted
run is wiped before the next build. The build still runs before
anything is stopped, so a failed build still never takes the fleet down.

Also fixed: the drain-gate abort message ("aborted -- nothing changed")
now names the staged jar when one exists, since staging already moves
the freshly built jar off the live path before that prompt runs.

Adds unit tests for stage_built_jar, swap_staged_jar, require_no_build_jar,
wait_for_daemon_exit, and a source-order test proving swap sits after the
wait and before start (sourcing stops before the main flow ever runs, so
the ordering itself can only be checked by reading the script's own call
sites).

ltms merged commit aa4c0b84c3 into main

2026-09-12 05:44:17 +02:00

ltms referenced this pull request

2026-09-12 05:48:58 +02:00

#493 follow-up: the drain-gate abort message names a recovery that does not work, and jar_id()'s default is unpinned #511

ltms referenced this pull request

2026-09-12 05:53:26 +02:00

redeploy-fleetd.sh says "no ERROR lines since restart" while blind to the exact failure #493 is about #512

ltms referenced this pull request

2026-09-12 05:53:50 +02:00

redeploy-fleetd.sh builds into the LIVE jar path, so the shutdown drain can die on a class it never loaded (#413) #493

ltms referenced this pull request