fleetd #521: extract should_swap so the swap guard can't be silently disabled #526
Reference in New Issue
Block a user
Delete Branch "worker/521-swap-guard-unpinned-28e931-5"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
fleetd #521 — the swap step's guard was unpinned the same way #517's drain-gate abort branch was:
test_swap_ordered_after_wait_and_before_startonly checks source POSITIONS, so mutatingif [ "$DO_BUILD" = 1 ](the swap guard) toif falseleaves every line position unchanged and the whole suite green.Fix
Extracted the decision into
should_swap(do_build)(same shape as #510'swait_for_daemon_exitand #517'sdrain_gate_refusal), with the main flow now callingif should_swap "$DO_BUILD"; then .... Addedtest_should_swap_true_when_build_ranandtest_should_swap_false_when_build_skipped, which call the function directly.Acceptance checks (real output)
1.
bash scripts/test-redeploy-fleetd.shexits 0.Test functions: 42 defined, 42 invoked (
grep -cE '^test_[a-zA-Z_]+\(\) \{'andgrep -cE '^test_[a-zA-Z_]+$'— was 40/40 onmainatc71ac23, +2 for the new tests). Ran under both/bin/bashandenv bash; both exit 0 with only the three internal mutation-cell lines (Recovery mutation: FAIL: …,Shared-counter mutation: FAIL: …,Unattributable mutation: FAIL: …) — no line matching^FAIL:.2.
bash -non both scripts, both bash versions./bin/bash(3.2.57) andenv bash(5.3.9):bash -n scripts/redeploy-fleetd.shandbash -n scripts/test-redeploy-fleetd.shall exit 0.3. The reported mutation is killed.
main@c71ac23 first: mutating line 723'sif [ "$DO_BUILD" = 1 ]; then(the swap guard) toif false; thenleft the suite green (exit 0, no^FAIL:) — confirms the ticket's finding before touching anything.should_swap. I mutated its body ([ "$do_build" = 1 ]→false, i.e. "never swap regardless of $DO_BUILD" — the same observable effect as the original PoC) and reran the suite:FAIL: should_swap 1 (a build ran and staged a jar) must return true, exit 1.diffagainst the pre-mutation copy showed no difference, and a control run was green again (exit 0, only the three internal mutation-cell lines).if should_swap "$DO_BUILD"; then→if false; then, the literal same transformation as the ticket's original PoC, now one line down at :739) — the suite stayed green.test_should_swap_*only calls the function directly and never proves the main-flowifstill invokes it;test_swap_ordered_after_wait_and_before_startonly checks the positions of three other lines (wait_for_daemon_exit,swap_staged_jar,say "start"), none of which move under this mutation. This is the identical residual gap already present in #517's precedent (nothing proves the drain gate's ownif [ "$reply" != "yes" ]call site is reached, either) — I did not fix it, since the ticket's ask was specifically "extract the decision + a test per value," matching the existing precedent shape, and this residual risk is structural to the sourced-testing boundary (the main flow after theSOURCEDguard can never be driven end-to-end by this harness). Flagging it rather than silently living with it.4. Mutation-applied proof (two greps, different strings, single-quoted; plus re-read).
For the primary (function-level) mutation:
grep -n '^ false$' scripts/redeploy-fleetd.sh→194: falsegrep -n '\[ "\$do_build" = 1 \]' scripts/redeploy-fleetd.sh→ only matches line 490 (a different function's owndo_buildparameter, unrelated) — confirmed by direct re-read of lines 192-195 showingfalsein the body.sed -n '192,195p'confirmed directly.5. Pristine-hash checks.
Confirmed
scripts/redeploy-fleetd.shonmain@c71ac23 hashed to2cb83dc380c7226191d657c40fccdfc856904e40b2d03d0851fb6e522cee2f41(matches the ticket) before any edits. Every restore during mutation testing was verified withdiffagainst a saved pre-mutation copy (byte-identical each time).Part 2 — does anything later report a differing jar id?
Traced with the swap guard disabled:
stage_built_jar(unconditional, in the build step — unaffected by the swap-guard mutation) always moves the freshly built jar off$JARonto$JAR_STAGEDthe moment the build succeeds. So by the time the swap step is reached,$JAR(the live path) is already absent, regardless of the mutation.$JAR/$JAR_STAGEDunder a temp dir, no daemon touched): afterstage_built_jar,jar_id(bare) reportsabsent, not a stale hash. If the script somehow reached the finalok "pid $NEW_PID, jar $(jar_id)"line (:865 in the current file), it would printjar absent— which would be noticeable, being a different word rather than a stale-looking hash.java -jar <missing-file>fails immediately (Error: Unable to access jarfile ..., exit 1 — confirmed generically on this box against a nonexistent path, no fleetd daemon involved). So starting a daemon whose jar was moved away and never swapped back fails to produce a lasting process. The script's own gates catch this before the final report: either[ -n "${NEW_PID:-}" ] || die "no process appeared. Last lines of $OUT: ..."(unsupervised case, most likely outcome — java exits near-instantly), or, if a supervisor's restart loop happens to letpgrepcatch a transient pid,/healthz never answered within ${HEALTH_WAIT}safter up to 60s of polling (the health endpoint can never come up, since no jar is running).Answer: No — nothing downstream reports a misleading jar id, because the run cannot reach the final report line at all; it dies loudly first, at "no process appeared" or "/healthz never answered", both of which print the tail of the daemon's own log. This is a real, visible failure, just not the one the removed
ok "jar in place: ..."line would have given, and not the "silent success" outcome I could not rule out without checking. I did not run this against the live daemon (prohibited); this is based on reading the script's control flow, a generic (non-fleetd)java -jar <missing>test, and a sourced-script trace of the jar-file state — not a live redeploy.Part 3 — sweep for the "grep-only test" shape (not fixed)
Searched the whole test file for every test that inspects
redeploy-fleetd.sh's own source text/position (grep -n '\$src\|ROOT/scripts/redeploy-fleetd.sh"' scripts/test-redeploy-fleetd.sh, excluding thesourcere-imports). Found exactly two, both already known:test_swap_ordered_after_wait_and_before_start(position-only; now supplemented by the newshould_swaptests, kept per the ticket)test_drain_gate_abort_message_says_no_no_build(wording-only; already supplemented by #517/#520's directdrain_gate_refusaltests)No third "the only test is a grep of the script's own source" instance exists in the test file today.
That said, the same underlying weakness (a main-flow decision, past the
SOURCEDguard, that this harness cannot drive end-to-end) shows up more broadly as branches with no test of any kind — not even a grep. Listed for awareness, not fixed::611if [ "$CHECK_ONLY" = 1 ](skip mutating flow entirely) — no test.:648if [ -n "$OLD_PID" ] && [ "$ASSUME_YES" = 0 ](drain-gate's own entry condition — whether the prompt fires at all) — no test;drain_gate_refusal()'s message is tested directly, but nothing proves this outer gate, or theif [ "$reply" != "yes" ]diecall site inside it, is reached.:549case "$SUPERVISOR_KIND" in ...(report-state, display only) — no test.:681case "$SUPERVISOR_KIND" in ...(stop-step dispatch:launchctl unload/systemctl --user stop/kill) — no test of the dispatch itself.:756case "$SUPERVISOR_KIND" in ...(start-step dispatch:launchctl load/systemctl --user start/nohup java) — no test.:820if [ -z "$HEALTH_BODY" ](healthz body vs. 503 vs. die) — no test.:866if [ "$REDEPLOY_ERROR_COUNT" -eq 0 ](result-summary ok/warn) — no test (the counters it reads are tested; this display decision built on them is not).Not fixing any of these — reporting per the ticket's ask to name the shape, not chase every instance.
Build / checks
bash scripts/test-redeploy-fleetd.sh— exit 0, both bash versions, 42/42 test functions, no^FAIL:lines.bash -n— exit 0, both scripts, both bash versions.mvnbuild involved — this ticket only touches shell scripts.Files changed
scripts/redeploy-fleetd.sh— addedshould_swap(do_build), changed the swap guard call site to use it.scripts/test-redeploy-fleetd.sh— addedtest_should_swap_true_when_build_ranandtest_should_swap_false_when_build_skipped, registered both in the run list.Correction to my own merge message, which was unfair to the implementer.
I wrote that I measured the call-site gap. I did — but the implementer found it first and reported it, in this PR's body under "Caveat for review":
That is the whole finding, measured, correctly attributed to the ticket's scope rather than quietly absorbed, and it also names the same residual gap on #517's
drain_gate_refusalprecedent — which is now #528. I reproduced it independently without having read the body first, so I arrived at the same place by a second route, but the report was already there. Worker behaviour that should be copied: it ran a mutation its brief did not ask for, found its own fix incomplete, and said so.Two more parts of that report worth keeping out of the archive, because I did not re-derive them and they answer questions #521 asked:
Part 2 — does anything downstream report a misleading jar id? #521 said someone should establish this rather than assume. The implementer's answer: no, and not because the report is right, but because the run cannot reach the final report line.
stage_built_jaris unconditional and has already moved the jar off$JARby then, so with the swap disabledjava -jarfails immediately and the script dies atno process appearedor/healthz never answered within ${HEALTH_WAIT}s, both of which print the daemon log tail. So the mutant is a loud failure, not a silent success — which narrows #521's severity from what I filed. Explicitly not run against the live daemon: this is from reading the control flow, a generic non-fleetdjava -jar <missing>test, and a sourced-script trace of the jar-file state. I have not re-verified it.Part 3 — the sweep. Exactly two source-text-only tests exist, both already known (
test_swap_ordered_after_wait_and_before_start, position-only;test_drain_gate_abort_message_says_no_no_build, wording-only). No third instance. But the implementer then reported something more useful than the thing asked for: seven main-flow decisions past theSOURCEDguard with no test of any kind, not even a grep —:611if [ "$CHECK_ONLY" = 1 ]— skips the mutating flow entirely:648if [ -n "$OLD_PID" ] && [ "$ASSUME_YES" = 0 ]— whether the drain prompt fires at all, plus theif [ "$reply" != "yes" ]die inside it:549/:681/:756the threecase "$SUPERVISOR_KIND"blocks — report display, stop dispatch (launchctl unload/systemctl --user stop/kill), start dispatch:820if [ -z "$HEALTH_BODY" ]— healthz body vs 503 vs die:866if [ "$REDEPLOY_ERROR_COUNT" -eq 0 ]— the ok/warn summary built on counters that are testedLine numbers are pre-merge and will have shifted; re-measure before using them. The
:681stop dispatch is the one I would rank first — it chooses between three different ways to stop the daemon, and #492's supervisor-detection work sits directly upstream of it.Not filing these as a ticket yet. Recorded here so they are not lost, and referenced from #528.