b6b006c651d7e178282863da4d325b40fea1ba6f
45 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
856dfc6318 |
fleetd #603 review: close the untested fall-through path
PR review (comment 17358) found a real gap via mutation testing: replacing the "pid never found" warn with die "no process appeared" left the whole suite green, because neither existing test drove the case the fall-through exists for -- running_pid() never finds anything (as its own doc comment says it eventually will) while /healthz answers anyway. Adds test_await_daemon_started_pid_never_found_but_healthy_warns_and_survives: running_pid always empty, poll_health_body succeeds. Asserts DIED_CALLED=0, the warn line is emitted, and NEW_PID stays empty (the honest "could not establish this" answer, never a guessed pid). Verified both halves myself: reverting the warn to die "no process appeared" turns this one test red (FAIL: await_daemon_started must not die...); restoring it returns the suite to green. |
||
|
|
81c1d8e91c |
fleetd #603: share HEALTH_WAIT between the pid poll and the health check
The start step gave the new process its own short, fixed 10s budget before a hard die, while the health check right after it waits a full HEALTH_WAIT (60s) for the same daemon. Under launchd, launchctl load returns before the java process exists, and on a slow host that took longer than 10s -- so the script reported "no process appeared" on a deploy that had fully succeeded. wait_for_new_pid/await_daemon_started fold the pid poll and the health check into one decision: the pid poll now shares HEALTH_WAIT instead of its own shorter budget, and a miss there falls through to the health check (direct proof the daemon is up) instead of killing the run. A genuine failure still dies, and still prints the log tail. Adds behavioural tests for both acceptance criteria (slow start succeeds, genuine failure still fails and prints the log tail) in one suite run, plus unit tests for wait_for_new_pid and a call-site test for the new function. |
||
|
|
4b9ebda1b3 |
fleetd #593 CORRECTION 1: allowlist comm=java, not a denylist of shells
The round-1 fix excluded known shell names (sh/bash/zsh/dash/ksh) from running_pid()'s pgrep candidates. Two holes remained, both the same false-positive shape the ticket exists to remove: 1. A pid pgrep lists can exit before the following `ps -o comm=` lookup runs. On a gone pid, ps prints nothing, comm is empty, and an empty string matches no denied shell name -- so a dead pid was still counted. 2. The denylist only knows the shells someone thought to name. ssh, perl, python3, ruby, tail -- anything else carrying the pattern in its own argv -- was still counted alongside the real daemon. The ticket names ssh as a live route. Both close with one change: allowlist comm=java instead of denying shells. The daemon is always `java -jar target/fleetd.jar`, so its comm is always `java`; an empty comm (hole 1) is not `java` either, closing that hole for free. Answers the objection in the code comment: an allowlist can under-count if fleetd ever stops being launched by `java` (a native image, a renamed launcher). That's a false negative, the worse direction for a guard -- but it is not a new assumption: PATTERN='target/fleetd.jar' already assumes a jar run by java, and that pattern breaks before this allowlist would. Replaces the round-1 "real second process" test (which gave its exec -a standin an argv[0] holding the pattern, but not comm=java) with one that forces comm=java via `exec -a java sh -c '...'`. Adds two stubbed pgrep/ps tests pinning the two holes directly (a non-java, non-shell comm such as perl; an empty comm from an already-exited pid) -- deterministic on every platform, unlike a live-process fixture, and immune to the BSD vs Linux difference in how `comm` is derived from a fabricated process. Adds a stubbed positive backstop (comm=java is counted). Confirmed the regression is caught: reverted to the round-1 denylist, reran the suite, watched the new non-shell-comm test fail at `set -e`'s first failure, then isolated the exited-pid test separately and confirmed it also fails against the same broken code. Restored the fix and reran green. Branch merged with origin/main (3 commits: hunter role + CLAUDE.md addendum) before this commit; unrelated, no conflicts. |
||
|
|
42820fbe75 |
fleetd #593 (pid-count half): running_pid() no longer matches the caller
running_pid() was a bare `pgrep -f "$PATTERN"`, which matches ANY process whose full command line contains the pattern text -- including a shell that merely embeds it as literal text (a hand-typed investigation, an ssh-shaped `sh -c '...; ...'`, or a pipeline) rather than being the daemon. That self-match turns a working redeploy into a reported "racing supervisor" failure via assert_single_daemon. pgrep -c does not exist on BSD/macOS, so this can't be fixed by switching flags. running_pid() now keeps pgrep to find candidates (portable), then drops any candidate whose process name (comm) names a shell -- the daemon is always `java`, so a self-matching wrapper of this shape is always excluded while a genuine second daemon-shaped process still counts. assert_single_daemon's die message no longer hands the operator a bare `pgrep -f "$PATTERN"` as remediation -- that was exactly the self-matching invocation -- and now says in words that a pattern can match the caller. Adds three tests: a self-matching wrapper shell must be excluded, a real second daemon-shaped process must still be found, and the die message must not recommend the self-matching command. Verified the first test fails against the pre-fix implementation (confirmed the regression is caught). Leaves instance 1 (the fleetd.out log source, systemd-only) for a Linux host, per the ticket's scope split. |
||
|
|
8f80d267a0 |
fleetd #555 rework: catch function definitions after the SOURCED guard
Comment 17012 on #555 found a hole in test_no_untested_main_flow_conditionals: the guard's function-body detection treats anything inside a function as "fine, out of scope for this scan" — but a function DEFINED after the SOURCED guard line can never be reached by sourcing this script (sourcing stops before the main flow runs), so its body is untestable by construction while still reading to the guard as safely inside a function. mainflow_bare_conditionals now also emits a FUNC record for every function opened after the guard line (reusing the same open-brace detection already used for depth tracking), and test_no_untested_main_flow_conditionals treats any such record as a violation on its own, independent of what the function's body contains or whether the allowlist would otherwise excuse a bare conditional inside it. Proof (redeploy-fleetd.sh restored to 4ffacc5185807d39720a3484d85b922413806eb5347318265bd8897dfd61e8d9 after each): - CONTROL — a bare conditional appended to the main flow is still caught: EXIT=1, "found 1 untested main-flow if/elif/case line(s) ... line 1341: if [ "$MY_CONTROL_BARE" = 1 ]; then :; fi" - CANDIDATE — the same conditional wrapped in a function defined after the boundary, previously invisible (EXIT=0), is now caught: EXIT=1, "line 1341: function defined after the SOURCED guard (line 1038) — it cannot be sourced, so it cannot be tested: newfunc_below_the_boundary() {" Full suite re-run green on both bash 5.3.9 and /bin/bash 3.2.57 (macOS system bash): exit 0, 0 FAIL lines, reached the final PASS line, on both. No change to redeploy-fleetd.sh; the 8 lifted decisions, their mutation proofs, and the allowlist all stand as before. |
||
|
|
8d79d229ff |
fleetd #555: lift 8 main-flow decisions into tested predicate/dispatch functions
redeploy-fleetd.sh's main flow had 8 bare if/case decisions (CHECK_ONLY
short-circuit, drain-gate entry+confirm, supervisor report/stop/start
dispatch, health-poll decision, HAD_OLD_PID computation) that lived outside
any function, so the 67-test suite could not reach them and any one could be
silently inverted with the whole suite green.
Follows the existing swap_if_built/refuse_drain_gate pattern: each bare
guard becomes a small predicate or dispatch function (should_stop_for_check,
drain_gate_required/drain_confirmed/run_drain_gate, report_supervisor_state,
dispatch_stop, dispatch_start, health_is_up/report_health,
compute_had_old_pid), called unconditionally by the main flow so the
decision itself is unit-testable in isolation.
Adds a structural guard, test_no_untested_main_flow_conditionals, that scans
the main flow (everything after the SOURCED guard) for bare if/elif/case
lines outside any function body, tracking function boundaries via this
file's one consistent name() { / } convention. It fails on any new bare
conditional not covered by MAIN_FLOW_ALLOWED_CONDITIONALS, an explicit
exact-text allowlist of the report-only/display conditionals and the two
#504-family supervisor elif branches that stay out of scope for this
ticket. This is the "shape, not the eight sites" guard the ticket asked
for: a ninth bare decision fails immediately, naming its line.
Out of scope, not touched: #504 items 2/3/4 and #528 item 2 (same
untested-main-flow family) — the seam here generalizes to make them
testable too, but lifting them was left for their own tickets.
|
||
|
|
f188947750 |
fleetd #552: warn instead of aborting when the post-restart mktemp fails
By the time the fresh-log mktemp ran, the daemon had already been stopped, the jar swapped, and the new daemon started — an unguarded mktemp failure there aborted the whole script anyway, so a caller read the resulting non-zero exit as "the redeploy failed" and would restart an already-correctly-restarted daemon. Extract the mktemp into capture_fresh_log_region, guarded the same way unload_launchd_if_loaded/stop_systemd_if_loaded guard their own, but warn instead of die: there is nothing left to protect by refusing after a successful restart. The trap is now installed before the assignment it cleans up, using an FRESH_LOG="" sentinel readers can check. classify_amqp_connection_errors and report_shutdown_drain both gain a new state (REDEPLOY_AMQP_CHECK_SKIPPED / REDEPLOY_DRAIN_STATE=skipped) for an uncapturable log region, distinct from "captured a region with nothing in it" — and the result section gains a matching branch, so a skipped capture can never read as a clean bill of health. |
||
|
|
b8182c96c2 |
fleetd #550: pin hash256's algorithm against a literal SHA-256 test vector
test_jar_id_defaults_to_live_and_reports_explicit_path's reference hash is computed by calling hash256 itself (needed so it doesn't call the Linux-crashing bare shasum directly). That made subject and reference the same instrument: they agree no matter which algorithm hash256 actually runs, so a mutation swapping both of hash256's arms for the wrong algorithm was invisible to the suite. Adds test_hash256_computes_a_real_sha256, pinned against the published SHA-256 test vector for the 3-byte input "abc" (ba7816bf8f01...), written as a literal constant rather than computed by any hasher at test time. Verified the constant myself both ways (sha256sum and shasum -a 256) before writing it in. |
||
|
|
3da44eed63 |
fleetd #550: replace shasum with a portable hash256 helper, add a Linux CI job for the shell suite
jar_id() in redeploy-fleetd.sh called shasum directly, which does not exist on GNU coreutils Linux (Debian/Ubuntu/etc.) — there it silently reported an existing jar as "absent" with exit 0, because the missing command made `cut` succeed on empty input and pipefail's failure was then swallowed by the `|| echo "absent"` fallback. The shell test suite hit the same tool at test-redeploy-fleetd.sh:298-299 and died at exit 127 with zero FAIL lines printed — the same shape as a clean pass on the one channel anyone would check. Adds one hash256() helper (prefer sha256sum, fall back to shasum -a 256, same idiom already used in probe-member-credentials.sh) and points jar_id and the test suite's own reference hash at it. jar_id now has three distinct answers instead of two: absent, a hash, or "unhashable" when neither hasher is on PATH — "absent" is never used for a file that exists. Adds a CI job (shell-tests) that runs scripts/test-redeploy-fleetd.sh on ubuntu-latest, gated on the step's own exit code rather than a FAIL-line count, since a suite that dies before running is exactly what a green run also looks like by that count. New tests: test_jar_id_reports_unhashable_when_no_hasher_on_path (stubbed PATH with neither hasher) and test_no_unguarded_macos_only_hasher_calls (a shape check across every script under scripts/, not named lines — #545 already showed this idiom spreading from two sites to six). |
||
|
|
a476a14f1c |
fleetd #545: fix mktemp -t templates for GNU coreutils, split unclear-supervisor detail
Every mktemp -t template in redeploy-fleetd.sh lacked an X placeholder. BSD mktemp (macOS) tolerates that and appends its own suffix; GNU mktemp (every Linux distribution) refuses it and exits non-zero. All six sites now use .XXXXXX. detect_supervisor's 'unclear' detail used to cover two different facts with one message that always named 'systemctl exited non-zero and reported an error on stderr' — even when systemctl was never run, because mktemp failed first. The SYSTEMD_LOADED_ERRORED/SYSTEMD_INSTALLED_ERRORED flags now carry a third value (2 = the probe's own mktemp setup failed) alongside the existing 1 (systemctl ran and answered badly on stderr), and detect_supervisor gives each its own detail text. kind stays 'unclear' in both cases; require_drivable_supervisor is unchanged. Tests added to scripts/test-redeploy-fleetd.sh: - test_mktemp_dash_t_templates_have_x_placeholders: source-text check, fails if any mktemp -t template lacks an X. - test_detect_supervisor_systemd_probe_setup_failure_is_unclear: proves the SET-UP-FAILED detail when mktemp itself fails (systemctl never runs). - test_detect_supervisor_systemd_probe_error_is_unclear: extended with assertions that the PROBE-ANSWERED-WITH-STDERR detail is present and the SET-UP-FAILED wording is absent, so swapping the two messages fails a test in both directions. |
||
|
|
e4eb3dbed4 |
fleetd #504 item 1: stop swallowing real launchctl/systemctl failures on the 'loaded but not running' path
The two 'loaded but not currently running' branches in the stop step (launchd/systemd, reached when $OLD_PID is empty) ran 'launchctl unload'/'systemctl --user stop' with '2>/dev/null || true' and printed 'ok' unconditionally. That swallowed a real supervisor failure (e.g. launchd or the systemd user bus unreachable) exactly like a harmless already-stopped answer, and let the script proceed to start a new daemon believing nothing was loaded -- the two-daemons failure fleetd #492 exists to prevent. Adds unload_launchd_if_loaded/stop_systemd_if_loaded, applying systemd_loaded's own pattern (capture stderr separately; a non-zero exit WITH stderr is a real failure, a non-zero exit with empty stderr is a clean already-stopped answer) to the write side. The two call sites now use these functions instead of the bare '|| true'. Adds 5 tests: dies-on-real-failure and tolerates-clean-negative for each function, plus a source-text check that the main flow calls the new functions instead of the original bare '2>/dev/null || true'. All 5 verified by mutation (reintroducing the swallow, and separately over-correcting to die unconditionally) -- each goes red with its own message, restores byte-identical (full sha256), and passes a green control. |
||
|
|
7d711942fe |
Merge #534: detect a died shutdown drain the ERROR count is blind to (fleetd #512 part 2)
Verified independently. The branch is based on |
||
|
|
bec87f987c |
scripts: name the mechanism in detect_supervisor's constraint 2, not a line number
Constraint 2 read "This script runs under `set -euo pipefail` (line 50), so an unset variable is a loud failure." Two problems, both small and both the same family as fleetd #494 — a comment that states the wrong reason. The line number was stale: the `set` line is at 54, not 50. It was the only line-number citation in the file, and a citation like that goes stale on the next insert above it, silently, with nothing to catch it. The mechanism was also misattributed. What makes an unset variable a loud failure is `set -u`. Naming the whole `-euo pipefail` string invites the reader to credit pipefail for it, which is the mistake fleet01 flagged on a different cell this week: pipefail is insurance against a future pipeline stage, not what catches the current shape. Now names `set -u` and says where it is without a number, and records why the number is gone so nobody adds one back. Comment only. bash -n exit 0 under /bin/bash 3.2.57 and env bash 5.3.9; scripts/test-redeploy-fleetd.sh exit 0 with 0 lines matching ^FAIL:. |
||
|
|
190436c9cf |
fleetd #512 part 2: detect a died shutdown drain the ERROR count is blind to
The previous daemon's dead shutdown drain (an uncaught exception in a shutdown thread) never passes through the logger, so it never carries an ERROR/SEVERE token, so redeploy-fleetd.sh's existing ERROR-count classifier is structurally blind to it and prints a confident "no ERROR lines since restart" while the drain actually died. Add scan_uncaught_exceptions (greps the shutdown window for the failure's real shape: `Exception in thread`, `NoClassDefFoundError`) and find_drain_complete_line (checks for #522's SessionManager.drainAll completion line). Compose both in report_shutdown_drain, a single decision+action function the main flow calls unconditionally (same shape as swap_if_built/refuse_drain_gate from #521/#528), which resolves to one of four outcomes: complete, died, unknown ("cannot tell" — the line is absent for either of two reasons that need opposite handling: the previous daemon predates #522, or its drain failed without throwing), or n/a (no previous daemon was actually stopped this run). Never fails the redeploy; warns loudly instead. Gate the "no ERROR lines since restart" summary line on the new outcome so it never reads as reassurance when the drain died or the outcome is "cannot tell" (item 4 of the ticket). Tests: 11 new test functions (60 defined/invoked, was 49), covering both pure classifiers, all four report_shutdown_drain outcomes, a source-grep proof of the main-flow call site (sourcing stops before the main flow runs), an ordering check, and the item-4 gating. Full suite green (exit 0, 0 anchored FAIL lines). Five mutations applied and killed by hand during review, each restored to a byte-identical file afterward. |
||
|
|
7c34e8f4f9 |
fleetd #528: pin drain_gate_refusal's call site, not just the predicate
drain_gate_refusal composes the correct abort message and is well tested, but the main flow built its own `die "$(drain_gate_refusal ...)"` call — nothing proved that call site was ever consulted. Mutating it to a flat `die "aborted -- nothing changed"` left the whole suite green, silently reinstating the exact defect #517 was filed to fix. Same shape as #521/#526's should_swap/swap_if_built: the decision and the die() now live together in refuse_drain_gate, which the main flow calls unconditionally. drain_gate_refusal stays separate and separately tested for the message logic; four new behavioural tests stub die() to prove refuse_drain_gate calls it correctly for all four cases, and a fifth source-text test pins the main flow's call site itself (the only thing that can catch deleting the call, since sourcing stops before the main flow runs). |
||
|
|
3366590dbe |
Merge #526: pin the jar swap at its call site, not just its predicate (fleetd #521)
Adjudicated by me. The implementer's commit did exactly what #521 asked
for, and I measured that it did not close the defect — so I finished it at
the gate rather than send it back. The gap was in my ticket, not their work.
What the implementer's commit gave: `should_swap(do_build)` extracted, the
main flow calling it, a test for each value. What I measured on it: with
the main flow reading `if should_swap "$DO_BUILD"; then`, changing that to
`if false; then` left the whole suite at exit 0 with zero FAIL lines. The
swap still never ran. Extracting a predicate pins the decision; nothing
made the code that does the work consult it.
Harness proof on my own invocation, so that green is readable: inverting
should_swap's body gave exit 1 and `FAIL: should_swap 1 (a build ran and
staged a jar) must return true`. The suite can fail when I run it.
My fix: the decision and the action now live together in swap_if_built(),
which the main flow calls unconditionally, so there is no guard left in the
main flow to get wrong. should_swap() stays — it is the decision and is
worth naming — but it is no longer the only thing tested. Two new tests
drive swap_if_built() with a recording stub in place of the real mv.
Two more things I fixed, both found while verifying:
* The ordering test had to follow the call site to `swap_if_built
"$DO_BUILD"`. Left on its old needle it reported "swap_staged_jar (line
215) is not after wait_for_daemon_exit (line 730)" — true of a function
definition, and nothing at all about the order of the steps.
* That test's three `[ -n ... ] || fail "could not find ... call site"`
guards were dead code. Under `set -euo pipefail`, an absent needle fails
the assignment and `set -e` kills the suite before the guard runs.
Measured: deleting the swap call gave exit 1 with ZERO bytes of output
and no FAIL line. Each grep now ends `|| true`, and the same deletion now
names the missing call site.
Verified by me on the merged revision:
* suite exit 0, 0 `^FAIL:` lines, 44 test functions defined and 44 invoked
* bash -n rc=0 on both scripts under /bin/bash 3.2.57 and bash 5.3.9
* four mutations, each killed with its own named FAIL line, each restored
byte-identical by hash, green control after the battery:
- guard removed inside swap_if_built -> "must not swap, but it did"
- guard inverted -> "must perform the swap, and did not"
- should_swap's comparison changed -> "must return true"
- main-flow call deleted -> "could not find the swap call site"
* CI green on
|
||
|
|
01adc841fa |
Merge #523: test the policy probe's parsing guards (fleetd #519)
Adjudicated and verified by me, not taken from the PR body.
What I measured on the merged revision (099b2ecf… for the worker's own
commit,
|
||
|
|
08771e270b |
fleetd #521 gate fix: pin the swap at the call site, not just the predicate
The extraction in the previous commit did what #521 asked for — a should_swap() predicate with a test for each value — and I measured that it does not close the defect. With the main flow reading `if should_swap "$DO_BUILD"; then`, changing that line to `if false; then` left the whole suite at exit 0 with zero FAIL lines. The swap still never ran, and a redeploy would still report success while starting on no jar. That is my ticket's fault, not the implementer's: "extract the decision so the suite can call it" pins the decision and never the wiring. Extraction moved the untested decision up one level instead of removing it. Fix: the decision and the action now live together in swap_if_built(), which the main flow calls unconditionally — there is no guard left in the main flow to get wrong. should_swap() stays, because it is the decision and is worth naming and testing on its own. Two new tests call swap_if_built() with a recording stub in place of the real mv, so they fail if the guard is removed, inverted, or stops being consulted. Also fixed, found while verifying this: * test_swap_ordered_after_wait_and_before_start had to follow the call site to `swap_if_built "$DO_BUILD"`. Left on the old needle it reported "swap_staged_jar (line 215) is not after wait_for_daemon_exit (line 730)" — true of a function definition, and nothing about step order. * That test's three `[ -n ... ] || fail "could not find ... call site"` guards were dead code. Under `set -euo pipefail` an absent needle fails the assignment and `set -e` kills the suite before the guard runs. Measured: deleting the swap call gave exit 1 with ZERO bytes of output, no FAIL line, nothing naming what was missing. Each grep now ends in `|| true` so the assignment succeeds empty and the guard can speak. Verified by me on this revision: * suite exit 0, 0 `^FAIL:` lines, 44 tests defined and 44 invoked * bash -n rc=0 on both scripts under /bin/bash 3.2.57 and bash 5.3.9 * four mutations, each killed with its own named FAIL line, each restored byte-identical, green control after the battery: - guard removed inside swap_if_built -> "must not swap, but it did" - guard inverted -> "must perform the swap, and did not" - should_swap's comparison changed -> "must return true" - main-flow call deleted -> "could not find the swap call site in redeploy-fleetd.sh" (this one printed 0 bytes before the dead-guard fix, which is the before/after proof for it) Not fixed here, filed separately: drain_gate_refusal has the same shape. Replacing `die "$(drain_gate_refusal ...)"` with `die "aborted — nothing changed"` leaves the suite at exit 0 with output byte-identical to a clean run, which reinstates the exact wrong message #517 was filed to fix. |
||
|
|
b5843ab43f |
fleetd #519 review fix: widen two needles to the whole parenthetical
Both arity assertions matched on `jq) returned N field(s)` — a needle that starts in the middle of the script's `(parser name)` parenthetical. On a real failure the harness prints `missing <needle>`, so the line came out as: FAIL: empty parser output count: missing jq) returned 0 field(s) which reads as if the script's own message had an unbalanced paren. It does not; the needle was just sliced. Matching on `policy parser (jq) returned N field(s)` makes the failure readable and also pins that the refusal names the parser it used, which the narrower needle did not. make_jq() PATH-prefixes a fake jq, so `_PARSER_NAME` is deterministically "jq" in both tests; the wider needle cannot flake on a host without jq. Re-proved on this revision, because a disproof is about a revision and not a file: * suite exit 0, "PASS: probe member credentials guards" * bash -n rc=0 on the test under /bin/bash 3.2.57 and bash 5.3.9 * dropping the empty-parse special case -> FAIL: empty parser output count: missing policy parser (jq) returned 0 field(s) * arity threshold 5 -> 0 -> FAIL: short parser output status * script restored byte-identical after each, green control after both |
||
|
|
5b1e13ca3d |
fleetd #519 review fix: move the parser comments with the code they explain
PR #523 extracted parse_policy_fields() but left about 25 lines of explanatory comments at the old parse site in main(). That is the same defect class as fleetd #500 itself — a stated fact that no longer matches the code next to it — in the very file whose ticket history is about it. Three blocks moved, no code touched: * "One parse pass" + the mapfile/process-substitution reasoning now sits above parse_policy_fields(), which is what it describes. * The arity-check block now sits inside the function, directly above `if (( ${#_FIELDS[@]} < 5 ))`. At the old site it said "the slice just below this" and "every line below this expects", both pointing at a function call rather than the check. Reworded to name main() and its slice explicitly. * The pipefail note said the parser failure was "handled below"; the handling is now above it, in the function. The call site keeps a three-line pointer saying where the reasoning went. Checked myself, on this revision: * suite exit 0, "PASS: probe member credentials guards" * bash -n rc=0 under /bin/bash 3.2.57 and bash 5.3.9 * two mutations killed, each proven applied two ways (mutant present AND original gone), restored byte-identical, green control after each: - dropping the empty-parse special case -> FAIL: empty parser output count - arity threshold 5 -> 0 -> FAIL: short parser output status |
||
|
|
c89a375e5d |
fleetd #521: extract should_swap so the swap guard can't be silently disabled
Mutating the swap step's guard (if [ "$DO_BUILD" = 1 ] -> if false) left the whole test suite green: test_swap_ordered_after_wait_and_before_start only checks source positions, which an in-place if-condition edit never moves. Extracts the decision into should_swap(do_build), following the same shape as #510's wait_for_daemon_exit and #517's drain_gate_refusal, with a direct test for each value. |
||
|
|
a5ad7c6561 | fleetd #519: test policy probe guards | ||
|
|
3833d8e52b |
fleetd #517: pin the drain-gate abort branch and jar_id's absent case
Two mutation-testing survivors in scripts/redeploy-fleetd.sh: a source-text test pins what a message SAYS but never whether the branch that prints it is REACHED. - Extract the drain-gate abort decision into drain_gate_refusal(do_build, staged_path), a pure function the suite can call directly for all four build/staged combinations. The existing source-text grep test is kept alongside it (it catches a re-wording; the new tests catch a dead branch). - Extend jar_id's test to cover the missing-file path (both the no-argument default and an explicit path), which the #511 test never exercised. |
||
|
|
b37def9238 | Merge #516: the probe refuses with three distinct messages, each naming its own cause (fleetd #500) | ||
|
|
d59ece6dec |
fleetd #500: stop a wrong-interpreter or failed-parse reading a policy as empty
probe-member-credentials.sh used mapfile < <(producer) to parse the fetched policy. That
hides a producer failure three ways: mapfile is bash 4+ and missing on macOS's /bin/bash
3.2, a process substitution's exit status is never propagated to mapfile, and the
downstream reads (":-" defaults and a slice) never fire set -u on a short or unset array.
All three converge on the same "0 known names" refusal, which blames the policy for a
failure that is actually the interpreter or the parser.
Three distinct guards, each closing one cause with its own message:
- a BASH_VERSINFO gate at the top refuses outright on bash < 4 (exit 3)
- the parser's output is captured via command substitution instead of mapfile < <(...),
so a non-zero jq/python3 exit is caught at the call while the fact still exists (exit 4)
- an arity check before the field slice refuses a parse that exits 0 but returns fewer
than 5 fields (exit 5)
The existing "0 known names" guard is now honest: by the time it fires, the three causes
above are already ruled out, so it really does mean the policy has 0 known names.
|
||
|
|
6e23bf8309 |
fleetd #511: fix wrong --no-build wording in drain-gate abort, pin jar_id() default
The drain-gate abort message told the operator a rerun "with or without --no-build" would finish the restart. That is wrong: by the time this message can fire, stage_built_jar has already moved the jar off $JAR, so --no-build hits require_no_build_jar's own refusal. Reworded to say the rerun must NOT use --no-build, and why: the built jar is no longer at the live path that --no-build requires. Also added a test pinning jar_id()'s no-argument default (reports $JAR, the live path) and its explicit-argument behavior (reports that path instead), per fleetd #511 item 2. Not adding a test for the JAR_STAGED rm -f at line 578 (fleetd #511 documents it as an equivalent mutant — mvn clean install deletes target/ on the next line regardless). |
||
|
|
979adf82eb |
fleetd #493: never build into the path a running daemon holds
redeploy-fleetd.sh's build step wrote straight into fleetd/target/fleetd.jar
via `mvn clean install` while the OLD daemon was still running from that
exact path. A JVM loads classes lazily, so a class the daemon had not
touched yet could be read from a jar already replaced or removed -- the
failure landed on the shutdown drain (NoClassDefFoundError, exit 143,
looks clean).
Stage the freshly built jar at target/fleetd-new.jar (stage_built_jar),
confirm the OLD pid has actually exited (wait_for_daemon_exit, extracted
from the existing wait loop), and only then swap it into the live path
(swap_staged_jar) -- strictly after the wait, strictly before start. A
failed swap dies without starting. --no-build and --check keep their
existing, truthful behavior (require_no_build_jar; jar_id still reads
the live path by default). A leftover staged jar from an interrupted
run is wiped before the next build. The build still runs before
anything is stopped, so a failed build still never takes the fleet down.
Also fixed: the drain-gate abort message ("aborted -- nothing changed")
now names the staged jar when one exists, since staging already moves
the freshly built jar off the live path before that prompt runs.
Adds unit tests for stage_built_jar, swap_staged_jar, require_no_build_jar,
wait_for_daemon_exit, and a source-order test proving swap sits after the
wait and before start (sourcing stops before the main flow ever runs, so
the ordering itself can only be checked by reading the script's own call
sites).
|
||
|
|
599419f9e6 |
fleetd #492 follow-up: carry the unclear detail across detect_supervisor's subshell boundary
|
||
|
|
b17f37a683 |
fleetd #492 follow-up: detect_supervisor must never read "could not tell" as "none"
Two situations were silently landing in the "none" answer, which require_drivable_supervisor accepts and the script then falls back to a raw kill + nohup — exactly the wrong move when a supervisor actually IS present: - installed-but-not-loaded, on either supervisor. `systemctl --user is-active` answers "no" for activating/deactivating/failed and while an auto-restart is pending too, and every one of those is a host that IS under systemd (or launchd) and about to act again. `*_installed` already knew this; it was only ever consulted for a warning line, never by the decision itself. - a systemd probe that could not answer at all (e.g. systemctl cannot reach the user bus over a non-lingering ssh session) looked identical to a clean negative, because both probes redirected stderr straight to /dev/null. detect_supervisor now returns a fifth answer, "unclear", for both cases. systemd_loaded/systemd_installed capture systemctl's exit status and stderr separately and set their own *_ERRORED flag only on a real tool failure (non-zero exit WITH stderr), never on a clean negative. "none" now means only: neither supervisor installed, neither loaded, neither probe errored. require_drivable_supervisor die()s on "unclear" exactly like it already does on "ambiguous", naming the specific supervisor and reason via the new SUPERVISOR_UNCLEAR_DETAIL global. Tests: 4 new cases (systemd/launchd installed-but-not-loaded, a real systemd_loaded run through a systemctl stub that errors on stderr, and the die() refusal for "unclear" naming the unit). All 3 new guards were verified by mutation: each was removed from the real script, the suite caught it (a new FAIL line naming the exact broken assertion), then the file was restored byte-identically and the suite went green again. |
||
|
|
dcd505286f |
fleetd #492: teach redeploy-fleetd.sh systemd --user as a third supervisor
launchd, systemd --user, and unsupervised are three different answers, not two. Refuse (die) rather than fall through to kill+nohup when a supervisor is detected that this script cannot drive (e.g. both signals fire at once), and add a post-restart check that fails the run if more than one fleetd process is alive. detect_supervisor()/require_drivable_supervisor()/ count_daemon_pids()/assert_single_daemon() are pure, overridable functions so scripts/test-redeploy-fleetd.sh can exercise them without a real launchd or systemd. |
||
|
|
09159f2857 | Classify named AMQP recovery errors | ||
|
|
0241e0d3a8 | Keep unattributed AMQP errors loud | ||
|
|
e4973eb8a4 | Classify recovered AMQP redeploy errors | ||
|
|
51f7b0a3ca |
fleetd #111: probe reads the live memberCredentials policy, no hardcoded name list
scripts/probe-member-credentials.sh carried its own hand-maintained NAMES array (31 names, recorded 2026-08-16), so a name added later to fleetd.yaml's memberCredentials.known was never checked and the probe still exited 0 with a clean-looking table. Same drift shape as #114's tool catalogue. - New dev.ltms.fleet.member.MemberCredentialPolicyView: the single place that turns a MemberCredentials policy into names + counts (never a value). Reused by Fleetd.reportMemberCredentialsGap (startup log line) and by the new GET /member-credentials REST endpoint (FleetApp), so the two can no longer drift apart the way the probe and the policy did. - FleetApp gains one route + handler + a Supplier<MemberCredentialPolicyView> constructor param (legacy constructors default to ::absent, so existing call sites are unaffected). - probe-member-credentials.sh now fetches its name list from GET /member-credentials instead of carrying one. No local fallback: an unreachable daemon, an empty/absent policy, or a knownCount/known[] length mismatch all refuse with a non-zero exit rather than silently checking zero names. Prints "policy contains N; this run checked N" so the two numbers are visibly equal. |
||
|
|
2e138a199b |
CB-634: one shared "fleet" workspace + rename bridged -> fleetd cutover
Two changes ship together here.
1. One shared herdr workspace. The lead and every worker now live in one
workspace called "fleet", so the operator sees one "session" with many
windows, not two. Before, the lead sat in a "leads" workspace and workers
in "bridged-workers", which read as two sessions. The lead is still told
apart from workers by its exact tab label ("lead: <name>"), so putting them
in one space is safe. LeadTabScanner keeps the exclude-by-label mechanism
for split layouts; Fleetd now passes an empty exclude set.
2. Rename the daemon from "bridged" to "fleetd" (the binary, config, scripts,
launchd/systemd units, module dir, and MCP mount).
- Module dir bridged/ -> fleetd/; jar finalName -> fleetd.jar.
- Log line, comments, docs, and CLAUDE.md updated to say fleetd.
- Scripts renamed: redeploy-bridged.sh -> redeploy-fleetd.sh,
bridged-launchd-wrapper.sh -> fleetd-launchd-wrapper.sh.
- Deploy units renamed: dev.ltms.bridged.plist -> dev.ltms.fleetd.plist,
bridged.service -> fleetd.service; launchd Label -> dev.ltms.fleetd.
- Config default bridged.yaml -> fleetd.yaml; the legacy bridged.yaml is
still read as a fallback, and still gitignored.
- MCP: drop the deprecated bridge_* tool twins; only fleet_* remain. The
server name is "fleet". The mount name in the local .mcp.json becomes
"fleet" (gitignored, not in this commit).
- Env var defaults BRIDGED_API_TOKEN -> FLEETD_API_TOKEN, fixture
BRIDGED_WORKER_TOKEN -> FLEETD_WORKER_TOKEN.
Kept on purpose: the BRIDGED_MEMBER marker. Renaming it is a coupled change to
the credential-scrub security control (an operator secrets.sh may guard on it),
so it stays until that migration is done on its own.
Metrics were already fleet_* (CB-632); MetricNamesTest still guards that no
name says bridged_.
The canonical CLAUDE.md block and the wiki template stay byte-identical
(wiki working tree edited, committed to the wiki repo separately).
949 tests pass (mvn clean install). 4 fewer than before = the 4 removed
bridge_* alias tests.
|
||
|
|
3f4ac2b24e |
CB-635: --check reports whether broker.uriEnv resolves in a login shell
An empty uriEnv no longer stops the daemon (#152), so the failure is quiet: bridged starts, falls back to the in-memory reply inbox, and held reports stop surviving a restart. --check is the only thing that says so before the fact. The var name is read out of bridged.yaml so a renamed key cannot make the check lie. |
||
|
|
51bdec22e7 |
CB-624: report three more rename surfaces as report-only
--check gains section 6 counting ~/.claude.json (the projects entry keyed by the old absolute path), ~/.config/herdr/session.json, and JetBrains recentProjects.xml / trusted-paths.xml across every IntelliJIdea* version. Apply mode's closing summary lists the same three with manual follow-ups. None of the three is rewritten automatically, on purpose: ~/.claude.json is global live Claude Code config, herdr session.json is live process state, and JetBrains rewrites its own files when the project is reopened at the new path. Header comment states the reason for each so it does not read as an oversight. JetBrains stores these paths as its $USER_HOME$ macro rather than a literal absolute path, so the counter matches both forms and says which form the hits used. |
||
|
|
d43f670285 |
CB-624: add scripts/rename-checkout.sh to rename the checkout safely
Renames ~/LTMS/claude-bridge -> ~/LTMS/fleetd as one auditable command, modeled on redeploy-bridged.sh. --check reports every surface holding the old absolute path (checkout files, launchd plist, Claude Code project state, worktree .git pointers, running daemon). Apply mode stops the daemon first (launchctl-aware), moves the checkout and the Claude Code project-state slug dir derived from both paths, repairs worktree gitdir pointers, rewrites bridged.yaml and the installed plist if they exist, restarts, and verifies /healthz plus a fresh 'bridged listening' line anchored to a pre-stop marker. |
||
|
|
03473a286b | CB-622: rename bridge_* tools to fleet_* in scripts, e2e tests and config; mount name bridged -> fleetd | ||
|
|
837fed7690 |
CB-596: the credential probe, as one auditable command
Issue #82 step 1 is a measurement, and the classifier refuses an ad-hoc pipeline that enumerates credential names inside a member — correctly. This is the seam: one file the operator reads once and then runs, instead of approving a shell pipeline they have to take on trust. It never prints a credential value or any part of one. #82's criterion 1 asked for a 6-character prefix; this prints a truncated SHA-256 instead. A prefix of a short secret is most of the secret and would end up pasted into a ticket, while the hash answers every question the prefix was for — is it set, is it the same value as over there, is it the CB-592 sentinel. Refuses to run unless BRIDGED_MEMBER=1, since the finding is what a MEMBER holds; --allow-outside-member takes the comparison reading and labels it as such. I have not run the reading path. That is the operator's call, which is the whole point of the ticket. |
||
|
|
cec48832be |
CB-600: make it safe to install the launchd agent
- redeploy-bridged.sh now refuses (not warns) a supervised restart when its computed log path disagrees with the loaded plist's StandardOutPath — otherwise every post-restart check reads the wrong file and can report a clean restart while the daemon crash-loops. The check is a pure, testable function; the script gained a source-for-test guard so it can be exercised without installing the agent or touching launchd. - a failed 'launchctl load' after a successful 'unload' now retries once and, on ultimate failure, tells the operator the agent is stopped AND disabled plus the exact recovery command, instead of leaving that silently worse than the pre-redeploy state. - the plist documents honestly that the crash loop launchd retries is unbounded (ThrottleInterval only paces it), and what actually stops it. - fixed the requiredSecretEnvVars javadoc: the auth.tokenEnv startup throw is ~370 lines below its call site, not a few lines above it, and only fires in auth.mode: token. |
||
|
|
3ba6d6784c |
CB-594: make supervision and a working fleet possible at the same time
Adds scripts/bridged-launchd-wrapper.sh so the launchd-run daemon still gets WORKER_GITEA_TOKEN/AI_GATEWAY_TOKEN by execing through a login shell (launchd never sources secrets.sh itself). bridged now logs at startup which required token env vars (derived from each profile's tokenEnv/gitTokenEnv, not a hand-written list) resolved or are MISSING, by name only. Fills in the real paths in deploy/dev.ltms.bridged.plist for this host and points it at the wrapper. scripts/redeploy-bridged.sh now detects a loaded launchd agent and uses launchctl unload/load instead of a raw kill+nohup, because a bare SIGTERM exits this JVM at 143 (measured) which KeepAlive.SuccessfulExit=false reads as a crash and would race the script's own restart; --check reports installed/loaded state and stays read-only. |
||
|
|
f0095bf8b2 |
CB-591: plan the move onto the LLM/MCP gateway, and check AI_GATEWAY_TOKEN
The gateway (llm.ltms.dev) replaced Bifrost on 2026-08-15 and serves an Anthropic surface and an OpenAI surface, so both member kinds can point at it. The plan is in docs/CB-591-Gateway-Migration.md; gitea #76 tracks the work. The opencode half needs no code: OpenCodeLauncher already pins an OpenAI-compatible endpoint (CB-508), so baseUrl + tokenEnv + provider/model is a config change. That matters more than it looks — every opencode member today is sol or terra, and both sit on one OpenAI account via credentialId: openai-shared, so an exhaustion on either locks out both. A gateway-backed opencode profile is free and off that credential, which retires a single point of failure rather than only adding capacity. Also extends the redeploy script's --check to AI_GATEWAY_TOKEN. A profile's tokenEnv is resolved from the DAEMON's own environment by HerdrPeerLauncher.resolveEnv, so a token added to secrets.sh after the daemon started is simply absent: the launcher injects an empty token and the gateway answers 401, long after the restart and with nothing tying the two together. That is the same trap as WORKER_GITEA_TOKEN, and it gets the same login-shell check that never prints the value. |
||
|
|
5fe02b7c98 |
Fix redeploy script reporting failure on a successful restart
Found by running it. The script launched the daemon with a relative jar path (cwd is bridged/) but detected it with an absolute one, so pgrep never matched. The daemon restarted correctly and booted clean, and the script still failed with 'no process appeared' — the worst shape of bug for a deploy tool, because it invites a second restart on a daemon that is already healthy. Detection now matches both path forms, and the launch uses the absolute path so ps names which checkout is running. |
||
|
|
a1052f4fd1 |
Add scripts/redeploy-bridged.sh so the lead can deploy in one command
The lead already owned the redeploy, but the command classifier refuses a bare kill on the daemon, so in practice every deploy still needed the operator to approve a stop and a start by hand. A single script is the seam that fixes that: the operator allow-lists one auditable command instead of two ad-hoc ones. It also stops the procedure from living only in a checklist people read after things go wrong. It builds before it stops anything, so a failed build never leaves the fleet down; waits for the old process to exit instead of assuming; polls /healthz; and anchors its log checks to a line marker taken before the restart, so old errors cannot be misread as new ones. The check with no log line anywhere in the daemon is the reason --check exists: bridged inherits WORKER_GITEA_TOKEN from the shell that starts it, and starting from a non-login shell empties it. The daemon boots fine, healthz is green, and the failure only appears later as workers that cannot open a PR. --check tests whether the name resolves and never prints the value. |