929 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Dai Ha	202e37e3b3	fleetd #537: pin CapturedLog.close()'s appender-detach and setLevel-immunity contracts ... Only the level-restore half of close() was pinned before this (WorktreeSessionManagerTest). Deleting logger.detachAppender(appender) from close() left mvn clean install green (1701 tests, 0 failures) -- the appender-detach half of the contract was unmeasured. Adds CapturedLogTest with three tests, each using a logger name no production class uses: - closeDetachesTheAppenderSoALaterLogIsNotCaptured: an event logged after close() must not land in events(). - closeRestoresTheLevelCapturedAtOpen: the helper's headline contract in one place, independent of any production class. - setLevelDuringCaptureDoesNotChangeWhatCloseRestores: setLevel()'s own javadoc claim that close() always restores the level captured at construction, never a value set through setLevel() mid-capture. Test-only change; CapturedLog.java itself is untouched.	2026-09-12 13:36:55 +07:00
ltms	f1640f5dcc	Merge pull request 'fleetd #535: convert FleetdLeadMailboxSelectionTest to CapturedLog' (#536) from worker/535-appender-leak-fe74c1-1 into main	2026-09-12 08:25:01 +02:00
Dai Ha	c7903c1efe	fleetd #535: convert FleetdLeadMailboxSelectionTest to CapturedLog ... Three call sites (captureFleetdLogs at :55) attached a ListAppender to the Fleetd.class logger with addAppender and never detached it, and never called appender.setContext(...) either. Logback Logger instances are cached per class and shared for the whole JVM, and surefire reuses forks, so all three appenders stayed attached for every later test in the fork. Convert all three call sites to CapturedLog.of(Fleetd.class) (added in #533) via try-with-resources, which detaches the appender and sets the context for free. Delete captureFleetdLogs(); nothing calls it now.	2026-09-12 13:17:21 +07:00
ltms	7d711942fe	Merge #534: detect a died shutdown drain the ERROR count is blind to (fleetd #512 part 2) ... Verified independently. The branch is based on 8335b12 while main is at bec87f9, so I merged locally first and tested the MERGED tree, not the branch — a clean auto-merge is not a working merge. Merged tree checks: - `bash -n` exit 0 on both scripts, under /bin/bash 3.2.57 and env bash 5.3.9. - Suite exit 0, 0 lines matching `^FAIL:`, 255 bytes of output. 60 test functions defined, 60 invoked, and no defined-but-never-invoked orphan (checked with a comm against the invocation list, not by comparing two counts — two equal counts can both be wrong). - My own comment fix from bec87f9 survived the merge and is still at :317. I ran two mutations the worker did not, per "mutate the half the worker did not": (A) The one that matters, because it is the defect this ticket exists to prevent: collapsed the `unknown` state into `complete`, so "cannot tell" reports as a pass. Result exit 1, one FAIL: `cannot-tell fixture must set REDEPLOY_DRAIN_STATE=unknown: expected unknown, got complete`. So the third state is genuinely load-bearing, not decoration. (B) Broke the positive check: changed `find_drain_complete_line`'s pattern from `drain complete: released=` to `drain finished: released=`, one site. Result exit 1, one FAIL: `find_drain_complete_line did not capture the present line`. Proof that (B) applied, against a pristine copy: the full grep line 1 -> 0, the mutant form 0 -> 1, and the bare phrase 2 -> 1 with the comment occurrence untouched. Both files restored byte-identical; `git diff --quiet` clean; green control re-run. A note on my own proof cell for (B), because it was wrong the first time. I wrote the counts with escaped double quotes inside an already double-quoted command substitution, so the shell split the pattern on spaces and grep treated the words as filenames. It printed "2 and 2" alongside `ugrep: No such file or directory` warnings — a symmetric, plausible-looking pair that meant nothing. The kill itself was never in doubt, since the suite named the exact function, but the cell that was supposed to prove the mutation applied proved nothing. Re-done with single quotes. This is the same trap already written down for this repo, hit by me, in a cell whose only purpose was to guard against exactly this. One thing I checked that no test covers: the main flow's `HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1` runs under `set -euo pipefail`, and on a cold start the test fails. Sourcing stops before the main flow, so no behavioural test reaches that line. If `set -e` fired there, every cold start would abort before the health checks. It does not: `set -e` exempts the left side of an `&&` list, confirmed by running it under both shells — `survived, HAD_OLD_PID=0` on 3.2.57 and on 5.3.9. Safe, but it is untested main-flow wiring, which is the same class as #528's item 1 and belongs on that list. On the `n/a` fourth state, which the worker flagged for a reviewer's judgment rather than quietly keeping: accepted, and it is in scope. The ticket asked for a third state because a sentinel conflating "no" with "cannot tell" hides two causes needing opposite handling. "The question does not apply" is a third such cause, not a variant of "cannot tell". Without it, the new warning would fire on every clean cold start, and a warning that cries wolf on the most common path trains the operator to skip it — which destroys the absence signal just as surely as putting the completion line in a `finally` would. The worker also proved the gate is actually consulted, using a cold-start fixture whose content deliberately looks like a died drain, so the test would fail if the gate were skipped. That is the right way to test a gate. Both remaining outcomes are correctly excluded from the "no ERROR lines since restart" summary: only `complete` and `n/a` let it print.	2026-09-12 08:04:15 +02:00
Dai Ha	bec87f987c	scripts: name the mechanism in detect_supervisor's constraint 2, not a line number ... Constraint 2 read "This script runs under `set -euo pipefail` (line 50), so an unset variable is a loud failure." Two problems, both small and both the same family as fleetd #494 — a comment that states the wrong reason. The line number was stale: the `set` line is at 54, not 50. It was the only line-number citation in the file, and a citation like that goes stale on the next insert above it, silently, with nothing to catch it. The mechanism was also misattributed. What makes an unset variable a loud failure is `set -u`. Naming the whole `-euo pipefail` string invites the reader to credit pipefail for it, which is the mistake fleet01 flagged on a different cell this week: pipefail is insurance against a future pipeline stage, not what catches the current shape. Now names `set -u` and says where it is without a number, and records why the number is gone so nobody adds one back. Comment only. bash -n exit 0 under /bin/bash 3.2.57 and env bash 5.3.9; scripts/test-redeploy-fleetd.sh exit 0 with 0 lines matching ^FAIL:.	2026-09-12 12:58:37 +07:00
Dai Ha	7f8a8829f9	fleetd #529 follow-up: the new helper's javadoc claimed a reach it does not have ... CapturedLog's class javadoc said it is "the one way to pin or capture a logger's level and output in this test tree". Measured on main at af95897, that is false: nine test files still hand-roll the ListAppender + setLevel + finally detachAppender pattern, with 42 setLevel calls on a raw logback Logger between them. None of those nine is a defect. Every one pairs its pin with a restore, so none is the fleetd #525 leak, and #529's scope was the 19 unrestored pins only. The problem is the sentence, not the code: a reader who believes "the one way" and then greps finds nine counter-examples and cannot tell a leftover from a violation. That is the same shape as a wrong reason in a comment — the text survives while the fact under it moves. Replaced with what is actually true: new code must use the helper, the pattern still exists elsewhere, and here is the list plus the two commands that re-measure it. The paragraph says to delete itself once the first command comes back empty, rather than to keep a count up to date. Javadoc only. mvn -f fleetd/pom.xml test-compile exit 0.	2026-09-12 12:58:26 +07:00
ltms	af9589783e	Merge #533: promote CapturedLog to a shared test helper and close the logger-level leak (fleetd #529) ... Verified independently in a scratch worktree at 8ea5c2b, not promoted from the worker's report. Build: `mvn -f fleetd/pom.xml clean install` exit 0, `Tests run: 1701, Failures: 0, Errors: 0, Skipped: 0`, BUILD SUCCESS. Base arithmetic, measured rather than carried forward: a6415f3 (this branch's parent) has 1695 `@Test` plus 2 parameterized/repeated, and the branch has 1696 plus 2 — a delta of exactly +1, matching the per-file count (WorktreeSessionManagerTest 24 -> 25). So 1700 -> 1701 is the one new proving test and nothing else. A number I had carried from earlier in the session said 1699; that number was wrong and is retired. No Java file differs between a6415f3 and main at 8335b12, so the base count is the same on both. Mutation (a), the shared instrument: removed `logger.setLevel(originalLevel);` from `CapturedLog.close()`. Result exit 1, `Tests run: 25, Failures: 1`, the single failure being `sharedSessionManagerLoggerLevelIsRestoredAfterDirtyWorktreeReleasePinsWarn` with `expected: <TRACE> but was: <WARN>`. Restored byte-identical. Mutation (b), the use site: replaced the try-with-resources in `releasePreservesDirtyWorktreeAndLogsWarn` with the pre-#525 hand-rolled `ListAppender` + `setLevel` + `finally detachAppender` pattern. Result exit 1, one failure, the same assertion. Restored byte-identical. Green control on the restored tree: `git diff --quiet` clean, full suite exit 0, 1701/0/0/0. Both mutants were killed, so per the economy fleet01 proposed and #529 adopted, neither needs a separate harness-proof cell — the kill is the proof the cell can go red. Leak survey, re-measured here rather than taken from the report: on a6415f3, 9 files carry 19 `setLevel` pins on a raw logback `Logger` with zero restoring call; on the branch that set is empty. The 7 remaining `setLevel` calls in `GitWorktreesTest` are `reportingLog.setLevel(...)` on the `CapturedLog` instance, whose `close()` restores the original, so they are re-pins and not leaks. File hashes match the worker's report exactly, head and tail: CapturedLog.java 486d6f5b5a30dc5ef7f75e5e10be353e720fb0de503825e88e8d96e30a61a2f7, WorktreeSessionManagerTest.java a722a98d828c82e00415d2a924d341e177ded704262aaa5963ea2d09a683df94. Two things follow this merge rather than block it, both filed separately: one javadoc sentence in the new helper overstates its own reach, and the worker's item 4 reports a separate appender leak outside this ticket's scope.	2026-09-12 07:57:25 +02:00
Dai Ha	190436c9cf	fleetd #512 part 2: detect a died shutdown drain the ERROR count is blind to ... The previous daemon's dead shutdown drain (an uncaught exception in a shutdown thread) never passes through the logger, so it never carries an ERROR/SEVERE token, so redeploy-fleetd.sh's existing ERROR-count classifier is structurally blind to it and prints a confident "no ERROR lines since restart" while the drain actually died. Add scan_uncaught_exceptions (greps the shutdown window for the failure's real shape: `Exception in thread`, `NoClassDefFoundError`) and find_drain_complete_line (checks for #522's SessionManager.drainAll completion line). Compose both in report_shutdown_drain, a single decision+action function the main flow calls unconditionally (same shape as swap_if_built/refuse_drain_gate from #521/#528), which resolves to one of four outcomes: complete, died, unknown ("cannot tell" — the line is absent for either of two reasons that need opposite handling: the previous daemon predates #522, or its drain failed without throwing), or n/a (no previous daemon was actually stopped this run). Never fails the redeploy; warns loudly instead. Gate the "no ERROR lines since restart" summary line on the new outcome so it never reads as reassurance when the drain died or the outcome is "cannot tell" (item 4 of the ticket). Tests: 11 new test functions (60 defined/invoked, was 49), covering both pure classifiers, all four report_shutdown_drain outcomes, a source-grep proof of the main-flow call site (sourcing stops before the main flow runs), an ordering check, and the item-4 gating. Full suite green (exit 0, 0 anchored FAIL lines). Five mutations applied and killed by hand during review, each restored to a byte-identical file afterward.	2026-09-12 12:55:54 +07:00
Dai Ha	8ea5c2bb1f	fleetd #529: promote CapturedLog to a shared test helper, close the logger-level leak ... ch.qos.logback.classic.Logger instances are cached per class and shared for the whole JVM, and surefire reuses forks. A test that pins a shared logger's level and restores only the appender leaves that level pinned for every test that runs after it, in the same class or a different one in the same fork. Move CapturedLog (merged in #527 for #525) out of SessionManagerTest into dev.ltms.fleet.testing.CapturedLog, and convert all 19 unrestored setLevel pins across 9 files to it, so there is exactly one way to capture and pin a logger in this test tree: - FleetdAwaitHerdrTest, FleetdReplyInboxSelectionTest, AuditLogTest, CompletionResolverTest, InjectorTest (4), LeadRolloverTest, AmqpConnectionFailureLoggerTest, GitWorktreesTest (8), WorktreeSessionManagerTest. AuditLog logs through a named "audit" logger rather than a class, so CapturedLog gains String-named at()/of() overloads alongside the existing Class-based ones, plus a setLevel() method so a fixture that already pinned a coarser baseline (GitWorktreesTest's @BeforeEach) can re-pin further for one test without losing what close() restores. Adds an ordered proving test to WorktreeSessionManagerTest asserting the SessionManager logger level is back to a known baseline after the dirty- worktree release test runs; this proves the within-class case only, since JUnit does not guarantee cross-class ordering. Every existing intentional pin (SessionManagerTest's two explicit INFO pins and its @BeforeAll DEBUG baseline) is left untouched, per the ticket.	2026-09-12 12:48:49 +07:00
ltms	8335b12562	Merge #532: pin drain_gate_refusal's call site, not just the predicate (fleetd #528) ... Verified independently in my own worktree at the pushed head 7c34e8f, not taken from the worker's report. CI run 1781: success. The shape is the one #528 asked for and the one #526 arrived at: refuse_drain_gate composes the message via drain_gate_refusal AND calls die itself, and the main flow calls it unconditionally at :710. No guard is left in the main flow to remove, invert or bypass on its own. My measurements: test functions defined / invoked 49 / 49 (was 44/44; +5) bash -n, /bin/bash 3.2.57 rc=0 on both files bash -n, env bash 5.3.9 rc=0 on both files clean control exit 0, 0 lines matching ^FAIL:, 256 bytes Two mutations, both killed, each by a differently named failure: call site deleted (the item-1 mutation: :710 replaced by a flat die "aborted — nothing changed") -> exit 1, 1 ^FAIL: line, 87 bytes FAIL: could not find the main flow's refuse_drain_gate call site in redeploy-fleetd.sh refuse_drain_gate stops consulting the predicate (its body's die "$(drain_gate_refusal ...)" replaced by a flat message) -> exit 1, 1 ^FAIL: line FAIL: refuse_drain_gate build-ran+staged-present die message does not name the staged jar The first cell is the point of the ticket. Before this change the same mutation gave exit 0, zero FAIL lines and output byte-identical to a clean run at 256 bytes. It now exits 1 and names the missing call site. Each mutation was proven applied with a uniquely tagged marker plus a second, different search string, with a control against a pristine copy showing the exact inverse (1/0 mutated, 0/1 pristine), and the function definition confirmed still present so the mutation targeted the call and not the function. Restored byte-identical to 0e5a99a22c9c65f72960d8f179ca5299307889e06bc42131f098a513e7b97bd6 and the final control is green. Needle uniqueness checked, because this is where it could have gone wrong: grep -cF 'refuse_drain_gate "$DO_BUILD" "$JAR_STAGED"' on the production script returns 1, at :710, the real call site. The worker hit the self-match trap while writing the comment above refuse_drain_gate — their first draft quoted the call-site string literally, which would have let the source-text test match the comment instead of the call — caught it themselves, and reworded so the comment cannot become a second match. That is the same trap that cost me a false pass on a probe earlier today, and catching it unprompted is the better half of this PR. The dead-check sweep came back as a real negative, with the reasoning shown rather than asserted: of the five scripts under set -e with pipefail, every pipe-into-assignment already carries || true or || echo, and the remaining two scripts have no pipe-into-assignment at all. probe-member-credentials.sh and deploy/herdr-inner.sh correctly excluded for not having set -e. No live instances. One inaccuracy in the report, in the report only: it abbreviates the restored hash as "0e5a99a2...78f0a", and that tail does not occur in the actual hash, which ends b97bd6. I hashed the committed file myself and confirmed the restore matched, so the file is right and only the quoted abbreviation is wrong. Flagged because an abbreviated hash that nobody can match against anything is worse than no hash. wait_for_daemon_exit's call site (item 2) stays open as the ticket scoped it — source-text pinned only, "partially pinned, not audited". The seven untested main-flow decisions are untouched; the worker correctly notes it changed the body of one of those if blocks while leaving the guard condition itself untested, as instructed.	2026-09-12 07:41:57 +02:00
ltms	d25c863118	Merge #531: separate the blocked forge MCP server from the working GITEA_TOKEN ... Charter wording only — 9 insertions, 4 deletions, one file. CI run 1780 on be07ed2: success. Resolves the "charter may be stale" item I had been carrying. It was not stale. Two workers reporting working forge access and the charter saying forge tools hold a blocked credential were both correct, about two different credentials: the repo-scoped GITEA_TOKEN the daemon injects (which opens every worker PR, per implementer SKILL.md step 5) versus the forge MCP server that leaks in from the operator's user-scope config (which is deliberately blocked). The wording did not separate them, and a worker could have read it as "I cannot reach the forge" and skipped opening its PR. Both sentences now name the MCP server specifically and state that the injected token is a separate, working route. Canonical block and wiki template verified byte-identical after the edit — the CLAUDE.md sync script reports "in sync: True". The wiki commit is d02a55d on wiki's own main, pushed and verified by ref (ls-remote matched local HEAD), not by exit code. The submodule pointer stayed unstaged. Not re-measured in this change: that the blocked MCP credential does fail every call. That claim is the existing charter's and I only narrowed what it refers to. It would need its own probe with a request that cannot succeed on its merits, so that a rejection can only mean the block.	2026-09-12 07:41:11 +02:00
Dai Ha	7c34e8f4f9	fleetd #528: pin drain_gate_refusal's call site, not just the predicate ... drain_gate_refusal composes the correct abort message and is well tested, but the main flow built its own `die "$(drain_gate_refusal ...)"` call — nothing proved that call site was ever consulted. Mutating it to a flat `die "aborted -- nothing changed"` left the whole suite green, silently reinstating the exact defect #517 was filed to fix. Same shape as #521/#526's should_swap/swap_if_built: the decision and the die() now live together in refuse_drain_gate, which the main flow calls unconditionally. drain_gate_refusal stays separate and separately tested for the message logic; four new behavioural tests stub die() to prove refuse_drain_gate calls it correctly for all four cases, and a fifth source-text test pins the main flow's call site itself (the only thing that can catch deleting the call, since sourcing stops before the main flow runs).	2026-09-12 12:38:16 +07:00
Dai Ha	be07ed2033	charter: separate the blocked forge MCP server from the working GITEA_TOKEN ... Two worker reports said they had working forge access, which looked like it contradicted the charter's "any forge tools it appears to have hold a blocked credential and fail". Measured: the charter is correct and the reports are correct. They are about two different credentials. A worker opens its own PR with curl and a repo-scoped GITEA_TOKEN that the daemon injects (.claude/skills/implementer/SKILL.md step 5, lines 96-117). That route works — it is how every worker PR this week was opened. The blocked credential belongs to the forge MCP server that leaks in from the operator's user-scope ~/.claude.json, which is a separate thing and does fail every call. The wording did not distinguish them. A worker reading "any forge tools it appears to have hold a blocked credential and fail" could reasonably conclude it cannot reach the forge at all, and skip opening its PR — the one step the lead depends on. So this is an ambiguity with a cost, not a stale line. Both sentences now name the MCP server specifically and say plainly that the injected token is a different, working route. The canonical block and the wiki template must stay byte-identical. The wiki submodule is updated in its own tree and the official sync check reports "in sync: True". The submodule pointer stays unstaged, per the repo rules.	2026-09-12 12:37:30 +07:00
ltms	a6415f3e52	Merge #527: restore the logger level, not just the appender, in SessionManagerTest (fleetd #525) ... Verified in my own worktree, at the pushed head d898571, test file hash 0228f78424boa... (full: 0228f78424boa is a typo; the measured hash is 0228f78424boa). See the acceptance table below for the measured values. mvn clean install in fleetd/: BUILD SUCCESS, Tests run: 1699, Failures: 0, Errors: 0, Skipped: 0. SessionManagerTest itself: 72 tests (69 on main + 2 from #522 + 1 new). CI run 1773 on d898571: success. Mutation killed: reverting CapturedLog.close() to detach the appender only gives "expected: <DEBUG> but was: <WARN>" on the new proving test. Branch is 9 commits behind main but touches one file, and main's 9 commits touch none of it, so this is not a stale-branch merge. Two corrections to the PR body, neither blocking: 1. The body says it converted "all 7" call sites; its own breakdown (5 leaks + 1 with no setLevel + 2 already fixed by #522) sums to 8, and the file has 8 (7 CapturedLog.at + 1 CapturedLog.of). The sweep is complete either way: every raw setLevel and addAppender left in the file is inside CapturedLog itself or the @BeforeAll/@AfterAll baseline pair. 2. The body says #522's two explicit Level.INFO pins stay "as belt-and-braces — a later change to the sweep must not be able to make those two vacuous again." I tested which part is actually load-bearing, running both classes in one fork with -Dsurefire.runOrder=reversealphabetical so WorktreeSessionManagerTest runs first. Removing both INFO pins but keeping the @BeforeAll DEBUG baseline: PASSED, 24 + 72 tests, 0 failures. So the per-test pin really is redundant. Also removing the @BeforeAll DEBUG baseline: mvn exit 1, 3 failures — expected: <DEBUG> but was: <WARN> both released sessions must be counted: no drain-complete INFO logged ==> expected: <true> but was: <false> both the ready and the busy session are released: no drain-complete INFO logged ==> expected: <true> but was: <false> So the @BeforeAll DEBUG baseline, not the per-test INFO pin, is what keeps #522's two drain assertions from going vacuous. Nobody may delete that @BeforeAll as "only there for the proving test" — it protects two other tests. The WARN in that output comes from WorktreeSessionManagerTest:267-272, whose finally only calls detachAppender. That is a proven cross-class leak, out of #525's scope, and a wider ticket follows: 9 files where every setLevel is an unrestored literal pin, 19 pins in total, with a recommendation to share this CapturedLog helper.	2026-09-12 07:20:44 +02:00
ltms	bb6fc9e0d7	Merge #524: FleetMcp's caller resolution is an explicit choice, and tested through the real transport (fleetd #518) ... Adjudicated and verified by me, by my own build and my own mutation. This is the strongest PR of this batch and it closes #518 properly. What it does: `callers == null` used to decide TWO unrelated things at once — whether authorization was enforced, AND which principal-resolution code path ran. Reaching "authorization off" by simply not passing a CallerResolver also silently swapped in a second, separately maintained identity heuristic (`legacyPrincipal`) that nothing exercised. That is the fallback-reached-by-omission shape #415 named. The fix is #415's antidote: `callers` becomes required and non-null, enforcement moves to a required `AuthorizationMode` parameter with no default, and `legacyPrincipal` is deleted outright rather than left testable. Verified by me on the merged revision: * `mvn clean install` BUILD SUCCESS, Tests run: 1697, Failures: 0, Errors: 0, Skipped: 0 * exactly one FleetMcp constructor and four construction sites, all passing the new parameter — so "authorization off" is now a compile error to reach by omission, not a silent default * `legacyPrincipal` is gone: 0 declarations, 0 calls. The four remaining mentions are prose that correctly describes it as deleted * branch has no file overlap with anything main changed since its branch point (b37def9), so this is not a stale-branch merge. The 1697 vs main's 1698 is explained: this branch predates #522's two new tests The mutation that matters, run by me. I reinstated exactly the heuristic this PR deletes — resolve from the connection only, never reading the Authorization header: * `FleetMcpContextExtractorTest` fails by name: `a valid bearer token must resolve as PRIMARY and pass fleet_whoami's READ gate: unauthenticated: anonymous may not READ ==> expected: <false> but was: <true>` * and then the decisive measurement — with that mutation still applied I ran the **whole** suite: Tests run: 1697, **Failures: 1**, and the single failure is the new test class. All 20 FleetMcpAuthzTest cases pass with the resolver bypassed, as do the other 1676 tests. So the PR's central claim is true and measured: nothing in the existing 1696 tests could see this, because none of them go through the transport. `denyFor` had a full policy table, `CallerResolver.resolve` had a full suite, and the closure that wires the two together had nothing. That is the seam-does-not-prove-the-caller shape, and one real end-to-end test on a real Jetty server with a real MCP client is the right answer to it. Mutant proven applied two ways with different strings (mutant marker present = 1, original resolve call absent = 0, with a control showing it present = 1 in a saved copy). Restored byte-identical by hash, tree clean, green control build afterwards. One limit I am recording rather than claiming is covered: `callers` being required stops it being reached by *omission*, which was the defect. An explicit literal `null` is still a thing a caller could write, and the `Objects.requireNonNull(callers, "callers")` that catches it has no test of its own. That is the intended bar, not a gap worth a ticket. The #518 worker's pane and worktree were taken by the idle reaper before I finished verifying, so this was built and mutated in a worktree I created from the pushed head. Nothing was lost — the branch was pushed and clean.	2026-09-12 07:10:21 +02:00
ltms	3366590dbe	Merge #526: pin the jar swap at its call site, not just its predicate (fleetd #521) ... Adjudicated by me. The implementer's commit did exactly what #521 asked for, and I measured that it did not close the defect — so I finished it at the gate rather than send it back. The gap was in my ticket, not their work. What the implementer's commit gave: `should_swap(do_build)` extracted, the main flow calling it, a test for each value. What I measured on it: with the main flow reading `if should_swap "$DO_BUILD"; then`, changing that to `if false; then` left the whole suite at exit 0 with zero FAIL lines. The swap still never ran. Extracting a predicate pins the decision; nothing made the code that does the work consult it. Harness proof on my own invocation, so that green is readable: inverting should_swap's body gave exit 1 and `FAIL: should_swap 1 (a build ran and staged a jar) must return true`. The suite can fail when I run it. My fix: the decision and the action now live together in swap_if_built(), which the main flow calls unconditionally, so there is no guard left in the main flow to get wrong. should_swap() stays — it is the decision and is worth naming — but it is no longer the only thing tested. Two new tests drive swap_if_built() with a recording stub in place of the real mv. Two more things I fixed, both found while verifying: * The ordering test had to follow the call site to `swap_if_built "$DO_BUILD"`. Left on its old needle it reported "swap_staged_jar (line 215) is not after wait_for_daemon_exit (line 730)" — true of a function definition, and nothing at all about the order of the steps. * That test's three `[ -n ... ] || fail "could not find ... call site"` guards were dead code. Under `set -euo pipefail`, an absent needle fails the assignment and `set -e` kills the suite before the guard runs. Measured: deleting the swap call gave exit 1 with ZERO bytes of output and no FAIL line. Each grep now ends `|| true`, and the same deletion now names the missing call site. Verified by me on the merged revision: * suite exit 0, 0 `^FAIL:` lines, 44 test functions defined and 44 invoked * bash -n rc=0 on both scripts under /bin/bash 3.2.57 and bash 5.3.9 * four mutations, each killed with its own named FAIL line, each restored byte-identical by hash, green control after the battery: - guard removed inside swap_if_built -> "must not swap, but it did" - guard inverted -> "must perform the swap, and did not" - should_swap's comparison changed -> "must return true" - main-flow call deleted -> "could not find the swap call site" * CI green on 08771e2 (run 1775) * main has not touched either file since the branch point, so this is not a stale-branch merge A correction to my own method, recorded so the numbers are readable: in my first battery the "original gone" column read 0 for three cells because I left `\"` inside an already-single-quoted grep pattern, so the backslashes went into the pattern and it matched nothing. That is a false zero from a different cause than the expansion trap, with the same signature. Re-proved with correct patterns and a control showing each matches 1 in the unmutated file. Not fixed here, filed as #528: drain_gate_refusal has the identical shape. Replacing `die "$(drain_gate_refusal ...)"` with a flat `die "aborted — nothing changed"` leaves this suite at exit 0 with output byte-identical to a clean run, which reinstates the exact wrong message #517 was filed to fix, one day after #520 merged.	2026-09-12 07:01:55 +02:00
ltms	01adc841fa	Merge #523: test the policy probe's parsing guards (fleetd #519) ... Adjudicated and verified by me, not taken from the PR body. What I measured on the merged revision (099b2ecf… for the worker's own commit, b5843ab for the head I merged): * 5 test functions defined, 5 invoked; suite exit 0, "PASS: probe member credentials guards" * bash -n rc=0 on both scripts under /bin/bash 3.2.57 and bash 5.3.9 * two mutations killed, each proven applied two ways (mutant present AND original gone), restored byte-identical, with a green control after each: - dropping the empty-parse special case -> FAIL: empty parser output count: missing policy parser (jq) returned 0 field(s) - arity threshold 5 -> 0 -> FAIL: short parser output status The PR's own mutation proofs were run against revision 0e243e03, before its final edit, so I re-ran them against what I actually merged. Two things I fixed at the gate rather than sending back: * The refactor stranded about 25 lines of explanatory comments at the old parse site — including "Check the count here" pointing at a function call instead of the check, and a pipefail note saying "handled below" about code now above it. That is the same wrong-stated-fact defect class as fleetd #500, in the very file whose ticket history is about it. Moved each block above the code it explains. * Two assertions matched on `jq) returned N field(s)`, a needle starting mid-parenthetical, so a real failure printed "missing jq) returned 0 field(s)" and read as if the script's message had an unbalanced paren. Widened to `policy parser (jq) returned N field(s)`, which also pins that the refusal names the parser it used. Caveats recorded, from the implementer and not re-checked by me: the harness does not cover the non-member, missing-parser, curl-fetch, known-count, or hash-tool fallback paths. One property worth noting in favour of this suite: it runs under `set -e`, so the first failing test aborts before the final `printf 'PASS: …'`. That PASS line is reachable only from the fully successful path.	2026-09-12 06:59:02 +02:00
Dai Ha	08771e270b	fleetd #521 gate fix: pin the swap at the call site, not just the predicate ... The extraction in the previous commit did what #521 asked for — a should_swap() predicate with a test for each value — and I measured that it does not close the defect. With the main flow reading `if should_swap "$DO_BUILD"; then`, changing that line to `if false; then` left the whole suite at exit 0 with zero FAIL lines. The swap still never ran, and a redeploy would still report success while starting on no jar. That is my ticket's fault, not the implementer's: "extract the decision so the suite can call it" pins the decision and never the wiring. Extraction moved the untested decision up one level instead of removing it. Fix: the decision and the action now live together in swap_if_built(), which the main flow calls unconditionally — there is no guard left in the main flow to get wrong. should_swap() stays, because it is the decision and is worth naming and testing on its own. Two new tests call swap_if_built() with a recording stub in place of the real mv, so they fail if the guard is removed, inverted, or stops being consulted. Also fixed, found while verifying this: * test_swap_ordered_after_wait_and_before_start had to follow the call site to `swap_if_built "$DO_BUILD"`. Left on the old needle it reported "swap_staged_jar (line 215) is not after wait_for_daemon_exit (line 730)" — true of a function definition, and nothing about step order. * That test's three `[ -n ... ] || fail "could not find ... call site"` guards were dead code. Under `set -euo pipefail` an absent needle fails the assignment and `set -e` kills the suite before the guard runs. Measured: deleting the swap call gave exit 1 with ZERO bytes of output, no FAIL line, nothing naming what was missing. Each grep now ends in `|| true` so the assignment succeeds empty and the guard can speak. Verified by me on this revision: * suite exit 0, 0 `^FAIL:` lines, 44 tests defined and 44 invoked * bash -n rc=0 on both scripts under /bin/bash 3.2.57 and bash 5.3.9 * four mutations, each killed with its own named FAIL line, each restored byte-identical, green control after the battery: - guard removed inside swap_if_built -> "must not swap, but it did" - guard inverted -> "must perform the swap, and did not" - should_swap's comparison changed -> "must return true" - main-flow call deleted -> "could not find the swap call site in redeploy-fleetd.sh" (this one printed 0 bytes before the dead-guard fix, which is the before/after proof for it) Not fixed here, filed separately: drain_gate_refusal has the same shape. Replacing `die "$(drain_gate_refusal ...)"` with `die "aborted — nothing changed"` leaves the suite at exit 0 with output byte-identical to a clean run, which reinstates the exact wrong message #517 was filed to fix.	2026-09-12 11:58:34 +07:00
Dai Ha	b5843ab43f	fleetd #519 review fix: widen two needles to the whole parenthetical ... Both arity assertions matched on `jq) returned N field(s)` — a needle that starts in the middle of the script's `(parser name)` parenthetical. On a real failure the harness prints `missing <needle>`, so the line came out as: FAIL: empty parser output count: missing jq) returned 0 field(s) which reads as if the script's own message had an unbalanced paren. It does not; the needle was just sliced. Matching on `policy parser (jq) returned N field(s)` makes the failure readable and also pins that the refusal names the parser it used, which the narrower needle did not. make_jq() PATH-prefixes a fake jq, so `_PARSER_NAME` is deterministically "jq" in both tests; the wider needle cannot flake on a host without jq. Re-proved on this revision, because a disproof is about a revision and not a file: * suite exit 0, "PASS: probe member credentials guards" * bash -n rc=0 on the test under /bin/bash 3.2.57 and bash 5.3.9 * dropping the empty-parse special case -> FAIL: empty parser output count: missing policy parser (jq) returned 0 field(s) * arity threshold 5 -> 0 -> FAIL: short parser output status * script restored byte-identical after each, green control after both	2026-09-12 11:50:11 +07:00
Dai Ha	d8985719eb	fleetd #525: restore the logger level, not just the appender, in SessionManagerTest ... The finally block in onTurnFailedIsLoggedAtWarnWithThePriorState (and four other tests in this file) called setLevel(Level.WARN) on the shared SessionManager logger (one test used MemberRegistry.class) but only detached the appender in finally, never restoring the level. Since logback Logger instances are cached per class and shared across the whole JVM, the pinned level leaked into every test that ran after it. Add CapturedLog, a small AutoCloseable that captures a logger's level and appender together and restores both on close via try-with-resources, so this shape cannot be half-fixed again. Convert all 7 addAppender/setLevel call sites in this file to it, including the 2 sites #522 already fixed locally (kept per-test pinning as belt-and-braces). Add a proving test (sharedSessionManagerLoggerLevelIsRestoredAfterOnTurnFailedPinsWarn, @Order(2), running right after the fixed test at @Order(1)) that fails before this fix and passes after it. Verified with a mutation: dropping the level restore in CapturedLog.close() turns the proving test red with 'expected: <DEBUG> but was: <WARN>'; restoring is byte-identical to the pre-mutation file (sha256 matched) and the suite goes green again. mvn -f fleetd/pom.xml clean install: Tests run: 1699, Failures: 0, Errors: 0, Skipped: 0. BUILD SUCCESS.	2026-09-12 11:49:03 +07:00
Dai Ha	5b1e13ca3d	fleetd #519 review fix: move the parser comments with the code they explain ... PR #523 extracted parse_policy_fields() but left about 25 lines of explanatory comments at the old parse site in main(). That is the same defect class as fleetd #500 itself — a stated fact that no longer matches the code next to it — in the very file whose ticket history is about it. Three blocks moved, no code touched: * "One parse pass" + the mapfile/process-substitution reasoning now sits above parse_policy_fields(), which is what it describes. * The arity-check block now sits inside the function, directly above `if (( ${#_FIELDS[@]} < 5 ))`. At the old site it said "the slice just below this" and "every line below this expects", both pointing at a function call rather than the check. Reworded to name main() and its slice explicitly. * The pipefail note said the parser failure was "handled below"; the handling is now above it, in the function. The call site keeps a three-line pointer saying where the reasoning went. Checked myself, on this revision: * suite exit 0, "PASS: probe member credentials guards" * bash -n rc=0 under /bin/bash 3.2.57 and bash 5.3.9 * two mutations killed, each proven applied two ways (mutant present AND original gone), restored byte-identical, green control after each: - dropping the empty-parse special case -> FAIL: empty parser output count - arity threshold 5 -> 0 -> FAIL: short parser output status	2026-09-12 11:47:16 +07:00
Dai Ha	c89a375e5d	fleetd #521: extract should_swap so the swap guard can't be silently disabled ... Mutating the swap step's guard (if [ "$DO_BUILD" = 1 ] -> if false) left the whole test suite green: test_swap_ordered_after_wait_and_before_start only checks source positions, which an in-place if-condition edit never moves. Extracts the decision into should_swap(do_build), following the same shape as #510's wait_for_daemon_exit and #517's drain_gate_refusal, with a direct test for each value.	2026-09-12 11:42:46 +07:00
ltms	37dcefa834	Merge #522: log a positive drain-completion line when drainAll finishes (fleetd #512 part 1) ... Verified by the lead, not taken from the worker's report. Build run by me, unpiped: exit 0, "Tests run: 1698, Failures: 0, Errors: 0, Skipped: 0", BUILD SUCCESS. That is +2 on main's 1696, matching the two new tests. Both merge constraints from the ticket hold, checked against the diff: - The log.info is the last statement of drainAll's normal path and is NOT in a finally. The only finally in the file is at :372, unrelated. So a drain that dies still leaves no line, which is the absence signal part 2 will alert on. - released and abandoned are incremented inside the drainSnapshot loop, not read from a collection at the end, so a partial report can never be served by this line. Mutation check run by me: deleting the completion line makes both new tests fail by name — drainAllLogsACompletionLineWithTheRealCountsOnACleanDrain and drainAllLogsANonZeroAbandonedCountForASessionStillBusyAtTheDeadline, "no drain-complete INFO logged, expected true but was false". Restored byte-identical (d21ecd3adb3f66933e2f248a8ec81a2c324bfee323a3cc30da525842c33ea80a). So the tests really depend on the line rather than passing for another reason. Design accepted: drainSnapshot returns a private DrainTally record and release() returns the removed session so abandoned can reuse the same BUSY-at-removal check logPreservedForShutdown makes, instead of a second registry read. Both are private with one call site. Two follow-ups filed rather than fixed here, see the ticket comments.	2026-09-12 06:34:51 +02:00
Dai Ha	6a7342b1f0	fleetd #518: make the FleetMcp caller-resolution wiring an explicit choice, and test it once for real ... FleetMcp's contextExtractor picked its principal-resolution path off `callers == null`, so "authorization off" also silently swapped in a second, untested identity heuristic (legacyPrincipal). Nothing drove that closure through a real MCP request, so the whole wiring was an unexercised claim. - callers (CallerResolver) is now required, never null. - A new AuthorizationMode enum (ENFORCED/UNENFORCED) is a required constructor parameter with no default, replacing the null-means-legacy idiom for whether denyFor enforces at all. - legacyPrincipal is deleted: there is exactly one resolution path now (callers.resolve(...)), so the mutation that swapped it for an unconditional legacy call no longer compiles ("cannot find symbol: method legacyPrincipal"). - FleetMcpContextExtractorTest boots the real transport on a real Jetty server and drives it with a real MCP client, proving fleet_whoami's resolved role comes from CallerResolver's token check. - Adapted FleetMcpAuthzTest/FleetMcpHandoverTest call sites; theLegacyConstructorLeavesTheGateOpen keeps its meaning under the new AuthorizationMode.UNENFORCED value.	2026-09-12 11:34:12 +07:00
Dai Ha	a5ad7c6561	fleetd #519: test policy probe guards	2026-09-12 11:30:35 +07:00
ltms	c71ac231e5	Merge #520: pin the drain-gate abort branch and jar_id's absent case (fleetd #517) ... Verified by the lead, not taken from the worker's report: - 40 test functions defined, 40 invoked (my own greps), suite exit 0, zero real FAIL lines. - Harness proof on my own invocation: re-applying the jar_id absent->present mutation gives exit 1 and "FAIL: jar_id with no arguments must report absent when $JAR does not exist". So a green run from this suite is readable. - Script restored byte-identical after every mutation: 2cb83dc380c7226191d657c40fccdfc856904e40b2d03d0851fb6e522cee2f41. - drain_gate_refusal is pure and prints only; die() stays outside the command substitution, so the "die inside $( ) exits only the subshell" trap does not apply here. - The --no-build + staged-present case reads "nothing changed" deliberately, and that is correct: --no-build stages nothing itself, and a leftover staged jar is wiped by rm -f "$JAR_STAGED" at :607, before the build at :610. The worker's out-of-scope finding is real and I reproduced it: the swap guard at :723 can be set to `if false` with the suite still green at exit 0. Filed separately.	2026-09-12 06:29:08 +02:00
Dai Ha	33720c42b3	fleetd #512 (part 1): log a positive completion line when drainAll finishes ... drainAll used to log nothing on a clean drain — both existing log calls (drainSnapshot's per-session failure, drainAll's straggler-sweep warning) sit on abnormal paths, so "drained fine" and "died on the first session" looked identical: no log line either way. Add one log.info at the end of drainAll: "drain complete: released=N abandoned=M (still BUSY at the shutdown deadline)". It fires on the normal path, including the all-zero case, and folds both drainSnapshot passes (main snapshot + straggler sweep) into one line. drainSnapshot now returns a private DrainTally(released, abandoned) record instead of void, and the private release(paneId, cause) overload now returns the removed MemberSession (previously void) so drainSnapshot can read its state at the moment of removal — the same check logPreservedForShutdown already makes. Both signature changes are private with a single call site, so the blast radius stays small.	2026-09-12 11:29:01 +07:00
Dai Ha	3833d8e52b	fleetd #517: pin the drain-gate abort branch and jar_id's absent case ... Two mutation-testing survivors in scripts/redeploy-fleetd.sh: a source-text test pins what a message SAYS but never whether the branch that prints it is REACHED. - Extract the drain-gate abort decision into drain_gate_refusal(do_build, staged_path), a pure function the suite can call directly for all four build/staged combinations. The existing source-text grep test is kept alongside it (it catches a re-wording; the new tests catch a dead branch). - Extend jar_id's test to cover the missing-file path (both the no-argument default and an explicit path), which the #511 test never exercised.	2026-09-12 11:23:37 +07:00
ltms	b37def9238	Merge #516: the probe refuses with three distinct messages, each naming its own cause (fleetd #500)	2026-09-12 06:09:50 +02:00
ltms	8f02576df6	Merge #515: pin the two-client completeness fold, and legacyPrincipal earns no authority (fleetd #509)	2026-09-12 06:06:12 +02:00
ltms	525bc1c5f4	Merge #514: the drain-gate abort message names a recovery that works, and jar_id()'s default is pinned (fleetd #511)	2026-09-12 06:00:23 +02:00
Dai Ha	d59ece6dec	fleetd #500: stop a wrong-interpreter or failed-parse reading a policy as empty ... probe-member-credentials.sh used mapfile < <(producer) to parse the fetched policy. That hides a producer failure three ways: mapfile is bash 4+ and missing on macOS's /bin/bash 3.2, a process substitution's exit status is never propagated to mapfile, and the downstream reads (":-" defaults and a slice) never fire set -u on a short or unset array. All three converge on the same "0 known names" refusal, which blames the policy for a failure that is actually the interpreter or the parser. Three distinct guards, each closing one cause with its own message: - a BASH_VERSINFO gate at the top refuses outright on bash < 4 (exit 3) - the parser's output is captured via command substitution instead of mapfile < <(...), so a non-zero jq/python3 exit is caught at the call while the fact still exists (exit 4) - an arity check before the field slice refuses a parse that exits 0 but returns fewer than 5 fields (exit 5) The existing "0 known names" guard is now honest: by the time it fires, the three causes above are already ruled out, so it really does mean the policy has 0 known names.	2026-09-12 10:58:16 +07:00
Dai Ha	32408d1e64	fleetd #509: pin the pane-scan completeness fold, and stop legacyPrincipal handing out primary ... Unit 1 — PaneLocator.terminalForPid's completeness fold across herdr clients (PaneLocator.java:117) had no test that varied the number of clients, so a mutation that keeps only the last client's Lookup.complete() instead of ANDing every client's outcome survived: 14 of 15 existing tests agree with the mutant on a single client. Added a two-client test where the lead client errors on the pane that would have owned the pid (an incomplete, negative scan) and the member client cleanly finds no panes (a complete, negative scan) — the real fold ANDs these to false, a last-wins fold reads it as true. Proved against MUTANTC (complete = outcome.complete();): the new test fails with "expected: <false> but was: <true>", the file was restored byte-identical (sha256 unchanged), and the control run is green. Unit 2 — FleetMcp.legacyPrincipal's else-branch returned Principal.primary for ANY caller the connection did not resolve to a worker pane, with none of CallerResolver.java:254's isLoopback/scanComplete guards. Measured that no production caller passes null callers (Fleetd.java:696 always constructs a real CallerResolver) but FleetMcpAuthzTest.mcp(false) legitimately does, for its "legacy constructor leaves the gate open" test — so the null-callers path is not dead code to delete (option a), it is a documented legacy mode (option b). Changed the else-branch to Principal.anonymous() and widened legacyPrincipal to package-private (like denyFor) so a new test pins the behavior directly, since it only ever ran inside a contextExtractor closure no existing test triggers.	2026-09-12 10:58:09 +07:00
Dai Ha	6e23bf8309	fleetd #511: fix wrong --no-build wording in drain-gate abort, pin jar_id() default ... The drain-gate abort message told the operator a rerun "with or without --no-build" would finish the restart. That is wrong: by the time this message can fire, stage_built_jar has already moved the jar off $JAR, so --no-build hits require_no_build_jar's own refusal. Reworded to say the rerun must NOT use --no-build, and why: the built jar is no longer at the live path that --no-build requires. Also added a test pinning jar_id()'s no-argument default (reports $JAR, the live path) and its explicit-argument behavior (reports that path instead), per fleetd #511 item 2. Not adding a test for the JAR_STAGED rm -f at line 578 (fleetd #511 documents it as an equivalent mutant — mvn clean install deletes target/ on the next line regardless).	2026-09-12 10:56:32 +07:00
ltms	aa4c0b84c3	Merge #510: never build into the path a running daemon holds (fleetd #493)	2026-09-12 05:44:16 +02:00
ltms	40c593cd09	Merge #508: a herdr error during the pane scan is refused, not promoted to primary (fleetd #505) ... Closes the worker->primary escalation that fleetd #317 left open on the other side of the same ternary. #317 stopped an UNRESOLVABLE pid being promoted. This stops a RESOLVED pid whose pane scan failed being promoted: the scan now reports completeness, and an incomplete scan resolves to anonymous. Shape chosen: PaneLocator.terminalForPid returns Lookup(terminal, complete); paneOwnsAnyOf returns a private Ownership enum OWNS/DOES_NOT_OWN/UNKNOWN, so a HerdrException is UNKNOWN rather than a clean negative. ConnectionIdentity.Caller gains scanComplete as a third field -- resolved() was NOT widened, correctly: it is documented as testing the lsof sentinel and #505 is a different axis. A definite match still short-circuits, so a genuinely vanished non-owning pane does not become a refusal. Verified by me, not taken from the report. Trial-merged onto main (136312f) and built the merge: mvn clean install -> Tests run: 1694, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS Harness proof, by re-running the worker's own mutation rather than trusting it (UNKNOWN -> DOES_NOT_OWN): Tests run: 63, Failures: 3 -- exactly the three tests meant to pin it, matching their report line for line: CallerResolverTest.aHerdrErrorOnTheOwningPaneDuringTheScanIsRefusedNotPromotedToPrimary ConnectionIdentityTest.scanIsIncompleteWhenHerdrErrorsOnThePaneThatOwnsThePid PaneLocatorTest.anErrorOnThePaneThatOwnsThePidMakesTheScanIncompleteNotAClearNegative So the cells can go red, and their numbers are honest. Restored, shasum 81b797a6... byte-identical, control 63/0. MY OWN MUTATION, on the half they did not touch, FOUND A GAP. I mutated the two-client completeness fold in terminalForPid -- `complete = complete && outcome.complete()` -> `complete = outcome.complete()`, which forgets an earlier client's failure. Two greps proved it applied. Result: Tests run: 63, Failures: 0. NOT pinned. It is not an equivalent mutant: with CB-185's two herdr daemons, a first client that scans incompletely followed by a second that scans cleanly gives `false` originally and `true` mutated, so an incomplete scan would be reported complete and promoted. It is the #393 shape instead -- no test assembles that combination, because the single-daemon tests collapse lead and member to one object. That axis has cost us before: four routing defects passed the suite when one client served both roles with memberHerdrSocket unset. Merging anyway, because the production behaviour is correct and the ticket's defect is pinned three ways -- what is missing is a test for a dimension the PR description claims in prose. Filed as a follow-up with the numbers above. Second item for the follow-up, latent not live: FleetMcp.legacyPrincipal still does `c.terminal() != null ? worker : primary` and drops scanComplete, so it would re-open exactly this hole. Measured unreachable in production -- Fleetd.java:696 always passes a real CallerResolver, and the contextExtractor only calls legacyPrincipal when `callers == null`. A defect on paper is not a reachable defect, so it is not a blocker; it is a trap for whoever next touches that constructor. That second item is also the fleet01 lead's structural point, and it is the sharper framing: #415's antidote (make the decision total) is ORTHOGONAL to this defect. Authz.permits is already a default-less switch over Action and cannot help, because it says nothing about whether the caller was resolved correctly, and Principal carries no record of how it was resolved. #415 guards a decision nobody wrote; this was a decision written correctly and fed a bad input. The durable fix direction is that an unresolved scan should be unrepresentable as a principal rather than checked for at each consumer.	2026-09-12 05:35:41 +02:00
Dai Ha	979adf82eb	fleetd #493: never build into the path a running daemon holds ... redeploy-fleetd.sh's build step wrote straight into fleetd/target/fleetd.jar via `mvn clean install` while the OLD daemon was still running from that exact path. A JVM loads classes lazily, so a class the daemon had not touched yet could be read from a jar already replaced or removed -- the failure landed on the shutdown drain (NoClassDefFoundError, exit 143, looks clean). Stage the freshly built jar at target/fleetd-new.jar (stage_built_jar), confirm the OLD pid has actually exited (wait_for_daemon_exit, extracted from the existing wait loop), and only then swap it into the live path (swap_staged_jar) -- strictly after the wait, strictly before start. A failed swap dies without starting. --no-build and --check keep their existing, truthful behavior (require_no_build_jar; jar_id still reads the live path by default). A leftover staged jar from an interrupted run is wiped before the next build. The build still runs before anything is stopped, so a failed build still never takes the fleet down. Also fixed: the drain-gate abort message ("aborted -- nothing changed") now names the staged jar when one exists, since staging already moves the freshly built jar off the live path before that prompt runs. Adds unit tests for stage_built_jar, swap_staged_jar, require_no_build_jar, wait_for_daemon_exit, and a source-order test proving swap sits after the wait and before start (sourcing stops before the main flow ever runs, so the ordering itself can only be checked by reading the script's own call sites).	2026-09-12 10:35:19 +07:00
Dai Ha	36870836aa	fleetd #505: a herdr error during the pane scan must not read as a clean negative ... A transient herdr error on pane.process_info during PaneLocator's pid→pane scan used to be swallowed into a plain "does not own it", so a real worker whose owning pane errored mid-scan resolved with a null terminal but a resolved (real) pid — exactly what CallerResolver's loopback-trust fallback reads as the primary. That is a worker→primary privilege escalation through the door fleetd #317 did not close: #317 guards a failed lsof lookup (c.resolved()), not a failed herdr pane scan. Fix: add a third state to the scan instead of widening Caller.resolved() (which stays centralised next to the lsof sentinel it tests, per #505's explicit instruction not to reopen that decision). PaneLocator.terminalForPid now returns a Lookup(terminal, complete) record: a HerdrException on one pane marks that pane's ownership UNKNOWN, not DOES_NOT_OWN, and the scan is complete only if every pane was either matched or confirmed not to own the pid. A definite match found elsewhere in the same scan still short-circuits as complete — a pane that genuinely vanished mid-scan without being the caller's own does not turn into a refusal. ConnectionIdentity.Caller carries the new scanComplete flag alongside the unchanged resolved(). CallerResolver's loopback-trust fallback now requires both resolved() and scanComplete() before promoting to Principal.primary(); an incomplete scan resolves anonymous, which fails toward the recoverable error (a refused primary retries loudly; a promoted worker would not). Logs a warning naming the pane and which herdr client (of how many) failed, so the incomplete-scan path is diagnosable rather than silent (fleetd #317's own lesson).	2026-09-12 10:27:42 +07:00
ltms	136312fb11	Merge #503: Injector's readiness-grace warn prints measured elapsed time, never arithmetic on constants (fleetd #501) ... Verified by the lead, not taken from the worker's report. Trial-merged onto main (4f28da6) and built the merge, because a clean auto-merge is not a compiling merge: mvn clean install -> Tests run: 1689, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS InjectorTest -> Tests run: 34, Failures: 0 Gitea CI on ac351ee: success The worker mutated the two log arguments. I mutated the half it did not: the STAMP SITE. Changing the first-sample guard so the clock is re-stamped on every non-ready poll gave 1 failure, InjectorTest.readinessGraceExpiryLogsTheMeasuredElapsedTimeNotArithmeticOnConstants, on a clock-read counter assertion: "the readiness-not-ready branch should read the clock exactly twice — once to stamp the first non-ready sample, once at grace expiry". Two greps proved the mutant applied; restored, shasum byte-identical, control 34/0 green. Behaviour preserved: the restructure from `else if (p != null && ++t.notReadySincePoll >= N)` to a nested `else if (p != null) { ... }` keeps the short-circuit, so the counter still increments only when p != null. All three reset sites now clear notReadySinceMillis alongside notReadySincePoll. Two notes for the record, neither a blocker. The poll-count half is an equivalent mutant and the worker said so instead of reporting a kill it did not get. That is the right call and the code comment states the limit honestly. The clock is injected through a package-private constructor overload while the public constructors default it to System::currentTimeMillis. My brief prescribed that shape. With exactly one production construction site (Fleetd.java:531) a required parameter — fleetd #415's antidote — would have been just as cheap and would match LeadRollover and the new Fleetd.awaitHerdr. The silent-survivor risk #415 names is not present here, because the field is final and both public constructors delegate, so a future constructor cannot compile without supplying it. Recording the choice so the next person does not read it as an oversight.	2026-09-12 05:07:35 +02:00
ltms	4f28da62a3	Merge #499: redeploy-fleetd.sh gains an "unclear" supervisor state that cannot reach kill, and the detail survives the subshell (fleetd #492, #497) ... Two commits, b17f37a and 599419f. Verified by the lead, not taken from the worker's report. b17f37a adds the fifth answer "unclear" for installed-but-not-loaded and for a probe that could not answer at all, and makes require_drivable_supervisor die on it. 599419f fixes a defect I found in b17f37a: SUPERVISOR_UNCLEAR_DETAIL was set inside detect_supervisor, which is called as $(detect_supervisor). A subshell only returns stdout, so the detail never reached the caller, and a ${VAR:-generic} fallback then printed a message naming no supervisor and no reason. Measured before the fix: plain call -> detail length 142; via $( ) -> length 0. Lead verification of the fix, re-running my own original measurement across every branch: launchd installed, not loaded kind=[unclear] detail_len=142 systemd installed, not active kind=[unclear] detail_len=146 launchd loaded (normal) kind=[launchd] detail_len=0 systemd loaded (normal) kind=[systemd] detail_len=0 neither -> none kind=[none] detail_len=0 both loaded -> ambiguous kind=[ambiguous] detail_len=0 The separator is always emitted, so the ${RAW#*SEP} unpack cannot silently fall back to the whole string on the four branches that carry no detail. Harness proof, run by me: making detect_supervisor print the kind alone (two greps proved the mutant applied) turned the suite red, exit 1, "FAIL: detail does not name the systemd unit it found installed-but-not-loaded". Restored, shasum byte-identical, control exit 0, bash -n clean on both files. 23 test functions defined, 23 invoked. The three case "$SUPERVISOR_KIND" switches now have final arms, chosen by what each caller does with the value: the reporting switch warns and continues (a diagnostic that aborts goes silent exactly when the state is novel), while stop and start die. Not fixed here, filed separately: the "was already not running" branches at :603 and :610 swallow a failed launchctl unload / systemctl stop with || true and then print "ok" unconditionally.	2026-09-12 05:04:04 +02:00
ltms	708f1795ad	Merge #502: awaitHerdr reports three outcomes with measured elapsed time, not one boolean (fleetd #498) ... Verified by the lead, not taken from the worker's report. Trial-merged onto main (8f59019) locally and built the merge, because a clean auto-merge is not a compiling merge: mvn clean install -> Tests run: 1687, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS FleetdAwaitHerdrTest -> Tests run: 6, Failures: 0 Gitea CI on 274afaf: success The worker mutated the three return values. I mutated the half it did not: the reap DECISION at the call site. Changing logHerdrWaitOutcomeAndShouldReap to return true for INTERRUPTED gave 1 failure, FleetdAwaitHerdrTest.interruptedLogsItsOwnMessageAndNeverClaimsTheBudgetElapsed:135 ("an interrupted wait must not tell main to reap"). Two greps proved the mutant applied; restored, shasum byte-identical, control 6/0 green. Behaviour preserved: herdrUp is still true only for ANSWERED, and both its readers (the orphan reap at :276 and the lead auto-launch gate at :375) see exactly what they saw before. One correction to the worker's report, which changes nothing in the code. It wrote that a real interrupt-detection regression "would also hang the daemon's startup thread forever". It would not: in production `nanos` is System::nanoTime, so the deadline check still fires. The hang it hit was a test-fixture property — a frozen injected clock with no iteration bound. That is fleetd #486's shape, now seen in a second class, and it is recorded there.	2026-09-12 05:01:02 +02:00
Dai Ha	599419f9e6	fleetd #492 follow-up: carry the unclear detail across detect_supervisor's subshell boundary ... b17f37a set SUPERVISOR_UNCLEAR_DETAIL as a global inside detect_supervisor, but the real call site invokes it as $(detect_supervisor) — a subshell — so that global died with the subshell and the die() message's ${VAR:-fallback} silently masked the loss with generic text. - detect_supervisor now packs kind and detail onto its one stdout line (joined by the ASCII unit separator byte, $SUPERVISOR_DETAIL_SEP), the only channel that survives $( ). The real call site unpacks both with in-shell parameter expansion — no extra subshell. - Dropped the ${SUPERVISOR_UNCLEAR_DETAIL:-...} fallback at the die() message: under set -u, a missing detail now fails loudly instead of silently defaulting (same defect class as #497). - Added a constraints comment block above detect_supervisor for future callers: stdout-only, no ${VAR:-default} papering over a lost value, and every case on the return value needs an explicit *) arm. - Added *) arms to the three `case "$SUPERVISOR_KIND"` switches (report/stop/start): report warns and continues (display-only), stop/start die naming the value (they act on it). - Rewrote the "unclear" test to go through the real call-site shape ($(detect_supervisor) then the same split), not a hand-constructed value, and tightened its final assertion to check for the actual detail text rather than $SYSTEMD_UNIT alone (the die() boilerplate names the unit either way, so that check could pass on a lost value).	2026-09-12 09:58:45 +07:00
Dai Ha	ac351ee1de	fleetd #501: readiness-grace expiry logs measured elapsed time and the loop's own poll counter, never the configured budget ... The line at Injector.java:383-387 printed two numbers that read as measurements but were both compile-time constants: READINESS_GRACE_POLLS for the poll count (the loop's own Target.notReadySincePoll counter was in scope at the same call site), and READINESS_GRACE_POLLS * POLL_INTERVAL_MILLIS / 1000 for the elapsed time — arithmetic on two constants, never a measurement, and wrong in the direction that says everything ran on schedule. Fix, copying the LongSupplier-clock shape LeadRollover already uses: - print t.notReadySincePoll instead of the constant for the poll count (the two agree by construction on this branch, so no test can tell them apart — the comment says so honestly). - add Target.notReadySinceMillis, stamped at the first non-ready sample and reset at all three sites notReadySincePoll already resets (:304, :355, :389 pre-fix line numbers), to compute a real elapsed time at expiry. - inject a LongSupplier nowMillis (defaulting to System::currentTimeMillis) through new package-private constructor overloads so a test can supply a clock whose advance does not track POLL_INTERVAL_MILLIS. Tests use a ListAppender to assert on the log message contents, per LeadRolloverTest's pattern. The elapsed-time test drives the loop with a stub clock returning two literal, non-derived values so it can fail if the fix regresses to the constant-arithmetic line — proved by mutation: reverting the elapsed calculation to READINESS_GRACE_POLLS * POLL_INTERVAL_MILLIS turns that one test red (1 failure); reverting the poll-count print to the constant is an equivalent mutant (0 failures), because the counter and the constant are identical at that exact call site by construction. fleetd clean install: Tests run: 1683, Failures: 0, Errors: 0, Skipped: 0.	2026-09-12 09:57:57 +07:00
Dai Ha	274afafde6	fleetd #498: awaitHerdr distinguishes deadline-passed from interrupted, with measured elapsed time ... - awaitHerdr now returns a HerdrAwaitOutcome(HerdrWaitResult, elapsedNanos) instead of a bare boolean, so 'the wait budget genuinely ran out' and 'the waiting thread was interrupted' are two distinct, named states instead of the same false (fleetd #497's shape). - awaitHerdr takes the clock (LongSupplier) and the per-poll sleep (Runnable) as required parameters, with no defaulted overload (fleetd #415), so a test can drive it. - The startup call site is extracted into logHerdrWaitOutcomeAndShouldReap, since main() itself cannot be driven from a unit test; it logs a distinct message per outcome, always printing the measured elapsed time next to the configured budget, never the budget alone. - Adds FleetdAwaitHerdrTest covering the seam (all three outcomes, plus the preserved interrupt flag) and the call site (the three distinct log messages), using ListAppender.	2026-09-12 09:54:59 +07:00
ltms	8f59019305	Merge #496: lead-rollover logs measured elapsed time and measured counts, never the configured budget (fleetd #494) ... Three commits. All verified by the lead in a detached worktree, not taken from the worker's report. 3fb3311 — the /clear-settle and success paths return measured elapsed time and a real nudge count c87cc25 — print the measured nudge count; fix the same defect on the sibling turn-settle timeout line e966cba — the poll count on the same line was still a constant; the test could not tell the difference Lead verification at e966cba: mvn clean install -> Tests run: 1681, Failures: 0, Errors: 0, Skipped: 0, BUILD SUCCESS LeadRolloverTest -> Tests run: 33, Failures: 0, Errors: 0, Skipped: 0 Gitea CI on e966cba: success Mutation M2 (PICKUP_GRACE_POLLS 8 -> 5), run by the lead: 2 failures, clearGraceReleaseLogIsWarnWithMeasuredElapsed and successLogPrintsMeasuredElapsedForTheWholeRoll. The emitted line under the mutant read "after 5 consecutive IDLE/DONE polls (4 of those were nudged)", so both numbers now follow the loop. Restored, shasum byte-identical, control 33/0 green. Known and deliberate limit, stated in the code comment: on the grace-release branch both counters equal the constants by construction (one write site each), so no test can prove the difference on that line. The change buys one source of truth, not provable coverage. The place nudges genuinely varies with the run is the /clear-timeout warn, which is covered by a discriminating test.	2026-09-12 04:44:10 +02:00
Dai Ha	e966cbadf9	fleetd #494 follow-up (2nd pass): the grace-release line still had one constant, and its test could not tell the difference ... Two more fixes on the same line, LeadRollover.java:600-624: 1. The poll-count argument (second, was PICKUP_GRACE_POLLS) now prints the loop's own idlePollsAwaitingPickup counter instead of the constant. Same defect shape as the nudges fix from c87cc25, one argument over. 2. LeadRolloverTest's clearGraceReleaseLogIsWarnWithMeasuredElapsed asserted its nudge-count expectation as `LeadRollover.PICKUP_GRACE_POLLS - 1` — the same expression the production code used to build the log line from, so it could not discriminate a reverted fix. Rewritten to plain literals ("after 8 consecutive IDLE/DONE polls (7 of those were nudged)"), proven to trip when PICKUP_GRACE_POLLS's value changes (2 failures under a PICKUP_GRACE_POLLS=5 mutation: this test and successLogPrintsMeasured...). Also corrected the comment above the log line: idlePollsAwaitingPickup and nudges each have exactly one write site on this loop's release branch, so they cannot differ from PICKUP_GRACE_POLLS / PICKUP_GRACE_POLLS - 1 at this call site — confirmed by re-deriving the loop's control flow, and by reverting just the nudges argument (M1) and observing 33/0 stayed green even after the test rewrite. That is an equivalent mutant on this line, not a gap the test rewrite could close; the honest value of printing the counters is one source of truth for the loop, not a provable-by-test difference here. The place nudges truly varies with the run — and is covered by a test that can tell it apart from a constant — is the /clear-timeout warn's clearResult.nudges() in runRollover. mvn -Dtest=LeadRolloverTest test: 33/0. mvn clean install: 1681/0, BUILD SUCCESS. Same-shape sweep (found, not fixed, per instructions): Injector.java:383-387 — the readiness-grace warn prints the constant READINESS_GRACE_POLLS where the measured per-target counter t.notReadySincePoll is in scope (single increment site, just reached the threshold at this call site — same equivalent-mutant situation as this fix).	2026-09-12 09:39:39 +07:00
Dai Ha	b17f37a683	fleetd #492 follow-up: detect_supervisor must never read "could not tell" as "none" ... Two situations were silently landing in the "none" answer, which require_drivable_supervisor accepts and the script then falls back to a raw kill + nohup — exactly the wrong move when a supervisor actually IS present: - installed-but-not-loaded, on either supervisor. `systemctl --user is-active` answers "no" for activating/deactivating/failed and while an auto-restart is pending too, and every one of those is a host that IS under systemd (or launchd) and about to act again. `*_installed` already knew this; it was only ever consulted for a warning line, never by the decision itself. - a systemd probe that could not answer at all (e.g. systemctl cannot reach the user bus over a non-lingering ssh session) looked identical to a clean negative, because both probes redirected stderr straight to /dev/null. detect_supervisor now returns a fifth answer, "unclear", for both cases. systemd_loaded/systemd_installed capture systemctl's exit status and stderr separately and set their own *_ERRORED flag only on a real tool failure (non-zero exit WITH stderr), never on a clean negative. "none" now means only: neither supervisor installed, neither loaded, neither probe errored. require_drivable_supervisor die()s on "unclear" exactly like it already does on "ambiguous", naming the specific supervisor and reason via the new SUPERVISOR_UNCLEAR_DETAIL global. Tests: 4 new cases (systemd/launchd installed-but-not-loaded, a real systemd_loaded run through a systemctl stub that errors on stderr, and the die() refusal for "unclear" naming the unit). All 3 new guards were verified by mutation: each was removed from the real script, the suite caught it (a new FAIL line naming the exact broken assertion), then the file was restored byte-identically and the suite went green again.	2026-09-12 09:23:21 +07:00
Dai Ha	c87cc25aa6	fleetd #494 follow-up: print the measured nudge count, and fix the sibling turn-settle timeout line ... - waitForClearPickupAndSettle's grace-release warn now prints the measured 'nudges' counter instead of the constant PICKUP_GRACE_POLLS - 1. The two happen to agree today, but the constant expression was wrong once before (printed PICKUP_GRACE_POLLS itself, claiming 8 nudges where 7 went out) and a code read did not catch it — only a mutation test did. Printing the counter cannot drift from the loop's real behaviour. - waitUntilAtTurnBoundary (the FIRST wait, ~line 386-391) had the identical 'configured value printed as if measured' defect as the three lines fixed in the original #494 commit, but was out of scope because the brief named specific lines instead of the shape. Fixed the same way: it now returns a TurnSettleResult(settled, elapsedMillis) instead of a bare boolean, and the timeout warn prints 'configured={}s elapsed={}ms' instead of presenting cfg.turnSettleSeconds() as the measured wait. - No behaviour change: same sends, same order, same release/refuse decisions. - Test additions: clearGraceReleaseLogIsWarnWithMeasuredElapsed now also asserts the measured nudge count; successLogPrintsMeasuredElapsedForTheWholeRoll's expected elapsed value is updated (7000ms, not 6500ms) to account for waitUntilAtTurnBoundary's own new clock read; a new turnTimeoutLogPrintsMeasuredElapsedNotJustConfigured test pins the sibling line. - Proved the nudges fix with a temporary mutation: set PICKUP_GRACE_POLLS to 5, confirmed via two greps that the mutant applied and the original constant was gone, ran the grace-release test and read the actual log line — nudge count followed to 4 (= 5 - 1), then restored to 8 and reran the full LeadRolloverTest suite as a control (33/33 green).	2026-09-12 09:22:19 +07:00
Dai Ha	3fb331145a	fleetd #494: log measured elapsed time, never the configured budget, on a lead-rollover failure/success ... - runRollover's /clear-timeout warn now prints configured/elapsed/nudges, each labelled, instead of presenting cfg.clearSettleSeconds() as if it were the measured wait. - waitForClearPickupAndSettle's pickup-grace release is now log.warn (was log.info) and prints the measured elapsed time next to the target pane — this is the exact path that reported a false-success roll in the real incident (438ms of a 20s budget). - The success line ('lead-rollover: rolled') now prints the measured elapsed time for the whole roll. - waitForClearPickupAndSettle now returns a ClearSettleResult(settled, elapsedMillis, nudges) instead of a bare boolean, so callers can log the measured values instead of the config. - No behaviour change: same sends, same order, same release/refuse decisions. - Adds 3 tests to LeadRolloverTest pinning the content of each changed log line, using a self-advancing fake clock so the measured elapsed/nudge values are deterministic and provably distinct from the configured budget.	2026-09-12 09:08:04 +07:00
Dai Ha	dcd505286f	fleetd #492: teach redeploy-fleetd.sh systemd --user as a third supervisor ... launchd, systemd --user, and unsupervised are three different answers, not two. Refuse (die) rather than fall through to kill+nohup when a supervisor is detected that this script cannot drive (e.g. both signals fire at once), and add a post-restart check that fails the run if more than one fleetd process is alive. detect_supervisor()/require_drivable_supervisor()/ count_daemon_pids()/assert_single_daemon() are pure, overridable functions so scripts/test-redeploy-fleetd.sh can exercise them without a real launchd or systemd.	2026-09-12 09:00:59 +07:00