Review of PR #559 (issue comment #16944) found a surviving mutant: removing
watchdog.reset() from StatusPoller.start() (and the identical line in
SessionReaper.start()) passed the entire suite.
stoppedByCaller is sticky and reset() — called only from start() — is the
only thing that clears it. Both loops document start() as idempotent and
loop()'s own error log says "it can be restarted", so stop() followed by
start() is an anticipated path. Without reset() wired into start(), health()
would report STOPPED forever after a restart even though the loop is
genuinely running again.
Add aRestartedLoopReportsRunningAgainNotStoppedForever to both
StatusPollerWatchdogTest and SessionReaperWatchdogTest, pinning "an
intentional stop must not outlive the restart that follows it". Verified via
the standard mutation cycle: exact-line anchor (not regex, to avoid the
\Q-style false match the reviewer flagged) counted pristine 1 -> mutated 0,
test goes red with its own message, restored, shasum -a 256 byte-identical,
green again.
mvn clean install: exit 0, BUILD SUCCESS, Tests run: 1734, Failures: 0,
Errors: 0, Skipped: 0 (cross-checked against 130 surefire report files).
No production code changed — the reset() call under test was already
correct; it simply had nothing pinning it.
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Each loop's virtual-thread runner (StatusPoller, SessionReaper) can die or
get permanently parked in a herdr call with no read timeout, and nothing
observed it: /healthz stayed green and Thread.isAlive() kept reporting true
the whole time.
Add LoopWatchdog (dev.ltms.fleet.inject — see its javadoc for why not
dev.ltms.fleet.health, which would close a package cycle through session):
each loop now records a monotonic last-completed-round timestamp
(injectable LongSupplier clock, same pattern as Injector/SessionManager) and
exposes it as a three-state health() fact — RUNNING, STALLED (dead or
parked, indistinguishable from outside), STOPPED (stop() was called on
purpose, never an alarm). This is the fleetd #512 shape: one flag cannot
carry both "halted on purpose" and "halted unexpectedly", so stop() marks
its own state explicitly instead of leaving state() to infer it from
staleness.
Staleness thresholds are derived from each loop's own poll interval with a
documented multiplier: StatusPoller 40x (250ms -> 10s), SessionReaper 12x
(5000ms -> 60s).
Scope: observability only, per the ticket's own comment. No restart/recovery
mechanism, no /healthz or REST/MCP wiring beyond the public health() API, no
change to the per-item catch(Throwable) behavior (#543) or a process-wide
uncaught-exception handler (ruled out on #538), and no deadline added to the
herdr read itself (a separate, real ticket).
🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
Dai Ha