fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

CB-185: fix two blockers to switching on memberHerdrSocket #196

Merged
ltms merged 2 commits from worker/cb-185-blockers-027756-4 into main 2026-08-31 04:30:57 +02:00
Conversation 0 Commits 2 Files Changed 5
+254 -17
agent commented 2026-08-31 04:20:54 +02:00
Member
Copy Link
Copy Source

Fixes #185.

Two blockers that must land before memberHerdrSocket can ever be switched on safely:

  1. CompositePeerLauncher.stop() un-stoppable after a restart — spawnedBy is in-memory only, so a daemon restart empties it, and every member on a second daemon that survives the restart could never be stopped (fleet_stop refused it forever). Fixed with a probe-on-cache-miss: ask each distinct configured herdr daemon's agent.list() who owns the pane. Exactly one owner routes and caches; zero owners is treated as already-stopped (a no-op success, matching the tolerance HerdrPeerLauncher already gives an already-gone pane); more than one owner is the genuine per-daemon pane-id ambiguity and still refuses with the existing message.

  2. /healthz reported only the lead daemon's protocol — with memberHerdrSocket configured, a member-daemon protocol mismatch was invisible behind a green /healthz while every spawn silently failed. Added a separate "member" key (version/protocol) alongside the unchanged "herdr" key, plus a "protocolMismatch": true flag when they differ. Verified scripts/redeploy-fleetd.sh and scripts/rename-checkout.sh only check the HTTP status code and print the body verbatim — neither parses a specific field — so adding a key alongside herdr is safe.

Testing: every new test was written to fail without its fix (verified by stashing each fix, watching the new tests fail with the expected error, then restoring). Full mvn clean install: 1018 tests, 0 failures, 0 errors, BUILD SUCCESS.

Did not set memberHerdrSocket, touch fleetd.yaml, or redeploy the daemon, per the ticket.

Fixes #185. Two blockers that must land before memberHerdrSocket can ever be switched on safely: 1. **CompositePeerLauncher.stop() un-stoppable after a restart** — spawnedBy is in-memory only, so a daemon restart empties it, and every member on a second daemon that survives the restart could never be stopped (fleet_stop refused it forever). Fixed with a probe-on-cache-miss: ask each distinct configured herdr daemon's agent.list() who owns the pane. Exactly one owner routes and caches; zero owners is treated as already-stopped (a no-op success, matching the tolerance HerdrPeerLauncher already gives an already-gone pane); more than one owner is the genuine per-daemon pane-id ambiguity and still refuses with the existing message. 2. **/healthz reported only the lead daemon's protocol** — with memberHerdrSocket configured, a member-daemon protocol mismatch was invisible behind a green /healthz while every spawn silently failed. Added a separate "member" key (version/protocol) alongside the unchanged "herdr" key, plus a "protocolMismatch": true flag when they differ. Verified scripts/redeploy-fleetd.sh and scripts/rename-checkout.sh only check the HTTP status code and print the body verbatim — neither parses a specific field — so adding a key alongside herdr is safe. **Testing**: every new test was written to fail without its fix (verified by stashing each fix, watching the new tests fail with the expected error, then restoring). Full `mvn clean install`: 1018 tests, 0 failures, 0 errors, BUILD SUCCESS. Did not set memberHerdrSocket, touch fleetd.yaml, or redeploy the daemon, per the ticket.
agent added 1 commit 2026-08-31 04:20:54 +02:00
CB-185: fix two blockers to switching on memberHerdrSocket (#185)
CI / contract (pull_request) Successful in 46s
Details
CI / build (pull_request) Successful in 1m39s
Details
045d229728 
1. CompositePeerLauncher.stop() was permanently un-stoppable for any
   member that survived a daemon restart, because spawnedBy is in-memory
   only. On a cache miss with more than one configured herdr daemon, probe
   each distinct daemon's agent.list() for the pane instead of refusing
   outright: exactly one owner routes and caches; zero owners is treated
   as already-stopped (a no-op, matching the tolerance HerdrPeerLauncher
   already gives an already-gone pane); more than one owner is the
   genuine per-daemon-pane-id ambiguity and still throws.

2. FleetApp#healthz always reported the LEAD daemon's herdr version/
   protocol even when a second (member) daemon was configured, so a
   member-daemon protocol mismatch was invisible behind a green
   /healthz while every spawn silently failed. Added a separate "member"
   key alongside the unchanged "herdr" key, and a "protocolMismatch"
   flag when the two differ. Verified scripts/redeploy-fleetd.sh and
   scripts/rename-checkout.sh only check the HTTP status code and print
   the body verbatim — neither parses a specific field — so adding a key
   is safe.

Both fixes are covered by tests written to fail without the fix
(verified by reverting each fix and watching the new tests fail, then
restoring). Full `mvn clean install`: 1018 tests, 0 failures, 0 errors.
agent added 1 commit 2026-08-31 04:28:32 +02:00
CB-185: fix ambiguous-pane message and unreachable-daemon abort in probeOwner (review)
CI / build (pull_request) Successful in 1m6s
Details
CI / contract (pull_request) Successful in 1m5s
Details
bc99d64786 
Lead review of PR #196 found two issues in CompositePeerLauncher.probeOwner:

1. The "more than one daemon claims this pane" throw kept the old
   pre-fix message ("no owning herdr daemon was recorded"), which was
   only true of the code it replaced. Reworded to say what actually
   happened: N configured herdr daemons report this pane, so it is
   genuinely ambiguous. Updated the one test pinning the old string.

2. probeOwner let list() propagate straight out of the probe loop, so
   one unreachable daemon aborted the whole probe and made a pane on a
   DIFFERENT, healthy daemon un-stoppable too — resurrecting the exact
   bug blocker 1 fixes. Now catches HerdrException per daemon, logs the
   exception class only, and treats that daemon as not knowing the pane
   so probing continues. New test proves this: verified it fails with
   the try/catch removed (HerdrException propagates and the stop that
   should succeed via the healthy daemon throws instead), then restored.

Full mvn clean install: 1019 tests, 0 failures, 0 errors.
ltms merged commit 63c19dcba7 into main 2026-08-31 04:30:57 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#196
Delete Branch "worker/cb-185-blockers-027756-4"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?