fleet/fleetd
3
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

redeploy health gate must classify errors, not count them #354

Closed
agent wants to merge 0 commits from worker/errscan-bed2ca-2 into main
pull from: worker/errscan-bed2ca-2
merge into: fleet:main
Conversation 0 Commits - Files Changed -
agent commented 2026-09-05 00:30:58 +02:00
Member
Copy Link
Copy Source

Update: classify named AMQP failures after PR #356

This branch now includes current main, including PR #356. The classifier uses the connection names that PR #356 adds to AMQP failure messages.

Patterns and source

I read fleetd/src/main/java/dev/ltms/fleet/msg/AmqpReplyInbox.java and LeadMailbox.java on current main. AmqpConnectionFailureLogger defines fleetd-reply-inbox and fleetd-lead-mailbox once, and logs AMQP connection {}: {} through each owning class logger.

The classifier matches ERROR or SEVERE lines with either failure message:

  • AMQP connection fleetd-reply-inbox: An unexpected connection driver error occurred
  • AMQP connection fleetd-reply-inbox: Caught an exception during connection recovery!
  • the same two messages with fleetd-lead-mailbox.

It does not depend on the abbreviated logger package. This avoids the different d.l...AmqpReplyInbox and d.ltms...LeadMailbox layout forms.

Attribution

An inbox failure increments only the inbox pending count. A lead-mailbox failure increments only the lead pending count. Each recovery decrements only its matching count. Any candidate failure message without either stable name is unexplained and stays loud.

The source-derived fixture has three inbox errors, three lead errors, and matching later recoveries. It reports total 6, recovered 6, unexplained 0. The unattributable fixture has two unknown-connection errors plus lead recoveries. It reports total 2, recovered 0, unexplained 2.

Tests and mutations

The script test now checks the new source logger message and the recovery strings in Java source. Its new shapes are source-derived, not yet confirmed against a live post-redeploy log.

Mutation checks:

Shared-counter mutation: FAIL: cross-attributed recovered: expected 0, got 2
Unattributable mutation: FAIL: cross-unattributable recovered: expected 0, got 2
Recovery mutation: FAIL: unrecovered AMQP errors: expected 0, got 1

After redeploy, run this command to confirm the new live ERROR patterns while redacting any URI credentials:

sed -E "s#://[^@]*@#://<redacted>@#g" fleetd/fleetd.out | grep -E " (ERROR|SEVERE) .*AMQP connection fleetd-(reply-inbox|lead-mailbox):"

Checks

$ bash scripts/test-redeploy-fleetd.sh
Shared-counter mutation: FAIL: cross-attributed recovered: expected 0, got 2
Unattributable mutation: FAIL: cross-unattributable recovered: expected 0, got 2
PASS: redeploy log classifier

$ cd fleetd && /Users/dai.ha/Softwares/apache-maven/bin/mvn clean install
Tests run: 1379, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

No daemon restart and no redeploy script invocation occurred.

## Update: classify named AMQP failures after PR #356 This branch now includes current main, including PR #356. The classifier uses the connection names that PR #356 adds to AMQP failure messages. ### Patterns and source I read `fleetd/src/main/java/dev/ltms/fleet/msg/AmqpReplyInbox.java` and `LeadMailbox.java` on current main. `AmqpConnectionFailureLogger` defines `fleetd-reply-inbox` and `fleetd-lead-mailbox` once, and logs `AMQP connection {}: {}` through each owning class logger. The classifier matches ERROR or SEVERE lines with either failure message: - `AMQP connection fleetd-reply-inbox: An unexpected connection driver error occurred` - `AMQP connection fleetd-reply-inbox: Caught an exception during connection recovery!` - the same two messages with `fleetd-lead-mailbox`. It does not depend on the abbreviated logger package. This avoids the different `d.l...AmqpReplyInbox` and `d.ltms...LeadMailbox` layout forms. ### Attribution An inbox failure increments only the inbox pending count. A lead-mailbox failure increments only the lead pending count. Each recovery decrements only its matching count. Any candidate failure message without either stable name is unexplained and stays loud. The source-derived fixture has three inbox errors, three lead errors, and matching later recoveries. It reports total 6, recovered 6, unexplained 0. The unattributable fixture has two unknown-connection errors plus lead recoveries. It reports total 2, recovered 0, unexplained 2. ### Tests and mutations The script test now checks the new source logger message and the recovery strings in Java source. Its new shapes are source-derived, not yet confirmed against a live post-redeploy log. Mutation checks: ```text Shared-counter mutation: FAIL: cross-attributed recovered: expected 0, got 2 Unattributable mutation: FAIL: cross-unattributable recovered: expected 0, got 2 Recovery mutation: FAIL: unrecovered AMQP errors: expected 0, got 1 ``` After redeploy, run this command to confirm the new live ERROR patterns while redacting any URI credentials: ```bash sed -E "s#://[^@]*@#://<redacted>@#g" fleetd/fleetd.out | grep -E " (ERROR|SEVERE) .*AMQP connection fleetd-(reply-inbox|lead-mailbox):" ``` ### Checks ```text $ bash scripts/test-redeploy-fleetd.sh Shared-counter mutation: FAIL: cross-attributed recovered: expected 0, got 2 Unattributable mutation: FAIL: cross-unattributable recovered: expected 0, got 2 PASS: redeploy log classifier $ cd fleetd && /Users/dai.ha/Softwares/apache-maven/bin/mvn clean install Tests run: 1379, Failures: 0, Errors: 0, Skipped: 0 BUILD SUCCESS ``` No daemon restart and no redeploy script invocation occurred.
agent added 1 commit 2026-09-05 00:30:59 +02:00
Classify recovered AMQP redeploy errors
CI / contract (pull_request) Successful in 45s
Details
CI / build (pull_request) Successful in 1m51s
Details
e4973eb8a4
agent added 1 commit 2026-09-05 00:36:57 +02:00
Keep unattributed AMQP errors loud
CI / contract (pull_request) Successful in 52s
Details
CI / build (pull_request) Successful in 1m30s
Details
0241e0d3a8
agent added 2 commits 2026-09-05 01:05:57 +02:00
Merge remote-tracking branch 'origin/main' into worker/errscan-bed2ca-2 29cd1194c2
Classify named AMQP recovery errors
CI / contract (pull_request) Successful in 51s
Details
CI / build (pull_request) Successful in 2m16s
Details
09159f2857
ltms closed this pull request 2026-09-05 01:09:40 +02:00
Some checks are pending
CI / contract (pull_request) Successful in 51s
Details
CI / build (pull_request) Successful in 2m16s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
blocked
Cannot start until something else lands. The body says what.
 needs-live-proof
Merged and green, but never shown working on the running daemon. Not the same as done.
 ready-to-delegate
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
 silent-default
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent kevin ltms
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: fleet/fleetd#354
Delete Branch "worker/errscan-bed2ca-2"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?