#164: classify a backend-error scrape as WORKER_FAILED, not a completed reply #200

Closed
agent wants to merge 2 commits from worker/cb-164-rebase-885863-8 into main
Member

See full report below (also committed as REPORT-cb164.md in this branch).

CB-164 report — worker/cb-164-rebase-885863-8

Headline finding — please read this before the diff

main already ships a full fix for issue #164 points 1 and 2, done separately from the
rescued branch.
Commit 3bfa828 ("fleetd#164: an empty or suspiciously fast scrape must
fail, never resolve as a success") is already an ancestor of origin/main (verified with
git merge-base --is-ancestor 3bfa828 origin/main). It was written the same day as the
rescued commit (2026-08-28), by the same author, but on main directly. No later commit
touches CompletionResolver.java after it.

What main already has, before any change of mine:

  • CompletionResolver.MIN_TURN_NANOS = 2 seconds — the exact "too fast to be a real
    completion" floor the brief described, just under a different name and a different
    mechanism (an injectable LongSupplier nowNanos clock read at delivery and again at
    resolution, stored on the InFlight record — no interface or caller changes needed).
  • A hard fail on any empty or unreadable scrape, via the existing fail() →
    Rendezvous.resolveFailure() → MessageService.Outcome.WORKER_FAILED path — the same
    pattern already used for the CB-109 wedge case. FleetMcp.formatReply already renders
    WORKER_FAILED as "[worker failed — turn ended in an unrecoverable state]\n" + text,
    never as blank content. I read this chain end to end and confirmed it (not just from the
    commit message).

So "Part 2" as described in the brief — the one called out as the most important — was
already done.
What I did not find on main: the narrow BACKEND_ERROR pattern
((?i)\bAPI Error\s*:) from the rescued branch. That is genuinely new.

Why I did not cherry-pick 851ebca, and what I did instead

851ebca implements the same floor-check idea a second time, but via a structurally
different, more invasive mechanism: it threads a real measured elapsedNanos through
TurnListener.onTurnComplete, Injector (a new turnStartedAtNanos field, a new
constructor, a new LongSupplier clock), and Fleetd's anonymous TurnListener. Cherry-
picking that wholesale onto current main would have created two competing timing
mechanisms for the same floor check
in the same class — main's already-shipped
delivery-clock read, and a second, parameter-threaded one from the rescued branch — with
no clear answer for which one should win if they ever disagreed. Reintroducing the
Injector/TurnListener/Fleetd signature changes for that also seemed hard to justify: main's
already-tested approach measures essentially the same interval (delivery to resolution)
with no interface changes at all.

Given that, I read the instruction "if you cannot tell which behaviour a hunk is meant to
have, keep both behaviours and say so"
as pointing at genuinely uncertain hunks — not at
knowingly wiring in two redundant implementations of the identical check. So instead of a
mechanical cherry-pick, I hand-ported only the parts of 851ebca that are not already
on main:

  1. Added CompletionResolver.BACKEND_ERROR — the exact narrow pattern from the brief,
    (?i)\bAPI Error\s*:, with the same "keep it narrow" comment.
  2. Added a classification check for it, placed right after the existing BACKEND_EXHAUSTED
    pattern-match block (same position, same shape: firstMatchingLine against
    assistantBlock, then fail(target, turn, reason) naming the member and the matched
    line) and before the success path (fleetd/src/main/java/dev/ltms/fleet/inject/ CompletionResolver.java).

I did not port:

  • MIN_COMPLETED_TURN_NANOS / the elapsedNanos parameter threading through
    TurnListener/Injector/Fleetd — redundant with main's already-shipped
    MIN_TURN_NANOS/LongSupplier mechanism, and touching those three extra files for no
    behavioural gain seemed like avoidable risk.
  • The rescued branch's visibleTurn = assistantBlock.isBlank() ? raw : assistantBlock
    fallback (falls back to the raw pane when the parsed assistant block is blank, so a
    TUI-hidden error/exhausted line still classifies). This is a real, separate improvement,
    but on main's current check order the empty-tail fail already fires before the
    exhausted/backend-error pattern checks run, so the fallback would be dead code unless I
    also reordered those checks ahead of the empty-tail fail. That reorder is a bigger,
    separate behavioural change than either "Part 1" or "Part 2" asked for, so I left it out
    and am flagging it here rather than making that call silently. Out-of-scope note for
    the lead
    , not fixed: a raw-screen-only exhausted/backend-error line (no ⏺ marker, so
    lastAssistantBlock returns blank) is still swallowed into the generic "empty scrape"
    failure rather than being classified specifically.

I did not use fleet_ask for this: the brief's own instruction for exactly this kind of
conflict ("keep both, say so") already pre-authorized me to decide and document rather than
block, and a live fleet_ask has only a ~55s window per prior fleet notes, so I judged
documenting clearly here was more reliable than gambling on that window. If this call is
wrong, it's easy to undo — the diff is 3 files, +85/-0 lines.

Files changed (worktree-relative)

  • fleetd/src/main/java/dev/ltms/fleet/inject/CompletionResolver.java — added the
    BACKEND_ERROR pattern constant and the classification check (+17 lines).
  • fleetd/src/test/java/dev/ltms/fleet/inject/CompletionResolverTest.java — 3 new unit
    tests for the BACKEND_ERROR classification (+46 lines).
  • fleetd/src/test/java/dev/ltms/fleet/msg/MessageServiceTest.java — 1 new end-to-end test
    driving the same scenario through MessageService (+22 lines).

No changes to Fleetd.java, Injector.java, or TurnListener.java — see above for why.

Build — verbatim result

Full unpiped mvn clean install from fleetd/, exit code 0:

[INFO] Tests run: 1036, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

No [ERROR] lines anywhere in the full log (checked with grep -n "\[ERROR\]" over the
whole log, not a piped/truncated view).

Sabotage proof for the new tests

Commented out the new check in CompletionResolver.java:

// String backendError = firstMatchingLine(assistantBlock, BACKEND_ERROR);
// if (backendError != null) {
//     fail(target, turn, "member " + target + " ended on a backend error: " + backendError);
//     return;
// }

Ran just the 4 new tests:

mvn -q test -Dtest='CompletionResolverTest#classifiesABackendErrorLineAsAFailureInsteadOfACompletedReply+theBackendErrorReasonNamesTheMemberAndCarriesTheMatchedLine+aCaseInsensitiveApiErrorLineIsStillClassifiedAsABackendError,MessageServiceTest#backendErrorScrapeThroughMessageServiceFailsInsteadOfBecomingReplyText'

All 4 failed, exactly as expected without the fix:

[ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0
[ERROR]   CompletionResolverTest.aCaseInsensitiveApiErrorLineIsStillClassifiedAsABackendError:612
    the pattern is case-insensitive ==> expected: <FAILED> but was: <COMPLETION>
[ERROR]   CompletionResolverTest.classifiesABackendErrorLineAsAFailureInsteadOfACompletedReply:582
    a backend rejection is a failure, not a completed reply ==> expected: <FAILED> but was: <COMPLETION>
[ERROR]   CompletionResolverTest.theBackendErrorReasonNamesTheMemberAndCarriesTheMatchedLine:597
    the failure names the member: API Error: 400 invalid request body ==> expected: <true> but was: <false>
[ERROR]   MessageServiceTest.backendErrorScrapeThroughMessageServiceFailsInsteadOfBecomingReplyText:104
    a backend rejection must use the caller's failure outcome, not a completed reply
    ==> expected: <WORKER_FAILED> but was: <COMPLETED_UNREPLIED>

Then restored the fix (put the block back exactly as committed) and re-ran the full
mvn clean install — green again, see above (Tests run: 1036, Failures: 0).

What I could not confirm

  • I have no IDE tooling and no way to run the daemon live — only mvn in this worktree.
    Everything reported above is from that command, nothing else.
  • I did not attempt the "visibleTurn/raw-fallback" reorder described above — flagged as an
    open, separate item, not fixed, not tested.
  • Points 3 (broader backend-error surfacing) and 4 (spawn-time profile quarantine) of the
    issue are untouched, as instructed.
  • I have not verified this against a live daemon or a real crashed backend — only against
    the unit/integration test fixtures in this repo.
See full report below (also committed as REPORT-cb164.md in this branch). # CB-164 report — worker/cb-164-rebase-885863-8 ## Headline finding — please read this before the diff **`main` already ships a full fix for issue #164 points 1 and 2, done separately from the rescued branch.** Commit `3bfa828` ("fleetd#164: an empty or suspiciously fast scrape must fail, never resolve as a success") is already an ancestor of `origin/main` (verified with `git merge-base --is-ancestor 3bfa828 origin/main`). It was written the same day as the rescued commit (2026-08-28), by the same author, but on `main` directly. No later commit touches `CompletionResolver.java` after it. What `main` already has, before any change of mine: - `CompletionResolver.MIN_TURN_NANOS` = 2 seconds — the exact "too fast to be a real completion" floor the brief described, just under a different name and a different mechanism (an injectable `LongSupplier nowNanos` clock read at delivery and again at resolution, stored on the `InFlight` record — no interface or caller changes needed). - A hard fail on any empty or unreadable scrape, via the existing `fail()` → `Rendezvous.resolveFailure()` → `MessageService.Outcome.WORKER_FAILED` path — the same pattern already used for the CB-109 wedge case. `FleetMcp.formatReply` already renders `WORKER_FAILED` as `"[worker failed — turn ended in an unrecoverable state]\n" + text`, never as blank content. I read this chain end to end and confirmed it (not just from the commit message). So **"Part 2" as described in the brief — the one called out as the most important — was already done.** What I did **not** find on `main`: the narrow `BACKEND_ERROR` pattern (`(?i)\bAPI Error\s*:`) from the rescued branch. That is genuinely new. ### Why I did not cherry-pick 851ebca, and what I did instead `851ebca` implements the same floor-check idea a second time, but via a structurally different, more invasive mechanism: it threads a real measured `elapsedNanos` through `TurnListener.onTurnComplete`, `Injector` (a new `turnStartedAtNanos` field, a new constructor, a new `LongSupplier` clock), and `Fleetd`'s anonymous `TurnListener`. Cherry- picking that wholesale onto current `main` would have created **two competing timing mechanisms for the same floor check** in the same class — main's already-shipped delivery-clock read, and a second, parameter-threaded one from the rescued branch — with no clear answer for which one should win if they ever disagreed. Reintroducing the Injector/TurnListener/Fleetd signature changes for that also seemed hard to justify: main's already-tested approach measures essentially the same interval (delivery to resolution) with no interface changes at all. Given that, I read the instruction *"if you cannot tell which behaviour a hunk is meant to have, keep both behaviours and say so"* as pointing at genuinely uncertain hunks — not at knowingly wiring in two redundant implementations of the identical check. So instead of a mechanical cherry-pick, I hand-ported only the parts of `851ebca` that are **not** already on `main`: 1. Added `CompletionResolver.BACKEND_ERROR` — the exact narrow pattern from the brief, `(?i)\bAPI Error\s*:`, with the same "keep it narrow" comment. 2. Added a classification check for it, placed right after the existing `BACKEND_EXHAUSTED` pattern-match block (same position, same shape: `firstMatchingLine` against `assistantBlock`, then `fail(target, turn, reason)` naming the member and the matched line) and before the success path (`fleetd/src/main/java/dev/ltms/fleet/inject/ CompletionResolver.java`). I did **not** port: - `MIN_COMPLETED_TURN_NANOS` / the `elapsedNanos` parameter threading through `TurnListener`/`Injector`/`Fleetd` — redundant with `main`'s already-shipped `MIN_TURN_NANOS`/`LongSupplier` mechanism, and touching those three extra files for no behavioural gain seemed like avoidable risk. - The rescued branch's `visibleTurn = assistantBlock.isBlank() ? raw : assistantBlock` fallback (falls back to the raw pane when the parsed assistant block is blank, so a TUI-hidden error/exhausted line still classifies). This is a real, separate improvement, but on `main`'s current check order the empty-tail fail already fires *before* the exhausted/backend-error pattern checks run, so the fallback would be dead code unless I also reordered those checks ahead of the empty-tail fail. That reorder is a bigger, separate behavioural change than either "Part 1" or "Part 2" asked for, so I left it out and am flagging it here rather than making that call silently. **Out-of-scope note for the lead**, not fixed: a raw-screen-only exhausted/backend-error line (no `⏺` marker, so `lastAssistantBlock` returns blank) is still swallowed into the generic "empty scrape" failure rather than being classified specifically. I did not use `fleet_ask` for this: the brief's own instruction for exactly this kind of conflict ("keep both, say so") already pre-authorized me to decide and document rather than block, and a live `fleet_ask` has only a ~55s window per prior fleet notes, so I judged documenting clearly here was more reliable than gambling on that window. If this call is wrong, it's easy to undo — the diff is 3 files, +85/-0 lines. ## Files changed (worktree-relative) - `fleetd/src/main/java/dev/ltms/fleet/inject/CompletionResolver.java` — added the `BACKEND_ERROR` pattern constant and the classification check (+17 lines). - `fleetd/src/test/java/dev/ltms/fleet/inject/CompletionResolverTest.java` — 3 new unit tests for the `BACKEND_ERROR` classification (+46 lines). - `fleetd/src/test/java/dev/ltms/fleet/msg/MessageServiceTest.java` — 1 new end-to-end test driving the same scenario through `MessageService` (+22 lines). No changes to `Fleetd.java`, `Injector.java`, or `TurnListener.java` — see above for why. ## Build — verbatim result Full unpiped `mvn clean install` from `fleetd/`, exit code `0`: ``` [INFO] Tests run: 1036, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS ``` No `[ERROR]` lines anywhere in the full log (checked with `grep -n "\[ERROR\]"` over the whole log, not a piped/truncated view). ## Sabotage proof for the new tests Commented out the new check in `CompletionResolver.java`: ```java // String backendError = firstMatchingLine(assistantBlock, BACKEND_ERROR); // if (backendError != null) { // fail(target, turn, "member " + target + " ended on a backend error: " + backendError); // return; // } ``` Ran just the 4 new tests: ``` mvn -q test -Dtest='CompletionResolverTest#classifiesABackendErrorLineAsAFailureInsteadOfACompletedReply+theBackendErrorReasonNamesTheMemberAndCarriesTheMatchedLine+aCaseInsensitiveApiErrorLineIsStillClassifiedAsABackendError,MessageServiceTest#backendErrorScrapeThroughMessageServiceFailsInsteadOfBecomingReplyText' ``` All 4 failed, exactly as expected without the fix: ``` [ERROR] Tests run: 4, Failures: 4, Errors: 0, Skipped: 0 [ERROR] CompletionResolverTest.aCaseInsensitiveApiErrorLineIsStillClassifiedAsABackendError:612 the pattern is case-insensitive ==> expected: <FAILED> but was: <COMPLETION> [ERROR] CompletionResolverTest.classifiesABackendErrorLineAsAFailureInsteadOfACompletedReply:582 a backend rejection is a failure, not a completed reply ==> expected: <FAILED> but was: <COMPLETION> [ERROR] CompletionResolverTest.theBackendErrorReasonNamesTheMemberAndCarriesTheMatchedLine:597 the failure names the member: API Error: 400 invalid request body ==> expected: <true> but was: <false> [ERROR] MessageServiceTest.backendErrorScrapeThroughMessageServiceFailsInsteadOfBecomingReplyText:104 a backend rejection must use the caller's failure outcome, not a completed reply ==> expected: <WORKER_FAILED> but was: <COMPLETED_UNREPLIED> ``` Then restored the fix (put the block back exactly as committed) and re-ran the full `mvn clean install` — green again, see above (`Tests run: 1036, Failures: 0`). ## What I could not confirm - I have no IDE tooling and no way to run the daemon live — only `mvn` in this worktree. Everything reported above is from that command, nothing else. - I did not attempt the "visibleTurn/raw-fallback" reorder described above — flagged as an open, separate item, not fixed, not tested. - Points 3 (broader backend-error surfacing) and 4 (spawn-time profile quarantine) of the issue are untouched, as instructed. - I have not verified this against a live daemon or a real crashed backend — only against the unit/integration test fixtures in this repo.
agent added 1 commit 2026-08-31 05:50:57 +02:00
#164: classify a backend-error scrape as WORKER_FAILED, not a completed reply
CI / contract (pull_request) Successful in 51s
CI / build (pull_request) Successful in 1m33s
1fcda74596
main already ships fleetd#164 points 1 and 2 (MIN_TURN_NANOS floor + hard-fail on an
empty/unreadable scrape, commit 3bfa828, already an ancestor of main). The rescued
worker branch worker/cb-164-empty-scrape-false-success-1a80af-3 (851ebca) reimplemented
the same two behaviours via a separate, more invasive mechanism (elapsedNanos threaded
through TurnListener/Injector/Fleetd) that would conflict with main's already-shipped
clock-injection design if merged wholesale.

Port forward only what main is genuinely missing: the narrow BACKEND_ERROR pattern
(`(?i)\bAPI Error\s*:`) that classifies a cleanly-scraped-but-backend-rejected turn as a
failure naming the member, instead of letting the error text resolve as a completed
reply. See REPORT-cb164.md for the full reasoning, what was deliberately not ported, and
the sabotage proof for the new tests.
agent added 1 commit 2026-08-31 05:56:18 +02:00
#164: a backend-error failure must carry the whole scrape, not just the matched line
CI / contract (pull_request) Successful in 1m18s
CI / build (pull_request) Successful in 1m28s
8aaf1f7e44
The BACKEND_ERROR pattern is a heuristic. It also matches a member that forgot
fleet_reply while reporting *about* a backend error. Failing is still right --
the caller must never read a scrape as an answer -- but the reason carried only
the matched line, so the rest of the report was thrown away.

That is the same defect #164 exists to fix: information destroyed on the way to
the caller. Carry the full pane tail alongside the classification, so a genuine
backend error reads the same as before and a false positive keeps its report.
Owner

Merged to main as 4ac688b (squashed). Thanks — the staleness catch was the valuable part of this run.

What I checked myself, and how

I did not take the "already on main" claim on trust, because I got this exact thing wrong twice this week (#173, #174):

  • git merge-base --is-ancestor 3bfa828 origin/main → yes. The claim is correct: main already had the MIN_TURN_NANOS floor and the hard fail on an empty or unreadable scrape.
  • git log <branch>..origin/main → empty. The branch was current with main, so nothing was silently superseded underneath it.
  • mvn clean install from fleetd/, unpiped, in the worktree: exit 0, Tests run: 1036, 0 [ERROR] lines. Matches your report exactly.
  • Sabotage, re-run by me rather than read from your transcript: commented out the BACKEND_ERROR block, ran both suites → all 4 new tests failed with the outcomes you quoted (expected: <FAILED> but was: <COMPLETION>, expected: <WORKER_FAILED> but was: <COMPLETED_UNREPLIED>). Restored, clean.

Declining to cherry-pick 851ebca was the right call. Threading a second elapsedNanos measurement through TurnListener when main already has clock injection would have left two mechanisms for one check.

One thing I added before merging (8aaf1f7)

The failure reason carried only the matched line. The rest of the scrape was dropped.

The pattern is a heuristic, so it also matches a member that forgot fleet_reply while reporting about a backend error — plausible in this repo, where members investigate gateway 400s. In that case the fix threw the report away. My sabotage run showed it concretely: a three-line report collapsed to

member term_a ended on a backend error: The log line was: API Error: 400 invalid request body

That is the same defect #164 exists to fix — information destroyed between the member and the caller — just with a failure label on it. Failing is still correct; discarding the pane is not.

So the reason now carries the full tail after the matched line. A genuine backend error reads as before; a false positive keeps its report. Added aBackendErrorFailureStillCarriesTheRestOfTheScrape, sabotage-proven the same way. Final build: 1037 tests, exit 0, 0 [ERROR] lines.

Not merged

REPORT-cb164.md — dropped at merge. No REPORT-*.md exists on main, so it is not a convention there. It stays visible in this PR, which is what I actually wanted when I asked you to write it: a report that outlives its ticket.

Left open, correctly

Your visibleTurn raw-screen note is a real observation, and stopping rather than reordering the check chain silently was the right instinct. Recorded on #164; not this PR's job.

Merged to `main` as `4ac688b` (squashed). Thanks — the staleness catch was the valuable part of this run. ## What I checked myself, and how I did not take the "already on main" claim on trust, because I got this exact thing wrong twice this week (#173, #174): - `git merge-base --is-ancestor 3bfa828 origin/main` → **yes**. The claim is correct: `main` already had the `MIN_TURN_NANOS` floor and the hard fail on an empty or unreadable scrape. - `git log <branch>..origin/main` → **empty**. The branch was current with main, so nothing was silently superseded underneath it. - `mvn clean install` from `fleetd/`, unpiped, in the worktree: exit 0, `Tests run: 1036`, **0** `[ERROR]` lines. Matches your report exactly. - Sabotage, re-run by me rather than read from your transcript: commented out the `BACKEND_ERROR` block, ran both suites → all 4 new tests failed with the outcomes you quoted (`expected: <FAILED> but was: <COMPLETION>`, `expected: <WORKER_FAILED> but was: <COMPLETED_UNREPLIED>`). Restored, clean. Declining to cherry-pick `851ebca` was the right call. Threading a second `elapsedNanos` measurement through `TurnListener` when `main` already has clock injection would have left two mechanisms for one check. ## One thing I added before merging (`8aaf1f7`) The failure reason carried **only the matched line**. The rest of the scrape was dropped. The pattern is a heuristic, so it also matches a member that forgot `fleet_reply` while reporting *about* a backend error — plausible in this repo, where members investigate gateway 400s. In that case the fix threw the report away. My sabotage run showed it concretely: a three-line report collapsed to ``` member term_a ended on a backend error: The log line was: API Error: 400 invalid request body ``` That is the same defect #164 exists to fix — information destroyed between the member and the caller — just with a failure label on it. Failing is still correct; discarding the pane is not. So the reason now carries the full tail after the matched line. A genuine backend error reads as before; a false positive keeps its report. Added `aBackendErrorFailureStillCarriesTheRestOfTheScrape`, sabotage-proven the same way. Final build: **1037 tests, exit 0, 0 `[ERROR]` lines**. ## Not merged `REPORT-cb164.md` — dropped at merge. No `REPORT-*.md` exists on `main`, so it is not a convention there. It stays visible in this PR, which is what I actually wanted when I asked you to write it: a report that outlives its ticket. ## Left open, correctly Your `visibleTurn` raw-screen note is a real observation, and stopping rather than reordering the check chain silently was the right instinct. Recorded on #164; not this PR's job.
Owner

Already on main — closing as superseded, not rejected. The change landed in 4ac688b ("#164: classify a backend-error scrape as WORKER_FAILED, carrying the whole pane"), byte-identical to this branch including both commits: the BACKEND_ERROR pattern and the follow-up that carries the whole pane tail rather than only the matched line.

Both judgement calls in it were the right ones and are worth recording:

  • Keeping the pattern deliberately narrow. A growing list of ad-hoc backend error strings rots as backends change wording, and a false positive here fails a turn that actually succeeded.
  • Carrying the full scrape after the marker. The pattern is a heuristic — a member reporting about a backend error matches it too — so failing is right, but dropping the rest of the pane would destroy the report, which is the same defect #164 exists to fix.

The remaining half of #164 (points 3 and 4, a real classification mechanism instead of hard-coded strings) stays open as #201.

Already on `main` — closing as superseded, not rejected. The change landed in `4ac688b` ("#164: classify a backend-error scrape as WORKER_FAILED, carrying the whole pane"), byte-identical to this branch including both commits: the `BACKEND_ERROR` pattern and the follow-up that carries the whole pane tail rather than only the matched line. Both judgement calls in it were the right ones and are worth recording: - Keeping the pattern deliberately narrow. A growing list of ad-hoc backend error strings rots as backends change wording, and a false positive here fails a turn that actually succeeded. - Carrying the full scrape after the marker. The pattern is a heuristic — a member reporting *about* a backend error matches it too — so failing is right, but dropping the rest of the pane would destroy the report, which is the same defect #164 exists to fix. The remaining half of #164 (points 3 and 4, a real classification mechanism instead of hard-coded strings) stays open as #201.
ltms closed this pull request 2026-08-31 17:07:42 +02:00
Some checks are pending
CI / contract (pull_request) Successful in 1m18s
CI / build (pull_request) Successful in 1m28s

Pull request closed

Sign in to join this conversation.