Fleetd's TurnListener.onDelivered ordering is load-bearing and untested: swapping two lines makes a throw strand the caller for its full timeout #561
Open
opened 2026-09-12 11:01:31 +02:00 by ltms
·
3 comments
No Branch/Tag Specified
main
worker/fleetd-612-unita-87807e-1
worker/612-b3-mcpwirings-da2b58-3
worker/612-b2-cb185-176d3a-2
worker/612-b1-completion-457459-1
worker/612-agaps-73a926-2
worker/608-sleeps-3a64ff-3
worker/621-b4520b-1
worker/618-b83894-2
worker/fleetd-615-e05481-5
worker/lead-autocompact-5f1ab2-3
worker/fleetd-613-f85deb-3
worker/fleetd-608-flaky-nudge-test-d0c2d1-3
worker/lead-context-gauge-ad404f-1
worker/gauge-wiring-9158c1-4
worker/redeploy-slowstart-ead0e5-5
worker/charter-bytes-13668c-6
worker/rollover-outcome-291483-2
worker/589-f64303-2
worker/593-1a8025-5
worker/589-fcd2aa-1
worker/568-9fdaa2-3
worker/571-attempted-outcome-5739f7-2
worker/581-completionresolver-cas-sites-0542b7-6
worker/562-loop-health-wiring-test-99611c-5
worker/562-surface-loop-health-7df5cc-4
worker/575-waiter-cleanup-sites-62ad80-1
worker/572-answer-lock-release-46a9ae-5
worker/567-probe-channel-leak-a38fc5-6
worker/551-record-before-send-7cbf56-1
worker/561-listener-fanout-survives-a-throw-61d538-2
worker/555-redeploy-main-flow-seam-65c2f5-2
worker/556-injector-owns-registration-e027a5-1
worker/552-post-restart-mktemp-abort-bc2672-4
worker/553-onstatus-completion-leak-0da881-2
worker/550-shasum-linux-196132-1
worker/538-loop-dies-on-error-4a5eeb-6
worker/426-health-coverage-ef1fd4-4
worker/504-failed-reported-clean-3cfd66-3
worker/537-capturedlog-close-e4c437-2
worker/459-broken-link-targets-cadc17-5
worker/535-appender-leak-fe74c1-1
worker/512-part2-shutdown-detection-434701-9
worker/529-logger-level-sweep-2a5533-8
worker/528-drain-gate-call-site-5de83d-7
charter/forge-mcp-vs-token
worker/521-swap-guard-unpinned-28e931-5
worker/519-probe-test-harness-d25ab8-4
worker/525-logger-level-leak-1b4eb0-6
worker/518-fleetmcp-resolver-wiring-8ef96c-1
worker/512-drain-complete-line-7edd71-3
worker/517-abort-branch-and-jar-id-41b641-2
worker/500-9e52c9-3
worker/509-4912f4-2
worker/511-9a4b23-1
worker/493-479f45-2
worker/505-03f8b2-1
worker/492-followup-detect-unclear
worker/501-a31fa0-7
worker/498-451d1c-5
worker/494-1015ce-2
worker/492-209647-1
worker/489-001902-2
worker/480-relative-handover-path-906323-1
worker/480-b-handover-skill-45bf1f-5
worker/474-followup-source-pin-f54a55-17
worker/474-charter-check-on-reload-f54a55-17
worker/466-quarantine-repeatcount-report
worker/393-opencode-skill-seeding-71854b-13
worker/469-canonical-tool-names-2a472a-16
worker/466-quarantine-escalation-5ae9c1-15
worker/446-hot-exhausted-pattern-0af580-6
worker/464-charter-tool-name-guard-a85635-12
worker/463-listfleet-default-fails-open-f1c76c-11
worker/458-invariant-5-by-purpose-862f9a-10
worker/439-coordinator-row-gate-bc032a-8
worker/449-herdr-protocol-576015-4
worker/450-abstract-spawn-599e1c-5
worker/437-ack-refuses-177d91-1
worker/444-placement-window-feb56a-2
worker/440-helddurable-derived-d462d7-13
worker/425-rework-placement-resolve-c58ba1-9
worker/421-lead-peek-held-msgs-cdbad2-10
worker/435-fixed-policy-cap-fe11de-12
worker/422-gate-state-observability-9e79d6-11
worker/431-memberregistry-live-readers-cdbad2-10
worker/424-architect-slot-hot-038b41-7
worker/422-model-gate-spawn-c29f48-6
worker/425-default-profile-live-f55534-8
worker/415-coverage-wording-2cbf9c-5
worker/416-3ad1da-1
worker/418-588283-3
worker/deterministic-stamp-race-409-3cb7b6-10
worker/armed-reads-live-config-404-ed931f-9
worker/reply-peer-refusal-391-5a34bd-7
worker/models-allowlist-aa9e9b-3
worker/ttl-stamp-race-399-f1122f-8
worker/scrub-receipt-400-316b3e-5
worker/exhaustion-detection-395-105105-6
worker/scrub-abort-394-316b3e-5
fix/scrub-uid-abort
worker/task-scrub-517574-2
worker/t386-clock-bd5b78-4
worker/t384-scrub-813790-5
worker/t381-cc-748314-2
worker/t373-336973-2
worker/t365-3920c5-3
worker/t358-6e989b-1
worker/t355-8b321c-1
worker/fleetd-369-hermetic-git-tests-e8b19a-3
worker/fleetd-368-stale-lead-binding-f5682e-2
worker/fleetd-360-deploy-units-0d3793-1
worker/359-dead-lead-tabs-f1253b-4
worker/362-worktree-skills-c03e51-3
worker/361-coord-visibility-655144-1
362-plugin-visibility-and-drift
worker/errscan-bed2ca-2
worker/amqp-log-identity-bed2ca-2
worker/withdefaults-guard-561704
worker/sleepguard-82076d-1
worker/fd334-9ee1b6-5
worker/fd348-f1ab27-4
worker/fd335-a71c35-1
worker/fd342-174a17-2
worker/fd345-490d0f-3
worker/fleetd-337-5ec7d4-21
worker/fleetd-341-af5a6b-24
worker/fleetd-339-5ca0a2-23
worker/fleetd-338-83a4a1-22
worker/fleetd-333-281f46-18
worker/fleetd-329-11bdbb-16
worker/fleetd-330-2770fb-17
worker/fix-326-50506e-15
worker/fix-324-3e9bbf-14
worker/fix-323-b8287d-13
worker/fix-316b-bd0860-11
worker/fix-318-76ca36-9
worker/fix-317-486aec-8
worker/fix-315-ce47c5-6
worker/fix-307-275890-6
worker/fix-308-b4f664-7
worker/fix-309-ec3939-8
worker/fix-310-7a3974-9
worker/fix-302-52ad0e-9
worker/fix-298-ce1acb-8
worker/fix-297-66bd11-7
worker/fix-296-104622-6
worker/fix-293-bare-closetab-eb22b5-3
worker/fix-280-gone-ask-lapse-bca98e-2
worker/fix-290-reapidle-guard-coverage-9b0dd1-1
worker/fix-285-trust-seed-8f3565-10
worker/fix-284-backend-error-seat-85912c-11
worker/fix-282-chained-ask-e6d0bb-8
worker/fix-283-teardown-leaks-f40dfa-9
worker/fix-281-pin-handler-actions-4921ac-7
worker/audit-rendezvous-lifecycle-d072ae-2
worker/audit-health-placement-1a2476-6
worker/audit-teardown-exits-e207a5-3
worker/audit-launcher-asymmetry-27e370-4
worker/audit-rest-authz-6ca53c-5
worker/investigate-275-abandon-asking-fdef52-8
worker/fix-274-worktree-leak-b0095d-7
worker/fix-273-exhausted-pattern-9665b5-6
worker/fleetd-267-model-check-bd8068-1
worker/fleetd-131-archunit-18b834-7
worker/fleetd-266-sshagent-rename-a014ff-6
worker/fleetd-184-uid-claim-8e1f31-4
worker/fleetd-184-warn-b381ee-10
worker/fleetd-184-docs-be1d12-9
worker/fleetd-257-9bf010-7
worker/fleetd-103-23a113-6
worker/fleetd-247-342356-5
worker/fleetd-116-04dea8-4
worker/fleetd-252-a830e0-3
worker/fleetd-111-7e8673-9
worker/fleetd-155c-f8ef4b-8
worker/fleetd-176-b928ca-3
worker/fleetd-249-7a7878-2
worker/cb248-composition-root-b-9acdf7-15
worker/cb148-envrc-default-fa6c82-12
worker/cb201-unit5-wiring-6c12e6-8
worker/cb241-fallback-echo-1175e9-11
worker/cb149-trust-dialog-2392a5-9
worker/cb134-148-overlay-visible-c9b986-10
worker/cb234-session-id-keyed-04e1fc-1
worker/cb201-unit3-nudge-abdf5c-6
worker/cb201-unit2-policy-c1102c-5
worker/cb201-unit4-outcome-a13bfa-7
worker/cb201-unit1-classifier-91b9b1-4
worker/cb201-227-refine-831980-3
worker/cb175-model-readback-0f085f-1
worker/cb222-charter-tmpdir-17f013-1
worker/cb226-architect-slot-race-cd3aa8-3
worker/cb224-worktree-root-group-024523-2
worker/cb-123-role-demotion-c600f7-2
worker/cb-219-opencode-roots-1f677e-1
worker/cb214-claude-session-id-b9eab4-4
worker/cb213-zdotdir-wrong-process-dd6de4-3
worker/cb211-exhaustion-classification-9546e0-2
worker/cb137-ambiguous-task-4df3d8-4
worker/cb209-agentsessionid-4dfdb6-2
worker/cb185-hostenvnames-2692b5-3
worker/cb206-opencode-sqlite-128718-2
worker/cb185-worktree-group-fc0c99-1
worker/cb-137-ask-ticket-e7760c-2
worker/cb-172-broker-uri-d36ae4-4
worker/cb-175-model-readback-76ead6-3
worker/cb-161-pane-ancestry-293510-1
worker/cb-164-rebase-885863-8
worker/cb-164-empty-scrape-false-success-1a80af-3
fix/cb-197-ticket-ttl-from-completion
worker/cb-189-remote-url-coverage-4692f3-1
worker/cb-185-blockers-027756-4
worker/cb-192-gap-log-11b631-2
worker/cb-633-fix-5f4396-3
worker/cb185-router-d6436d-3
worker/cb185-router-routing-gaps-9e9d33-3
worker/cb185-paneids-992586-2
worker/cb-633-allow-list-union-ed374b-1
worker/cb-157-credential-in-remote-url-496e44-2
worker/cb-641-health-herdr-evidence-8f1f54-6
worker/cb-640-health-msg-evidence-99c9cd-1
worker/cb-642-fleets-status-skill-bbbc40-5
cb-634-ide-mcp
worker/lead-comms-wiring-c014b9-7
worker/lead-mailbox-c19577-6
worker/autocompact-window-82bc2f-5
worker/cb-634-probe-18056f-4
worker/cb635-broker-urienv
worker/cb-632-config-retry-8e0efa-7
lead/cb-622e-claude-md
lead/cb-622-followup
worker/cb-622a-165dff-1
lead/cb-622d-opencode-mount
worker/cb-622b-717c67-2
worker/cb-622c-ab7759-3
worker/cb-617b2-20ca4b-3
worker/cb-617a-5c2f4a-1
worker/cb596-4e49ef-3
worker/cb586-10500c-1
worker/cb-606-b9343a-25
worker/cb604-1445f8-24
worker/cb582-477374-21
worker/cb584-8c2281-22
worker/cb600-e6b9a9-20
worker/cb602-ce257f-19
worker/cb601-b42837-18
worker/cb598-6c7ba7-17
worker/cb599-740fe4-16
worker/cb597-282224-15
worker/cb590fix-185e9a-10
worker/cb528-recovery-race
worker/cb594-96bead-8
worker/cb590-916766-2
worker/cb527-997d99-3
worker/cb592-env-leak-3cbf9c-1
worker/cb588-async-ticket-nudge-3218f7-5
worker/cb578b-9dcb13-6
worker/cb581-d24826-5
worker/m2-u5-ef8c42-15
worker/cb578a-516499-2
worker/cb576-01a04b-17
worker/cb579-lead-tab-acba06-20
worker/cb580-terminal-health-ed6058-21
worker/cb577-f36fdc-18
worker/cb573b-3db06f-16
worker/cb568c-f36fdc-18
worker/cb568-drop-cause-c3ac1c
worker/cb575-cancelled-notification-c3ac1c
worker/m4-sol-a2cbec-3
worker/cb574-async-ask-c3ac1c
worker/cb573-health-model-8ca857-14
worker/cb572-unknown-target-7f2e35-13
worker/u4-700706-9
worker/u3-b9fcb6-6
worker/u2-ef5b68-4
worker/u1-469dce-1-clean
worker/u1-469dce-1
worker/cb-564-health-events-70cf7e-2
worker/cb-565-recycle-drops-role-98e58f-3
worker/cb-563-missing-reply-df2866-1
worker/cb-562-readiness-gate-silent-6c23c9-3
worker/cb-560-architect-presence-da8155-1
worker/cb-561-architect-silent-off-a71cab-2
worker/cb-548-bind-architect-slot-fe1b8c-1
worker/parity-overlay-settings-5fb711-1
secrets-central-store
cb-559-hot-key-correction
cb-557-fleet-role-pools
worker/cb-553-maxload-explicit-spawn-305ee3-6
worker/cb-551-idle-lead-heartbeat-f1633c-1
worker/cb-544-drain-preserves-worktree-925fad-3
worker/cb-552-docs-sync-1cb9cf-4
worker/cb-548-rendezvous-guard-rebased
worker/cb-548-rendezvous-guard-116b53-10
worker/cb-548-authz-v2-586df6-8
worker/cb-548-authz-264363-5
salvage/cb-528b-codex-home
salvage/cb-528a-codex-launcher
CB-518-primary-flow
feature/peer-launcher-spi
cb-103-injector
v1.1.0
v1.0.0
Labels
Clear labels
blocked
needs-live-proof
ready-to-delegate
silent-default
Cannot start until something else lands. The body says what.
Merged and green, but never shown working on the running daemon. Not the same as done.
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: fleet/fleetd#561
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Measured on
mainat7a3b2bb, after #553 merged (PR #557).Raised indirectly by the fleet01 lead, who argued that after #553's rework the surviving
sentHandledflag still conflates attempted with succeeded, so a throw part way throughonDeliveredyields "a success receipt over a possibly-unregistered waiter". I measured it and the live defect they described is not reachable — but the reason it is not reachable is an untested two-line ordering, which is a real fragility and is what this ticket is for. Their instinct found something; it is one layer over from where they placed it.Why the described defect is not live
Three measurements, each of which has to hold:
1. The caller does not block on the delivery future.
MessageService.java:919opens the rendezvous waiter before enqueue (CB-548), and:938blocks onreply.get(...). The delivery future is consulted only inside theTimeoutExceptionhandler:So
delivered()is not a success receipt the caller acts on. It is a delivery-fact label that picksTIMED_OUT_WORKINGoverTIMED_OUT_QUEUED. Completing it withcomplete(null)when the text really was typed into the pane is the honest value, not a lie.2. Completing it exceptionally would change nothing observable. On the exceptional branch the handler falls through to
:950:and
Injector.java:288-290:The Pending is already
DELIVERED, sowasDeliveredcomes back true anyway and the outcome isTIMED_OUT_WORKINGeither way. The proposed change is observationally a no-op that adds onecancel()call.3. The in-flight record cannot currently be missing when a later listener throws.
CompletionResolver.captureBaselinealready fails open around its only throwing step:and the production listener calls the resolver first:
So by the time anything in that method can realistically throw, the record is in place.
The actual defect: nothing protects that ordering
Every one of those three facts is load-bearing, and the third is two adjacent lines with no test. Swap
:514and:515— a refactor, an alphabetisation, an IDE "sort members", someone groupingsessions.*calls together — and the fleet01 lead's scenario becomes live immediately:sessions.onDeliveredthrows (it moves the session toBUSYand bumps the turn count atSessionManager.java:895),completion.onDeliverednever runs, soinFlighthas no record,Injector'ssentHandledis alreadytrue, so thefinallycorrectly suppresses the re-call and completes the delivery future,CompletionResolver.resolvefindsturn == nulland returns having resolved nobody,No test fails. The whole suite passes, because no test asserts anything about the order of those two calls.
This is the invariant-not-idiom shape the fleet01 lead named on #556, pointed at their own finding. Do not test the line order. Name the invariant and test that:
Suggested acceptance
TurnListenercomposed the wayFleetdcomposes it, where the session half throws, assertingCompletionResolver.inFlight(target)is non-null afterwards — i.e. the resolver half already ran.inFlight(String)atCompletionResolver.java:269is an existing package-private hook, so no new seam is needed.onTurnCompleteandonTurnFailedatFleetd.java:518-524, which have the identical two-call shape and the same unprotected ordering.tryso one throwing delegate cannot skip the others. Note this trades one silent failure for another if it swallows, so whatever escapes must still reachStatusPoller'scatch (Throwable). That was the same tension #553 had to resolve.shasum -a 256, green control, and run the proof cell against the un-mutated tree first to confirm it reports not-applied.What I am explicitly not doing
Not reopening #553 and not changing how the delivery future completes. Measurements 1 and 2 above say
complete(null)is the accurate value and the alternative is a no-op wearing a correction's clothes.Credit
The fleet01 lead, from their own reasoning with no access to this tree — they flagged that #553's surviving flag records intent and asked, correctly, whether
delivered()'s consumers handle exceptional completion, naming that as the grep they could not run. That grep is measurement 1 and 2 above, and running it is what located the real fragility one layer over.Related: #553, #556 (the same invariant-not-idiom test design), #551, CB-116, CB-548.
This ticket's test is a regression guard, not a fix — and it can entrench the real defect
Raised by the fleet01 lead, whose finding this ticket already credits. Recording it here because it changes what "done" means for #561, and because the trap is invisible once the test is green.
Their point, in their words:
They are right, and it is their own invariant-not-idiom rule pointed back at this ticket.
What #561's test can prove: that
Fleetd.java:514and:515are in that order today, and that swapping them goes red. That is real and worth having — a two-line swap by an IDE sort-members currently costs the caller its full timeout with a green suite.What it cannot prove: the invariant this ticket names. The completion resolver's in-flight record must be registered before any other
onDeliveredlistener can throw is not a fact about line order. It is a fact about who owns the registration. Today theInjectorstates an invariant that only a listener callback can satisfy, so any listener anyone adds later can break it — from any position — and the order test stays green while they do.So this ticket is explicitly scoped down
:514/:515swap.The distinction matters because the two tests fail on different days. #561's goes red when someone reorders two lines. #556's goes red today, and turns green only when the contract is fixed — which is what an acceptance criterion is for.
Cross-referenced on #556.
Re-measured on
mainatba2f4d1, after #556 merged. This ticket is now half done and half still live. The body is out of date in a way that would send a worker to write a test that already exists, so read this comment as the current spec — it is newer than the body and it wins.The
onDeliveredhalf is structurally fixed. Do not work on it.#556 moved registration off the
TurnListenerfan-out entirely.Fleetd.java:531-535:The
Injectornow callscompletion.registeritself, unconditionally, beforeonDeliveredruns at all. So the scenario in the body — swap the two lines inonDelivered,sessionsthrows,inFlighthas no record — can no longer happen. The order of those two lines stopped being load-bearing.It is also already pinned, by a test that exists:
InjectorTest.aTurnListenerThatThrowsFromEveryCallbackStillLeavesTheDeliveredTurnRegisteredAndResolvable. I mutated the registration call away and it goes red on its own. There is nothing left to add there.Three callbacks still have the exact shape the body describes, and #556 did nothing for them
Fleetd.java, measured today:Every one is two unguarded calls where the
completionhalf is the one that resolves the waiter. Today the order is safe becausecompletiongoes first. Swap either pair, or have the first call throw, and the second never runs.The consequence is the same one the body names, and it is worse than a lost registration because there is no later chance to recover it:
CompletionResolver.onTurnCompleteis what readsinFlightand starts the resolving virtual thread (CompletionResolver.java:301-307). If it never runs, nothing ever resolves that waiter. The caller burns its full timeout and the answer strands in the inbox.No test fails if you swap them. I checked: nothing asserts the order, and nothing asserts the survival.
The invariant to test — not the line order
Same framing the body already got right, restated for what is actually left:
Note this is a different invariant from the one #556 fixed. #556's is about a record being written on delivery; this one is about a resolution being triggered on completion. A test for one proves nothing about the other — they are separate instantiations, and that is exactly why #556 landing did not close this.
Acceptance
Fleetdcomposes it, where the session half throws, asserting the completion half still had its effect. ForonTurnCompletethat means the waiter is resolved, not merely thatinFlightwas read.StatusPoller'scatch (Throwable)and log at ERROR. A listener bug stays loud. This is the tension #553 had to resolve; do not resolve it by swallowing.sedonly, anchor counted withgrep -Fxc(notawk -v— it escape-processes the value and silently counts 0 on any line containing\tor\n), count down by exactly one, red with the test's own assertion message and observed value, restored byte-identical undershasum -a 256, green control, and the proof cell run against the un-mutated tree first to confirm it reports not-applied.Credit unchanged
Still the fleet01 lead's finding. Their instinct was right and was one layer over from where they placed it; #556 has since taken the first layer, and this is what remains of the second.
Merged as
ed2fd66(PR #570).Lead verification on a merged tree, re-running everything rather than accepting the worker's
numbers:
mvn -o clean installfromfleetd/, exit 0, 1761 tests from Maven and from anindependent sum over 131 surefire reports. The tree I built is byte-identical to
origin/main(
git rev-parse main^{tree}matches), so that result ismain's, not a candidate's.The part worth keeping
Round 1 looked complete — a real composition fix, five new tests, full green suite, and the
production wiring at
Fleetd.java:498genuinely calling the new factory, so the seam under test wasthe real caller. I mutated three lines the worker had not:
bothMustRunbothMustRunKeepingSecondResultfailure.addSuppressed(t)Two helpers maintain one invariant — "the second half always runs". Five tests existed. Four
asserted the direction "the completion half still resolves when the SESSION half throws". Exactly
one asserted the reverse, and it called
onTurnComplete, which routes through the first helperonly. So the second helper could be reverted to the broken form with the suite green.
The worker's own kills (the two
throwUnchecked(failure)lines) proved a different property —does not swallow — which is true and is not the claim the ticket exists for.
Both survivors are now killed, and I re-ran both myself after the fix rather than trusting the
report. Note the line moved 1191 → 1193 with the comment edit, so I located it fresh; assuming the
old number would have mutated the wrong line and produced a meaningless green.
The rule: count assertions PER SITE, not per invariant. The non-zero total is what hides the
zero. Extracting a shared helper makes this worse rather than better — it does not reduce the number
of sites, only how many are visible. Predicted as a general shape by the fleet01 lead before an
instance was found; a second, independent instance is #572.
The negative case
onDeliveredstays deliberately unguarded and that is correct.CompletionResolver.captureBaselinealready catches
RuntimeExceptionaround its scrape and fails open, so that half does notrealistically throw. The comment now says that, instead of its previous reason — which was true, but
about registration surviving via the #556
Injectorwiring, not about why this pair is safe. Acorrect conclusion resting on a wrong premise reads exactly like a verified one, which is why the
filter yields candidates and the callee decides.