fleetd/AUDIT.md

Audit: async ticket / rendezvous lifecycle (fleetd/src/main/java/dev/ltms/fleet/msg/)

Scope: Rendezvous.java and MessageService.java — the lifecycle of an async ticket (Task) and a rendezvous waiter: create, send, ask, answer, resolve, timeout, abandon, prune.

Main finding

1. fleetd/src/main/java/dev/ltms/fleet/msg/MessageService.java:938
2. issue: answer() completes an async ticket's future with a QUESTION outcome when the worker
   asks a second fleet_ask in the same resumed turn, permanently mislabeling a live delegation
   as failed and losing its real reply to fleet_poll.
3. fix: guard the finishAsyncTask(turnId, result) call at line 938 the same way sendAsync's
   lambda already guards its own call (lines 999-1005): skip it when result.outcome() ==
   Outcome.QUESTION, and instead re-associate the task with the new turnId (as markAsyncQuestion
   does on the first ask).
4. severity: high

Call sequence that reaches it

1. Lead: fleet_send{sessionId: W, content: "task", wait:false} → sendAsync creates task1 / ticket1. Its worker thread calls send(W, content, ASYNC_TIMEOUT_MS, onAccepted, task1), which does asyncTasksByWaiter.put(reply, task1) (line 802) before blocking on reply.get().
2. Worker W calls fleet_ask{"Q1"} → ask(W, "Q1", t). markAsyncQuestion finds task1 via asyncTasksByWaiter, stamps task1.turnId = turnId1, asyncTasksByTurn[turnId1] = task1. resolveQuestion wakes step 1's send(), which returns Outcome.QUESTION; sendAsync's lambda sees QUESTION and deliberately does not call finishAsyncTask (lines 1000-1005) — ticket1 correctly polls Phase.ASKING.
3. Lead polls, sees ASKING, answers: fleet_send{turnId: turnId1, content: "A1"} → answer(turnId1, "A1", t). This opens a new waiter via rendezvous.open(workerSession) (line 929) but — unlike send() — never puts it into asyncTasksByWaiter. answerAsk(turnId1, "A1") unblocks the worker's ask() call. clearAsyncQuestion(turnId1, false) clears task1.question but keeps asyncTasksByTurn[turnId1] = task1 (deliberate, per its own javadoc). answer() then blocks on its own reply.get() (line 936).
4. Worker W, still in the same resumed turn, calls fleet_ask{"Q2"} again before replying → a second ask(W, "Q2", t). openAsk mints turnId2. markAsyncQuestion looks up asyncTasksByWaiter.get(waiter) for the waiter answer() opened in step 3 — not found (never registered), so task == null; task1.turnId stays turnId1, no asyncTasksByTurn[turnId2] entry is ever created. resolveQuestion still succeeds (it only needs a live waiter, not a Task) and wakes answer(turnId1,...)'s blocked reply.get() with Resolution(QUESTION, "Q2", turnId2).
5. answer(turnId1,...) (line 936-939): result = Reply(Outcome.QUESTION, "Q2", turnId2); finishAsyncTask(turnId1, result) looks up task1 by the original turnId1 (still stamped from step 3) and unconditionally does task1.future.complete(result) — completing ticket1's future with a QUESTION outcome, then removes asyncTasksByTurn[turnId1]. answer() returns Outcome.QUESTION to the lead's own fleet_send{turnId1,...} call (correct, and separately answerable via turnId2), but ticket1 is now terminally done.

What goes wrong

• fleet_poll{ticket1} now hits the f.isDone() branch in poll() permanently. Outcome.QUESTION is not REPLIED/COMPLETED_UNREPLIED (r.completed() is false) and carries no WORKER_FAILED/BACKEND_EXHAUSTED reason, so it falls through to Phase.FAILED, detail = "no reply — question" — even though the worker is alive and only waiting on turnId2.
• asyncTasksByTurn no longer has any entry for task1/target, so hasAsyncQuestion(target) goes back to false immediately, and hasOrphanedDelegation no longer excludes this target's real state correctly either.
• If the worker's eventual real fleet_reply (after turnId2 is answered, or times out and it finishes on its own) is not captured by a chained direct answer(turnId2,...) call, reply()'s fast path (rendezvous.resolve) finds no live waiter, askAnsweredAsyncTasks finds no candidate (task1.future.isDone() is already true, so it is excluded), and the reply is silently dropped into the inbox as a stranded reply — unreachable from ticket1 and from abandon()'s stranded-reply recovery (no open matching task exists any more).

Confidence

High. I traced this with no races or interleavings assumed beyond the documented, deterministic CB-205 chained-ask protocol that outcomeOf/answer() already generically support (mapping Kind.QUESTION through answer()'s own return value is clearly intentional — see the Reply.turnId() javadoc). I did not run the suite, but grepped src/test/java/dev/ltms/fleet/msg/MessageServiceTest.java for a test exercising a second fleet_ask inside one resumed (answered) turn and found none — asyncQuestionBelongsToTheTaskThatOwnsItsForwardWaiter and askSurfacesAsAQuestionAndTheAnswerResumesTheSameTurn both cover only a single ask per turn. A git log -p on this file also turned up the CB-588 comment (now at sendAsync, describing the whenComplete hook) which explicitly frames answer()'s finishAsyncTask(turnId, result) as firing "once a QUESTION is resolved" — i.e. the author modeled that call as inherently terminal, which is exactly the assumption this bug violates when the resumed turn asks again.

Secondary (much shorter)

1. Root-cause detail, same defect as above — answer() (line 929) never puts its freshly opened waiter into asyncTasksByWaiter, unlike send() (line 802). Even if the outcome guard above is added, a chained second ask still can't be re-attached to task1 via the normal markAsyncQuestion path without also fixing this registration gap.
2. Low / shape only — pruneTerminalTickets() (line ~1092) is only invoked from inside sendAsync(). A fleet whose sessions stop receiving new async sends (e.g. everything now goes through blocking send(), or the target churns and workers are torn down) never prunes its already-terminal tasks entries past TICKET_TTL_NANOS. Not reachable as a "stuck" ticket (tickets still resolve correctly), only as unbounded tasks/asyncTasksByWaiter-adjacent memory growth over a long-lived daemon with no further sendAsync traffic; did not verify this is realistic in production traffic patterns, flagging as a shape only.