diff --git a/docs/CB-308-Multi-Host-Federation.md b/docs/CB-308-Multi-Host-Federation.md index 1a78ad7..7d24230 100644 --- a/docs/CB-308-Multi-Host-Federation.md +++ b/docs/CB-308-Multi-Host-Federation.md @@ -197,7 +197,9 @@ CB-308 doesn't have to repaint the topology. Settled in a design review of this note + wiki chapter 10. The broker-level operational rules (inbox caps, TLS + private broker, schema versioning, trace id, exclusive consumers, U8 broadcast) -are recorded in wiki 10 §10; the CB-308-side decisions are below. +are recorded in wiki 10 §10; the CB-308-side decisions are below. Entries 1–6 are the first-pass +decisions; 7–10 came out of the adversarial second-pass review (same day) and supersede 1–6 where +they overlap (notably: the envelope is no longer optional, and dedup is split by path). 1. **Sender authenticity — sign every message.** Each gateway holds its own signing key and signs what it publishes (sender gid, `msgId`, timestamp). The receiving gateway verifies the @@ -230,17 +232,72 @@ are recorded in wiki 10 §10; the CB-308-side decisions are below. deliver). Gateways auto-reconnect; remote hosts read as unknown in the roster meanwhile. Broker HA is a later ops choice, not a design requirement. +7. **Turn state — split by where the signals are.** The *worker's* gateway owns the turn record + (turnId minting, ask coalescing, STALE_TURN, the completion/failure fallbacks, CB-516 abandon): + every input to those decisions — pane status, injection, teardown — is local to it. The + *sender's* gateway owns only the waiter. The two are stitched by terminal-outcome envelope + kinds (`REPLY` / `FAILED` / `ABANDONED`) published to the sender's inbox: a worker dying on B + fails A's waiter fast because gateway B sees the death synchronously and says so. + **`ABANDONED` is belt-and-braces over the waiter's own timeout and roster expiry, never a + replacement** — the case where the waiter hangs longest is gateway B itself dying, which is + exactly when B can publish nothing. +8. **Dual ack model + spawn idempotence by construction.** Forward path (a brief into a worker): + ack **before** the inject — at-most-once, duplicates structurally impossible; the loss window + is closed by an `INJECTED` confirmation published after the inject lands (no `INJECTED` within + a bound = loud fast failure at the sender, not a silent send-timeout). Reply/pull path keeps + ack-after-drain — a duplicate reply is benign, deduped by `msgId`. Spawn: the requester mints + **spawn id = the new worker's gid**; the target checks it against the **live pane registry**, + and the gid is **stored in the herdr pane itself** (label/env, readable back), so a restarted + gateway rebuilds gid↔pane from herdr and the check survives restarts with *no persisted + ledger* — this storage point is the load-bearing detail of the no-ledger position. An + **in-flight reservation set**, entered before the launcher call, absorbs a redelivery arriving + while the first spawn is still inside CB-306's readiness gate; a crash mid-spawn leaves a + half-built pane, which is exactly what CB-117 reaps. +9. **Publish is enforced, not fire-and-forget.** Publisher confirms + the `mandatory` flag + a + return listener, on a **publish channel separate from the consume/ack channel** — synchronous + confirms on the single shared channel would hold its lock across a broker round trip and + serialize acks fleet-wide. Ordering caveat: a *return* (unroutable) arrives **before** the + confirm, so "confirmed" ≠ "routed"; the sender checks the returned-set at confirm time. + `mandatory` is false only for `BROADCAST`, where an empty group is legal silence. +10. **Queue lifecycle is session lifecycle.** `bridge_stop`/reap deletes the worker's inbox queue + (its `broadcast.*` bindings die with it — no broadcasts to the dead); `x-expires` collects + queues orphaned by a crashed gateway (long for main/orchestrator inboxes, short for workers). + Queue names carry a version suffix (`.v2`): AMQP refuses to redeclare an existing durable + queue with new arguments (`PRECONDITION_FAILED` — a crash loop on an in-place upgrade from + v1.0.0), and the suffix keeps old sender-keyed and new recipient-keyed queues apart during + the keying migration (wiki 10 §3 footnote). + ## 8. Still open - **Directory ground-truth:** pure soft-state presence (heartbeats) vs. also treating broker queue existence as authoritative. Lean soft-state to preserve the persistence boundary; revisit if split-brain roster views cause mis-routing. -- **Global id scheme:** `/` (human-legible, leaks host) vs. opaque UUID (clean, needs - the directory to resolve host). Probably UUID in the protocol, host as directory metadata. - **Gateway discovery:** how gateways find the broker and each other (static config vs. discovery). -- **Control authorization:** signing (§7.1) settles *who sent it*; still open is *who may do - what* — which leaders may spawn or stop on which hosts (allowlist vs. a capability token per - control action — ties into CB-401 Stage-C). +- **Control authorization — THE GATE ON U4.** Signing (§7.1) settles *who sent it*; authorization + is *who may do what*. **Cross-host spawn must not land before the minimal version exists**: a + per-host allowlist in `bridged.yaml` — beside the peer public keys — of gateway ids permitted to + publish control to this host, checked against the verified signature. A few lines of config and + check; without them, any principal holding broker credentials can start processes on every host + in the fleet. +- **Key distribution & rotation:** static config (host → public key in each `bridged.yaml`) is + fine at the current 2–3 host scale; rotation is manual. A refinement, not a blocker. - **Gateway death mid-turn:** the roster reaps it by missed heartbeat, and in-flight primary-bound messages survive by broker durability; still open is reconciling *worker* state when the dead gateway's host comes back (orphaned panes vs. still-valid sessions). + +*(Resolved and moved up: the global id scheme — an opaque UUID minted by the spawn requester as +the spawn id, host carried as roster metadata; §7.8.)* + +## 9. Implementation order (each step verifiable single-host) + +1. **As-built fixes, independent of CB-308** (v1.0.x tickets): `basicQos` prefetch on the AMQP + consumer (today the queue drains into gateway heap, so any cap would guard an empty queue); + publisher confirms + `mandatory` (§7.9); the `drainReplies` javadoc that claims "the ack is + local" — false for the AMQP adapter. +2. Envelope + signing (wiki 10 §2.1) — testable against the single-host broker. +3. Recipient-keyed queue migration (`.v2` names, drain-by-`from`, `ReplyPushLoop` rekeyed). +4. Global id + queue lifecycle (§7.8, §7.10). +5. Roster: host-level heartbeat + signed presence; then the routing fork (§3.2). +6. U2 cross-host with the terminal-outcome kinds (§7.4, §7.7). +7. U4 cross-host spawn — **gated on the control allowlist (§8)**. +8. U8 broadcast **last** — it is the feature that punishes an unfinished queue lifecycle. diff --git a/wiki b/wiki index 36bb865..4320c1c 160000 --- a/wiki +++ b/wiki @@ -1 +1 @@ -Subproject commit 36bb86588aa550e51c4227fc6edc1eebb1b62d6c +Subproject commit 4320c1ca5263fb40ce1ca8637a99cc60d5998ffe