CB-308: fold in the adversarial review; bump wiki to 4320c1c
Four new resolved decisions (§7.7-7.10): turn state split by where the signals are, with ABANDONED explicitly belt-and-braces over the waiter timeout; the dual ack model with spawn idempotence by construction (gid stored IN the herdr pane — the load-bearing detail of the no-ledger position — plus an in-flight reservation for redelivery during a slow spawn); enforced publish semantics (confirms + mandatory on a separate channel, return-before-confirm caveat); queue lifecycle = session lifecycle with .v2 names for the redeclare hazard. §8 reworked: global id scheme resolved and moved up; control authorization sharpened into the hard gate on U4 (per-host allowlist beside the peer keys); key distribution/rotation added. New §9: implementation order, each step verifiable single-host, U4 gated, U8 last. Wiki pointer bumped to 4320c1c (chapter 10 same-pass changes). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ZGgxLQ2VpwZhEYoru8rkf
This commit is contained in:
@@ -197,7 +197,9 @@ CB-308 doesn't have to repaint the topology.
|
||||
|
||||
Settled in a design review of this note + wiki chapter 10. The broker-level operational rules
|
||||
(inbox caps, TLS + private broker, schema versioning, trace id, exclusive consumers, U8 broadcast)
|
||||
are recorded in wiki 10 §10; the CB-308-side decisions are below.
|
||||
are recorded in wiki 10 §10; the CB-308-side decisions are below. Entries 1–6 are the first-pass
|
||||
decisions; 7–10 came out of the adversarial second-pass review (same day) and supersede 1–6 where
|
||||
they overlap (notably: the envelope is no longer optional, and dedup is split by path).
|
||||
|
||||
1. **Sender authenticity — sign every message.** Each gateway holds its own signing key and signs
|
||||
what it publishes (sender gid, `msgId`, timestamp). The receiving gateway verifies the
|
||||
@@ -230,17 +232,72 @@ are recorded in wiki 10 §10; the CB-308-side decisions are below.
|
||||
deliver). Gateways auto-reconnect; remote hosts read as unknown in the roster meanwhile. Broker
|
||||
HA is a later ops choice, not a design requirement.
|
||||
|
||||
7. **Turn state — split by where the signals are.** The *worker's* gateway owns the turn record
|
||||
(turnId minting, ask coalescing, STALE_TURN, the completion/failure fallbacks, CB-516 abandon):
|
||||
every input to those decisions — pane status, injection, teardown — is local to it. The
|
||||
*sender's* gateway owns only the waiter. The two are stitched by terminal-outcome envelope
|
||||
kinds (`REPLY` / `FAILED` / `ABANDONED`) published to the sender's inbox: a worker dying on B
|
||||
fails A's waiter fast because gateway B sees the death synchronously and says so.
|
||||
**`ABANDONED` is belt-and-braces over the waiter's own timeout and roster expiry, never a
|
||||
replacement** — the case where the waiter hangs longest is gateway B itself dying, which is
|
||||
exactly when B can publish nothing.
|
||||
8. **Dual ack model + spawn idempotence by construction.** Forward path (a brief into a worker):
|
||||
ack **before** the inject — at-most-once, duplicates structurally impossible; the loss window
|
||||
is closed by an `INJECTED` confirmation published after the inject lands (no `INJECTED` within
|
||||
a bound = loud fast failure at the sender, not a silent send-timeout). Reply/pull path keeps
|
||||
ack-after-drain — a duplicate reply is benign, deduped by `msgId`. Spawn: the requester mints
|
||||
**spawn id = the new worker's gid**; the target checks it against the **live pane registry**,
|
||||
and the gid is **stored in the herdr pane itself** (label/env, readable back), so a restarted
|
||||
gateway rebuilds gid↔pane from herdr and the check survives restarts with *no persisted
|
||||
ledger* — this storage point is the load-bearing detail of the no-ledger position. An
|
||||
**in-flight reservation set**, entered before the launcher call, absorbs a redelivery arriving
|
||||
while the first spawn is still inside CB-306's readiness gate; a crash mid-spawn leaves a
|
||||
half-built pane, which is exactly what CB-117 reaps.
|
||||
9. **Publish is enforced, not fire-and-forget.** Publisher confirms + the `mandatory` flag + a
|
||||
return listener, on a **publish channel separate from the consume/ack channel** — synchronous
|
||||
confirms on the single shared channel would hold its lock across a broker round trip and
|
||||
serialize acks fleet-wide. Ordering caveat: a *return* (unroutable) arrives **before** the
|
||||
confirm, so "confirmed" ≠ "routed"; the sender checks the returned-set at confirm time.
|
||||
`mandatory` is false only for `BROADCAST`, where an empty group is legal silence.
|
||||
10. **Queue lifecycle is session lifecycle.** `bridge_stop`/reap deletes the worker's inbox queue
|
||||
(its `broadcast.*` bindings die with it — no broadcasts to the dead); `x-expires` collects
|
||||
queues orphaned by a crashed gateway (long for main/orchestrator inboxes, short for workers).
|
||||
Queue names carry a version suffix (`.v2`): AMQP refuses to redeclare an existing durable
|
||||
queue with new arguments (`PRECONDITION_FAILED` — a crash loop on an in-place upgrade from
|
||||
v1.0.0), and the suffix keeps old sender-keyed and new recipient-keyed queues apart during
|
||||
the keying migration (wiki 10 §3 footnote).
|
||||
|
||||
## 8. Still open
|
||||
|
||||
- **Directory ground-truth:** pure soft-state presence (heartbeats) vs. also treating broker queue
|
||||
existence as authoritative. Lean soft-state to preserve the persistence boundary; revisit if
|
||||
split-brain roster views cause mis-routing.
|
||||
- **Global id scheme:** `<host>/<paneId>` (human-legible, leaks host) vs. opaque UUID (clean, needs
|
||||
the directory to resolve host). Probably UUID in the protocol, host as directory metadata.
|
||||
- **Gateway discovery:** how gateways find the broker and each other (static config vs. discovery).
|
||||
- **Control authorization:** signing (§7.1) settles *who sent it*; still open is *who may do
|
||||
what* — which leaders may spawn or stop on which hosts (allowlist vs. a capability token per
|
||||
control action — ties into CB-401 Stage-C).
|
||||
- **Control authorization — THE GATE ON U4.** Signing (§7.1) settles *who sent it*; authorization
|
||||
is *who may do what*. **Cross-host spawn must not land before the minimal version exists**: a
|
||||
per-host allowlist in `bridged.yaml` — beside the peer public keys — of gateway ids permitted to
|
||||
publish control to this host, checked against the verified signature. A few lines of config and
|
||||
check; without them, any principal holding broker credentials can start processes on every host
|
||||
in the fleet.
|
||||
- **Key distribution & rotation:** static config (host → public key in each `bridged.yaml`) is
|
||||
fine at the current 2–3 host scale; rotation is manual. A refinement, not a blocker.
|
||||
- **Gateway death mid-turn:** the roster reaps it by missed heartbeat, and in-flight primary-bound
|
||||
messages survive by broker durability; still open is reconciling *worker* state when the dead
|
||||
gateway's host comes back (orphaned panes vs. still-valid sessions).
|
||||
|
||||
*(Resolved and moved up: the global id scheme — an opaque UUID minted by the spawn requester as
|
||||
the spawn id, host carried as roster metadata; §7.8.)*
|
||||
|
||||
## 9. Implementation order (each step verifiable single-host)
|
||||
|
||||
1. **As-built fixes, independent of CB-308** (v1.0.x tickets): `basicQos` prefetch on the AMQP
|
||||
consumer (today the queue drains into gateway heap, so any cap would guard an empty queue);
|
||||
publisher confirms + `mandatory` (§7.9); the `drainReplies` javadoc that claims "the ack is
|
||||
local" — false for the AMQP adapter.
|
||||
2. Envelope + signing (wiki 10 §2.1) — testable against the single-host broker.
|
||||
3. Recipient-keyed queue migration (`.v2` names, drain-by-`from`, `ReplyPushLoop` rekeyed).
|
||||
4. Global id + queue lifecycle (§7.8, §7.10).
|
||||
5. Roster: host-level heartbeat + signed presence; then the routing fork (§3.2).
|
||||
6. U2 cross-host with the terminal-outcome kinds (§7.4, §7.7).
|
||||
7. U4 cross-host spawn — **gated on the control allowlist (§8)**.
|
||||
8. U8 broadcast **last** — it is the feature that punishes an unfinished queue lifecycle.
|
||||
|
||||
+1
-1
Submodule wiki updated: 36bb86588a...4320c1ca52
Reference in New Issue
Block a user