CB-611: our checkers keep covering less than they look like they cover — three instances in one day #113
Closed
opened 2026-08-17 14:24:44 +02:00 by ltms
·
3 comments
No Branch/Tag Specified
main
worker/fleetd-612-unita-87807e-1
worker/612-b3-mcpwirings-da2b58-3
worker/612-b2-cb185-176d3a-2
worker/612-b1-completion-457459-1
worker/612-agaps-73a926-2
worker/608-sleeps-3a64ff-3
worker/621-b4520b-1
worker/618-b83894-2
worker/fleetd-615-e05481-5
worker/lead-autocompact-5f1ab2-3
worker/fleetd-613-f85deb-3
worker/fleetd-608-flaky-nudge-test-d0c2d1-3
worker/lead-context-gauge-ad404f-1
worker/gauge-wiring-9158c1-4
worker/redeploy-slowstart-ead0e5-5
worker/charter-bytes-13668c-6
worker/rollover-outcome-291483-2
worker/589-f64303-2
worker/593-1a8025-5
worker/589-fcd2aa-1
worker/568-9fdaa2-3
worker/571-attempted-outcome-5739f7-2
worker/581-completionresolver-cas-sites-0542b7-6
worker/562-loop-health-wiring-test-99611c-5
worker/562-surface-loop-health-7df5cc-4
worker/575-waiter-cleanup-sites-62ad80-1
worker/572-answer-lock-release-46a9ae-5
worker/567-probe-channel-leak-a38fc5-6
worker/551-record-before-send-7cbf56-1
worker/561-listener-fanout-survives-a-throw-61d538-2
worker/555-redeploy-main-flow-seam-65c2f5-2
worker/556-injector-owns-registration-e027a5-1
worker/552-post-restart-mktemp-abort-bc2672-4
worker/553-onstatus-completion-leak-0da881-2
worker/550-shasum-linux-196132-1
worker/538-loop-dies-on-error-4a5eeb-6
worker/426-health-coverage-ef1fd4-4
worker/504-failed-reported-clean-3cfd66-3
worker/537-capturedlog-close-e4c437-2
worker/459-broken-link-targets-cadc17-5
worker/535-appender-leak-fe74c1-1
worker/512-part2-shutdown-detection-434701-9
worker/529-logger-level-sweep-2a5533-8
worker/528-drain-gate-call-site-5de83d-7
charter/forge-mcp-vs-token
worker/521-swap-guard-unpinned-28e931-5
worker/519-probe-test-harness-d25ab8-4
worker/525-logger-level-leak-1b4eb0-6
worker/518-fleetmcp-resolver-wiring-8ef96c-1
worker/512-drain-complete-line-7edd71-3
worker/517-abort-branch-and-jar-id-41b641-2
worker/500-9e52c9-3
worker/509-4912f4-2
worker/511-9a4b23-1
worker/493-479f45-2
worker/505-03f8b2-1
worker/492-followup-detect-unclear
worker/501-a31fa0-7
worker/498-451d1c-5
worker/494-1015ce-2
worker/492-209647-1
worker/489-001902-2
worker/480-relative-handover-path-906323-1
worker/480-b-handover-skill-45bf1f-5
worker/474-followup-source-pin-f54a55-17
worker/474-charter-check-on-reload-f54a55-17
worker/466-quarantine-repeatcount-report
worker/393-opencode-skill-seeding-71854b-13
worker/469-canonical-tool-names-2a472a-16
worker/466-quarantine-escalation-5ae9c1-15
worker/446-hot-exhausted-pattern-0af580-6
worker/464-charter-tool-name-guard-a85635-12
worker/463-listfleet-default-fails-open-f1c76c-11
worker/458-invariant-5-by-purpose-862f9a-10
worker/439-coordinator-row-gate-bc032a-8
worker/449-herdr-protocol-576015-4
worker/450-abstract-spawn-599e1c-5
worker/437-ack-refuses-177d91-1
worker/444-placement-window-feb56a-2
worker/440-helddurable-derived-d462d7-13
worker/425-rework-placement-resolve-c58ba1-9
worker/421-lead-peek-held-msgs-cdbad2-10
worker/435-fixed-policy-cap-fe11de-12
worker/422-gate-state-observability-9e79d6-11
worker/431-memberregistry-live-readers-cdbad2-10
worker/424-architect-slot-hot-038b41-7
worker/422-model-gate-spawn-c29f48-6
worker/425-default-profile-live-f55534-8
worker/415-coverage-wording-2cbf9c-5
worker/416-3ad1da-1
worker/418-588283-3
worker/deterministic-stamp-race-409-3cb7b6-10
worker/armed-reads-live-config-404-ed931f-9
worker/reply-peer-refusal-391-5a34bd-7
worker/models-allowlist-aa9e9b-3
worker/ttl-stamp-race-399-f1122f-8
worker/scrub-receipt-400-316b3e-5
worker/exhaustion-detection-395-105105-6
worker/scrub-abort-394-316b3e-5
fix/scrub-uid-abort
worker/task-scrub-517574-2
worker/t386-clock-bd5b78-4
worker/t384-scrub-813790-5
worker/t381-cc-748314-2
worker/t373-336973-2
worker/t365-3920c5-3
worker/t358-6e989b-1
worker/t355-8b321c-1
worker/fleetd-369-hermetic-git-tests-e8b19a-3
worker/fleetd-368-stale-lead-binding-f5682e-2
worker/fleetd-360-deploy-units-0d3793-1
worker/359-dead-lead-tabs-f1253b-4
worker/362-worktree-skills-c03e51-3
worker/361-coord-visibility-655144-1
362-plugin-visibility-and-drift
worker/errscan-bed2ca-2
worker/amqp-log-identity-bed2ca-2
worker/withdefaults-guard-561704
worker/sleepguard-82076d-1
worker/fd334-9ee1b6-5
worker/fd348-f1ab27-4
worker/fd335-a71c35-1
worker/fd342-174a17-2
worker/fd345-490d0f-3
worker/fleetd-337-5ec7d4-21
worker/fleetd-341-af5a6b-24
worker/fleetd-339-5ca0a2-23
worker/fleetd-338-83a4a1-22
worker/fleetd-333-281f46-18
worker/fleetd-329-11bdbb-16
worker/fleetd-330-2770fb-17
worker/fix-326-50506e-15
worker/fix-324-3e9bbf-14
worker/fix-323-b8287d-13
worker/fix-316b-bd0860-11
worker/fix-318-76ca36-9
worker/fix-317-486aec-8
worker/fix-315-ce47c5-6
worker/fix-307-275890-6
worker/fix-308-b4f664-7
worker/fix-309-ec3939-8
worker/fix-310-7a3974-9
worker/fix-302-52ad0e-9
worker/fix-298-ce1acb-8
worker/fix-297-66bd11-7
worker/fix-296-104622-6
worker/fix-293-bare-closetab-eb22b5-3
worker/fix-280-gone-ask-lapse-bca98e-2
worker/fix-290-reapidle-guard-coverage-9b0dd1-1
worker/fix-285-trust-seed-8f3565-10
worker/fix-284-backend-error-seat-85912c-11
worker/fix-282-chained-ask-e6d0bb-8
worker/fix-283-teardown-leaks-f40dfa-9
worker/fix-281-pin-handler-actions-4921ac-7
worker/audit-rendezvous-lifecycle-d072ae-2
worker/audit-health-placement-1a2476-6
worker/audit-teardown-exits-e207a5-3
worker/audit-launcher-asymmetry-27e370-4
worker/audit-rest-authz-6ca53c-5
worker/investigate-275-abandon-asking-fdef52-8
worker/fix-274-worktree-leak-b0095d-7
worker/fix-273-exhausted-pattern-9665b5-6
worker/fleetd-267-model-check-bd8068-1
worker/fleetd-131-archunit-18b834-7
worker/fleetd-266-sshagent-rename-a014ff-6
worker/fleetd-184-uid-claim-8e1f31-4
worker/fleetd-184-warn-b381ee-10
worker/fleetd-184-docs-be1d12-9
worker/fleetd-257-9bf010-7
worker/fleetd-103-23a113-6
worker/fleetd-247-342356-5
worker/fleetd-116-04dea8-4
worker/fleetd-252-a830e0-3
worker/fleetd-111-7e8673-9
worker/fleetd-155c-f8ef4b-8
worker/fleetd-176-b928ca-3
worker/fleetd-249-7a7878-2
worker/cb248-composition-root-b-9acdf7-15
worker/cb148-envrc-default-fa6c82-12
worker/cb201-unit5-wiring-6c12e6-8
worker/cb241-fallback-echo-1175e9-11
worker/cb149-trust-dialog-2392a5-9
worker/cb134-148-overlay-visible-c9b986-10
worker/cb234-session-id-keyed-04e1fc-1
worker/cb201-unit3-nudge-abdf5c-6
worker/cb201-unit2-policy-c1102c-5
worker/cb201-unit4-outcome-a13bfa-7
worker/cb201-unit1-classifier-91b9b1-4
worker/cb201-227-refine-831980-3
worker/cb175-model-readback-0f085f-1
worker/cb222-charter-tmpdir-17f013-1
worker/cb226-architect-slot-race-cd3aa8-3
worker/cb224-worktree-root-group-024523-2
worker/cb-123-role-demotion-c600f7-2
worker/cb-219-opencode-roots-1f677e-1
worker/cb214-claude-session-id-b9eab4-4
worker/cb213-zdotdir-wrong-process-dd6de4-3
worker/cb211-exhaustion-classification-9546e0-2
worker/cb137-ambiguous-task-4df3d8-4
worker/cb209-agentsessionid-4dfdb6-2
worker/cb185-hostenvnames-2692b5-3
worker/cb206-opencode-sqlite-128718-2
worker/cb185-worktree-group-fc0c99-1
worker/cb-137-ask-ticket-e7760c-2
worker/cb-172-broker-uri-d36ae4-4
worker/cb-175-model-readback-76ead6-3
worker/cb-161-pane-ancestry-293510-1
worker/cb-164-rebase-885863-8
worker/cb-164-empty-scrape-false-success-1a80af-3
fix/cb-197-ticket-ttl-from-completion
worker/cb-189-remote-url-coverage-4692f3-1
worker/cb-185-blockers-027756-4
worker/cb-192-gap-log-11b631-2
worker/cb-633-fix-5f4396-3
worker/cb185-router-d6436d-3
worker/cb185-router-routing-gaps-9e9d33-3
worker/cb185-paneids-992586-2
worker/cb-633-allow-list-union-ed374b-1
worker/cb-157-credential-in-remote-url-496e44-2
worker/cb-641-health-herdr-evidence-8f1f54-6
worker/cb-640-health-msg-evidence-99c9cd-1
worker/cb-642-fleets-status-skill-bbbc40-5
cb-634-ide-mcp
worker/lead-comms-wiring-c014b9-7
worker/lead-mailbox-c19577-6
worker/autocompact-window-82bc2f-5
worker/cb-634-probe-18056f-4
worker/cb635-broker-urienv
worker/cb-632-config-retry-8e0efa-7
lead/cb-622e-claude-md
lead/cb-622-followup
worker/cb-622a-165dff-1
lead/cb-622d-opencode-mount
worker/cb-622b-717c67-2
worker/cb-622c-ab7759-3
worker/cb-617b2-20ca4b-3
worker/cb-617a-5c2f4a-1
worker/cb596-4e49ef-3
worker/cb586-10500c-1
worker/cb-606-b9343a-25
worker/cb604-1445f8-24
worker/cb582-477374-21
worker/cb584-8c2281-22
worker/cb600-e6b9a9-20
worker/cb602-ce257f-19
worker/cb601-b42837-18
worker/cb598-6c7ba7-17
worker/cb599-740fe4-16
worker/cb597-282224-15
worker/cb590fix-185e9a-10
worker/cb528-recovery-race
worker/cb594-96bead-8
worker/cb590-916766-2
worker/cb527-997d99-3
worker/cb592-env-leak-3cbf9c-1
worker/cb588-async-ticket-nudge-3218f7-5
worker/cb578b-9dcb13-6
worker/cb581-d24826-5
worker/m2-u5-ef8c42-15
worker/cb578a-516499-2
worker/cb576-01a04b-17
worker/cb579-lead-tab-acba06-20
worker/cb580-terminal-health-ed6058-21
worker/cb577-f36fdc-18
worker/cb573b-3db06f-16
worker/cb568c-f36fdc-18
worker/cb568-drop-cause-c3ac1c
worker/cb575-cancelled-notification-c3ac1c
worker/m4-sol-a2cbec-3
worker/cb574-async-ask-c3ac1c
worker/cb573-health-model-8ca857-14
worker/cb572-unknown-target-7f2e35-13
worker/u4-700706-9
worker/u3-b9fcb6-6
worker/u2-ef5b68-4
worker/u1-469dce-1-clean
worker/u1-469dce-1
worker/cb-564-health-events-70cf7e-2
worker/cb-565-recycle-drops-role-98e58f-3
worker/cb-563-missing-reply-df2866-1
worker/cb-562-readiness-gate-silent-6c23c9-3
worker/cb-560-architect-presence-da8155-1
worker/cb-561-architect-silent-off-a71cab-2
worker/cb-548-bind-architect-slot-fe1b8c-1
worker/parity-overlay-settings-5fb711-1
secrets-central-store
cb-559-hot-key-correction
cb-557-fleet-role-pools
worker/cb-553-maxload-explicit-spawn-305ee3-6
worker/cb-551-idle-lead-heartbeat-f1633c-1
worker/cb-544-drain-preserves-worktree-925fad-3
worker/cb-552-docs-sync-1cb9cf-4
worker/cb-548-rendezvous-guard-rebased
worker/cb-548-rendezvous-guard-116b53-10
worker/cb-548-authz-v2-586df6-8
worker/cb-548-authz-264363-5
salvage/cb-528b-codex-home
salvage/cb-528a-codex-launcher
CB-518-primary-flow
feature/peer-launcher-spi
cb-103-injector
v1.1.0
v1.0.0
Labels
Clear labels
blocked
needs-live-proof
ready-to-delegate
silent-default
Cannot start until something else lands. The body says what.
Merged and green, but never shown working on the running daemon. Not the same as done.
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
2.0 — one operation centre, many hosts
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: fleet/fleetd#113
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Filed on 2026-08-17 after a pre-tag scan. This is a pattern ticket, not a single defect. Three separate checkers were examined today and all three passed while the thing they exist to catch was present.
The three instances
1. CB-602's config guard —
BridgedConfigTest#everyKnownTopLevelKeyIsDocumentedInTheExample, commit32bf324.It checks one direction only (code → example) and matches only keys anchored at column 0 (regex
^(?:#\s*)?<key>:,BridgedConfigTest.java:1308-1311). Consequences:profiles.<name>.subscription) got through — the test passes today with that gap open. Verified by running it:Tests run: 1, Failures: 0.health.workingSuspectAfterSecondsandhealth.paneProbeIntervalSecondsare both parsed into theHealthrecord and read by nothing.The reverse direction is nominally covered by
everyOptionalKnobDocumentedInTheExampleBinds(:1162), but that test carries a hand-written list of keys, so it has the same failure mode one level up.2.
scripts/probe-member-credentials.sh— already filed as CB-608 (#111). Hardcodes 31 names, never reads the policy, reported 26 blocked against a policy of 29, exit 0.3. CB-586's unit tests — already closed, recorded here because it completes the shape. Every test called the seam directly and walked around the interval gate, so a sweep that never ran once had a fully green test suite.
The shape
In all three, the checker is narrower than the thing it is understood to check, and nothing says so. A green run is then read as "covered", which is worse than no checker, because it stops anyone looking.
Two recurring mechanisms:
everyOptionalKnobDocumentedInTheExampleBindshas one. A list written once drifts silently from the thing it mirrors — which is the same defect CB-596 was filed to fix in the credential policy itself.What to do
subscriptionand on the two deadhealth.*keys.everyOptionalKnobDocumentedInTheExampleBinds, or derive it. A guard whose coverage is a literal is not a guard.26 blockednext to a policy of29is only detectable because someone knew to compare. A checker that prints "checked 26 of 29" reports its own gap.Acceptance criteria
Milestone
2.0. Nothing is misbehaving on one host; these are gaps in how we verify, and the specific defect the config guard let through is filed separately as CB-610. Grouped here so the next person sees the pattern rather than three unrelated tickets.
Related
CB-608 (#111) · CB-610 (#112) · CB-586 (#67, closed) · CB-602 (#96, closed) · CB-596 (#82, closed)
Three more instances from one session (2026-09-03), all found while verifying #201/#227 Units 1-4 and #234. Adding them because each is a different way a checker reports more coverage than it has.
1. A
defaultoverload makes a fix invisible to every existing lambda#234's first round added
onExhausted(target, reason, profile)as adefaultmethod.Fleetd.java:177held a two-argument lambda, so it kept compiling, kept calling the old path, and the profile hint never reached production. The fix was dead and every test passed.Round two was worse: the new tests built their own forwarder rather than using the production one. I mutated production back to a plain lambda and got 52 tests, 0 failures. The test could not fail, because it never held the object it claimed to test.
The fix that actually worked was to invert the interface — make the 3-argument method abstract and the 2-argument one
default— so the compiler rejects the bad shape. The denominator lesson: when you add an overload, the unit of work is every implementation, and adefaultmethod is invisible to a lambda.2. A compile error reads as a green mutation run
Twice I mutated production, saw
Tests run: N, Failures: 0, and nearly recorded "not covered". Both times the build had actually failed to compile — a mutation left a dangling@Override. A compile failure prints zero test failures, so it is indistinguishable from a passing run if you only read the test line.Every mutation run now prints its own denominator first:
A red is only a kill when that count is
0. This is the same shape as the ticket's core complaint, applied to the checking process itself.3. A clean auto-merge is not a compiling merge
Units 1-4 were file-disjoint in production but shared
CompletionResolverTest. Merging #234's interface inversion and then Unit 1 produced no conflict markers at all —git mergereported clean — and the build failed with exactly one error:A merge tool compares text; lambda arity is a type fact. Only a compiler can see it. So "the merge was clean" is not evidence of anything, and each parallel unit now gets its own build after merging rather than one build at the end.
What I changed in how units are briefed
Every delegation now carries three lines, because each maps to one of the above:
The third one caught a real gap in Unit 4 that the full suite could not see.
Instance 1 (the config guard) is fixed on
main—21c539f.First: two parts of this ticket had gone stale
The ticket was filed 2026-08-17. Checked today before starting:
bf616e1documentedsubscription:, so it is no longer available as the demonstration case criterion 1 asks for. I used a different key instead (below).health.workingSuspectAfterSecondsis no longer dead. It is read by one file outsideFleetConfignow.health.paneProbeIntervalSecondsis still read by nothing — and the example already says so in its own text ("parsed, but nothing reads it yet — changing it changes nothing"), so that one is documented honestly rather than hidden.The structural defect was still entirely real, which is what got fixed.
What changed
The old guard walked
KNOWN_TOP_LEVEL_KEYSwith a regex anchored at column 0. It could see 22 top-level keys. The record tree holds 83 distinct key names across 17 records, so roughly three quarters of the schema was outside its scope, and neither its name nor its output said so.Two derived guards now replace that:
everyNestedConfigKeyIsDocumentedInTheExampleeveryLiveKeyInTheExampleBindsToARecordComponentNeither carries a list, so a key added to any nested record is covered the moment it compiles.
Acceptance criteria
1 — both directions fail on an injected key, proved through the real caller. The ticket is explicit that asserting the helper in isolation is the defect, so both were run end to end:
0 compile errors in both runs. Both reverted, confirmed with
diff -q.Worth recording: my first attempt at mutation 1 stayed green, and was right to. I had removed only the
paneProbeIntervalSeconds:line, and the key was still documented in a prose block, so it was still documented. An incomplete mutation proves nothing about the guard. Redone removing every mention, it failed as it should.2 — no check in this family establishes coverage with a literal.
everyOptionalKnobDocumentedInTheExampleBindskeeps its hand-written list, because it asserts real values bind, which no name-matching guard can do. It is now documented as a value-binding spot check that is explicitly not a coverage guard, with a pointer to the two derived tests that are. Leaving a knob out of that list is no longer a coverage gap.3 — each prints its own denominator. Both report how many keys they examined. The "did the walk actually descend?" floor is itself derived (
> KNOWN_TOP_LEVEL_KEYS.size() * 2) rather than a literal — an under-counting walk passes every subset check vacuously, which is this ticket's own failure mode aimed at the fix.That floor earned its place immediately: I first guessed
>= 100and the guard failed at 83. I then counted the components by parsing the source independently — 83 distinct across 17 records — and the walk was complete. The guess was wrong, not the walk.One finding worth keeping
The new guard's first real run flagged
broker.urias undocumented. It is not: it is documented in the example's prose convention (# uri → AMQP connection URI…) but never as a copy-pasteableuri:key.That omission looks deliberate. Writing out
uri: amqp://user:pass@hostinvites an operator to paste a password into a file, which is the thinguriEnvexists to avoid. A guard that demanded the key form would have pushed the file toward doing exactly that. So the matcher accepts both conventions — it encodes the convention the example really uses instead of imposing a new one.Scope, stated rather than implied
Written into the javadoc so the next reader does not have to infer it:
# mode: tokenthat are indistinguishable from keys by text alone; parsing them would produce false failures.paneProbeIntervalSecondspasses both guards. Proving a key is live code needs a call graph.Still open on this ticket
Instance 2 (
scripts/probe-member-credentials.sh, #111) is in progress separately. Instance 3 is closed. Leaving this open until #111 lands.Full build at
21c539f: 1236 tests, 0 failures, 0 compile errors.All three instances are now closed. Closing this pattern ticket.
FleetConfigTest)21c539f— walks the record tree, both directions, no listscripts/probe-member-credentials.sh(#111)6b5f3f4, verified live: 34 checked / 29 blocked / 5 presentWhat the three had in common, restated now they are all fixed
Both live fixes worked the same way: delete the second copy, derive it instead. The config guard derives its key set from
FleetConfig's record components; the probe derives its name list from the daemon. Neither can drift, because neither holds a copy any more.And both now print a denominator — "checked 83 config key(s)", "policy contains 34 name(s); this run checked 34 — they match." That is the cheap half of the fix and it is what would have caught all three originally.
26 blockedwas only detectable as wrong because somebody happened to know the policy said 29.Two things this ticket got wrong by the time it was picked up
Recorded because a stale premise costs the next person a run:
bf616e1), sosubscription:was no longer available as criterion 1's demonstration case. A different key was used.health.workingSuspectAfterSecondsis no longer dead.paneProbeIntervalSecondsstill is — and the example already says so in its own text, so it is documented rather than hidden.A caveat that stays open on purpose
Neither new guard proves a parsed key is read by anything.
paneProbeIntervalSecondspasses both. That needs a call graph, and pretending otherwise would recreate this ticket's own defect — a checker that looks wider than it is. It is stated in the javadoc instead.A fourth instance turned up while fixing this one, in my own tooling rather than the codebase: the
find -newermt '-5 minutes'check used to tell whether a member is making progress matches nothing on macOS, because BSDfinddoes not parse the GNU relative form. It does not error — it returns 0 every time, which reads as "this member is stalled". I reported two healthy members as dead before a one-line denominator probe (toucha file, then look for it) showed the check itself could not detect anything. Same shape, same fix: make the checker prove it can see something before believing its zero.