"Extract the decision so the suite can call it" pins the decision and never the wiring — drain_gate_refusal's call site can be bypassed with the suite green #528
Closed
opened 2026-09-12 07:01:05 +02:00 by ltms
·
2 comments
No Branch/Tag Specified
main
worker/fleetd-612-unita-87807e-1
worker/612-b3-mcpwirings-da2b58-3
worker/612-b2-cb185-176d3a-2
worker/612-b1-completion-457459-1
worker/612-agaps-73a926-2
worker/608-sleeps-3a64ff-3
worker/621-b4520b-1
worker/618-b83894-2
worker/fleetd-615-e05481-5
worker/lead-autocompact-5f1ab2-3
worker/fleetd-613-f85deb-3
worker/fleetd-608-flaky-nudge-test-d0c2d1-3
worker/lead-context-gauge-ad404f-1
worker/gauge-wiring-9158c1-4
worker/redeploy-slowstart-ead0e5-5
worker/charter-bytes-13668c-6
worker/rollover-outcome-291483-2
worker/589-f64303-2
worker/593-1a8025-5
worker/589-fcd2aa-1
worker/568-9fdaa2-3
worker/571-attempted-outcome-5739f7-2
worker/581-completionresolver-cas-sites-0542b7-6
worker/562-loop-health-wiring-test-99611c-5
worker/562-surface-loop-health-7df5cc-4
worker/575-waiter-cleanup-sites-62ad80-1
worker/572-answer-lock-release-46a9ae-5
worker/567-probe-channel-leak-a38fc5-6
worker/551-record-before-send-7cbf56-1
worker/561-listener-fanout-survives-a-throw-61d538-2
worker/555-redeploy-main-flow-seam-65c2f5-2
worker/556-injector-owns-registration-e027a5-1
worker/552-post-restart-mktemp-abort-bc2672-4
worker/553-onstatus-completion-leak-0da881-2
worker/550-shasum-linux-196132-1
worker/538-loop-dies-on-error-4a5eeb-6
worker/426-health-coverage-ef1fd4-4
worker/504-failed-reported-clean-3cfd66-3
worker/537-capturedlog-close-e4c437-2
worker/459-broken-link-targets-cadc17-5
worker/535-appender-leak-fe74c1-1
worker/512-part2-shutdown-detection-434701-9
worker/529-logger-level-sweep-2a5533-8
worker/528-drain-gate-call-site-5de83d-7
charter/forge-mcp-vs-token
worker/521-swap-guard-unpinned-28e931-5
worker/519-probe-test-harness-d25ab8-4
worker/525-logger-level-leak-1b4eb0-6
worker/518-fleetmcp-resolver-wiring-8ef96c-1
worker/512-drain-complete-line-7edd71-3
worker/517-abort-branch-and-jar-id-41b641-2
worker/500-9e52c9-3
worker/509-4912f4-2
worker/511-9a4b23-1
worker/493-479f45-2
worker/505-03f8b2-1
worker/492-followup-detect-unclear
worker/501-a31fa0-7
worker/498-451d1c-5
worker/494-1015ce-2
worker/492-209647-1
worker/489-001902-2
worker/480-relative-handover-path-906323-1
worker/480-b-handover-skill-45bf1f-5
worker/474-followup-source-pin-f54a55-17
worker/474-charter-check-on-reload-f54a55-17
worker/466-quarantine-repeatcount-report
worker/393-opencode-skill-seeding-71854b-13
worker/469-canonical-tool-names-2a472a-16
worker/466-quarantine-escalation-5ae9c1-15
worker/446-hot-exhausted-pattern-0af580-6
worker/464-charter-tool-name-guard-a85635-12
worker/463-listfleet-default-fails-open-f1c76c-11
worker/458-invariant-5-by-purpose-862f9a-10
worker/439-coordinator-row-gate-bc032a-8
worker/449-herdr-protocol-576015-4
worker/450-abstract-spawn-599e1c-5
worker/437-ack-refuses-177d91-1
worker/444-placement-window-feb56a-2
worker/440-helddurable-derived-d462d7-13
worker/425-rework-placement-resolve-c58ba1-9
worker/421-lead-peek-held-msgs-cdbad2-10
worker/435-fixed-policy-cap-fe11de-12
worker/422-gate-state-observability-9e79d6-11
worker/431-memberregistry-live-readers-cdbad2-10
worker/424-architect-slot-hot-038b41-7
worker/422-model-gate-spawn-c29f48-6
worker/425-default-profile-live-f55534-8
worker/415-coverage-wording-2cbf9c-5
worker/416-3ad1da-1
worker/418-588283-3
worker/deterministic-stamp-race-409-3cb7b6-10
worker/armed-reads-live-config-404-ed931f-9
worker/reply-peer-refusal-391-5a34bd-7
worker/models-allowlist-aa9e9b-3
worker/ttl-stamp-race-399-f1122f-8
worker/scrub-receipt-400-316b3e-5
worker/exhaustion-detection-395-105105-6
worker/scrub-abort-394-316b3e-5
fix/scrub-uid-abort
worker/task-scrub-517574-2
worker/t386-clock-bd5b78-4
worker/t384-scrub-813790-5
worker/t381-cc-748314-2
worker/t373-336973-2
worker/t365-3920c5-3
worker/t358-6e989b-1
worker/t355-8b321c-1
worker/fleetd-369-hermetic-git-tests-e8b19a-3
worker/fleetd-368-stale-lead-binding-f5682e-2
worker/fleetd-360-deploy-units-0d3793-1
worker/359-dead-lead-tabs-f1253b-4
worker/362-worktree-skills-c03e51-3
worker/361-coord-visibility-655144-1
362-plugin-visibility-and-drift
worker/errscan-bed2ca-2
worker/amqp-log-identity-bed2ca-2
worker/withdefaults-guard-561704
worker/sleepguard-82076d-1
worker/fd334-9ee1b6-5
worker/fd348-f1ab27-4
worker/fd335-a71c35-1
worker/fd342-174a17-2
worker/fd345-490d0f-3
worker/fleetd-337-5ec7d4-21
worker/fleetd-341-af5a6b-24
worker/fleetd-339-5ca0a2-23
worker/fleetd-338-83a4a1-22
worker/fleetd-333-281f46-18
worker/fleetd-329-11bdbb-16
worker/fleetd-330-2770fb-17
worker/fix-326-50506e-15
worker/fix-324-3e9bbf-14
worker/fix-323-b8287d-13
worker/fix-316b-bd0860-11
worker/fix-318-76ca36-9
worker/fix-317-486aec-8
worker/fix-315-ce47c5-6
worker/fix-307-275890-6
worker/fix-308-b4f664-7
worker/fix-309-ec3939-8
worker/fix-310-7a3974-9
worker/fix-302-52ad0e-9
worker/fix-298-ce1acb-8
worker/fix-297-66bd11-7
worker/fix-296-104622-6
worker/fix-293-bare-closetab-eb22b5-3
worker/fix-280-gone-ask-lapse-bca98e-2
worker/fix-290-reapidle-guard-coverage-9b0dd1-1
worker/fix-285-trust-seed-8f3565-10
worker/fix-284-backend-error-seat-85912c-11
worker/fix-282-chained-ask-e6d0bb-8
worker/fix-283-teardown-leaks-f40dfa-9
worker/fix-281-pin-handler-actions-4921ac-7
worker/audit-rendezvous-lifecycle-d072ae-2
worker/audit-health-placement-1a2476-6
worker/audit-teardown-exits-e207a5-3
worker/audit-launcher-asymmetry-27e370-4
worker/audit-rest-authz-6ca53c-5
worker/investigate-275-abandon-asking-fdef52-8
worker/fix-274-worktree-leak-b0095d-7
worker/fix-273-exhausted-pattern-9665b5-6
worker/fleetd-267-model-check-bd8068-1
worker/fleetd-131-archunit-18b834-7
worker/fleetd-266-sshagent-rename-a014ff-6
worker/fleetd-184-uid-claim-8e1f31-4
worker/fleetd-184-warn-b381ee-10
worker/fleetd-184-docs-be1d12-9
worker/fleetd-257-9bf010-7
worker/fleetd-103-23a113-6
worker/fleetd-247-342356-5
worker/fleetd-116-04dea8-4
worker/fleetd-252-a830e0-3
worker/fleetd-111-7e8673-9
worker/fleetd-155c-f8ef4b-8
worker/fleetd-176-b928ca-3
worker/fleetd-249-7a7878-2
worker/cb248-composition-root-b-9acdf7-15
worker/cb148-envrc-default-fa6c82-12
worker/cb201-unit5-wiring-6c12e6-8
worker/cb241-fallback-echo-1175e9-11
worker/cb149-trust-dialog-2392a5-9
worker/cb134-148-overlay-visible-c9b986-10
worker/cb234-session-id-keyed-04e1fc-1
worker/cb201-unit3-nudge-abdf5c-6
worker/cb201-unit2-policy-c1102c-5
worker/cb201-unit4-outcome-a13bfa-7
worker/cb201-unit1-classifier-91b9b1-4
worker/cb201-227-refine-831980-3
worker/cb175-model-readback-0f085f-1
worker/cb222-charter-tmpdir-17f013-1
worker/cb226-architect-slot-race-cd3aa8-3
worker/cb224-worktree-root-group-024523-2
worker/cb-123-role-demotion-c600f7-2
worker/cb-219-opencode-roots-1f677e-1
worker/cb214-claude-session-id-b9eab4-4
worker/cb213-zdotdir-wrong-process-dd6de4-3
worker/cb211-exhaustion-classification-9546e0-2
worker/cb137-ambiguous-task-4df3d8-4
worker/cb209-agentsessionid-4dfdb6-2
worker/cb185-hostenvnames-2692b5-3
worker/cb206-opencode-sqlite-128718-2
worker/cb185-worktree-group-fc0c99-1
worker/cb-137-ask-ticket-e7760c-2
worker/cb-172-broker-uri-d36ae4-4
worker/cb-175-model-readback-76ead6-3
worker/cb-161-pane-ancestry-293510-1
worker/cb-164-rebase-885863-8
worker/cb-164-empty-scrape-false-success-1a80af-3
fix/cb-197-ticket-ttl-from-completion
worker/cb-189-remote-url-coverage-4692f3-1
worker/cb-185-blockers-027756-4
worker/cb-192-gap-log-11b631-2
worker/cb-633-fix-5f4396-3
worker/cb185-router-d6436d-3
worker/cb185-router-routing-gaps-9e9d33-3
worker/cb185-paneids-992586-2
worker/cb-633-allow-list-union-ed374b-1
worker/cb-157-credential-in-remote-url-496e44-2
worker/cb-641-health-herdr-evidence-8f1f54-6
worker/cb-640-health-msg-evidence-99c9cd-1
worker/cb-642-fleets-status-skill-bbbc40-5
cb-634-ide-mcp
worker/lead-comms-wiring-c014b9-7
worker/lead-mailbox-c19577-6
worker/autocompact-window-82bc2f-5
worker/cb-634-probe-18056f-4
worker/cb635-broker-urienv
worker/cb-632-config-retry-8e0efa-7
lead/cb-622e-claude-md
lead/cb-622-followup
worker/cb-622a-165dff-1
lead/cb-622d-opencode-mount
worker/cb-622b-717c67-2
worker/cb-622c-ab7759-3
worker/cb-617b2-20ca4b-3
worker/cb-617a-5c2f4a-1
worker/cb596-4e49ef-3
worker/cb586-10500c-1
worker/cb-606-b9343a-25
worker/cb604-1445f8-24
worker/cb582-477374-21
worker/cb584-8c2281-22
worker/cb600-e6b9a9-20
worker/cb602-ce257f-19
worker/cb601-b42837-18
worker/cb598-6c7ba7-17
worker/cb599-740fe4-16
worker/cb597-282224-15
worker/cb590fix-185e9a-10
worker/cb528-recovery-race
worker/cb594-96bead-8
worker/cb590-916766-2
worker/cb527-997d99-3
worker/cb592-env-leak-3cbf9c-1
worker/cb588-async-ticket-nudge-3218f7-5
worker/cb578b-9dcb13-6
worker/cb581-d24826-5
worker/m2-u5-ef8c42-15
worker/cb578a-516499-2
worker/cb576-01a04b-17
worker/cb579-lead-tab-acba06-20
worker/cb580-terminal-health-ed6058-21
worker/cb577-f36fdc-18
worker/cb573b-3db06f-16
worker/cb568c-f36fdc-18
worker/cb568-drop-cause-c3ac1c
worker/cb575-cancelled-notification-c3ac1c
worker/m4-sol-a2cbec-3
worker/cb574-async-ask-c3ac1c
worker/cb573-health-model-8ca857-14
worker/cb572-unknown-target-7f2e35-13
worker/u4-700706-9
worker/u3-b9fcb6-6
worker/u2-ef5b68-4
worker/u1-469dce-1-clean
worker/u1-469dce-1
worker/cb-564-health-events-70cf7e-2
worker/cb-565-recycle-drops-role-98e58f-3
worker/cb-563-missing-reply-df2866-1
worker/cb-562-readiness-gate-silent-6c23c9-3
worker/cb-560-architect-presence-da8155-1
worker/cb-561-architect-silent-off-a71cab-2
worker/cb-548-bind-architect-slot-fe1b8c-1
worker/parity-overlay-settings-5fb711-1
secrets-central-store
cb-559-hot-key-correction
cb-557-fleet-role-pools
worker/cb-553-maxload-explicit-spawn-305ee3-6
worker/cb-551-idle-lead-heartbeat-f1633c-1
worker/cb-544-drain-preserves-worktree-925fad-3
worker/cb-552-docs-sync-1cb9cf-4
worker/cb-548-rendezvous-guard-rebased
worker/cb-548-rendezvous-guard-116b53-10
worker/cb-548-authz-v2-586df6-8
worker/cb-548-authz-264363-5
salvage/cb-528b-codex-home
salvage/cb-528a-codex-launcher
CB-518-primary-flow
feature/peer-launcher-spi
cb-103-injector
v1.1.0
v1.0.0
Labels
Clear labels
blocked
needs-live-proof
ready-to-delegate
silent-default
Cannot start until something else lands. The body says what.
Merged and green, but never shown working on the running daemon. Not the same as done.
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: fleet/fleetd#528
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found while adjudicating #526 (fleetd #521). The fix I have prescribed three times in this file is incomplete, and this is my defect, not any implementer's. I measured both halves myself before filing.
The remedy, and what is wrong with it
Three times now, for a branch in
scripts/redeploy-fleetd.shthat no test could reach, the ticket said: extract the decision into a function so the suite can call it.wait_for_daemon_exitdrain_gate_refusalshould_swapEach time the new tests call the extracted function directly and pass. Each time nothing makes the main flow consult it. Extraction does not remove the untested decision; it moves it up one level, from "a branch no test can reach" to "a call site no test can reach". Coverage goes up while a new uncovered decision appears.
This is the same shape as the Java finding already recorded against
Fleetd.main— extraction moved the untested surface up and five call sites survived — so it is not specific to shell.Measured: #521's prescribed fix did not close #521
PR #526 did exactly what #521 asked:
should_swap(do_build), plus a test for each value. With the main flow then readingif should_swap "$DO_BUILD"; then:Proven applied: mutant string present (1), original string absent (0),
should_swapitself untouched (still defined). Restored byte-identical.Harness proof on my own invocation, so the green is readable: inverting
should_swap's own body gaveexit=1andFAIL: should_swap 1 (a build ran and staged a jar) must return true. The suite can fail when I run it. It does not fail for the call site.That is fixed in #526 as merged — the decision and the action now live together in
swap_if_built, which the main flow calls unconditionally, and two tests drive it with a recording stub. This ticket is about the other two.1.
drain_gate_refusal's call site is not pinned at allscripts/redeploy-fleetd.sh:683:Mutation — the main flow stops consulting it and goes back to the old flat message:
Proven applied: mutant present (1),
drain_gate_refusal "$DO_BUILD"absent (0), the function itself still defined (1). Restored byte-identical.This reinstates the exact defect #517 was filed to fix, with every test #520 added still green. #520's tests prove the four cases of
drain_gate_refusalproduce the right strings. Nothing proves the abort path uses it. Notetest_drain_gate_refusal_*and the kept source-text test at the--no-buildclaim both survive: the source-text test greps this script for the claim wording, and the wording is still in the function.Severity: the same as #517's — a wrong abort message during an abort the operator chose. Lower than #521's silent wrong outcome, but it is a regression that a merge one day old already permits.
2.
wait_for_daemon_exit's call site is pinned, but only by texttest_swap_ordered_after_wait_and_before_startgreps forwait_for_daemon_exit "$STOP_WAIT"and compares line positions, so deleting the call is caught. That is a source-text pin, with the known limit: it proves what the file says, never what runs. I did not find a mutation that makes the call present-but-ineffective while keeping that needle, but I also did not search exhaustively — treat this row as "partially pinned, not audited", not as clear.The fix
Do not add another extracted predicate. Use the shape that #526 ended up with:
For
drain_gate_refusalconcretely: arefuse_drain_gate(do_build, staged_path)that composes the message and callsdieitself, with the main flow calling that one function, and tests asserting the message the composite produces for each of the four cases.Then state plainly in a comment what the remaining hole is — deleting the call line altogether — and which test covers it.
Also fix: the test that was supposed to report a missing call site could not
While measuring the above, deleting the swap call made the suite exit 1 having printed zero bytes — no
FAIL:line, nothing naming what was missing. Cause, measured with a control:The file runs under
set -euo pipefail.pipefailmakes the pipeline's statusgrep's status, so an absent needle fails the assignment andset -ekills the suite on the spot — before the guard written to report exactly that case. Withpipefailoff, the guard is reached and prints. So all three|| fail "could not find …"guards in that test were dead code, in precisely the situation they existed for.Already fixed in #526 (each grep now ends
|| true), and the fix is proven: the same deletion now reportsFAIL: could not find the swap call site in redeploy-fleetd.sh. Flagged here because the pattern is likely elsewhere — anyx="$(cmd | cmd)"followed by an emptiness check, in any script underset -ewithpipefail, has a dead check. Sweep for it and report; do not fix outside this file in this ticket.Acceptance
bash scripts/test-redeploy-fleetd.shexits 0; report test functions defined and invoked, and they must match.bash -npasses under both/bin/bash(3.2.57) andenv bash(5.x).die "$(drain_gate_refusal …)"call with a flatdie "aborted — nothing changed", show the suite red, quote the FAIL line and exit code, restore, confirm byte-identical withshasum -a 256, then a green control.grep -nre-read of the line. Two traps, both of which have bitten me this week: a$-variable inside a double-quoted pattern is expanded by your own shell, and a\"left inside a single-quoted pattern stays in the pattern as a literal backslash. Both give a false zero that looks like an un-applied mutation. A genuinely un-applied mutation gives 0 and 1, never 0 and 0.pipefaildead-check pattern described above and list what you find. Do not fix those here.scripts/redeploy-fleetd.shagainst the live daemon, with any flag. It is the channel the fleet talks through.Related
should_swaphalf is fixed there.Attribution correction to the ticket body above.
I wrote that I measured the call-site gap. I did, but #526's implementer found it first and reported it in that PR's body under "Caveat for review" — including that the same residual gap already existed on #517's
drain_gate_refusalprecedent, which is item 1 of this ticket. I reproduced it independently before reading the body, so it is measured twice by two routes, but the finding was not mine. Full quote and credit in #526 (comment).Two corrections to this ticket's own content that follow from that report:
Severity of the
should_swapcase was overstated in #521, and I am carrying that error into this ticket's framing. #521 called it a "silent wrong outcome during a redeploy the operator believes succeeded". The implementer traced it:stage_built_jaris unconditional, so by the swap step$JARis already absent,java -jarfails at once, and the script dies atno process appearedor/healthz never answered within ${HEALTH_WAIT}s— both of which print the daemon log tail. It is a loud failure, not a silent one. The defect is still real (a guard that can be disabled with a green suite), but "reports success" was wrong. That trace was not run against the live daemon and I have not re-verified it.This does not change item 1's severity.
drain_gate_refusalproduces a wrong message on a path that still completes, so there is no loud failure to save it.A larger list exists and is not in this ticket. The sweep #521 asked for turned up only the two source-text-only tests already known. But the implementer also listed seven main-flow decisions past the
SOURCEDguard with no test of any kind — the threecase "$SUPERVISOR_KIND"dispatches (report display, stop, start),if [ "$CHECK_ONLY" = 1 ], the drain-gate's own entry condition and theif [ "$reply" != "yes" ]die inside it,if [ -z "$HEALTH_BODY" ], and the ok/warn summary. Those are listed in the comment linked above, with pre-merge line numbers that need re-measuring.Whoever takes this ticket: do not widen into those seven. They are a bigger piece of work and the stop dispatch in particular deserves its own ticket. Stay on the two items here.
Fixed in PR #532, merged. Closing.
Item 1 — fixed and pinned two ways
refuse_drain_gatecomposes the message viadrain_gate_refusaland callsdieitself; the main flow calls it unconditionally at:710. Verified in my own worktree at the pushed head7c34e8f, independently of the worker's report.^FAIL:, 256 bytes:710→ flatdie "aborted — nothing changed")^FAIL:line, 87 bytes —FAIL: could not find the main flow's refuse_drain_gate call site in redeploy-fleetd.shrefuse_drain_gatestops consulting the predicate^FAIL:line —FAIL: refuse_drain_gate build-ran+staged-present die message does not name the staged jar0e5a99a22c9c65f72960d8f179ca5299307889e06bc42131f098a513e7b97bd6, control greenbash -n/bin/bash3.2.57 andenv bash5.3.9The first cell is the whole point of this ticket. Before the change, that same mutation gave exit 0, zero FAIL lines, and output byte-identical to a clean run. The regression window #517 had reopened is closed, and closed in a way that reports rather than passes silently.
Each mutation was proven applied with a uniquely tagged marker plus a second different search string, with a control against a pristine copy showing the exact inverse, and the function definition confirmed still present so the mutation hit the call and not the function.
The self-match trap, caught by the implementer
The worker's first draft of the comment above
refuse_drain_gatequoted the call-site string literally. That would have given the source-text test a second match — in the comment — so the test would have passed with the real call deleted. They caught it themselves and reworded the comment to avoid the literal.grep -cF 'refuse_drain_gate "$DO_BUILD" "$JAR_STAGED"'now returns 1, at:710.That is the same family as a probe grepping a file that contains its own needle, which cost me a false pass earlier today. Catching it unprompted is the better half of this PR, and worth more than the fix.
The sweep — a real negative, with reasoning
No live instances of the
pipefaildead-check pattern. Of the five scripts underset -ewithpipefail, everyvar="$(cmd | cmd)"already carries|| trueor|| echo; the remaining two have no pipe-into-assignment at all.probe-member-credentials.sh(set -uo pipefail, deliberately no-e) anddeploy/herdr-inner.sh(noset -e) were correctly excluded rather than silently counted.A negative result with the exclusions named is worth recording, because the next person will otherwise re-run the sweep.
What stays open, and where
Closing this ticket does not close these — stated explicitly, because a ticket with several items is otherwise closed by the first one that produces a green run:
wait_for_daemon_exit's call site — unchanged, and still exactly what this ticket said it was: source-text pinned only, "partially pinned, not audited." I did not find a mutation that makes the call present-but-ineffective while keeping the needle, and I did not search exhaustively. Nobody should read this closure as that row being clear.CHECK_ONLY, the drain-gate's own entryifand itsreply != yescondition, the threecase "$SUPERVISOR_KIND"dispatches,HEALTH_BODY, the error-count summary) — untouched. This PR changed the body of one of thoseifblocks while leaving the guard condition itself untested, as instructed. Line numbers shifted in this merge, so they need re-measuring before anyone ticket them.^Exception in threadandNoClassDefFoundError. It was sequenced behind this ticket because they share both files. That file is now free and #512 part 2 is next.One inaccuracy, in the report only
The worker's report abbreviates the restored hash as
0e5a99a2...78f0a. That tail does not occur in the real hash, which endsb97bd6. I hashed the committed file myself and confirmed the restore matched, so the file is correct and only the quoted abbreviation is wrong. Flagged because an abbreviated hash nobody can match against anything is worse than quoting none.