redeploy-fleetd.sh: the 67-test suite covers the functions and barely touches the main flow #555
Closed
opened 2026-09-12 10:07:26 +02:00 by ltms
·
2 comments
No Branch/Tag Specified
main
worker/fleetd-612-unita-87807e-1
worker/612-b3-mcpwirings-da2b58-3
worker/612-b2-cb185-176d3a-2
worker/612-b1-completion-457459-1
worker/612-agaps-73a926-2
worker/608-sleeps-3a64ff-3
worker/621-b4520b-1
worker/618-b83894-2
worker/fleetd-615-e05481-5
worker/lead-autocompact-5f1ab2-3
worker/fleetd-613-f85deb-3
worker/fleetd-608-flaky-nudge-test-d0c2d1-3
worker/lead-context-gauge-ad404f-1
worker/gauge-wiring-9158c1-4
worker/redeploy-slowstart-ead0e5-5
worker/charter-bytes-13668c-6
worker/rollover-outcome-291483-2
worker/589-f64303-2
worker/593-1a8025-5
worker/589-fcd2aa-1
worker/568-9fdaa2-3
worker/571-attempted-outcome-5739f7-2
worker/581-completionresolver-cas-sites-0542b7-6
worker/562-loop-health-wiring-test-99611c-5
worker/562-surface-loop-health-7df5cc-4
worker/575-waiter-cleanup-sites-62ad80-1
worker/572-answer-lock-release-46a9ae-5
worker/567-probe-channel-leak-a38fc5-6
worker/551-record-before-send-7cbf56-1
worker/561-listener-fanout-survives-a-throw-61d538-2
worker/555-redeploy-main-flow-seam-65c2f5-2
worker/556-injector-owns-registration-e027a5-1
worker/552-post-restart-mktemp-abort-bc2672-4
worker/553-onstatus-completion-leak-0da881-2
worker/550-shasum-linux-196132-1
worker/538-loop-dies-on-error-4a5eeb-6
worker/426-health-coverage-ef1fd4-4
worker/504-failed-reported-clean-3cfd66-3
worker/537-capturedlog-close-e4c437-2
worker/459-broken-link-targets-cadc17-5
worker/535-appender-leak-fe74c1-1
worker/512-part2-shutdown-detection-434701-9
worker/529-logger-level-sweep-2a5533-8
worker/528-drain-gate-call-site-5de83d-7
charter/forge-mcp-vs-token
worker/521-swap-guard-unpinned-28e931-5
worker/519-probe-test-harness-d25ab8-4
worker/525-logger-level-leak-1b4eb0-6
worker/518-fleetmcp-resolver-wiring-8ef96c-1
worker/512-drain-complete-line-7edd71-3
worker/517-abort-branch-and-jar-id-41b641-2
worker/500-9e52c9-3
worker/509-4912f4-2
worker/511-9a4b23-1
worker/493-479f45-2
worker/505-03f8b2-1
worker/492-followup-detect-unclear
worker/501-a31fa0-7
worker/498-451d1c-5
worker/494-1015ce-2
worker/492-209647-1
worker/489-001902-2
worker/480-relative-handover-path-906323-1
worker/480-b-handover-skill-45bf1f-5
worker/474-followup-source-pin-f54a55-17
worker/474-charter-check-on-reload-f54a55-17
worker/466-quarantine-repeatcount-report
worker/393-opencode-skill-seeding-71854b-13
worker/469-canonical-tool-names-2a472a-16
worker/466-quarantine-escalation-5ae9c1-15
worker/446-hot-exhausted-pattern-0af580-6
worker/464-charter-tool-name-guard-a85635-12
worker/463-listfleet-default-fails-open-f1c76c-11
worker/458-invariant-5-by-purpose-862f9a-10
worker/439-coordinator-row-gate-bc032a-8
worker/449-herdr-protocol-576015-4
worker/450-abstract-spawn-599e1c-5
worker/437-ack-refuses-177d91-1
worker/444-placement-window-feb56a-2
worker/440-helddurable-derived-d462d7-13
worker/425-rework-placement-resolve-c58ba1-9
worker/421-lead-peek-held-msgs-cdbad2-10
worker/435-fixed-policy-cap-fe11de-12
worker/422-gate-state-observability-9e79d6-11
worker/431-memberregistry-live-readers-cdbad2-10
worker/424-architect-slot-hot-038b41-7
worker/422-model-gate-spawn-c29f48-6
worker/425-default-profile-live-f55534-8
worker/415-coverage-wording-2cbf9c-5
worker/416-3ad1da-1
worker/418-588283-3
worker/deterministic-stamp-race-409-3cb7b6-10
worker/armed-reads-live-config-404-ed931f-9
worker/reply-peer-refusal-391-5a34bd-7
worker/models-allowlist-aa9e9b-3
worker/ttl-stamp-race-399-f1122f-8
worker/scrub-receipt-400-316b3e-5
worker/exhaustion-detection-395-105105-6
worker/scrub-abort-394-316b3e-5
fix/scrub-uid-abort
worker/task-scrub-517574-2
worker/t386-clock-bd5b78-4
worker/t384-scrub-813790-5
worker/t381-cc-748314-2
worker/t373-336973-2
worker/t365-3920c5-3
worker/t358-6e989b-1
worker/t355-8b321c-1
worker/fleetd-369-hermetic-git-tests-e8b19a-3
worker/fleetd-368-stale-lead-binding-f5682e-2
worker/fleetd-360-deploy-units-0d3793-1
worker/359-dead-lead-tabs-f1253b-4
worker/362-worktree-skills-c03e51-3
worker/361-coord-visibility-655144-1
362-plugin-visibility-and-drift
worker/errscan-bed2ca-2
worker/amqp-log-identity-bed2ca-2
worker/withdefaults-guard-561704
worker/sleepguard-82076d-1
worker/fd334-9ee1b6-5
worker/fd348-f1ab27-4
worker/fd335-a71c35-1
worker/fd342-174a17-2
worker/fd345-490d0f-3
worker/fleetd-337-5ec7d4-21
worker/fleetd-341-af5a6b-24
worker/fleetd-339-5ca0a2-23
worker/fleetd-338-83a4a1-22
worker/fleetd-333-281f46-18
worker/fleetd-329-11bdbb-16
worker/fleetd-330-2770fb-17
worker/fix-326-50506e-15
worker/fix-324-3e9bbf-14
worker/fix-323-b8287d-13
worker/fix-316b-bd0860-11
worker/fix-318-76ca36-9
worker/fix-317-486aec-8
worker/fix-315-ce47c5-6
worker/fix-307-275890-6
worker/fix-308-b4f664-7
worker/fix-309-ec3939-8
worker/fix-310-7a3974-9
worker/fix-302-52ad0e-9
worker/fix-298-ce1acb-8
worker/fix-297-66bd11-7
worker/fix-296-104622-6
worker/fix-293-bare-closetab-eb22b5-3
worker/fix-280-gone-ask-lapse-bca98e-2
worker/fix-290-reapidle-guard-coverage-9b0dd1-1
worker/fix-285-trust-seed-8f3565-10
worker/fix-284-backend-error-seat-85912c-11
worker/fix-282-chained-ask-e6d0bb-8
worker/fix-283-teardown-leaks-f40dfa-9
worker/fix-281-pin-handler-actions-4921ac-7
worker/audit-rendezvous-lifecycle-d072ae-2
worker/audit-health-placement-1a2476-6
worker/audit-teardown-exits-e207a5-3
worker/audit-launcher-asymmetry-27e370-4
worker/audit-rest-authz-6ca53c-5
worker/investigate-275-abandon-asking-fdef52-8
worker/fix-274-worktree-leak-b0095d-7
worker/fix-273-exhausted-pattern-9665b5-6
worker/fleetd-267-model-check-bd8068-1
worker/fleetd-131-archunit-18b834-7
worker/fleetd-266-sshagent-rename-a014ff-6
worker/fleetd-184-uid-claim-8e1f31-4
worker/fleetd-184-warn-b381ee-10
worker/fleetd-184-docs-be1d12-9
worker/fleetd-257-9bf010-7
worker/fleetd-103-23a113-6
worker/fleetd-247-342356-5
worker/fleetd-116-04dea8-4
worker/fleetd-252-a830e0-3
worker/fleetd-111-7e8673-9
worker/fleetd-155c-f8ef4b-8
worker/fleetd-176-b928ca-3
worker/fleetd-249-7a7878-2
worker/cb248-composition-root-b-9acdf7-15
worker/cb148-envrc-default-fa6c82-12
worker/cb201-unit5-wiring-6c12e6-8
worker/cb241-fallback-echo-1175e9-11
worker/cb149-trust-dialog-2392a5-9
worker/cb134-148-overlay-visible-c9b986-10
worker/cb234-session-id-keyed-04e1fc-1
worker/cb201-unit3-nudge-abdf5c-6
worker/cb201-unit2-policy-c1102c-5
worker/cb201-unit4-outcome-a13bfa-7
worker/cb201-unit1-classifier-91b9b1-4
worker/cb201-227-refine-831980-3
worker/cb175-model-readback-0f085f-1
worker/cb222-charter-tmpdir-17f013-1
worker/cb226-architect-slot-race-cd3aa8-3
worker/cb224-worktree-root-group-024523-2
worker/cb-123-role-demotion-c600f7-2
worker/cb-219-opencode-roots-1f677e-1
worker/cb214-claude-session-id-b9eab4-4
worker/cb213-zdotdir-wrong-process-dd6de4-3
worker/cb211-exhaustion-classification-9546e0-2
worker/cb137-ambiguous-task-4df3d8-4
worker/cb209-agentsessionid-4dfdb6-2
worker/cb185-hostenvnames-2692b5-3
worker/cb206-opencode-sqlite-128718-2
worker/cb185-worktree-group-fc0c99-1
worker/cb-137-ask-ticket-e7760c-2
worker/cb-172-broker-uri-d36ae4-4
worker/cb-175-model-readback-76ead6-3
worker/cb-161-pane-ancestry-293510-1
worker/cb-164-rebase-885863-8
worker/cb-164-empty-scrape-false-success-1a80af-3
fix/cb-197-ticket-ttl-from-completion
worker/cb-189-remote-url-coverage-4692f3-1
worker/cb-185-blockers-027756-4
worker/cb-192-gap-log-11b631-2
worker/cb-633-fix-5f4396-3
worker/cb185-router-d6436d-3
worker/cb185-router-routing-gaps-9e9d33-3
worker/cb185-paneids-992586-2
worker/cb-633-allow-list-union-ed374b-1
worker/cb-157-credential-in-remote-url-496e44-2
worker/cb-641-health-herdr-evidence-8f1f54-6
worker/cb-640-health-msg-evidence-99c9cd-1
worker/cb-642-fleets-status-skill-bbbc40-5
cb-634-ide-mcp
worker/lead-comms-wiring-c014b9-7
worker/lead-mailbox-c19577-6
worker/autocompact-window-82bc2f-5
worker/cb-634-probe-18056f-4
worker/cb635-broker-urienv
worker/cb-632-config-retry-8e0efa-7
lead/cb-622e-claude-md
lead/cb-622-followup
worker/cb-622a-165dff-1
lead/cb-622d-opencode-mount
worker/cb-622b-717c67-2
worker/cb-622c-ab7759-3
worker/cb-617b2-20ca4b-3
worker/cb-617a-5c2f4a-1
worker/cb596-4e49ef-3
worker/cb586-10500c-1
worker/cb-606-b9343a-25
worker/cb604-1445f8-24
worker/cb582-477374-21
worker/cb584-8c2281-22
worker/cb600-e6b9a9-20
worker/cb602-ce257f-19
worker/cb601-b42837-18
worker/cb598-6c7ba7-17
worker/cb599-740fe4-16
worker/cb597-282224-15
worker/cb590fix-185e9a-10
worker/cb528-recovery-race
worker/cb594-96bead-8
worker/cb590-916766-2
worker/cb527-997d99-3
worker/cb592-env-leak-3cbf9c-1
worker/cb588-async-ticket-nudge-3218f7-5
worker/cb578b-9dcb13-6
worker/cb581-d24826-5
worker/m2-u5-ef8c42-15
worker/cb578a-516499-2
worker/cb576-01a04b-17
worker/cb579-lead-tab-acba06-20
worker/cb580-terminal-health-ed6058-21
worker/cb577-f36fdc-18
worker/cb573b-3db06f-16
worker/cb568c-f36fdc-18
worker/cb568-drop-cause-c3ac1c
worker/cb575-cancelled-notification-c3ac1c
worker/m4-sol-a2cbec-3
worker/cb574-async-ask-c3ac1c
worker/cb573-health-model-8ca857-14
worker/cb572-unknown-target-7f2e35-13
worker/u4-700706-9
worker/u3-b9fcb6-6
worker/u2-ef5b68-4
worker/u1-469dce-1-clean
worker/u1-469dce-1
worker/cb-564-health-events-70cf7e-2
worker/cb-565-recycle-drops-role-98e58f-3
worker/cb-563-missing-reply-df2866-1
worker/cb-562-readiness-gate-silent-6c23c9-3
worker/cb-560-architect-presence-da8155-1
worker/cb-561-architect-silent-off-a71cab-2
worker/cb-548-bind-architect-slot-fe1b8c-1
worker/parity-overlay-settings-5fb711-1
secrets-central-store
cb-559-hot-key-correction
cb-557-fleet-role-pools
worker/cb-553-maxload-explicit-spawn-305ee3-6
worker/cb-551-idle-lead-heartbeat-f1633c-1
worker/cb-544-drain-preserves-worktree-925fad-3
worker/cb-552-docs-sync-1cb9cf-4
worker/cb-548-rendezvous-guard-rebased
worker/cb-548-rendezvous-guard-116b53-10
worker/cb-548-authz-v2-586df6-8
worker/cb-548-authz-264363-5
salvage/cb-528b-codex-home
salvage/cb-528a-codex-launcher
CB-518-primary-flow
feature/peer-launcher-spi
cb-103-injector
v1.1.0
v1.0.0
Labels
Clear labels
blocked
needs-live-proof
ready-to-delegate
silent-default
Cannot start until something else lands. The body says what.
Merged and green, but never shown working on the running daemon. Not the same as done.
Scope, files and acceptance criteria are written. A worker can be briefed from the body alone.
A feature that compiles, passes tests, and ships turned off. Nine recurrences and counting.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: fleet/fleetd#555
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Measured on
mainat93a9ed3, after #548 merged.The shape
67 defined, 67 invoked, no orphans. The coverage is genuinely good — but it is coverage of the helper functions.
jar_id,detect_supervisor,refuse_drain_gate,report_shutdown_drain,swap_if_built,stage_built_jar,wait_for_daemon_exit,scan_uncaught_exceptionsare each tested hard, several with their own mutation checks.Exactly three tests reach into the main flow, and all three do it the same way — by grepping the script's source for a call site:
A call-site grep proves the line exists. It proves nothing about the guard around it. A function can be perfect, its call site present and correctly ordered, and the
ifthat decides whether to reach it can be inverted with every one of the 67 tests still green.The decisions with no test at all
Each is a branch in the main flow (not inside any function), so no existing test can reach it.
if [ "$CHECK_ONLY" = 1 ](:846)--checkperforms a real redeploy. This is the worst one:--checkis documented as read-only and the skill tells the lead to run it first.if [ -n "$OLD_PID" ] && [ "$ASSUME_YES" = 0 ](:883) — the drain-gate entry--yesstarts prompting.refuse_drain_gateitself is tested four ways and its call site is pinned; nothing pins that thisifstill guards it.if [ "$reply" != "yes" ](:892)yesaborts; typing anything else proceeds.case "$SUPERVISOR_KIND"— the report dispatch (:784)check_log_path_matches_plist, whose own comment says every check after it is worthless if it does not run.case "$SUPERVISOR_KIND"— the stop dispatch (:917)launchctl unload/systemctl --user stop, so the supervisor restarts the OLD jar — the exact CB-594 / #492 failure both branches were written to prevent.case "$SUPERVISOR_KIND"— the start dispatch (:992)launchdthe branch also carries the retry that stops the agent being left stopped-and-disabled.HEALTH_BODYpoll loop andif [ -z "$HEALTH_BODY" ](:1049-1068)/healthz 200, or a live one reports failure.HAD_OLD_PID=0; [ -n "$OLD_PID" ] && HAD_OLD_PID=1(:1099)report_shutdown_drain'sn/adecision. All four outcomes are tested; nothing tests that the input is computed right.One suspicion I measured and killed — do not re-derive it
Item 8's idiom looks like a
set -ehazard: whenOLD_PIDis empty the&&list's status is 1, underset -euo pipefail, after the daemon has already been swapped and restarted. That would be the same severity as #552. It is not a bug:[ -n "$OLD_PID" ]is not the command following the final&&, soset -eexempts it. The idiom is safe at both shell versions this script must run under. It still has no test.Why this is worth fixing rather than noting
The suite's own strongest tests are mutation checks (
test_recovery_requirement_mutation_is_caught,test_shared_counter_mutation_is_caught,test_unattributable_quiet_mutation_is_caught). Those exist because the author already knew a passing assertion is not a proof. The same standard has simply never been applied to the main flow, because the main flow is not callable — sourcing the script runs it.That is the actual blocker, and it is the thing to solve first: there is no seam that lets a test execute one main-flow branch. Until there is, every new guard added to the main flow arrives untestable by construction, which is how the list above reached eight.
Suggested approach
Not prescriptive — whoever takes this should measure before committing to one.
REDEPLOY_SOURCED_FOR_TESTguard (orreturnwhen sourced) so the test harness can source it for its functions and call small main-flow steps, the way the function tests already source it today.should_stop_here,health_is_up,drain_gate_required— following theswap_if_built/refuse_drain_gatepattern this script already uses: decision and action together in one function the main flow calls unconditionally, so there is no guard left in the main flow to invert.Acceptance: pick the shape, not the eight sites. A test that greps for main-flow
if/casekeywords outside any function and fails when one appears without a matching predicate test — otherwise a ninth arrives next month. That is the lesson #545 already paid for withmktemp -t: six named lines would have missed the seventh.Related
mktempabort after the restart, in the same file. Both touchscripts/redeploy-fleetd.sh; sequence them.jar_id/shasumon Linux, same file, in flight now.running_pid, the threezsh -lcprobes,curl … || echo 000are the same untested-main-flow family.wait_for_daemon_exit's call site is "partially pinned, not audited", which is item-2's shape one function over.Rework on PR #565 — one demonstrated hole in the new structural guard
The refactor itself verifies clean. One gap in
test_no_untested_main_flow_conditionals, found by a reviewer and then reproduced by me with a control.What I verified first (all clean)
Merged into
origin/main(f4f5f31) →84212b8.bash 5.3.9and/bin/bash 3.2.57(macOS system bash)commboth directions on the sorted definition and call-site name sets is empty, so there is no defined-but-never-called testredeploy-fleetd.shsha4ffacc51…61e8d9matches the sha you reported:1009[ "$code" = "503" ]→!=(anchor 1 → 0) ⇒FAIL: report_health did not report the 503-degraded case:1014die→warnon the never-answered branch (anchor 1 → 0) ⇒FAIL: report_health must die when the body is empty and the code is not 503--checkrun on the pre-refactor and post-refactor scripts produces 20 lines each, identical apart from the commit sha and the worktree path, same exit code. That is the strongest evidence available that the 8 lifted decisions did not change observable behaviour.mktemp: cannot create temp filelines in the suite output are expected — stubmktempbinaries the tests install on PATH.origin/mainemits exactly five too. Not a defect; I checked before assuming.The hole: a conditional wrapped in a function defined below the sourcing boundary
The guard skips anything inside a function body. But a function defined after the
SOURCEDguard (redeploy-fleetd.sh:1032) can never be sourced by the suite, so it is untestable by construction — while still reading to the guard as "inside a function, therefore fine".Reproduced with a control so the two states are distinguishable:
File restored to
4ffacc51…61e8d9after each.The control matters: it proves the guard is not simply inert. It catches the bare form at a position of my choosing and emits a precise message naming the line. The guard is real — it just has one shape it cannot see.
Independent confirmation of the same hole, from the reviewer, including the part I did not measure: the allowlist holds 18 entries against 19 candidate lines the scan actually finds, so it is not vacuous; and today no function definitions exist after the guard line at all, so boundary tracking is currently inert rather than wrong. The hole is latent, not live. It becomes live the first time someone adds a function down there — which is exactly the edit this guard exists to catch.
Acceptance for the rework
SOURCEDguard line. Such a function cannot be sourced, so it cannot be tested, and the existing check treats it as if it were. One added assertion in the same test closes the shape completely — it does not need a second scan.newfunc_below_the_boundary() { if [ "$X" = 1 ]; then :; fi }below the boundary, run the suite, and paste the exact failure text. Then restore and confirm the sha returns to4ffacc5185807d39720a3484d85b922413806eb5347318265bd8897dfd61e8d9.bashand/bin/bash, reporting the exit code next to each result.Nothing else changes. The 8 lifted decisions, their mutation proofs and the allowlist all stand.
One correction to your report's numbers
You reported "236 test_ function definitions … up from 67". Measured on your branch: 118 definitions and 118 invocations;
origin/mainhas 79 of each. The 236 is the two sets added together — the pattern counted definitions and call sites both. The direction was right and it does not affect the work, but the count is worth correcting since it is the kind of number that gets quoted later.Merged as
ba2f4d1(PR #565). Closing.What the rework had to fix
The first pass lifted 8 main-flow decisions into predicate and dispatch functions, and added
test_no_untested_main_flow_conditionalsto stop new bare conditionals appearing. That guard had a hole: it tracks whether a line sits inside a function, and waves through anything that does. So a conditional wrapped in a function defined below theSOURCEDguard was invisible.The hole matters because such a function can never be sourced by the suite — sourcing returns at the
SOURCEDguard, before any code below it runs. So its body is untestable by construction, and the old guard read it as "inside a function, therefore fine". I reproduced this with a control before filing it.The fix emits a
FUNCrecord for every function opened after the guard line, and treats anyFUNCrecord as a violation on its own — independent of what the body contains and of the allowlist.Verification (lead, on a tree with main merged in, and again on real main after the merge)
scripts/redeploy-fleetd.shat its pristine sha4ffacc5185807d39720a3484d85b922413806eb5347318265bd8897dfd61e8d9:^FAIL:bash5.3.9/bin/bash3.2.57 (macOS system bash)PASS: redeploy log classifieris printed unconditionally at the end of the suite, so it is not evidence. The exit code and the^FAIL:count are.Three mutations, each with the subject restored to its pristine sha afterwards:
1. Control — a bare conditional appended to the main flow. Must still be caught.
2. Candidate — the same conditional wrapped in a function below the guard. This is the hole.
3. The fix itself removed — delete the one line that emits the
FUNCrecord, then re-run case 2. It returns to exit 0. So the new assertion is what catches it, not something else that happened to be in the way. Anchor count 1 -> 0;test-redeploy-fleetd.shrestored to0d713a3092a0c0ea8c05663ffa0cb1595e21b9d73872e662eb99b5035fd38151.One thing to know about the new guard's reach
The function-definition pattern only matches
name() {with the brace on the same line.function name {and a brace on the next line are not matched. That direction is safe: when the pattern misses a definition, the depth tracker never enters "inside a function", so conditionals in that body are reported as bare conditionals instead. The miss produces a catch, not a hole. Worth knowing if anyone later widens the pattern and wonders why nothing changed.Correction carried over from the worker's report
An earlier reply from the worker gave "236
test_definitions". That number was wrong — it summed definitions and invocations. The worker flagged it themselves rather than let it stand. The measured figures are 118 test definitions and 118 invocations, withcommproving every defined test is actually invoked.