From 13482872bbd4c6900c5ff58fb896fff4bf23f75c Mon Sep 17 00:00:00 2001 From: Dai Ha Date: Thu, 10 Sep 2026 18:37:41 +0700 Subject: [PATCH] contract test: keep the loaded run's passes, discard only its failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The javadoc threw away the whole loaded run as "not clean evidence". The fixed cell order makes that too strong in one direction. A cell that runs last on a climbing load has a free explanation for FAILING. It has no free explanation for PASSING: surviving a worse condition than a fair order would have given it is evidence in the safe direction. So the old version's 0 of 3 is still discarded, and the three passes are kept with the load each one ran at. Also names the cell that tests the swallow explanation head-on, which the old text left as "nobody has managed that yet". SHELL_READY_TIMEOUT_MS at 0 types input at once — the worst case for "typed before the prompt" — and it passed 3 of 3 at load 18.42 to 23.65. The wider read window cannot explain that away, because a swallowed keystroke is lost, not late: the command never runs, so no amount of polling makes its output appear. And a warning not to carry the raw load average to another host. Load average counts differently per core and per operating system, so only load per core compares. I broke that rule myself when comparing this run with another host's numbers. Both points came from the fleet01 lead reviewing 2af13ab. --- .../fleet/herdr/AgentControlContractTest.java | 28 +++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/fleetd/src/test/java/dev/ltms/fleet/herdr/AgentControlContractTest.java b/fleetd/src/test/java/dev/ltms/fleet/herdr/AgentControlContractTest.java index 8ec0c82..e11c2c0 100644 --- a/fleetd/src/test/java/dev/ltms/fleet/herdr/AgentControlContractTest.java +++ b/fleetd/src/test/java/dev/ltms/fleet/herdr/AgentControlContractTest.java @@ -36,16 +36,28 @@ import static org.junit.jupiter.api.Assumptions.assumeTrue; * typing at 1000ms, and it is not. * *

Where this was measured. A 12-core macOS host, load average 2.6 to 5.9, - * on commit 20c1094. The same four cells were also run under load and gave the same answer, but - * that run is not clean evidence: the load average climbed from 7 to 50 while the cells ran, and - * the old version ran last, at the top of that climb. Above about load 20 everything here is - * slow for reasons that have nothing to do with this seam. So read the claim as "measured near - * idle on a 12-core host", and nothing stronger. If this test fails on a smaller or busier - * machine, raise {@link #OUTPUT_TIMEOUT_MS} before you suspect the seam. + * on commit 20c1094. The same four cells were also run under load, with 8 spinners on 12 cores. + * The failure and the passes from that run are not worth the same. The cells ran in a fixed + * order while the load climbed from 7 to 50. The old version ran last, at the top of that climb, + * so it has a free explanation for failing and its 0 of 3 is discarded. A pass has no such free + * explanation: a cell that survives a worse condition than a fair order would have given it is + * evidence in the safe direction. So keep the three passes, each with the load it ran at: the + * fixed version 3 of 3 at load 7.42 to 18.42, the 0ms-write cell 3 of 3 at 18.42 to 23.65, the + * 1000ms-write cell 3 of 3 at 23.65 to 46.40. Above about load 20 everything here is slow for + * reasons that have nothing to do with this seam, so read the positive claim — that the read + * deadline was the whole cause — as "measured near idle on a 12-core host", and nothing + * stronger. Do not carry that raw load average to another host either: load average counts + * differently per core and per operating system, so only load per core compares. If this test + * fails on a smaller or busier machine, raise {@link #OUTPUT_TIMEOUT_MS} before you suspect the + * seam. * *

{@link #waitUntilSettled} is kept as cheap insurance against that swallow case, not because - * anyone showed it was needed. If you want to delete it, the honest test is whether you can make - * this test fail by typing early. Nobody has managed that yet. + * anyone showed it was needed. The cell that tests the swallow case head-on is the one with + * {@link #SHELL_READY_TIMEOUT_MS} at 0: input is typed at once, which is the worst case for + * "typed before the prompt is ready". It passed 3 of 3 at load 18.42 to 23.65. The wider read + * window cannot explain that pass away, because a swallowed keystroke is LOST, not late — the + * command never runs, so no amount of polling makes its output appear. So the swallow mechanism + * was tested and did not show up. If you want to delete this call, that is the cell to re-run. * *

Tagged {@code contract}; run with {@code mvn test -Pcontract}. */