Recorded per-test traces, phase 1: results on TestPrune's own suite
Measured 2026-09-26 with FsHotWatch CLI 0.14.0-alpha.69, tests.traces =
{"record": "full-runs", "fingerprintInputs": ["global.json", "Directory.Packages.props"]}, macOS arm64.
The host was also running another repository's release gate at the time, so CPU numbers are noisy.
Runs
Run |
Tree |
Result |
Recorded |
|---|---|---|---|
|
both projects traced |
exit 1: 10 TestPrune.Trace.Tests failures (see below) |
|
|
TestPrune.Trace.Tests |
exit 0 |
|
|
unchanged |
exit 0: run-2 verdict reused, no tests ran |
nothing new |
Runs 1 and 2 recorded identical TestPrune.Tests traces (same counters, 848 complete).
trace_runs.status is recorded with no reason for every run: preparation refused nothing.
TestPrune.Trace.Tests is not traced
Traced, the suite's outcome changes: 10 tests that pass untraced fail. The shadow bin replaces the app's
own TestPrune.Trace.Recorder build with the weaver's bundled one, so the suite that tests the recorder
runs against a different, live recorder:
Failure |
Cause |
|---|---|
|
the process recorder is live |
|
the shadow bin ships the recorder without its PDB |
|
the bundled recorder predates the child-scope fix |
5 |
the bundled shim gives the woven fixture's process the parent's scope, so the fixture run has no main dump |
The run also overflowed (10,370 overflow hits, 259 of 280 tests recorder-overflow), because the suite's
own tests swap the process recorder for small test recorders. The project is marked "traces": false.
Phase 1 never changes a verdict, so the opt-out is required rather than a choice.
Bars (TestPrune.Tests, run 2bdf1176…)
Bar |
Pass when |
Measured |
Verdict |
|---|---|---|---|
census |
traced ≥ 0.99; ambient < 0.1 % or listed |
traced 1024/1024 (1.0000); ambient 0/14,771,003 |
pass |
audit |
|
sampled 11, extra 15 (one test, see below) |
fail: tool defect |
overhead |
CPU ratio ≤ 1.15 |
1.101 (untraced median 83.0 s, traced 91.4 s); traced max RSS 1,910.8 MiB |
pass |
file-census |
ratio ≤ 0.05 |
0.048 (reads-repo 21, fail-outside 20, fail-in-repo 0) |
pass |
coverage parity |
identical cobertura with traces removed |
not measured: |
open |
PDB / JIT |
|
no refused run |
pass |
Incomplete traces (99 of 1024 tests; 848 of 947 stored traces complete)
Reason |
Tests |
Cause |
|---|---|---|
|
72 |
7 closure methods in |
|
13 |
|
|
14 |
tests that start non-woven processes; expected by design |
Audit
The one test with extras is the theory row
rich types and patterns without anon records produce no diagnostics(body: "let f (p: int * string) = p").
Its 15 "extra" ids are everything it executes, its own method included, and the audit reports it as
"not recorded when run alone". The isolated --filter-display-name run did not select the row, so the
comparison was against nothing. It is an audit defect, not cross-test leakage. The missing entries are
static constructors, closures and union cases that an earlier test already ran in the parallel run.
The 8 missing user entries in SymbolOccurrenceStoreTests are AstAnalyzer record constructors and
union-case getters. They run once per process, so in the parallel run an earlier test ran them first.
Defects found
Owner |
Defect |
|---|---|
TestPrune.Trace (released recorder) |
the |
TestPrune.Trace |
the shadow bin silently substitutes its bundled recorder for an app's own build of the same version |
TestPrune.Trace |
joiner: closures in |
TestPrune.Core / joiner |
F# |
TestPrune.Trace |
audit: a sampled test the isolated run does not select is counted as extras |
TestPrune.Trace |
census reports a project's last recorded run after the project opted out |
Second run (2026-09-27)
Measured with FsHotWatch CLI 0.14.0-alpha.71, which bundles TestPrune.Core 13.2.2 (index schema 18: F#
exception types are indexed) and TestPrune.Trace 0.2.0. The "traces": false opt-out on
TestPrune.Trace.Tests is removed. Same tests.traces, macOS arm64. The host was loaded again (load average
17–39), so CPU numbers are noisy.
Runs
Run |
Result |
Recorded |
|---|---|---|
|
exit 0, green full-suite run: TestPrune.Tests 1033 passed + 1 skipped, TestPrune.Trace.Tests 303 passed |
|
trace_runs for 8af7b8f8…:
Project |
Status |
Reason |
|---|---|---|
TestPrune.Tests |
|
none |
TestPrune.Trace.Tests |
|
the app ships its own TestPrune.Trace.Recorder build; tracing would replace it (its deps.json lists the recorder as a project) |
The shadow bin now refuses TestPrune.Trace.Tests by itself. The suite runs untraced and stays green, with no config opt-out. The measurement verbs refuse it with the same reason, so the bars below cover TestPrune.Tests only.
Bars (TestPrune.Tests)
Bar |
Pass when |
First run (alpha.69, |
Second run (alpha.71, |
Verdict |
|---|---|---|---|---|
census: traced |
≥ 0.99 |
1024/1024 (1.0000) |
1030/1030 (1.0000) |
pass |
census: ambient |
< 0.001, or listed | 0/14,771,003 |
0/14,851,732 |
pass |
audit |
|
sampled 11, extra 15 (display-filter defect) |
sampled 11, extra 0; missing only |
pass |
overhead |
CPU ratio ≤ 1.15 |
1.101 (83.0 s → 91.4 s); max RSS 1,910.8 MiB |
1.068 (67.4 s → 72.0 s); max RSS 1,781.3 MiB; see below |
pass |
file-census |
ratio ≤ 0.05 |
0.048 (reads-repo 21, fail-outside 20, fail-in-repo 0) |
0.048 (reads-repo 21, fail-outside 20, fail-in-repo 0) |
pass |
coverage parity |
identical cobertura with traces removed |
not measured |
lines identical, branches not; see "Coverage parity" below |
fail |
PDB / JIT |
|
no refused run |
0 invalid of 5,774 verified methods; the one refusal is the intended own-recorder refusal |
pass |
The theory row that produced the first run's 15 false extras
(rich types and patterns without anon records produce no diagnostics(body: "let f (p: int * string) = p"))
was sampled again and is now selected alone: extra 0, missing none. No missing user entries remain.
Incomplete traces
Reason |
First run (tests) |
Second run (tests) |
|---|---|---|
|
72 |
0 |
|
13 |
0 |
|
14 |
14 (dotnet 6, sleep 5, sh 2, chmod 1) |
total incomplete / stored traces |
99 / 947 (848 complete) |
14 / 953 (939 complete) |
unmappedIds is 0. The only incomplete traces left are tests that start non-woven processes, which is
expected by design.
Overhead: an unexplained traced failure
The first overhead --reps 3 exited 1. The ratio was 1.049, but traced samples 2 and 3 exited 2 (untraced
runs all exited 0). The verb discards the launched app's output, so the failing tests are unknown. The
failure did not come back in 8 more traced launches: 2 back-to-back runs sharing one dump directory, 3 runs
interleaved with untraced ones as the verb does, and a second overhead run (all 6 samples exit 0; its
numbers are in the table). The fshw daemon was idle during the failing run, so no rebuild overlapped it.
Host load average was 25–39 at the time.
Owner |
Defect |
|---|---|
TestPrune.Trace |
|
What still blocks the phase-1 bar
Item |
State |
|---|---|
coverage parity |
measured below: line coverage is identical, branch coverage is not |
3 consecutive traced full runs (D6 census bar) |
this run recorded one |
intermittent traced exit 2 under load |
2 of 12 traced launches, not reproduced; open until a failing launch's output is kept |
Follow-up (2026-09-27): overhead reruns and coverage parity
overhead and audit now print a nonzero-exit launch's output (TestPrune.Trace Unreleased), which fixes
the defect above.
Overhead reruns with output capture
Run |
Load average |
Ratio |
Untraced median |
Traced median |
Peak RSS |
Exits |
|---|---|---|---|---|---|---|
1 |
24–28 |
1.022 |
66.2 s |
67.7 s |
1,831.7 MiB |
all 0 |
2 |
24–31 |
1.027 |
69.1 s |
71.0 s |
1,802.5 MiB |
all 0 |
3 |
22–27 |
1.043 |
65.6 s |
68.4 s |
1,840.9 MiB |
all 0 |
4 |
19–25 |
1.030 |
67.0 s |
69.0 s |
1,777.0 MiB |
all 0 |
None of the 12 traced launches exited nonzero, so the earlier exit 2 did not recur. Over the day, 2 of 23
traced measurement launches exited nonzero, both in the same overhead run. It stays open: the next failure will print its tests.
Coverage parity
The second run's "confirm collects no coverage" was wrong. fshw collects coverage by default
(tests.projects[].coverage defaults to true). Each confirm writes
coverage/<project>/coverage.baseline.cobertura.xml per project and a merged, line-only
coverage/coverage.cobertura.xml. Two confirm runs on the same tree, traced (run ef14ad88…, TestPrune.Tests
recorded, TestPrune.Trace.Tests refused) and with tests.traces removed, were compared per file on lines
covered/valid and branches covered/valid:
Report |
Files |
Line counts |
Branch counts |
Recorder module |
|---|---|---|---|---|
TestPrune.Tests (traced) |
31 = 31 |
identical in every file (5,504 / 5,903) |
differ in 8 files: traced 1,196 / 1,548, untraced 1,283 / 1,646 |
none |
TestPrune.Trace.Tests (refused, so untraced both times) |
32 = 32 |
identical |
identical (696 / 712) |
none |
merged |
63 = 63 |
identical (8,594 / 8,994) |
none reported |
none |
The traced report adds no branch and changes no line hit. It drops every branch point on 31 lines, 98 branch
points in all: AstAnalyzer.fs (38), Orchestration.fs (28), SymbolDiff.fs (16), Database.fs (8),
AuditSink.fs, EdgeEmission.fs, ImpactAnalysis.fs, Program.fs (2 each). Every such line is a union
match (match change with, match r.Outcome with) or a union-case test (if sym.IsExtern then). Those
are the sites where the site-probe pass instruments the union's tag read before the switch. The loss is
deterministic: two traced runs wrote byte-identical reports.
Owner |
Defect |
|---|---|
TestPrune.Trace (weaver, site probes) |
coverage of a woven assembly loses the branch points of union matches and union-case tests: 98 of 1,646 in TestPrune.Tests. Line coverage is unaffected. A consumer whose coverage ratchet reads per-project branch rates from a traced run would see them drop. Cause not yet confirmed; the tag-read probe before the |
Coverage parity with the weaver fix (2026-09-27)
confirm collects per-project cobertura by default. A traced and an untraced confirm on alpha.71 matched on
lines but not on branches: the traced TestPrune.Tests report lost 98 of 1,646 branch points (1,196 / 1,548
against 1,283 / 1,646), all on lines with a union match, a field comparison or a type test.
Cause
MS CodeCoverage counts a conditional branch under a hidden sequence point only while no call lies between it and its line's visible point. F# puts a match's or comparison's test under a hidden point, and a site probe is a call inserted right there, so the line keeps its hits and loses its branch points. Observed on the IL:
Unwoven shape |
Branch counted |
|---|---|
|
yes |
|
no, even untraced |
the same |
no |
a probe, then a hidden copy of the enclosing point where the probe resumes |
no |
a probe, then a visible copy of the line's point where the probe resumes |
yes |
Fix (TestPrune.Trace, Unreleased)
Where a probe precedes a conditional branch in a hidden chain that falls through from its line (before any
call, transfer or jump target), the site-probe pass gives the instruction the probe resumes at a copy of the
line's visible sequence point. Probes and ids are unchanged; only the woven PDB gains points.
CoverageParityTests weaves the FxTests fixture and requires every FxLib line's hits and condition-coverage
to match the untraced run.
Measurement
The pinned fshw (alpha.71) bundles the released weaver, so confirm cannot measure an unreleased weaver
fix. TestPrune.Tests was therefore woven with this tree's weaver and run twice with fshw's default coverage
arguments (--coverage --coverage-output-format cobertura): the shadow apphost traced, and bin/Debug
untraced. Both runs: 1,037 tests, 0 failed.
Report (repository files) |
Lines covered / valid |
Branches covered / valid |
Files differing |
|---|---|---|---|
untraced |
5,552 / 5,949 |
1,283 / 1,644 |
— |
traced, released weaver ( |
identical |
1,196 / 1,548 (98 points lost) |
8 |
traced, fixed weaver |
identical, hits identical |
1,279 / 1,640 (4 points lost) |
1 |
traced, weaver closing the copy's range (below) |
identical, hits identical |
1,283 / 1,644 (0 points lost) |
0 |
The 4 remaining points were Orchestration.fs lines 679 and 741 (match selection with): the list test of
| RunSubset [] -> sits under a later hidden point, after calls, and counts unwoven. Rewriting only the PDB of
a woven fixture of that shape (FxLib.Logic.selected) showed MS CodeCoverage drops such a branch when the
range of the visible point heading its chain holds a call. The copy covered the probes that follow the case
test, not just the case test. A hidden point right after the copy's last branch keeps its range call-free and
restores the list test's points; hidden points later in the chain do not. The site-probe pass now adds that
point, and CoverageParityTests covers the shape. Measured as above, with the weave cache cleared: the cache
key hashes the woven inputs, not the weaver, so two local weaver builds of one version share a weave. The traced
report has no TestPrune.Trace.Recorder package when the recorder's PDB is absent (the released recorder).
A recorder built in this tree has its PDB at its build path, and MS CodeCoverage then reports it.
Bar |
Pass when |
Measured |
Verdict |
|---|---|---|---|
coverage parity |
identical per-file lines and branches; no recorder module |
lines identical; branches 1,640 / 1,644 (2 lines differ); no recorder module with the released recorder |
fail (4 points), until a |
Third run (alpha.74, 2026-09-27)
FsHotWatch CLI 0.14.0-alpha.74, which bundles TestPrune.Trace 0.3.0: the site-probe branch-point fix and the
output capture of a failing measurement launch. Same tests.traces, macOS arm64, fresh workspace off main.
Several other agents shared the host, so each launch is listed with its load average.
Runs
Run |
Load (start → end) |
Result |
TestPrune.Tests |
TestPrune.Trace.Tests |
|---|---|---|---|---|
|
31.2 → 40.6 |
exit 0 |
1036 passed + 1 skipped; recorded |
307 passed; refused (own recorder) |
|
23.6 → 23.2 |
exit 0 |
1036 passed + 1 skipped; recorded |
307 passed; refused (own recorder) |
|
21.6 → 37.3 |
exit 0 |
1036 passed + 1 skipped; recorded |
307 passed; refused (own recorder) |
|
43.7 → 29.7 |
exit 0 |
1036 passed + 1 skipped |
307 passed |
.fshw.json was restored byte-identical after the untraced run.
Recorded stats, TestPrune.Tests |
Run 1 |
Run 2 |
Run 3 |
|---|---|---|---|
executed / traced / complete |
1033 / 1033 / 942 |
1033 / 1033 / 942 |
1033 / 1033 / 942 |
test-scope hits |
19,389,838 |
19,389,838 |
19,389,838 |
static-init / ambient / overflow hits |
410 / 0 / 0 |
410 / 0 / 0 |
410 / 0 / 0 |
unmapped ids, rejected dumps |
0, none |
0, none |
0, none |
recorder CPU |
67.7 s |
61.9 s |
58.1 s |
Bars (TestPrune.Tests; TestPrune.Trace.Tests is refused by every verb with the own-recorder reason, exit 2)
Bar |
Pass when |
Measured |
Load |
Verdict |
|---|---|---|---|---|
census: traced |
≥ 0.99 on 3 consecutive full runs |
1033/1033 (1.0000) on runs 1, 2 and 3 |
see Runs |
pass |
census: ambient |
< 0.001, or listed | 0 / 19,390,250 on each run |
see Runs |
pass |
audit |
|
sampled 11, extra 0; missing only |
29.1 → 29.0 |
pass |
overhead |
CPU ratio ≤ 1.15 |
1.066 (untraced median 75.8 s, traced 80.8 s); peak RSS 1,872.9 MiB; all 6 samples exit 0 |
19.7 → 21.2 |
pass |
file-census |
ratio ≤ 0.05 |
0.048 (reads-repo 21, fail-outside 20, fail-in-repo 0) |
29.0 → 19.7 |
pass |
coverage parity |
identical per-file lines and branches; no recorder module |
lines and every line's hit count identical in all 63 files; branches differ in 1 file (4 points, below); no recorder module in the traced report |
43.7 → 29.7 |
fail (4 points, known) |
PDB / JIT |
|
|
— |
pass |
Coverage parity (run 3 against the untraced run)
Report |
Files |
Lines covered / valid |
Branches covered / valid |
Differing files |
|---|---|---|---|---|
TestPrune.Tests traced |
31 = 31 |
5,552 / 5,949 = |
1,279 / 1,640 against 1,283 / 1,644 |
1 |
TestPrune.Trace.Tests (refused, untraced both times) |
32 = 32 |
3,160 / 3,161 = |
718 / 734 = |
0 |
merged |
63 = 63 |
8,712 / 9,110 = |
none reported |
0 |
File |
Line |
Traced |
Untraced |
Explanation |
|---|---|---|---|---|
|
679 |
hits 1, branches 2/2 |
hits 1, branches 4/4 |
the recorded gap after the weaver fix ( |
|
741 |
hits 1, branches 2/2 |
hits 1, branches 4/4 |
same |
The alpha.71 traced report lost 98 branch points in 8 files. alpha.74 loses the 4 above and nothing else. The three
traced runs wrote identical per-file reports. TestPrune.Trace.Tests reports a TestPrune.Trace.Recorder package
traced and untraced alike: that suite tests the recorder, and it is refused, so the package is its own code under
test, not a woven-in module.
Other observations
Observation |
Detail |
|---|---|
no traced launch exited nonzero |
3 traced |
|
|
census of an older run loses its incomplete reasons |
|
audit's parallel run has 17 |
audit: dotnet 9, sleep 5, sh 2, chmod 1; |
Fourth run (alpha.78, 2026-09-29)
FsHotWatch CLI 0.14.0-alpha.78, which bundles TestPrune.Core 13.3.0 and TestPrune.Trace 0.4.1. That includes
0.3.1's fix for branch points under a match's later hidden sequence points and 0.4.0's static-init inheritance.
Same tests.traces, macOS arm64 on AC power, a fresh workspace off main with only the pin bump. Each traced
full run is dotnet fshw confirm --fresh. The measurement verbs ran from this tree's src/TestPrune.Trace.Cli,
whose TestPrune.Trace is the released 0.4.1. CPU busy is the mean over two 1 s top samples just before each
launch.
Runs
Run |
Start (UTC) |
Load at start (1/5/15 min) |
CPU busy |
Result |
TestPrune.Tests |
TestPrune.Trace.Tests |
|---|---|---|---|---|---|---|
|
12:24:14 |
9.7 / 13.0 / 38.3 |
93 % |
exit 0 |
1041: 0 failed, 1 skipped; recorded |
328: 0 failed; refused (own recorder) |
|
12:30:27 |
29.0 / 38.1 / 43.5 |
72 % |
exit 0 |
1041: 0 failed, 1 skipped; recorded |
328: 0 failed; refused (own recorder) |
|
12:48:51 |
9.7 / 21.4 / 30.7 |
35 % |
exit 0 |
1041: 0 failed, 1 skipped; recorded |
328: 0 failed; refused (own recorder) |
untraced full run ( |
12:49:48 |
14.5 / 21.2 / 30.1 |
21 % |
exit 0 |
1041: 0 failed, 1 skipped |
328: 0 failed |
All three traced runs have the same input tree (sha256:8e2e8784…). .fshw.json was restored byte-identical
after the untraced run.
Recorded stats, TestPrune.Tests |
Run 1 |
Run 2 |
Run 3 |
|---|---|---|---|
executed / traced / complete |
1037 / 1037 / 946 |
1037 / 1037 / 946 |
1037 / 1037 / 946 |
test-scope hits |
19,785,966 |
19,785,966 |
19,785,966 |
static-init / ambient / overflow hits |
770 / 0 / 0 |
770 / 0 / 0 |
770 / 0 / 0 |
unmapped ids, rejected dumps |
0, none |
0, none |
0, none |
recorder CPU |
70.2 s |
67.7 s |
70.1 s |
Against alpha.74 there are 946 complete traces instead of 942, and 770 static-init hits instead of 410. Both changes come from 0.4.0, which records each type initializer in its own scope and links it to the tests that use the type.
Bars (TestPrune.Tests; every verb refuses TestPrune.Trace.Tests with the own-recorder reason, exit 2)
Bar |
Pass when |
Measured |
Start (UTC), load, CPU busy |
Verdict |
|---|---|---|---|---|
census: traced |
≥ 0.99 on 3 consecutive full runs |
1037/1037 (1.0000) on runs 1, 2 and 3 |
see Runs |
pass |
census: ambient |
< 0.001, or listed | 0 / 19,786,736 on each run |
see Runs |
pass |
audit |
|
sampled 11, extra 0; missing only |
12:50:48, 17.2, 17 % |
pass |
overhead |
CPU ratio ≤ 1.15 |
1.067 (untraced median 78.1 s, range 78.0–78.2; traced 83.4 s, range 81.8–84.5); peak RSS 1,856.5 MiB; all 6 samples exit 0 |
12:51:40, 9.9, 22 % |
pass |
file-census |
ratio ≤ 0.05 |
0.000 (fail-outside 20, reach-outside 1, reads-repo 21, fail-in-repo 0) |
12:54:42, 6.2, 30 % |
pass |
coverage parity |
identical per-file lines and branches; no recorder module |
identical in every file, line hit counts included (below) |
see Runs |
pass |
PDB / JIT |
|
|
— |
pass |
The file-census ratio fell from 0.048 to 0. The one test that used to account for the difference reads a
repository file at an absolute path. 0.4.0's file-census now counts that read, made from outside the
repository, as a dependency on the repository (reach-outside 1).
Coverage parity (traced run 3 against the untraced run)
Report |
Files |
Lines covered / valid |
Branches covered / valid |
Differing files |
|---|---|---|---|---|
TestPrune.Tests traced |
31 = 31 |
5,561 / 5,957 = |
2,160 / 2,690 = |
0 |
TestPrune.Trace.Tests (refused, untraced both times) |
33 = 33 |
3,250 / 3,251 = |
1,051 / 1,067 = |
0 |
merged |
64 = 64 |
8,811 / 9,208 = |
none reported |
0 |
Every line's hit count and condition coverage is identical as well. The traced TestPrune.Tests report has no
TestPrune.Trace.Recorder package. TestPrune.Trace.Tests reports one traced and untraced alike, because that
suite's code under test is the recorder. The totals are not comparable with earlier rounds (the tree
and the pinned SDK band both changed since then); parity is measured within this round.
Observations
Observation |
Detail |
|---|---|
audit and census count |
explained: the audit's parallel run counts 9 |
the untraced full run was launched by the daemon, not by |
removing |
census of an older run loses its incomplete reasons |
unchanged from the third run: re-recording a test replaces its reasons |
no traced launch exited nonzero |
3 traced |
Phase-1 bar on TestPrune's own suite
Every D6 bar holds for TestPrune.Tests on a released toolchain. TestPrune.Trace.Tests is intentionally not traced (ADR 0008). No unexplained item remains, so no defect was filed.
TestPrune