A regression harness for programs that don't give the same answer twice. It runs every test case several times, measures how far the results move on their own, and only calls a change a regression when it moves them further than that.
Eval tools answer “what score does this get?”. The question that blocks a merge is “did this change make it worse?”, and on a system that samples, two runs of unchanged code already disagree. So noisefloor measures the spread first, then judges the change against it.
Plays the six stages below in order, about 60 seconds. Scroll or press Esc to stop.
All three are real runs. Unchanged compares the committed capture of a real RAG agent, offline. Regressed uses an illustrative target written for this page. First run is the repo's quickstart.
A suite is a YAML file: one target command, and a list of cases, each with an input and the scorers that judge it. The command is a list of arguments, and the target is started without a shell.
{{input}}, {{case_id}} and {{repeat}} are filled in one pass per argument, so an input that contains the literal text {{repeat}} is not filled a second time.The cost is no pipes, globs or variable expansion in the command line. Those are exactly the features that make injection possible.
One hostile input, one argument Illustrative input
Paris"; touch pwned.txt; echo "$(touch pwned2.txt)
The suite's command, one box per argument
pythonecho_argv.py{{input}}Paris"; touch pwned.txt; echo "$(touch pwned2.txt){"argv": ["Paris\"; touch pwned.txt; echo \"$(touch pwned2.txt)"]}
noisefloor run started a script that prints its own arguments, and the stored output and the directory listing are exactly as printed. Neither pwned.txt nor pwned2.txt exists.Each case runs N times, five by default, one after another. Every repeat's stdout, stderr, exit code and duration is stored as JSON under .noisefloor/runs/, so nothing a target printed is thrown away.
noisefloor rescore applies new or fixed scorers to runs already paid for, and noisefloor diff compares stored runs, without calling the target again.--jobs above 1. Timings taken in parallel contend with each other, so they aren't measurements.A real RAG agent, 5 cases, 3 repeats each
runs/20260903T014508.400182Z-rag-knowledge-agent-3dbc2f92/cases/
examples/rag-knowledge-agent/baseline/: rag-knowledge-agent answering with claude-sonnet-4-6, 15 calls in 3 minutes 9 seconds, all ok. Played back 40 times faster.Scorers are deterministic, so judging an answer never needs a model or a key. Each one returns a value and a pass flag for every repeat, and the repeats then collapse into one band per scorer: the noise floor that a change will be judged against.
contains, regex and json_path_equals, and two continuous ones, json_path_number and latency.0.543 [0.437–0.597] n=3.json_path_number takes the lowest value by default, so every citation has to clear the bar, not just the best one.The range over a handful of repeats is not a confidence interval, and the README says so. It is a heuristic, and every report prints the counts behind it so a person can overrule it.
charging-bays, scored three times
QuestionHow many charging bays should we provision for a two-shift operation?
Scorer {type: json_path_number, path: "citations[].score", min: 0.35}
This is the line. It doesn't hold back code; it holds back the claim “this got worse” until the change moves the results further than they moved on their own. The baseline, measured before the change, is always the yardstick.
--min-effect, which is 0 unless you set it.No t-test, on purpose: at five samples a p-value claims a precision the data can't support. A tolerance of 1e-9 keeps two drops of exactly 0.2, which floating point computes as 0.20000000000000007 and 0.19999999999999996, on the same side of the threshold.
One baseline band, three candidates
Binary scorers: all 36 outcomes at five repeats
Rows are the baseline's passes, columns the candidate's. Hover or focus a cell for the reason the report prints.
compare() run on that pair of pass counts. Before issue #7 was fixed, the 4/5 to 5/5 cell read improved; it is the only cell that changed.ClearedAbove this line noisefloor only measures. Below it, the claim “this got worse” leaves the tool: a verdict, the numbers behind it, and an exit code that fails the build.
Each scorer gets its own verdict, and a case takes the worst of them: any regressed scorer makes the case regressed, then any improved one. Cases are listed worst first, so the line that fails the build is the first one you read.
The real capture, judged by two versions of the rule
5 unchangedexit 0
524d5cd, just before the fix. The old rule measured the baseline against the candidate's zero-width band (three repeats at 0.5965) by making the call with the two swapped, which is also why its reason reads backwards. Fixed in ade90b8, and the wording in 206e4a9.The exit code is the verdict CI acts on: 0 when nothing regressed, 1 when a case regressed, 2 when a case broke, and 3 when the harness or its configuration is wrong.
--allow-target-change. Comparing two different programs is occasionally what you want and usually a mistake.check with no baseline adopts its own run and exits 0. That is a pass by absence, so commit a baseline before trusting the gate.Four changes, four exit codes Illustrative input
IndexError: list index out of range. Choose a card to see its run.The README publishes one real capture: rag-knowledge-agent on claude-sonnet-4-6 with no temperature set, 5 cases, 3 repeats, captured twice with nothing changed in between, 30 model calls. Every number here can be re-checked offline from the committed run files.
Fifteen binary scorers held at 3/3 in both captures. The one number that moved was a lowest citation score, because in one repeat the model chose to cite a second document.
Retrieval scored each document the same in every repeat: 0.5965 for the product spec, 0.4374 for the FAQ whenever it appeared.
The first comparison of the two captures said 1 improved, 4 unchanged, on an agent that had not changed. 182 tests had passed; one 15-call capture found the hole.
Tests collected at each commit. The fix was checked against the stored runs, so it cost no extra model calls.
The suite checks structure and substrings, so a reworded answer passes as stable. The stored output shows how often the wording changed: distinct answers across the six stored repeats.
The refusals came back word for word; the answers did not. The README names this as a limit of the suite, not a finding about the agent.
A deterministic fake target reaches every failure mode from flags. Its flakiness is a function of the repeat index, so a three-out-of-five baseline is exactly reproducible without a random seed.
CI runs ruff, pytest and the quickstart from a clean checkout on Python 3.11 and 3.12.
git clone https://github.com/ahmed-hashim-pro/noisefloor.git cd noisefloor python -m venv .venv && . .venv/bin/activate pip install -e . # The first check adopts a baseline. Run it again to compare. noisefloor check examples/quickstart/suite.yaml noisefloor check examples/quickstart/suite.yaml # The real RAG capture, compared offline. No key needed. noisefloor --root examples/rag-knowledge-agent/baseline diff pip install -e ".[dev]" pytest
Two runtime dependencies, both pinned: pydantic and pyyaml. noisefloor itself never needs an API key; only your target might.