flaky-hunt

Finds flaky pytest tests and says why each one flakes. It re-runs the suite while changing one thing at a time, names the factor that makes each test fail, measures how often, and gives a plain pytest command that reproduces the failure.

A flaky test is usually reported as "fails sometimes", which is the least useful thing to know about it. Rerunning until green, or marking it @flaky, hides it and the bug ships. What a developer needs is the cause: it fails after one particular test, in the last minutes of the day, or when the hash seed is 0. Each of those is a short fix once it is known.

View on GitHub How reliable the diagnosis is

Plays the six stages below in order, about 60 seconds. Scroll or press Esc to stop.

Language: Python 3.10 or newer; a pytest plugin and a CLI
Tests: 47 passing with pytest: 42 fast, 5 that run whole hunts
Dependencies: pytest and time-machine
Factors: 7: order, time, time zone, random, timing, environment, network
Statistics: One-sided Fisher exact test, Wilson 95% intervals
License: MIT

pytest -q

Real runs of flaky-hunt 0.1.0 on examples/planted, the repository's small shop with nine planted flaky tests, output verbatim; the lines starting with a dot are its progress on stderr. This machine's zone is Africa/Cairo, so test_receipt_epoch, which is right only in UTC, failed in every plain run here. The last tab is the repository's own recorded run in docs/real.

1Run the whole suite 39 times, one change at a time

Three runs as-is, then one under each setting: four test orders, four clock times, five time zones, ten random and hash seeds, four timing perturbations tried twice each, four environment changes, and the network switched off.

Every run is a fresh pytest process with its own empty TMPDIR, so nothing leaks from one run to the next unless that is the thing being tested.
The perturbations live in a pytest plugin, each an ordinary pytest option such as --fh-seed=5 or --fh-switch-interval=1e-6. Installed but unused, the plugin changes nothing.
With order among the factors and up to 200 tests, every test is also run on its own, to find tests that need an earlier one.

Every probe run of the planted suite, test by test

PassedFailedNot in that run

One hunt of the planted suite on this machine, recorded run by run: 16 tests down the side, the 39 suite runs across, grouped by factor. Bold names are the nine planted flaky tests. The empty column is "run from another directory": with no test paths given, that run collected no tests at all a problem in flaky-hunt found while making this page.

2Suspect a factor where one value fails and another passes

Settings that are alternative values of one thing share a knob: seed 5 and seed 1, TZ=Pacific/Kiritimati and TZ=UTC. A test that failed under one value and passed under another, or under a setting and as-is, is a suspect for that knob.

A test that failed in every run, perturbed or not, isn't flaky. It is listed apart as failing every time.
A test that fails at random is a suspect under nearly every knob. That is expected; stage 3 is what clears the innocent ones.

Suspects per test, and what became of them

Confirmed in stage 3Not confirmed

The same hunt: each square is one cause tried for that test in stage 3, against another value of the same knob or as-is. test_render_tags fails at random about half the time as-is, so it collected ten suspects. Nine reached stage 3 and only PYTHONHASHSEED survived; the tenth, an order suspect, was dropped when the bisection found no single test that breaks it. Hover a square for the pair compared.

3Confirm it in fresh processes, both sides at once

Each suspect is run on its own, 8 times under the failing setting and 8 under the passing one, alternating between the two so a change in the machine's load hits both sides alike. The knob is reported only if the failing side fails significantly more often, by a one-sided Fisher exact test.

Inconclusive but leaning (p under 0.25): both sides go to 32 runs. That is for a race that also fails now and then as-is, where 8 of 8 against 3 of 8 proves nothing and 32 of 32 against 10 of 32 does.
The 1% significance level is split over both stages and over every cause tried for that test (Bonferroni), so for each test the chance of blaming a cause that isn't there stays under 1%.
Rates carry Wilson 95% intervals: 8 of 8 means at least 68%, not always.

Every cause tried for test_render_tags

Each dot is one fresh pytest process running only that test, in the order the hunt started them; red failed. The significance needed is 1% split over two stages and the ten causes tried for this test, 0.0005. The time-zone comparison leaned far enough at 8 a side to earn 32, and still came out no.

4Bisect an order failure down to one test

A test that fails in some order but passes on its own is bisected over the tests that ran before it: keep whichever half still breaks it, then try the last few one by one. The pair is then confirmed the same way as any other cause.

The other direction too: a test that fails on its own but passes in the suite is bisected to the test it silently depends on.
The search is tested on its own against an oracle with the culprit at every position in lists of 1 to 333 tests.

Who breaks test_checkout_hides_debug_details?

Each bar is the 14 tests that ran before the victim in the shuffled order where it failed; the filled ones ran ahead of it in that process. Eight runs, the last a single test: test_enable_debug_mode, which sets settings.DEBUG and never resets it. Recorded from the same hunt; the report lines are from a plain flaky-hunt run.

ReportedAbove this line flaky-hunt probes and proves. It never reruns a test to green and never marks one flaky; it exits 1 and hands each cause, with a command, to a person.

5A person gets a command that fails

Each diagnosis ends with a reproducing command in plain pytest: an environment variable, a plugin option, or two test ids in order. A developer runs it, sees the failure on demand, and fixes the cause.

Every plugin option is an ordinary pytest option, so the command works anywhere flaky-hunt is installed.
Tests that fail for one reason are grouped, the way one polluter breaking 24 tests is one problem. --json lists every test one by one.
Exit 1 when it finds a flaky test, 0 when it finds none, 2 when pytest itself can't run. It takes dozens of suite runs, so it belongs in a nightly job.

The reproducing commands, run by hand

The commands are the ones the hunt printed, run by hand in the planted project, with and without the setting. Only pytest's assertion lines and summary are kept; exit codes are pytest's. The race passed three times out of three on its own and failed three out of three with frequent thread switches.

6Check the checker

examples/planted is a small shop with nine tests that are flaky in nine known ways and seven that aren't. expected.json says which factor each flaky one must be blamed on. The self-tests run whole hunts and require exactly that: every planted cause found, nothing blamed on anything else, no stable test reported.

Six deliberately broken versions of flaky-hunt, switched on by an environment variable, must each fail the self-tests.
CI runs ten more whole hunts on every push and scores each against expected.json.

Nine planted causes, six mutants

The nine rows are what the recorded hunt blamed; each matches expected.json. The mutant table is python scripts/mutants.py, run on this machine: each mutant runs only the self-tests aimed at what it breaks, and is caught when they fail.

Measured, not assumed

Five releases of four well-known projects, each hunted with every factor in a clean python:3.12-slim container by make real. Plain pytest passes all five.

The repository's own runs, in docs/real, on a 12-core machine with 3 workers. click and both humanize releases were hunted again for this page with the same script: click 0 flaky in 25 s; humanize 4.10.0 the same 25 tests with the same causes and rates, line for line, in 4m 50s; humanize 4.11.0 clean in 1m 50s. Neither finding is a new bug report: humanize fixed its pair upstream, and the arrow pair was found with pytest-randomly in 2021 and is still open. No time, zone, seed, timing, environment or network dependence turned up in any of them, which is not proof that none exists.

The planted suite, on this machine

One flaky-hunt run with every factor, four workers, on a 12-core Linux box that was also running other jobs:

39 suite runs392 single-test runs1m 40s

All nine planted causes found, nothing else blamed, exit 1. Most of the single-test runs are stage 3, which had 18 suspects to settle. --factors order alone took 17 s.

Right, hunt after hunt

Whole hunts of the planted suite, each scored against expected.json: nine causes, nothing else blamed.

This machine5 / 5

First release, starved8 / 12

Now, starved10 / 10

The first row is python scripts/reliability.py 5 here: 56 s to 227 s a hunt, 9 min 18 s in all. The other two are the README's, on 4 CPUs with 4 busy loops; it also reports 10 of 10 on 2 CPUs and on GitHub's runners. 30 of 30 is a 95% interval of 89 to 100%: good, not proven perfect.

Why the statistics matter

With the confirmation step switched off (the no-confirm mutant), the hash-order test is also blamed on the time zone and on thread timing. Before the second stage and the split significance level, the README records 8 of 12 hunts right on a starved machine, and a seed once "confirmed" as a race's cause, 13 of 32 against 1 of 32.

8 / 8 vs 0 / 8: p = 0.000088 / 8 vs 3 / 8: p = 0.013

Both computed with flaky-hunt's own fisher_greater. The second isn't enough at a level split over two stages and several causes, which is why a race gets 32 runs a side.

47 tests, five of them whole hunts

Counted with pytest -q: 47 tests, all passing. CI splits them: pytest -m "not slow" runs 42 in about 18 s here, and pytest -m slow runs the 5 whole hunts, which took 4 min 57 s on this machine while other work was running.

Tested the way it tests others

Whole hunts: the planted suite, every factor, scored against expected.json; then order, environment and time zone alone; and a test that fails at random must not be blamed on anything.
Every plugin option in a real pytest process, through pytester: seeds, the clock, thread switching, sleep jitter, the busy thread, the fast clock, the network block.
The hunt logic against a fake runner, including tests that need an earlier test, and both sides of a comparison run alternately.
The bisection against an oracle with the culprit at every position in lists of 1 to 333 tests.
Statistics: Wilson intervals at 0 of n and n of n, and the Fisher test against known values and in the right direction.

CI lints with ruff and runs the fast tests on Python 3.10, 3.12 and 3.13. A separate job runs the whole hunts, the six mutants, ten scored hunts and the README quickstart.

Deliberately missing

Two causes at once: It finds causes that one factor triggers. A test that fails only when a shuffled order and a slow machine coincide is reported as flaky with no single factor, not diagnosed.
Real CPU starvation: Timing is simulated in-process: frequent thread switches, longer sleeps, a busy thread, a fast monotonic clock. No cgroups, no slow disk, nothing inside C extensions.
Clocks outside Python: time-machine moves time.time, datetime.now and friends. Code that reads the clock in C, or in a subprocess, still sees the real time.
Every order: Three shuffles and a reversal won't find every order dependency in a large suite, and running each test alone is skipped above 200 tests unless --isolated-max is raised.
xdist, nose, tox: Runs are serial inside each pytest process; flaky-hunt runs several processes at once instead. Plain pytest only, and not on PyPI.

Run it

git clone https://github.com/ahmed-hashim-pro/flaky-hunt
cd flaky-hunt
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'
pytest -q -m "not slow"        # 42 tests, seconds

cd examples/planted
flaky-hunt run                 # exits 1: it finds 9
flaky-hunt run --factors order

# Your project, where you run pytest:
flaky-hunt run tests/ --json flaky.json

Python 3.10 or newer. Install it into the environment your tests run in. make check adds the whole hunts, make mutants the broken versions, and make real repeats the real-project runs in Docker.