Replays past SLI data through a proposed alert rule before it goes live. For each rule it reports which incidents it would have caught and how late, how much error budget burned before it paged, and how often it would have woken someone for nothing.
Alert rules are usually tested by waiting. Ship the rule, then learn at 3 a.m. that it pages on every overnight traffic dip, or learn in a post-mortem that it slept through the slow burn that ate half the month's budget. The data to answer both questions is already in your metrics.
Plays the seven stages below in order, about 75 seconds. Scroll or press Esc to stop.
from-prom converts two range-query exportsAll three are real runs of slo-proof 0.1.0 on the repo's example month. Refused feeds it two small files written for this page: a CSV with a missing minute and a rules file with a misspelt key.
The SLI is one CSV row per minute: good events and total events. Incidents are a second CSV, and the rules are TOML. Every file is parsed strictly, and a bad one stops the run with exit code 2 and a message that names the file and, for the CSVs and the rules, the line.
windows would otherwise make a rule that silently never fires.Rules errors give the line of the key at fault, and also name the rule and the condition, so a typo in a long file is one jump away.
Five files it won't replay Illustrative input
[slo] table.The rule is evaluated at the end of every minute. A window of W minutes covers the last W complete minutes, and its error ratio is bad events over total events, what sum(increase(bad[W])) / sum(increase(total[W])) gives over per-minute recording rules.
for works as in Prometheus: the first true evaluation makes the alert pending, it fires at the first evaluation at least for later, and every evaluation in between must hold. A false one resets the clock.One stated simplification: Prometheus's increase() extrapolates to the window's edges, while slo-proof sums whole minutes exactly. The difference is a fraction of a minute's events at each edge.
A 4-minute outage against for = "5m"
Rulestatic-1pct: error ratio above 1% over 5m, for 5m
holds() and evaluate(). Bars are the share of that minute's requests that failed. The row for fast-burn-only is the same data under the second rule in examples/rules.toml.A rule fires if any of its when conditions holds, and a condition holds only if the error ratio is above its threshold over every one of its windows. That one rule shape covers a static threshold, a single-window burn rate and the SRE Workbook's multiwindow, multi-burn-rate alert.
The eight hours after the outage
6x-over-6hworkbook-multiwindowevaluate(). The outage lasted from 13:41 to 13:45. Over the month this one shape costs 6x-over-6h 18h43m of firing with no incident open.This is the line. Before a rule can wake anyone, each page it would have sent is held against the incidents you know about. A page during an incident is true. One within an hour after an incident ended is late: about something real, but nobody could act on it. Anything else is false.
page are the ones a rule has to catch. ticket incidents were real but could wait: a page during one isn't false, and missing one isn't held against the rule.The incident records are taken as true. If a recorded start is when someone noticed rather than when users did, every delay is measured from the wrong moment, which is why the repo ships a synthetic month where the truth is known.
Every page static-1pct would have sent in 28 days
The first false page: 8 March, 02:00 to 02:40, requests per minute
score() on the committed example month. Incidents are drawn above the axis, pages below it. Traffic in the month averages 548 requests a minute; during the three night dips it falls to about 6 while health checks fail, which the synthetic month adds on purpose.ClearedAbove this line slo-proof only replays the past. Below it, each rule leaves with its record: what it missed, what the delay cost, and an exit code that can keep it off the pager.
Catching an incident isn't the whole answer. For each incident the replay reports the bad events from its start until the page, as a share of the period's error budget at the data's average traffic, next to what the whole incident cost.
(1 - target) times average traffic per minute times the minutes in the period, 30 days unless the rules say otherwise.The slow burn: 0.7% errors for 8 hours
score() and evaluate() on 18 March of the committed example month; the four percentages match slo-proof replay. Even the rule that catches everything lets 5.5% of the month's budget go first. The table says so before an incident does.compare replays every rule in the file and ranks them: incidents caught first, then false pages per week, then median delay. It is a default order, not a verdict, and the table carries the numbers for any other.
Four rules from examples/rules.toml, ranked
slo-proof compare printed. Every number is from that run, verbatim.With --fail-on-miss, replay and compare exit 1 when a rule misses an incident that deserved a page. A pull request that changes an alert rule can then be gated in CI on last quarter's incidents.
--fail-on-miss. 2: bad input.Three runs, three exit codes
slo-proof synth builds a month with a daily and weekly traffic rhythm, Poisson noise, and one of each kind of trouble that tells rules apart. The seed moves every incident by up to an hour. The cards below replay the four example rules on seeds 1 to 8 with the real CLI.
On all eight months, the multiwindow rule caught all six page-worthy incidents with no false pages, and fast-burn-only missed exactly the slow burn.
The static threshold missed the hard outage, the slow burn and the short blip on every seed. A repo test checks the same claims on the same eight seeds.
Hours 6x-over-6h spent firing with no incident open, by seed.
Every month: over 18 hours. The multiwindow rule stayed between 1h38m and 1h41m.
Each synthetic month has three overnight dips where traffic falls to about 6 requests a minute while health checks fail. A 1% static threshold reads them as outages.
False pages a week from static-1pct. The dip on 8 March spent 0.19% of the month's budget.
The evaluator is checked against a second implementation written another way: it sums each window by hand and fires wherever the expression held at every evaluation of the last for minutes. Hypothesis feeds both up to 400 random series and rules, with up to three conditions of up to three windows each.
for, a false evaluation resetting the pending clock, still firing at the end of the data, all windows required, any condition enough, no traffic never firing.for never pages more.synth --seed 7 writes, and the README's findings hold on eight seeds.CI runs the tests on Python 3.11 and 3.13, ruff and mypy --strict, and checks that the committed examples and pictures match what the current code generates.
git clone https://github.com/ahmed-hashim-pro/slo-proof && cd slo-proof
pip install -e .
slo-proof compare examples/sli.csv examples/incidents.csv examples/rules.toml
slo-proof replay examples/sli.csv examples/incidents.csv examples/rules.toml --rule workbook-multiwindow
slo-proof check examples/rules.toml
slo-proof synth --seed 3 --out mymonth/
# Tests
pip install -e '.[dev]'
pytest
Python 3.11 or newer, in a virtualenv. The tool itself has no dependencies; the dev extra adds pytest, Hypothesis, ruff and mypy, all pinned.