trace-diff

A command-line tool that compares OpenTelemetry traces from before and after a change, and reports the structural regressions that tests and latency dashboards miss: a query that now runs once per item, a call that is now retried, a fan-out that grew, a new call the request has to wait for.

A change can pass every test and leave the latency graph flat, or even better, while it doubles the work behind each request. In the repository's example the p95 fell from 73.4 to 50.3 ms and the spans nearly doubled, from 2,128 to 4,203. Each finding names the operation, says how often and how much, and points at trace ids you can open.

View on GitHub What it reports

Plays the eight stages below in order, about 70 seconds. Scroll or press Esc to stop.

Language: Go 1.24 or newer
Tests: 37 passing, 4 of them in the demo module
Dependencies: The OTLP protobuf types only; the statistics are written in the repository
Reads: OTLP JSON from the Collector's file exporter, gzipped or not, or OTLP/HTTP directly
Reports: 11 kinds of finding at three severities
Exit codes: 1 when something should block a merge, 0 otherwise, 2 on bad input
License: MIT

trace-diff check examples/shop/before.jsonl.gz examples/shop/after.jsonl.gz before: examples/shop/before.jsonl.gz, 300 traces, 2128 spans after: examples/shop/after.jsonl.gz, 300 traces, 4203 spans FAIL n-plus-one frontend GET /orders/{id} FAIL retry-storm frontend GET /orders/{id} FAIL new-sync-call frontend GET /orders/{id}

Real runs of trace-diff 0.1.0, output verbatim. The traces are the repository's committed example: a small shop of three Go services instrumented with the OpenTelemetry SDK, 300 requests a run, written by a real OpenTelemetry Collector. The change under review makes orders load items lazily and the frontend retry pricing.

1Read the traces you already collect

trace-diff reads OTLP JSON: what the OpenTelemetry Collector's file exporter writes, or OTLP/HTTP request bodies, one object or many per file, gzipped or not. Nothing has to change in the services.

Add a file exporter to the Collector you already run, or point the services straight at trace-diff collect, which speaks OTLP/HTTP in protobuf, the SDKs' default, or JSON.
A span delivered twice by a Collector retry is dropped, and a trace whose root was lost takes its earliest server span as the root.
Input it can't read is an error that names the file and the object, with exit code 2.

Three runs of the shop, and the receiver

The three files are examples/shop/ in the repository: the code before the change, the same code run again, and the change. Sizes are the gzipped files. The receiver was sent the first five Collector batches of the before run as OTLP/HTTP JSON posts, each answered 200, then stopped with an interrupt, as Ctrl-C would.

2Name each call by what it did

Span ids differ between runs, and so do the values inside them. So a span is identified by its service, its kind, and a description with the values taken out. That is what makes a call in one run the same call in the other.

A database span is its statement with literals and bind parameters replaced: id = 42 and id = $1 both become id = ?, and IN (1, 2, 3) becomes IN (?).
An HTTP server span is its route. An HTTP client span is the host and the path, with numbers, UUIDs and other ids replaced.
Without this step, every request's calls look new. The keep-literals mutant leaves the values in, and the retry storm below disappears.

Spans as recorded, and the operation trace-diff names

The first four spans are from trace d322190d of the shop's after run. The rest come from trace-diff's own synthetic shop (trace-diff synth --scenario n-plus-one --seed 101), which inlines literal ids the way some drivers do. The operations are what trace-diff summary prints for those runs. The terminal compares the synthetic retry-storm scenario with the deliberately broken keep-literals comparison: only the latency is left.

3Compare requests to the same endpoint

Traces are grouped by their root span, such as frontend GET /orders/{id}, and every comparison is between requests to the same endpoint. The traffic mix can differ between the runs without hiding anything.

For each request and each operation: how many calls, the most under one parent, how many of those ran at once, and how many followed a failed attempt.
The critical path, walked back from the end of the root: the child that finished last, then the same inside it. Time spent waiting inside a call is what "the request waits on it" reports.
An endpoint with fewer than 20 requests on either side is listed as a note and not compared.

Four requests to GET /orders/{id}

The operation the finding is aboutEnded with an errorOther spansThe response was sent

Each waterfall is drawn from trace-diff show for that trace id, on its own time scale; the request durations are in the caption above it. The four are the ones the report points at: the median request before the change, a typical request after it, and the worst after it for each of the two findings. The counts below a waterfall are read off the same output.

4Ask whether the change is real

Every question is a statistical test, per endpoint and per operation. Fisher's exact test asks whether more requests make a call, or retry it. Mann-Whitney U asks whether requests make more of a call, and whether they got slower. Fisher again asks whether more requests now land past the old p95.

A comparison runs many tests at once, so they are corrected together with Benjamini-Hochberg, at a false discovery rate of 0.001.
The tests are written in the repository, about 140 lines with no dependencies. Mann-Whitney matches scipy's to five decimal places on the test's sample.
Two runs of the same code should give nothing to report. On the shop they don't.

Every test in one comparison, smallest p first

trace-diff doesn't print its p-values; these were listed by a scratch test added to a copy of the repository, which calls the same functions check does and prints each test with its raw and adjusted p. Bars are on a log scale and stop at 10^-12; the exact value is beside each. Drift is the synthetic scenario with small real shifts, seed 101, 3,000 requests a side.

5Ask whether it is big enough to matter

With enough traffic, a 2% shift is significant and still not worth blocking a merge. A significant change is reported only if it also clears a floor.

Ten points more of the requests making a call.
Half a call more per request, and 1.5 times as many.
Two points more of the requests retried, and at least three of them.
Latency: 20% and 2 ms on the median; or on the p95, with at least a tenth of the requests past the old one.

On the synthetic benchmark, the deliberately broken no-floors comparison reports drift on 76 seeds of 200. With the floors, none of the drift seeds fail the check.

Drift, seed 101: with the floors and without

Both runs are real trace-diff check output on trace-diff synth --scenario drift --seed 101: a pricing cache that hits 64% of the time instead of 70%, user lookups 8% slower, payments retried on 4% of checkouts instead of 3%. Trace id lines are left out here. The cache miss rate cleared the ten-point floor by chance plus the planted shift, so it is reported, as a warning.

6Name the change by its shape

A call that went up is named by how it went up, in this order. The name decides the severity, and the severity decides the exit code.

A new client call brings its server span and its queries with it, and a retried call repeats everything beneath it. Those are one change, so findings about a subtree are folded into the finding about its top.
A removed call and a new one under the same parent are paired, so the report says what the new call replaces.
A latency finding lists where the time went: the operations whose time on the critical path grew most.

Five planted changes, five names

The rules are the order in classify(); latency is decided separately, from the whole request. Each terminal is the first finding of trace-diff check on that synthetic scenario, seed 101, with the trace id lines left out. The scenario descriptions are trace-diff synth's own.

BlockedAbove this line trace-diff reads, matches, tests and names. Below it, a failing finding stops the merge, and the person who wrote the change opens the traces it points at.

7Fail the check, and point at the traces

check exits 1 when a finding should block the merge, 0 otherwise, and 2 on bad input. diff prints the same report and always exits 0. In CI, record the base branch and the pull request under the same load and compare them.

--fail-on warn makes warnings fail too. --json writes the report as JSON, and --svg draws the two traces behind a finding.
Each finding ends with trace ids: the worst request after, a typical one, and the median request before. trace-diff show opens any of them as a tree.
The comparison is only as good as the load. An endpoint only one run exercised is a note, not a finding.

Six runs, three exit codes

# a CI step, as the README gives it
- run: ./trace-diff check base.jsonl pr.jsonl

Every row is a real run, with examples/shop/ left out of the paths, and the line under it is its last line of output. The first three use the shop's committed traces; the drift rows use the synthetic scenario from stage 5.

8Catch it in a real shop, not only a synthetic one

The repository's demo is a frontend, an orders service on Postgres and a pricing service, instrumented with the official OpenTelemetry Go SDK and exporting to a real OpenTelemetry Collector. make demo runs it three times in Docker: before the change, the same code again, and after.

The change passes the shop's own tests: lazy and eager loading return identical orders, and a slow first quote is recovered by the retry.
The p95 improves because the frontend stops waiting for pricing's slow path, not because the slow path went away: pricing still finishes the abandoned request.
CI runs the whole demo on every push and requires both the N+1 and the retry storm in the report.

A fresh run on this machine

Measured, not assumed

The repository's own benchmark, rerun here: a synthetic shop with four endpoints, a cache that hits 70% of the time, retries on 3% of payments and a traffic mix that moves by up to 50% between runs. Each scenario plants one change, or none, and is compared over 200 seeds.

Every planted change, every seed

A finding counts only with the right kind about the right operation, 300 requests a side (3,000 for drift).

No seed produced a finding that would fail check by mistake. One new-sync-call seed and five drift seeds reported something besides the planted change, and none of it would fail the check.

Five broken versions, all caught

Each mutant breaks one decision in the comparison and runs the same 1,600 comparisons. The table must change, and does.

What it costs

The full benchmark, 1,600 comparisons and the five mutants, on one machine shared with other jobs:

The README gives 15 to 20 minutes on four cores; GitHub's runner took 15 and 20 minutes on the two pushes. A single check of the shop's committed runs takes 0.1 s, and of drift, 3,000 requests a side, 0.8 s.

37 tests, counted with their subtests

33 in the main module with go test -race ./...: 21 test functions and the 12 subtests of the benchmark's two tests. 4 in the demo module, where the lazy-loading test needs the demo's Postgres, as CI's demo job provides it. All 37 pass.

What they hold it to

Statistics: Fisher's tea-tasting table to twelve decimals, Mann-Whitney against scipy, Benjamini-Hochberg on a known case, quantiles that don't sort the input in place.
Matching and features: SQL and path normalization, critical path times that add up to the root, concurrency and retries, duplicate spans dropped, and endpoints with too few requests skipped rather than guessed.
Input: the Collector's file format, base64 ids and zero parents, gzip round trips, the receiver in protobuf and JSON, and bad input that says where.
The benchmark in miniature: every scenario on its first five seeds, and every mutant visibly wrong within ten.
The CLI and the demo: exit codes 0, 1 and 2, usage errors, summary and show; the real OpenTelemetry SDK exporting to the receiver; the shop's own tests.

CI also checks that docs/ is what the current code makes of the committed runs, reruns the 200-seed benchmark against docs/bench.txt, and runs the real demo.

Deliberately missing

Running your load: It compares what it is given. Two runs under different traffic, data or configuration differ for those reasons, and it will say so.
Skewed sampling: Both runs must sample the same way. Tail sampling that keeps slow or failed traces skews both counts and latency.
Tiny endpoints: Under 20 requests a side is skipped, and a change that affects one request in a hundred needs many hundreds to be seen.
Invisible retries: A retry inside a client library without its own span can't be seen. Neither can OTLP protobuf files or gRPC: files are JSON, the receiver is OTLP/HTTP.

Run it

git clone https://github.com/ahmed-hashim-pro/trace-diff
cd trace-diff
go build -o trace-diff ./cmd/trace-diff

S=examples/shop
./trace-diff check $S/before.jsonl.gz $S/rerun.jsonl.gz  # exit 0
./trace-diff check $S/before.jsonl.gz $S/after.jsonl.gz  # exit 1
./trace-diff show $S/after.jsonl.gz 08f7b4dbb263

go test ./...      # a few seconds
make demo          # the real shop, needs Docker

Go 1.24 or newer; the demo module needs 1.25, as the OpenTelemetry SDK does. make bench reruns the 200-seed tables and takes 15 to 20 minutes.