tsz

A Rust library and command-line tool that compresses time series of (i64, f64) points without losing a single bit. For each block of 4,096 points it measures Gorilla, Chimp, Chimp128 and a decimal codec, and writes whichever is smallest.

Gorilla, the usual starting point, XORs each value with the one before and stores the bits that changed. That works when neighbouring floats share most of their bits. Most metrics reach a database as text, though: a tide gauge prints 1.327, and parsed to a double that is a repeating binary fraction. On five of the six real series here, Gorilla comes out bigger than gzip of the CSV. tsz checks, bit for bit, whether a block's values were printed decimals, and if so stores the integers.

View on GitHub Against other crates

Plays the seven stages below in order, about 65 seconds. Scroll or press Esc to stop.

Language: Rust 1.74+, no dependencies
Tests: 21 passing with cargo test, plus a million-series soak
Codecs: Gorilla, Chimp, Chimp128, decimal, and auto per block
Lossless: Every i64 timestamp and every f64 bit pattern, NaN payloads included
Data: Six real series, 1.3 million points, from NOAA, UCI and Wikimedia
License: MIT

tsz compare *.csv file points gorilla chimp chimp128 auto bits/point; auto's value codecs noaa-sf-air-temp.csv 87,637 34.10 32.85 14.29 4.36 decimal 22

Real runs of the release binary, output verbatim. The CSV files are the repository's six datasets, unzipped from data/. The values in the last two tabs were written for this page Illustrative input.

1Read the text, keep the exact bits

tsz encode reads one timestamp,value per line, from a file or stdin. Timestamps are integers in any unit. A value is anything Rust's f64 parser accepts, or 0x and 16 hex digits for an exact bit pattern.

From here on a value is its 64 bits, not a number. -0 keeps its sign, and a NaN keeps its payload.
tsz decode --bits prints the 0x form back, so even a NaN payload survives a trip through text.
Bad input names the file and the line, and exits 2. Nothing is guessed or skipped, apart from one header line.

Six values in, six bit patterns out Illustrative input

The six values were written for this page, encoded with tsz encode and decoded both ways; every pattern is the decoder's own output. The bad files are a few lines each, made to trip the parser. The errors and exit codes are real.

2Gorilla: XOR with the value before

Gorilla, from Facebook's in-memory time-series database, XORs each value with the previous one. A repeat costs one bit. Otherwise it stores the run of bits between the leading and trailing zeros, reusing the last window when the new bits fit inside it.

Timestamps store the delta of the delta. A steady scrape costs one bit per point.
Two changes from the paper, so that every i64 sequence round-trips: a fifth bucket for a full 64-bit jump, and wrapping arithmetic, so i64::MIN can sit next to i64::MAX.
The values follow the paper exactly. Within 2% of the existing tsz crate's size on five datasets is some evidence that both do.

A printed decimal is the problem. 1.327 and 1.297 differ in a few digits, but as doubles their XOR has bits set almost all the way down.

The first ten values of a real series, as Gorilla sees them

Bit set, storedZero, but inside the stored windowZero, not stored

Rows are the start of data/noaa-sf-water-level.csv and noaa-sf-air-temp.csv. The bits per value come from a line-by-line port of the crate's Gorilla value coder, checked against the binary: for the first block of 4,096 points both give exactly the byte counts that tsz encode --codec gorilla printed.

3Chimp and Chimp128: spend the bits differently

Chimp (VLDB 2022) noticed that XORs of real values rarely have many trailing zeros. It pays for a trailing-zero count only when there are more than six, and rounds the leading-zero count to one of eight values so it fits in three bits.

Chimp128 also lets each value pick its reference from the previous 128, through a table keyed on the low 14 bits. An exact repeat costs 9 bits.
Sensors revisit the same readings, so on the five sensor series Chimp128 is 45 to 66% smaller than Gorilla.
Both follow the paper's flag layout. A test pins the papers' costs: a constant series is exactly 1, 2 and 9 bits per value under Gorilla, Chimp and Chimp128.

On the Wikipedia counts Gorilla beats both. They are integers stored as floats, so their XORs end in long runs of zeros, and Chimp pays 11 bits of header where Gorilla usually reuses its window for 2.

Bits per point on six real series, against gzip of the text

gorillachimpchimp128csv.gz, the file itself

Every bar is from tsz compare data/*.csv, and the csv.gz marks from the bench's size table, which this page re-ran: both match the README to the last digit. Bits per point counts the timestamp and the value together; uncompressed is 128.

4Decimal: check that every value was printed

For d from 0 to 15, the encoder tries n = round(v × 10^d) on every value in the block. It keeps the first d for which n / 10^d is the same 64 bits as v. Division and parsing are both correctly rounded, so 1327 / 1000 is exactly the double that "1.327" parses to.

Comparing bits, not values, is what keeps it lossless. -0.0 and NaN never match, and neither does a computed value such as 0.1 + 0.2.
The integers go to an adaptive code: first or second differences, zigzagged, with two bucket widths fitted to the block from all 2,016 pairs.
The same integer code competes for the timestamps, so gaps and irregular clocks get fitted widths too.

One value that doesn't fit sends the whole block to a float codec. There are no per-value exceptions, which pco and ALP have and this doesn't.

The same water levels, stored as integers

Break one value in a block of power readings Illustrative input

The integers and residuals are the decimal coder's, from the same port, which matches the binary's 3,890 bytes for this block. The block is 4,096 points, so 18 bits of header and the first value's 64 are spread over all of them. The broken files are the first 8,192 points of uci-power-active-2007.csv with line 10 changed; the lines are tsz compare output.

5Auto: measure every codec, write the cheapest

Every encoder is generic over a bit sink. auto runs each candidate into a counter that only adds up lengths, and writes only the cheapest. Blocks are independent, so each one chooses its timestamp codec and its value codec on its own.

The counter gives up as soon as it passes the best size so far. Decimal goes first, so the float codecs usually stop early. That took auto from about 17 to about 26 million points a second on encode, by the README's measurement.
Ties go to the earlier candidate, so the same input always gives the same bytes. A test hashes the bytes every codec writes for a fixed series.
Because auto chooses from a superset of what each fixed codec uses, it is never bigger than any of them. The property tests check that on every random series.

One file, three blocks, three choices Illustrative input

The file was generated for this page: 4,096 printed readings with one decimal place, 4,096 draws from a palette of 20 computed floats, and 4,096 points of a computed sine. Each block's bars are tsz compare on that block alone; the terminal is tsz inspect on the whole file encoded with auto.

Bit for bitAbove this line tsz chooses how to store each block. Below it, what comes back must be the same 64 bits that went in, and bytes from anywhere are read as if they were hostile.

6Decode untrusted bytes without panicking

The format is a magic number, then blocks of a point count, one codec byte and two length-prefixed bit streams, then a zero. The decoder checks every one of those against what the encoder could have written, and a mismatch is an error, never a panic.

A header claiming a block of more than 2²⁰ points is refused before anything is allocated.
Every truncation of a valid stream must be an error. The tests cut 300 streams at every byte, flip bits in 2,000, and append random bytes after the magic 20,000 times.
Two cargo-fuzz targets run in CI on every push: arbitrary bytes into the decoder, and arbitrary series through every codec and back.

There are no checksums. A flipped bit that keeps the structure intact decodes to wrong values with no error, so wrap the stream in something with a CRC if that matters.

Seven damaged streams

Six are copies of the first 5,000 points of the water-level series, encoded with auto to 5,373 bytes and then damaged as described; huge.tsz is the header the test suite uses. Messages and exit codes are what tsz decode printed. For the flipped bit, the values were compared bit for bit with the original: every timestamp survived, and 1,925 of the 5,000 values did not.

7Prove the round trip, every codec

tsz verify encodes a file under every codec, decodes it, and compares each value bit for bit. It exits 1 on any mismatch. The test suite does the same on series built to break float handling.

Every NaN class with random payloads, both zeros, both infinities, subnormals and the extremes, next to every other.
Timestamps that are regular, jittered, backwards, duplicated, random, or i64::MIN and MAX; lengths of 0, 1, 2 and a block plus or minus one.
cargo test runs 20,000 random series. The soak in CI runs a million, and checks on each that auto is no bigger than the best fixed codec.

Six datasets, four codecs, 1.3 million points

tsz verify data/*.csv on the release binary, then make soak, both verbatim. The soak took 156 s here, on a machine running other jobs, and 97 s in CI.

Measured, not assumed

The repository benchmarks tsz against zstd, the existing tsz crate and pco, and CI fails if the chart drifts from what the code produces. This page re-ran the size table: every figure matched the README, and the chart file came out byte-identical.

A year of household power

521,669 one-minute readings from uci-power-active-2007, in bits per point.

tsz crate52.54

csv.gz33.39

zstd22.80

tsz auto9.18

pco6.75

13.9 times smaller than 16 bytes a point, and 2.5 times smaller than zstd on the raw columns.

pco is smaller, everywhere

tsz auto against pco 1.0.4 at its default level, bits per point:

pco also notices the decimal step, then entropy-codes over bins fitted to the whole chunk. If data can go over in chunks, the README says to use pco.

The suite, and what CI adds

cargo test, warm, twice in a row on this machine:

11.9 s13.4 s

Most of it is the 20,000 random series. CI runs the suite on Rust 1.74 and stable, rustfmt and clippy with warnings as errors, the million-series soak, two minutes of each fuzz target, and the size chart check.

A two-minute fuzz run of decode here made 13.1 million attempts and found nothing.

21 tests, and one long one

Counted with cargo test --locked, as CI runs it: 21 passing across unit tests, three integration files and one doc test. The soak is a 22nd test, ignored by default and run with make soak; it passed here too.

Weighted toward breaking it

Round trips: seeded generators for adversarial series, every codec, bit for bit; every special value next to every other; block boundaries; a streaming encoder that must write the same bytes as encode.
The format: FNV hashes of the exact bytes each codec writes for a fixed series, so a format change can't slip in, and the papers' bit costs for a constant series and a regular clock.
Untrusted input: every truncation, flipped bits, random bytes, and framing errors that name the problem.
The CLI as a user runs it: exact bits through text, stdin and stdout, errors that name the line, and decode | head exiting quietly instead of panicking.

While building it, nine bugs were planted by hand, one at a time, and the README records that eight made the suite fail. The ninth turned out to change no output at all.

Deliberately missing

Checksums: A flipped bit inside a payload can decode to wrong values without an error, as stage 6 shows. Wrap the stream in something with a CRC if that matters.
An index: Blocks are independent, but finding the block for a time range means walking the framing from the start.
Exceptions in the decimal codec: One value that isn't n / 10^d sends the whole block to a float codec.
Other shapes and lossy modes: One series of (i64, f64): no integer or string values, no columns or tags. Nothing rounds, quantizes or drops points.
crates.io: The package is tsz-codec and lives on GitHub. tsz on crates.io is the older Gorilla crate it is compared with.

Run it

git clone https://github.com/ahmed-hashim-pro/tsz.git && cd tsz
cargo build --release
export PATH=$PWD/target/release:$PATH

gunzip -k data/*.csv.gz
tsz compare data/*.csv
tsz encode data/uci-power-active-2007.csv -o power.tsz
tsz inspect power.tsz
tsz decode power.tsz | head -3
tsz verify data/*.csv

# The suite, then the soak and the size check CI adds.
cargo test
make soak bench-sizes

Rust 1.74 or newer and nothing else for the crate. The bench in bench/ is its own package and pulls zstd, pco and the tsz crate for comparison; fuzzing needs nightly and cargo-fuzz.