ota-safe

A/B firmware updates for microcontrollers, with a trial boot and automatic rollback, in no_std Rust with no allocation. Beside it is the tool that checks the design: a NOR flash simulator that cuts power at every erase and program of a device's life, in every way that operation can be left half-done, and checks the device still boots.

A firmware update is the riskiest thing a deployed device does. It happens in the field, often on a battery, and a device that can't boot after a power cut needs someone to come and reflash it. Update schemes are usually tested by pulling the plug a few times. A device's update path is a few hundred flash operations, and the dangerous moments are usually one or two of them, so this cuts power at all of them.

View on GitHub How it works

Plays the seven stages below in order, about 60 seconds. Scroll or press Esc to stop.

Language: Rust 1.81 or newer; the core is no_std and forbids unsafe
Tests: 33 passing with cargo test, plus the depth-2 search in make deep
Dependencies: sha2 and ed25519-dalek, without default features
Searched: 998 single power cuts and 356,028 lives with two, every one ok
Images: SHA-256 hashed, Ed25519 signed, verified before every boot
Runs on: A Cortex-M3 in QEMU, deciding as the host did on all 998 cut states
License: MIT

ota-sim matrix good: v1 installs v2, v2 installs v3; every version healthy

Real runs of the release build of ota-sim, output verbatim. The middle two tabs cut power at the same operation, op 44, an erase, in two different schemes. The simulated images are signed with a fixed test key, so every run gives the same result.

1Model the flash, and every way a cut leaves it

SimFlash models NOR flash. Erasing sets a 4 KiB sector to 0xFF. Programming stays inside one 256-byte page and can only clear bits, so programming over data that isn't erased ANDs the two. A power cut is armed at the n-th erase or program after power-on; when it fires, that operation is left torn and every later one fails until power returns.

A torn program leaves the first quarter, half or three quarters of its bytes, with the next byte partly programmed, or each bit that should clear done at random.
A torn erase leaves each byte either erased or with only some bits set back to 1, in two different patterns.
before is power going just before the operation, which also covers just after the one before it.

The tears are seeded from the cut, so every result replays exactly.

What a cut leaves in one 32-byte record

The flash at 0x01000 as each cut left it, from the simulator's own snapshot of the good workload: op 48 erases that sector and op 49 programs the record that confirms v2. The record it should have become is the last button. The verdicts are Meta::decode on those bytes and the bootloader's decision at the next power-on.

2Commit with one record write

Two metadata sectors hold 32-byte records. A record says which slot is confirmed and at which version, any trial with its boots left, and the last version rolled back, with a sequence number and a CRC-32. The current record is the valid one with the highest sequence number. Every change erases the sector that does not hold it and programs the new record there.

If power goes during that erase or program, the old record is still whole in the other sector, and it still wins.
The CRC rejects a record that was cut short; the sequence number picks the newer of two whole ones.
Each step has exactly one commit point, this write, and the record that was current before it stays valid until the new one is complete.

The seven records of one device's life

The good workload with no cut: v1 installs v2, v2 confirms and installs v3, v3 confirms. Each record is the simulator's flash read back with read_meta right after the write, and the op numbers are the life's own. The first record is written at provisioning.

3Try the new image, and roll back without help

An update refuses an image that is unsigned, not newer than the confirmed version, or the version last rolled back. It erases the slot that isn't confirmed, writes the body and then the header page, reads the slot back and verifies it, and only then writes the record that starts a trial with three boots. The confirmed slot is never touched.

The header holds the version, the body length, its SHA-256 and an Ed25519 signature over those 44 bytes. The header page goes last, so a half-written slot never verifies.
The bootloader spends a boot before jumping to the trial. If the firmware never confirms, the third boot is the last.
Then it writes a record that ends the trial and marks that version rejected, and boots the confirmed slot. A confirmed slot that doesn't verify is reported as a brick, never guessed around.

v3 crashes before it can confirm itself

ota-sim life --workload bad-update, from where v3 is written, output verbatim. Beside it is the record each line leaves, read back from the simulator's flash.

4Cut the power at every operation

The search runs one life per cut: every erase and program of the uncut life, with every tear that operation allows, and the device carries on from whatever its flash says. After every boot the checker reads the flash itself instead of trusting the bootloader, and asks three things: is the booted slot a valid signed image of that version, is it at least the newest confirmed one, and has an unconfirmed version booted more often than the trial allows.

Naive in-place rewrites the only slot. Power lost anywhere between its first erase and last page leaves nothing to boot.
Naive A/B writes the other slot, then erases and rewrites a one-byte flag. All 14 cuts that brick it fall in those two flag rewrites. With a bad update it has no way back.
ota-safe survives all 998 single cuts, and all 356,028 lives where a second cut lands after the device comes back.

Worst result of a cut at each operation, v1 to v3

Every column is one erase or program of the uncut life, coloured by the worst outcome over its tears, the chart ota-sim chart draws, read from the same search through the crate's own search function. Choose a column to see each tear. The totals are ota-sim matrix.

5Plant bugs the search has to catch

This is the line. A search that never fails proves nothing about the search, so six deliberate bugs sit behind a mutants feature that firmware never enables, and the same search runs against each. ota-sim matrix and ota-sim mutants exit 1 if ota-safe fails a cut or a bug goes uncaught, and CI fails with them.

Three are logic bugs that an ordinary bad-update test finds with the power on.
The other three pass every test with the power on. Only a cut finds single-meta-copy, trial-before-image and no-crc.
no-crc fails on 3 of 998 cuts. Half of a trial boot's record is believed, a garbage version is marked rejected instead of v3, and v3 gets three fresh boots on top of the one it had.

A test campaign that pulls the plug at random would need luck to find that one.

Six planted bugs, and the life that catches each

Rows are ota-sim mutants, both workloads at depth 1, verbatim. Choosing one replays its first failure with ota-sim life; the exit code is the shell's.

ClearedAbove this line the update code runs inside the simulator, where every power cut can be tried and every planted bug must be caught. Below it, the same bootloader code leaves the host.

6Make the same decision on a Cortex-M3

The core crate is no_std, forbids unsafe code and allocates nothing, so the bootloader decision can run where a bootloader runs. qemu/ builds ota_safe::boot for thumbv7m-none-eabi and runs it on QEMU's mps2-an385 board.

scripts/qemu-check.sh writes every depth-1 cut state as a 40 KiB flash image, with the host simulator's decision for each.
The board loads each image, decides what to boot, makes any metadata write that needs, and prints the decision and the record over semihosting. The script compares every line.
On the board, flash is RAM with the same AND-only programming rule. No real flash controller and no real power cut are involved.

The QEMU build is 55,644 bytes of code at opt-level = "s", most of it Ed25519 and output formatting.

Eight of the 998 cut states, decided twice

Each row's host line is from ota-sim states and its board line is qemu-system-arm 8.2.2 running the release ELF on that image, both verbatim. The summary is make qemu over all 998.

7Call three functions from firmware

A device implements the Flash trait for its part's driver: read, erase a sector, program within one page. The bootloader calls boot and jumps to the slot it names. The application calls confirm once it is up and healthy, and update with a downloaded image.

The simulator drives exactly these calls, so each one's effect can be read off a device's life.
update takes the whole image as a slice, so the image has to fit in RAM.
The bootloader picks a slot and doesn't copy or swap, so each image must be linked for its slot or be position-independent.

Each call, and what it did in a life

The code is the README's sketch of the API, which isn't compiled as part of the build. The lines under it are ota-sim life output for the good and bad-update workloads, verbatim, picked out where that call ran.

Measured, not assumed

Run on one 12-thread Linux machine that other jobs were loading heavily at the time, so these are slower than an idle machine would give. Every count matched the README exactly.

Two cuts, every pair

ota-sim matrix --depth 2 runs a second cut at every operation after the device comes back, for all three schemes and both workloads.

356,028 ota-safe lives0 failures234 s

The README says about two and a half minutes on 12 cores. Naive A/B, for comparison, bricks in 4,634 of its 153,654 lives in each workload.

Every state on the board

make qemu: builds the ELF, writes the 998 images, and boots QEMU once per image.

Decided alike998

Differed0

108 s in all. Of the 998 decisions, 926 boot a confirmed image and 72 a trial; none is a brick.

The suite, twice

cargo test --locked on stable 1.99 and on 1.81, the minimum version CI also tests:

33 passed1 ignored25 s warm

The single-cut search in the tests takes most of it. ota-sim matrix alone runs in 0.8 s and ota-sim mutants in 9.2 s.

33 tests, and one that takes minutes

Counted with cargo test --locked: 33 passed and 1 ignored, the same on Rust 1.81 and stable. The ignored one is the depth-2 search, which make deep and CI run in release mode.

Weighted toward the claim

The search: every single cut settles, the naive schemes fail exactly where the README says, every planted bug is caught, three of them only under a cut, and a cut replays exactly.
The protocol, power on: three trial boots then rollback and refusal, the updater's refusals, the confirmed slot never touched, records alternating sectors, any flipped bit failing verification.
The flash model: programming only clears bits and never crosses a page, a cut keeps power off, a torn program never sets a bit, a torn erase leaves a mix.
Records: the CRC-32 check value, and a round trip where every single-bit flip of a record is rejected.
The CLI: the matrix, the mutants, replaying a cut, dumping a flash image, and exit 2 on bad arguments.

CI also runs rustfmt and clippy with warnings as errors on both crates, a check that docs/ matches what the code produces, the depth-2 search, and the QEMU check. Run 38657517378 on 79b2335 passed all six jobs; the depth-2 job took 11 minutes.

Deliberately missing

Real hardware: The flash is a model of NOR behaviour, and the Cortex-M3 runs in QEMU with RAM standing in for flash. Marginal erases that lose bits later and slow brown-outs aren't modelled.
NAND: No bad blocks, no ECC, no rule against reprogramming a page.
Everything else: The search is exhaustive for this life and these tears: one or two cuts at every operation of v1 to v2 to v3. Not three cuts, other update orders or other tear shapes.
Secure boot: The signature stops an unsigned image, but nothing protects the bootloader, key or metadata from someone who can write the flash. There is no monotonic hardware counter.
Streaming and swapping: The image must be in RAM to update, and it runs from the slot it was written to.
Boot time: Every boot hashes and verifies the whole image; on a slow MCU that cost is real, and it isn't measured.

Run it

git clone https://github.com/ahmed-hashim-pro/ota-safe && cd ota-safe
cargo build --release
export PATH=$PWD/target/release:$PATH

ota-sim matrix
ota-sim life --scheme naive-ab --cut 44:erase-0
ota-sim life --cut 44:erase-0
ota-sim mutants

# The suite, two cuts deep, then the Cortex-M3.
cargo test
make deep
make qemu

Rust 1.81 or newer. make qemu also needs rustup target add thumbv7m-none-eabi and qemu-system-arm; make docker-check needs only Docker. Not published to crates.io.