geohashim

fieldlog

An offline-first Android notebook for logging boreholes and outcrops on site. Every phone holds the whole site and works with no signal. When phones reach each other again, their edits merge with nothing to resolve by hand and no server deciding who wins.

Merging is only half of it. Two loggers can both call their new hole BH41, and a perfect merge keeps both. So the app checks every borehole with the rules strata uses, in strata's own words, and won't export until a person has fixed what merging can't. The export then goes straight into strata, which turns it into a ground model with error bars.

View on GitHub How it works

Plays the seven stages below in order, about 60 seconds. Scroll or press Esc to stop.

Language
Kotlin 2.0, for the app, the core and the CLI
App
Jetpack Compose, Android 8.0 or newer
Tests
39 passing: 33 on the JVM, 3 app unit tests, 3 UI tests on an emulator
Merge
Three CRDTs on one Lamport clock, no library
Sync
A relay with no authority, two HTTP calls
Exports to
strata, checked against the real strata in CI
License
MIT
fieldlog demo -o merged.csv relay listening on http://127.0.0.1:33845 (store-and-forward, no authority) desk imports strata's example site: 40 boreholes, 994 operations to the relay pixel-a has 40 holes, pixel-b has 40

Real runs of fieldlog at bcb9199, output verbatim; fieldlog is the built CLI, run in a Java 17 container. fieldlog demo plays the README's story over real HTTP, with two in-process phones and a relay. The relay tab is curl against a real relay, and the strata tab is strata itself, built from GitHub by the repository's check script.

1Log a hole with no signal

The app works the same with sync off. A hole gets a name, a location in metres on the site grid, a collar level, a log of intervals from the site's column, flags such as water strike, and free-text notes.

  • Every change is appended to the notebook's file on the phone at once, so killing the app loses nothing. A UI test checks exactly that.
  • The status line counts what hasn't reached the relay yet: offline · 66 changes not sent.
  • Each borehole is checked while it is typed, in strata's words. A gap, an overlap or a unit out of order shows under the log.
  • Numbers accept a decimal comma: a collar typed as 50,5 exports as 50.5.

BH41 on an Android emulator, and the lines it wrote

The BH41 editor: name BH41, x 600, y 480, collar 64.62, the water-strike flag on, four intervals from topsoil to gravel, and the note Water strike 7.4 m.
Logged offline
The same hole after sand's base was changed from 10.3 to 10: a red line under the log reads gap between 10 and 10.3 m.
A gap, in strata's words

The phone's op log, as typed

    Screenshots of the debug APK, built from the repository, on an API 36 emulator. The typing is the repository's own UI script, DemoScript; the gap was made by hand afterwards, deleting .3 from sand's base. The lines on the right are the first ones that phone sent the relay, verbatim, with tabs shown as spaces.

    2Every edit is one line of text

    An edit becomes an operation with a Lamport timestamp, counter@device, written as one tab-separated line. The op log is the notebook. The app appends to it, the relay stores it, and a sync sends the lines the other side doesn't have yet.

    • Five kinds: set a field, add to a set, remove from a set, insert a character, delete one.
    • Ids are op ids, so two phones can never mint the same hole id. They can still pick the same hole name.
    • The phone's counter starts at 100 here because it had already received the desk's 80 operations, numbered up to 98, and its clock jumps past every id it sees.

    The five kinds, and what typing does

      Typing BH41 into the name field

        Typing the note, one character after another

          Every line and id is from the same phone's log. Remove and delete didn't occur in that run; their formats are from the codec's documentation. A field keystroke is a whole new value, and the higher id wins, so BH41 at 105 replaces BH4 at 104 everywhere.

          3Three CRDTs, one clock

          Each kind of data merges the way a logger would expect, and all three share the Lamport clock, so "later" means the same thing everywhere.

          • Fields are last-writer-wins registers. Name, kind, x, y, collar, and each interval's unit and depths. An edit made after seeing another edit always wins over it.
          • Membership is an observed-remove set. Holes on the site, intervals in a hole, flags on a hole. A removal removes only the adds it had seen, so a concurrent flag survives, and Undo brings a hole back with all its data.
          • Notes are an RGA sequence. Two people typing at the same spot get both sentences whole, not interleaved letters.
          • An operation that refers to one not yet arrived waits in a buffer indexed by what it waits for, so a whole log arriving shuffled is applied in one pass.

          Two phones offline, then one sync each

          pixel-a

          Holes
          State

          BH16 notes

          pixel-b

          Holes
          State

          BH16 notes

          Lines from fieldlog demo, verbatim; the cards show what each phone reports at that point. The state is the first six bytes of a SHA-256 of the replica's full dump, tombstones and set tags included, so equal hashes mean equal state underneath, not just equal text.

          4A relay with no authority

          The relay is an append-only list of lines, deduplicated by operation id and saved to a file. It parses each line and refuses malformed ones, but it never interprets an operation, so it can't resolve a conflict and it can't be the reason two phones disagree.

          • Sync is two calls: POST this phone's new operations, then GET everything after a cursor, which comes back in X-Fieldlog-Next.
          • Losing the cursor costs a bigger transfer, never data: the relay and the phones both ignore operations they already have.
          • A batch with one malformed line is refused whole, with a 400 that names the line.

          No authentication and no TLS. Anyone who can reach the relay can read and write the site, so the README says to run it on your own machine or LAN.

          curl against a fresh relay Illustrative input

          Relay holds0 operations

          The relay is the repository's, started with fieldlog relay --port 8791 --data relay.ops. five.ops is five lines of an import of examples/small-site.csv, and bad.ops was written for this page. Responses are verbatim; the # HTTP lines are curl's status codes, and the date header is left out.

          5A fuzzer, and six bugs it has to catch

          Two to four replicas make 80 to 260 random edits on small pools of names, values and notes, so their writes collide. Meanwhile the network drops 8% of messages, duplicates 8%, delays the rest by 1 to 40 ticks, and phones go offline. Reconnecting sends the whole log in a scrambled order. Then three properties must hold.

          • Local intent: every edit shows up at once on the phone that made it.
          • Convergence: every replica ends in the same full state, tombstones and set tags included.
          • Order independence: a fresh replica given every operation in another random order ends in that state too.
          • A run is a pure function of its seed, so fuzz run --seed 1 --bug clock-no-bump --trace replays a failure step by step.

          clock-no-bump is the quiet one: every phone still converges, just not on what you typed. Only the local intent check catches it.

          The correct merge and six deliberate bugs, 2,000 seeds each

          Seed 1 under four of the bugs, and a clean seed

          The table is fieldlog fuzz mutants --seeds 2000, verbatim in the hero's second tab, and identical to the README's. Each bug is one switch in the real merge code, such as keeping a field's last arrival instead of its highest id.

          ConvergedAbove this line the merge is automatic, and a fuzzer shows it holds. Below it, a notebook every phone agrees on can still be wrong, and a person fixes it before strata sees it.

          6Converged is not the same as correct

          The CRDT guarantees every phone ends up with the same notebook, not that the notebook makes geological sense. Two loggers can both create a BH41. Two people can edit one interval until it overlaps the next. Quietly picking one version would hide a real problem.

          • Every borehole is checked with strata's rules: gaps and overlaps, units out of order, units missing from the column, invalid names, two holes at one location, two holes with one name.
          • The problem is marked on the hole, and export is refused until it is fixed. Every phone shows the same problems, because they all have the same notebook.
          • Outcrops and boreholes with nothing logged are left out of the export and listed, not refused.

          One phone, from no signal to an export Illustrative input

          1. The hole list offline: four holes, four ready for strata, and the status pixel-a, offline, 66 changes not sent.

            1 Offline, BH41 logged

          2. After syncing: five holes, two of them called BH41, both marked name clash in red.

            2 Synced: two BH41s

          3. After the other device renamed its hole: BH41 and BH42, every hole marked ready.

            3 The other one is renamed

          4. The export dialog: Ready for strata, 5 boreholes ready, 0 left out, with the start of the CSV.

            4 Ready for strata

          Frames from one emulator running the repository's DemoScript as pixel-a, captured every 0.6 s. The other BH41 was sent by the fieldlog CLI posing as pixel-b, and its rename to BH42 was one operation POSTed by hand; the README's own recording uses two emulators. The terminal lines are fieldlog demo's ending, verbatim.

          7Hand it to strata, and let strata check

          The export is strata's borehole-log CSV. make strata-check builds strata from GitHub at a pinned commit and runs it on what fieldlog wrote, and CI runs the same script.

          • strata validates the merged export from the demo, all 42 holes.
          • At (600, 480), where strata refused the base of gravel because the nearest hole was 120 m away, BH41 turns the refusal into an answer.
          • Importing strata's 40-hole example site and exporting it again changes nothing strata can see: strata cv output is byte-identical.
          • A log with a gap is refused by both tools, with the same words.

          strata query at (600, 480), before and after the merge

          SurfaceExample siteAfter the merge
          Both queries and the four check lines are verbatim output of scripts/strata-check.sh and of strata built by it at 047ca52. The ranges are strata's 95% intervals; at a logged hole they collapse to the logged value.

          Measured, not assumed

          Everything below was run on one Linux machine from a fresh copy of the repository, while other heavy jobs shared it (load average 10 to 47 on 12 threads). CI's numbers are from GitHub's API for run 38620812427 on bcb9199, where all five jobs passed.

          25,000 seeds, none failed

          The correct merge under the fuzzer, in two runs: seeds 1 to 20,000, then 20,001 to 25,000.

          82.4 s16.3 s0 failed

          3.3 to 4.1 ms a seed here, depending on the load. CI's runner did its 20,000 in 40 s. The mutant table, seven variants at 2,000 seeds each, took 53.8 s here.

          On an emulator, three UI tests

          Compose tests on the real activity: log a borehole offline until strata would accept it, survive the app being killed, and bring a removed hole back with Undo.

          Here, API 3690.5 s
          CI, API 34190 s

          Here is the test run alone. CI's figure is the whole emulator step, including the boot. Nobody has run it on a physical phone yet, and the README says so.

          Five CI jobs

          Durations from GitHub's API, rounded to whole seconds.

          39 tests, on the JVM and on a device

          Counted the way CI runs them: ./gradlew -Pjvm-only :core:test :cli:test (33), :app:testDebugUnitTest (3), and :app:connectedDebugAndroidTest on an emulator (3). A fourth androidTest, the README's two-phone recording script, skips itself unless it is given a role, and isn't counted.

            What they pin

            • The merge: a write that saw another wins over it, one winner everywhere for concurrent writes, a concurrent add survives a remove, typing at one spot interleaves by word, characters outside the basic plane stay whole, duplicates ignored, an operation waits for the one it refers to.
            • The fuzzer itself: 300 seeds of the correct merge per run, every deliberate bug caught, the same result for the same seed, and runs that really do drop, duplicate and reorder.
            • strata: import then export gives strata the same site, the import rejects what strata rejects, and the export refuses what strata would refuse, with the reason.
            • The relay: two devices converge over real HTTP, each operation kept once across a restart, malformed operations rejected whole, offline is an IOException that loses nothing.
            • The app: the notebook and sync position survive a restart, the device name freezes after its first edit, and the three UI tests above.

            CI also runs 20,000 fuzz seeds, the mutant table, the demo, make strata-check, and make test and make apk in Docker, as the README says to.

            Deliberately missing

            A physical phone
            It has run on emulators only: API 36, API 34 in CI, and the two-emulator recording. Real radios dropping in and out are untested.
            GPS
            x and y are typed in metres on the site grid, the way strata reads them.
            Security on the relay
            No authentication and no TLS. Run it on your own machine or LAN.
            A warning on field conflicts
            If two people correct the same collar differently while offline, one value wins on every phone and the other isn't flagged. Notes and sets don't lose edits; single-value fields can.
            Compaction
            The log only grows, and every typed character is an operation. Thousands of operations are fine; years of notes in one notebook haven't been tried.

            Run it

            git clone https://github.com/ahmed-hashim-pro/fieldlog.git
cd fieldlog
make test     # 33 JVM tests, in Docker
make apk      # the debug APK, built in a container
make cli

B=cli/build/install/fieldlog/bin/fieldlog
$B demo -o merged.csv
$B fuzz mutants --seeds 2000
make strata-check

            You need only Docker; the Makefile runs Gradle, the Android SDK and strata in containers. Start a relay with fieldlog relay and install the APK on an emulator: its default relay address, 10.0.2.2:8787, is the host as an emulator sees it.