Skip to content
BEACON
Documentationv0.1.0

Beacon reference

An ultra-lightweight time-series database for IoT, edge devices and homelabs. This page is the complete reference: build, ingest, query, stream, and what the storage layer does underneath.

Binary
2.89 MB
Embedded UI
35.1 KB
Tests
71/71
License
MIT

Overview

Beacon is a small time-series database you can drop onto anything with a CPU. It ingests sensor readings over HTTP, compresses them heavily, and serves them back with live streaming — all from a single executable with no runtime and no external services.

Reach for it when the data is small, the machine is small and the network may not be there: ESP32 fleets, home-lab monitoring, offline edge gateways, bench instrumentation. It is a deliberate trade: a compact single-node store with numeric fields only, not a distributed analytics cluster.

What you get out of the box

  • Ingest — InfluxDB-compatible line protocol, single points or multi-line batches, validated atomically before anything is written.
  • Store — Gorilla compression (delta-of-delta timestamps + XOR floats) inside immutable segments, protected by a CRC-framed write-ahead log.
  • Query — time-range reads with optionalavg/min/max/count, plus block-skip pruning by time range.
  • Stream — server-sent events push every ingested point to browsers, with a lag notice instead of back-pressure.
  • Watch — a 35.1 KB dashboard compiled into the binary, served at /. No CDN, no npm, no fonts.

One process owns one data directory: a write-ahead log and immutable Gorilla-compressed segments. The HTTP API, the SSE stream and the dashboard all live in that same process.

RouteMethodPurpose
/api/v1/writePOSTIngest line protocol, single line or batch.
/api/v1/queryGETTime-range read with optional aggregation.
/api/v1/metricsGETEvery known series with counts and bounds.
/api/v1/statsGETEngine statistics and storage footprint.
/api/v1/healthGETLiveness probe.
/api/v1/streamGETServer-Sent Events of ingested points.
/GETEmbedded dashboard (404 with --no-ui).

Errors are JSON with a single error string. Protocol and query problems return 400; anything else is500.

Installation & prerequisites

There is nothing to set up beyond getting the binary on the machine. Pick the path that matches the box you are deploying to.

RequirementRequired forNotes
Rust 1.85+ via rustup.rsMethods 1 · 2 · 3Any source-based path. Edition 2024 is required.
CargoMethods 1 · 2 · 3Ships with rustup, so installing Rust covers this.
gitMethod 2Or download the source archive from Releases.
Nothing — prebuilt binarySkipping the toolchainWindows x64, Linux x64, macOS arm64 from GitHub Releases.
Nothing at runtimeAny methodNo runtime, no Node, no CDN, no external assets.

1. Install with Cargo RECOMMENDED

Pulls the crate from GitHub and drops a beaconexecutable into ~/.cargo/bin:

Shell
cargo install --git https://github.com/prxmel4/beacon

2. Build from source

Shell
git clone https://github.com/prxmel4/beacon.git && cd beacon
cargo build --release

# the binary lands at:
#   Windows        .\target\release\beacon.exe
#   Linux / macOS  ./target/release/beacon

3. Run without installing

Straight from the crate, no install step. Handy while iterating:

Shell
cargo run --release -- serve --port 8086 --data-dir ./beacon-data

Confirm it worked

Any of the three paths should answer with the version embedded at build time:

Shell
beacon --version
# -> beacon 0.1.0

Prefer a prebuilt binary

Download the archive for your platform fromGitHub Releases, extract it, and runbeacon serve. No toolchain, no compilation — the only file you ship is the executable.

Your first 60 seconds

Four steps, in order. Each one takes a few seconds; the whole walkthrough is under a minute on a laptop.

  1. Start the server. Pick a port and a data directory:
    Shell
    beacon serve --port 8086 --data-dir ./beacon-data
    
    # -> listening on http://0.0.0.0:8086
    # -> dashboard: http://0.0.0.0:8086/
  2. Open the console. Browse tohttp://127.0.0.1:8086. The KPI strip shows TOTAL POINTS 0 — nothing has been written yet, which is correct.
  3. Press MOCK SENSOR in the top-right corner. The button turns red and three synthetic series (temperature, humidity, CPU) start arriving every 500 ms: the KPI counters climb, the LIVE dot pulses per frame, and the canvas draws a 60fps trace. Leave it running — it is also your live data source for the next step.
  4. Write something yourself, then read it back. This is the same request path a real device would use:
    curl
    curl -sS -X POST http://127.0.0.1:8086/api/v1/write \
      -H 'Content-Type: text/plain' \
      --data-binary 'temperature,device=bench-01,room=lab value=21.5 1732250000000'
    
    # -> {"points_written":1}
    curl
    curl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&tag=device=bench-01'
    
    # -> {"series":[{"measurement":"temperature",
    #               "tags":[["device","bench-01"],["room","lab"]],
    #               "field":"value",
    #               "points":[{"timestamp":1732250000000,"value":21.5}]}]}

That is the full loop: write → store → query. Ctrl+C on the server triggers a graceful flush; the next start replays the WAL and picks up where it left off.

CLI reference

Two subcommands: serve and bench.

beacon serve

FlagDefaultDescription
--port <u16>8086TCP port to listen on.
--host <addr>0.0.0.0Bind address. Use 127.0.0.1 to stay loopback-only.
--data-dir <path>./beacon-dataDirectory for the WAL and segment files.
--flush-threshold <n>1000Flush the MemTable to a segment after n buffered points.
--no-uioffHeadless: GET / returns 404, the embedded UI is never served.

beacon bench

FlagDefaultDescription
--points <n>1000000Total points to ingest across all series.
--devices <n>8Simulated devices; each produces temperature, humidity and cpu.
--batch-size <n>1000Lines per write batch — one WAL fsync per batch.
--flush-threshold <n>50000MemTable flush threshold during the benchmark run.
--queries <n>200Random 1%-window aggregation queries for the latency report.
--data-dir <path>tempKeep the generated data here instead of a cleaned-up temp dir.

Headless mode

Pass --no-ui and GET /returns 404. The API, the SSE stream and the storage engine are untouched — only the dashboard route is disabled. Use it on fleet nodes where the UI is noise, or where a reverse proxy owns the root path.

Shell
beacon serve --host 127.0.0.1 --port 8086 --data-dir /var/lib/beacon --no-ui

Line protocol

InfluxDB-compatible syntax, numeric fields only:

Grammar
measurement[,tag=value...] field=value[,field=value...] [timestamp_ms]

# timestamps are Unix milliseconds.
# omit the timestamp and the server fills in wall-clock now().

Tags are sorted canonically, so a=1,b=2 andb=2,a=1 address the same series. A batch is validated in full before anything is written: one bad line rejects the entire request and nothing is broadcast to subscribers.

Line protocol
# measurement only
cpu usage=0.64 1732250000000

# tags, multiple fields
temperature,device=sensor-01,room=kitchen value=28.5,humidity=61i 1732250000000

# no timestamp: the server stamps it with wall-clock now()
temperature,device=sensor-01 value=28.5

# comment lines and blank lines are ignored
# this whole batch is validated before any point is written
humidity,device=sensor-01 value=61i 1732250001000

Field types

LiteralStored asNotes
28.5, -3.2e2f64Standard float syntax.
28if64Signed integer suffix, parsed as i64.
28uf64Unsigned integer suffix, parsed as u64.
true / t1.0Also True, TRUE, T.
false / f0.0Also False, FALSE, F.
"text"—Rejected: string fields are not supported.
NaN, inf—Rejected: non-finite values never enter the WAL.

Escaping & errors

Backslash escapes \,, \ ,\= and \\ are honoured in measurement names, tag keys, tag values and field keys:

Line protocol
weather\,station,city=New\ York,sensor=a\=b temp\=out=21.5 1732250000000

Parse failures name the offending line, 1-based, inside the batch:line 2: invalid numeric field 'bad'. Blank lines and lines starting with # are skipped.

POST /api/v1/write

Body is raw line protocol as text/plain — one line or a multi-line batch. Every acknowledged write is fsynced to the WAL before the response is sent.

Request
curl -sS -X POST http://127.0.0.1:8086/api/v1/write \
  -H 'Content-Type: text/plain' \
  --data-binary $'temperature,device=sensor-01,room=kitchen value=28.5 1732250000000\nhumidity,device=sensor-01 value=61i 1732250000000'
200 OK
{"points_written": 2}

Parsed lines are also broadcast to every SSE subscriber. When no subscriber is connected, the ingest path skips the broadcast entirely and does zero extra work.

GET /api/v1/query

ParameterRequiredDescription
metricrequiredMeasurement name, e.g. temperature.
tagoptionalOne filter as key=value, e.g. device=sensor-01.
fieldoptionalRestrict to a single field key.
fromoptionalInclusive lower bound, Unix milliseconds.
tooptionalInclusive upper bound, Unix milliseconds.
aggoptionalavg · min · max · count. Presence switches the response to a scalar aggregate.

Without agg the response carries rawpoints. With agg it carries a single aggregation object instead —points is omitted entirely.

Request
# raw points in a 60-second window
curl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&from=1732250000000&to=1732250060000'
200 OK
{
  "series": [
    {
      "measurement": "temperature",
      "tags": [["device", "sensor-01"], ["room", "kitchen"]],
      "field": "value",
      "points": [
        { "timestamp": 1732250000000, "value": 28.5 },
        { "timestamp": 1732250010000, "value": 28.7 }
      ]
    }
  ]
}
Request + response
# same window, aggregated
curl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&tag=device=sensor-01&agg=avg'

{
  "series": [
    {
      "measurement": "temperature",
      "tags": [["device", "sensor-01"], ["room", "kitchen"]],
      "field": "value",
      "aggregation": { "function": "avg", "value": 28.6, "point_count": 2 }
    }
  ]
}

from greater than to is a400. Series with no points in range are omitted from the response rather than returned empty. Blocks whose[min_ts, max_ts] does not overlap the range are skipped without touching the disk.

GET /api/v1/metrics

Every known series, with point counts and time bounds. This is what the dashboard's explorer is built from.

200 OK
{
  "series": [
    {
      "measurement": "temperature",
      "tags": [["device", "sensor-01"], ["room", "kitchen"]],
      "field": "value",
      "points": 2,
      "min_ts": 1732250000000,
      "max_ts": 1732250010000
    }
  ]
}

GET /api/v1/stats

Live engine statistics. The dashboard polls this every two seconds.

200 OK
{
  "ingested_points": 1000000,
  "stored_points": 1000000,
  "series_count": 24,
  "memtable_points": 0,
  "flushed_points": 1000000,
  "raw_bytes": 16000000,
  "compressed_bytes": 1533456,
  "compression_ratio": 10.4352,
  "segment_count": 1,
  "wal_bytes": 0,
  "disk_usage_bytes": 1625984
}
FieldMeaning
ingested_pointsLifetime acknowledged points, restored from disk on startup.
stored_pointsPoints currently retained: segments + MemTable.
series_countDistinct (measurement, tags, field) keys known to the engine.
memtable_pointsPoints buffered in RAM, not yet flushed.
flushed_pointsPoints living in immutable compressed segments.
raw_bytesstored_points × 16 — the uncompressed cost.
compressed_bytesGorilla payload bytes across all segments.
compression_ratioflushed raw ÷ compressed; 0.0 before the first flush.
segment_countOpen segment files.
wal_bytesCurrent write-ahead log length.
disk_usage_bytesSegments + WAL on disk.

GET /api/v1/health

Constant-time liveness probe. It does not touch the engine or the disk.

200 OK
{"status": "ok", "service": "beacon"}

GET /api/v1/stream

Server-Sent Events. Each ingested line becomes a pointevent carrying the parsed line as JSON: measurement,tags as [[key, value]],fields as [[key, number]], and timestamp in Unix milliseconds.

Wire format
GET /api/v1/stream
Accept: text/event-stream

event: point
data: {"measurement":"temperature","tags":[["device","sim-01"],["location","edge"]],"fields":[["value",23.9]],"timestamp":1732250000123}

event: point
data: {"measurement":"humidity","tags":[["device","sim-01"],["location","edge"]],"fields":[["value",55.0]],"timestamp":1732250000123}

event: lagged
data: {"dropped":37}

: keep-alive

Frames are fanned out through a bounded broadcast channel of 1024 entries. A subscriber that cannot keep up does not stall ingestion: it receives alagged event with the number of dropped frames and should re-read the range it cares about. A keep-alive comment is emitted every 15 seconds so proxies hold the connection open.

Browser
const es = new EventSource("/api/v1/stream");

es.addEventListener("point", (e) => {
  const { measurement, tags, fields, timestamp } = JSON.parse(e.data);
  console.log(measurement, fields, new Date(timestamp).toISOString());
});

// The browser was too slow: buffered frames were dropped. Re-sync history.
es.addEventListener("lagged", async (e) => {
  const { dropped } = JSON.parse(e.data);
  console.warn(dropped, "frames dropped — reloading range");
  await reloadHistory();
});

The dashboard uses exactly this contract: pointframes append to the live canvas, lagged triggers a history re-sync.

Gorilla compression

Timestamps are encoded with delta-of-delta: the second difference between consecutive samples is bucket-coded into a variable number of bits, which collapses a fixed-cadence IoT stream to a handful of bits per point. Values are encoded with XOR: XOR the f64 against its predecessor, then store the leading and trailing zero counts plus only the significant bits.

All bit packing runs through a hand-written BitWriter /BitReader — no compression crate, no unsafe. A raw point costs 16 bytes (8-byte timestamp + 8-byte f64); the benchmark measures 1.53 B/point on realistic sensor data, a 10.44× reduction.

WAL & recovery

The write-ahead log is append-only with CRC framing. Records are written and fsynced before the HTTP response acknowledges the write.

Record layout
record := payload_len: u32le | crc32: u32le | payload
          ^ CRC32 covers the payload only; a bad or short record ends replay.

On startup the log is scanned from offset zero. The first short header, short payload, CRC mismatch or implausible length (over 64 MiB) ends the replay; the file is truncated back to the last valid boundary. A crash can therefore lose only an unacknowledged tail — never an acknowledged write.

Recovery sequence
1. open WAL read/write, scan records from offset 0
2. stop at the first short header, short payload, or CRC mismatch
3. truncate the file back to the last valid record boundary
4. replay every valid record into the MemTable
5. scan segments, rebuild SeriesKey -> Vec<BlockMeta>, truncate a torn tail block

Flush ordering matters: compress → append blocks → fsync the segment → truncate the WAL. A crash in the middle replays points that are already durable; queries and the next flush deduplicate by timestamp, so recovery is idempotent.

Segments & index

Block layout
block := magic: u32le ("BEAC") | header_len: u32le | header_json | payload | crc32: u32le
header_json := { series_key, min_ts, max_ts, point_count, payload_len }

The JSON header carries the series key and the block's time range, so the in-memory index (SeriesKey → Vec<BlockMeta>) can skip blocks without reading them. The index is rebuilt on startup by scanning every segment; a torn tail block from a crash mid-flush is detected by CRC and truncated. The active segment rolls to a new file at 64 MiB.

Benchmark

beacon bench simulates temperature, humidity and CPU streams with delivery jitter, pushes them through the real engine (WAL fsync + Gorilla flush) and reports throughput, compression and query latency percentiles. The PRNG is deterministic, so runs are comparable.

Shell
beacon bench --points 1000000

# keep the generated data for inspection
beacon bench --points 1000000 --data-dir ./bench-data

Reference machine, three runs: 552K / 608K / 615Kpoints/sec, 10.44× compression, avg 52.1 µs (p50 26 µs, p99 71 µs) over 200 random 1%-window aggregations.

Verification

The checks this project holds itself to:

Shell
cargo build --release
cargo test                      # 71 tests
cargo clippy --all-targets -- -D warnings
cargo run --release -- bench --points 500000
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\smoke-test.ps1

Current state: 71/71 tests passing, 0 clippy warnings with-D warnings, 0 unsafe blocks insrc/.

End of reference · v0.1.0Source on GitHub