SenFlat v2

SenFlat v2 is designed for gateways: one message carries many sources (id) and many time points, including history data sent after a connection recovers. Every point carries the time it was created on the source device, not the time the gateway sent it. A gateway only collects and forwards, and must not change a point’s time.

The parsed result is the same as SenML and SenFlat v1, so rules, dashboards, alarms, and storage are unchanged. SenFlat v1 still applies.

Shape

{
  "f": "flat",
  "v": 2,
  "q": 1791755578291,
  "h": 0,
  "tx": 1790003600000,
  "s": [
    {
      "id": "001a2b3c4d5e",
      "t0": 1789999000000,
      "ts": 1,
      "m": {
        "temp_inlet":      [[0, 22.4], [10000, 22.5], [20000, 22.6]],
        "voltage":         [[3000, 230.1], [13000, 230.0]],
        "door_front_open": [[18000, true]],
        "state":           "online"
      }
    },
    {
      "id": "001a2b3c4d5f",
      "t0": 1789999400000,
      "m": { "temp_inlet": [[0, 23.1], [10000, 23.2]] }
    }
  ]
}

Field	Required	Description
`f`	yes	Exactly `"flat"`.
`v`	yes	`2`.
`q`	yes	A message sequence number per publisher (gateway), used to drop resends.
`h`	yes	`1` = history (resent after a connection recovers), `0` = a current reading.
`tx`	no	The time the gateway sent it (epoch milliseconds UTC). Only recorded.
`s`	yes	The list of sources, at least one.
`s[].id`	yes	The source identity. Metrics are stored as `<id>:<name>`.
`s[].t0`	yes	The base time of this source, epoch milliseconds UTC, from the source device’s clock.
`s[].ts`	no	`1` = the device clock is synchronized (NTP); `0` = it is not. When absent, it is assumed to be `1`.
`s[].m`	yes	The metrics. A scalar value = one point at `t0`; or an array of points `[[dt, value], ...]`.

dt is an integer number of milliseconds from t0 (it may be negative). A point’s time = t0 + dt.
Points do not have to line up across metrics. A failed reading is not written (no null).
The order of points within a series is free.
Values: finite numbers, booleans, or strings (truncated to 500 characters).
v2 is a single object, not an array of objects. Without a valid q or h, the payload is not a metric.

Size limit

A payload is at most 512 KiB per message (counted per byte), with a safety cap of 50,000 points. A larger payload is rejected on every path; it is never truncated and then parsed. A gateway must split large history into several messages, each with its own q.

Trusted time

A source is considered to have an untrusted time when ts is 0, or any point’s t0 + dt falls outside −1 year to +1 day of the server’s receive time.

All points of that source are still stored, with an estimated time from the receive time and marked untrusted. The source’s last point is anchored to the receive time and the spacing between points is preserved:

time = receive_time − (dt_max − dt)

The mark appears in Messages as the Estimated time badge and in the time_trusted column of CSV downloads. Other sources in the same message are unaffected.

Last Seen and latest values

Last Seen and the online status of the sending device (the gateway) use the server receive time.
The latest value per metric uses the highest data time: history that arrives later does not overwrite a newer value.
For a child source on a history message (h=1), last seen uses the newest data time, not the receive time, so old history does not make a sensor that has long been silent look newly active.

Duplicate removal (`q`)

The rules engine remembers q per (rule, publisher, the set of source ids) for 72 hours (the administrator can change it). A message with the same q and set of sources from the same publisher is skipped before the rule runs, so every output (storage, alarms, webhooks, e-mail) does not do the work twice.

Requirements for the sender:

q increases for every new message from the same sender, and must not go back to an old value, including after a restart. Keep the counter in persistent storage, or start from the time (milliseconds).
A message that is resent must use the same q as the original.
A node behind a gateway may have its own q counters; the same q from different sources is not a duplicate.
When Redis is unavailable, messages are still processed (a duplicate is better than lost data).

`tx` and the gateway clock

tx is recorded and is not used to correct data times. When tx deviates by more than 5 seconds from server time, the server logs a warning. The payload as is appears in the Last messages tab.

Time synchronization

For ts: 1 to mean something, device clocks must be accurate:

A gateway should have its own NTP server.
Every source device synchronizes its clock to the gateway, or to Celestian’s NTP server.
When neither is available, use public NTP pools (for example 0.id.pool.ntp.org).
While the clock is not yet synchronized, the device sends ts: 0.

Topology: through a gateway or directly

	Through a gateway	Directly
The authenticated publisher	The gateway	The device itself
`s[].id`	The node’s MAC or serial	The device’s own MAC or serial
The `q` counter	Per node, or per aggregation component on the gateway	Per device (persistent)
History when Celestian is unreachable	A queue on the gateway	None, unless the device has its own buffer

Using it in rules

The SenFlat v2 splitter → rows (rule nodes) splits a message into wide rows without Lua.
Alarm and Device alarm have a history window and a retroactive option (Alarms).
A complete example of storing to PostgreSQL: Rule to PostgreSQL.