# Events and telemetry streams

> How to follow rt-control's event log over polling or Server-Sent Events, and how to read the full-rate binary telemetry stream, with cursors, loss reporting and the exact record layout.

URL: https://advancedmetalresearch.com/docs/apis/rt-control-events-telemetry
Section: RosieOS docs / APIs
Last updated: 2026-10-10

rt-control publishes two observation streams. Neither needs a lease, so any client that can reach the socket can watch the cell while another client controls it.

- **Events** are a JSON log of state changes: grants, enable and arm, handle and execution transitions, jog sessions, faults, Home epochs and restarts. They are good for reacting to changes.
- **Telemetry** is binary: one record per control cycle with positions, targets, drive state and fault masks. It is good for plotting, logging and replay.

Both use the same cursor model: you pass `after`, the last sequence you handled, and the server tells you exactly what you missed.

## Follow the event stream

**curl**

```bash
# Server-Sent Events from the beginning of the retained log; Ctrl-C to stop.
curl -sN --unix-socket /run/rosie-rt-core/control.sock \
  'http://localhost/v1/events/stream?after=0'
```
**Go**

```go
c, err := control.Dial("/run/rosie-rt-core/control.sock")
if err != nil {
	log.Fatal(err)
}
defer c.Close()
var cursor uint64
err = c.StreamEvents(ctx, cursor, func(e control.Event) error {
	if e.Type == "events_dropped" {
		// History was lost: re-read Status before trusting your local view.
		log.Printf("lost events %d..%d", e.Dropped.FirstLostSequence, e.Dropped.LastLostSequence)
	}
	log.Printf("%d %s", e.Sequence, e.Type)
	cursor = e.Sequence
	return nil
})
```

SSE output:

```text
: rt-core events

id: 1
event: adapter_incarnation_changed
data: {"sequence":1,"time_ns":81230000000000,"type":"adapter_incarnation_changed","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","incarnation":{"previous":"","current":"a41c…"}}

id: 2
event: grant_acquired
data: {"sequence":2,"time_ns":81234567890123,"type":"grant_acquired","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","grant":{"generation":1,"stopping":false}}
```

## Events

**Endpoint: `GET /v1/events?after=<sequence>`**

One batch of up to 64 events after the cursor, in the normal `Response` envelope with an `EventBatch` in `data`.

**Endpoint: `SSE /v1/events/stream?after=<sequence>`**

The same events as Server-Sent Events. Each message is `id` (the sequence), `event` (the type) and `data` (one `Event` as JSON, with no envelope).

### Cursors and loss

- `after` is the sequence of the last event you handled. Omit it (or send 0) to start at the oldest retained event. It must be one unsigned decimal number, otherwise you get 409 `invalid_event_cursor`. A cursor newer than the log gets 409 `event_cursor_ahead`.
- Sequences increase by one within an `adapter_incarnation`. A poll returns `next_sequence` (use it as your next `after`) and `latest_sequence`.
- The log keeps the last **256** events. If your cursor is older than that, the first record you receive is a synthetic `events_dropped` event whose `first_lost_sequence`..`last_lost_sequence` range (inclusive) is what you missed. Its own `sequence` equals `last_lost_sequence`, so it is also your resume cursor. The retained events that follow keep their original sequence numbers.
- The SSE handler does not read `Last-Event-ID`. To resume, reconnect with `?after=` set to the last `id` you handled.
- When rt-control restarts, the sequence starts again under a new `adapter_incarnation`. Discard your cursor, read Status and start from 0.

Events are observations, not a lossless trace. `time_ns` is the core's publication time, and native snapshots can merge transitions that happen between publications. After any loss or incarnation change, reconcile from `GET /v1/status` rather than replaying motion.

### Event types

Every event has `sequence`, `time_ns`, `type`, `daemon_incarnation` and `adapter_incarnation`, plus one payload field for its type:

| Type | Payload field | Payload type | Emitted when |
|---|---|---|---|
| `grant_acquired` | `grant` | [`GrantEvent`](/docs/apis/rt-control-http#type-grantevent) | `acquire` succeeded. |
| `grant_renewed` | `grant` | [`GrantEvent`](/docs/apis/rt-control-http#type-grantevent) | `renew` succeeded. `stopping: true` while a Stop drains. |
| `grant_released` | `grant` | [`GrantEvent`](/docs/apis/rt-control-http#type-grantevent) | `release`. |
| `grant_expired` | `grant` | [`GrantEvent`](/docs/apis/rt-control-http#type-grantevent) | The lease ran out. |
| `grant_revoked` | `grant` | [`GrantEvent`](/docs/apis/rt-control-http#type-grantevent) | `stop`, or authority lost to a native or transport failure. |
| `enable_changed` | `enable` | [`EnableEvent`](/docs/apis/rt-control-http#type-enableevent) | `axis_enable_mask` changed. |
| `arm_changed` | `enable` | [`EnableEvent`](/docs/apis/rt-control-http#type-enableevent) | `armed` changed. |
| `handle_transition` | `handle` | [`HandleTransitionEvent`](/docs/apis/rt-control-http#type-handletransitionevent) | A trajectory handle changed state (`from` → `to`). |
| `execution_started` | `execution` | [`ExecutionEvent`](/docs/apis/rt-control-http#type-executionevent) | A handle started. |
| `execution_completed` | `execution` | [`ExecutionEvent`](/docs/apis/rt-control-http#type-executionevent) | A started handle was consumed: it completed, or a later execution replaced it. |
| `execution_aborted` | `execution` | [`ExecutionEvent`](/docs/apis/rt-control-http#type-executionevent) | A started handle was retired with no fault bits set. |
| `execution_faulted` | `execution` | [`ExecutionEvent`](/docs/apis/rt-control-http#type-executionevent) | A started handle was retired with fault bits set; `fault_bits` and `recovery` say which. |
| `jog_begin` | `jog` | [`JogEvent`](/docs/apis/rt-control-http#type-jogevent) | A jog session opened. |
| `jog_end` | `jog` | [`JogEvent`](/docs/apis/rt-control-http#type-jogevent) | A jog session closed for a reason other than expiry. |
| `jog_expired` | `jog` | [`JogEvent`](/docs/apis/rt-control-http#type-jogevent) | Jog input expired. |
| `jog_ramping` | `jog` | [`JogEvent`](/docs/apis/rt-control-http#type-jogevent) | The jog is ramping down. |
| `jog_limited` | `jog` | [`JogEvent`](/docs/apis/rt-control-http#type-jogevent) | The jog is being slowed at a position limit (an accepted state, not a failure). |
| `fault_latched` | `fault` | [`FaultEvent`](/docs/apis/rt-control-http#type-faultevent) | An execution fault bit was set. |
| `fault_cleared` | `fault` | [`FaultEvent`](/docs/apis/rt-control-http#type-faultevent) | An execution fault bit was cleared. |
| `home_epoch_changed` | `epoch` | [`EpochEvent`](/docs/apis/rt-control-http#type-epochevent) | Home evidence changed. |
| `daemon_incarnation_changed` | `incarnation` | [`IncarnationEvent`](/docs/apis/rt-control-http#type-incarnationevent) | The core process changed (and once at adapter start). |
| `adapter_incarnation_changed` | `incarnation` | [`IncarnationEvent`](/docs/apis/rt-control-http#type-incarnationevent) | Emitted once when rt-control starts. |
| `publisher_overflow` | `overflow` | [`PublisherOverflowEvent`](/docs/apis/rt-control-http#type-publisheroverflowevent) | Native observations were dropped before they reached the event log. |
| `events_dropped` | `events_dropped` | [`EventsDropped`](/docs/apis/rt-control-http#type-eventsdropped) | Your cursor fell behind the 256-event ring; the lost range is inclusive. |
| `telemetry_mark` | `mark` | [`TelemetryMarkEvent`](/docs/apis/rt-control-http#type-telemetrymarkevent) | `mark_telemetry` was accepted. |

> [!NOTE] In `FaultEvent`, `bit` is the bit's **value** (for example 32 for bit 5), not its index. In `RecoveryStatus.faults[]`, `bit` is the index. The bits are listed in [Execution fault bits](https://advancedmetalresearch.com/docs/reference/error-codes#fault-bits).

Events never contain the session token. The field tables for every payload type are in the [types section](https://advancedmetalresearch.com/docs/apis/rt-control-http#types-events) of the HTTP reference.

## Telemetry

**Endpoint: `GET /v1/telemetry?after=<sequence>`**

One binary batch after the cursor. Sends gzip when you ask for it with `Accept-Encoding: gzip`.

**Endpoint: `GET /v1/telemetry/stream?after=<sequence>`**

Concatenated binary batches over chunked HTTP, one complete batch per flush, about every 20 ms.

Telemetry is `application/octet-stream`, never JSON or base64. A batch is a 312-byte `TelemetryBatchHeaderV1` followed by exactly `record_count` records of 5392 bytes (`CycleCaptureRecordV2`), all little-endian. A batch is at most 312 + 4096 × 5392 = 22085944 bytes.

Go: follow telemetry:

```go
err := c.TelemetryStream(ctx, 0, func(b control.TelemetryBatch) error {
	if b.Header.Dropped != 0 {
		log.Printf("lost %d records", b.Header.Dropped)
	}
	for _, r := range b.Records {
		j1 := r.Ax[0]
		_ = j1.Position // drive counts; convert with Describe counts_per_unit and sign
	}
	return nil
})
```

### Cursors and loss

- `after` is the last record sequence you consumed. 0 (the default) starts at sequence 1, and history that the ring has already overwritten is reported in `dropped`.
- A non-empty batch starts at `after + dropped + 1` and its sequences are contiguous. An empty batch has `first_sequence` 0 and `last_sequence` equal to `after + dropped`. Always resume from `last_sequence`, after you have handled the batch and its loss.
- Errors come back as JSON with HTTP 409: `invalid_telemetry_cursor`, `telemetry_cursor_ahead`, `telemetry_unavailable`, or `telemetry_busy` with `Retry-After: 1` (retry the same cursor). On an established stream, a busy read becomes an empty batch with the cursor unchanged.
- A stream closes when the core disconnects or a write blocks for 5 s. HTTP proxies may re-chunk the stream, so find batch boundaries from the header, not from chunks.
- If `adapter_incarnation` or `daemon_incarnation` changes, stop. Re-read Describe and Status before you reset the cursor.

Telemetry needs no session, but on the remote listener it still needs a valid client certificate.

### Batch header

`TelemetryBatchHeaderV1`, 312 bytes. Check `magic`, `version`, `header_bytes`, `record_bytes`, the layout digest, `axis_count` (1..16) and `record_count` (≤ 4096) before reading the body.

| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
| `magic` | 0 | 4 | `u32` | `0x31425452` (`RTB1`). |
| `version` | 4 | 2 | `u16` | 1. |
| `header_bytes` | 6 | 2 | `u16` | 312. |
| `adapter_incarnation` | 8 | 32 | `u8[32]` | Lowercase hex, rt-control process identity. |
| `machine_sha256` | 40 | 64 | `u8[64]` | Lowercase hex machine digest. |
| `deployment_sha256` | 104 | 64 | `u8[64]` | Lowercase hex deployment digest. |
| `record_layout_digest` | 168 | 64 | `u8[64]` | Must equal `CycleCaptureRecordV2LayoutDigest`. |
| `cycle_period_ns` | 232 | 8 | `u64` | Cycle period, ns. |
| `first_sequence` | 240 | 8 | `u64` | First record's sequence; 0 when the batch is empty. |
| `last_sequence` | 248 | 8 | `u64` | Last record's sequence, or `after + dropped` when empty. Your next cursor. |
| `dropped` | 256 | 8 | `u64` | Records lost after your cursor and before this batch. |
| `axis_count` | 264 | 4 | `u32` | Active axes, 1..16. |
| `record_count` | 268 | 4 | `u32` | Records in this batch, at most min(ring capacity, 4096). |
| `record_bytes` | 272 | 4 | `u32` | 5392. |
| `reserved` | 276 | 4 | `u32` | Reserved, zero. |
| `daemon_incarnation` | 280 | 32 | `u8[32]` | Lowercase hex, core process identity; owns the sequence numbers. |

The current record layout digest is `1173439681672348f35f9652f2a4821d251638c70b60362ff59b37032a794b6b`. A reader that sees a different digest must refuse the batch: the Go, C++ and TypeScript decoders all raise `TelemetryLayoutMismatchError` with both digests.

### Cycle record

`CycleCaptureRecordV2`, 5392 bytes: 52 group fields followed by `ax`, 16 per-axis entries. Positions and targets are in drive counts, not radians; convert them with the axis's `counts_per_unit` and `sign` from Describe.

| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
| `t_ns` | 0 | 8 | `u64` | Cycle time, host monotonic ns. |
| `seq` | 8 | 8 | `u64` | Record sequence. |
| `cycle` | 16 | 8 | `u64` | Cycle counter. |
| `work_ns` | 24 | 8 | `u64` | Cycle work time, ns. |
| `axes` | 32 | 4 | `u32` | Active axis count. |
| `reserved` | 36 | 4 | `u32` | Reserved, zero. |
| `cycle_jitter_ns` | 40 | 8 | `i64` | Wake time minus scheduled time, ns (signed). |
| `active_traj_id` | 48 | 8 | `u64` |  |
| `active_command_seq` | 56 | 8 | `u64` |  |
| `jog_generation` | 64 | 8 | `u64` |  |
| `app_grant_generation` | 72 | 8 | `u64` |  |
| `control_generation` | 80 | 8 | `u64` |  |
| `configuration_epoch` | 88 | 8 | `u64` |  |
| `home_epoch` | 96 | 8 | `u64` |  |
| `grant_deadline_host_ns` | 104 | 8 | `u64` |  |
| `jog_input_deadline_host_ns` | 112 | 8 | `u64` |  |
| `jog_source_sequence` | 120 | 8 | `u64` |  |
| `bus_submission_return_code` | 128 | 8 | `i64` |  |
| `armed` | 136 | 4 | `u32` | 1 when armed. |
| `axis_enable_mask` | 140 | 4 | `u32` | Enabled axes. |
| `active_mode` | 144 | 4 | `u32` | 0 idle, 2 trajectory, 3 jog. |
| `state` | 148 | 4 | `u32` |  |
| `current_point_index` | 152 | 4 | `u32` |  |
| `queue_depth` | 156 | 4 | `u32` |  |
| `last_event_code` | 160 | 4 | `u32` |  |
| `underrun_count` | 164 | 4 | `u32` |  |
| `stale_command_flag` | 168 | 4 | `u32` |  |
| `motion_done` | 172 | 4 | `u32` |  |
| `capability_flags` | 176 | 4 | `u32` |  |
| `jog_session_open` | 180 | 4 | `u32` | 1 while jog input is accepted. |
| `jog_has_input` | 184 | 4 | `u32` |  |
| `jog_axis_mask` | 188 | 4 | `u32` |  |
| `jog_ramping` | 192 | 4 | `u32` |  |
| `grant_active` | 196 | 4 | `u32` | 1 while the effective grant is active. |
| `grant_axis_mask` | 200 | 4 | `u32` |  |
| `safety_fault_mask` | 204 | 4 | `u32` | Non-zero blocks motion. |
| `execution_fault_reasons` | 208 | 4 | `u32` | Latched fault bits. |
| `submitted_axis_mask` | 212 | 4 | `u32` |  |
| `bus_submission_operation` | 216 | 4 | `u32` |  |
| `pdo_fresh_axis_mask` | 220 | 4 | `u32` | Axes with fresh feedback this cycle. |
| `config_verified_mask` | 224 | 4 | `u32` |  |
| `home_valid_mask` | 228 | 4 | `u32` | Axes with valid Home. |
| `service_mode_axis_mask` | 232 | 4 | `u32` |  |
| `wkc_actual` | 236 | 4 | `u32` |  |
| `wkc_expected` | 240 | 4 | `u32` |  |
| `master_state` | 244 | 4 | `u32` |  |
| `io_configured` | 248 | 4 | `u32` |  |
| `io_input_word` | 252 | 4 | `u32` |  |
| `io_output_word` | 256 | 4 | `u32` |  |
| `io_input_valid` | 260 | 4 | `u32` |  |
| `io_armed` | 264 | 4 | `u32` |  |
| `io_reason` | 268 | 4 | `u32` |  |
| `ax` | 272 | 5120 | `CycleCaptureAxisV2[16]` | One entry per axis; unused axes are zero. |

### Axis entry

`CycleCaptureAxisV2`, 320 bytes, 16 per record. Check `pdo_fresh` and each field's validity flag before you use a value; invalid values are not meaningful, and `di_valid = 0` means unavailable, not "inputs off".

| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
| `position` | 0 | 4 | `i32` | Feedback position, drive counts. |
| `target` | 4 | 4 | `i32` | Commanded target, drive counts. |
| `statusword` | 8 | 2 | `u16` | CiA402 statusword. |
| `error_code` | 10 | 2 | `u16` | CiA402 error code. |
| `manufacturer_error_code` | 12 | 4 | `u32` |  |
| `absolute_value` | 16 | 64 | `i32[16]` |  |
| `absolute_valid` | 80 | 16 | `u8[16]` |  |
| `velocity_actual_counts_per_s` | 96 | 4 | `i32` | Actual velocity, counts/s; valid only with `velocity_actual_valid`. |
| `following_error_counts` | 100 | 4 | `i32` | Following error, counts; valid only with `following_error_valid`. |
| `velocity_actual_valid` | 104 | 4 | `u32` |  |
| `following_error_valid` | 108 | 4 | `u32` |  |
| `di_bits` | 112 | 4 | `u32` |  |
| `di_valid` | 116 | 4 | `u32` |  |
| `external_enable_active` | 120 | 4 | `u32` |  |
| `external_enable_valid` | 124 | 4 | `u32` |  |
| `coordinate_counts` | 128 | 8 | `i64` |  |
| `coordinate_epoch` | 136 | 8 | `u64` |  |
| `pdo_observed_time_ns` | 144 | 8 | `u64` |  |
| `collision_peak_following_error_counts` | 152 | 8 | `u64` |  |
| `collision_trip_count` | 160 | 8 | `u64` |  |
| `collision_samples` | 168 | 8 | `u64` |  |
| `collision_last_trip_peak_following_error_counts` | 176 | 8 | `u64` |  |
| `absolute_source_counts` | 184 | 4 | `i32` |  |
| `coordinate_reason` | 188 | 4 | `u32` |  |
| `target_velocity_counts_per_s` | 192 | 4 | `i32` |  |
| `collision_armed` | 196 | 4 | `u32` |  |
| `collision_torque_cycles` | 200 | 4 | `u32` |  |
| `collision_following_error_cycles` | 204 | 4 | `u32` |  |
| `collision_peak_torque_raw` | 208 | 4 | `u32` |  |
| `collision_last_trip_quantity` | 212 | 4 | `u32` |  |
| `collision_trip_sustained_cycles` | 216 | 4 | `u32` |  |
| `collision_last_trip_peak_torque_raw` | 220 | 4 | `u32` |  |
| `brake_state` | 224 | 4 | `u32` |  |
| `native_home_position_offset` | 228 | 4 | `i32` |  |
| `native_home_last_abort_code` | 232 | 4 | `u32` |  |
| `ext_position_error_counts` | 236 | 4 | `i32` |  |
| `ext_multi_turn_lo` | 240 | 4 | `i32` |  |
| `ext_multi_turn_hi` | 244 | 4 | `i32` |  |
| `max_abs_velocity_actual_counts_per_s` | 248 | 4 | `u32` |  |
| `max_abs_following_error_counts` | 252 | 4 | `u32` |  |
| `torque_raw` | 256 | 2 | `i16` | Torque, signed per-mille of rated; valid only with `torque_valid`. |
| `ext_bus_voltage_raw` | 258 | 2 | `u16` |  |
| `ext_load_rate_raw` | 260 | 2 | `u16` |  |
| `ext_igbt_temp_raw` | 262 | 2 | `i16` |  |
| `ext_motor_temp_raw` | 264 | 2 | `i16` |  |
| `ext_drive_not_ready_bits` | 266 | 2 | `u16` |  |
| `ext_motor_not_rotating_code` | 268 | 2 | `u16` |  |
| `ext_valid_mask` | 270 | 2 | `u16` | Bits 0..8 qualify the `ext_*` fields; ignore any field whose bit is clear. |
| `mode_display` | 272 | 1 | `i8` | Drive mode (8 = CSP). |
| `ds402_state` | 273 | 1 | `u8` | DS402 state code (see the real-time core page). |
| `torque_valid` | 274 | 1 | `u8` |  |
| `coordinate_valid` | 275 | 1 | `u8` | 1 when the coordinate is trustworthy. |
| `coordinate_source_valid` | 276 | 1 | `u8` |  |
| `native_home_state` | 277 | 1 | `u8` |  |
| `slave_al_state` | 278 | 1 | `u8` |  |
| `slave_online` | 279 | 1 | `u8` |  |
| `slave_operational` | 280 | 1 | `u8` |  |
| `pdo_fresh` | 281 | 1 | `u8` | 1 when this axis's feedback is fresh. |
| `target_velocity_valid` | 282 | 1 | `u8` |  |
| `ext_valid` | 283 | 1 | `u8` |  |
| `calibration_valid` | 284 | 1 | `u8` |  |
| `position_state` | 285 | 1 | `u8` |  |
| `audit_reason` | 286 | 2 | `u16` |  |
| `audit_last_verified_ns` | 288 | 8 | `u64` |  |
| `audit_started_ns` | 296 | 8 | `u64` |  |
| `audit_completed_ns` | 304 | 8 | `u64` |  |
| `audit_coherence_error_counts` | 312 | 8 | `u64` |  |

Decoders for these layouts are generated from `protocol/control.json`: Go in `rosieos/rt-core/ipcclient` (used by `Client.TelemetryBatches` and `TelemetryStream`), C++ in `protocol_generated.hpp` (used by `RtControlClient::telemetry`), and TypeScript in [`rt_protocol_generated.ts`](/docs/apis/typescript-types#telemetry-decoders).

## Resources

Describe's `robot.resources` lists the compiled robot description files (manifest, URDF, meshes, calibration) with their SHA-256 digests. Fetch one with [`GET /v1/resources/<sha256>`](/docs/apis/rt-control-http#resource). The bytes are immutable, and an unknown digest returns 404 `resource_unknown`.

## Related pages

- [rt-control HTTP API](https://advancedmetalresearch.com/docs/apis/rt-control-http): `subscribe_events`, `telemetry`, `mark_telemetry` and `resource`.
- [The real-time core](https://advancedmetalresearch.com/docs/concepts/real-time-core#telemetry): where the telemetry ring comes from.
- [TypeScript contracts](https://advancedmetalresearch.com/docs/apis/typescript-types): decoding telemetry in the browser or Node.

## Sources

Written from these files in the RosieOS repository (https://github.com/advanced-metal-research/RosieOS):

- `rt-core/adapters/rosie/control/events.go:17-421`
- `rt-core/adapters/rosie/control/telemetry_publication.go:18-216`
- `rt-core/adapters/rosie/control/remote_listener.go:88-104`
- `rt-core/protocol/control.json (native_records TelemetryBatchHeaderV1, CycleCaptureRecordV2, CycleCaptureAxisV2)`
- `rt-core/protocol/application-v1.schema.json (rules.events`
- `rules.telemetry, types Event, EventBatch)`
- `rt-core/clients/ts/rt_protocol_generated.ts:3`
- `rt-core/sdk/control/events.go:19-80`
- `rt-core/sdk/control/telemetry.go:52-121`
- `rt-core/ipcclient/telemetry_batch.go:18-21`