Events and telemetry streams
On this page
rt-control publishes two observation streams. Neither needs a lease, so any client that can reach the socket can watch the cell while another client controls it.
- Events are a JSON log of state changes: grants, enable and arm, handle and execution transitions, jog sessions, faults, Home epochs and restarts. They are good for reacting to changes.
- Telemetry is binary: one record per control cycle with positions, targets, drive state and fault masks. It is good for plotting, logging and replay.
Both use the same cursor model: you pass after, the last sequence you handled, and the server tells you exactly what you missed.
Follow the event stream#
# Server-Sent Events from the beginning of the retained log; Ctrl-C to stop.
curl -sN --unix-socket /run/rosie-rt-core/control.sock \
'http://localhost/v1/events/stream?after=0'c, err := control.Dial("/run/rosie-rt-core/control.sock")
if err != nil {
log.Fatal(err)
}
defer c.Close()
var cursor uint64
err = c.StreamEvents(ctx, cursor, func(e control.Event) error {
if e.Type == "events_dropped" {
// History was lost: re-read Status before trusting your local view.
log.Printf("lost events %d..%d", e.Dropped.FirstLostSequence, e.Dropped.LastLostSequence)
}
log.Printf("%d %s", e.Sequence, e.Type)
cursor = e.Sequence
return nil
}): rt-core events
id: 1
event: adapter_incarnation_changed
data: {"sequence":1,"time_ns":81230000000000,"type":"adapter_incarnation_changed","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","incarnation":{"previous":"","current":"a41c…"}}
id: 2
event: grant_acquired
data: {"sequence":2,"time_ns":81234567890123,"type":"grant_acquired","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","grant":{"generation":1,"stopping":false}}Events#
/v1/events?after=<sequence> One batch of up to 64 events after the cursor, in the normal Response envelope with an EventBatch in data.
/v1/events/stream?after=<sequence> The same events as Server-Sent Events. Each message is id (the sequence), event (the type) and data (one Event as JSON, with no envelope).
Cursors and loss#
afteris the sequence of the last event you handled. Omit it (or send 0) to start at the oldest retained event. It must be one unsigned decimal number, otherwise you get 409invalid_event_cursor. A cursor newer than the log gets 409event_cursor_ahead.- Sequences increase by one within an
adapter_incarnation. A poll returnsnext_sequence(use it as your nextafter) andlatest_sequence. - The log keeps the last 256 events. If your cursor is older than that, the first record you receive is a synthetic
events_droppedevent whosefirst_lost_sequence..last_lost_sequencerange (inclusive) is what you missed. Its ownsequenceequalslast_lost_sequence, so it is also your resume cursor. The retained events that follow keep their original sequence numbers. - The SSE handler does not read
Last-Event-ID. To resume, reconnect with?after=set to the lastidyou handled. - When rt-control restarts, the sequence starts again under a new
adapter_incarnation. Discard your cursor, read Status and start from 0.
Events are observations, not a lossless trace. time_ns is the core's publication time, and native snapshots can merge transitions that happen between publications. After any loss or incarnation change, reconcile from GET /v1/status rather than replaying motion.
Event types#
Every event has sequence, time_ns, type, daemon_incarnation and adapter_incarnation, plus one payload field for its type:
| Type | Payload field | Payload type | Emitted when |
|---|---|---|---|
grant_acquired | grant | GrantEvent | acquire succeeded. |
grant_renewed | grant | GrantEvent | renew succeeded. stopping: true while a Stop drains. |
grant_released | grant | GrantEvent | release. |
grant_expired | grant | GrantEvent | The lease ran out. |
grant_revoked | grant | GrantEvent | stop, or authority lost to a native or transport failure. |
enable_changed | enable | EnableEvent | axis_enable_mask changed. |
arm_changed | enable | EnableEvent | armed changed. |
handle_transition | handle | HandleTransitionEvent | A trajectory handle changed state (from → to). |
execution_started | execution | ExecutionEvent | A handle started. |
execution_completed | execution | ExecutionEvent | A started handle was consumed: it completed, or a later execution replaced it. |
execution_aborted | execution | ExecutionEvent | A started handle was retired with no fault bits set. |
execution_faulted | execution | ExecutionEvent | A started handle was retired with fault bits set; fault_bits and recovery say which. |
jog_begin | jog | JogEvent | A jog session opened. |
jog_end | jog | JogEvent | A jog session closed for a reason other than expiry. |
jog_expired | jog | JogEvent | Jog input expired. |
jog_ramping | jog | JogEvent | The jog is ramping down. |
jog_limited | jog | JogEvent | The jog is being slowed at a position limit (an accepted state, not a failure). |
fault_latched | fault | FaultEvent | An execution fault bit was set. |
fault_cleared | fault | FaultEvent | An execution fault bit was cleared. |
home_epoch_changed | epoch | EpochEvent | Home evidence changed. |
daemon_incarnation_changed | incarnation | IncarnationEvent | The core process changed (and once at adapter start). |
adapter_incarnation_changed | incarnation | IncarnationEvent | Emitted once when rt-control starts. |
publisher_overflow | overflow | PublisherOverflowEvent | Native observations were dropped before they reached the event log. |
events_dropped | events_dropped | EventsDropped | Your cursor fell behind the 256-event ring; the lost range is inclusive. |
telemetry_mark | mark | TelemetryMarkEvent | mark_telemetry was accepted. |
Note
In FaultEvent, bit is the bit's value (for example 32 for bit 5), not its index. In RecoveryStatus.faults[], bit is the index. The bits are listed in Execution fault bits.
Events never contain the session token. The field tables for every payload type are in the types section of the HTTP reference.
Telemetry#
/v1/telemetry?after=<sequence> One binary batch after the cursor. Sends gzip when you ask for it with Accept-Encoding: gzip.
/v1/telemetry/stream?after=<sequence> Concatenated binary batches over chunked HTTP, one complete batch per flush, about every 20 ms.
Telemetry is application/octet-stream, never JSON or base64. A batch is a 312-byte TelemetryBatchHeaderV1 followed by exactly record_count records of 5392 bytes (CycleCaptureRecordV2), all little-endian. A batch is at most 312 + 4096 × 5392 = 22085944 bytes.
err := c.TelemetryStream(ctx, 0, func(b control.TelemetryBatch) error {
if b.Header.Dropped != 0 {
log.Printf("lost %d records", b.Header.Dropped)
}
for _, r := range b.Records {
j1 := r.Ax[0]
_ = j1.Position // drive counts; convert with Describe counts_per_unit and sign
}
return nil
})Cursors and loss#
afteris the last record sequence you consumed. 0 (the default) starts at sequence 1, and history that the ring has already overwritten is reported indropped.- A non-empty batch starts at
after + dropped + 1and its sequences are contiguous. An empty batch hasfirst_sequence0 andlast_sequenceequal toafter + dropped. Always resume fromlast_sequence, after you have handled the batch and its loss. - Errors come back as JSON with HTTP 409:
invalid_telemetry_cursor,telemetry_cursor_ahead,telemetry_unavailable, ortelemetry_busywithRetry-After: 1(retry the same cursor). On an established stream, a busy read becomes an empty batch with the cursor unchanged. - A stream closes when the core disconnects or a write blocks for 5 s. HTTP proxies may re-chunk the stream, so find batch boundaries from the header, not from chunks.
- If
adapter_incarnationordaemon_incarnationchanges, stop. Re-read Describe and Status before you reset the cursor.
Telemetry needs no session, but on the remote listener it still needs a valid client certificate.
Batch header#
TelemetryBatchHeaderV1, 312 bytes. Check magic, version, header_bytes, record_bytes, the layout digest, axis_count (1..16) and record_count (≤ 4096) before reading the body.
| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
magic | 0 | 4 | u32 | 0x31425452 (RTB1). |
version | 4 | 2 | u16 | 1. |
header_bytes | 6 | 2 | u16 | 312. |
adapter_incarnation | 8 | 32 | u8[32] | Lowercase hex, rt-control process identity. |
machine_sha256 | 40 | 64 | u8[64] | Lowercase hex machine digest. |
deployment_sha256 | 104 | 64 | u8[64] | Lowercase hex deployment digest. |
record_layout_digest | 168 | 64 | u8[64] | Must equal CycleCaptureRecordV2LayoutDigest. |
cycle_period_ns | 232 | 8 | u64 | Cycle period, ns. |
first_sequence | 240 | 8 | u64 | First record's sequence; 0 when the batch is empty. |
last_sequence | 248 | 8 | u64 | Last record's sequence, or after + dropped when empty. Your next cursor. |
dropped | 256 | 8 | u64 | Records lost after your cursor and before this batch. |
axis_count | 264 | 4 | u32 | Active axes, 1..16. |
record_count | 268 | 4 | u32 | Records in this batch, at most min(ring capacity, 4096). |
record_bytes | 272 | 4 | u32 | 5392. |
reserved | 276 | 4 | u32 | Reserved, zero. |
daemon_incarnation | 280 | 32 | u8[32] | Lowercase hex, core process identity; owns the sequence numbers. |
The current record layout digest is 1173439681672348f35f9652f2a4821d251638c70b60362ff59b37032a794b6b. A reader that sees a different digest must refuse the batch: the Go, C++ and TypeScript decoders all raise TelemetryLayoutMismatchError with both digests.
Cycle record#
CycleCaptureRecordV2, 5392 bytes: 52 group fields followed by ax, 16 per-axis entries. Positions and targets are in drive counts, not radians; convert them with the axis's counts_per_unit and sign from Describe.
| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
t_ns | 0 | 8 | u64 | Cycle time, host monotonic ns. |
seq | 8 | 8 | u64 | Record sequence. |
cycle | 16 | 8 | u64 | Cycle counter. |
work_ns | 24 | 8 | u64 | Cycle work time, ns. |
axes | 32 | 4 | u32 | Active axis count. |
reserved | 36 | 4 | u32 | Reserved, zero. |
cycle_jitter_ns | 40 | 8 | i64 | Wake time minus scheduled time, ns (signed). |
active_traj_id | 48 | 8 | u64 | |
active_command_seq | 56 | 8 | u64 | |
jog_generation | 64 | 8 | u64 | |
app_grant_generation | 72 | 8 | u64 | |
control_generation | 80 | 8 | u64 | |
configuration_epoch | 88 | 8 | u64 | |
home_epoch | 96 | 8 | u64 | |
grant_deadline_host_ns | 104 | 8 | u64 | |
jog_input_deadline_host_ns | 112 | 8 | u64 | |
jog_source_sequence | 120 | 8 | u64 | |
bus_submission_return_code | 128 | 8 | i64 | |
armed | 136 | 4 | u32 | 1 when armed. |
axis_enable_mask | 140 | 4 | u32 | Enabled axes. |
active_mode | 144 | 4 | u32 | 0 idle, 2 trajectory, 3 jog. |
state | 148 | 4 | u32 | |
current_point_index | 152 | 4 | u32 | |
queue_depth | 156 | 4 | u32 | |
last_event_code | 160 | 4 | u32 | |
underrun_count | 164 | 4 | u32 | |
stale_command_flag | 168 | 4 | u32 | |
motion_done | 172 | 4 | u32 | |
capability_flags | 176 | 4 | u32 | |
jog_session_open | 180 | 4 | u32 | 1 while jog input is accepted. |
jog_has_input | 184 | 4 | u32 | |
jog_axis_mask | 188 | 4 | u32 | |
jog_ramping | 192 | 4 | u32 | |
grant_active | 196 | 4 | u32 | 1 while the effective grant is active. |
grant_axis_mask | 200 | 4 | u32 | |
safety_fault_mask | 204 | 4 | u32 | Non-zero blocks motion. |
execution_fault_reasons | 208 | 4 | u32 | Latched fault bits. |
submitted_axis_mask | 212 | 4 | u32 | |
bus_submission_operation | 216 | 4 | u32 | |
pdo_fresh_axis_mask | 220 | 4 | u32 | Axes with fresh feedback this cycle. |
config_verified_mask | 224 | 4 | u32 | |
home_valid_mask | 228 | 4 | u32 | Axes with valid Home. |
service_mode_axis_mask | 232 | 4 | u32 | |
wkc_actual | 236 | 4 | u32 | |
wkc_expected | 240 | 4 | u32 | |
master_state | 244 | 4 | u32 | |
io_configured | 248 | 4 | u32 | |
io_input_word | 252 | 4 | u32 | |
io_output_word | 256 | 4 | u32 | |
io_input_valid | 260 | 4 | u32 | |
io_armed | 264 | 4 | u32 | |
io_reason | 268 | 4 | u32 | |
ax | 272 | 5120 | CycleCaptureAxisV2[16] | One entry per axis; unused axes are zero. |
Axis entry#
CycleCaptureAxisV2, 320 bytes, 16 per record. Check pdo_fresh and each field's validity flag before you use a value; invalid values are not meaningful, and di_valid = 0 means unavailable, not "inputs off".
| Field | Offset | Size | Type | Notes |
|---|---|---|---|---|
position | 0 | 4 | i32 | Feedback position, drive counts. |
target | 4 | 4 | i32 | Commanded target, drive counts. |
statusword | 8 | 2 | u16 | CiA402 statusword. |
error_code | 10 | 2 | u16 | CiA402 error code. |
manufacturer_error_code | 12 | 4 | u32 | |
absolute_value | 16 | 64 | i32[16] | |
absolute_valid | 80 | 16 | u8[16] | |
velocity_actual_counts_per_s | 96 | 4 | i32 | Actual velocity, counts/s; valid only with velocity_actual_valid. |
following_error_counts | 100 | 4 | i32 | Following error, counts; valid only with following_error_valid. |
velocity_actual_valid | 104 | 4 | u32 | |
following_error_valid | 108 | 4 | u32 | |
di_bits | 112 | 4 | u32 | |
di_valid | 116 | 4 | u32 | |
external_enable_active | 120 | 4 | u32 | |
external_enable_valid | 124 | 4 | u32 | |
coordinate_counts | 128 | 8 | i64 | |
coordinate_epoch | 136 | 8 | u64 | |
pdo_observed_time_ns | 144 | 8 | u64 | |
collision_peak_following_error_counts | 152 | 8 | u64 | |
collision_trip_count | 160 | 8 | u64 | |
collision_samples | 168 | 8 | u64 | |
collision_last_trip_peak_following_error_counts | 176 | 8 | u64 | |
absolute_source_counts | 184 | 4 | i32 | |
coordinate_reason | 188 | 4 | u32 | |
target_velocity_counts_per_s | 192 | 4 | i32 | |
collision_armed | 196 | 4 | u32 | |
collision_torque_cycles | 200 | 4 | u32 | |
collision_following_error_cycles | 204 | 4 | u32 | |
collision_peak_torque_raw | 208 | 4 | u32 | |
collision_last_trip_quantity | 212 | 4 | u32 | |
collision_trip_sustained_cycles | 216 | 4 | u32 | |
collision_last_trip_peak_torque_raw | 220 | 4 | u32 | |
brake_state | 224 | 4 | u32 | |
native_home_position_offset | 228 | 4 | i32 | |
native_home_last_abort_code | 232 | 4 | u32 | |
ext_position_error_counts | 236 | 4 | i32 | |
ext_multi_turn_lo | 240 | 4 | i32 | |
ext_multi_turn_hi | 244 | 4 | i32 | |
max_abs_velocity_actual_counts_per_s | 248 | 4 | u32 | |
max_abs_following_error_counts | 252 | 4 | u32 | |
torque_raw | 256 | 2 | i16 | Torque, signed per-mille of rated; valid only with torque_valid. |
ext_bus_voltage_raw | 258 | 2 | u16 | |
ext_load_rate_raw | 260 | 2 | u16 | |
ext_igbt_temp_raw | 262 | 2 | i16 | |
ext_motor_temp_raw | 264 | 2 | i16 | |
ext_drive_not_ready_bits | 266 | 2 | u16 | |
ext_motor_not_rotating_code | 268 | 2 | u16 | |
ext_valid_mask | 270 | 2 | u16 | Bits 0..8 qualify the ext_* fields; ignore any field whose bit is clear. |
mode_display | 272 | 1 | i8 | Drive mode (8 = CSP). |
ds402_state | 273 | 1 | u8 | DS402 state code (see the real-time core page). |
torque_valid | 274 | 1 | u8 | |
coordinate_valid | 275 | 1 | u8 | 1 when the coordinate is trustworthy. |
coordinate_source_valid | 276 | 1 | u8 | |
native_home_state | 277 | 1 | u8 | |
slave_al_state | 278 | 1 | u8 | |
slave_online | 279 | 1 | u8 | |
slave_operational | 280 | 1 | u8 | |
pdo_fresh | 281 | 1 | u8 | 1 when this axis's feedback is fresh. |
target_velocity_valid | 282 | 1 | u8 | |
ext_valid | 283 | 1 | u8 | |
calibration_valid | 284 | 1 | u8 | |
position_state | 285 | 1 | u8 | |
audit_reason | 286 | 2 | u16 | |
audit_last_verified_ns | 288 | 8 | u64 | |
audit_started_ns | 296 | 8 | u64 | |
audit_completed_ns | 304 | 8 | u64 | |
audit_coherence_error_counts | 312 | 8 | u64 |
Decoders for these layouts are generated from protocol/control.json: Go in rosieos/rt-core/ipcclient (used by Client.TelemetryBatches and TelemetryStream), C++ in protocol_generated.hpp (used by RtControlClient::telemetry), and TypeScript in rt_protocol_generated.ts.
Resources#
Describe's robot.resources lists the compiled robot description files (manifest, URDF, meshes, calibration) with their SHA-256 digests. Fetch one with GET /v1/resources/<sha256>. The bytes are immutable, and an unknown digest returns 404 resource_unknown.
Related pages#
- rt-control HTTP API:
subscribe_events,telemetry,mark_telemetryandresource. - The real-time core: where the telemetry ring comes from.
- TypeScript contracts: decoding telemetry in the browser or Node.