Advanced Metal Research
GitHub Contact AMR

Events and telemetry streams

On this page
  1. Follow the event stream
  2. Events
  3. Cursors and loss
  4. Event types
  5. Telemetry
  6. Cursors and loss
  7. Batch header
  8. Cycle record
  9. Axis entry
  10. Resources
  11. Related pages

rt-control publishes two observation streams. Neither needs a lease, so any client that can reach the socket can watch the cell while another client controls it.

  • Events are a JSON log of state changes: grants, enable and arm, handle and execution transitions, jog sessions, faults, Home epochs and restarts. They are good for reacting to changes.
  • Telemetry is binary: one record per control cycle with positions, targets, drive state and fault masks. It is good for plotting, logging and replay.

Both use the same cursor model: you pass after, the last sequence you handled, and the server tells you exactly what you missed.

Follow the event stream#

# Server-Sent Events from the beginning of the retained log; Ctrl-C to stop.
curl -sN --unix-socket /run/rosie-rt-core/control.sock \
  'http://localhost/v1/events/stream?after=0'
c, err := control.Dial("/run/rosie-rt-core/control.sock")
if err != nil {
	log.Fatal(err)
}
defer c.Close()
var cursor uint64
err = c.StreamEvents(ctx, cursor, func(e control.Event) error {
	if e.Type == "events_dropped" {
		// History was lost: re-read Status before trusting your local view.
		log.Printf("lost events %d..%d", e.Dropped.FirstLostSequence, e.Dropped.LastLostSequence)
	}
	log.Printf("%d %s", e.Sequence, e.Type)
	cursor = e.Sequence
	return nil
})
SSE outputText
: rt-core events

id: 1
event: adapter_incarnation_changed
data: {"sequence":1,"time_ns":81230000000000,"type":"adapter_incarnation_changed","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","incarnation":{"previous":"","current":"a41c…"}}

id: 2
event: grant_acquired
data: {"sequence":2,"time_ns":81234567890123,"type":"grant_acquired","daemon_incarnation":"3b9d…","adapter_incarnation":"a41c…","grant":{"generation":1,"stopping":false}}

Events#

GET /v1/events?after=<sequence>

One batch of up to 64 events after the cursor, in the normal Response envelope with an EventBatch in data.

SSE /v1/events/stream?after=<sequence>

The same events as Server-Sent Events. Each message is id (the sequence), event (the type) and data (one Event as JSON, with no envelope).

Cursors and loss#

  • after is the sequence of the last event you handled. Omit it (or send 0) to start at the oldest retained event. It must be one unsigned decimal number, otherwise you get 409 invalid_event_cursor. A cursor newer than the log gets 409 event_cursor_ahead.
  • Sequences increase by one within an adapter_incarnation. A poll returns next_sequence (use it as your next after) and latest_sequence.
  • The log keeps the last 256 events. If your cursor is older than that, the first record you receive is a synthetic events_dropped event whose first_lost_sequence..last_lost_sequence range (inclusive) is what you missed. Its own sequence equals last_lost_sequence, so it is also your resume cursor. The retained events that follow keep their original sequence numbers.
  • The SSE handler does not read Last-Event-ID. To resume, reconnect with ?after= set to the last id you handled.
  • When rt-control restarts, the sequence starts again under a new adapter_incarnation. Discard your cursor, read Status and start from 0.

Events are observations, not a lossless trace. time_ns is the core's publication time, and native snapshots can merge transitions that happen between publications. After any loss or incarnation change, reconcile from GET /v1/status rather than replaying motion.

Event types#

Every event has sequence, time_ns, type, daemon_incarnation and adapter_incarnation, plus one payload field for its type:

TypePayload fieldPayload typeEmitted when
grant_acquiredgrantGrantEventacquire succeeded.
grant_renewedgrantGrantEventrenew succeeded. stopping: true while a Stop drains.
grant_releasedgrantGrantEventrelease.
grant_expiredgrantGrantEventThe lease ran out.
grant_revokedgrantGrantEventstop, or authority lost to a native or transport failure.
enable_changedenableEnableEventaxis_enable_mask changed.
arm_changedenableEnableEventarmed changed.
handle_transitionhandleHandleTransitionEventA trajectory handle changed state (from → to).
execution_startedexecutionExecutionEventA handle started.
execution_completedexecutionExecutionEventA started handle was consumed: it completed, or a later execution replaced it.
execution_abortedexecutionExecutionEventA started handle was retired with no fault bits set.
execution_faultedexecutionExecutionEventA started handle was retired with fault bits set; fault_bits and recovery say which.
jog_beginjogJogEventA jog session opened.
jog_endjogJogEventA jog session closed for a reason other than expiry.
jog_expiredjogJogEventJog input expired.
jog_rampingjogJogEventThe jog is ramping down.
jog_limitedjogJogEventThe jog is being slowed at a position limit (an accepted state, not a failure).
fault_latchedfaultFaultEventAn execution fault bit was set.
fault_clearedfaultFaultEventAn execution fault bit was cleared.
home_epoch_changedepochEpochEventHome evidence changed.
daemon_incarnation_changedincarnationIncarnationEventThe core process changed (and once at adapter start).
adapter_incarnation_changedincarnationIncarnationEventEmitted once when rt-control starts.
publisher_overflowoverflowPublisherOverflowEventNative observations were dropped before they reached the event log.
events_droppedevents_droppedEventsDroppedYour cursor fell behind the 256-event ring; the lost range is inclusive.
telemetry_markmarkTelemetryMarkEventmark_telemetry was accepted.

Note

In FaultEvent, bit is the bit's value (for example 32 for bit 5), not its index. In RecoveryStatus.faults[], bit is the index. The bits are listed in Execution fault bits.

Events never contain the session token. The field tables for every payload type are in the types section of the HTTP reference.

Telemetry#

GET /v1/telemetry?after=<sequence>

One binary batch after the cursor. Sends gzip when you ask for it with Accept-Encoding: gzip.

GET /v1/telemetry/stream?after=<sequence>

Concatenated binary batches over chunked HTTP, one complete batch per flush, about every 20 ms.

Telemetry is application/octet-stream, never JSON or base64. A batch is a 312-byte TelemetryBatchHeaderV1 followed by exactly record_count records of 5392 bytes (CycleCaptureRecordV2), all little-endian. A batch is at most 312 + 4096 × 5392 = 22085944 bytes.

Go: follow telemetrygo
err := c.TelemetryStream(ctx, 0, func(b control.TelemetryBatch) error {
	if b.Header.Dropped != 0 {
		log.Printf("lost %d records", b.Header.Dropped)
	}
	for _, r := range b.Records {
		j1 := r.Ax[0]
		_ = j1.Position // drive counts; convert with Describe counts_per_unit and sign
	}
	return nil
})

Cursors and loss#

  • after is the last record sequence you consumed. 0 (the default) starts at sequence 1, and history that the ring has already overwritten is reported in dropped.
  • A non-empty batch starts at after + dropped + 1 and its sequences are contiguous. An empty batch has first_sequence 0 and last_sequence equal to after + dropped. Always resume from last_sequence, after you have handled the batch and its loss.
  • Errors come back as JSON with HTTP 409: invalid_telemetry_cursor, telemetry_cursor_ahead, telemetry_unavailable, or telemetry_busy with Retry-After: 1 (retry the same cursor). On an established stream, a busy read becomes an empty batch with the cursor unchanged.
  • A stream closes when the core disconnects or a write blocks for 5 s. HTTP proxies may re-chunk the stream, so find batch boundaries from the header, not from chunks.
  • If adapter_incarnation or daemon_incarnation changes, stop. Re-read Describe and Status before you reset the cursor.

Telemetry needs no session, but on the remote listener it still needs a valid client certificate.

Batch header#

TelemetryBatchHeaderV1, 312 bytes. Check magic, version, header_bytes, record_bytes, the layout digest, axis_count (1..16) and record_count (≤ 4096) before reading the body.

FieldOffsetSizeTypeNotes
magic04u320x31425452 (RTB1).
version42u161.
header_bytes62u16312.
adapter_incarnation832u8[32]Lowercase hex, rt-control process identity.
machine_sha2564064u8[64]Lowercase hex machine digest.
deployment_sha25610464u8[64]Lowercase hex deployment digest.
record_layout_digest16864u8[64]Must equal CycleCaptureRecordV2LayoutDigest.
cycle_period_ns2328u64Cycle period, ns.
first_sequence2408u64First record's sequence; 0 when the batch is empty.
last_sequence2488u64Last record's sequence, or after + dropped when empty. Your next cursor.
dropped2568u64Records lost after your cursor and before this batch.
axis_count2644u32Active axes, 1..16.
record_count2684u32Records in this batch, at most min(ring capacity, 4096).
record_bytes2724u325392.
reserved2764u32Reserved, zero.
daemon_incarnation28032u8[32]Lowercase hex, core process identity; owns the sequence numbers.

The current record layout digest is 1173439681672348f35f9652f2a4821d251638c70b60362ff59b37032a794b6b. A reader that sees a different digest must refuse the batch: the Go, C++ and TypeScript decoders all raise TelemetryLayoutMismatchError with both digests.

Cycle record#

CycleCaptureRecordV2, 5392 bytes: 52 group fields followed by ax, 16 per-axis entries. Positions and targets are in drive counts, not radians; convert them with the axis's counts_per_unit and sign from Describe.

FieldOffsetSizeTypeNotes
t_ns08u64Cycle time, host monotonic ns.
seq88u64Record sequence.
cycle168u64Cycle counter.
work_ns248u64Cycle work time, ns.
axes324u32Active axis count.
reserved364u32Reserved, zero.
cycle_jitter_ns408i64Wake time minus scheduled time, ns (signed).
active_traj_id488u64
active_command_seq568u64
jog_generation648u64
app_grant_generation728u64
control_generation808u64
configuration_epoch888u64
home_epoch968u64
grant_deadline_host_ns1048u64
jog_input_deadline_host_ns1128u64
jog_source_sequence1208u64
bus_submission_return_code1288i64
armed1364u321 when armed.
axis_enable_mask1404u32Enabled axes.
active_mode1444u320 idle, 2 trajectory, 3 jog.
state1484u32
current_point_index1524u32
queue_depth1564u32
last_event_code1604u32
underrun_count1644u32
stale_command_flag1684u32
motion_done1724u32
capability_flags1764u32
jog_session_open1804u321 while jog input is accepted.
jog_has_input1844u32
jog_axis_mask1884u32
jog_ramping1924u32
grant_active1964u321 while the effective grant is active.
grant_axis_mask2004u32
safety_fault_mask2044u32Non-zero blocks motion.
execution_fault_reasons2084u32Latched fault bits.
submitted_axis_mask2124u32
bus_submission_operation2164u32
pdo_fresh_axis_mask2204u32Axes with fresh feedback this cycle.
config_verified_mask2244u32
home_valid_mask2284u32Axes with valid Home.
service_mode_axis_mask2324u32
wkc_actual2364u32
wkc_expected2404u32
master_state2444u32
io_configured2484u32
io_input_word2524u32
io_output_word2564u32
io_input_valid2604u32
io_armed2644u32
io_reason2684u32
ax2725120CycleCaptureAxisV2[16]One entry per axis; unused axes are zero.

Axis entry#

CycleCaptureAxisV2, 320 bytes, 16 per record. Check pdo_fresh and each field's validity flag before you use a value; invalid values are not meaningful, and di_valid = 0 means unavailable, not "inputs off".

FieldOffsetSizeTypeNotes
position04i32Feedback position, drive counts.
target44i32Commanded target, drive counts.
statusword82u16CiA402 statusword.
error_code102u16CiA402 error code.
manufacturer_error_code124u32
absolute_value1664i32[16]
absolute_valid8016u8[16]
velocity_actual_counts_per_s964i32Actual velocity, counts/s; valid only with velocity_actual_valid.
following_error_counts1004i32Following error, counts; valid only with following_error_valid.
velocity_actual_valid1044u32
following_error_valid1084u32
di_bits1124u32
di_valid1164u32
external_enable_active1204u32
external_enable_valid1244u32
coordinate_counts1288i64
coordinate_epoch1368u64
pdo_observed_time_ns1448u64
collision_peak_following_error_counts1528u64
collision_trip_count1608u64
collision_samples1688u64
collision_last_trip_peak_following_error_counts1768u64
absolute_source_counts1844i32
coordinate_reason1884u32
target_velocity_counts_per_s1924i32
collision_armed1964u32
collision_torque_cycles2004u32
collision_following_error_cycles2044u32
collision_peak_torque_raw2084u32
collision_last_trip_quantity2124u32
collision_trip_sustained_cycles2164u32
collision_last_trip_peak_torque_raw2204u32
brake_state2244u32
native_home_position_offset2284i32
native_home_last_abort_code2324u32
ext_position_error_counts2364i32
ext_multi_turn_lo2404i32
ext_multi_turn_hi2444i32
max_abs_velocity_actual_counts_per_s2484u32
max_abs_following_error_counts2524u32
torque_raw2562i16Torque, signed per-mille of rated; valid only with torque_valid.
ext_bus_voltage_raw2582u16
ext_load_rate_raw2602u16
ext_igbt_temp_raw2622i16
ext_motor_temp_raw2642i16
ext_drive_not_ready_bits2662u16
ext_motor_not_rotating_code2682u16
ext_valid_mask2702u16Bits 0..8 qualify the ext_* fields; ignore any field whose bit is clear.
mode_display2721i8Drive mode (8 = CSP).
ds402_state2731u8DS402 state code (see the real-time core page).
torque_valid2741u8
coordinate_valid2751u81 when the coordinate is trustworthy.
coordinate_source_valid2761u8
native_home_state2771u8
slave_al_state2781u8
slave_online2791u8
slave_operational2801u8
pdo_fresh2811u81 when this axis's feedback is fresh.
target_velocity_valid2821u8
ext_valid2831u8
calibration_valid2841u8
position_state2851u8
audit_reason2862u16
audit_last_verified_ns2888u64
audit_started_ns2968u64
audit_completed_ns3048u64
audit_coherence_error_counts3128u64

Decoders for these layouts are generated from protocol/control.json: Go in rosieos/rt-core/ipcclient (used by Client.TelemetryBatches and TelemetryStream), C++ in protocol_generated.hpp (used by RtControlClient::telemetry), and TypeScript in rt_protocol_generated.ts.

Resources#

Describe's robot.resources lists the compiled robot description files (manifest, URDF, meshes, calibration) with their SHA-256 digests. Fetch one with GET /v1/resources/<sha256>. The bytes are immutable, and an unknown digest returns 404 resource_unknown.