Use the virtual pendant

On this page

Start it
Drive the simulated robot
Keyboard and mouse
What it cannot do
Run the bridge by hand

The virtual pendant is a browser copy of the Steam Deck v4 pendant's screen, drawn at the Deck's native 1280×800. A small loopback bridge connects it to the simulated core. Use it to try the pendant workflow, or to work on its layout, without a Deck or a robot.

It is simulation-only by construction:

The bridge binds only to a loopback address.
It checks that the backend it drives is the simulation, with the expected configuration digest and pair.
The browser never sees a control socket or a credential.

Start it#

Start the simulated core, rt-control and the bridge with the dev stack. Then start the browser UI:

export DEV_STACK_UI_HOST=127.0.0.1 DEV_STACK_FIREWALL=0
./dev-stack.sh start rt-sim rt-control pendant
cd steamdeck/virtual/v1/ui
npm ci
npm run dev

Open http://127.0.0.1:51711/steamdeck/virtual/v1/. Vite serves the page on port 51711. It proxies /healthz and /api/v1 (including the WebSocket session) to the bridge on 127.0.0.1:51712.

The pendant's 3D view loads the Rosie 1400 model from urdf/v1/ui/public/robot/rosie_1400_v3.

Drive the simulated robot#

On the rt-core backend, the pendant does joint jog only. The usual sequence:

Take the leader. The bridge acquires the rt-control lease for your browser session and renews it every 100 ms. A second browser gets control_already_owned.
Home. This sends a public home for all axes. The dev-stack simulation requires Home before anything moves.
Arm. This sends enable for all axes, then arm.
Jog. Select a joint, J1 to J9, and push the left stick up or down, or the right stick sideways. Holding one trigger scales the speed to 50%, both triggers to 10%.
Disarm or Stop. Either one sends a Stop and releases the lease. Release the leader when you are done.

Jog speed is capped at 10% of the joint's maximum velocity from Describe, times the stick deflection and the trigger scale. The bridge refuses any input sample older than 150 ms. If samples stop arriving, the core's jog deadline ramps the joint to a hold.

Clear Fault maps to reset_fault, which is an interim capability in rt-core.

Keyboard and mouse#

Key	Deck control
`[` / `]`, or Page Up / Page Down	LB / RB
Arrow keys	D-pad
A (or Space, Enter), B (or Esc), X, Y	Face buttons
Tab	Select (View)
M	Menu
Shift (hold)	LT
Ctrl (hold)	RT
1 / 2	Latch LT / RT on or off
G	Toggle the layout grid

Drag the on-screen sticks with the mouse, or plug in a gamepad. The browser Gamepad API maps its sticks, face buttons, bumpers and D-pad.

What it cannot do#

On the rt-core backend, these return a typed refusal instead of acting:

Feature	Refusal code
Cartesian or TCP jog	`rt_core_cartesian_unavailable`
Program list, program editing, planning	`rt_core_planning_unavailable`
Reading or applying drive configuration	`rt_core_configuration_unavailable`
Log sessions	`rt_core_logs_unavailable`

Use the offline programming app for Cartesian jog and programs. Use rtctl and the events and telemetry streams for logs and configuration.

Run the bridge by hand#

The dev-stack pendant service builds the bridge from steamdeck/virtual and starts it with the simulator's binding. To do the same yourself:

(cd steamdeck/virtual && go build -o ../../rt-core/build/virtual-deck ./cmd/local)
source rt-core/build/dev-stack/binding.env
rt-core/build/virtual-deck bridge start --adapter robot-v4-sim --backend rt_core \
  --rt-core-socket "$ROSIE_RT_CONTROL_SOCKET" --rt-core-pair "$ROSIE_RT_PAIR_ID" \
  --rt-core-revision "$ROSIE_RT_PAIR_REVISION" --rt-core-sha256 "$ROSIE_RT_CONFIGURATION_SHA256"

bridge start flags:

Flag	Default	Description
`--adapter`	`fixture`	`robot-v4-sim` drives the simulated core. `fixture` serves a deterministic, command-disabled fixture set.
`--backend`	`rt_core`	The only supported backend. Any other value is refused as `backend_retired`.
`--rt-core-socket`	—	The public `control.sock`. Required for `robot-v4-sim`.
`--rt-core-pair`, `--rt-core-revision`	—	The pair binding. Required for `robot-v4-sim`.
`--rt-core-sha256`	—	The configuration digest, 64 hex. Required for `robot-v4-sim`.
`--fixture`	`v1-complete`	Fixture set for the `fixture` adapter
`--host`	`127.0.0.1`	Bind host. Must be a loopback address.
`--port`	`51712`	Bind port
`--dry-run`	off	Validate the flags and print the address without starting

bridge status [--host] [--port] probes a running bridge's /healthz. bridge contract prints the bridge's protocol contract as JSON. The rt-core adapter needs Linux or WSL. On other platforms it fails with rt_core_platform_unavailable.

To point the UI at a bridge on another port, set STEAMDECK_VIRTUAL_BRIDGE_URL before npm run dev, for example http://127.0.0.1:51713. Start the bridge on that port with DEV_STACK_PENDANT_PORT=51713.