In this tutorial, we work through the retargeting engine at the core of NVIDIA IsaacTeleop, the framework that turns XR hand tracking and motion-controller input into commands for simulated and real robots. Rather than plugging in a headset, we build every input ourselves in NumPy, so each step runs on a plain Colab CPU and prints what it computes. We start with the type system every node speaks, generate synthetic hand and controller data, write our own retargeter with live-tunable parameters, and then drive the built-in gripper and SE(3) retargeters with it. From there, we compose a full graph that emits one action vector per step, apply a world-frame transform, step through the run, pause and kill the state machine, and finish with a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts.
import os
import sys
import json
import math
import tempfile
import traceback
import subprocess
import numpy as np
RESULTS = {}
def banner(title):
print("n" + "=" * 78)
print(title)
print("=" * 78)
def section(name):
def wrap(fn):
def run(*a, **kw):
banner(name)
try:
out = fn(*a, **kw)
RESULTS[name] = out if isinstance(out, str) else "ok"
return out
except Exception as e:
RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}"
print(f"n[!] {name} did not complete: {type(e).__name__}: {e}")
traceback.print_exc(limit=3)
return None
return run
return wrap
banner("0. Install isaacteleop and check the environment")
subprocess.run(
[sys.executable, "-m", "pip", "install", "-q", "isaacteleop[retargeters-lite]==1.4.145"],
check=True,
)
import pkgutil
import isaacteleop
from isaacteleop import schema
print(f" isaacteleop {isaacteleop.__version__} | Python {sys.version.split()[0]} | numpy {np.__version__}")
print(" top-level modules :", ", ".join(sorted(m.name for m in pkgutil.iter_modules(isaacteleop.__path__))))
message_types = [n for n in dir(schema) if n[0].isupper()]
print(f" schema message types: {len(message_types)}, e.g. {', '.join(message_types[:6])}")
print(" no headset, no OpenXR runtime, no simulator is used anywhere below.")
We install the stable isaacteleop wheel from PyPI with the retargeters-lite extra, which adds only SciPy, and report the version, interpreter, and NumPy build. The package splits into device I/O modules that wrap OpenXR and CloudXR, a schema module holding the FlatBuffer message types every tracker emits, and the pure-Python retargeting engine we work with here. Listing the schema types shows the vocabulary of the data layer, but nothing below opens a headset session; from here on, every input is a tensor we build by hand.
from isaacteleop.retargeting_engine.interface import (
TensorGroup,
OptionalTensorGroup,
TensorGroupType,
OptionalType,
)
from isaacteleop.retargeting_engine.tensor_types import (
HandInput,
ControllerInput,
HandInputIndex,
ControllerInputIndex,
HandJointIndex,
)
@section("1. The type contract: TensorGroupType, TensorGroup, Optional")
def type_contract():
hand_t = HandInput()
ctrl_t = ControllerInput()
print(" HandInput :", hand_t)
print(" slot index :", ", ".join(f"{m.name}={m.value}" for m in HandInputIndex))
print(f" ControllerInput: {len(ctrl_t)} slots ->",
", ".join(t.name.replace("controller_", "") for t in ctrl_t.types))
print(f" OpenXR hand joints: {len(HandJointIndex)} "
f"(WRIST={int(HandJointIndex.WRIST)}, THUMB_TIP={int(HandJointIndex.THUMB_TIP)}, "
f"INDEX_TIP={int(HandJointIndex.INDEX_TIP)})")
hand = TensorGroup(hand_t)
hand[HandInputIndex.JOINT_POSITIONS] = np.zeros((26, 3), dtype=np.float32)
print("n wrote a (26, 3) float32 array into JOINT_POSITIONS ->", hand)
try:
hand[HandInputIndex.JOINT_POSITIONS] = np.zeros((26, 3))
except TypeError as e:
print(" float64 rejected at write time :", str(e)[:110])
try:
_ = hand[HandInputIndex.JOINT_VALID]
except ValueError as e:
print(" reading a slot nobody wrote :", e)
maybe = OptionalTensorGroup(OptionalType(ctrl_t))
print(f"n Optional group starts absent : is_none={maybe.is_none} {maybe}")
maybe[ControllerInputIndex.TRIGGER_VALUE] = 0.0
print(f" one write flips it to present : is_none={maybe.is_none} {maybe}")
return f"{len(hand_t)} hand slots, {len(ctrl_t)} controller slots"
type_contract()
The engine’s contract is a TensorGroupType, an ordered list of typed slots, and a TensorGroup, the runtime container that holds one value per slot and validates each write. HandInput carries four NumPy arrays for the 26 OpenXR hand joints, and ControllerInput carries fourteen slots for poses, buttons, and axes, addressed through generated IntEnum indices rather than magic numbers. Writing a float64 array where float32 is declared fails at the write, and reading a slot nobody wrote raises instead of returning stale data. OptionalType marks inputs a tracker may not deliver; the matching OptionalTensorGroup starts absent and becomes present on its first write, which is how every downstream node learns that a hand has left the tracking volume.
J = HandJointIndex
C = ControllerInputIndex
RIGHT_WRIST = np.array([0.30, 1.05, -0.45], dtype=np.float32) # OpenXR: x right, y up, z back
def make_hand(pinch_m, wrist=RIGHT_WRIST, quat=(0.0, 0.0, 0.0, 1.0)):
"""A synthetic right hand in HandInput layout: 26 joints in OpenXR order."""
pos = np.zeros((26, 3), dtype=np.float32)
pos[J.WRIST] = wrist
pos[J.PALM] = wrist + [0.0, 0.0, -0.06]
fingers = [(J.INDEX_METACARPAL, 0.03), (J.MIDDLE_METACARPAL, 0.01),
(J.RING_METACARPAL, -0.01), (J.LITTLE_METACARPAL, -0.03)]
for base, x_off in fingers: # metacarpal .. tip, five joints each
for k in range(5):
pos[base + k] = wrist + [x_off, 0.0, -0.05 - 0.02 * k]
for k in range(4): # thumb: metacarpal .. tip, four joints
pos[J.THUMB_METACARPAL + k] = wrist + [0.05, -0.01, -0.02 - 0.015 * k]
pos[J.THUMB_TIP] = pos[J.INDEX_TIP] + [pinch_m, 0.0, 0.0]
g = TensorGroup(HandInput())
g[HandInputIndex.JOINT_POSITIONS] = pos
g[HandInputIndex.JOINT_ORIENTATIONS] = np.tile(np.asarray(quat, dtype=np.float32), (26, 1))
g[HandInputIndex.JOINT_RADII] = np.full(26, 0.01, dtype=np.float32)
g[HandInputIndex.JOINT_VALID] = np.ones(26, dtype=np.uint8)
return g
def make_controller(pos, quat=(0.0, 0.0, 0.0, 1.0), trigger=0.0, squeeze=0.0,
thumbstick=(0.0, 0.0), valid=True):
"""A synthetic controller snapshot in ControllerInput layout."""
p = np.asarray(pos, dtype=np.float32)
q = np.asarray(quat, dtype=np.float32)
g = TensorGroup(ControllerInput())
g[C.GRIP_POSITION], g[C.GRIP_ORIENTATION], g[C.GRIP_IS_VALID] = p, q, bool(valid)
g[C.AIM_POSITION] = p + np.array([0.0, 0.0, -0.05], dtype=np.float32)
g[C.AIM_ORIENTATION], g[C.AIM_IS_VALID] = q.copy(), bool(valid)
for idx in (C.PRIMARY_CLICK, C.SECONDARY_CLICK, C.THUMBSTICK_CLICK, C.MENU_CLICK):
g[idx] = 0.0
g[C.THUMBSTICK_X], g[C.THUMBSTICK_Y] = float(thumbstick[0]), float(thumbstick[1])
g[C.SQUEEZE_VALUE], g[C.TRIGGER_VALUE] = float(squeeze), float(trigger)
return g
@section("2. Synthetic tracking data: a hand and a controller in numpy")
def synthetic_inputs():
hand = make_hand(pinch_m=0.06)
pos = hand[HandInputIndex.JOINT_POSITIONS]
print(" joint x y z")
for j in (J.WRIST, J.PALM, J.THUMB_TIP, J.INDEX_TIP, J.LITTLE_TIP):
print(f" {j.name:12s} {pos[j][0]:6.3f} {pos[j][1]:6.3f} {pos[j][2]:6.3f}")
pinch = np.linalg.norm(pos[J.THUMB_TIP] - pos[J.INDEX_TIP])
print(f" thumb-to-index distance: {pinch * 100:.1f} cm "
f"valid joints: {int(hand[HandInputIndex.JOINT_VALID].sum())}/26")
ctrl = make_controller((0.40, 1.20, -0.30), trigger=0.8, thumbstick=(0.0, 0.5))
print(f"n controller grip {np.round(ctrl[C.GRIP_POSITION], 2)} aim {np.round(ctrl[C.AIM_POSITION], 2)} "
f"trigger {ctrl[C.TRIGGER_VALUE]} squeeze {ctrl[C.SQUEEZE_VALUE]} "
f"thumbstick ({ctrl[C.THUMBSTICK_X]}, {ctrl[C.THUMBSTICK_Y]})")
return "synthetic HandInput + ControllerInput builders"
synthetic_inputs()
We build the tracking data that a headset would normally supply. make_hand lays out 26 joints in OpenXR order around a wrist position, runs four finger chains and a thumb chain away from the palm, and places the thumb tip a chosen pinch distance from the index tip, so the one number the later steps depend on is under our control. make_controller fills every ControllerInput slot: grip and aim poses with validity flags, four buttons, the thumbstick axes, and the analog squeeze and trigger values. Both return ordinary TensorGroups, which is all a retargeter ever sees, whether the numbers came from OpenXR or from NumPy.
from isaacteleop.retargeting_engine.interface import (
BaseRetargeter,
ParameterState,
FloatParameter,
BoolParameter,
)
from isaacteleop.retargeting_engine.tensor_types import FloatType, BoolType
class PinchRetargeter(BaseRetargeter):
"""Thumb-to-index distance -> (distance_cm, is_pinching), with live-tunable parameters."""
def __init__(self, name, config_file=None):
params = [
FloatParameter("threshold_cm", "Pinching when closer than this",
default_value=3.0, min_value=0.5, max_value=10.0,
sync_fn=lambda v: setattr(self, "threshold_cm", v)),
BoolParameter("use_distal_joints", "Measure between distal joints, not tips",
default_value=False,
sync_fn=lambda v: setattr(self, "use_distal_joints", v)),
]
super().__init__(name, parameter_state=ParameterState(name, params, config_file=config_file))
def input_spec(self):
return {"hand_right": OptionalType(HandInput())}
def output_spec(self):
return {"pinch": TensorGroupType("pinch", [FloatType("distance_cm"), BoolType("is_pinching")])}
def _compute_fn(self, inputs, outputs, context):
hand = inputs["hand_right"]
if hand.is_none: # tracking lost: say so, do not guess
outputs["pinch"][0] = -1.0
outputs["pinch"][1] = False
return
pos = np.from_dlpack(hand[HandInputIndex.JOINT_POSITIONS])
a, b = (J.THUMB_DISTAL, J.INDEX_DISTAL) if self.use_distal_joints else (J.THUMB_TIP, J.INDEX_TIP)
d_cm = float(np.linalg.norm(pos[a] - pos[b]) * 100.0)
outputs["pinch"][0] = d_cm
outputs["pinch"][1] = bool(d_cm < self.threshold_cm)
@section("3. Write a retargeter: pinch detection with live-tunable parameters")
def custom_retargeter():
pinch = PinchRetargeter("pinch")
for d in (0.06, 0.02):
out = pinch({"hand_right": make_hand(d)})
print(f" hand at {d * 100:.0f} cm -> distance_cm={out['pinch'][0]:.2f} is_pinching={out['pinch'][1]}")
out = pinch({}) # optional input omitted entirely
print(f" no hand tracked -> distance_cm={out['pinch'][0]:.1f} is_pinching={out['pinch'][1]}")
state = pinch.get_parameter_state()
print("n tunable parameters:", state.get_all_values())
for threshold in (2.5, 3.5):
state.set({"threshold_cm": threshold}) # what the tuning UI does from its own thread
out = pinch({"hand_right": make_hand(0.028)})
print(f" threshold_cm={threshold}: a 2.8 cm pinch -> is_pinching={out['pinch'][1]}")
state.set({"use_distal_joints": True})
out = pinch({"hand_right": make_hand(0.028)})
print(f" use_distal_joints=True: distance_cm={out['pinch'][0]:.2f} is_pinching={out['pinch'][1]}")
return "custom retargeter + 2 tunable parameters"
custom_retargeter()
A retargeter is a BaseRetargeter subclass that declares input_spec and output_spec and implements _compute_fn; the framework fills missing optional inputs with absent groups, validates types, syncs parameters, and only then calls our code. PinchRetargeter measures the thumb-to-index distance and emits a float and a bool, reporting -1 when no hand is tracked instead of guessing. Its two parameters live in a ParameterState whose sync functions write onto the instance before every compute, so a value set from another thread by the tuning UI takes effect on the next frame. We reproduce that by calling set directly: the same 2.8 cm pinch flips between not pinching and pinching as the threshold moves, and switching the measurement to the distal joints changes the distance itself.
from isaacteleop.retargeters import (
GripperRetargeter,
GripperRetargeterConfig,
Se3AbsRetargeter,
Se3RelRetargeter,
Se3RetargeterConfig,
)
@section("4. Built-in retargeters: gripper hysteresis and SE(3) end-effector pose")
def builtin_retargeters():
gripper = GripperRetargeter(
GripperRetargeterConfig(hand_side="right", gripper_close_meters=0.03, gripper_open_meters=0.05),
name="gripper",
)
print(" pinch sweep (close below 3 cm, open above 5 cm, hold in between):")
for d in (0.07, 0.045, 0.028, 0.040, 0.052, 0.020):
cmd = gripper({"hand_right": make_hand(d)})["gripper_command"][0]
print(f" {d * 100:4.1f} cm -> {cmd:+.0f} {'CLOSED' if cmd < 0 else 'open'}")
cmd = gripper({"hand_right": make_hand(0.07),
"controller_right": make_controller((0.3, 1.0, -0.4), trigger=0.9)})["gripper_command"][0]
print(f" open hand + trigger 0.9 -> {cmd:+.0f} (a present controller outranks hand tracking)")
se3 = Se3AbsRetargeter(
Se3RetargeterConfig(input_device="controller_right", target_offset_roll=90.0, zero_out_xy_rotation=True),
name="ee_pose",
)
ctrl = make_controller((0.40, 1.20, -0.30))
pose = np.from_dlpack(se3({"controller_right": ctrl})["ee_pose"][0])
print(f"n Se3Abs: grip {np.round(ctrl[C.GRIP_POSITION], 2)} -> ee_pose pos {np.round(pose[:3], 3)} "
f"quat(xyzw) {np.round(pose[3:], 3)}")
lost = make_controller((9.0, 9.0, 9.0), valid=False)
pose2 = np.from_dlpack(se3({"controller_right": lost})["ee_pose"][0])
print(f" grip_is_valid=False -> holds the last pose: {np.round(pose2[:3], 3)}")
rel = Se3RelRetargeter(
Se3RetargeterConfig(input_device="controller_right", delta_pos_scale_factor=10.0, alpha_pos=0.5),
name="ee_delta",
)
print("n Se3Rel: the controller moves +2 cm in x every frame (delta x10, EMA alpha 0.5):")
for i in range(4):
ctrl = make_controller((0.40 + 0.02 * i, 1.20, -0.30))
delta = np.from_dlpack(rel({"controller_right": ctrl})["ee_delta"][0])
print(f" frame {i}: ee_delta [dx, dy, dz, rx, ry, rz] = {np.round(delta, 3)}")
return "gripper, Se3Abs and Se3Rel driven from synthetic input"
builtin_retargeters()
We drive two of the built-in retargeters with the same synthetic groups. GripperRetargeter turns pinch distance into the -1/+1 gripper command Isaac Lab expects, with hysteresis: the gripper closes below 3 cm, opens above 5 c,m and holds its state in between, so 4.0 cm stays closed after a close and the sweep back through 5.2 cm reopens it. A present controller takes priority, and a trigger above the threshold closes an open hand. Se3AbsRetargeter maps the controller grip pose to a 7-D end-effector target, applying the configured roll offset and keeping only yaw when zero_out_xy_rotation is set. It holds the last pose when the grip becomes invalid rather than passing a zero quaternion downstream. Se3RelRetargeter emits deltas instead: a constant 2 cm step per frame appears scaled by ten and smoothed by the EMA, converging toward 0.2.
from isaacteleop.retargeting_engine.interface import ValueInput, OutputCombiner
from isaacteleop.retargeters import TensorReorderer
class CountingInput(ValueInput):
"""A ValueInput leaf that counts how many times the graph asked it to compute."""
def __init__(self, name, tensor_type):
super().__init__(name, tensor_type)
self.calls = 0
def _compute_fn(self, inputs, outputs, context):
self.calls += 1
super()._compute_fn(inputs, outputs, context)
def build_pipeline():
controller = CountingInput("controller_right", OptionalType(ControllerInput()))
hand = CountingInput("hand_right", OptionalType(HandInput()))
ee_pose = Se3AbsRetargeter(Se3RetargeterConfig(input_device="controller_right"), name="ee_pose").connect(
{"controller_right": controller.output("value")}
)
gripper = GripperRetargeter(GripperRetargeterConfig(hand_side="right"), name="gripper").connect(
{"controller_right": controller.output("value"), "hand_right": hand.output("value")}
)
ee = ["pos_x", "pos_y", "pos_z", "quat_x", "quat_y", "quat_z", "quat_w"]
action = TensorReorderer(
input_config={"ee_pose": ee, "gripper_command": ["gripper"]},
output_order=ee + ["gripper"],
name="action",
input_types={"ee_pose": "array", "gripper_command": "scalar"},
).connect({"ee_pose": ee_pose.output("ee_pose"), "gripper_command": gripper.output("gripper_command")})
return OutputCombiner({"action": action.output("output")}), controller, hand
@section("5. Compose the graph: leaves -> retargeters -> one action vector per step")
def compose_graph():
pipeline, controller, hand = build_pipeline()
print(" leaf nodes :", [n.name for n in pipeline.get_leaf_nodes()])
print(" outputs :", {k: str(v) for k, v in pipeline.output_types().items()})
print("n frame trigger action = [x, y, z, qx, qy, qz, qw, gripper]")
for f in range(6):
t = f / 6
trigger = 1.0 if f >= 4 else 0.0
ctrl = make_controller((0.40 + 0.10 * math.cos(2 * math.pi * t), 1.20,
-0.30 + 0.10 * math.sin(2 * math.pi * t)), trigger=trigger)
leaf_inputs = {"controller_right": {"value": ctrl}, "hand_right": {"value": make_hand(0.06)}}
action = np.from_dlpack(pipeline.execute_pipeline(leaf_inputs)["action"][0])
print(f" {f:3d} {trigger:.1f} {np.round(action, 3)}")
print(f"n the controller leaf feeds two retargeters, yet computed {controller.calls} times "
f"in 6 frames: one ExecutionCache per step")
return f"8-D action vector; {controller.calls} leaf computes for 6 frames"
compose_graph()
This is the shape of a real Isaac Teleop pipeline. ValueInput nodes stand in for the DeviceIO source nodes as graph leaves, connect wires each retargeter’s inputs to upstream outputs with type checking at connect time, TensorReorderer flattens the 7-D pose and the gripper scalar into the action layout an environment expects, and OutputCombiner exposes the result under a single action key. execute_pipeline takes inputs keyed by leaf name and runs the DAG once per step against a shared ExecutionCache, so the controller leaf, although wired into both the SE(3) and the gripper retargeter, computes exactly once per frame, which our counting subclass confirms. Swapping the leaves for HandsSource and ControllersSource is the only change needed to run this graph against a headset.
from scipy.spatial.transform import Rotation
from isaacteleop.retargeting_engine.utilities import ControllerTransform
from isaacteleop.retargeting_engine.tensor_types import TransformMatrix
@section("6. Coordinate frames: ControllerTransform with a world_T_anchor matrix")
def coordinate_frames():
world_T_anchor = np.eye(4, dtype=np.float32)
world_T_anchor[:3, :3] = Rotation.from_euler("z", 90, degrees=True).as_matrix().astype(np.float32)
world_T_anchor[:3, 3] = [1.0, 0.0, 0.5]
xf = TensorGroup(TransformMatrix())
xf[0] = world_T_anchor
ctrl = make_controller((0.40, 1.20, -0.30), trigger=0.7)
out = ControllerTransform("controller_xform")({"controller_right": ctrl, "transform": xf})
p_in, p_out = ctrl[C.GRIP_POSITION], out["controller_right"][C.GRIP_POSITION]
print(f" grip position anchor frame {np.round(p_in, 3)} -> world frame {np.round(p_out, 3)}")
print(f" check R @ p + t = {np.round(world_T_anchor[:3, :3] @ p_in + world_T_anchor[:3, 3], 3)}")
print(f" grip orientation (xyzw) {np.round(ctrl[C.GRIP_ORIENTATION], 3)} -> "
f"{np.round(out['controller_right'][C.GRIP_ORIENTATION], 3)}")
print(f" trigger passes through {ctrl[C.TRIGGER_VALUE]} -> {out['controller_right'][C.TRIGGER_VALUE]}")
print(f" left controller not given -> {out['controller_left']}")
return "yaw 90 deg + translation applied to grip and aim, inputs untouched"
coordinate_frames()
Headsets report poses in the runtime’s anchor frame and the robot lives in the world frame, so every pipeline that talks to a simulator applies a world_T_anchor transform first. ControllerTransform takes the two optional controller groups plus a TransformMatrix group holding a 4×4 homogeneous matrix, rewrites the grip and aim positions as R p + t and the orientations by the rotation part, and passes buttons, axes and validity flags through untouched. We verify the position against the same matrix product in NumPy and see the absent left controller propagate as absent, which is what lets a single-controller session flow through a two-controller graph.
from isaacteleop.teleop_session_manager import DefaultTeleopStateManager, bool_signal
from isaacteleop.retargeting_engine.interface import ComputeContext, ExecutionEvents, ExecutionState
from isaacteleop.retargeters import LocomotionRootCmdRetargeter, LocomotionRootCmdRetargeterConfig
def button(name, pressed):
g = TensorGroup(bool_signal(name))
g[0] = bool(pressed)
return g
def state_name(out):
st = out["teleop_state"]
return next(t.name for i, t in enumerate(st.group_type.types) if st[i])
@section("7. The control state machine: run, pause, kill, and the reset pulse")
def state_machine():
sm = DefaultTeleopStateManager("state")
script = [("idle", 0, 0, 0), ("press run", 1, 0, 0), ("release", 0, 0, 0), ("press run", 1, 0, 0),
("hold run", 1, 0, 0), ("release", 0, 0, 0), ("press reset", 0, 0, 1), ("release", 0, 0, 0),
("KILL", 0, 1, 0), ("release", 0, 0, 0)]
print(" input run kill reset | state reset_event")
for label, run, kill, reset in script:
out = sm({"run_toggle_button": button("run_toggle_button", run),
"kill_button": button("kill_button", kill),
"reset_button": button("reset_button", reset)})
print(f" {label:12s} {run} {kill} {reset} | {state_name(out):8s} {out['reset_event'][0]}")
out = sm({"run_toggle_button": button("run_toggle_button", 0)})
print(f" kill signal lost | {state_name(out):8s} (fail-safe: required input absent)")
loco = LocomotionRootCmdRetargeter(LocomotionRootCmdRetargeterConfig(initial_hip_height=0.72), name="loco")
print("n right thumbstick Y=+1 raises the hip height each frame; a reset pulse snaps it back:")
for i, reset in enumerate([False, False, False, True, False]):
ctx = ComputeContext(execution_events=ExecutionEvents(reset=reset, execution_state=ExecutionState.RUNNING))
cmd = loco({"controller_left": make_controller((0, 1, 0), thumbstick=(0.0, 0.6)),
"controller_right": make_controller((0, 1, 0), thumbstick=(0.2, 1.0))},
context=ctx)["root_command"][0]
print(f" frame {i} reset={str(reset):5s} root_command [vx, vy, wz, hip] = {np.round(cmd, 4)}")
return "STOPPED -> PAUSED -> RUNNING -> STOPPED, reset pulses delivered through ComputeContext"
state_machine()
Teleoperation needs a way to arm, pause, and kill the robot that doesn’t depend on the retargeters behaving. DefaultTeleopStateManager is itself a retargeter: three optional bool inputs go in, a one-hot teleop_state and a reset_event pulse come out. A rising edge on the run toggle walks STOPPED to PAUSED to RUNNING and back to PAUSED, the reset button emits a one-frame pulse without changing state, kill forces STOPPED and pulses reset, and losing the kill or run signal fails safe to STOPPED. Those events reach every other node through ComputeContext.execution_events; LocomotionRootCmdRetargeter integrates the right thumbstick into a hip height each frame and, on the frame carrying reset=True, snaps back to its initial height before integrating again.
from isaacteleop.retargeters import TriHandMotionControllerRetargeter, TriHandMotionControllerConfig
@section("8. Controller -> dexterous hand joints, and tuned parameters that survive a restart")
def trihand_and_persistence():
joints = ["thumb_rotation", "thumb_proximal", "thumb_distal", "index_proximal",
"index_distal", "middle_proximal", "middle_distal"]
tri = TriHandMotionControllerRetargeter(
TriHandMotionControllerConfig(hand_joint_names=joints, controller_side="right"), name="trihand_right"
)
print(" trigger squeeze |" + "".join(f"{j[:10]:>11s}" for j in joints))
for trig, sq in ((0.0, 0.0), (1.0, 0.0), (0.0, 1.0), (1.0, 1.0)):
out = tri({"controller_right": make_controller((0.3, 1.0, -0.4), trigger=trig, squeeze=sq)})["hand_joints"]
print(f" {trig:.1f} {sq:.1f} |" + "".join(f"{out[i]:11.2f}" for i in range(7)))
cfg_path = os.path.join(tempfile.mkdtemp(), "ee_pose_tuning.json")
se3 = Se3AbsRetargeter(Se3RetargeterConfig(input_device="controller_right", parameter_config_path=cfg_path),
name="ee_pose")
se3.get_parameter_state().set({"rotation_offset_rpy": np.array([90.0, 0.0, 45.0]),
"position_offset_xyz": np.array([0.0, 0.0, 0.10])})
print()
se3.get_parameter_state().save_to_file()
print(" saved :", json.load(open(cfg_path)))
ctrl = make_controller((0.4, 1.2, -0.3))
pose_a = np.from_dlpack(se3({"controller_right": ctrl})["ee_pose"][0])
se3_b = Se3AbsRetargeter(Se3RetargeterConfig(input_device="controller_right", parameter_config_path=cfg_path),
name="ee_pose_restarted")
pose_b = np.from_dlpack(se3_b({"controller_right": ctrl})["ee_pose"][0])
print(f" tuned instance ee_pose = {np.round(pose_a, 3)}")
print(f" restarted instance ee_pose = {np.round(pose_b, 3)} <- same tuning, read back from JSON")
return "7-DOF trihand mapping; parameter JSON round trip"
trihand_and_persistence()
Two production concerns close the tutorial. TriHandMotionControllerRetargeter is the controller-only route to a dexterous hand: trigger drives the index finger, squeeze drives the middle finger, the larger of the two curls the thumb and their difference rotates it, giving seven named joint angles with no hand tracking and no optimization library. Then we make tuning stick: setting the rotation and position offsets on Se3AbsRetargeter’s ParameterState and calling save_to_file writes a JSON file, and a fresh instance constructed with the same parameter_config_path loads it on startup and produces an identical end-effector pose, which is how a calibration tuned in the UI survives a restart of the session.
banner("SUMMARY")
for name, res in RESULTS.items():
print(f" {name:<78s} {res}")
print("""
Where to go next
- Put a headset on the graph: swap the ValueInput leaves for HandsSource /
ControllersSource and run the same pipeline inside TeleopSession with
CloudXRLauncher (examples/teleop/python/gripper_retargeting_example_simple.py).
- Record and replay: McapRecordingConfig / McapReplayConfig on TeleopSessionConfig
replay a .mcap through this exact graph with no headset attached.
- Drive a simulator: the action vector from Step 5 is what Isaac Lab's
IsaacTeleopDevice consumes; the TensorReorderer order must match the env.
- Tune live: MultiRetargeterTuningUI (isaacteleop[ui]) edits the same
ParameterState objects you set by hand in Steps 3 and 8.
""")
The summary collects the headline of every section from the RESULTS dictionary that the section decorator filled in, so a skipped or failed step shows up here with its error instead of silently disappearing from the run.
In conclusion, we built a working Isaac Teleop pipeline without any of the hardware the framework is designed around, because everything below the device layer is plain Python operating on typed tensor groups. The type system caught wrong dtypes and unwritten slots at the point of the mistake; absent optional groups gave every node one explicit way to learn that tracking was lost; and ParameterState let us tune behavior between frames and keep it on disk. The built-in gripper, SE(3), locomotion and TriHand retargeters behaved as documented under synthetic input, and composing them with ValueInput leaves, TensorReorderer and OutputCombiner produced the same action vector a headset session would feed to Isaac Lab, with each leaf computed once per frame. Where hardware enters, the graph does not change: the leaves become HandsSource and ControllersSource inside a TeleopSession, a recording becomes an MCAP replay, and the anchor transform we applied by hand arrives from the simulator.
Check out the FULL CODES here. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
