Robot Show Runtime

The system behind the Robot Darkroom. Where Motion-Core plays keyframed animation across several UFactory arms in real time, this composes a show as it runs: one arm, modular clips, a state machine choosing between them, with the lights and the people in the room as inputs.

1. The room, as a twin

The lab was measured with the arm itself: its tool touched to the floor and walls, its own kinematics agreeing with the controller to about 1.1 mm. A RoomPlan scan filled in the rest, aligned to the probed walls within 0.47°. The room is kept as a USD stage that Isaac Sim loads as it is, lit to match the room from photographs, so a render reads like the place.

Every clip is checked against this room before it plays: the arm as capsules fitted to its meshes, against the measured floor, walls and furniture. A clip that comes too close fails the build.

2. Rehearsal in Isaac Sim

The whole show runs in Isaac Sim before it runs on the arm, and the light runs with it. The UV strip was modelled on the tool before it was ever mounted, so the twin shows where each LED sits as the arm moves, what TouchDesigner's animation looks like along it, how a scan writes the paper, and how the ceiling light plays against all of it, live, as TouchDesigner sends it. The cable for the strip is simulated too, to find where it can be clipped to the arm without binding. The full library replayed with 0 contacts and a worst tracking error of 0.96°.

The scan's layout was chosen here too, from where the guests would stand.

3. A library, not a timeline

Nothing in the show is keyframed end to end. The clips are generated in Houdini with the Houdini Robot Toolkit: gestures aimed at a guest at one position, and at another, dance phrases built from Laban's efforts (punch, glide, float, wring and the rest), each labelled afterwards by what it measures as, not only by what it was asked to be.

4. Composing the show live

A build compiles them into 85 segments (70 idle clips at four positions, the moves between them, and the scan) and checks every one against the room. Profiling cut the build from 33 min to 141s, with the output identical to the bit.

At run time a state machine picks the next clip, weighted, never twice in a row, along an energy arc over 3 min, and finishes the running clip before a scan or a guest takes over. The arm gets one continuous ServoJ stream at 125 Hz.

Clips are timed against the arm's measured accelerations, 300 deg/s² on the first three joints and 600 on the wrist, so they play at their designed length. Timed for velocity alone, an earlier export of the same curve would have played 8.54× slower than drawn.

5. The arm, inside TouchDesigner

The runtime tells TouchDesigner where the arm is and what it is doing over OSC: the scan position, the strip's distance to the paper, the beat of the current clip. TouchDesigner answers with the strip's LEDs and the room's ceiling light. The arm becomes one more thing in a scene the studio already lights and cues.

What broke on the way

• A 30 min stream stuttered on the controller simulator: 159 skipped ticks in bursts. The cause was Python's garbage collector walking the run's own log. Flat arrays and a frozen heap after start-up made the rerun 0 skips.
• The floor came out tilted 6° when the room was probed on the real arm: three nearly colinear points put the plane 17 cm off. Probing now warns on a poor spread, and fits floors level and walls vertical.
• A move was refused with a 11.28° mismatch between the arm and its model: a workpiece frame had been left active on the controller. The player now refuses to move with one applied and says which to reset.
• The arm stopped 3.4 h into the party on a singular pose: moves between the low and high positions cross the wrist singularity, and the FR20 halts there about once in 317 crossings. The runtime now recovers by itself: it clears the fault, makes a checked move to the end of the move it was in, and the show goes on. Never during a scan, at most three times a run, and any other fault, a collision above all, stays stopped.