Teaching a Robot Arm to Pick and Place

I teach an SO-101 arm to pick a cube off a table and drop it in a bowl, from 50 teleoperated demonstrations. Six degrees of freedom, two 640×480 RGB streams at 30 frames per second, no reward signal and no simulator — the policy either moves the real arm correctly or it does not.

The headline is 92% success over 50 scored physical rollouts. Most of the work below is about why I do not fully trust that number yet, and what I built to find out.

The result

Protocol	Trials	Success	Grasp rate	Mean placement error
Fixed cube position	50 scored	92% (46/50)	92%	2.49 cm
Randomized cube position	50 planned	pending	pending	pending

Placement error is measured on successful trials only.

The failures are more informative than the successes. All four were no-grasp: the gripper closed on nothing and the trial ended there. Zero trials grasped the cube and then dropped it, and zero placed it somewhere other than the bowl. Once the policy had the cube, it finished the task every single time.

That is a clean split, and it points at perception rather than control. The arm knows how to move. What it sometimes fails to do is decide where.

Why 92% may not mean what it looks like

Before running anything else, I template-matched the first frame of all 50 training episodes to find where the cube actually starts.

It starts in the same place every time. Every demonstration puts the cube on the same paper marker, with under a centimetre of spread across the entire dataset.

Which means the policy has never once seen the cube anywhere else. The most likely thing it learned is not a visually-conditioned grasp but a near open-loop trajectory to one location — a location that happens to be where the cube always is. Run that trajectory against a cube sitting exactly there and it succeeds 92% of the time. That is consistent with a policy that understands the scene, and equally consistent with one that has memorised a single arm motion.

A fixed-position protocol cannot tell those apart. It would report 92% and call the policy solved.

So the randomized protocol exists to separate them: the same policy, scored against the cube placed at 50 measured offsets from the trained position. If success falls off sharply with distance, the hypothesis holds and the fix is data collection rather than more gradient steps — demonstrations with the cube deliberately scattered across the workspace. If it holds up, the policy is doing more than I am giving it credit for.

Either way the number that matters next is the randomized one, and it is a hypothesis with a test attached rather than a caveat in a footnote.

Two policies, one protocol

I trained a second policy to have something to compare against. ACT trains from scratch on an ImageNet-pretrained ResNet18 backbone. SmolVLA finetunes lerobot/smolvla_base, updating 99.9M parameters while the SmolVLM2-500M backbone stays frozen. Same 50 demonstrations, same task, same protocol.

	ACT v2	SmolVLA
Initialization	from scratch	finetune of smolvla_base
Training steps	60,000	20,000
Trainable parameters	51.7M	99.9M of 450M
Wall-clock	~13 h	5.7 h
Action MAE, 200 frames	1.49	1.06
Joint accuracy (±5)	94.8%	97.8%
Frame accuracy (±5)	72.0%	89.0%
Inference latency	18 ms	251 ms
Fixed-position success	92%	not yet scored

SmolVLA fits the demonstrations substantially better on a third of the gradient steps. On every offline metric it is the better policy.

It also answers 14 times slower. At 251 ms per query, closed-loop behaviour on a moving scene is an open risk rather than a known quantity, and that is exactly the kind of thing an offline metric cannot see. So the offline win stays an offline win. I am not reporting SmolVLA as better than ACT until it has been scored on the same physical protocol.

The checkpoint validation loss would have thrown away

A third run held out 5 of the 50 episodes — every tenth index rather than the last five, so that operator fatigue and lighting drift late in a recording session do not get quietly confounded with generalization.

Validation loss bottomed at 0.122 somewhere around steps 5,000 to 7,500, then climbed to 0.153 by step 20,000 while training loss kept falling. Textbook overfitting, and the textbook response is to ship the early checkpoint.

The early checkpoint was worse. On held-out action accuracy the final checkpoint beat the best-val/loss checkpoint by a wide margin: 1.20 MAE against 2.01.

Flow-matching validation loss and teacher-forced action accuracy do not pick the same checkpoint, and on this run they disagreed by enough to matter. Selecting on validation loss alone — the default, the thing every tutorial tells you to do — would have shipped the worse policy and produced a curve that looked responsible while doing it.

The harness

Most of the engineering here is measurement, not training.

• Resumable rollouts. The evaluation script resumes mid-protocol, refuses to run against a stale eval cache, and returns the arm to a known start pose between trials. A 50-trial run interrupted by a camera dropout does not mean rescoring from zero.
• Placement error from video, not from eyeballing it. Two clicks on the bowl rim calibrate centimetres-per-pixel against its known 11.5 cm diameter, and each trial’s final frame is measured for cube-to-bowl offset. That is where 2.49 cm comes from — and why the failure modes separate cleanly: every success lands under 5 cm, while the four no-grasp trials sit at 13 to 16 cm.
• A scoresheet locked before scoring starts. Task spec, trial count, and the success definition — autonomous grasp and release in the bowl — are fixed in advance, with a failure taxonomy of no-grasp, grasped-then-dropped, and wrong-placement. The harness counts failure modes rather than summarising them afterwards, which is what makes the paragraph above possible.
• Data validated before any GPU time. Episode and frame alignment, H.264 decodability, resolution, and finite action and proprioceptive tensors, all checked up front. All 50 episodes and 49,633 frames of the v1 dataset passed with zero integrity failures before training started.
• Long runs that survive interruption. Configurable augmentation, Weights & Biases tracking, 5,000-step checkpointing and resume support, so a 13-hour run is not a 13-hour bet on nothing going wrong.

Two runs I threw out

SmolVLA’s physical scoring is not done, and twice that was my own doing.

The first attempt I voided because a 3-trial sanity check showed the policy stalling for 11 to 51 seconds before initiating any motion at all. The cause was not the policy: objects were sitting in the top camera’s field of view that appear in no demonstration anywhere in the training set. The background was out of distribution and the policy waited. The second attempt I voided when a camera dropped out at episode 17 of 50.

Voiding a run costs a day. Reporting a contaminated one costs the result, and every conclusion drawn from it afterwards. The scoresheet is locked before scoring precisely so that this decision is made in advance rather than while staring at a number I would like to keep.

What’s mine, and what’s upstream

The repository is a fork of LeRobot, and src/lerobot/ is upstream code except where noted below. Mine is the training, evaluation and measurement tooling in scripts/; the experiment write-ups and scoresheets; the committed calibration for this pair of arms; the Windows bring-up guide; and one patch to the motor bus.

That patch is worth a paragraph, because it cost a week before it was understood. After a power cycle, the gripper (ID 6) and wrist_roll (ID 5) could both come back as ID 5 — a bus collision that surfaces as the thoroughly unhelpful Missing motor IDs: 5, 6. STS3215 EEPROM has to be unlocked, written, and then explicitly re-locked for the write to commit. Upstream did not always re-lock, so on some firmware batches the ID assignment silently reverted the next time the arm was powered on, and the arm that worked yesterday does not work today.

Where it stands

ACT is trained and scored on the fixed protocol. The randomized protocol is generated and the harness is ready; those trials are not yet scored. SmolVLA is trained and its offline metrics are measured, with physical trials pending.

The next experiment is the randomized run, followed by re-recording demonstrations with scattered cube placement if it confirms the data-coverage hypothesis.

Until then the honest summary of this work is: 92% at one cube position, and a measurement setup built to find out whether that is the ceiling.