This Robot Watched a Short Video and Figured Out the Rest Itself

Key points

• Generalist AI, a Cambridge, Massachusetts startup, has built robot arms that can learn a new physical task after watching one short video clip.
• The robots currently complete tasks successfully about 59 percent of the time, well below the 99 percent threshold the company says it needs for real-world use.
• All three co-founders previously worked at Google DeepMind or Boston Dynamics, two of the most respected names in robotics.
• Generalist built its AI model entirely from scratch rather than adapting an existing open-source system.
• A Georgia Tech roboticist who reviewed the work called Generalist "the closest to something that's deployable" among rivals chasing general-purpose robots.

Robot arms that can pick up new skills on the spot, without months of specialised training, have been one of the hardest problems in machine learning. Wired AI recently got a first-hand look at a startup that claims to be closer to solving it than almost anyone else.

Generalist AI, based in Cambridge, Massachusetts, showed a reporter its robot arms performing tasks they had never been explicitly trained on: stacking cups, sorting blocks into bowls, and more. The machines were given a short instructional video instead of thousands of practice repetitions, which is the usual method.

What did the robots actually do?

The demonstrations were concrete and, in a few moments, genuinely surprising. One arm was asked to sweep a block into a bowl using a dustpan and brush. When the brush was taken away, it picked up the dustpan and used it like a brush instead, flicking the block into the bowl. Nobody programmed that workaround.

A two-armed robot watched a short clip of someone unzipping a purse and removing banknotes. It then unzipped a different style of purse and pulled the notes out. When its right gripper couldn't get a good grip, it switched to its left, trying a new angle. An engineer watching said the robot had never done that before.

In a separate late-night session caught on video, an engineer left cups on a table in front of a two-armed robot just to see what would happen. The robot started stacking them alongside the engineer, finishing the pile neatly on its own.

Co-founder and CEO Pete Florence compared the moment to the release of GPT-3, the large language model, meaning a text-generating AI system, that OpenAI published in 2020. That model could attempt tasks it had never seen simply by reading a description. Generalist is trying to do something similar but in the physical world.

How does the training work?

Generalist collects movement data at scale using custom gloves that look like robot pincers and carry small cameras. Workers in Mexico and elsewhere wear the gloves to perform everyday physical chores, recording exactly how a human hand moves through a task. The company has gathered a large library of this data and built its AI model on top of it from scratch.

The bet, according to Stanford roboticist Karen Liu, is that collecting broad physical-interaction data without tying it tightly to one specific robot body gives the model a more general understanding of how objects behave.

Should ordinary people care yet?

Not immediately, but the trajectory matters. A 59 percent success rate means the robot fails nearly half the time, which is not good enough for a factory floor or a hospital supply room. Generalist says it knows this.

The longer-term picture is different. If a robot can learn a new task from a single video rather than weeks of programming, the cost of deploying robots in manufacturing or warehousing drops sharply. That will eventually affect jobs that involve repetitive physical handling, packing, or sorting.

For now, the technology is still in the lab. But Danfei Xu, a roboticist at Georgia Tech who follows the company closely, says Generalist has "pushed this to the extreme" in a way that few rivals have, and called it the closest team to a genuinely deployable product.