ART

Robots Learn to Grab by Watching Humans

University Research Laboratory in Computer Science and RoboticsThu Nov 19 2026

Robots need to know how to grab things, but they don't see the world the same way people do. When a person reaches for a cup, their hand shape tells them what to do. A robot's hand looks different, so the same clues don't work. That mismatch makes it hard for robots to copy human moves. Kids can picture it like a puzzle where the pieces fit only when the shapes match.

To solve this, we built a huge collection of 3D shape guides called HPAD. It holds 2,137 items spread over 80 groups, each tagged with both a person's guess and a robot's verified label. The set includes 13 basic move types and more than 27,000 pairings of objects with actions.

Our method, named ART, first runs a trust filter to drop wrong human clues. Then it rebuilds the clues step by step and smooths them with a network trick that matches the robot's shape layout. This fixes the shape layout problem between human and robot views.

We tested ART on a six‑move joint arm with a two‑finger hand, trying 12 objects and four move types across 2,400 tries. The system succeeded 86.5 percent of the time, showing that the idea works in real life.

actions