DEXTEROUS MANIPULATION · HAND–OBJECT INTERACTION
Scaling Dexterous Manipulation Through
Iterative Hand-Object Interaction Generation
* Corresponding authors
01 / THE IDEA
Learning dexterous manipulation calls for continuous, fine-grained hand–object interactions. Capturing every motion for every object is impractical.
DexBeyond brings together a large-scale real-world dataset, a conditional interaction generator, and a self-improving data flywheel. We expand motion diversity for captured objects, then turn proposals for unseen objects into feasible robot interactions—without additional real-world demonstrations.
DEXBEYOND IN MOTION
From captured human motion to generated interactions and real-world dexterous manipulation.
02 / DEXBEYOND-D
Fine-grained finger coordination, diverse geometries, and synchronized 3D hand and object poses.
Geometric primitives, everyday rigid objects, and articulated or multi-part objects across 77 categories.
Reorientation, translation, and object-specific functional behaviors under four wrist settings.
6.26 million frames at 30 fps, with reconstructed hand poses, object poses, and corresponding object meshes.
03 / GENERATE & IMPROVE
Expand how known objects move. Learn to manipulate objects never demonstrated.
DEXBEYOND-G
A conditional diffusion model synthesizes continuous hand–object trajectories from object geometry and motion descriptions. Reconstruction, smoothness, and geometric objectives preserve contacts and reduce penetration.
Object geometry + motion description Interaction sequence
DEXBEYOND-FW
For unseen objects, geometric refinement and physical feasibility detection prepare generated references for controller training. Diverse successful simulation rollouts update the generator for the next round.
Generate Refine Execute Update
04 / EXPERIMENTS
Generated interactions improve downstream controller success in simulation and the real world.
Mean success rate with captured data versus captured + generated data.
+8.9 percentage pointsMean success rate on the 22-DoF SharpaWave anthropomorphic hand.
+12.3 percentage pointsUNSEEN-OBJECT GENERALIZATION
Across four held-out objects, mean controller success rises over four flywheel rounds. No captured demonstrations from the held-out categories are used.
REFERENCE
BibTeX for this work.
@article{zuo2026dexbeyond,
title={DexBeyond: Scaling Dexterous Manipulation Through
Iterative Hand-Object Interaction Generation},
author={Zuo, Yi and Wu, Yingying and Chen, Zeyuan and
Li, Puhao and Liu, Fang and Jiao, Licheng and
Jia, Baoxiong and Huang, Siyuan},
year={2026}
}