Publikation
In-Hand Object Pose Estimation via Visual-Tactile Fusion
Felix Nonnengießer; Alap Kshirsagar; Boris Belousov; Jan Peters
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2506.10787, Pages 1-8, arXiv, 2025.
Zusammenfassung
Accurate in-hand pose estimation is crucial for
robotic object manipulation, but visual occlusion remains
a major challenge for vision-based approaches. This paper
presents an approach to robotic in-hand object pose estima-
tion, combining visual and tactile information to accurately
determine the position and orientation of objects grasped by
a robotic hand. We address the challenge of visual occlusion
by fusing visual information from a wrist-mounted RGB-
D camera with tactile information from vision-based tactile
sensors mounted on the fingertips of a robotic gripper. Our
approach employs a weighting and sensor fusion module to
combine point clouds from heterogeneous sensor types and
control each modality’s contribution to the pose estimation
process. We use an augmented Iterative Closest Point (ICP)
algorithm adapted for weighted point clouds to estimate the 6D
object pose. Our experiments show that incorporating tactile
information significantly improves pose estimation accuracy,
particularly when occlusion is high. Our method achieves an
average pose estimation error of 7.5 mm and 16.7 degrees,
outperforming vision-only baselines by up to 20%. We also
demonstrate the ability of our method to perform precise object
manipulation in a real-world insertion task.