Publication
Capturing Task-related Information for Text-based Grasp Classification using Fine-tuned Embeddings
Niko Kleer; Leon Weyand; Michael Feld; Klaus Berberich
In: Elmar Nöth; Aleš Horák; Petr Sojka (Hrsg.). 27th International Conference, TSD 2024, Brno, Czech Republic, September 9–13, 2024. International Conference on Text, Speech and Dialogue (TSD-2024), September 9-13, Brno, Czech Republic, Pages 288-299, ISBN 978-3-031-70566-3, Springer International Publishing, Switzerland, 9/2024.
Abstract
Manipulating objects with a robotic hand or gripper is a challenging task that can be supported by knowledge about the object, such as textual descriptions. Even with such knowledge, there remain numerous possibilities for applying an appropriate grasping gesture. This ambiguity can be reduced by providing information about the intended task, aiding robots in making the choice of a suitable grasp less arbitrary and more robust. This work investigates using word embeddings in the context of grasp classification for multi-fingered robots. Instead of predicting grasping gestures without specifying the intended task, our work combines a description of the properties of an object and task-related information. We demonstrate that a systematically generated dataset and fine-tuned context embeddings can compete with existing models that do not consider object manipulation. Our best model achieves a micro f1 score of 0.774 and macro f1 score of 0.731 while distinguishing between over 40 tasks.
Projekte
CAMELOT - Continuous Adaptive Machine-Learning of Transfer of Control Situations,
SC_Kleer - MultiGrasp: Multimodal Grasp Type Prediction for Dexterous Multi-fingered Robotic Grasping
SC_Kleer - MultiGrasp: Multimodal Grasp Type Prediction for Dexterous Multi-fingered Robotic Grasping