Skip to main content Skip to main navigation

Publication

Context-Aware Robotic Assistance for Workers Using Intention Recognition and Semantic Digital Twin

Snehal Walunj; Parsha Pahlevannejad; Michael Sintek; Juan Carlos Saborio; Prerna Garg; Hooman Tavakoli Ghinani; Christiane Plociennik; Nicolas Grossmann; Ansgar Bernardi; Martin Ruskowski; Joachim Hertzberg; Andreas Dengel
In: Dhananjay Singh; Jan-Willem van 't Klooster; Uma Shanker Tiwary (Hrsg.). Intelligent Human Computer Interaction. IADIS International Conference on Interfaces and Human Computer Interaction (IHCI-2024), May 29, Cham, Pages 92-104, ISBN 978-3-031-88705-5, Springer Nature Switzerland, 5/2025.

Abstract

Context-aware assistance systems are becoming essential for aiding decision-making and task management for the workers. However, integrating these systems with various other entities such that they interact seamlessly, poses a challenge. This paper proposes a solution that leverages advanced interaction protocols and context-aware robotic assistance to ensure smooth and effective collaboration in the factory environment supported by three pillars: 1) an ontology-based Digital Twin (DT) representing the semantic dimension and providing object-specific information, 2) an egocentric perception system using HoloLens2 to observe the worker with individually tailored dataset for context recognition, and 3) a robotic assistant guided by an intention-recognition module that responds to emerging worker needs, based on behavioral and environmental observations. In a manual assembly scenario, our framework provides useful contextual information as well as automated and adaptive robotic assistance. We present the results from a user experiment which was conducted to compare the stress level of participants with and without the use of the assistance system in a manual assembly scenario. Physiological sensors from the Empatica E4 wrist-watch were used to collect physiological data on which the stress levels were detected using a deep learning model. Additionally, the user ratings on the System Usability Scale (SUS) were used to assess the usability and acceptance of robotic-assistance technology.

Projects

More links