Multilinguality and Language Technology

TR Team

We are a research group at the MLT-Lab at DFKI specialized in dialogue processing for human-robot collaboration and for teamwork support. 

The goal of dialogue processing is to build systems that can interpret and participate in natural spoken dialogue interaction. Dialogue processing comprises automatic speech recognition, natural language understanding and interpretation of multimodal inputs, dialogue state modeling, dialogue management, generation and synthesis of verbal and multimodal outputs. Dialogue is situated, social and personal. We therefore also model how what is said relates to and is grounded in the context of preceding interaction(s), knowledge about the world and the physical situation; what roles the dialogue participants have and what relationships exit between them; and how the current interaction reflects accumulated interpersonal experience.

We have extensive experience with dialogue processing in a variety of applications, including: home and office service; healthcare; human-robot collaboration in hybrid industrial production teams; robot-assisted disaster response.  We have integrated dialogue processing into mobile robot-based systems in various EU and national projects. In these contexts we have worked intensively with end users, including first responders (e.g., firefighters), patients (e.g., diabetic children) and caregivers.


Some current projects:

A-DRZ

A-DRZ is setting up a competence center bundling scientific and technical expertise in robot-assisted disaster response. The TR group develops methods for interpreting verbal communication between the members of the response team and anchoring it in mission process models.  

More details

Project website: rettungsrobotik.de

Video: https://www.youtube.com/watch?v=ZWD6XDIzc8w

YouTube-Channel: https://www.youtube.com/channel/UCdFmco0w3RFACmCtygMNiMA

NotAs

NotAs investigates the use of language technology to support emergency call operators in handling foreign language calls. The TR group develops machine translation and interpretation of emergency calls.


More details
 

IMPRESS

IMPRESS (BMBF; 2020-2023) investigates the integration of semantic knowledge into linguistic and multimodal embeddings and the impact on selected downstream tasks in the processing of speech and image/video.  

More details

CORA4NLP

Co(n)textual Reasoning and Adaptation for Natural Language Processing

Period: 01.10.2020-30.09.2023, funded by BMBF.

Project website

XAINES

XAINES (01.09.2020 - 31.08.2024): Explaining AI with Narratives

Funded by BMBF (01IW20005).

Project website


Selected past projects

INTUITIV

INTUITIV investigates intuitive nonverbal and informative verbal robot-human communication. The TR group develops methods for situated spoken natural language interaction with focus on indoor navigation dialogue.

More details
 

PAL

Personal Assistant for Healthy Lifestyle

PAL (EU; 2015-2019) developed an integrated system providing comprehensive, prolonged, personalized and context-sensitive support in order to advance the self-management of children with Type 1 diabetes, so that  adequate shared patient-caregiver responsibility is established before the child reaches adolescence. The TR group contributed dialogue management, long-term interaction memory and spoken input and output processing.
More details


Concept overview video: https://www.youtube.com/watch?v=v_7RqBq6tv0 
Youtube channel: https://www.youtube.com/channel/UC7gsnMoFvw9Nztvk0XmSSBQ

TRADR

Long-Term Human-Robot Teaming for Robot-Assisted Disaster Response

TRADR (EU; 2013-2017) investigated how a team consisting of humans and several ground and airborne robots develops situation awareness and experience gradually over multiple synchronous and asynchronous sorties in a  long-term mission. TRADR addressed persistence of environment models, multi-robot action models and human-robot collaboration over time. The TR group contributed dialogue processing for team communication and interactive mission reporting. 

More details

Project website: http://www.tradr-project.eu

Videos:
TRADR Joint Exercise Dortmund 2015: mission concept: https://www.youtube.com/watch?v=4z86nUlgEqc

TRADR Joint Exercise 2016 Prague: overview of technology: https://www.youtube.com/watch?v=mJpLdiC75ns

Final review: https://www.youtube.com/watch?v=W8onMryGTEA

View more on TRADR youtube channel: https://www.youtube.com/channel/UCvrPijjrCFGWdJIHkKbFK-A
 

HySociaTea

Hybrid Social Teams for Long-Term Collaboration in Cyber-Physical Environments

HySocieTea 
(BMBF, 2014-2016) examined collaboration and communication in teams consisting of technologically augmented humans, autonomous robots, virtual characters and softbots, working on joint tasks in flexible production processes. The TR group contributed to the development of an integrated team collaboration system, including the processing for spoken dialogue between humans and artificial agents and otology-based modeling and reasoning for contextual reference resolution and generation. 


More details

Video: https://youtu.be/mdu60HFZrP8


Selected publications

  • Ivana Kruijff-Korbayová; Robert Grafe; Nils Heidemann; Alexander Berrang; Cai Hussung; Christian Willms; Peter Fettke; et al. (2021): German Rescue Robotics Center (DRZ): A Holistic Approach for Robotic Systems Assisting in Emergency Response. In: 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR-2021), October 25-27, New York City, NY, USA, Pages 138-145, ISBN 978-1-6654-1764-8, DOI 10.1109/SSRR53300.2021.9597869, IEEE Xplore, 10/2021.
  • Anindita Ghosh; Noshaba Cheema; Cennet Oguz; Christian Theobalt; Philipp Slusallek (2021): Synthesis of Compositional Animations from Textual Descriptions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). International Conference on Computer Vision (ICCV-2021), October 11-17, Virtual, Pages 1396-1406, IEEE, 10/2021. https://github.com/anindita127/Complextext2animation / https://openaccess.thecvf.com/content/ICCV2021/supplemental/Ghosh_Synthesis_of_Compositional_ICCV_2021_supplemental.pdf / https://openaccess.thecvf.com/content/ICCV2021/papers/Ghosh_Synthesis_of_Compositional_Animations_From_Textual_Descriptions_ICCV_2021_paper.pdf
  • Ivana Kruijff-Korbayová; Luigi Freda; Mario Gianni; Valsamis Ntouskos; Vaclav Hlavac; Vladimir Kubelka; Erik Zimmermann; Hartmut Surmann; Kresimir Dulic; Wolfgang Rottner; Emanuele Gissi: Deployment of Ground and Aerial Robots in Earthquake-Struck Amatrice in Italy (brief report). In: Kamilo Melo (Hrsg.). Proceedings of the 2016 IEEE International Symposium on Safety, Security and Rescue Robotics. IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR-2016), October 23-27, EPFL, Lausanne, Switzerland, Pages 278-279, ISBN 978-1-5090-4349-1, IEEE, 10/2016.
  • N. Skachkova, and I. Kruijff-Korbayová, I. (2020) Reference in Team Communication for Robot-Assisted Disaster Response: An Initial Analysis. In Proceedings of the 3rd CRAC Workshop on Computational Models of Reference, Anaphora and Coreference, COLING 2020. https://www.aclweb.org/anthology/2020.crac-1.13
  • C. Willms, C. Houy, J.-R. Rehse, P. Fettke, I. Kruijff-Korbayová, I. (2019): Team communication processing and process analytics for supporting robot-assisted emergency response. Proceedings of 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR-2019), Würzburg, Germany. https://ieeexplore.ieee.org/document/8848976
  • T. Anakina, I. Kruijff-Korbayová (2019): Dialogue act classification in team communication for robot assisted disaster response. Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Stockholm, Sweden, September 11-13, 2019.
 https://www.aclweb.org/anthology/W19-5946/
  • Kiefer, B., Welker, A., Biwer, C. (2019). VOnDA: A Framework for Ontology-Based Dialogue Management. Proc. of International Workshop on Spoken Dialogue Systems Technology (IWSDS), Springer. https://arxiv.org/abs/1910.00340
  • Schwartz, T., Zinnikus, I., Krieger, H. U., Bürckert, C., Folz, J., Kiefer, B., et al. (2016). Hybrid teams: 
flexible collaboration between humans, robots and virtual agents. In German Conference on Multiagent System Technologies (pp. 131-146). Springer, Cham. 
 https://www.dfki.de/web/forschung/projekte-publikationen/publikationen-uebersicht/publikation/8634/
  • Kruijff-Korbayová, F. Colas, M. Gianni, F. Pirri, J. de Greeff, K. Hindriks, M. Neerincx, P. Ögren, T. Svoboda and R. Worst. TRADR Project: Long-Term Human-Robot Teaming for Robot Assisted Disaster Response. In KI - Künstliche Intelligenz, German Journal on Artificial Intelligence, June 2015, Volume 29, Issue 2, pp 193-201. https://hal.archives-ouvertes.fr/hal-01143484/document
  • G.-J. Kruijff, I. Kruijff-Korbayová, S. Keshavdas, B. Larochelle, M. Janíček, F. Colas, M. Liu, F. Pomerleau, R. Siegwart, M. A. Neerincx, R. Looije, N. J. J. M. Smets, T. Mioch, J. van Diggelen, F. Pirri, M. Gianni, F. Ferri, M. Menna, R. Worst, T. Linder, V. Tretyakov, H. Surmann, T. Svoboda, M. Reinstein, K. Zimmermann, T. Petříček, V. Hlaváč.  Designing, developing, and deploying systems to support human-robot teams in disaster response.  Advanced Robotics, 28 (23), Pages 1547-1570, Taylor & Francis, London, 12/2014. https://hal.archives-ouvertes.fr/hal-01143476/document
  • T. Belpaeme, P. Baxter, R. Read,  R. Wood, H. Cuayáhuitl, B. Kiefer, et al. Multimodal Child-Robot Interaction: Building Social Bonds. Journal of Human-Robot Interaction, 1(2), 33-53. 2013. https://dl.acm.org/doi/10.5555/3109688.3109691

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz