A Multimodal Dialogue System for Medical Decision Support inside Virtual Reality

Alexander Prange, Margarita Chikobava, Peter Poller, Michael Barz, Daniel Sonntag

In: Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. Annual SIGdial Meeting on Discourse and Dialogue (SIGdial-17) 18th August 15-17 Saarbrücken Germany Pages 23-26 Association for Computational Linguistics 8/2017.


We present a multimodal dialogue system that allows doctors to interact with a medical decision support system in virtual reality (VR). We integrate an interactive visualization of patient records and radiology image data, as well as therapy predictions. Therapy predictions are computed in real-time using a deep learning model.


2017_A_Multimodal_Dialogue_System_for_Medical_Decision_Support_in_Virtual_Reality.pdf (pdf, 1 MB )

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz