Publikation
Do You (Dis)agree With Me? Modelling Implicit User Disagreement in Human–AI Interaction Using Gaze Data
Abdulrahman Mohamed Selim; Omair Shahzad Bhatti; Amr Gomaa; Michael Barz; Daniel Sonntag
In: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. ACM International Conference on Human Factors in Computing Systems (CHI-2026), April 13-17, Barcelona, Spain, Association for Computing Machinery (ACM), 2026.
Zusammenfassung
The widespread use of generative AI has led to increased focus on human–AI interaction. However, AI systems can generate unexpected outputs, leading to disagreement or human–AI conflict. This paper focuses on modelling user disagreement using machine learning (ML) by observing users’ implicit viewing behaviour. We conducted a controlled study with 30 participants evaluating captions from a simulated ML image-captioning system. Participants indicated agreement or disagreement with each caption while we recorded their gaze and facial-expression data, which we used to predict (dis)agreement. We show that unimodal gaze-based personalised modelling (0.684 average balanced accuracy) outperforms generalised modelling (0.570), whereas multimodal approaches did not improve performance. Our exploratory post hoc gaze-based analysis highlights the importance of feature selection and temporal dynamics, which help guide system design and future work. We release the dataset to support reproducibility and further work. Due to the nature of this research, we also discuss the potential ethical and privacy implications of continuous passive gaze and facial monitoring.
Projekte
- MASTER - MASTER: Mixed reality ecosystem for teaching robotics in manufacturing
- No-IDLE - Interactive Deep Learning Enterprise
- NoIDLEChatGPT - No-IDLE meets ChatGPT