Publication
Dynamically Updating and Interrelating Representations of Visual and Linguistic Discourse
John Kelleher; Josef A. van Genabith; Fintan Costello
In: Artificial Intelligence (AIJ), Vol. 67, No. 1-2, Pages 62-102, Elsevier, 2005.
Abstract
The fundamental claim of this paper is that salience—both visual and linguistic—is an important
overarching semantic category structuring visually situated discourse. Based on this we argue
that computer systems attempting to model the evolving context of a visually situated discourse
should integrate models of visual and linguistic salience within their natural language processing
(NLP) framework. The paper highlights the importance of dynamically updating and interrelating
visual and linguistic discourse context representations. To support our approach, we have developed a real-time, natural language virtual reality (NLVR) system (called LIVE, for Linguistic Interaction with Virtual Environments) that implements an NLP framework based on both visual and linguistic salience. Within this framework saliency information underpins two of the core subtasks of NLP: reference resolution and the generation of referring expressions. We describe the theoretical basis and architecture of the LIVE NLP framework and present extensive evaluation results comparing the system’s performance with that of human participants in a number of experiments.