Publikation
Extrinsic Summarization Evaluation: A Decision Audit Task
Gabriel Murray; Thomas Kleinbauer; Peter Poller; Steve Renals; Jonathan Kilgour; Tilman Becker
In: Andrei Popescu-Belis; Rainer Stiefelhagen (Hrsg.). Machine Learning for Multimodal Interaction. Machine Learning and Multimodal Interaction (MLMI), 5th International Workshop, MLMI 2008, September 8-10, Utrecht, Netherlands, Pages 349-361, Lecture Notes in Computer Science (LNCS), Vol. 5237, ISBN 978-3-540-85852-2, Springer, Heidelberg, 2008.
Zusammenfassung
In this work we describe a large-scale extrinsic evaluation of
automatic speech summarization technologies for meeting speech. The
particular task is a decision audit, wherein a user must satisfy a complex
information need, navigating several meetings in order to gain an understanding
of how and why a given decision was made. We compare the
usefulness of extractive and abstractive technologies in satisfying this
information need, and assess the impact of automatic speech recognition
(ASR) errors on user performance. We employ several evaluation
methods for participant performance, including post-questionnaire data,
human subjective and objective judgments, and an analysis of participant
browsing behaviour.