Extrinsic Summarization Evaluation: A Decision Audit Task

Gabriel Murray, Thomas Kleinbauer, Peter Poller, Steve Renals, Jonathan Kilgour, Tilman Becker

In: Andrei Popescu-Belis, Rainer Stiefelhagen (Hrsg.). Machine Learning for Multimodal Interaction. Machine Learning and Multimodal Interaction (MLMI) 5th International Workshop, MLMI 2008 September 8-10 Utrecht Netherlands Seiten 349-361 Lecture Notes in Computer Science (LNCS) 5237 ISBN 978-3-540-85852-2 Springer Heidelberg 2008.


In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and an analysis of participant browsing behaviour.

Weitere Links

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence