Extrinsic Summarization Evaluation: A Decision Audit Task

Gabriel Murray, Thomas Kleinbauer, Peter Poller, Tilman Becker, Steve Renals, Jonathan Kilgour

In: ACM Transactions on Speech and Language Processing (TLSP) 6 2 Seiten 1-29 ACM 10/2009.


In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must sat- isfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recog- nition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and a detailed analysis of participant browsing behavior. We find that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily. Results also indicate that users consider extractive summaries to be intu- itive and useful tools for browsing multimodal meeting data. We discuss areas in which automatic summarization techniques can be improved in comparison with gold-standard meeting abstracts.

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence