Better Metrics to Automatically Predict the Quality of a Text Summary

Abstract

In this paper we demonstrate a family of metrics for estimating the quality of a text summary relative to one or more human-generated summaries. The improved metrics are based on features automatically computed from the summaries to measure content and linguistic quality. The features are combined using one of three methods—robust regression, non-negative least squares, or canonical correlation, an eigenvalue method. The new metrics significantly outperform the previous standard for automatic text summarization evaluation, ROUGE.

Keywords

• multi-document summarization
• update summarization
• evaluation
• computational linguistics
• text processing