Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation
報告題目：Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation
時 間：2021年10月25日（周一）上午09:30
地 點：計算所948會議室
報 告 人：孫飛
報告人簡介：孫飛，畢業于中科院計算所，目前在阿里巴巴達摩院智能計算實驗室從事推薦系統、自然語言處理等領域研發工作。主要研究方向：推薦系統中用戶行為序列表示學習、隱私保護；文本表示學習、文本摘要等。在ACL、SIGIR、WWW、TOIS等頂級會議期刊發表論文40余篇，獲RecSys 2019 會議Best Long Paper Runner-up獎，Google學術引用1000余次。擔任ACL、SIGIR、WWW、IJCAI等國際會議PC member及senior PC。
摘要：Despite significant progress has been achieved in text summarization, factual inconsistency in generated summaries still severely limits its practical applications. Among the key factors to ensure factual consistency, a reliable automatic evaluation metric is the first and the most crucial one. However, existing metrics either neglect the intrinsic cause of the factual inconsistency or rely on auxiliary tasks, leading to an unsatisfied correlation with human judgments or increasing the inconvenience of usage in practice. In light of these challenges, we propose a novel metric to evaluate the factual consistency in text summarization via counterfactual estimation, which formulates the causal relationship among the source document, the generated summary, and the language prior. We remove the effect of language prior, which can cause factual inconsistency, from the total causal effect on the generated summary, and provides a simple yet effective way to evaluate consistency without relying on other auxiliary tasks. We conduct a series of experiments on three public abstractive text summarization datasets, and demonstrate the advantages of the proposed metric in both improving the correlation with human judgments and the convenience of usage.