Cargando…
Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization
Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks, including automatic summarization. However, most work has focused on (relatively) data-rich single-document summarization settings. In this paper, we explore highly-abstractiv...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7720861/ https://www.ncbi.nlm.nih.gov/pubmed/33293900 |
_version_ | 1783619926065414144 |
---|---|
author | Goodwin, Travis R. Savery, Max E. Demner-Fushman, Dina |
author_facet | Goodwin, Travis R. Savery, Max E. Demner-Fushman, Dina |
author_sort | Goodwin, Travis R. |
collection | PubMed |
description | Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks, including automatic summarization. However, most work has focused on (relatively) data-rich single-document summarization settings. In this paper, we explore highly-abstractive multi-document summarization, where the summary is explicitly conditioned on a user-given topic statement or question. We compare the summarization quality produced by three state-of-the-art transformer-based models: BART, T5, and PEGASUS. We report the performance on four challenging summarization datasets: three from the general domain and one from consumer health in both zero-shot and few-shot learning settings. While prior work has shown significant differences in performance for these models on standard summarization tasks, our results indicate that with as few as 10 labeled examples, there is no statistically significant difference in summary quality, suggesting the need for more abstractive benchmark collections when determining state-of-the-art. |
format | Online Article Text |
id | pubmed-7720861 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
record_format | MEDLINE/PubMed |
spelling | pubmed-77208612020-12-07 Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization Goodwin, Travis R. Savery, Max E. Demner-Fushman, Dina Proc Int Conf Comput Ling Article Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks, including automatic summarization. However, most work has focused on (relatively) data-rich single-document summarization settings. In this paper, we explore highly-abstractive multi-document summarization, where the summary is explicitly conditioned on a user-given topic statement or question. We compare the summarization quality produced by three state-of-the-art transformer-based models: BART, T5, and PEGASUS. We report the performance on four challenging summarization datasets: three from the general domain and one from consumer health in both zero-shot and few-shot learning settings. While prior work has shown significant differences in performance for these models on standard summarization tasks, our results indicate that with as few as 10 labeled examples, there is no statistically significant difference in summary quality, suggesting the need for more abstractive benchmark collections when determining state-of-the-art. 2020-12 /pmc/articles/PMC7720861/ /pubmed/33293900 Text en https://creativecommons.org/licenses/by/4.0/This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Article Goodwin, Travis R. Savery, Max E. Demner-Fushman, Dina Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title | Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title_full | Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title_fullStr | Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title_full_unstemmed | Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title_short | Flight of the PEGASUS? Comparing Transformers on Few-Shot and Zero-Shot Multi-document Abstractive Summarization |
title_sort | flight of the pegasus? comparing transformers on few-shot and zero-shot multi-document abstractive summarization |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7720861/ https://www.ncbi.nlm.nih.gov/pubmed/33293900 |
work_keys_str_mv | AT goodwintravisr flightofthepegasuscomparingtransformersonfewshotandzeroshotmultidocumentabstractivesummarization AT saverymaxe flightofthepegasuscomparingtransformersonfewshotandzeroshotmultidocumentabstractivesummarization AT demnerfushmandina flightofthepegasuscomparingtransformersonfewshotandzeroshotmultidocumentabstractivesummarization |