Cargando…
BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection
In recent years, text sentiment analysis has attracted wide attention, and promoted the rise and development of stance detection research. The purpose of stance detection is to determine the author’s stance (favor or against) towards a specific target or proposition in the text. Pre-trained language...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8432858/ https://www.ncbi.nlm.nih.gov/pubmed/34506549 http://dx.doi.org/10.1371/journal.pone.0257130 |
_version_ | 1783751254431760384 |
---|---|
author | Li, Yang Sun, Yuqing Zhu, Nana |
author_facet | Li, Yang Sun, Yuqing Zhu, Nana |
author_sort | Li, Yang |
collection | PubMed |
description | In recent years, text sentiment analysis has attracted wide attention, and promoted the rise and development of stance detection research. The purpose of stance detection is to determine the author’s stance (favor or against) towards a specific target or proposition in the text. Pre-trained language models like BERT have been proven to perform well in this task. However, in many reality scenes, they are usually very expensive in computation, because such heavy models are difficult to implement with limited resources. To improve the efficiency while ensuring the performance, we propose a knowledge distillation model BERTtoCNN, which combines the classic distillation loss and similarity-preserving loss in a joint knowledge distillation framework. On the one hand, BERTtoCNN provides an efficient distillation process to train a novel ‘student’ CNN structure from a much larger ‘teacher’ language model BERT. On the other hand, based on the similarity-preserving loss function, BERTtoCNN guides the training of a student network, so that input pairs with similar (dissimilar) activation in the teacher network have similar (dissimilar) activation in the student network. We conduct experiments and test the proposed model on the open Chinese and English stance detection datasets. The experimental results show that our model outperforms the competitive baseline methods obviously. |
format | Online Article Text |
id | pubmed-8432858 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-84328582021-09-11 BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection Li, Yang Sun, Yuqing Zhu, Nana PLoS One Research Article In recent years, text sentiment analysis has attracted wide attention, and promoted the rise and development of stance detection research. The purpose of stance detection is to determine the author’s stance (favor or against) towards a specific target or proposition in the text. Pre-trained language models like BERT have been proven to perform well in this task. However, in many reality scenes, they are usually very expensive in computation, because such heavy models are difficult to implement with limited resources. To improve the efficiency while ensuring the performance, we propose a knowledge distillation model BERTtoCNN, which combines the classic distillation loss and similarity-preserving loss in a joint knowledge distillation framework. On the one hand, BERTtoCNN provides an efficient distillation process to train a novel ‘student’ CNN structure from a much larger ‘teacher’ language model BERT. On the other hand, based on the similarity-preserving loss function, BERTtoCNN guides the training of a student network, so that input pairs with similar (dissimilar) activation in the teacher network have similar (dissimilar) activation in the student network. We conduct experiments and test the proposed model on the open Chinese and English stance detection datasets. The experimental results show that our model outperforms the competitive baseline methods obviously. Public Library of Science 2021-09-10 /pmc/articles/PMC8432858/ /pubmed/34506549 http://dx.doi.org/10.1371/journal.pone.0257130 Text en © 2021 Li et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Li, Yang Sun, Yuqing Zhu, Nana BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title | BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title_full | BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title_fullStr | BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title_full_unstemmed | BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title_short | BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection |
title_sort | berttocnn: similarity-preserving enhanced knowledge distillation for stance detection |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8432858/ https://www.ncbi.nlm.nih.gov/pubmed/34506549 http://dx.doi.org/10.1371/journal.pone.0257130 |
work_keys_str_mv | AT liyang berttocnnsimilaritypreservingenhancedknowledgedistillationforstancedetection AT sunyuqing berttocnnsimilaritypreservingenhancedknowledgedistillationforstancedetection AT zhunana berttocnnsimilaritypreservingenhancedknowledgedistillationforstancedetection |