Pre-trained language models with domain knowledge for biomedical extractive summarization

Loading...
Thumbnail Image
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
Date
2022-09-27
Major/Subject
Mcode
Degree programme
Language
en
Pages
12
1-12
Series
KNOWLEDGE-BASED SYSTEMS, Volume 252
Abstract
Biomedical text summarization is a critical task for comprehension of an ever-growing amount of biomedical literature. Pre-trained language models (PLMs) with transformer-based architectures have been shown to greatly improve performance in biomedical text mining tasks. However, existing methods for text summarization generally fine-tune PLMs on the target corpora directly and do not consider how fine-grained domain knowledge, such as PICO elements used in evidence-based medicine, can help to identify the context needed for generating coherent summaries. To fill the gap, we propose KeBioSum, a novel knowledge infusion training framework, and experiment using a number of PLMs as bases, for the task of extractive summarization on biomedical literature. We investigate generative and discriminative training techniques to fuse domain knowledge (i.e., PICO elements) into knowledge adapters and apply adapter fusion to efficiently inject the knowledge adapters into the basic PLMs for fine-tuning the extractive summarization task. Experimental results from the extractive summarization task on three biomedical literature datasets show that existing PLMs (BERT, RoBERTa, BioBERT, and PubMedBERT) are improved by incorporating the KeBioSum knowledge adapters, and our model outperforms the strong baselines.
Description
Funding Information: This research is supported by the Alan Turing Institute, United Kingdom and the Biotechnology and Biological Sciences Research Council (BBSRC), United Kingdom , BB/P025684/1 . Publisher Copyright: © 2022 The Author(s)
Keywords
Domain knowledge, Extractive summarization, PICO elements, Pre-trained language models
Other note
Citation
Xie , Q , Bishop , J A , Tiwari , P & Ananiadou , S 2022 , ' Pre-trained language models with domain knowledge for biomedical extractive summarization ' , KNOWLEDGE-BASED SYSTEMS , vol. 252 , 109460 , pp. 1-12 . https://doi.org/10.1016/j.knosys.2022.109460