Improving BERT Pretraining with Syntactic Supervision

Bidirectional masked Transformers have become the core theme in the current NLP landscape. Despite their impressive benchmarks, a recurring theme in recent research has been to question such models’ capacity for syntactic generalization. In this work, we seek to address this question by adding a supervised, token-level supertagging objective to standard unsupervised pretraining, enabling the explicit incorporation of syntactic biases into the network’s training dynamics. Our approach is straightforward to implement, induces a marginal computational overhead and is general enough to adapt to a variety of settings. We apply our methodology on Lassy Large, an automatically annotated corpus of written Dutch. Our experiments suggest that our syntax-aware model performs on par with established baselines, despite Lassy Large being one order of magnitude smaller than commonly used corpora.

Citation

Tziafas, G, Kogkalidis, K, Wijnholds, G & Moortgat, M 2023, Improving BERT Pretraining with Syntactic Supervision . in Proceedings of the 2023 CLASP Conference on Learning with Small Data . CLASP Papers in Computational Linguistics, vol. 5, Association for Computational Linguistics, pp. 176-184, Learning with Small Data, Gothenburg, Sweden, 11/09/2023 . < https://arxiv.org/abs/2104.10516 >

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202501171256

Collections

[article-cris] Perustieteiden korkeakoulu / SCI

Show all metadata

Improving BERT Pretraining with Syntactic Supervision

Access rights

URL

Journal Title

Journal ISSN

Volume Title

Authors

Date

Department

Major/Subject

Mcode

Degree programme

Language

Pages

Series

Abstract

Description

Keywords

Other note

Citation

Permanent link to this item

Collections