On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation
dc.contributor | Aalto-yliopisto | fi |
dc.contributor | Aalto University | en |
dc.contributor.author | Moisio, Anssi | en_US |
dc.contributor.author | Creutz, Mathias | en_US |
dc.contributor.author | Kurimo, Mikko | en_US |
dc.contributor.department | Department of Information and Communications Engineering | en |
dc.contributor.groupauthor | Speech Recognition | en |
dc.date.accessioned | 2024-01-04T09:05:41Z | |
dc.date.available | 2024-01-04T09:05:41Z | |
dc.date.issued | 2023-12-06 | en_US |
dc.description.abstract | Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems’ capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca. | en |
dc.description.version | Peer reviewed | en |
dc.format.extent | 10 | |
dc.format.mimetype | application/pdf | en_US |
dc.identifier.citation | Moisio, A, Creutz, M & Kurimo, M 2023, On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation . in Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP . Association for Computational Linguistics, GenBench: Workshop on generalisation (benchmarking) in NLP, Singapore, Singapore, 06/12/2023 . < https://aclanthology.org/2023.genbench-1.17 > | en |
dc.identifier.isbn | 979-8-89176-042-4 | |
dc.identifier.other | PURE UUID: a61184fe-ef63-4972-8d34-6e989438b4c9 | en_US |
dc.identifier.other | PURE ITEMURL: https://research.aalto.fi/en/publications/a61184fe-ef63-4972-8d34-6e989438b4c9 | en_US |
dc.identifier.other | PURE LINK: https://aclanthology.org/2023.genbench-1.pdf | en_US |
dc.identifier.other | PURE LINK: https://aclanthology.org/2023.genbench-1.17 | en_US |
dc.identifier.other | PURE FILEURL: https://research.aalto.fi/files/131423680/2023.genbench-1.17.pdf | en_US |
dc.identifier.uri | https://aaltodoc.aalto.fi/handle/123456789/125491 | |
dc.identifier.urn | URN:NBN:fi:aalto-202401041180 | |
dc.language.iso | en | en |
dc.relation.ispartof | GenBench: Workshop on generalisation (benchmarking) in NLP | en |
dc.relation.ispartofseries | Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP | en |
dc.rights | openAccess | en |
dc.title | On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation | en |
dc.type | A4 Artikkeli konferenssijulkaisussa | fi |
dc.type.version | publishedVersion |