Investigating wav2vec2 context representations and the effects of fine-tuning, a case-study of a Finnish model

Self-supervised speech models, such as the wav2vec2, have become extremely popular in the past few years. Their main appeal is that after their pre-training on a large amount of audio, they require only a small amount of supervised, finetuning data to achieve outstanding results. Despite their immense success, very little is understood about the pre-trained models and how finetuning changes them. In this work, we take the first steps towards a better understanding of wav2vec2 systems using model interpretation tools such as visualization and latent embedding clustering. Through our analysis, we gain new insights into the abilities of the pre-trained networks and the effect that finetuning has on them. We demonstrate that the clusters learned by the pre-trained model are just as important a factor as the supervised training data distribution in determining the accuracy of the finetuned system, which could aid us in selecting the most suitable pre-trained model for the supervised data.

DOI

10.21437/interspeech.2023-837

Citation

Grosz, T, Getman, Y, Al-Ghezi, R, Rouhe, A & Kurimo, M 2023, Investigating wav2vec2 context representations and the effects of fine-tuning, a case-study of a Finnish model . in Proceedings of Interspeech 2023 . Interspeech, International Speech Communication Association (ISCA), pp. 196-200, Interspeech, Dublin, Ireland, 20/08/2023 . https://doi.org/10.21437/interspeech.2023-837

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202311296987

Collections

[article-cris] Palvelut / Services

Show all metadata

Investigating wav2vec2 context representations and the effects of fine-tuning, a case-study of a Finnish model

Access rights

URL

Journal Title

Journal ISSN

Volume Title

Authors

Date

Department

Major/Subject

Mcode

Degree programme

Language

Pages

Series

Abstract

Description

Keywords

Other note

DOI

Citation

Permanent link to this item

Collections