Automated Assessment of Task Completion in Spontaneous Speech for Finnish and Finland Swedish Language Learners

Loading...
Thumbnail Image
Journal Title
Journal ISSN
Volume Title
Conference article in proceedings
This publication is imported from Aalto University research portal.
View publication in the Research portal
View/Open full text file from the Research portal
Date
2023-05-16
Major/Subject
Mcode
Degree programme
Language
en
Pages
9
102-110
Series
Proceedings of the 12th Workshop on Natural Languages Processing for Computer Assisted Language Learning, Linköping Electronic Conference Proceedings, issue 197, NEALT Proceedings Series, issue 53
Abstract
This study investigates the feasibility of automated content scoring for spontaneous spoken responses from Finnish and Finland Swedish learners. Our experiments reveal that pretrained Transformer-based models outperform the tf-idf baseline in automatic task completion grading. Furthermore, we demonstrate that pre-fine-tuning these models to differentiate between responses to distinct prompts enhances subsequent task completion finetuning. We observe that task completion classifiers exhibit accelerated learning and produce predictions with stronger correlations to human grading when accounting for task differences. Additionally, we find that employing similarity learning, as opposed to conventional classification fine-tuning, further improves the results. It is especially helpful to learn not just the similarities between the responses in one score bin, but the exact differences between the average human scores responses received. Lastly, we demonstrate that models applied to both manual and ASR transcripts yield comparable correlations to human grading.
Description
Keywords
Other note
Citation
Voskoboinik , E , Getman , Y , Al-Ghezi , R , Kurimo , M & Grosz , T 2023 , Automated Assessment of Task Completion in Spontaneous Speech for Finnish and Finland Swedish Language Learners . in Proceedings of the 12th Workshop on Natural Languages Processing for Computer Assisted Language Learning . Linköping Electronic Conference Proceedings , no. 197 , NEALT Proceedings Series , no. 53 , Linköpings universitet , pp. 102-110 , Workshop on Natural Language Processing for Computer Assisted Language Learning , Tórshavn , Faroe Islands , 22/05/2023 . https://doi.org/10.3384/ecp197012