aalto1 untyped-item.component.html
CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI
Loading...
Access rights
openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Date
Major/Subject
Mcode
Degree programme
Language
en
Pages
2
Series
Interspeech 2024, pp. 5212-5213, Interspeech
Abstract
We introduce the CaptainA mobile app, designed to meet the needs of second language (L2) learners engaged in self- study of Finnish, with potential applicability to other languages. Our app can provide automatic speaking assessment (ASA) of task completion in picture-based tasks, along with grading ex- planations and corrective feedback. It can also automatically generate pictures for visual tasks, providing users with unlim- ited practice opportunities. The mobile app is based on our framework that combines visual natural language generation (NLG), automatic speech recognition (ASR), and prompting large language model (LLM) for low-resource language. Our goal is to promote the development of next-generation speech- based computer-assisted language learning (CALL) systems ca- pable of providing automatic scoring with feedback for learn- ers, even when minimal speech data of L2 learners is available. While the mobile app demonstration is designed for Finnish, the app can also be tested in English.
Description
Other note
Citation
Chi, N P, von Zansen, A, Kautonen, M, Grósz, T & Kurimo, M 2024, CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI. in Interspeech 2024. Interspeech, International Society for Computers and Their Applications (ISCA), pp. 5212-5213, Interspeech, Kos Island, Greece, 01/09/2024. < https://www.isca-archive.org/interspeech_2024/phan24b_interspeech.pdf >