aalto1 untyped-item.component.html
Solving Proof Block Problems Using Large Language Models
Loading...
Access rights
openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Date
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
7
Series
SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education, pp. 1063-1069
Abstract
Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs. In this paper, we investigate the ability of LLMs to solve mathematical proofs by using Proof Blocks, a tool previously shown to efficaciously teach proofs to students. Our results show that GPT-3.5 is almost completely unable to provide correct solutions (11.4%), while GPT-4 shows a significant increase in correctness (64.8%). However, even given this improvement, current models still struggle to correctly order lines in a proof. It remains an open question whether this is a temporary situation or if LLMs will continue to struggle to solve these types of exercises in the future.
Description
Publisher Copyright: © 2024 Owner/Author.
Other note
Citation
Poulsen, S, Sarsa, S, Prather, J, Leinonen, J, Becker, B A, Hellas, A, Denny, P & Reeves, B N 2024, Solving Proof Block Problems Using Large Language Models. in SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education. ACM, pp. 1063-1069, ACM Technical Symposium on Computer Science Education, Portland, Oregon, United States, 20/03/2024. https://doi.org/10.1145/3626252.3630928