Solving Proof Block Problems Using Large Language Models

Loading...
Thumbnail Image

Access rights

openAccess

URL

Journal Title

Journal ISSN

Volume Title

A4 Artikkeli konferenssijulkaisussa

Date

2024-03-07

Major/Subject

Mcode

Degree programme

Language

en

Pages

7

Series

Abstract

Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs. In this paper, we investigate the ability of LLMs to solve mathematical proofs by using Proof Blocks, a tool previously shown to efficaciously teach proofs to students. Our results show that GPT-3.5 is almost completely unable to provide correct solutions (11.4%), while GPT-4 shows a significant increase in correctness (64.8%). However, even given this improvement, current models still struggle to correctly order lines in a proof. It remains an open question whether this is a temporary situation or if LLMs will continue to struggle to solve these types of exercises in the future.

Description

Publisher Copyright: © 2024 Owner/Author.

Keywords

ai, algorithms, artificial intelligence, chatgpt, code generation, generative ai, gpt-3, gpt-4, large language models, openai, proof blocks, proofs

Other note

Citation

Poulsen, S, Sarsa, S, Prather, J, Leinonen, J, Becker, B A, Hellas, A, Denny, P & Reeves, B N 2024, Solving Proof Block Problems Using Large Language Models . in SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education . ACM, pp. 1063-1069, ACM Technical Symposium on Computer Science Education, Portland, United States, 20/03/2024 . https://doi.org/10.1145/3626252.3630928