Solving Proof Block Problems Using Large Language Models

Thumbnail Image
Journal Title
Journal ISSN
Volume Title
Conference article in proceedings
Degree programme
Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs. In this paper, we investigate the ability of LLMs to solve mathematical proofs by using Proof Blocks, a tool previously shown to efficaciously teach proofs to students. Our results show that GPT-3.5 is almost completely unable to provide correct solutions (11.4%), while GPT-4 shows a significant increase in correctness (64.8%). However, even given this improvement, current models still struggle to correctly order lines in a proof. It remains an open question whether this is a temporary situation or if LLMs will continue to struggle to solve these types of exercises in the future.
Publisher Copyright: © 2024 Owner/Author.
ai, algorithms, artificial intelligence, chatgpt, code generation, generative ai, gpt-3, gpt-4, large language models, openai, proof blocks, proofs
Other note
Poulsen, S, Sarsa, S, Prather, J, Leinonen, J, Becker, B A, Hellas, A, Denny, P & Reeves, B N 2024, Solving Proof Block Problems Using Large Language Models . in SIGCSE 2024 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education . ACM, pp. 1063-1069, ACM Technical Symposium on Computer Science Education, Portland, United States, 20/03/2024 .