Comparing Code Explanations Created by Students and Large Language Models
Loading...
Access rights
openAccess
URL
Journal Title
Journal ISSN
Volume Title
A4 Artikkeli konferenssijulkaisussa
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Other link related to publication (opens in new window)
Date
2023-06-29
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
7
124-130
124-130
Series
ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education
Abstract
Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.Description
Funding Information: We are grateful for the grant from the Ulla Tuominen Foundation to the first author. Publisher Copyright: © 2023 Owner/Author.
Keywords
ChatGPT, code comprehension, code explanations, CS1, foundation models, GPT-3, GPT-4, large language models, natural language generation, resource generation
Other note
Citation
Leinonen, J, Denny, P, Macneil, S, Sarsa, S, Bernstein, S, Kim, J, Tran, A & Hellas, A 2023, Comparing Code Explanations Created by Students and Large Language Models . in ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education . ACM, pp. 124-130, Annual Conference on Innovation and Technology in Computer Science Education, Turku, Finland, 08/07/2023 . https://doi.org/10.1145/3587102.3588785