Evaluating the Use of Large Language Models to Create High Quality Questions for Programming Assignments
Loading...
URL
Journal Title
Journal ISSN
Volume Title
Perustieteiden korkeakoulu |
Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Authors
Date
2024-07-31
Department
Major/Subject
Security and Cloud Computing
Mcode
SCI3113
Degree programme
Master’s Programme in Security and Cloud Computing (SECCLO)
Language
en
Pages
53+7
Series
Abstract
Large Language Models (LLMs) are becoming increasingly important in almost every domain, including education technology, due to their adaptive and scalable abilities. Despite this, there exists a gap in the use and evaluation of LLMs in the generation of high-quality questions in terms of matching the quality of human-generated questions. Specifically, this thesis aims to address three primary gaps: the effectiveness of the LLMs in generating diverse questions, the human perception of LLM-generated high-quality questions, and verifying whether people can distinguish between the sources of the questions. To address these gaps, this work proposes an approach to generate high-quality questions with prompt engineering. This approach consists of generating information or topic knowledge, using that as the basis for the generation of the questions, and subsequently generating the test cases and their respective solutions. This study employs a mixed methods approach, incorporating both quantitative and qualitative analysis. The quantitative analysis makes use of metrics BLEU, ROUGE, and cosine similarity scores, while the qualitative analysis explores the human perception of the LLM-generated questions, analyzing various aspects of question quality such as clarity, depth of understanding, uniqueness, complexity, and relevance. The study examines the distinction between human-generated and machine-generated questions along these aspects by using surveys.Description
Supervisor
Hellas, ArtoThesis advisor
Leinonen, JuhoKeywords
automated question generation, large language models, educational technology, high quality questions, machine learning in education, natural language processing