Browsing by Author "Lehtinen, Teemu"

Now showing 1 - 20 of 21
  • No Thumbnail Available
    Item
    Advantages and disadvantages of vertical integration in the implementation of systemic process innovations: Case studies on implementing building information modeling (BIM) in the Finnish construction industry
    (Aalto-yliopisto, 2010) Lehtinen, Teemu; Hirvensalo, Antero; Informaatio- ja luonnontieteiden tiedekunta; Smeds, Riitta
    Vertikaalinen integraatio tarkoittaa usean tai kaikkien arvoketjun toimintojen yhdistymistä samaan yritykseen. Monet tutkijat ovat olleet sitä mieltä, että vertikaalinen integraatio edistää systeemisten innovaatioiden synnyttämistä ja käyttöönottoa. Kun tavalliset innovaatiot voidaan ottaa käyttöön sellaisenaan ilman sen suurempia muutoksia laajempaan toimintaympäristöön, systeemisten innovaatioiden menestyksekäs käyttöönotto vaatii merkittäviä muutoksia koko toimintaympäristössä. Usein systeemiset mnnovaatiot ovat kuitenkin liian monimutkaisia ja laajoja yhden yrityksen hallittavaksi. Rakennusten tietomallinnus (BIM) on esimerkki systeemisestä prosessi-innovaatiosta rakennusalalla. Tietomallinnus tarkoittaa laajaa kokonaisuutta, joka koostuu yhteisistä käytännöistä, prosesseista ja teknologioista. Tietomallinnuksen tarkoituksena on hallita rakennukseen liittyvää tietoa digitaalisessa muodossa rakennuksen koko elinkaaren ajan aina suunnittelusta toteutukseen ja käyttöön. Tietomallinnuksen on odotettu tuovan merkittäviä parannuksia rakennusalan tuottavuuteen jo 80-luvulta lähtien, mutta sen käyttöönotto ja leviäminen on osoittautunut ennakoitua hitaammaksi ja vaikeammaksi. Keskeisenä syynä tähän on ollut tietomallinnuksen systeeminen ja organisaatioiden rajat ylittävä luonne. Samalla kuitenkin, myös kiinnostus vertikaaliseen integraation on lisääntynyt rakennusalalla. Tämä tutkimus pyrkiikin valottamaan tietomallinnuksen käyttöönoton ja organisaatiorakenteiden välistä yhteyttä tutkimalla vertikaalisen integraation hyötyjä ja haittoja tietomallinnuksen käyttöönotossa. Tutkimuksen empiirinen osa perustuu kahteen vastakkaiseen tapaustutkimukseen Suomen rakennusteollisuudessa. Toisessa tapauksessa projektiverkosto koostui erillisistä yrityksistä, kun taas toisessa tapauksessa projektiverkosto oli vertikaalisti integroitunut. Molemmissa tapauksissa tarkasteltiin yksittäistä rakennushanketta, jossa testattiin tietomallinnusta. Laadullinen tutkimusaineisto koostuu projektidokumentaatiosta, haastatteluista sekä osapuolien havainnoinnista. Tutkimuksessa luodaan ensin teoreettinen malli vertikaalisen integraation hyödyistä ja haitoista perustuen kirjallisuuskatsaukseen. Tämän jälkeen mallia testataan tutkimusaineistolla ja parannellaan analyysin perusteella. Tutkimuksen tulosten perusteella tietomallinnuksen käyttöönottoon liittyy seitsemän organisaatiorakenteen kannalta merkittävää tekijää. Nämä tekijät ovat (1) johdon tuki, (2) koordinointi ja hallinta, (3) oppiminen ja kokemus, (4) teknologian hallinta, (5) viestintä, (6) motivaatio sekä (7) roolien määrittely. Kaikkiin näihin tekijöihin liittyy vertikaalisen integraation tuomia hyötyjä ja haittoja. Tämän vuoksi yritysjohdon pitäisikin ymmärtää organisaatiorakenteen mahdolliset vaikutukset tietomallinnuksen käyttöönotossa, jotta käyttöönottoprojektit osattaisiin suunnitella ja toteuttaa mahdollisimman sujuvasti.
  • Thumbnail Image
    Item
    Automated Questionnaires About Students’ JavaScript Programs: Towards Gauging Novice Programming Processes
    (2023-01-30) Lehtinen, Teemu; Haaranen, Lassi; Leinonen, Juho; Department of Computer Science; Denny, Paul; Herbert, Nicole; Professorship Malmi L.; Computer Science Lecturers; Lecturer Haaranen Lassi group; Computer Science - Computing education research and educational technology (CER)
    Students sometimes manage to produce functionally correct program code while having a fragile understanding of the related learning goals. Such unproductive success could be intercepted by an educator who asks questions that target the structure and evaluation of the student’s program using the constructs and identifiers in the code. We provide a tool that automatically generates multiple-choice questions of seven different types for this purpose. We integrated these questions into a web-based program writing exercises, which we also publish as a part of this work, and successfully used them on an introductory programming course. In our pilot evaluation of the tool, we found that the students who answer these questions repeatedly incorrectly are likely to drop out, have more challenges while writing a program, and resort to tinkering behavior.
  • Thumbnail Image
    Item
    Automated Questions about Learners' Own Code Help to Detect Fragile Prerequisite Knowledge
    (2023-06-29) Lehtinen, Teemu; Seppälä, Otto; Korhonen, Ari; Department of Computer Science; Professorship Malmi L.; Computer Science Lecturers; Computer Science - Computing Systems (ComputingSystems); Computer Science - Computing education research and educational technology (CER); Lecturer Seppälä Otto group; Lecturer Korhonen Ari group
    Students are able to produce correctly functioning program code even though they have a fragile understanding of how it actually works. Questions derived automatically from individual exercise submissions (QLC) can probe if and how well the students understand the structure and logic of the code they just created. Prior research studied this approach in the context of the first programming course. We replicate the study on a follow-up programming course for engineering students which contains a recap of general concepts in CS1. The task was the classic rainfall problem which was solved by 90% of the students. The QLCs generated from each passing submission were kept intentionally simple, yet 27% of the students failed in at least one of them. Students who struggled with questions about their own program logic had a lower median for overall course points than students who answered correctly.
  • Thumbnail Image
    Item
    Automatic Assessment of Programming Exercises
    (2021-06-14) Rutanen, Timo; Lehtinen, Teemu; Perustieteiden korkeakoulu; Malmi, Lauri
    Programming assignments are a key part in teaching programming skills. The students' solutions to the assignments have to be collected and graded and as course sizes grow, so does the amount of work required for the teaching staff to organize, analyze and grade them as well as provide guidance or feedback to the students. It is therefore reasonable to seek ways to automate at least parts of the process of grading. An automatic grading system carries some of the burden of repetitive tasks for the one-time investment of developing such a system. The students also benefit by gaining access to immediate feedback on their submissions, aiding them in the development of the solutions depending on the system's functionality. Of course, the system may need upgrades later, like many systems do, and the requirements may also change as courses and programming languages evolve over time. On the other hand, a next generation grading system can prove to be an even more significant aid to teaching. This thesis aims to showcase various different methods an automatic grading system could analyze programming assignments as well as implement one analyzer of its own in Python for Python assignments for teaching purposes. Though there are dynamic methods as well, this thesis concentrates on static methods of analyzing software complexity. The complexity metrics used in the analyzer are: Lines of Code, Cyclomatic complexity, the ABC-metric and five Halstead's metrics: Volume, Bugs, Difficulty, Effort and Programming Time. The anonymized test data consists of one 300-student course's worth of python programming assignments and their unit test results, a total amount of over 5000 submission results gathered by another assessment system outside this project. Single submissions are not the focus of this thesis though the data does exist for future development purposes. It is discovered that Halstead's software metrics produce very similar results to each other in terms of the analysis, therefore for their usage here they are interchangeable. No radically different major solutions were found in the data, though it does point toward alternative solution groups in some exercises. A clear evolution of metrics from faulty submissions to perfect submissions is not visible in the analysis. The reasoning behind a student submitting multiple perfect solutions to an exercise is noted as a curious question perhaps worth further work.
  • Thumbnail Image
    Item
    Bootstrapping Learning Analytics Case: Aalto Online Learning
    (2017-12-11) Lehtinen, Teemu; Korhonen, Ari; Perustieteiden korkeakoulu; Malmi, Lauri
    The digital transformation of learning brings forth data having unprecedented granularity and coverage of learning activity. The research area of Learning Analytics (LA) uses this data to understand and improve learning. The practice of LA is a cyclic process where learning data is collected from different sources and analytics is developed according to stakeholder objectives. Finally, current results are delivered that lead into action which improves learning and produces new data. The goal of this thesis is to bootstrap LA in multiple courses that implement different weekly online learning activities. The term bootstrap underlines the aim to support continuity, further development, and expansion of LA. The research questions were: what learning data the courses currently instrument, and what LA objectives the course staff find most important. This thesis conducts software engineering to construct an LA solution for the research case. Requirements are defined via examination of the case and interviews of the course staff. The developed solution enables real time access to learning data and possibility to integrate data from both Moodle and A-plus learning environments for joined analysis. Novel interactive visualizations are developed according to the user requirements. The work in bootstrapping LA at course level lead to two general findings. First, the integration of learning data from multitude of sources is a common challenge that requires design. Second, teachers' initial LA objectives include aims to monitor expected progress, improve allocation of learning material, identify problematic areas in learning material, and improve interaction with learners.
  • Thumbnail Image
    Item
    Challenges in building information modeling: Insights from a pioneering process development workshop in Finland
    (2015-03-23) Löfgren, Jenny; Lehtinen, Teemu; Smeds, Matilda; Insinööritieteiden korkeakoulu; Smeds, Riitta
    The purpose of this thesis was firstly, to find the challenges related to BIM process, and secondly to find the solutions to the challenges. The solutions were found from multi-party agreements (MPAs), where building construction and the contract is designed in collaboration between the architect, the other designers, main contractor, and the owner. The contract model is based on the fact that the profits and risks during the process are allocated between the participants, which will motivate the team to pursue as effective cooperation as possible, to share information also about incomplete work, and thus to enhance understanding about different discipline’s work, challenges and the whole design and construction process. The subject is relevant, as the industry evolves so slowly even though there are solutions for the challenges. Building information modeling (BIM) refers to the process in which different disciplines are designing 3-4D designs within a construction project. It provides development opportunities when there is competence to utilize them. The model can be used for visualization for the owner and the authorities, and as a tool for the site workers in the constructing phase. The as-built model can also be used in the maintenance phase, as all the materials and particles used in the building are in the final model. The challenges found during this study were especially related to collaboration, coordination, contractual interests, and the lack of competence in both using the models, as well as modeling it-self. The results show that these challenges are not only related to BIM, but construction processes in general. Thus, it is important to emphasize all of the most obvious solutions available. This thesis provides for a basis for future research concentrating on operational challenges related to construction process. It also serves the field work, especially in the planning phase, when struggling with the challenges presented in this thesis. The empirical part of this thesis was based on a focus-group workshop arranged for the Finnish pioneers in BIM, from which the qualitative material was collected by observations and recordings. In the research, an initial theoretical framework is constructed from the BIM related challenges and MPA related solutions found in the literature, which are tested with the collected empirical data. The result is an enhanced constructed framework, which shows that the BIM related challenges can in fact be solved with implementing MPAs as well as the so called Last Planner® System.
  • Thumbnail Image
    Item
    Chatbots: Characteristics increasing customer satisfaction
    (2023) Lehtinen, Teemu; Hekkala, Riitta; Tieto- ja palvelujohtamisen laitos; Kauppakorkeakoulu; School of Business
    The number of chatbots used as customer service agents has increased significantly over the last decade. Even though chatbots lack the ability to simulate human speech and do not perform as well as human agents do, their implementation as customer service agents has several benefits. Therefore, it is important to understand which characteristics customers find appealing and engage them into satisfying interaction. This study examines the characteristics and features which are found to positively affect the customer satisfaction. Moreover, this research investigates if there is a conflict between these characteristics and if it is possible to incorporate them all into a single chatbot to construct the perfect chatbot. The research method used in this study was a literature review, which mainly focused on publications on the subject published in the last few years. The research questions were twofold: which characteristics of chatbots have a positive impact on customer satisfaction and how they conflict with each other. The research questions were addressed as broadly as possible, taking into account the significant and recent literature. The study found multiple characteristics positively affecting customer satisfaction. From the results a table was created where all the characteristics not in conflict can be found. The study also discovered conflicting characteristics that should be considered when a company is designing a chatbot for their target audience. The most significant result was that characteristics found to be appealing to customers were anthropomorphic which indicates that customers want the chatbot to be as human as possible and communicative cues should be carefully considered.
  • No Thumbnail Available
    Item
    Co-creating Digital Services with and for Facilities Management
    (Emerald Group Publishing Ltd., 2017-08-09) Lavikka, Rita; Lehtinen, Teemu; Hall, Daniel; Department of Industrial Engineering and Management; Stanford University
    This study aims to increase understanding about the co-creation of digital facilities management (FM) services with and for FM during a construction project. The paper reports a case study on the co-creation of a digital facilities management service during the Mission Bay medical center construction project of the University of California, San Francisco. The sub-contractors and the FM team co-created a quick-response (QR) code system for valves on the project. This digital service is now used by the FM team for training purposes and in emergency situations when information on valves and their service areas is quickly needed. Researchers made on-site observations, conducted 84 interviews, and reviewed archival data in 2012, 2014, and 2016. The findings show that a successful co-creation process consists of three elements: 1) A dialogue between the project parties, 2) The creation of shared context between the project parties, and 3) The creation of shared understanding about the FM’s needs and the service providers’ ways of satisfying those needs. The study describes ways to promote these elements. Previous studies emphasize the need for digital FM but few explain how FM teams can be involved in creating digital services for them. This paper describes how to co-create digital FM services with and for FM in the context of a construction project.
  • No Thumbnail Available
    Item
    Co-creation of digital services with and for facilities management in construction projects
    (2016-06-28) Lavikka, Rita; Lehtinen, Teemu; Hall, Daniel; Department of Industrial Engineering and Management; Stanford University; Kaminsky, Jessica; Zerjav, Vedran
    The building construction sector is slowly recognizing the financial opportunities of the growing digital service business for facilities management (FM). Digital FM provides service business opportunities for construction project parties, reduces building life-cycle costs, and increases the performance of the building. However, research is needed to understand how the construction sector parties can create digital services with and for FM. This paper presents a case study on the co-creation of a digital FM service during the medical center construction project of the University of California, San Francisco (UCSF) at Mission Bay in California. The sub-contractors and the FM team co-created a quick-response (QR) code system for valves on the project. This digital service is now used by the FM team for training purposes and in emergency situations when information on valves and their service areas is quickly needed. In the future, the service will be linked to building information modeling (BIM) to better benefit from BIM in the operations and maintenance of the center. Researchers made on-site observations, conducted 84 interviews, and reviewed archival data in 2012, 2014, and 2016. Drawing from the case study, this paper provides a framework for successful co-creation of digital FM services within a construction project. The framework consists of three processes. 1) Dialogue process is promoted by co-located project parties and early and continuous involvement of project parties, a liaison, and the FM team’s experience both in construction and FM. 2) Trust building process can be accomplished by providing the FM team with access to project documentation and receiving constructive feedback from the FM team. 3) The creation of a shared understanding about the FM’s needs and the service providers’ ways of satisfying those needs is enabled by project parties learning from each other, FM’s empathy skills, risk assessment, and cost transparency.
  • Thumbnail Image
    Item
    Complex Online Material Development in CS Courses
    (2020-11-19) Haaranen, Lassi; Mariani, Giacomo; Sormunen, Peter; Lehtinen, Teemu; Department of Computer Science; Falkner, Nick; Seppala, Otto; Professorship Malmi L.; Computer Science - Computing education research and educational technology (CER); Department of Computer Science
    Computer Science (CS) education has a tradition of using online materials to teach courses, whether as a part of a course or having fully online courses. Commonly, the materials to present the subject matter are not just static objects like text or images but also contain automatically assessed exercises and other interactive content. This makes the course systems inherently tied to teaching – limiting pedagogical approaches, types of exercises, and available functionality. Increasingly, CS courses utilize multiple systems to handle various learning and course management related activities. The use of multiple systems brings about traditional software engineering problems, such as development, integration, maintenance, and testing. We present two small studies (case study and interviews) to highlight the issues in developing and running modern online CS courses. Based on these two studies we argue that online courses should be developed with a stronger software engineering approach considering the development process and tools. In addition, we define the term Complex Online Learning Material (COLM) to foster discussion and further research into improving instructor practices in online education.
  • No Thumbnail Available
    Item
    Esineiden internetin hyödyntäminen energiatehokkaissa ja älykkäissä rakennuksissa
    (2015-04-19) Lehtinen, Teemu; Främling, Kary; Tietotekniikan laitos; Perustieteiden korkeakoulu; Rousu, Juho
  • No Thumbnail Available
    Item
    Estimation of the costs of primary repairs of HPAC systems in dwelling houses
    (1988) Lehtinen, Teemu; Santaniemi, Ulla-Maija; Polvinen, Martti; Konetekniikan osasto; Teknillinen korkeakoulu; Helsinki University of Technology; Siren, Kai
  • Thumbnail Image
    Item
    An Event Listener or an Event Handler? Students Explain Event-drivenness in JavaScript
    (2021-11-17) Lukkarinen, Aleksi; Lehtinen, Teemu; Haaranen, Lassi; Malmi, Lauri; Professorship Malmi L.; Department of Computer Science; Computer Science Lecturers; Computer Science Professors; Seppälä, Otto; Petersen, Andrew
    When students in programming courses are taught event-driven programming (EDP) for the first time, they face new terminology and concepts that they should internalize. Moreover, they learn a fully new approach for reasoning about program logic and execution order. However, there is a lack of research in students’ understanding of these concepts. In this paper, we describe a study, in which we asked web development students to explain their conception of EDP: what are the main concepts involved and how they interact. Moreover, we asked them to explain the execution of a short piece of JavaScript code that focuses on basic usage of events and event listeners. The answers, which we requested as concept maps and text, were analyzed using inductive content analysis. Our results clearly demonstrate shortcomings in the students’ learning and illustrate various misunderstandings that they may have regarding EDP. Based on the findings, we give suggestions for improving the teaching of EDP.
  • No Thumbnail Available
    Item
    Improving alliance projects through facilitation
    (2016-05-30) Kokkonen, Anne; Lehtinen, Teemu; Lavikka, Rita; Department of Industrial Engineering and Management; Prins, Matthijs; Wamelink, Hans; Giddings, Bob; Ku, Kihong; Feenstra, Manon
    Alliance and other collaborative project delivery models such as integrated project delivery (IPD) represent a solution to decrease the fragmentation in the construction industry. New technology such as building information modelling (BIM) is also claimed to introduce more integration into the design and construction processes. However, an intensified collaboration is required for successful alliancing and BIM processes. The intensified collaboration does not seem to occur automatically after committing to a contract, but might often require help in the daily project activities. Facilitation is an activity used in other industries to help in accomplishing tasks by concentrating on the social processes of groups performing the activities. Currently, very little is known about the facilitation in the context of construction projects. Facilitation literature suggests that facilitation can support collaborative task performance with three elements: 1) the management of group process, 2) the management of content, and 3) facilitator’s substantive expertise. With a qualitative case study approach, we studied what kind of facilitation occurs in a Finnish alliance project and how current facilitation methods meet the project needs. The results suggest the presence of all the three facilitation elements, but also suggest possibilities for improvement. The results connect facilitation literature to collaborative construction projects. The empirical analysis also offers practical suggestions on how to apply facilitation in construction projects efficiently.
  • Thumbnail Image
    Item
    Improving Performance in Developing and Serving Complex Online Learning Materials
    (2020-10-20) Sormunen, Peter; Lehtinen, Teemu; Perustieteiden korkeakoulu; Haaranen, Lassi
    Nowadays, a major part of teaching and learning happens online. Course teachers and educators are using interactive learning content, such as simulations, visualizations and automatically assessed exercises, to improve and enhance the pedagogy of their courses. Especially in the field of computer science, these tools and resources are used to enrich the experience when learning programming. Serving this so called "Smart Learning Content" and other online material can produce performance issues which can cause a bad user experience for both course teachers and students. This thesis studies the current situation of serving online material in computer science and programming courses, focusing on the course teacher's workflow especially in a FITech course called Introduction to Web Development and Programming (IWDAP). This thesis also studies the current system (A+) used in IWDAP course and major issues with it. A prototype application was developed and realized based on the requirement elicitation, which was done by interviewing instructors mostly from universities. Based on interviews, we built evaluation criteria which was used to evaluate the prototype in addition to quantitative evaluation i.e. performance testing. Based on the evaluation of the prototype, we found that the new prototype makes teachers' workflow easier when developing online learning material. Furthermore, a better website quality and performance was achieved compared to the current platform. Also, the instructors' experiences and opinions about material development were gathered in the thesis.
  • Thumbnail Image
    Item
    Jask: Generation of Questions About Learners' Code in Java
    (2022-07-07) Santos, André; Soares, Tiago; Garrido, Nuno; Lehtinen, Teemu; Department of Computer Science; Professorship Malmi L.; ISCTE - Instituto Universitário de Lisboa
    We present Jask, a system capable of generating questions about a learner's code written in Java. Given Java code as input, Jask provides a set of meaningful questions formulated in terms of the actual code (using its constructs and identifiers) and the corresponding correct answers. We integrated Jask in a web-based system where students submit their code (e.g., from lab exercises), answer questions about it, and obtain immediate formative feedback with the correct answers. An initial study involving 123 distinct introductory programming students providing 2274 answers revealed that questions pertaining to program dynamics tend to register low scores, possibly evidencing fragile comprehension of programming constructs. Participants were surveyed, revealing a positive view towards the usefulness of Jask, especially with respect to consolidating terminology.
  • Thumbnail Image
    Item
    Learning Analytics for Teacher’s Dashboard in a Course Result System
    (2023-01-23) Ma, Mengmeng; Lehtinen, Teemu; Perustieteiden korkeakoulu; Korhonen, Ari
    With the digital transformation of learning and the spread of online learning under the impact of COVID-19, huge amounts of learning data are being generated. How to utilize and analyze this learning data has led to a new field -- Learning Analytics (LA). LA demonstrates the collection, use and analysis of data generated by students in the learning process to predict student behavior and provide feedback. The goal of this thesis is to redesign a course result system (OSR) with LA functions. OSR is a results registration system used by the Department of Computer Science at Aalto University for over a decade. The development technology used by OSR is outdated and does not have the satisfying LA features for teachers. Meanwhile, this thesis tries to build a prototype of the new system based on the cloud platform (Salesforce). The research questions are: what is the state of art of the old system, what LA functions the new system needs, and what are the requirements of the new system. In order to answer these three research questions, this thesis first conducts a literature review. This thesis reviews the literature on OSR, A+, LA, cloud service models, and Salesforce. Through literature review and practical experience of OSR, this thesis obtains the state of art of the existing system as well as possible functional requirements and LA requirements. This thesis uses the constructive research method to design the new system. To gather functional needs and LA requirements from users, this thesis performed semi-structured interviews with teachers. After analyzing and summarizing the interview results, this thesis concludes the user needs. After comparing, merging and screening the requirements from the OSR and LA reviews in the Literature Review chapter, the requirements of the new system are summarized in this thesis. As for final requirements, the following three findings are made in this thesis. First of all, teachers are satisfied with the final grade calculation function and database function provided by the old system. The new system can follow the design of final grade calculation function of the old system. Second, although teachers were satisfied with most of the features of the old system, they spent a lot of time verifying the results. This suggests that they need a more reassuring system. Third, the old system provided only some statistical functions, and few teachers used them. Teachers hope the new system will provide them with more LA features, such as producing periodic reports for them to monitor students' progress in real time.
  • Thumbnail Image
    Item
    Let's Ask AI About Their Programs : Exploring ChatGPT's Answers To Program Comprehension Questions
    (2024-05-24) Lehtinen, Teemu; Koutcheme, Charles; Hellas, Arto; Department of Computer Science; Professorship Malmi L.; Computer Science Lecturers; Computer Science - Computing education research and educational technology (CER); Lecturer Hellas Arto group
    Recent research has explored the creation of questions from code submitted by students. These Questions about Learners' Code (QLCs) are created through program analysis, exploring execution paths, and then creating code comprehension questions from these paths and the broader code structure. Responding to the questions requires reading and tracing the code, which is known to support students' learning. At the same time, computing education researchers have witnessed the emergence of Large Language Models (LLMs) that have taken the community by storm. Researchers have demonstrated the applicability of these models especially in the introductory programming context, outlining their performance in solving introductory programming problems and their utility in creating new learning resources. In this work, we explore the capability of the state-of-the-art LLMs (GPT-3.5 and GPT-4) in answering QLCs that are generated from code that the LLMs have created. Our results show that although the state-of-the-art LLMs can create programs and trace program execution when prompted, they easily succumb to similar errors that have previously been recorded for novice programmers. These results demonstrate the fallibility of these models and perhaps dampen the expectations fueled by the recent LLM hype. At the same time, we also highlight future research possibilities such as using LLMs to mimic students as their behavior can indeed be similar for some specific tasks.
  • Thumbnail Image
    Item
    Questions About Learners' Code: Extending Automated Assessment Towards Program Comprehension
    (Aalto University, 2024) Lehtinen, Teemu; Korhonen, Ari, Senior university lecturer, Aalto University, Department of Computer Science, Finland; Tietotekniikan laitos; Department of Computer Science; Learning + Technology Research Group (LeTech); Perustieteiden korkeakoulu; School of Science; Malmi, Lauri, Prof., Aalto University, Department of Computer Science, Finland
    Novice programmers have a limited understanding of the program code they produce. Their programs are often based on code snippets from examples and internet searches. Recently and rather suddenly, artificial intelligence has changed programming environments that can now suggest and complete entire programs based on the available context. However, the ability to comprehend and discuss programs is essential in becoming a programmer who is responsible for their work and can reliably solve problems as a member of a team. Many introductory programming courses have hundreds of students per teacher. Therefore, automated systems are often used to produce immediate feedback and assessment for programming exercises. Current systems focus on the created program and its requirements. Unfortunately, their feedback helps students in iterating toward acceptable code rather than acquiring a deep understanding of the program. This dissertation addresses that gap. The dissertation defines and introduces questions about learners' code (QLCs). After a student has submitted a program, they are asked automated, personal QLCs about the structure and the logic of their program. The dissertation describes a system to generate QLCs and contributes three open-source implementations supporting Java, JavaScript, and Python. The empirical contributions of the dissertation are based on multiple studies that research both quantitatively and qualitatively how novice programmers answer various types of QLCs. From the students, who create a correct program, as many as 20% may answer incorrectly about concepts that are critical to systematically reason about their program code. More than half of the students fail to mentally trace the execution of their program. This confirms that novices' program comprehension needs improvement and instructors may overestimate their abilities. The more students answer incorrectly to QLCs, the more they tinker with their code and have less success on the course. Current artificial intelligence systems respond to QLCs better than the average novice. However, they also lapse into humanlike errors producing failed reasoning about the code they generated, which could present an important learning opportunity for the critical use of AI in programming.
  • No Thumbnail Available
    Item
    Similarity Detection of Student Assignments in Computer Science
    (2021-10-18) Huusko, Henri; Sarolahti, Pasi; Lehtinen, Teemu; Perustieteiden korkeakoulu; Korhonen, Ari