Exploring the Utility of Large Language Models for Producing Insights from Discussion Recordings

Thumbnail Image

Files

master_Saj_Sneha_2025.pdf (410.13 KB)

URL

Journal Title

Journal ISSN

Volume Title

School of Science | Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2024-12-27

Department

Major/Subject

Computer Science

Mcode

Degree programme

Master's Programme in Computer, Communication and Information Sciences

Language

en

Pages

64

Series

Abstract

This thesis explores the application of large language models (LLMs), specifically GPT-4o and o1-preview, for the analysis of professional meeting transcripts, with a focus on identifying patterns of collaboration. Through structured experimentation with automated coding and prompt engineering techniques such as zero-shot, few-shot, and chain-of-thought prompting, this study evaluates the capabilities of LLMs to conduct detailed communication analysis. A systematic prompt design methodology was employed, comprising detailed initial instructions, a structured codebook, and illustrative examples. This approach enabled the evaluation of model performance in classifying themes, including interaction patterns, argumentative acts, emotional expressions, and cognitive processes. Experimental results reveal both strengths and critical limitations in the models' performance. LLMs demonstrate proficiency in identifying straightforward collaboration dynamics; however, they often struggle to accurately categorize complex behaviors and subtle emotional cues, particularly within the domains of emotional expression and cognitive depth. These findings indicate structural constraints in current LLM architectures for advanced communication analysis, suggesting a need for further model refinement. Notably, o1-preview outperformed GPT-4o in consistency and adaptability when employing few-shot and chain-of-thought prompts, suggesting the effectiveness of tailored prompting strategies in enhancing model interpretability. The insights provided by this thesis highlight the potential for LLMs to contribute to automated communication analysis within organizational settings, supporting productivity and decision-making through data-driven insights into team interactions. However, the study also emphasizes that achieving accurate, detailed analysis of collaborative behavior will require ongoing advancements in both model training and prompt design strategies. In conclusion, this thesis lays the foundation for future advancement in LLM-based communication analysis, identifying key areas where improvements in model accuracy and reliability could enhance the capture of complex collaboration themes across professional environments.

Description

Supervisor

Hellas, Arto

Keywords

Large Language Models (LLMs), meeting transcription analysis, collaboration patterns, prompt engineering, automated coding, zero-shot prompting, chain-of-thought prompting

Other note

Citation

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202501302209

Collections

[dipl] Perustieteiden korkeakoulu / SCI