aalto1 untyped-item.component.html
Toward a unified framework for information retrieval in large language model applications: Balancing textual and graph-based knowledge sources
Loading...
URL
Journal Title
Journal ISSN
Volume Title
School of Engineering |
Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Authors
Date
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
54
Series
Abstract
This research investigates how the performance of Large Language Model (LLM) based applications can be improved by augmenting them with structured knowledge, specifically Knowledge Graphs (KGs). It is critical to integrate contextually correct information to mitigate hallucination in such LLM based retrieval systems. To answer this, the study focuses on automated KG generation using LLMs, and evaluates three retrieval methods: Semantic Text Retrieval (Vector Retriever), Hybrid Retrieval (Vector+Cypher Retriever), and Graph Traversal (Graph Retriever). These methods are systematically evaluated on manually curated datasets of simple and multi-step reasoning questions. The assessment is done both qualitatively and quantitatively, particularly assessing the correctness, clarity, informativeness, and faithfulness of the answers. Further, autonomous KG generation is critically evaluated to identify its impact on context retrieval. The findings highlight distinction in performance among retrievers. Graph traversal outperformed hybrid and semantic text retrieval in all the metrics for multi-step reasoning scenarios that required relational reasoning and gathering information from scattered sources. In contrast, for simple fact based queries hybrid retrieval gave the best results with 92\% correct answers. Based on these results, this research underscores Knowledge Augmented Generation (KAG)'s potential to boost accuracy in information retrieval, improved breath of coverage and robust adaptability. However, some limitations of KAG were also identified. These include consistency of autonomously generated KGs, proper handling of numeric data and scalability which point towards avenues for future research on validation frameworks, Knowledge Graph maintenance strategies and enhanced numeric data representation of data in KGs.