Automating Information Extraction from Financial Reports Using LLMs

Thumbnail Image

Files

master_Gomes_Ziegler_Gabriel_2024.pdf (3.02 MB)

URL

Journal Title

Journal ISSN

Volume Title

School of Science | Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2024-09-23

Department

Major/Subject

Data Science

Mcode

Degree programme

Master's Programme in ICT Innovation

Language

en

Pages

50

Series

Abstract

This thesis investigates the application of the latest Large Language Model (LLM)s for the automated extraction of Environmental, Social, and Governance (ESG) indicators from financial reports, a critical task for companies focused on keeping up-to-date sustainability reporting. The study explores the trade-offs in three distinct approaches: text-only, image-only, and multimodal, to evaluate their effectiveness in a real-world diverse dataset of financial reports. The text-only approach, although effective for documents with structured textual data, struggled with visual-rich content, leading to high residual difference between the extracted and actual values. The image-only approach, while adept at interpreting visual elements, faced challenges with hallucinations and lacked the accuracy of text-based methods that tend to get more perfect matches than the image-based counterparts. The multimodal approach, which combines text and image data, demonstrated superior performance, achieving the highest accuracy with a perfect match rate exceeding 85% for key indicators like Scope 1 and Scope 3 emissions. This method effectively mitigated errors through cross-validation between text and image data, resulting in minimal residuals and reliable data extraction. The study’s findings underscore the possibility of using the three approaches according to the nature of the dataset having a more confident usage when applying the combined multimodal approach in Document AI, particularly in format-agnostic scenarios such as the one presented in this study. The thesis also identifies challenges, such as the ambiguity in sub-indicators under Scope 2 emissions, and the importance of examining the tolerance for errors in the context of the application, where residuals can be acceptable or make the system completely unusable. This work contributes to the growing field of DocumentAI, offering insights into the capabilities and limitations of LLM for financial report analysis, and suggests pathways for future advancements in automated information retrieval systems.

Description

Supervisor

Zhao, Bo

Thesis advisor

Shakhpazyan, Liliya

Keywords

large language models, document AI, information extraction, GPT-4, deep learning, machine learning, natural language processing

Other note

Citation

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202411217264

Collections

[dipl] Perustieteiden korkeakoulu / SCI