Agent-based multimodal sentiment analysis with vision-language models

Thumbnail Image

Files

master_Hao_Rijie_2025.pdf (2.56 MB)

URL

Journal Title

Journal ISSN

Volume Title

School of Electrical Engineering | Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.

Authors

Date

2025-11-22

Department

Major/Subject

Autonomous Systems

Mcode

Degree programme

Master's Programme in ICT Innovation
Master's Programme in ICT Innovation
Master's Programme in ICT Innovation

Language

en

Pages

49

Series

Abstract

Multimodal sentiment analysis (MSA) aims to understand human emotions by integrating cues from text, audio, and visual data. The rise of large language models (LLMs) and visual language models (VLMs) has significantly enhanced multimodal understanding capabilities. However, these models require substantial computational resources and are often difficult to interpret, making them challenging to adapt to specific tasks or domains. This paper introduces a agent-based MSA framework. This framework can effectively coordinate local fine-tuned VLMs and specialized traditional models. The system employs simple feature extraction, parameter-efficient fine-tuning(PEFT) LoRA, and modular agents to handle text, audio, and videos. It not only focuses on local fine-tuned models but also lays the foundation for future connections with online multimodal systems based on API. Experiments on the CMU-MOSEI dataset show that the locally fine-tuned models, especially Qwen2-VL-LoRA, can achieve performance levels comparable to traditional multimodal systems (such as Transformer-based architectures), while maintaining control and privacy integrity. These results indicate that reducing parameters to fine-tune large models can make them have the same efficacy as smaller, specialized multimodal models. This provides a scalable and understandable foundation for future integration with online multimodal reasoning systems. This study also examines the performance of the model on different datasets, the impact of removing system parts, and how the modular agent design supports flexible multimodal processing. The research results show that the efficient local coordination of VLMs can almost match specialized systems. This provides a scalable, understandable, and privacy-protected solid foundation for real-world emotion computing applications.

Description

Supervisor

Zhou, Quan

Thesis advisor

Wang, Haoming

Keywords

multimodal large language modal, vision-language model, agent framework, parameter-efficient fine-tuning, large language model, sentiment analysis

Other note

Citation

Permanent link to this item

https://urn.fi/URN:NBN:fi:aalto-202512159193

Collections

[dipl] Sähkötekniikan korkeakoulu / ELEC