Deciphering Multimodal Correspondence using Exploratory Data Analysis
Loading...
URL
Journal Title
Journal ISSN
Volume Title
Perustieteiden korkeakoulu |
Master's thesis
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Authors
Date
2023-06-12
Department
Major/Subject
Human-Computer Interaction
Mcode
SCI3097
Degree programme
Master's Programme in Computer, Communication and Information Sciences
Language
en
Pages
78+29
Series
Abstract
Artistic and creative processes rely on integrating information from multiple sensory modalities. However, understanding the complex interplay between these modalities and how they correlate remains a challenge. The methods followed in conventional behavioral and psychological experiments have been consistently qualitative and the correlations/correspondence have been traditionally found on the basis of the choices that the human participant thinks (pair-matching). These have proven to be the existential foundation of multimodal correlation studies however, a lack of a quantitative approach limits this experimental methodology to test only a few numbers of participants. Conventional pair/pattern matching experiments may not fully capture the underlying correlations in sensory multimodal data and Exploratory Data Analysis (EDA) based approaches can reveal hidden trends and insights. This thesis proposes Primary Evaluator for Multimodal Correlation (PEMC), a novel framework which provides a data-driven approach for exploring correlations between two or more sensory modalities. The framework emphasizes the importance of EDA techniques in identifying hidden patterns in sensory multimodal data, which may not be captured through conventional pair/pattern matching experiments. Utilizing various EDA techniques, such as dimensionality reduction, unsupervised clustering, and correlation analysis, we propose the Correlation Analyzer (CA), an integral part of PEMC. CA is used to identify correlations between two modalities. PEMC framework tries to conduct a preliminary evaluation of the existence of underlying correlations in sensory data using CA in 3 unique test settings. The results suggest that there exist multimodal correlations and recommend whether more controlled experiments are needed to establish the presence of universal multimodal correlations. In this thesis, we conduct an in-depth analysis of sensory multimodal data extracted from audio responses, pen movement responses, and colour transition data as stimulus data using the PEMC. Our findings reveal moderate to strong correlations in the features of audio and pen movement data in response to colour transition data, providing valuable insights into how different modalities interact and influence each other. Potential limitations of the framework, best practices and many applications of the correlation analysis are also discussed giving directions to future studies.Description
Supervisor
Takala, TapioThesis advisor
Okulov, JaanaKeywords
multimodal correlation, machine learning, exploratory data analysis, data analysis