Audiovisual matching of room acoustics in virtual reality

Abstract

Ensuring audio-visual consistency is crucial for building a credible virtual reality (VR) and augmented reality (AR) experience, as perceptual inconsistency can weaken the sense of presence and reality. Although V-to-A matching has been studied in VR, where users adjust auditory rendering based on visual scenes, there are still many issues that have not been fully explored. This paper systematically studies the indoor acoustic matching performance under four experimental conditions: Visual-to-Audio (V-to-A), Audiovisual-to-Audio (AV-to-V), Audio-to-Audio (A-to-A), and Audio-to-visual (A-to-V) further address the current research gaps.

A high-fidelity VR experiment was conducted using the measured spatial room impulse response and binaural audio presented by the panoramic visual scene. Our results demonstrate that matching performance improves significantly when acoustic references are available. In contrast, relying solely on V-to-A information leads to substantially poorer performance, whereas both AV-to-A and A-to-A conditions provide clear benefits. Furthermore, inconsistent source vision does not significantly undermine perceptual consistency. Although slight asymmetry was observed between the A-to-V and V-to-A tasks, it was not statistically significant. In terms of analysis, we determined that reverberation and clarity are the main auditory cues (loads on PC1) for perceiving the size and distance of space, while the secondary factor (PC2) is irrelevant in perception.

This work provides a fundamental framework for designing a coherent spatial audio rendering system in immersive media, emphasizing the relative importance of auditory and visual references in achieving perceptual consistency in AR/VR.