| Title |
Emotion-aware Multimodal Music Recommendation via Semantic Image-audio Alignment |
| Authors |
김재현(Jaehyun Kim) ; 임헌터(Hunter Lim) ; 양수영(Suyeong Yang) ; 이계식(Gyesik Lee) |
| DOI |
https://doi.org/10.5573/ieie.2026.63.7.95 |
| Keywords |
Image-based music recommendation; Semantic alignment; Emotional information; CLAP-based embeddings; Multimodal context-aware systems |
| Abstract |
This paper proposes an image-based music recommendation system that leverages semantic and emotional relationships between images and music. Conventional music recommendation approaches primarily rely on users' past listening histories or explicit text queries, which face limitations in reflecting users' immediate situational contexts or subtle emotional states in real-time. To address these challenges, this study utilizes images as intuitive inputs that encapsulate user context and proposes an Indirect Alignment framework that translates visual and emotional information into textual representations, subsequently linking them with a CLAP-based music embedding space. To verify the effectiveness of the proposed system, three prototype models with different text generation strategies were designed, and their recommendation performance was analyzed through a comparative study with existing audio-feature-based and text-based search models. In particular, an Ablation Study was conducted to define the specific problems each module aims to solve?such as bridging the domain gap and summarizing core contexts?and to analyze their respective contributions. Experimental results demonstrate that explicitly extracting and incorporating emotional information from images, rather than simply increasing the detail of visual descriptions, reduces bias in recommendation results and yields more balanced performance in terms of subjective emotional congruence compared to conventional methods. This study validates the practical feasibility of image-based music recommendation through efficient linkage between multimodal models and is expected to serve as a foundational reference for future research on advanced context-aware recommendation systems. |