Shintami Chusnul Hidayati, Kevin Davi Samuel, James Rafferty Lee, Yeni Anistyasari, Tse-Yu Pan
Emotion recognition is a cornerstone of human-computer interaction, enabling systems to respond in ways that are adaptive, empathetic, and contextually appropriate. While multimodal emotion recognition has gained interest by leveraging complementary cues from speech and text, two persistent challenges remain: effectively aligning heterogeneous modalities and mitigating class imbalance in large-scale datasets. This paper, therefore, introduces BiMER (Bidirectional Multimodal Emotion Recognition), a framework designed to jointly learn modality-invariant audio-text representations while addressing imbalance through targeted augmentation. BiMER employs a bidirectional joint encoder that encourages shared representations across audio and textual modalities. A representation alignment loss further ensures semantic consistency. For classification, an optimized multi-layer perceptron is applied to the joint representation space. To mitigate class imbalance, synthetic samples for minority emotions in harder-to-learn regions of the distribution are generated to the textual feature space. Experimental results on a challenging largescale benchmark show that BiMER consistently outperforms baselines, achieving the highest accuracy and recall across all configurations. © 2025 IEEE.
Institut Teknologi Sepuluh Nopember, Department of Informatics, Surabaya, Indonesia; Universitas Negeri Surabaya, Department of Informatics, Surabaya, Indonesia; Graduate Institute of A.I. Cross-disciplinary Tech., National Taiwan University of Science and Technology, Taipei, Taiwan
