BiMER: Rethinking Multimodal Emotion Recognition with Bidirectional Representation Learning

Emotion recognition is a cornerstone of human-computer interaction, enabling systems to respond in ways that are adaptive, empathetic, and contextually appropriate. While multimodal emotion recognition has gained interest by leveraging complementary cues from speech and text, two persistent challenges remain: effectively aligning heterogeneous modalities and mitigating class imbalance in large-scale datasets. This paper, therefore, introduces BiMER (Bidirectional Multimodal Emotion Recognition), a framework designed to jointly learn modality-invariant audio-text representations while addressing imbalance through targeted augmentation. BiMER employs a bidirectional joint encoder that encourages shared representations across audio and textual modalities. A representation alignment loss further ensures semantic consistency. For classification, an optimized multi-layer perceptron is applied to the joint representation space. To mitigate class imbalance, synthetic samples for minority emotions in harder-to-learn regions of the distribution are generated to the textual feature space. Experimental results on a challenging largescale benchmark show that BiMER consistently outperforms baselines, achieving the highest accuracy and recall across all configurations.