Emot-CM-BERT: Adaptive Attention and Class-Aware Cross-Modal Learning for Emotion Recognition from Audio and Text

Emotion recognition from multimodal data has become increasingly vital for affective computing applications such as virtual assistants, social robots, and mental health monitoring. Among various modalities, audio and text offer complementary emotional signals, but effectively integrating them remains a challenge due to asynchronous signals, modality imbalance, and static fusion strategies. This paper presents Emot-CM-BERT, a cross-modal framework designed to recognize emotion from audio and text inputs. The model integrates acoustic features - such as intonation, pitch, and rhythm - with semantic features that capture the contextual meaning of language, using a masked multimodal attention mechanism that dynamically fuses information across modalities. To further optimize learning, curriculum learning is employed to progressively introduce training complexity, while class-aware sampling addresses imbalances in emotion label distributions. We evaluate Emot-CM-BERT on three benchmark datasets - CREMA-D, IEMOCAP, and CMU-MOSEI - encompassing both acted and naturalistic emotional expressions. Experimental results demonstrate that the proposed model achieves competitive or superior performance compared to baselines, particularly in real-world, conversational settings. The analysis further confirms its effectiveness in aligning heterogeneous features and generalizing across diverse emotions.