TY - GEN
T1 - MEASURING EMOTION PRESERVATION IN EXPRESSIVE SPEECH-TO-SPEECH TRANSLATION
AU - Atmaja, Bagus Tris
AU - Shirai, Toru
AU - Sakti, Sakriani
N1 - Publisher Copyright:
© 2025 IEEE.
PY - 2025
Y1 - 2025
N2 - Measuring how well emotions are preserved in speech-to-speech translation is a difficult task. Previous research focused on measuring the similarity between speech embeddings related to emotion or prosody, which may not accurately capture the emotional contents. This study proposes more direct approaches by evaluating emotion preservation using metrics derived from speech emotion recognition (SER), including balanced accuracy ratio, emotion preservation rate, and Cohen's Kappa. The results show that about half of the original emotion is preserved in the translation process in the MELD-ST dataset, with metrics such as balanced accuracy ratio, valence-arousal similarity, and pause rate serving as reliable indicators of emotion preservation. We also found that high similarities between emotion embeddings do not necessarily mean emotion preservation, since the same acoustic embeddings used for SER lead to distinct performance. The analysis highlights the challenges of maintaining emotional consistency during speech-to-speech translation.
AB - Measuring how well emotions are preserved in speech-to-speech translation is a difficult task. Previous research focused on measuring the similarity between speech embeddings related to emotion or prosody, which may not accurately capture the emotional contents. This study proposes more direct approaches by evaluating emotion preservation using metrics derived from speech emotion recognition (SER), including balanced accuracy ratio, emotion preservation rate, and Cohen's Kappa. The results show that about half of the original emotion is preserved in the translation process in the MELD-ST dataset, with metrics such as balanced accuracy ratio, valence-arousal similarity, and pause rate serving as reliable indicators of emotion preservation. We also found that high similarities between emotion embeddings do not necessarily mean emotion preservation, since the same acoustic embeddings used for SER lead to distinct performance. The analysis highlights the challenges of maintaining emotional consistency during speech-to-speech translation.
KW - emotion preservation
KW - evaluation metrics
KW - expressive speech
KW - speech emotion recognition
KW - speech-to-speech translation
UR - https://www.scopus.com/pages/publications/105034882833
U2 - 10.1109/O-COCOSDA68185.2025.11385054
DO - 10.1109/O-COCOSDA68185.2025.11385054
M3 - Conference contribution
AN - SCOPUS:105034882833
T3 - 2025 28th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2025
BT - 2025 28th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2025
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 28th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques, O-COCOSDA 2025
Y2 - 12 November 2025 through 14 November 2025
ER -