Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition

Bagus Tris Atmaja

doi:10.1145/3689062.3689082

Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition

Bagus Tris Atmaja^*

^*Corresponding author for this work

National Institute of Advanced Industrial Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Automatic social perception recognition is a new task to mimic the measurement of human traits, which was previously done by humans via questionnaires. We evaluated unimodal and multimodal systems to predict agentive and communal traits from the LMU-ELP dataset. We optimized variants of recurrent neural networks from each feature from audio and video data and then fused them to predict the traits. Results on the development set show a consistent trend that multimodal fusion outperforms unimodal systems. The performance-weighted fusion also consistently outperforms mean and maximum fusions. We found two important factors that influence the performance of performance-weighted fusion. These factors are normalization and the number of models.

Original language	English
Title of host publication	MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop
Subtitle of host publication	Social Perception and Humor, Co-Located with: MM 2024
Publisher	Association for Computing Machinery, Inc
Pages	28-35
Number of pages	8
ISBN (Electronic)	9798400711992
DOIs	https://doi.org/10.1145/3689062.3689082
Publication status	Published - 28 Oct 2024
Externally published	Yes
Event	5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, MuSe 2024, in conjunction with ACM Multimedia 2024 - Melbourne, Australia Duration: 28 Oct 2024 → 1 Nov 2024

Publication series

Name	MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024

Conference

Conference	5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, MuSe 2024, in conjunction with ACM Multimedia 2024
Country/Territory	Australia
City	Melbourne
Period	28/10/24 → 1/11/24

Keywords

multimodal fusion
parameter optimization
sentiment analysis
social perception

Access to Document

10.1145/3689062.3689082

Cite this

Atmaja, B. T. (2024). Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition. In MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024 (pp. 28-35). (MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024). Association for Computing Machinery, Inc. https://doi.org/10.1145/3689062.3689082

Atmaja, Bagus Tris. / Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition. MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024. Association for Computing Machinery, Inc, 2024. pp. 28-35 (MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024).

@inproceedings{65518fef05b24d37b2a8b1cce64931c6,

title = "Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition",

abstract = "Automatic social perception recognition is a new task to mimic the measurement of human traits, which was previously done by humans via questionnaires. We evaluated unimodal and multimodal systems to predict agentive and communal traits from the LMU-ELP dataset. We optimized variants of recurrent neural networks from each feature from audio and video data and then fused them to predict the traits. Results on the development set show a consistent trend that multimodal fusion outperforms unimodal systems. The performance-weighted fusion also consistently outperforms mean and maximum fusions. We found two important factors that influence the performance of performance-weighted fusion. These factors are normalization and the number of models.",

keywords = "multimodal fusion, parameter optimization, sentiment analysis, social perception",

author = "Atmaja, {Bagus Tris}",

note = "Publisher Copyright: {\textcopyright} 2024 Copyright held by the owner/author(s).; 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, MuSe 2024, in conjunction with ACM Multimedia 2024 ; Conference date: 28-10-2024 Through 01-11-2024",

year = "2024",

month = oct,

day = "28",

doi = "10.1145/3689062.3689082",

language = "English",

series = "MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024",

publisher = "Association for Computing Machinery, Inc",

pages = "28--35",

booktitle = "MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop",

}

Atmaja, BT 2024, Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition. in MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024. MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024, Association for Computing Machinery, Inc, pp. 28-35, 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, MuSe 2024, in conjunction with ACM Multimedia 2024, Melbourne, Australia, 28/10/24. https://doi.org/10.1145/3689062.3689082

Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition. / Atmaja, Bagus Tris.
MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024. Association for Computing Machinery, Inc, 2024. p. 28-35 (MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition

AU - Atmaja, Bagus Tris

PY - 2024/10/28

Y1 - 2024/10/28

N2 - Automatic social perception recognition is a new task to mimic the measurement of human traits, which was previously done by humans via questionnaires. We evaluated unimodal and multimodal systems to predict agentive and communal traits from the LMU-ELP dataset. We optimized variants of recurrent neural networks from each feature from audio and video data and then fused them to predict the traits. Results on the development set show a consistent trend that multimodal fusion outperforms unimodal systems. The performance-weighted fusion also consistently outperforms mean and maximum fusions. We found two important factors that influence the performance of performance-weighted fusion. These factors are normalization and the number of models.

AB - Automatic social perception recognition is a new task to mimic the measurement of human traits, which was previously done by humans via questionnaires. We evaluated unimodal and multimodal systems to predict agentive and communal traits from the LMU-ELP dataset. We optimized variants of recurrent neural networks from each feature from audio and video data and then fused them to predict the traits. Results on the development set show a consistent trend that multimodal fusion outperforms unimodal systems. The performance-weighted fusion also consistently outperforms mean and maximum fusions. We found two important factors that influence the performance of performance-weighted fusion. These factors are normalization and the number of models.

KW - multimodal fusion

KW - parameter optimization

KW - sentiment analysis

KW - social perception

UR - http://www.scopus.com/inward/record.url?scp=85210830945&partnerID=8YFLogxK

U2 - 10.1145/3689062.3689082

DO - 10.1145/3689062.3689082

M3 - Conference contribution

AN - SCOPUS:85210830945

T3 - MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024

SP - 28

EP - 35

BT - MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop

PB - Association for Computing Machinery, Inc

T2 - 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, MuSe 2024, in conjunction with ACM Multimedia 2024

Y2 - 28 October 2024 through 1 November 2024

ER -

Atmaja BT. Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition. In MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024. Association for Computing Machinery, Inc. 2024. p. 28-35. (MuSe 2024 - Proceedings of the 5th Multimodal Sentiment Analysis Challenge and Workshop: Social Perception and Humor, Co-Located with: MM 2024). doi: 10.1145/3689062.3689082

Feature-wise Optimization and Performance-weighted Multimodal Fusion for Social Perception Recognition

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this