This study is led by Prof. Jianhua Tao(Department of Automation，Tsinghua University) . Affective computing is an important branch in the field of artificial intelligence. Multimodal emotion understanding and interaction technologies aim to fully model multidimensional information from audio,video,and physiological signals to achieve more accurate emotion understanding, and receive increasing attention from researchers in fully exploiting the complementary nature of different modalities. Based on the detailed analyses of recent SOTA works, an overview is given in the three dimensions: an overview of multimodal sentiment recognition, multimodal sentiment understanding, and detection and assessment of emotional disorders such as depression.
Multimodal sentiment recognition and understanding are discussed in the context of emotion feature extraction, multimodal fusion, and the representation and models involved in sentiment recognition under the umbrella of pre-trained large models. LLMs, through self-supervised learning or contrastive learning, can learn more expressive multimodal representations that capture the correlations between different modalities and emotional information.
The authors believe that the main challenge in multimodal emotion recognition is data scarcity, which hinders the creation of robust models based on deep neural network methods. Addressing this issue involves constructing large-scale multimodal emotion databases and exploring transfer learning methods based on large models. The role of multimodal emotion computing in addressing emotional disorders like depression and anxiety is becoming increasingly significant, with future research focusing on dataset construction, algorithm development, and the design of intelligent psychological intervention systems.
See the article:
Development of multimodal sentiment recognition and understanding
https://doi.org/10.11834/jig.240017
