Automatic emotion recognition is a burgeoning field of research and has its roots in psychology and cognitive science. This article comprehensively reviews multimodal emotion recognition, covering various aspects such as emotion theories, discrete and dimensional models, emotional response systems, datasets, and current trends. This article reviewed 179 multimodal emotion recognition literature papers from 2017 to 2023 to reflect on the current trends in multimodal affective computing. This article covers various modalities used in emotion recognition based on the emotional response system under four categories: subjective experience comprising text and self‐report; peripheral physiology comprising electrodermal, cardiovascular, facial muscle, and respiration activity; central physiology comprising EEG, neuroimaging, and EOG; behavior comprising facial, vocal, whole‐body behavior, and observer ratings. This review summarizes the measures and behavior of each modality under various emotional states. This article provides an extensive list of multimodal datasets and their unique characteristics. The recent advances in multimodal emotion recognition are grouped based on the research focus areas such as emotion elicitation strategy, data collection and handling, the impact of culture and modality on multimodal emotion recognition systems, feature extraction, feature selection, alignment of signals across the modalities, and fusion strategies. The recent multimodal fusion strategies are detailed in this article, as extracting shared representations of different modalities, removing redundant features from different modalities, and learning critical features from each modality are crucial for multimodal emotion recognition. This article summarizes the strengths and weaknesses of multimodal emotion recognition based on the review outcome, along with challenges and future work in multimodal emotion recognition. This article aims to serve as a lucid introduction, covering all aspects of multimodal emotion recognition for novices.This article is categorized under: Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction Technologies > Cognitive Computing Technologies > Artificial Intelligence

中文翻译：

---

多模态情绪识别：全面回顾、趋势和挑战


自动情绪识别是一个新兴的研究领域，起源于心理学和认知科学。本文全面回顾了多模态情绪识别，涵盖了情绪理论、离散和维度模型、情绪反应系统、数据集和当前趋势等各个方面。本文回顾了 2017 年至 2023 年的 179 篇多模态情绪识别文献论文，以反思当前多模态情感计算的趋势。本文涵盖了基于情绪反应系统的情绪识别中使用的各种模式，分为四类：包括文本和自我报告的主观体验;外周生理学包括皮肤电、心血管、面部肌肉和呼吸活动;中枢生理学包括脑电图、神经影像学和 EOG;行为包括面部、发声、全身行为和观察者评分。本文总结了各种情绪状态下每种模式的测量和行为。本文提供了多模态数据集及其独特特征的广泛列表。多模态情感识别的最新进展根据研究重点领域进行分组，如情绪诱发策略、数据收集和处理、文化和模态对多模态情绪识别系统的影响、特征提取、特征选择、跨模态信号对齐和融合策略。本文详细介绍了最近的多模态融合策略，因为提取不同模态的共享表示，从不同模态中删除冗余特征，以及从每种模态中学习关键特征对于多模态情绪识别至关重要。 本文根据综述结果总结了多模态情绪识别的优缺点，以及多模态情绪识别面临的挑战和未来工作。本文旨在作为一个清晰的介绍，涵盖新手多模态情绪识别的各个方面。本文分为： 数据和知识的基本概念 > 以人为本和用户交互技术 > 认知计算技术 > 人工智能