A Multimodal Emotion Recognition Approach for Socially Assistive Robots

Social assistive robots need emotion recognition to enable adaptive and user-centered interactions. Traditional facial expression recognition is often unreliable, especially in real-world settings. Moreover, most literature studies face the classification of binary conditions considering the Russell affective state model. This study proposes a multimodal emotion recognition system, integrating physiological monitoring with kinematic analysis from RGB-D skeletal tracking to capture both autonomic and postural markers of emotion to estimate the seven basic emotions of Ekman’s model. Results show that Random Forest achieves the highest accuracy (71.30±3.47%) when using Multimodal features to classify the seven basic emotions, significantly outperforming physiological and kinematic data alone. The findings confirm that body movements and autonomic responses provide complementary information, enhancing emotion classification. This approach offers a robust alternative to methods that rely on a single modality, improving the feasibility of real-time emotion recognition in social robotics.