Statistical emotion control: Comparing intensity and duration of emotional reactions based on facial expressions

Abstract

The aim is to develop an intelligent automatic facial expression recognition and emotion analysis (AFEREA) algorithm that, first, characterizes the time-based raw signals of biosensors in quantitative indicators of the emotional state of the individuals participating in an experiment and, second, compares the emotional reactions across them in terms of intensity and duration. The proposed Statistical Emotion Control (SEC) intelligent algorithm is based on statistical process control (SPC) theory. After representing the individuals’ baseline behaviour in a non-normal I-chart and describing the output per subject in emotional peaks with their corresponding duration in terms of relative cutoffs, SEC uses Poisson c-charts to compare across subjects in terms of the quantity of peaks and binomial p-charts in terms of length of the emotional reactions. To validate the datadriven algorithm, the state-of-the-art iMotions software and its AFFDEX face recognition and emotion analysis algorithm is used to record the individuals while receiving the results of their economic decisions when playing an experimental business game. The SEC intelligent algorithm is proven to be useful to take the raw output of the biosensors, to characterize the intensity and duration of the emotional reactions as well as to compare across subjects by emotion. SEC recognizes “out of control” negative emotions more often (7.25% vs. 2.00%) and positive emotions as often (15.63%) by setting relative cutoffs instead of traditional absolute thresholds. The results show significant pairwise discrepancies among both tested settings in 7.86% of the recorded 560 combinations of emotions and individuals, with a high 43.59% among those timeseries with the maximum recorded value above the traditional threshold of 50.