class: center, middle, inverse, title-slide # .center[Disenchantment with Emotion Recognition Technologies] ## .center[A Comparison Between Humans Observers and Automatic Classifiers] ### Damien Dupré ### Dublin City University ### December 7th, 2020 --- layout: true <div class="custom-footer"><span>AICS - Dupré (2020) </span></div> --- # Why Measuring Emotions? The emotional experience **determines our perceptions** and **leads our decisions** in every life (e.g., the Phineas Gage Case; see also [Bechara, Damasio, & Damasio, 2000](https://doi.org/10.1093/cercor/10.3.295)) <img src="aics_2020_files/media/phineas_gage.jpg" width="50%" style="display: block; margin: auto;" /> .center.tiny[Modeling the path of the tamping iron through the Gage skull and its effects on white matter structure.<br />Credit: Van Horn, Irimia, Torgerson, Chambers, Kikinis & Toga (2012) [🔗](https://doi.org/10.1371/journal.pone.0037454)] --- # Characteristics of Emotions .left-column[ <img src="aics_2020_files/media/emo_event.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_appraisal.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_sync.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_change.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_behaviour.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_intensity.png" width="27%" style="display: block; margin: auto;" /> <img src="aics_2020_files/media/emo_rapidity.png" width="27%" style="display: block; margin: auto;" /> .center.tiny[Adapted from Scherer (2005) [🔗](https://doi.org/10.1177/0539018405058216)] ] .right-column[ ] -- ### Event Focus -- ### Appraisal Driven -- ### Response Synchronization -- ### Rapidity of Change -- ### Behavioural Impact -- ### Intense Response -- ### Short Duration --- # Affective Computing Research on emotions have lead to a "**conceptual and definitional chaos**" ([Buck, 1990, p. 330](https://doi.org/10.1207/s15327965pli0104_15)): * There is still no consensual agreement between researchers * Some assumptions of the broad audience are not supported by scientific evidences -- .pull-left[ However, multiple tools and databases have been developed to investigate emotions. With the **increase in computer processing power** and the **development of machine learning algorithm**, computer scientists have created models to automatically recognize emotions... **What Could Possibly Go Wrong?** ] .pull-right[ <img src="aics_2020_files/media/rise_affective_computing.png" width="80%" style="display: block; margin: auto;" /> .center.tiny[Credit: The Guardian (2019) [🔗](https://www.theguardian.com/technology/2019/mar/06/facial-recognition-software-emotional-science)] ] --- # Automatic Facial Expression Recognition **Development of the technology:** * First attempt by reported by [Suwa, Sugie, & Fujimura (1978)](https://books.google.ie/books?id=P4s-AQAAIAAJ) * Numerous academic systems since (see revue by [Zeng, Pantic, Roisman, & Huang, 2009]()) * VicarVision to develop the first commercial automatic classifier ([den Uyl & van Kuilenburg, 2005](http://www.vicarvision.nl/pub/fc_denuyl_and_vankuilenburg_2005.pdf)) * Today more than 20 companies for applications to automotive, sport, health, human resources, security or marketing purposes ([Dupré, Andelic, Morrison, & McKeown, 2018](https://doi.org/10.1109/PERCOMW.2018.8480127)) -- .pull-left[ **A process in 3 steps:** * Face Detection * Facial Landmark Detection * Classification Result is a recognition probability for a labelled category (e.g., Action Unit, Basic Emotion, Dimensions) ] .pull-right[ <img src="aics_2020_files/media/automatic_steps.png" width="70%" style="display: block; margin: auto;" /> .center.tiny[Credit: Dupré, Andelic, Morrison & McKeown (2018) [🔗](https://doi.org/10.1109/PERCOMW.2018.8480127)] ] --- # Facial Expression Categorization Emotion categories/dimensions are inferred from facial expressions either: .pull-left[ * **Directly, by matching Action Units** to prototypical expressions of emotions (Emotion coded by the FACS; [Ekman, Friesen, & Hager, 2002](https://www.paulekman.com/facial-action-coding-system/)) <img src="aics_2020_files/media/emfacs_example.jpg" width="85%" style="display: block; margin: auto;" /> .center.tiny[Credit: Bartlett, Littlewort, Frank, Lainscsek, Fasel, & Movellan (2006) [🔗](https://www.doi.org/10.1109/FGR.2006.55)] ] .pull-right[ * **Indirectly, by generalizing features** learnt from training with specific databases (pictures or video, posed or spontaneous) <img src="aics_2020_files/media/affdex_example.jpg" width="42%" style="display: block; margin: auto;" /> .center.tiny[Credit: ThinkApps [🔗](http://thinkapps.com/blog/development/machine-intelligence-affectiva-interview/)] ] --- # Paper Objective In April 24, 2020 we have published in **Plos One** a paper entitled "**A performance comparison of eight commercially available automatic classifiers for facial affect recognition**" ([DOI: 10.1371/journal.pone.0231968](https://doi.org/10.1371/journal.pone.0231968)) -- Comparison of 8 commercially available automatic emotion recognition classifier: <table class="table" style="font-size: 13px; margin-left: auto; margin-right: auto;"> <caption style="font-size: initial !important;">Detail of the eight automatic classifiers tested</caption> <thead> <tr> <th style="text-align:left;"> Company </th> <th style="text-align:left;"> Classifier </th> <th style="text-align:left;"> Access </th> <th style="text-align:left;"> Version </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> Affectiva </td> <td style="text-align:left;"> Affdex </td> <td style="text-align:left;"> SDK </td> <td style="text-align:left;"> v3.4.1 </td> </tr> <tr> <td style="text-align:left;"> CrowdEmotion </td> <td style="text-align:left;"> FaceVideo </td> <td style="text-align:left;"> API </td> <td style="text-align:left;"> v1.0 </td> </tr> <tr> <td style="text-align:left;"> Emotient </td> <td style="text-align:left;"> Facet </td> <td style="text-align:left;"> SDK </td> <td style="text-align:left;"> v6.3 </td> </tr> <tr> <td style="text-align:left;"> Microsoft </td> <td style="text-align:left;"> Cognitive Services </td> <td style="text-align:left;"> API </td> <td style="text-align:left;"> v1.0 </td> </tr> <tr> <td style="text-align:left;"> MorphCast </td> <td style="text-align:left;"> EmotionalTracking </td> <td style="text-align:left;"> SDK </td> <td style="text-align:left;"> v1.0 </td> </tr> <tr> <td style="text-align:left;"> Neurodata Lab </td> <td style="text-align:left;"> EmotionRecognition </td> <td style="text-align:left;"> API </td> <td style="text-align:left;"> v1.0 </td> </tr> <tr> <td style="text-align:left;"> VicarVison </td> <td style="text-align:left;"> FaceReader </td> <td style="text-align:left;"> Software </td> <td style="text-align:left;"> v7.0 </td> </tr> <tr> <td style="text-align:left;"> VisageTechnologies </td> <td style="text-align:left;"> FaceAnalysis </td> <td style="text-align:left;"> SDK </td> <td style="text-align:left;"> v1.0 </td> </tr> </tbody> </table> .center[**Evaluation of classifiers' accuracy compared to human observers to recognise emotions from facial expressions**] --- # Method Two databases were selected, **each including videos supposedly expressing one of the six following emotions: happiness, sadness, anger, fear, surprise, and disgust** .pull-left[ ### **BU4DFE** - 467 Videos - Posed Expressions (asked to express emotions) - Age range of 18-45 years (Participants of Western European descent mostly) ] .pull-right[ ### **UT-Dallas** - 470 Videos - Spontaneous Expressions (watching eliciting videos) - Age range of 18-25 years (Participants of Western European descent mostly) ] -- All the video were processed by: - 8 automatic classifiers producing a frame-by-frame probability of recognition for all six emotion categories - 14 human observers (10 females, `\(M_{age} = 24.0\)`, `\(SD = 6.62\)`) who were asked to watch all the videos an to categorize it as expressing one of the six emotion categories --- # Data Analysis Both results from human observers and from automatic classifiers needed to be **pre-processed to identify the emotion label attributed to each video**: -- - Human Observers: Among the 14 chosen labels, the label being the most used to describe the video is the label recognized by human observers `\begin{equation} {EmoRec_{i,j}} = \max\left(\frac{1}{K}\sum_{k=1}^{K}EmoRec_{i,j,k}\right) \end{equation}` -- - Automatic Classifiers: For each video, the highest sum of the frame-by-frame probabilities is the label recognized by a classifier `\begin{equation} {EmoRec_{i,j}} = \max\left(\frac{\sum_{x = 0}^{T}\psi_{x,i,j}}{\sum_{j = 1}^{J}\sum_{x = 0}^{T}\psi_{x,i,j}}\right) \end{equation}` -- **Once a recognized label is obtained for human observers and for each classifier, it is possible to compare them with the label corresponding to the emotion expressed.** --- # Overall Results ### For how many videos is the emotion supposedly expressed corresponding to the emotion recognized? <img src="aics_2020_files/slides_files/figure-html/unnamed-chunk-13-1.png" width="720" style="display: block; margin: auto;" /> --- # Overall Results ### What about the distinction between posed and spontaneous expressions? <img src="aics_2020_files/slides_files/figure-html/unnamed-chunk-14-1.png" width="720" style="display: block; margin: auto;" /> --- # Recognition Accuracy ### Comparison of classification performance using Receiver Operating Characteristic curves <img src="aics_2020_files/slides_files/figure-html/unnamed-chunk-15-1.png" width="720" style="display: block; margin: auto;" /> --- # Recognition Error Automation facial expression recognition algorithms are designed to infer emotions in a controlled laboratory setting. They may not be accurate once applied to the real world or to different context. -- .pull-left[ Face recognition depends on: - Face orientation (e.g., inclination, rotation) - Face features (e.g., glasses, beard, face mask) - Context light - Morphological facial configurations ] .pull-right[ <img src="aics_2020_files/media/interstellar_affdex.gif" width="100%" style="display: block; margin: auto;" /> .center.tiny[Interstellar by Affdex. Credit: Affectiva [🔗](https://www.youtube.com/watch?v=NsmAldoVwDs)] ] --- # Prototypical Expressions Both facial expressions and physiological rhythms are proxies to infer emotions **based on theoretical assumptions** .pull-left[ In the case of facial expressions, a majority of databases used to train automatic classifiers considers: .small[ - Six emotions are universal (happiness, surprise, sadness, disgust, fear, anger) - These 6 emotions have prototypical representations ] As a result, automatic classifiers cannot recognize the diversity of facial expressions: .small[ - More than 6 categories of facial expressions - Difficulty to identify subtle and mixed expressions ] ] .pull-right[ <img src="aics_2020_files/media/six_basic_emotion.jpg" width="100%" style="display: block; margin: auto;" /> .center.tiny[Credit: Ekman, Friesen, & Hager (2002) [🔗](https://www.paulekman.com/facial-action-coding-system/)] ] --- # Meaning is Context Dependent A same facial expression can be interpreted differently according to the context in which the expression is produced Examples of athletes' victory (e.g., raging or crying after wining; see [Martinez, 2019](https://doi.org/10.1073/pnas.1902661116)) <img src="aics_2020_files/media/automatic_rec.png" width="100%" style="display: block; margin: auto;" /> .center.tiny[Emotion recognized as 'Anger' but the context reveals an experience closer to 'Intense Joy'.] --- # Absence of Scientific Support Despite the development of automatic classifiers on the idea that emotional categories can be inferred from sensors, there is **no scientific evidence** of reliable expressive and physiological patterns corresponding to emotional categories: .pull-left[ * No one-to-one mapping between patterns and categories ([Kappas, 2003](https://doi.org/10.1007/978-1-4615-1063-5_11)) * Facial expression often communicates something other than an emotional state ([Barrett, Adolphs, Marsella, Martinez, & Pollak, 2019](https://doi.org/10.1177/1529100619832930)) ] .pull-right[ <img src="aics_2020_files/media/tweet_truthbegolduk.png" width="100%" style="display: block; margin: auto;" /> ] --- # Current Challenges With regard to what has been said, we have two problems here, and data/emotion privacy is not one of them: 1. Expressive measures are prone to errors 2. Models used by automatic classifiers to categorise emotions are not reliable Therefore, **should we use these automatic classifiers?** -- .pull-left[ > *"Regulators should ban the use of affect recognition in important decisions that impact people's lives and access to opportunities. Until then, AI companies should stop deploying it."* - [AI Now Institute (2019)](https://ainowinstitute.org/AI_Now_2019_Report.html) ] .pull-right[ <img src="aics_2020_files/media/ai_now.png" width="80%" style="display: block; margin: auto;" /> .center.tiny[Credit: Tech Xplore (2019) [🔗](https://techxplore.com/news/2019-12-ai-watchdogs-rips-emotion-tech.html)] ] --- # Future Directions The precision of devices is should be monitored and standard to provide safe guards to users should be defined -- Most of automatic classifiers of facial expressions have already moved from a classification in categories to a classification in dimensions such as Valence/Pleasure and Arousal/Activation: * More reliable scientific evidences for a dimensional perspective * Not restricted to specific patterns -- .pull-left[ Additionally, errors in face and physiological measures are reducing with improved techniques and materials. > *"All models are wrong, but some are useful"* - [Box (1979)](https://doi.org/10.1016/B978-0-12-438150-6.50018-2) ] .pull-right[ <img src="aics_2020_files/media/tweet_digitaltrends.png" width="100%" style="display: block; margin: auto;" /> ] --- class: inverse, mline, left, middle <img class="circle" src="https://github.com/damien-dupre.png" width="250px"/> # Thanks for your attention, find me at... [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> @damien_dupre](http://twitter.com/damien_dupre) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 496 512"><path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"/></svg> @damien-dupre](http://github.com/damien-dupre) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M326.612 185.391c59.747 59.809 58.927 155.698.36 214.59-.11.12-.24.25-.36.37l-67.2 67.2c-59.27 59.27-155.699 59.262-214.96 0-59.27-59.26-59.27-155.7 0-214.96l37.106-37.106c9.84-9.84 26.786-3.3 27.294 10.606.648 17.722 3.826 35.527 9.69 52.721 1.986 5.822.567 12.262-3.783 16.612l-13.087 13.087c-28.026 28.026-28.905 73.66-1.155 101.96 28.024 28.579 74.086 28.749 102.325.51l67.2-67.19c28.191-28.191 28.073-73.757 0-101.83-3.701-3.694-7.429-6.564-10.341-8.569a16.037 16.037 0 0 1-6.947-12.606c-.396-10.567 3.348-21.456 11.698-29.806l21.054-21.055c5.521-5.521 14.182-6.199 20.584-1.731a152.482 152.482 0 0 1 20.522 17.197zM467.547 44.449c-59.261-59.262-155.69-59.27-214.96 0l-67.2 67.2c-.12.12-.25.25-.36.37-58.566 58.892-59.387 154.781.36 214.59a152.454 152.454 0 0 0 20.521 17.196c6.402 4.468 15.064 3.789 20.584-1.731l21.054-21.055c8.35-8.35 12.094-19.239 11.698-29.806a16.037 16.037 0 0 0-6.947-12.606c-2.912-2.005-6.64-4.875-10.341-8.569-28.073-28.073-28.191-73.639 0-101.83l67.2-67.19c28.239-28.239 74.3-28.069 102.325.51 27.75 28.3 26.872 73.934-1.155 101.96l-13.087 13.087c-4.35 4.35-5.769 10.79-3.783 16.612 5.864 17.194 9.042 34.999 9.69 52.721.509 13.906 17.454 20.446 27.294 10.606l37.106-37.106c59.271-59.259 59.271-155.699.001-214.959z"/></svg> damien-datasci-blog.netlify.app](https://damien-datasci-blog.netlify.app) [<svg style="height:0.8em;top:.04em;position:relative;" viewBox="0 0 512 512"><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg> damien.dupre@dcu.ie](mailto:damien.dupre@dcu.ie)