Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration

Boneh-Shitrit, Tali, Feighelstein, Marcelo, Bremhorst, Annika , Amir, Shir, Distelfeld, Tomer, Dassa, Yaniv, Yaroshetsky, Sharon, Riemer, Stefanie, Shimshoni, Ilan, Mills, Daniel and Zamansky, Anna (2022) Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration. Nature, 12 (22611). pp. 1-16. ISSN 0028-0386

Full content URL: https://doi.org/10.1038/s41598-022-27079-w

Documents
Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration
Published manuscript
[img]
[Download]
[img]
Preview
PDF
BONEH-~1.PDF - Whole Document
Available under License Creative Commons Attribution 4.0 International.

2MB
Item Type:Article
Item Status:Live Archive

Abstract

In animal research, automation of affective states recognition has so far mainly addressed pain in a few species. Emotional states remain uncharted territories, especially in dogs, due to the complexity of their facial morphology and expressions. This study contributes to fill this gap in two aspects. First, it is the first to address dog emotional states using a dataset obtained in a controlled experimental setting, including videos from (n = 29) Labrador Retrievers assumed to be in two experimentally induced emotional states: negative (frustration) and positive (anticipation). The dogs’ facial expressions were measured using the Dogs Facial Action Coding System (DogFACS). Two different approaches are compared in relation to our aim: (1) a DogFACS-based approach with a two-step pipeline consisting of (i) a DogFACS variable detector and (ii) a positive/negative state Decision Tree classifier; (2) An approach using deep learning techniques with no intermediate representation. The approaches reach accuracy of above 71% and 89%, respectively, with the deep learning approach performing better. Secondly, this study is also the first to study explainability of AI models in the context of emotion in animals. The DogFACS-based approach provides decision trees, that is a mathematical representation which reflects previous findings by human experts in relation to certain facial expressions (DogFACS variables) being correlates of specific emotional states. The deep learning approach offers a different, visual form of explainability in the form of heatmaps reflecting regions of focus of the network’s attention, which in some cases show focus clearly related to the nature of particular DogFACS variables. These heatmaps may hold the key to novel insights on the sensitivity of the network to nuanced pixel patterns reflecting information invisible to the human eye.

Keywords:Animal Behaviour, Facial Expressions
Subjects:D Veterinary Sciences, Agriculture and related subjects > D300 Animal Science
D Veterinary Sciences, Agriculture and related subjects > D390 Veterinary Sciences not elsewhere classified
Divisions:College of Science > School of Life and Environmental Sciences > Department of Life Sciences
Related URLs:
ID Code:53467
Deposited On:17 Feb 2023 17:10

Repository Staff Only: item control page