Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms

Choi, Taeyeong, Pyenson, Benjamin, Liebig, Juergen and Pavlic, Theodore P. (2022) Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms. Artificial Life and Robotics . ISSN 1433-5298

Full content URL: https://doi.org/10.1007/s10015-022-00753-y

This is the latest version of this item.

Documents
Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms
Accepted manuscript
[img] PDF
DARS-SWARM2020_paper_140.pdf - Whole Document
Restricted to Repository staff only until 23 March 2023.

4MB
Item Type:Article
Item Status:Live Archive

Abstract

Most deep-learning frameworks for understanding biological swarms are designed to fit perceptive models of group behavior to individual-level data (e.g., spatial coordinates of identified features of individuals) that have been separately gathered from video observations. Despite considerable advances in automated tracking, these methods are still very expensive or unreliable when tracking large numbers of animals simultaneously. Moreover, this approach assumes that the human-chosen features include sufficient features to explain important patterns in collective behavior. To address these issues, we propose training deep network models to predict system-level states directly from generic graphical features from the entire view, which can be relatively inexpensive to gather in a completely automated fashion. Because the resulting predictive models are not based on human-understood predictors, we use explanatory modules (e.g., Grad-CAM) that combine information hidden in the latent variables of the deep-network model with the video data itself to communicate to a human observer which aspects of observed individual behaviors are most informative in predicting group behavior. This represents an example of augmented intelligence in behavioral ecology—knowledge co-creation in a human–AI team. As proof of concept, we utilize a 20-day video recording of a colony of over 50 Harpegnathos saltator ants to showcase that, without any individual annotations provided, a trained model can generate an "importance map" across the video frames to highlight regions of important behaviors, such as dueling (which the AI has no a priori knowledge of), that play a role in the resolution of reproductive-hierarchy re-formation. Based on the empirical results, we also discuss the potential use and current challenges to further develop the proposed framework as a tool to discover behaviors that have not yet been considered crucial to understand complex social dynamics within biological collectives.

Keywords:Deep learning, behavioral ecology, Swarm Behavior, Explainable AI, Augmented Intelligence, knowledge co-production, Ant colony, Animal Behaviour, Computer Vision, collective behaviour, Generative adversarial network, biological systems, Monitoring system, Anomaly Detection
Subjects:G Mathematical and Computer Sciences > G700 Artificial Intelligence
G Mathematical and Computer Sciences > G400 Computer Science
D Veterinary Sciences, Agriculture and related subjects > D300 Animal Science
C Biological Sciences > C180 Ecology
C Biological Sciences > C120 Behavioural Biology
G Mathematical and Computer Sciences > G760 Machine Learning
G Mathematical and Computer Sciences > G740 Computer Vision
C Biological Sciences > C100 Biology
Divisions:College of Science > Lincoln Institute for Agri-Food Technology
ID Code:48680
Deposited On:29 Mar 2022 14:52

Available Versions of this Item

Repository Staff Only: item control page