Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms

Choi, Taeyeong, Pyenson, Benjamin, Liebig, Juergen and Pavlic, Theodore P. (2021) Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms. In: 4th International Symposium on Swarm Behavior and Bio-Inspired Robotics 2021 (SWARM2021), 01-04 Jun 2021, Kyoto, Japan.

WarningThere is a more recent version of this item available.

Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms
Accepted Manuscript of conference paper
DARS-SWARM2020_paper_140.pdf - Whole Document

Item Type:Conference or Workshop contribution (Paper)
Item Status:Live Archive


Most deep-learning frameworks for understanding biological swarms are designed to fit perceptive models of group behavior to individual-level data (e.g., spatial coordinates of identified features of individuals) that have been separately gathered from video observations. Despite considerable advances in automated tracking, these methods are still very expensive or unreliable when tracking large numbers of animals simultaneously. Moreover, this approach assumes that the human-chosen features include sufficient features to explain important patterns in collective behavior. To address these issues, we propose training deep network models to predict system-level states directly from generic graphical
features from the entire view, which can be relatively inexpensive to gather in a completely automated fashion. Because the resulting predictive models are not based on human-understood predictors, we use explanatory modules (e.g., Grad-CAM) that combine information hidden in the latent variables of the deep-network model with the video data itself to communicate to a human observer which aspects of observed individual behaviors are most informative in predicting group behavior. This represents an example of augmented intelligence in behavioral ecology -- knowledge co-creation in a human-AI team. As proof of concept, we utilize a 20-day video recording of a colony of over 50 Harpegnathos saltator ants to showcase that, without any individual annotations provided, a trained model can generate an "importance map" across the video frames to highlight regions of important behaviors, such as dueling
(which the AI has no a priori knowledge of), that play a role in the resolution of reproductive-hierarchy re-formation. Based on the empirical results, we also discuss the potential use and current challenges to further develop the proposed framework as a tool to discover behaviors that have not yet been considered crucial to understand complex social dynamics within biological collectives.

Keywords:Deep learning, behavioral ecology, Swarm Behavior, Explainable AI, Augmented Intelligence, knowledge co-production
Subjects:C Biological Sciences > C180 Ecology
C Biological Sciences > C120 Behavioural Biology
G Mathematical and Computer Sciences > G700 Artificial Intelligence
G Mathematical and Computer Sciences > G760 Machine Learning
G Mathematical and Computer Sciences > G740 Computer Vision
Divisions:College of Science > Lincoln Institute for Agri-Food Technology
ID Code:46374
Deposited On:21 Oct 2021 14:37

Available Versions of this Item

Repository Staff Only: item control page