Combined frame- and event-based detection and tracking

Liu, Hongjie, Moeys, Diederik Paul, Das, Gautham , Neil, Daniel, Liu, Shih-Chii and Delbruck, Tobi (2016) Combined frame- and event-based detection and tracking. In: 2016 IEEE International Symposium on Circuits and Systems (ISCAS), Montreal, QC, Canada.

Full content URL:

Combined frame- and event-based detection and tracking
Published Manuscript
[img] PDF
2016_ISCAS_Liu_et_al.pdf - Whole Document
Restricted to Repository staff only

Item Type:Conference or Workshop contribution (Presentation)
Item Status:Live Archive


This paper reports an object tracking algorithm for a moving platform using the dynamic and active-pixel vision sensor (DAVIS). It takes advantage of both the active pixel sensor (APS) frame and dynamic vision sensor (DVS) event outputs from the DAVIS. The tracking is performed in a three step-manner: regions of interest (ROIs) are generated by a cluster-based tracking using the DVS output, likely target locations are detected by using a convolutional neural network (CNN) on the APS output to classify the ROIs as foreground and background, and finally a particle filter infers the target location from the ROIs. Doing convolution only in the ROIs boosts the speed by a factor of 70 compared with full-frame convolutions for the 240x180 frame input from the DAVIS. The tracking accuracy on a predator and prey robot database reaches 90% with a cost of less than 20ms/frame in Matlab on a normal PC without using a GPU.

Keywords:Event-based tracking and detection, DVS, DAVIS, particle filtering, Convolutional Neural Networks
Subjects:G Mathematical and Computer Sciences > G740 Computer Vision
Divisions:College of Science > Lincoln Institute for Agri-Food Technology
ID Code:40825
Deposited On:30 Sep 2020 10:40

Repository Staff Only: item control page