机器人系列animaltracking for behavior

Robotics and Intelligent Machines Georgia Tech Animal Tracking for Behavior Modeling James M. Rehg Georgia Tech (standing in for Tucker Balch) Robotics and Intelligent Machines Georgia Tech People PI Tucker Balch Faculty Jim Rehg, Computer Science Collaborator on animal tracking Aaron Bobick, Computer Science Collaborator on animal behavior modeling Bruce Walker, Psychology Collaborator on aquarium project Atsushi Nakazawa, Computer Science (Osaka Univ., Japan) Collaborator on animal tracking PhD Students Michael Novitzky Jin Lee Matthew Flagg Robotics and Intelligent Machines Georgia Tech Goals Track social animals reliably under natural conditions (in vivo) from video Estimate behavioral parameters from tracking data Construct executable models of social animal behavior Develop a biologically-inspired protocol for dynamic team formation Robotics and Intelligent Machines Georgia Tech Year 1 Overview Goal Develop reliable multi-target tracking algorithms for animals in video Approach Jointly estimate segmentation and motion of a nonrigid, deformable target Key requirements Reliably estimate target shape over time, to support behavioral analysis Minimize the amount of human effort required Robotics and Intelligent Machines Georgia Tech Challenges Accurate segmentation of target Set-point tracking is not sufficient Track reliably with significant camera motion Background subtraction is not sufficient Handle a wide range of animals Scalable solution for the animal kingdom Track multiple interacting targets Multiple instances of target type (e.g. wolf pack) Occlusions (with self, other targets, and background) Track long sequences, track across cuts Robotics and Intelligent Machines Georgia Tech Approach Basic research in video object segmentation and tracking Modular software architecture Easily change features, models, and solver Testing in two scenarios Aquarium Monitoring Long sequences, static camera, some modeling effort Tracking Animal Planet videos Shorter sequences, moving camera, minimal human effort Preliminary application Accessible Aquarium Project Robotics and Intelligent Machines Georgia Tech Tracking App Basic Trackers: -Contour feature w/ Iterated Closest Point -Color feature w/ Mean Shift Robotics and Intelligent Machines Georgia Tech Color Histogram Tracker Fish Model: Appearance model For each species of fish, multiple HSV Color Histograms on Image patches Off-line Model Selection by Human Detection and Tracking: Histogram-based mean shift approach Maximization of Bhattacharyya Coefficient between Observation and Model Selection of a model with the highest coefficient and update track Robotics and Intelligent Machines Georgia Tech Histogram-based Model Selection Model Image Patches (to build histogram) Input Image Similarity between shifted & model patches 0. 6 0. 5 0. 7 0. 8 0. 3 Best Model * Image Patch after Mean-shifting Robotics and Intelligent Machines Georgia Tech Accessible Aquarium Project Provide a meaningful and informative aquarium experience for visually-impaired or blind visitors Approach Track the movement of individual fish within tank Sonify the fish movement Example: 65 gallon marine aquarium Track yellow tangs and blue chromis Music structure is Bach chorale Each fish type is same instrument, different registers Movement speed mapped to note density (tempo) Horizontal dimension is stereo, vertical is timbre Robotics and Intelligent Machines Georgia Tech Example Play video Robotics and Intelligent Machines Georgia Tech Tracking in the Video Volume Robotics and Intelligent Machines Georgia Tech Graphcut Tracking Robotics and Intelligent Machines Georgia Tech Tracking Results Robotics and Intelligent Machines Georgia Tech Social Game Retrieval Robotics and Intelligent Machines Georgia Tech Overview Robotics and Intelligent Machines Georgia Tech Summary of Progress Robust long-life tracking of multiple targets (fish) Tracking under controlled (but realistic) conditions Virtual Aquarium Project Novel experience of animal behavior via sonification New state-of-art motion segmentation algorithm Accurate segmentation of (shorter) video sequences under a wide range conditions Robotics and Intelligent Machines Georgia Tech Year 2 Pl