Social Tracking Animals with Deep Learning

Swiss Federal Institute of Technology  EPFL   Deep Learning Tracking 

Time to get social: tracking animals with deep learning

Researchers at EPFL have made strides in computer-aided animal tracking by expanding their software, DeepLabCut, to offer high-performance tracking of multiple animals in videos.

The ability to capture the behavior of animals is critical for neuroscience, ecology, and many other fields. Cameras are ideal for capturing fine-grained behavior, but developing computer vision techniques to extract the animal’s behavior is challenging even though this seems effortless for our own visual system.

One of the key aspects of quantifying animal behavior is “pose estimation”, which refers to the ability of a computer to identify the pose (position and orientation of different body parts) of an animal. In a lab setting, it’s possible to assist pose estimation by placing markers on the animal’s body like in motion-capture techniques used in movies (think Gollum in the Lord of the Rings). But as one can imagine, getting animals to wear specialized equipment is not the easiest task, and downright impossible and unethical in the wild.

For this reason, Professors Alexander Mathis and Mackenzie Mathis at EPFL have been pioneering “markerless” tracking for animals. Their software relies on deep-learning to “teach” computers to perform pose estimation without the need for physical or virtual markers.

Their teams have been developing DeepLabCut, an open-source, deep-learning “animal pose estimation package” that can perform markerless motion capture of animals. In 2018 they released DeepLabCut, and the software has gained significant traction in life sciences: over 350,00 downloads of the software and nearly 1400 citations. Then, in 2020, the Mathis teams released DeepLabCut-Live!, a real-time low-latency version of DeepLabCut that allows researchers to rapidly give feedback to animals they are studying.

Now, the scientists have expanded DeepLabCut to address another challenge in pose estimation: tracking social animals, even closely interacting ones; e.g., parenting mice or schooling fish. The challenges here are obvious: the individual animals can be so similar looking that they confuse the computer, they can obscure each other, and there can be many “keypoints” that researchers wish to track, making it computationally difficult to process efficiently.

To tackle this challenge, they first created four datasets of varying difficulty for benchmarking multi-animal pose estimation networks. The datasets, collected with colleagues at MIT and Harvard University, consist of three mice in an open field, home-cage parenting in mice, pairs of marmosets housed in a large enclosure, and fourteen fish in a flow tank. With these datasets in hand, the researchers were able to develop novel methods to deal with the difficulties of real-world tracking. .....