Name: a step-by-step implementation of deepbehavior, deep learning toolbox for automated behavior analysis
Uploaded: 2020-02-06T15:00:00
Description: Watch this Scientific Journal Video about A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis at JoVE.com

We would like to thank Pingping Zhao and Peyman Golshani for providing the raw data for two-mouse social interaction tests used in the original paper2. This study was supported by NIH NS109315 and NVIDIA GPU grants (AA).

1. GPU and Python Setup
<ol>
	<li>GPU Software 
	When the computer is first setup for deep learning applications, GPU-appropriate software and drivers should be installed which can be found on the GPU&#39;s respective website. (see the Table of Materials for those used in this study).</li>
	<li>Python 2.7 Installation 
	Open a command line prompt on your machine. 
	Command line: sudo apt-get install python-pip python-dev python-virtualenv</li>
<...

<ol>
	<li>Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., Poeppel, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Neuroscience+Needs+Behavior:+Correcting+a+Reductionist+Bias.">Neuroscience Needs Behavior: Correcting a Reductionist Bias.</a> Neuron. 93 (3), 480-490 (2017).</li><li>Arac, A., Zhao, P., Dobkin, B. H., Carmichael, S. T., Golshani, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=DeepBehavior:+A+Deep+Learning+Toolbox+for+Automated+Analysis+of+Animal+and+Human+Behavior+Imaging+Data.">DeepBehavior: A Deep Learning Toolbox for Automated Analysis of Animal and Human Behavior Imaging Data.</a> Front Syst Neurosci. 13, 20 (2019).</li><li>Pereira, T. D., Aldarondo, D. E., Willmore, L., Kislin, M., Wang, S. S., Murthy, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+animal+pose+estimation+using+deep+neural+networks.">Fast animal pose estimation using deep neural networks.</a> Nat Methods. 16 (1), 117-125 (2019).</li><li>Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=DeepLabCut:+markerless+pose+estimation+of+user-defined+body+parts+with+deep+learning.">DeepLabCut: markerless pose estimation of user-defined body parts with deep learning.</a> Nat Neurosci. 21 (9), 1281-1289 (2018).</li><li>Stern, U., He, R., Yang, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Analyzing+animal+behavior+via+classifying+each+video+frame+using+convolutional+neural+networks.">Analyzing animal behavior via classifying each video frame using convolutional neural networks.</a> Sci Rep. 5, 14351 (2015).</li><li>Tinbergen, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=On+aims+and+methods+of+ethology.">On aims and methods of ethology.</a> Zeitschrift fu&#776;r Tierpsychologie. 20, 410-433 (1963).</li><li>LeCun, Y., Bengio, Y., Hinton, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+Learning.">Deep Learning.</a> Nature. 521 (7553), 436-444 (2015).</li><li>Zhao, Z., Zheng, P., Xu, S., Wu, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Object+Detection+With+Deep+Learning:+A+Review.">Object Detection With Deep Learning: A Review.</a> IEEE Transactions on Neural Networks and Learning Systems. , 1-21 (2019).</li><li>He, K., Zhang, X., Ren, S., Deep Sun, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Residual+Learning+for+Image+Recognition.">Residual Learning for Image Recognition.</a> arXiv. , (2015).</li><li>Krizhevsky, A., Sutskever, I., Hinton, G. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ImageNet+classification+with+deep+convolutional+neural+networks.">ImageNet classification with deep convolutional neural networks.</a> Proceedings of the 25th International Conference on Neural Information Processing Systems. 1, 1097-1105 (2012).</li><li>Szegedy, C., Wei, L., Yangqing, J., Sermanet, P., Reed, S., Anguelov, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Going+deeper+with+convolutions.">Going deeper with convolutions.</a> 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , 7-12 (2015).</li><li>Stewart, R., Andriluka, M., Ng, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=End-to-End+People+Detection+in+Crowded+Scenes.">End-to-End People Detection in Crowded Scenes.</a> 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , 27-30 (2016).</li><li>Redmon, J., Farhadi, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=YOLOv3:+An+Incremental+Improvement.">YOLOv3: An Incremental Improvement.</a> arXiv. , (2018).</li><li>Cao, Z., Simon, T., Wei, S. E., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Realtime+Multi-Person+2D+Pose+Estimation+using+Part+Affinity+Fields.">Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.</a> arXiv. , (2017).</li><li>Simon, T., Joo, H., Matthews, I., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hand+Keypoint+Detection+in+Single+Images+using+Multiview+Bootstrapping.">Hand Keypoint Detection in Single Images using Multiview Bootstrapping.</a> arXiv. , (2017).</li><li>Wei, S. E., Ramakrishna, V., Kanade, T., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Convolutional+Pose+Machines.">Convolutional Pose Machines.</a> arXiv. , (2016).</li></ol>

Here, we provide a step-by-step guide for implementation of DeepBehavior, our recently developed deep learning based toolbox for animal and human behavior imaging data analysis2. We provide detailed explanations for each step for installation of the frameworks for each network architecture, and provide links for installation of the open-source requirements to be able to run these frameworks. We demonstrate how to install them, how to create training data, how to train the network, and how to proce...

A detailed analysis of behavior is key to understanding the brain and behavior relationships. There have been many exciting advances in methodologies for recording and manipulating neuronal populations with high temporal resolution, however, behavior analysis methods have not developed at the same rate and are limited to indirect measurements and a reductionist approach1. Recently, deep learning based methods have been developed to perform automated and detailed behavior analysis2,3,4,5. This protocol provides a step-by-step implementation guide for the DeepBehavior toolbox.
Traditional behavioral analysis methods often include manually labeling data by multiple evaluators, leading to variance in how experimenters define a behavior6. Manual labeling of the data requires time and resources that increase disproportionately to the amount of data collected. Moreover, manually labelled data reduce the behavior outcomes into categorical measurements which do not capture the richness of the behavior, and will be more subjective. Thus, the current traditional methods may be limited in capturing the details in the natural behaviors.
The DeepBehavior toolbox presents a precise, detailed, highly temporal, and automated solution using deep learning for behavioral analysis. Deep learning has quickly become accessible to all with open-source tools and packages. Convolutional neural networks (CNNs) are proven to be highly effective in object recognition and tracking tasks7,8. Using modern day CNNs and high-performance graphics-processing-units (GPUs), large image and video datasets can be processed quickly with high precision7,9,10,11. In DeepBehavior, there are three different convolutional neural net architectures, TensorBox, YOLOv3, and OpenPose2.
The first framework, Tensorbox, is a versatile framework that incorporates many different CNN architectures for object detection12. TensorBox is best suited for detecting only one object class per image. The resulting outputs are bounding boxes of the object of interest (Figure 1) and the cartesian coordinates of the bounding box.
The second CNN framework is YOLOv3, which stands for &#34;You Only Look Once&#34;13. YOLOv3 is advantageous when there are multiple objects of interest that must be tracked separately. The output of this network includes the bounding box with the associated object label class as well as the bounding box cartesian coordinates of the object in the video frame (Figure 2).
The previous two frameworks are advantageous for generalized behavioral data collected from standard laboratory experiments in animal subjects. The last CNN framework is OpenPose14,15,16 which is used for human joint pose estimation. OpenPose detects human body, hand, facial, and foot key points on images. The outputs of the framework are labeled images of the human subject as well as the coordinates of all the 25 key points in the body and 21 key points of each hand (Figure 3).
This detailed step-by-step guide for implementation of our recently developed open-source DeepBehavior toolbox employs state-of-the-art convolutional neural nets to track animal behavior (e.g. movement of a paw) or human behavior (e.g. reaching tasks). By tracking the behavior, useful kinematics can be derived from the behavior such as position, velocity, and acceleration. The protocol explains the installation of each CNN architecture, demonstrates how to create training datasets, how to train the networks, how to process new videos on the trained network, how to extract the data from the network on the new videos, and how to post-process the output data to make it useful for further analysis.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>CUDA v8.0.61</td>
			<td>NVIDIA</td>
			<td>n/a</td>
			<td>GPU Software</td>
		</tr>
		<tr>
			<td>MATLAB R2016b</td>
			<td>Mathworks</td>
			<td>n/a</td>
			<td>Matlab</td>
		</tr>
		<tr>
			<td>Python 2.7</td>
			<td>Python</td>
			<td>n/a</td>
			<td>Python Version</td>
		</tr>
		<tr>
			<td>Quadro P6000</td>
			<td>NVIDIA</td>
			<td>n/a</td>
			<td>GPU Processor</td>
		</tr>
		<tr>
			<td>Ubuntu v16.04</td>
			<td>Ubuntu</td>
			<td>n/a</td>
			<td>Operating System</td>
		</tr>
	</tbody>
</table>

a step-by-step implementation of deepbehavior, deep learning toolbox for automated behavior analysis

Understanding behavior is the first step to truly understanding neural mechanisms in the brain that drive it. Traditional behavioral analysis methods often do not capture the richness inherent to the natural behavior. Here, we provide detailed step-by-step instructions with visualizations of our recent methodology, DeepBehavior. The DeepBehavior toolbox uses deep learning frameworks built with convolutional neural networks to rapidly process and analyze behavioral videos. This protocol demonstrates three different frameworks for single object detection, multiple object detection, and three-dimensional (3D) human joint pose tracking. These frameworks return cartesian coordinates of the object of interest for each frame of the behavior video. Data collected from the DeepBehavior toolbox contain much more detail than traditional behavior analysis methods and provides detailed insights to the behavior dynamics. DeepBehavior quantifies behavior tasks in a robust, automated, and precise way. Following the identification of behavior, post-processing code is provided to extract information and visualizations from the behavioral videos.

When the protocol is followed, the data for each network architecture should be similar to the following. For TensorBox, it outputs a bounding box around the object of interest. In our example, we used videos from a food pellet reaching task, and labeled the right paws to track their movement. As seen in Figure 1, the right paw can be detected in different positions in both the front view and side view cameras. After post-processing with camera calibration, 3...

Watch this Scientific Journal Video about A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis at JoVE.com

A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

Understanding behavior is the first step to truly understanding neural mechanisms in the brain that drive it. Traditional behavioral analysis methods ...

Performing a detailed behavior analysis is crucial to understanding brain behavior relationship. One of the best ways to evaluate behavior is through careful observations. However, quantifying the observed behavior is time consuming and challenging.Classical methods of behavior analysis are not easily quantifiable and are inherently subjective. Recent developments in deep learning, a branch of machine learning and artificial intelligence fields, provide opportunities for automated and objective quantification of images and videos. Here we present our recently developed methods utilizing deep neural networks to perform detailed behavior analysis in rodents and humans.The main advantage of this technique are its flexibility and applicability to any imaging data for behavior analysis. The DeepBehavior Toolbox supports single object identification, multi object detection, and human pose tracking. We also provide the post processing code in MATLAB for more in-depth kinematic analysis methods.Begin by setting up Tensor Box. Activate the environment, then use GitHub to clone Tensor Box and install it on the machine and on additional dependencies. Next, launch the labeling graphical user interface and label at least 600 images from a wide distribution of behavior frames.To label an image click the top left corner of the object of interest and then the bottom right corner. Then make sure that the bounding box captures the entire object. Click next to move to the next frame.To link the training images to a network hyper parameters file, open overfeat_rezoom. json in a text editor and replace the file path under train_idl to labels.json. Then add the same file path under test-idl and save the changes.Initiate the training script which will begin training for 600, 000 iterations and generate the resulting trained weights of the convolutional neural network in the output folder. Then perform prediction on new images, and view the outputs of the network as labeled images and as bounding box coordinates. Install YOLOv3.Then label the training data with Yolo_mark by placing the images in the Yolo_mark-data-obga folder and labeling them one by one in the graphical user interface. Label approximately 200 images. Next, set up the configuration file.To modify the configuration file open the YOLO-obj. cfg folder. Modify the batch, subdivision, and classes lines.Then change the filter for each convolution layer before a YOLO layer. Download the network weights and place them into the darknet-build. x64 folder.Run the training algorithm, and once it is complete view the iterations. To track multiple body parts in a human subject, install OpenPose then use it to process the desired video. The capabilities of the DeepBehavior Toolbox were demonstrated on videos of mice performing a food pellet reaching task.Their right paws were labeled and movement was tracked with front and side view cameras. After post processing with camera calibration, 3D trajectories of the reach were obtained. The outputs of YOLOv3 are multiple bounding boxes because multiple objects can be tracked.The bounding boxes are around the objects of interest which can be parts of the body. In OpenPose, the network detected the joint positions and after post processing with camera calibration, a 3D model of the subject was created. One critical step not covered in this protocol is ensuring that your device has the appropriate Python versions and dependencies as well as a GPU configured device before beginning.After successfully obtaining the track behavior from the network additional post processing can be done to further analyze the kinematics and patterns of the behavior. Why the DeepBehavior Toolbox is applicable for diagnostic approaches in disease models of rodents and human subjects is not direct therapeutic benefit. Use of these techniques as a diagnostic or prognostic tool is under active research within our laboratory.This technique is being used to investigate the neural mechanisms of skilled motor behavior in rodents as well as being used in clinical studies to evaluate motor recovery in patients with neurological diseases.

Research

JoVE Journal

Behavior

Contextual and Cued Fear Conditioning Test Using a Video Analyzing System in Mice

The contextual and cued fear conditioning test is one of the behavioral tests that assesses the ability of mice to learn and remember an association ...

Assessment of Social Cognition in Non-human Primates Using a Network of Computerized Automated Learning Device (ALDM) Test Systems

Assessment of Social Cognition in Non-human Primates Using a Network of Computerized Automated Learning Device ALDM Test Systems

Fagot &#38; Paleressompoulle1 and Fagot &#38; Bonte2 have published an automated learning device (ALDM) for the study of cognitive abilities of monkeys ...

Quantifying Learning in Young Infants: Tracking Leg Actions During a Discovery-learning Task

Task-specific actions emerge from spontaneous movement during infancy. It has been proposed that task-specific actions emerge through a discovery-learning ...

A General Method for Evaluating Deep Brain Stimulation Effects on Intravenous Methamphetamine Self-Administration

Substance use disorders, particularly to methamphetamine, are devastating, relapsing diseases that disproportionally affect young people.  There is a need ...

Protocol for Data Collection and Analysis Applied to Automated Facial Expression Analysis Technology and Temporal Analysis for Sensory Evaluation

We demonstrate a method for capturing emotional response to beverages and liquefied foods in a sensory evaluation laboratory using automated facial ...

Automated Analysis of a Nematode Population-based Chemosensory Preference Assay

The nematode, Caenorhabditis elegans' compact nervous system of only 302 neurons underlies a diverse repertoire of behaviors. To facilitate the dissection ...

Stress-Enhanced Fear Learning, a Robust Rodent Model of Post-Traumatic Stress Disorder

Fear behaviors are important for survival, but disproportionately high levels of fear can increase the vulnerability for developing psychiatric disorders ...

Paw-Print Analysis of Contrast-Enhanced Recordings (PrAnCER): A Low-Cost, Open-Access Automated Gait Analysis System for Assessing Motor Deficits

Paw-Print Analysis of Contrast-Enhanced Recordings PrAnCER: A Low-Cost, Open-Access Automated Gait Analysis System for Assessing Motor Deficits

Gait analysis is used to quantify changes in motor function in many rodent models of disease. Despite the importance of assessing gait and motor function ...

A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers

With the advent of new technological advances, it is possible to study social interactions at a microlevel with unprecedented accuracy. High frequency ...

Fully Automated Leg Tracking in Freely Moving Insects using Feature Learning Leg Segmentation and Tracking (FLLIT)

Fully Automated Leg Tracking in Freely Moving Insects using Feature Learning Leg Segmentation and Tracking FLLIT

The Drosophila model has been invaluable for the study of neurological function and for understanding the molecular and cellular mechanisms that underlie ...