We would like to thank Pingping Zhao and Peyman Golshani for providing the raw data for two-mouse social interaction tests used in the original paper2. This study was supported by NIH NS109315 and NVIDIA GPU grants (AA).

1. GPU and Python Setup
<ol>
	<li>GPU Software 
	When the computer is first setup for deep learning applications, GPU-appropriate software and drivers should be installed which can be found on the GPU&#39;s respective website. (see the Table of Materials for those used in this study).</li>
	<li>Python 2.7 Installation 
	Open a command line prompt on your machine. 
	Command line: sudo apt-get install python-pip python-dev python-virtualenv</li>
</ol>
2. TENSORBOX
<ol>
	<li>Tensorbox Setup
	<ol>
		<li>Create Virtual Environment for Tensorbox 
		Command line: cd ~ 
		Command line: virtualenv --system-site-packages ~/tensorflow 
		NOTE: &#39;~/tensorflow&#39; is the name of the environment and is arbitrary</li>
		<li>Activate environment 
		Command line: source ~/tensorflow/bin/activate</li>
	</ol>
	</li>
	<li>Tensorbox Installation 
	We will be using GitHub to clone TensorBox from <a href="http://github.com/aarac/TensorBox" target="_blank">http://github.com/aarac/TensorBox</a> and install it on our machine as well as installing additional dependencies. 
	Command line: cd ~ 
	Command line: git clone <a href="http://github.com/aarac/TensorBox" target="_blank">http://github.com/aarac/TensorBox</a> 
	Command line: cd TensorBox 
	Command line: pip install -r requirements.txt</li>
	<li>Label Data
	<ol>
		<li>Create a folder of images of behavior 
		Open source tools such as ffmpeg are useful to accomplish converting videos to individual frames We recommend labeling at least 600 images from a wide-distribution of behavior frames for training. Put these images in a folder.</li>
		<li>Launch labeling graphical user interface 
		Command line: python make_json.py &#60;path to image folder&#62; labels.json 
		To label an image, click the top left corner of the object of interest (i.e. paw) first and then click the bottom right corner of the object of interest (Figure 4). Inspect that the bounding box captures the entire object of interest. Press &#39;undo&#39; to re-label the same image or press &#39;next&#39; to move onto the next frame.</li>
	</ol>
	</li>
	<li>Train TensorBox
	<ol>
		<li>Link training images to network hyperparameters file 
		Within the tensorbox folder, open the following folder in a text editor: 
		/TensorBox/hypes/overfeat_rezoom.json. Navigate to the attribute under data named train_idl and replace the file path from ./data/brainwash/train_boxes.json to the labels.json filepath. Save the changes to file.</li>
		<li>Begin training script 
		Command line: cd ~/TensorBox 
		Command line: python train.py --hypes hypes/overfeat_rezoom.json --gpu 0 --logdir output 
		The network will then begin training for 600,000 iterations. In the output folder, the resulting trained weights of the convolutional neural network will be generated.</li>
	</ol>
	</li>
	<li>Predict on New Images 
	For image labeling: 
	Command line: cd ~/TensorBox 
	Command line: python label_images.py --folder &#60;path to image folder&#62; --weights output/overfeat_rezoom_&#60;timestamp&#62;/save.ckpt-600000 --hypes /hypes/overfeat_rezoom.json --gpu 0 
	To get coordinates of bounding boxes: 
	Command line: cd ~/TensorBox 
	Command line: python predict_images_to_json.py --folder &#60;path to image folder&#62; --weights 
	output/overfeat_rezoom_&#60;timestamp&#62;/save.ckpt-600000 --hypes 
	/hypes/overfeat_rezoom.json --gpu 0</li>
	<li>MATLAB Post-Processing for TensorBox 
	Additional MATLAB code has been provided to extract kinematics and visualizations of the coordinates using the resulting JSON coordinate file from the model 
	Run the &#34;Process_files_3Dreaching_mouse.m&#34; script for 3D kinematic analysis of single food pellet reaching task.</li>
</ol>
3. YOLOv3
<ol>
	<li>Install YOLOv3 
	Command line: cd ~ 
	Command line: git clone cd darknet 
	For GPU usage, open &#39;Makefile&#39; and change the following lines: GPU=1; CUDNN=1. 
	Command line: make</li>
	<li>Labeling Training Data using Yolo_mark 
	Command line: cd ~ 
	Command line: git clone cd ~/Yolo_Mark 
	Command line: cmake . 
	Command line: make 
	Place the training images in ~/Yolo_mark/data/obj folder 
	Command line: chmod +x ./linux_mark.sh 
	Command line: ./linux_mark.sh 
	Label the images one by one in the graphical user interface (Figure 5). The recommended amount of images is approximately 200.</li>
	<li>Training YOLOv3
	<ol>
		<li>Setup configuration file 
		Command line: cd ~/Yolo_mark 
		Command line: scp -r ./data ~/darknet 
		Command line: cd ~/darknet/cfg 
		Command line: cp yolov3.cfg yolo-obj.cfg</li>
		<li>Modify the configuration file 
		Open the yolo-obj.cfg folder and modify the following lines: batch=64, subdivision=8, classes=(# of class to detect), and for each convolutional layer before a yolo layer change the filter=(classes+5)x3. Details on these changes can be found at <a href="https://github.com/aarac/darknet/blob/master/README.md" target="_blank">https://github.com/aarac/darknet/blob/master/README.md</a></li>
		<li>Download network weights 
		Download the network weights from <a href="https://www.dropbox.com/s/613n2hwm5ztbtuf/darknet53.conv.74?dl=0" target="_blank">https://www.dropbox.com/s/613n2hwm5ztbtuf/darknet53.conv.74?dl=0</a> 
		Place the downloaded weight file into ~/darknet/build/darknet/x64</li>
		<li>Run training algorithm 
		Command line: cd ~/darknet 
		Command line: ./darknet detector train data/obj.data cfg/yolo-obj.cfg darknet53.conv.74</li>
		<li>YOLOv3 Evaluation 
		After the training is complete based on a set number of iterations (ITERATIONNUMBER), you can view them by 
		Command line: ./darknet detector test data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_ITERATIONNUMBER.weights &#60;IMAGE&#62;.jpg</li>
	</ol>
	</li>
	<li>Predict on new videos and get coordinates 
	This command can be run to obtain the coordinates of the labels in the new video: 
	Command line: ./darknet detector demo data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_ITERATIONNUMBER.weights VIDEO.avi -ext_output &#60;VIDEO.avi&#62; FILENAME.txt</li>
	<li>YOLOv3 PostProcessing in MATLAB 
	Take the FILENAME.txt file to MATLAB, and run the &#34;Process_socialtest_mini.m&#34; script for two mice social interaction test. See results in Figure 2</li>
</ol>
4. OpenPose
OpenPose is ideal to track multiple body parts in a human subject. The setup and installation processes are very similar to the previous two frameworks. However, there is no training step as the network is already trained on human data.
<ol>
	<li>OpenPose Installation 
	Navigate to <a href="https://github.com/aarac/openpose" target="_blank">https://github.com/aarac/openpose</a> and follow the installation instructions.</li>
	<li>Process Video 
	./build/examples/openpose/openpose.bin --video VIDEONAME.avi --net_resolution &#34;1312x736&#34; --scale_number 4 --scale_gap 0.25 --hand --hand_scale_number 6 --hand_scale_range 0.4 --write_json JSONFOLDERNAME --write_video RESULTINGVIDEONAME.avi 
	Here the --net_resolution, --scale_number, --scale_gap, --hand_scale_number and --hand_scale_range handles can be omitted if a high precision detection is not needed (this would decrease the processing time).</li>
	<li>OpenPose Post-Processing 
	In MATLAB folder, please use &#39;process_files_human3D.m&#39; script to run the code after adding the appropriate folder containing json files from cameras 1 and 2, as well as the calibration file. This will create a &#34;cell&#34; file with all the 3D poses of the joints. It will also make a movie of the 3D skeletal view. For camera calibration, please follow the instructions at this link: <a href="http://www.vision.caltech.edu/bouguetj/calib_doc/" target="_blank">http://www.vision.caltech.edu/bouguetj/calib_doc/</a></li>
</ol>

<ol>
	<li>Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., Poeppel, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Neuroscience+Needs+Behavior:+Correcting+a+Reductionist+Bias.">Neuroscience Needs Behavior: Correcting a Reductionist Bias.</a> Neuron. 93 (3), 480-490 (2017).</li><li>Arac, A., Zhao, P., Dobkin, B. H., Carmichael, S. T., Golshani, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=DeepBehavior:+A+Deep+Learning+Toolbox+for+Automated+Analysis+of+Animal+and+Human+Behavior+Imaging+Data.">DeepBehavior: A Deep Learning Toolbox for Automated Analysis of Animal and Human Behavior Imaging Data.</a> Front Syst Neurosci. 13, 20 (2019).</li><li>Pereira, T. D., Aldarondo, D. E., Willmore, L., Kislin, M., Wang, S. S., Murthy, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+animal+pose+estimation+using+deep+neural+networks.">Fast animal pose estimation using deep neural networks.</a> Nat Methods. 16 (1), 117-125 (2019).</li><li>Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=DeepLabCut:+markerless+pose+estimation+of+user-defined+body+parts+with+deep+learning.">DeepLabCut: markerless pose estimation of user-defined body parts with deep learning.</a> Nat Neurosci. 21 (9), 1281-1289 (2018).</li><li>Stern, U., He, R., Yang, C. H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Analyzing+animal+behavior+via+classifying+each+video+frame+using+convolutional+neural+networks.">Analyzing animal behavior via classifying each video frame using convolutional neural networks.</a> Sci Rep. 5, 14351 (2015).</li><li>Tinbergen, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=On+aims+and+methods+of+ethology.">On aims and methods of ethology.</a> Zeitschrift fu&#776;r Tierpsychologie. 20, 410-433 (1963).</li><li>LeCun, Y., Bengio, Y., Hinton, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+Learning.">Deep Learning.</a> Nature. 521 (7553), 436-444 (2015).</li><li>Zhao, Z., Zheng, P., Xu, S., Wu, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Object+Detection+With+Deep+Learning:+A+Review.">Object Detection With Deep Learning: A Review.</a> IEEE Transactions on Neural Networks and Learning Systems. , 1-21 (2019).</li><li>He, K., Zhang, X., Ren, S., Deep Sun, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Residual+Learning+for+Image+Recognition.">Residual Learning for Image Recognition.</a> arXiv. , (2015).</li><li>Krizhevsky, A., Sutskever, I., Hinton, G. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=ImageNet+classification+with+deep+convolutional+neural+networks.">ImageNet classification with deep convolutional neural networks.</a> Proceedings of the 25th International Conference on Neural Information Processing Systems. 1, 1097-1105 (2012).</li><li>Szegedy, C., Wei, L., Yangqing, J., Sermanet, P., Reed, S., Anguelov, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Going+deeper+with+convolutions.">Going deeper with convolutions.</a> 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , 7-12 (2015).</li><li>Stewart, R., Andriluka, M., Ng, A. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=End-to-End+People+Detection+in+Crowded+Scenes.">End-to-End People Detection in Crowded Scenes.</a> 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , 27-30 (2016).</li><li>Redmon, J., Farhadi, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=YOLOv3:+An+Incremental+Improvement.">YOLOv3: An Incremental Improvement.</a> arXiv. , (2018).</li><li>Cao, Z., Simon, T., Wei, S. E., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Realtime+Multi-Person+2D+Pose+Estimation+using+Part+Affinity+Fields.">Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.</a> arXiv. , (2017).</li><li>Simon, T., Joo, H., Matthews, I., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Hand+Keypoint+Detection+in+Single+Images+using+Multiview+Bootstrapping.">Hand Keypoint Detection in Single Images using Multiview Bootstrapping.</a> arXiv. , (2017).</li><li>Wei, S. E., Ramakrishna, V., Kanade, T., Sheikh, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Convolutional+Pose+Machines.">Convolutional Pose Machines.</a> arXiv. , (2016).</li></ol>

The authors have nothing to disclose.

Here, we provide a step-by-step guide for implementation of DeepBehavior, our recently developed deep learning based toolbox for animal and human behavior imaging data analysis2. We provide detailed explanations for each step for installation of the frameworks for each network architecture, and provide links for installation of the open-source requirements to be able to run these frameworks. We demonstrate how to install them, how to create training data, how to train the network, and how to process new video files on the trained network. We also provide the post-processing code to extract the basic necessary information needed for further analysis.
For single object detection, we recommend using TensorBox. If the goal is to track multiple objects at once, we recommend using YOLOv3. Finally, to obtain human kinematic data, we recommend using OpenPose. In this protocol we have shown that deep learning methods are able to process hundreds of thousands of frames while tracking objects with a high degree of precision. Using the post-processing code provided, we can derive meaningful ways of analyzing the tracked behavior of interest. This provides a more detailed way of capturing behavior. It also provides an automated, robust way of defining behavior that is generalizable to many different types of behavioral tasks.
It is quite common to get a &#39;ModuleNotFoundError&#39; when starting with a new virtual environment or code that has been downloaded from the internet. In the case that this occurs, open up your terminal, activate the source environment and type &#39;pip install &#60;missing module name&#62;&#39;. If the problem persists, you will need to check your python version as well as other dependency packages.
Limitations to this technique include the technical troubleshooting to properly set up GPU processing units compatible with open-source code. It is advantageous to have past programming experience within a linux environment to properly set up the necessary project dependencies and environments that are compatible with the computer&#39;s hardware.
We demonstrate the DeepBehavior toolbox installations and processing of in a linux environment, however, this toolbox can also be run on a Windows and Mac machines with GPUs by following the respective installation guides on github.
Using deep learning methods for imaging data analysis is a very efficient way to automate behavior analysis. In comparison to traditional behavior analysis methods, DeepBehavior captures much more information to quantify, automate, and evaluate the behavior at a more precise and temporally detailed way. With the further advances in the deep learning field, the utilization and extent of the use of this technology in behavior analysis will likely continue to improve. The applications of DeepBehavior can be expanded beyond the demonstrated reaching tasks to identify objects of interest in any behavioral images. In this protocol, we provide detailed instructions to implement three neural networks for behavior analysis. With this kind of automated and unbiased behavior analysis methods, hopefully, the neuroscience field will be able to do more detail behavior analysis.

A detailed analysis of behavior is key to understanding the brain and behavior relationships. There have been many exciting advances in methodologies for recording and manipulating neuronal populations with high temporal resolution, however, behavior analysis methods have not developed at the same rate and are limited to indirect measurements and a reductionist approach1. Recently, deep learning based methods have been developed to perform automated and detailed behavior analysis2,3,4,5. This protocol provides a step-by-step implementation guide for the DeepBehavior toolbox.
Traditional behavioral analysis methods often include manually labeling data by multiple evaluators, leading to variance in how experimenters define a behavior6. Manual labeling of the data requires time and resources that increase disproportionately to the amount of data collected. Moreover, manually labelled data reduce the behavior outcomes into categorical measurements which do not capture the richness of the behavior, and will be more subjective. Thus, the current traditional methods may be limited in capturing the details in the natural behaviors.
The DeepBehavior toolbox presents a precise, detailed, highly temporal, and automated solution using deep learning for behavioral analysis. Deep learning has quickly become accessible to all with open-source tools and packages. Convolutional neural networks (CNNs) are proven to be highly effective in object recognition and tracking tasks7,8. Using modern day CNNs and high-performance graphics-processing-units (GPUs), large image and video datasets can be processed quickly with high precision7,9,10,11. In DeepBehavior, there are three different convolutional neural net architectures, TensorBox, YOLOv3, and OpenPose2.
The first framework, Tensorbox, is a versatile framework that incorporates many different CNN architectures for object detection12. TensorBox is best suited for detecting only one object class per image. The resulting outputs are bounding boxes of the object of interest (Figure 1) and the cartesian coordinates of the bounding box.
The second CNN framework is YOLOv3, which stands for &#34;You Only Look Once&#34;13. YOLOv3 is advantageous when there are multiple objects of interest that must be tracked separately. The output of this network includes the bounding box with the associated object label class as well as the bounding box cartesian coordinates of the object in the video frame (Figure 2).
The previous two frameworks are advantageous for generalized behavioral data collected from standard laboratory experiments in animal subjects. The last CNN framework is OpenPose14,15,16 which is used for human joint pose estimation. OpenPose detects human body, hand, facial, and foot key points on images. The outputs of the framework are labeled images of the human subject as well as the coordinates of all the 25 key points in the body and 21 key points of each hand (Figure 3).
This detailed step-by-step guide for implementation of our recently developed open-source DeepBehavior toolbox employs state-of-the-art convolutional neural nets to track animal behavior (e.g. movement of a paw) or human behavior (e.g. reaching tasks). By tracking the behavior, useful kinematics can be derived from the behavior such as position, velocity, and acceleration. The protocol explains the installation of each CNN architecture, demonstrates how to create training datasets, how to train the networks, how to process new videos on the trained network, how to extract the data from the network on the new videos, and how to post-process the output data to make it useful for further analysis.

<table><tbody><tr><td>CUDA v8.0.61</td><td>NVIDIA</td><td>n/a</td><td>GPU Software</td></tr><tr><td>MATLAB R2016b</td><td>Mathworks</td><td>n/a</td><td>Matlab</td></tr><tr><td>Python 2.7</td><td>Python</td><td>n/a</td><td>Python Version</td></tr><tr><td>Quadro P6000</td><td>NVIDIA</td><td>n/a</td><td>GPU Processor</td></tr><tr><td>Ubuntu v16.04</td><td>Ubuntu</td><td>n/a</td><td>Operating System</td></tr></tbody></table>

a step-by-step implementation of deepbehavior, deep learning toolbox for automated behavior analysis

Understanding behavior is the first step to truly understanding neural mechanisms in the brain that drive it. Traditional behavioral analysis methods often do not capture the richness inherent to the natural behavior. Here, we provide detailed step-by-step instructions with visualizations of our recent methodology, DeepBehavior. The DeepBehavior toolbox uses deep learning frameworks built with convolutional neural networks to rapidly process and analyze behavioral videos. This protocol demonstrates three different frameworks for single object detection, multiple object detection, and three-dimensional (3D) human joint pose tracking. These frameworks return cartesian coordinates of the object of interest for each frame of the behavior video. Data collected from the DeepBehavior toolbox contain much more detail than traditional behavior analysis methods and provides detailed insights to the behavior dynamics. DeepBehavior quantifies behavior tasks in a robust, automated, and precise way. Following the identification of behavior, post-processing code is provided to extract information and visualizations from the behavioral videos.

When the protocol is followed, the data for each network architecture should be similar to the following. For TensorBox, it outputs a bounding box around the object of interest. In our example, we used videos from a food pellet reaching task, and labeled the right paws to track their movement. As seen in Figure 1, the right paw can be detected in different positions in both the front view and side view cameras. After post-processing with camera calibration, 3D trajectories of the reach can be obtained (Figure 1B).
In Yolov3, as there are multiple objects, the output is also multiple bounding boxes. As seen in Figure 2B, there are multiple bounding boxes around the objects of interest. These can be parts of the body.
In OpenPose, the network detects the joint positions as seen in Figure 3A. After post-processing with camera calibration, a 3D model of the subject can be created (Figure 3B).
In conclusion, these representative results showcase the rich details of behavior that can be captured using the DeepBehavior toolbox.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/60763/60763fig1.jpg" /> 
Figure 1: Bounding boxes with TensorBox seen on the paws of video frames during a reaching task in mice. (Adapted from Arac et al 2019). <a href="https://www.jove.com/files/ftp_upload/60763/60763fig1large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/60763/60763fig2.jpg" /> 
Figure 2: Bounding boxes with Yolov3 seen on the regions of interest in video frames during a two mice social interaction test (A raw image, B analyzed image). (Adapted from Arac et al 2019). <a href="https://www.jove.com/files/ftp_upload/60763/60763fig2large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/60763/60763fig3.jpg" /> 
Figure 3: Human pose detection with OpenPose in two camera views (A) and 3D model created from these two images (B). (Adapted from Arac et al 2019). <a href="https://www.jove.com/files/ftp_upload/60763/60763fig3large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/60763/60763fig4.jpg" /> 
Figure 4: TensorBox&#39;s make_json GUI used to label training data. <a href="https://www.jove.com/files/ftp_upload/60763/60763fig4large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/60763/60763fig5.jpg" /> 
Figure 5: GUI of Yolo_Mark to label images in a format acceptable for Yolov3. <a href="https://www.jove.com/files/ftp_upload/60763/60763fig5large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis at JoVE.com

A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

Understanding behavior is the first step to truly understanding neural mechanisms in the brain that drive it. Traditional behavioral analysis methods ...

a-step-step-implementation-deepbehavior-deep-learning-toolbox-for

Nanyang Technological University

Research

JoVE Journal

Behavior

9.3K Views.  David Geffen School of Medicine, University of California, Los Angeles. The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

Video: A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

Intra-Operative Behavioral Tasks in Awake Humans Undergoing Deep Brain Stimulation Surgery

Deep brain stimulation (DBS) is a surgical procedure that directs chronic, high frequency electrical stimulation to specific targets in the brain through implanted electrodes. Deep brain stimulation was first implemented as a therapeutic modality by Benabid et al. in the late 1980s, when he used this technique to stimulate the ventral intermediate nucleus of the thalamus for the treatment of tremor 1. Currently, the procedure is used to treat patients who fail to respond adequately to medical management for diseases such as Parkinson's, dystonia, and essential tremor. The efficacy of this procedure for the treatment of Parkinson's disease has been demonstrated in well-powered, randomized controlled trials 2. Presently, the U.S. Food and Drug Administration has approved DBS as a treatment for patients with medically refractory essential tremor, Parkinson's disease, and dystonia. Additionally, DBS is currently being evaluated for the treatment of other psychiatric and neurological disorders, such as obsessive compulsive disorder, major depressive disorder, and epilepsy. 
DBS has not only been shown to help people by improving their quality of life, it also provides researchers with the unique opportunity to study and understand the human brain. Microelectrode recordings are routinely performed during DBS surgery in order to enhance the precision of anatomical targeting. Firing patterns of individual neurons can therefore be recorded while the subject performs a behavioral task. Early studies using these data focused on descriptive aspects, including firing and burst rates, and frequency modulation 3. More recent studies have focused on cognitive aspects of behavior in relation to neuronal activity 4,5. This article will provide a description of the intra-operative methods used to perform behavioral tasks and record neuronal data with awake patients during DBS cases. Our exposition of the process of acquiring electrophysiological data will illuminate the current scope and limitations of intra-operative human experiments.

Deep brain stimulation surgery offers a unique opportunity to examine information encoding in the awake human brain. This article will describe intra-operative methods used to perform cognitive and behavioral tasks while simultaneously acquiring physiological data such as EMG, single-unit neuronal activity and/or local field potentials.

Deep brain stimulation (DBS) is a surgical procedure that directs chronic, high frequency electrical stimulation to specific targets in the brain through ...

Medicine

Automated High-throughput Behavioral Analyses in Zebrafish Larvae

We have created a novel high-throughput imaging system for the analysis of behavior in 7-day-old zebrafish larvae in multi-lane plates. This system measures spontaneous behaviors and the response to an aversive stimulus, which is shown to the larvae via a PowerPoint presentation. The recorded images are analyzed with an ImageJ macro, which automatically splits the color channels, subtracts the background, and applies a threshold to identify individual larvae placement in the lanes. We can then import the coordinates into an Excel sheet to quantify swim speed, preference for edge or side of the lane, resting behavior, thigmotaxis, distance between larvae, and avoidance behavior. Subtle changes in behavior are easily detected using our system, making it useful for behavioral analyses after exposure to environmental toxicants or pharmaceuticals.

Our laboratory developed a novel high-throughput automated imaging system that is useful for the detection of several different behaviors in 7-day-old zebrafish larvae. The system can be used for detecting subtle changes in behavior after the larvae have been exposed to environmental toxicants or pharmaceuticals.

We have created a novel high-throughput imaging system for the analysis of behavior in 7-day-old zebrafish larvae in multi-lane plates. This system ...

Neuroscience

Tracking Rats in Operant Conditioning Chambers Using a Versatile Homemade Video Camera and DeepLabCut

Operant conditioning chambers are used to perform a wide range of behavioral tests in the field of neuroscience. The recorded data is typically based on the triggering of lever and nose-poke sensors present inside the chambers. While this provides a detailed view of when and how animals perform certain responses, it cannot be used to evaluate behaviors that do not trigger any sensors. As such, assessing how animals position themselves and move inside the chamber is rarely possible. To obtain this information, researchers generally have to record and analyze videos. Manufacturers of operant conditioning chambers can typically supply their customers with high-quality camera setups. However, these can be very costly and do not necessarily fit chambers from other manufacturers or other behavioral test setups. The current protocol describes how to build an inexpensive and versatile video camera using hobby electronics components. It further describes how to use the image analysis software package DeepLabCut to track the status of a strong light signal, as well as the position of a rat, in videos gathered from an operant conditioning chamber. The former is a great aid when selecting short segments of interest in videos that cover entire test sessions, and the latter enables analysis of parameters that cannot be obtained from the data logs produced by the operant chambers.

This protocol describes how to build a small and versatile video camera, and how to use videos obtained from it to train a neural network to track the position of an animal inside operant conditioning chambers. This is a valuable complement to standard analyses of data logs obtained from operant conditioning tests.

Operant conditioning chambers are used to perform a wide range of behavioral tests in the field of neuroscience. The recorded data is typically based on ...

An Open-Source, Fully Customizable 5-Choice Serial Reaction Time Task Toolbox for Automated Behavioral Training of Rodents

The 5-choice serial reaction time task (5-CSRTT) is a behavioral test often used to study visuospatial attention and impulsiveness in rodents. The task requires animals to allocate attention to a horizontal array of five small apertures equipped with light sources and, within a limited time window, nose-poke one illuminated target aperture to get a food reward at the food magazine located in the opposite wall of the chamber. The task considers behavioral control measures such as response accuracy and reaction times and allows to infer selective attention and impulsivity. Task difficulty can be controlled by modifying the stimulus duration and task design in general. Commercially available apparatus usually consists of an experimental chamber and particular software to specify task parameters, but due to fixed hard- and software, they pose many limitations on changes in the general experimental design and specific task requirements and the related data output. This article explains a fully customizable alternative based on an easy-to-use single-board microcontroller and standard electrotechnical components, an open-access Arduino script, and a Matlab-toolbox for hardware control and behavioral task specifications, respectively. The toolbox includes an optional staircase procedure, enabling automated behavioral training. The complete hardware setup, which can be installed in customized chambers, and the freely adaptable software encourage non-standardized task and chamber design. The design of the system and the open-source code for hardware control and experimental setup are described.

The present protocol describes the development of an open-source 5-choice serial reaction time task toolbox for rodent animal models, using Arduino and related hardware and a versatile Matlab toolbox, including an optional script for automated behavioral training. The scripts are customizable and facilitate the implementation of different trial-and-test designs.

The 5-choice serial reaction time task (5-CSRTT) is a behavioral test often used to study visuospatial attention and impulsiveness in rodents. The task ...

Pipeline for Analyzing Behavioral Responses to External Cues in MRI Environments

Profiling Maternal Behavior Responses During Whole-Brain Imaging

Recent advancements in whole-brain imaging tools have enabled neuroscientists to investigate how coordinated brain activity processes external cues, influencing internal state changes and eliciting behavioral responses. For example, functional magnetic resonance imaging (fMRI) is a noninvasive technique that allows for the measurement of whole-brain activity in awake, behaving mice using the blood oxygenation-level-dependent (BOLD) response. However, to fully understand BOLD responses evoked by external stimuli, it is crucial that experimenters also assess behavioral responses during scans. The MRI environment poses challenges to this goal, rendering commonly employed methods of behavioral monitoring incompatible. These challenges include (1) a restricted field of view and (2) the limited availability of equipment without ferromagnetic components. Presented here is a behavioral video analysis pipeline that overcomes these limitations by extracting valuable information from videos acquired within these environmental constraints, enabling the evaluation of behavior during the acquisition of whole-brain neural data. Employing methods such as optical flow estimation and dimensionality reduction, robust differences can be detected in behavioral responses to stimuli presented during fMRI scans. For example, representative results suggest that mouse pup vocalizations, but not pure tones, evoke significantly different behavioral responses in maternal versus virgin female mice. Moving forward, this behavioral analysis pipeline, initially tailored to overcome challenges in fMRI experiments, can be extended to various neural recording methods, providing versatile behavioral monitoring in constrained environments. The coordinated evaluation of behavioral and neural responses will offer a more comprehensive understanding of how the perception of stimuli leads to the coordination of complex behavioral outputs.

Presented here is a video analysis pipeline that overcomes challenges in behavioral monitoring within MRI environments, allowing for the detection of uninstructed behavioral responses to external cues. This analysis will facilitate a more comprehensive understanding of evoked internal state changes and brain-wide activity.

Recent advancements in whole-brain imaging tools have enabled neuroscientists to investigate how coordinated brain activity processes external cues, ...

Automated Behavioral Analysis of Large C. elegans Populations Using a Wide Field-of-view Tracking Platform

Caenorhabditis elegans is a well-established animal model in biomedical research, widely employed in functional genomics and ageing studies. To assess the health and fitness of the animals under study, one typically relies on motility readouts, such as the measurement of the number of body bends or the speed of movement. These measurements usually involve manual counting, making it challenging to obtain good statistical significance, as time and labor constraints often limit the number of animals in each experiment to 25 or less. Since high statistical power is necessary to obtain reproducible results and limit false positive and negative results when weak phenotypic effects are investigated, efforts have recently been made to develop automated protocols focused on increasing the sensitivity of motility detection and multi-parametric behavioral profiling. In order to extend the limit of detection to the level needed to capture the small phenotypic changes that are often crucial in genetic studies and drug discovery, we describe here a technological development that enables the study of up to 5,000 individual animals simultaneously, increasing the statistical power of the measurements by about 1,000-fold compared to manual assays and about 100-fold compared to other available automated methods.

Automated Behavioral Analysis of Large C. elegans Populations Using a Wide Field-of-view Tracking Platform

We describe protocols for using the wide field-of-view nematode tracking platform (WF-NTP), which enables high-throughput phenotypic characterization of large populations of Caenorhabditis elegans. These protocols can be used to characterize subtle behavioral changes in mutant strains or in response to pharmacological treatment in a highly scalable fashion.

Caenorhabditis elegans is a well-established animal model in biomedical research, widely employed in functional genomics and ageing studies. To assess the ...

Immunology and Infection

A Flexible Platform for Monitoring Cerebellum-Dependent Sensory Associative Learning

Climbing fiber inputs to Purkinje cells provide instructive signals critical for cerebellum-dependent associative learning. Studying these signals in head-fixed mice facilitates the use of imaging, electrophysiological, and optogenetic methods. Here, a low-cost behavioral platform (~$1000) was developed that allows tracking of associative learning in head-fixed mice that locomote freely on a running wheel. The platform incorporates two common associative learning paradigms: eyeblink conditioning and delayed tactile startle conditioning. Behavior is tracked using a camera and the wheel movement by a detector. We describe the components and setup and provide a detailed protocol for training and data analysis. This platform allows the incorporation of optogenetic stimulation and fluorescence imaging. The design allows a single host computer to control multiple platforms for training multiple animals simultaneously.

We have developed a single platform to track animal behavior during two climbing fiber-dependent associative learning tasks. The low-cost design allows integration with optogenetic or imaging experiments directed towards climbing fiber-associated cerebellar activity.

Climbing fiber inputs to Purkinje cells provide instructive signals critical for cerebellum-dependent associative learning. Studying these signals in ...

A Pipeline using Bilateral In Utero Electroporation to Interrogate Genetic Influences on Rodent Behavior

As genome-wide association studies shed light on the heterogeneous genetic underpinnings of many neurological diseases, the need to study the contribution of specific genes to brain development and function increases. Relying on mouse models to study the role of specific genetic manipulations is not always feasible since transgenic mouse lines are quite costly and many novel disease-associated genes do not yet have commercially available genetic lines. Additionally, it can take years of development and validation to create a mouse line. In utero electroporation offers a relatively quick and easy method to manipulate gene expression in a cell-type specific manner in vivo that only requires developing a DNA plasmid to achieve a particular genetic manipulation. Bilateral in utero electroporation can be used to target large populations of frontal cortex pyramidal neurons. Combining this gene transfer method with behavioral approaches allows one to study the effects of genetic manipulations on the function of prefrontal cortex networks and the social behavior of juvenile and adult mice.

The role of recently discovered disease-associated genes in the pathogenesis of neuropsychiatric disorders remains obscure. A modified bilateral in utero electroporation technique allows for the gene transfer in large populations of neurons and examination of the causative effects of gene expression changes on social behavior.

As genome-wide association studies shed light on the heterogeneous genetic underpinnings of many neurological diseases, the need to study the contribution ...

Biology

Analyzing Signaling Modules in Alzheimer's Disease with Deep Learning

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data

Large omics datasets are becoming increasingly available for research into human health. This paper presents DeepOmicsAE, a workflow optimized for the analysis of multi-omics datasets, including&#160;proteomics, metabolomics, and clinical data. This workflow employs a type of neural network called autoencoder, to extract a concise set of features from the high-dimensional multi-omics input data. Furthermore, the workflow provides a method to optimize the key parameters needed to implement the autoencoder. To showcase this workflow, clinical data were analyzed from a cohort of 142 individuals who were either healthy or diagnosed with Alzheimer&#39;s disease, along with the proteome and metabolome of their postmortem brain samples. The features extracted from the latent layer of the autoencoder retain the biological information that separates healthy and diseased patients. In addition, the individual extracted features represent distinct molecular signaling modules, each of which interacts uniquely with the individuals&#39; clinical features,&#160;providing for a mean to integrate&#160;the proteomics, metabolomics, and clinical data.

Author Spotlight: Advancing Alzheimer's Research &#8211; Exploring Early Detection and Multi-Omics Approaches

DeepOmicsAE is a workflow centered on the application of a deep learning method (i.e., an autoencoder) to reduce the dimensionality of multi-omics data, providing a foundation for predictive models and signaling modules representing multiple layers of omics data.

Large omics datasets are becoming increasingly available for research into human health. This paper presents DeepOmicsAE, a workflow optimized for the ...

Animal Behavior

<ol class="note-items"><li data-note-item="1" data-parent-collapse="">D. Melanogaster Care
<button class="expand">Expand</button><ul class="lab-items"><li data-sub-note-item="1-1"><div class="lab-step">Obtain the Drosophila melanogaster specimens at least two weeks before the experiment to allow the specimens to acclimate to the lab room conditions. NOTE: 25 Drosophila adults produce about 50 new adults in two weeks. Each colony-housing container should consist of a clean vial, 1.5 &#8211; 3 cm of prepared food medium, and a stopper.
</div></li><li data-sub-note-item="1-2"><div class="lab-step">Use a funnel to transfer the flies to the colony-housing container. NOTE: House the colonies so there will be approximately 40 flies per student or student group (plus spares) on the day of the experiment.
</div></li><li data-sub-note-item="1-3"><div class="lab-step">On the day of the experiment, place a dish of apple cider vinegar, a razor blade, a dish of distilled water, two sterile cotton balls, a funnel, scissors, Sharpie pens, a timer, four water bottle caps, and two water bottles onto each lab table.
</div></li><li data-sub-note-item="1-4"><div class="lab-step">After the equipment is in place, use a funnel to randomly move colonies of 40 adult flies each into the clean vials without food or substrate.
</div></li><li data-sub-note-item="1-5"><div class="lab-step">NOTE: Vials without food or substrate make it easier for the students to collect only the adults and to avoid adding food to the experimental chambers.
</div></li></ul></li></ol>

Animal Behavior: Studying Taxis Behavior of Drosophila - Prep

D. Melanogaster Care
ExpandObtain the Drosophila melanogaster specimens at least two weeks before the experiment to allow the specimens to acclimate to ...

A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

Summary

Explore More Videos

Chapters in this video

Intra-Operative Behavioral Tasks in Awake Humans Undergoing Deep Brain Stimulation Surgery

Automated High-throughput Behavioral Analyses in Zebrafish Larvae

Tracking Rats in Operant Conditioning Chambers Using a Versatile Homemade Video Camera and DeepLabCut

An Open-Source, Fully Customizable 5-Choice Serial Reaction Time Task Toolbox for Automated Behavioral Training of Rodents

Profiling Maternal Behavior Responses During Whole-Brain Imaging

Automated Behavioral Analysis of Large C. elegans Populations Using a Wide Field-of-view Tracking Platform

A Flexible Platform for Monitoring Cerebellum-Dependent Sensory Associative Learning

A Pipeline using Bilateral In Utero Electroporation to Interrogate Genetic Influences on Rodent Behavior

DeepOmicsAE: Representing Signaling Modules in Alzheimer's Disease with Deep Learning Analysis of Proteomics, Metabolomics, and Clinical Data

Animal Behavior