Fully Automated Leg Tracking in Freely Moving Insects using Feature Learning Leg Segmentation and Tracking (FLLIT)

Summary

We describe detailed protocols for using FLLIT, a fully automated machine learning method for leg claw movement tracking in freely moving Drosophila melanogaster and other insects. These protocols can be used to quantitatively measure subtle walking gait movements in wild type flies, mutant flies and fly models of neurodegeneration.

Abstract

The Drosophila model has been invaluable for the study of neurological function and for understanding the molecular and cellular mechanisms that underlie neurodegeneration. While fly techniques for the manipulation and study of neuronal subsets have grown increasingly sophisticated, the richness of the resultant behavioral phenotypes has not been captured at a similar detail. To be able to study subtle fly leg movements for comparison amongst mutants requires the ability to automatically measure and quantify high-speed and rapid leg movements. Hence, we developed a machine-learning algorithm for automated leg claw tracking in freely walking flies, Feature Learning-based Limb segmentation and Tracking (FLLIT). Unlike most deep learning methods, FLLIT is fully automated and generates its own training sets without a need for user annotation, using morphological parameters built into the learning algorithm. This article describes an in depth protocol for carrying out gait analysis using FLLIT. It details the procedures for camera setup, arena construction, video recording, leg segmentation and leg claw tracking. It also gives an overview of the data produced by FLLIT, which includes raw tracked body and leg positions in every video frame, 20 gait parameters, 5 plots and a tracked video. To demonstrate the use of FLLIT, we quantify relevant diseased gait parameters in a fly model of Spinocerebellar ataxia 3.

Introduction

In the last few decades, neurodegenerative diseases and movement disorders have grown more prevalent in our aging populations. Although our understanding of many neurodegenerative diseases has advanced at the molecular and cellular level, fundamental features of the affected neuronal circuitry underlying disease remain poorly understood. Recently developed behavioral tracking tools^1,2,3,4 now allow us to study movement abnormalities in animal disease models in order to identify molecular, cellular and circuit dysregulation underlying disease.

Molecular pathways involved in many neurodegenerative diseases are conserved in the fruit fly Drosophila melanogaster, and Drosophila disease models have helped to elucidate fundamental mechanisms underlying neurodegeneration^5,6. We recently showed that fly models of Parkinson’s Disease (PD) and Spinocerebellar ataxia 3 (SCA3) exhibit distinct, conserved gait signatures that resemble those of the respective human diseases¹, demonstrating that the fly model can be used to understand circuit mechanisms underlying movement dysfunction in specific movement disorders. The rich and continually growing arsenal of tools in the fly model for targeted manipulation and visualization of neurons at the single gene and single cell level^7,8,9,10 makes the fly an ideal model one to probe the relationship between disease pathways, neuronal circuitry and behavioral phenotypic manifestation in vivo. To enable precise, automated insect gait analysis, we recently developed a machine learning method, Feature Learning-based LImb segmentation and Tracking (FLLIT)¹.

FLLIT consists of a fully automated multi-stage algorithm that first segments the leg pixels, which are subsequently used to locate and track the corresponding leg claws. FLLIT employs a boosting algorithm for segmentation, in contrast to deep learning algorithms used in recent work^2,3. There are some similarities with convolutional neural networks in that for both frameworks, feature extraction is done automatically through learning convolutional kernels. The first step in FLLIT involves using morphological operations (edge and skeletonization) to automatically generate positive (pixels on the legs) and negative (background or pixels on the fly body) training samples with high confidence. Hence, FLLIT is fully automated and does not require user annotated training samples. Using the above training samples, a classifier is then trained in the framework of a boosting algorithm. An ensemble of weak classifiers is iteratively learnt, with each consisting a set of convolutional kernels for feature extraction and a decision tree. The final learnt classifier is then used for leg segmentation and is able to better discern difficult regions/hard samples better than morphological operations, producing an overall much more accurate segmentation for tracking¹. From the segmented legs, we locate the tips and track them using the Hungarian algorithm: by matching tips across frames such that the sum of the distance moved by each tip is minimized. FLLIT can handle occlusion cases by remembering the last seen location (in fly centered coordinates) so that a leg tip is recovered once it is no longer under occlusion.

We previously showed that FLLIT can automatically and accurately track leg movements and analyze gait in an unmarked, freely moving fly or spider from high-speed video¹; FLLIT should hence be broadly applicable for arthropod leg tracking. By extracting machine learning training sets using morphological parameters, FLLIT automatically trains itself to segment and track insect legs without the need for laborious manual annotation, which is required for most deep learning methods. FLLIT is hence fully automated. After leg segmentation and tracking, FLLIT automatically produces raw tracked body and leg positions in every video frame, 20 gait parameters, 5 plots and a tracked video for gait analysis and visualization of gait movements. This protocol provides a step-by-step guide to using FLLIT.

Protocol

1. System setup

1. Ensure that the recording station has a high-speed camera and a stage over it to hold the arena chamber (Figure 1). Adjust the camera to record at a minimum of 250 frames per second (fps), with an accordingly fast shutter speed (in this case, recording is performed at 1000 fps with a shutter speed of 1 ms).
   NOTE: Check that the video is suitable for tracking by ensuring that there is minimal or no motion blur of moving legs in all frames. If the moving leg is so blurred that a human annotator cannot track it, then the camera recording speed and/or shutter speed must be increased.
2. Place the infrared LED lights at the top of the stage with a diffuser (translucent sheet) between the camera and sample (Figure 1A,B).
3. Make the recording chamber by cutting a 1.6 mm thick acrylic sheet. In this experiment, use a field-of-view of 11 mm x 11 mm. Place the chamber between two glass slides (Figure 1C).

2. Preparation of flies for recording

1. Transfer the flies to be recorded into a new food vial 24 h prior to recording.
   NOTE: Do not use CO₂ (usually used to anesthetize the flies upon first collection) on the flies less than 24 h before recording.
2. About 40 min before recording, transfer the flies into empty vials and keep on ice for 5-7  min. 
3. In the meantime, wipe the arena and clear glass slides with water and a wipe.
   NOTE: Do not use ethanol to clean the chambers and the slides.
4. Prepare the recording chamber. Fix one of the microscopic glass slides below the chamber with tape.
5. When the flies have been anesthetized on ice, transfer one fly in each chamber using a clean brush.
   NOTE: Both male and female flies can be used in this setup, and, as far as possible, flies of both sexes should be analyzed to avoid sex-specific bias.
6. Secure the chamber with another microscopic glass slide with tape (Figure 1C).
7. Keep the chambered flies at room temperature for 15 - 20 min for acclimatization.

3. Generation of videos for FLLIT analysis

NOTE: This step is specific to the video camera used. In this case, a commercially available video camera is used (see Table of Materials).

1. Turn on the power source. Wait for the green LED for power and the orange LED for the ethernet interface connection to stabilize. Switch on the power for the infrared LED. Ensure that the voltage remains at 12.5 V.
2. Open the Viewer application on the connected computer system.
3. Change the recording frame rate to 1000 fps. Set the shutter speed to 1/1000 s (1 ms).
4. Place the chamber with the fly on the recording arena and select the LIVE button. Ensure that the camera is focused on the leg tips when the fly is walking upright on the floor of the chamber; the leg tips should be in sharp focus.
5. Click Record (Figure 2).
6. Record the fly walking, ensuring that: 
   The fly walked in a relatively straight trajectory without touching the edge of the arena.
   The fly walked at least three strides per leg.
   The fly does not pause during the walk.
   The distance walked is equivalent to at least one body length. 
   
   NOTE: Having the background subtracted cleanly is critical for accurate segmentation. The automated background subtraction algorithm employed by FLLIT requires that the imaged fly moves at least one body length in distance.
7. Click Rec Done to stop recording (Figure 2).
8. Crop the video to ensure that the recording encompasses only a straight walk of the fly (as described in Step 3.6).
9. Click Save (Figure 2). Save the files in ‘.mraw’ or ‘.tiff’ format in the respective folders.
   NOTE: ‘.mraw’ format gives greater flexibility to change the file name (if required) and for storage of videos compared to ‘.tiff’ file format.

4. Installation of FLLIT program

NOTE: Up-to-date instructions can be found at: https://github.com/BII-wushuang/FLLIT/blob/master/Compiled/Readme.pdf

1. Download FLLIT on any operating system
   1. Download FLLIT from the following Github link: https://github.com/BII-wushuang/FLLIT/archive/master.zip. Extract the contents of the zip file.
   2. Download sample datasets from the following Google Drive link: https://bit.ly/2EibvNY. Create a folder Data under FLLIT-master/Compiled and place dataset folders in this Data directory.
2. Install FLLIT In Ubuntu
   1. Navigate to the FLLIT/Compiled directory.
   2. Right-click and select Open in Terminal.
   3. Issue the following command to download and install the MATLAB runtime libraries to $HOME/MCR:
      bash MCR_2016a.sh
   4. After installation of the MATLAB runtime libraries has been completed, issue the following command to ensure that executable rights are accorded to FLLIT:
      chmod +x FLLIT
   5. Open a terminal in the FLLIT/Compiled directory and issue the following command to execute FLLIT:
      bash run_FLLIT.sh $HOME/MCR/v901
3. Install FLLIT in Windows
   1. For 7 and 10th Home Edition, install Docker Toolbox at:
      (https://github.com/docker/toolbox/releases/download/v19.03.1/DockerToolbox-19.03.1.exe).
   2. For Windows 10 Pro or Enterprise Edition, install Docker Desktop for Windows at: (https://download.docker.com/win/stable/Docker\%20Desktop\%20Installer.exe).
   3. To enable execution of GUI applications in a Docker container on Windows, first install VcXSrV (https://sourceforge.net/projects/vcxsrv). On starting VcXsrv, configure the settings as in Figure S1.
      NOTE: Make sure Docker and VcXsrv are running before starting FLLIT.
   4. Double click FLLIT.bat to run FLLIT.
      NOTE: When executing for the first time, it will take some time to pull the Docker image from Docker Hub.
4. Install FLLIT in MacOS
   1. Download Docker Desktop for MacOS at https://download.docker.com/mac/stable/Docker.dmg
   2. Install socat by opening a terminal and issuing the following command:
      brew install socat
   3. Start socat with:
      socat TCP-LISTEN:6000,reuseaddr,fork UNIX-CLIENT:\"$DISPLAY\" & disown
   4. Install XQuartz (https://www.xquartz.org) to enable execution of GUI applications in a Docker container on MacOS. Start XQuartz and change the preferences by checking the Allow connections from network clients in the Security tab as shown in Figure S2.
      NOTE: Make sure that Docker, socat and XQuartz are all running before starting FLLIT. 
   5. Open a terminal in the FLLIT/Compiled directory and execute FLLIT with the following command:
      bash FLLIT_Mac.sh
      NOTE: When executing for the first time, it will take some time to pull the Docker image from Docker Hub.

5. Running FLLIT for automated leg tracking

1. Segmentation
   1. Convert the video into individual TIFF files and copy into the FLLIT data folder.
   2. Run FLLIT (In Ubuntu, right click to open FLLIT in Terminal).
   3. Select the folder containing the frame-by-frame TIFF images of the video to be tracked and click the Add button.
   4. In the pop-up window choose 0 for carrying out leg segmentation only, or 1 to include leg tracking with leg segmentation.
   5. Click Done to initiate segmentation and tracking of the selected video. 
2. Tracking
   1. To check the accuracy of tracking and carry out error corrections (if any), click on Select Data Folder. Select the folder to be tracked and click Open.
   2. Click on View Tracking.
      NOTE: Check that Viewer Mode remains selected while viewing the tracked leg positions. Otherwise, any previous corrections made will be over-ridden.
   3. Check labeling for all the legs in the first frame
      NOTE: Since the leg labels are placed according to the position on the screen, if the fly is walking upright, the fly’s RIGHT side is labelled as L1 (fore-leg), L2 (mid-leg), L3 (hind-leg) and the fly’s LEFT side is labelled as R1 (fore-leg), R2 (mid-leg), R3 (hind-leg), respectively (Figure 3). If the fly is walking upside, down the leg labels will be correctly annotated.
   4. If a leg is wrongly labeled and a correction is required, click Pause viewing, followed by Adjust Prediction (Figure 3).
   5. From the right panel headed Leg to Adjust, select the leg that requires correction. 
   6. Double click on the correct position for this leg in the image window, click on Save, and then Exit. To go to the previous frame or subsequent frame, click Pause viewing followed by the forward and backward <I and I> buttons, respectively (Figure 3).
   7. To correct any incorrectly tracked videos, open the Data folder of the video to be retracked and select Manually Initiate Tracking.
   8. Click the Tracking button, which will then change its label to Initial.
   9. Click Adjust Prediction and correct the leg labels by double-clicking each leg tip and then assigning it with the correct leg label. Click Save and Exit.
   10. Click on Resume to initiate tracking.
3. Data processing and video generation
   1. Click on Data Process. In the popup window, type in the number of frames per second (fps) at which the videos were recorded (e.g., 1,000 fps). 
   2. Use the following equation to calculate the actual field of view of the captured video so that gait parameters can be measured in millimeters:
      
      
      
      NOTE: For example, if the actual chamber size is 7 mm, the width of the image frame is 137 mm, the width of the chamber in the image frame on the computer screen is 81 mm, and the width of the field of view was 11.83 mm (Figure S3).
   3. To see the tracking results, go to the Tracking folder under the Results folder.
   4. To generate a video of the tracked fly, select Make video. The video will be saved in the same Results folder as that of the original video data analyzed.
      NOTE: The Start (first) and End (last) frames of the video can be selected.
4. Normalization to body length of each fly.
   NOTE: As each fly may be slightly different in size, certain gait parameters should be normalized to the body length of each fly to facilitate comparisons (e.g., stride length may be longer in larger flies, and shorter in smaller flies).
   1. Open three still images from the video of each fly (usually first, middle and last frames) using image software.
   2. Magnify each image frame to 800% and label the anterior-most pixel of the head and posterior-most pixel of the abdomen at the midline using a bright color (e.g., yellow).
   3. Open the labeled images in ImageJ.
   4. Use Set Scale to input the scale accordingly: Set Distance in pixels: 512; Known distance: Actual Field of view (mm) as measured in step 5.3.2.; Unit of length: mm.
   5. Draw a straight line between the labelled head and abdomen tip pixels to obtain the body length.
   6. Open Set Scale again to obtain the value in a known distance, which is the body length in mm.
   7. Take an average of the length determined in each of the three images to obtain the average body size in mm.

Discussion

In this manuscript, we describe in detail the steps involved in using FLLIT, an automated machine-learning program¹, to analyze gait in freely walking Drosophila. After tracking and data analysis, FLLIT automatically generates raw data for the positional information of the body and leg claws, producing twenty body and gait features as well as a video of the tracked fly to enable gait visualization.

There are now a number of methods for leg movement tracking of Drosophila and other animals^{1,2,3,4,14,15,16}, giving researchers a wide range of options depending on the goals of the experiment. Some of these are foot printing-based approaches, which are highly accurate but which report only claw contact points with the detection surface^4,14. On the other hand, recent deep learning approaches^2,3,16 are highly versatile, allowing analysis of behaviors that require tracking of leg joints and other body parts in any animal, with the caveat that the algorithms need to first be trained with user annotated datasets. A third type of approach uses morphology or image-contrast-based methods^1,15,17 to find the outline of each leg to identify claw positions. In general, these methods deal poorly with behaviors where the legs cross over (e.g. during grooming). FLLIT combines the second and third approaches, using morphological parameters to train a boosting algorithm for leg segmentation. This allows FLLIT to bypass the laborious task of user annotation to generate the training dataset, while enhancing accuracy using machine learning. Future improvements to FLLIT will have to deal with instances where legs cross over, to allow for analysis of more complex behaviors.

FLLIT is robust to slight changes in illumination, recording resolution and frame speed¹. However, frame speed of recorded videos should not fall below 250 fps, and FLLIT runs optimally for videos recorded at 1000 fps. If there is motion blur in the images, such that it is challenging for a human annotator to identify the leg position, FLLIT will not be able to accurately identify leg tips in those frames. In the light of this, it is essential that the camera be focused sharply on the leg tips. To prevent segmentation artifacts, the arena should be cleaned thoroughly, and should not be moved during the recording. For accurate background subtraction and clean segmentation, the fly should move at least one body length during the recording, without pausing. After automatic segmentation and tracking the labeling of the all legs must be checked. If the fly gait is not tracked or tracked wrongly, the file should be retracked manually using the Manually Initiate Tracking option (step 5.2.7 – 5.2.10).

Neurodegenerative diseases and movement disorders are increasingly prevalent in our aging societies. Fly models of neurodegeneration have been studied for more than 2 decades, during which advances have been made regarding the molecular and cellular aspects of disease pathophysiology. However, specific behavioral consequences of disease have been technically difficult to assess. For example, while reports of trembling movements in the fly have been made^18,19, these had not been quantitatively studied until recently¹. The climbing assay has been a useful and quantitative, yet relatively coarse measure⁶. This technical deficit has similarly hampered high-resolution movement analysis in other animal models. The advent of new tools for behavioral analysis, hence, has promise to rejuvenate the field of movement disorders to enable researchers to study how molecular and cellular mechanisms of neuromuscular diseases lead to specific behavioral outcomes in animal models. In this paper and in our previous work¹, we showed using FLLIT that fly models of SCA3 exhibit a hyperkinetic ataxic gait, while PD fly models exhibit a hypokinetic rigid gait, recapitulating movement hallmarks of the respective human diseases¹. Gait analysis also enabled us to identify distinct neuronal populations underlying specific movement dysfunctions. Going forward, detailed movement analysis, combined with the powerful imaging and functional tools available in the fly, will allow us to gain novel insight into mechanisms of locomotor dysfunction, illuminating our understanding of neurodegenerative diseases with respect to circuit mechanisms.

FLLIT should be widely applicable to study gait in other small arthropods, as it was previously demonstrated to be highly accurate for tracking spider leg movements¹. While we focus here on the use of detailed movement phenotyping for quantifying pathogenic gait and its underlying circuitry, advances in movement tracking have already revolutionized, and will have continuing impact on, the understanding of normal walking coordination and gait and its underlying circuits, especially in myriad different branches of the evolutionary tree. 

Materials

Name	Company	Catalog Number	Comments
Acrylic Sheets	Dama	1.6 mm thickness, Clear sheets	Singapore, Singapore
Clear Glass slides	Biomedia	BMH 880101	Singapore, Singapore
High speed camera	Photron	Fastcam MC2.1	Tokyo, Japan. A shutter speed of 1 msec or faster is ideal to reduce motion blur of captured images
Infra Red LED	Any - Generic from hardware store	940nm Infrared Light Board	Singapore, Singapore
Kimwipe	Kimberly Clark	34155-01LS	Irving, Texas, USA