Name: from voxels to knowledge: a practical guide to the segmentation of complex electron microscopy 3d-data
Uploaded: 2014-08-13T16:00:00
Duration: 12 min 8 s
Description: Watch this Scientific Journal Video about From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data at JoVE.com

We would like to acknowledge and thank Tom Goddard at University of California San Francisco for his endless help with Chimera, Joel Mancuso and Chris Booth at Gatan, Inc. for their help with SBF-SEM data collection of bacteria dataset, Doug Wei at Zeiss, Inc. for his help with the FIB-SEM data collection of epithelial cell dataset, Kent McDonald at University of California Berkeley Electron Microscopy Lab for advice on sample preparation, TEM imaging and tomography, Roseann Csencsits at Lawrence Berkeley National Laboratory for her help taking the cryo-TEM image, Elena Bosneaga for cryo-sectioning of the plant dataset, Jocelyn Krey at Oregon Health and Science University for the dissection of utricle tissue, David Skinner at National Energy Research Scientific Computing Center (NERSC) and Jitendra Malik at University of California Berkeley for their advice in software infrastructure, and Pablo Arbelaez at&#160;University of California Berkeley&#160;for his codes contributions to the custom-tailored script presented in this article. 
Research was supported by the U.S. Department of Energy, Office of Science under contract No. DE-AC02-05CH11231 [David Skinner], as well as U.S. National Institutes of Health (NIH) grant No. P01 GM051487 [M.A.] for the inner ear hair cell project and microscopy instrumentation use.

1. Manual Abstracted Model Generation
Note: The details of the methodology described below are specific to Chimera, but other software packages may be used instead. Use this approach when the sole objective is to create a geometrical model (e.g., a ball and stick model) in order to make geometric measurements, rather than displaying the volume shape of the objects.
<ol>
	<li>Import the data volume into a suitable program for manual abstracted model generation.
	<ol>
		<li>Select File &#62; Open Map to pull up the Open File dialog. Navigate to the file location of the desired map.</li>
		<li>Pull up the Volume Viewer (Tools &#62; Volume Data &#62; Volume Viewer) and select Features &#62; Display Style to display data with different rendering styles.</li>
		<li>Adjust the threshold for the display by dragging the vertical bar on the histogram in the Volume Viewer window.</li>
	</ol>
	</li>
	<li>Navigate through the 3D volume (e.g., slice by slice) to select an area of interest for segmentation and crop out a smaller sub-volume if necessary.
	<ol>
		<li>In the Volume Viewer dialog, click Axis, then select X, Y, or Z.</li>
		<li>In the Volume Viewer dialog, select Features &#62; Planes. Click One to set Depth to display the plane corresponding to the number in the left box, and click All to display all planes.</li>
		<li>In the Volume Viewer dialog, select Features &#62; Subregion selection.
		<ol>
			<li>Click and drag to create a rectangular box around the region of interest.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Place markers along the feature of interest and connect them with linkers where appropriate (often done automatically by the program) until the model is complete.
	<ol>
		<li>From the Volume Viewer menu bar, select Tools &#62; Volume Tracer dialog to open the Volume Tracer dialog. In the Volume Tracer dialog, select File &#62; New Marker Set.</li>
		<li>In the Volume Tracer dialog, check Mouse &#62; Place markers on high quality, Place markers on data planes, Move and resize markers, Link new marker to selected marker, and Link consecutively selected markers.</li>
		<li>Click on the Marker Color swatch, and select a color. Repeat this step for Link color.</li>
		<li>Enter radii for the marker and link model-building elements.</li>
		<li>In the Volume Tracer Window, select Place markers using [right] mouse button, and insert radii for markers and links.</li>
		<li>Right click on the volume data to begin laying down markers. Markers will be connected automatically.</li>
		<li>In the Volume Tracer dialog, select File &#62; Save current marker set, then File &#62; Close marker set.</li>
	</ol>
	</li>
	<li>Open a new marker set (Step 1.3.1) to begin building a model into a second desired feature of interest. Utilize contrasting colors between marker sets to emphasize differences in features.</li>
</ol>
2. Manual Tracing of Features of Interest
Note: The details of the methodology described below are specific to Amira, but other software packages may be used instead. Use this approach when the population density is relatively small and when accuracy of feature extraction is paramount, as manual tracing is a time-consuming approach.
<ol>
	<li>Import volume data into a program with manual tracing options. Software with this capability generally offer at least a basic paintbrush tool.
	<ol>
		<li>For large volumes or tomograms (e.g., 16-bit 2,048 x 2,048 or larger .rec or .mrc tomograms generated in IMOD): Select Open Data &#62; Right click on filename.rec &#62; Format&#8230; &#62;Select Raw as LargeDiskData &#62; Ok &#62; Load. Select appropriate Raw Data Parameters from header information &#62; Ok. Toggle and Save As a new <a href="http://filename.am/">filename.am</a> file for use in the following steps.</li>
		<li>For smaller 3D image stack files (e.g., 3D .tif or .mrc or .rec): Open Data &#62; Select filename.tif or filename.mrc. Toggle and Right Click &#62; Save As <a href="http://filename.am/">filename.am</a>. If an error is generated or the program is unresponsive, the file may be too large and can be opened by following Step 2.1.1.</li>
	</ol>
	</li>
	<li>Navigate through the slices to select a 3D sub-volume for segmentation, and then crop to this area of interest.
	<ol>
		<li>In 3D Viewer window, select Orthoslice to open image file. Use slider at bottom to navigate through slices.</li>
		<li>To crop larger data opened as LargeDiskData, toggle file name in Pool window &#62; Right click &#62; LatticeAccess. Enter desired box size &#62; Apply. Save new file.</li>
	</ol>
	</li>
	<li>Create segmentation file.
	<ol>
		<li>Toggle the file in the Pool window &#62; Right Click &#62; Labeling &#62; LabelField. A new file will be created and automatically loaded in the Segmentation Editor tab.</li>
	</ol>
	</li>
	<li>Trace the border of the first feature of interest, then fill the trace by hand or by using a command specific to the software used. Follow the feature of interest through all slices and repeat the manual tracing segmentation. Use the following commands when using Amira:
	<ol>
		<li>To use the Paintbrush tool, alter brush size as desired, then use the mouse pointer to trace the border of the feature of interest.</li>
		<li>Fill the traced area with shortcut &#8220;f&#8221;. Add the selection by clicking the button with the plus symbol, or the shortcut &#8220;a&#8221;. If necessary, press &#8220;u&#8221; to undo, and &#8220;s&#8221; to subtract or erase.</li>
	</ol>
	</li>
	<li>Generate a surface rendering for visualization and basic qualitative or quantitative analysis per software user guide instruction.
	<ol>
		<li>In the Object Pool tab, toggle the filename-labels.am in the Pool window &#62; Right click &#62; SurfaceGen.</li>
		<li>Select desired Surface properties &#62; Apply. A new file filename.surf will be created in the Pool.</li>
		<li>To visualize the segmented volume, toggle filename.surf in Pool window &#62; Right click &#62; SurfaceView.</li>
		<li>Use the tools in the 3DViewer window to move, rotate, and zoom in the 3D volume.</li>
	</ol>
	</li>
	<li>Extract the exact densities and determine measurements such as volume or surface area. Export to other programs for more advanced display, analysis and simulation.
	<ol>
		<li>On 3DViewer window, click Measure tool &#62; Select appropriate option (2D length and 2D angle for measurements on a single 2D plane, 3D length and 3D Angle for measurements on a 3D volume).</li>
		<li>Click on mesh surface to measure desired length, distance, and angles. The values will be listed in the Properties window.</li>
	</ol>
	</li>
</ol>
3. Automated Density-based Segmentation
Note: The details of the methodology described below are specific to Amira, but other software packages may be used instead.
<ol>
	<li>Use this approach on data sets with any variety of contrast, crispness, or crowdedness to withdraw the densities of interest.</li>
	<li>Import volume data into a program equipped with thresholding, magic wand, or other density-based tools for automatic segmentation. Follow steps outlined in 2.1-2.1.2 in the directions for manual tracing.</li>
	<li>Navigate through slices and select area for segmentation. If necessary, crop out a smaller 3D sub-volume for segmentation. Follow steps outlined in 2.2-2.2.2 in the directions for manual tracing.</li>
	<li>Select the density of a feature of interest, usually by clicking or placing a mark or anchor point on the feature. If allowed in the software, enter a number range encompassing the feature&#8217;s pixel intensity and adjust this tolerance as desired. Densities belonging to the feature will be picked up in accordance to the intensity of the anchor&#8217;s pixel or tolerance value. Use the following commands when using Amira.
	<ol>
		<li>Use the Magic Wand Tool for features with distinguishable margins.
		<ol>
			<li>Click on the area of interest, then adjust sliders in Display and Masking to capture correct range of values so that the feature is fully highlighted. Add selection with shortcut &#8220;a&#8221;.</li>
		</ol>
		</li>
		<li>Use the Threshold Tool for features without clearly distinguishable margins.</li>
		<li>Select the Threshold icon. Adjust slider to adjust density within desirable range so that only the features of interest are masked. Click Select button, then add selection with shortcut &#8220;a&#8221;.</li>
		<li>To segment entire volume, select All slices before adding selection.</li>
		<li>To remove noise, select Segmentation &#62; Remove Islands and/or Segmentation &#62; Smooth labels.</li>
	</ol>
	</li>
	<li>Generate a surface for visualization and qualitative analysis as described in the manual tracing section 2.6-2.6.2. If desired, export to other programs for adequate 3D display, quantitative analysis and simulations.</li>
</ol>
4. Custom-tailored Automated Segmentation
Note: Use this approach to create customized scripts for automatic segmentation, which requires background experience in computer science, but allows the ability to create a precise density model from a large volume.
<ol>
	<li>Tools (Specific Example of Shape-Supervised Segmentation in MATLAB27)
	<ol>
		<li>Image pre-processing: Perform de-noising, background removal and image enhancement by using the following pipeline:
		<ol>
			<li>Load the image using the imread command.
			<ol>
				<li>In the command line, enter: &#62;&#62; im = imread($image_path), where $image_path is the location of the image to be analyzed.</li>
			</ol>
			</li>
			<li>From the Image Processing toolbox, call Wiener Filter using an estimated or known Noise-power-to-signal ratio (NSR).</li>
			<li>On the previously processed image, call the image opening function imopen to estimate the background layer, then allocate the outcome as a different mask.
			<ol>
				<li>In the command line, enter: &#62;&#62; background = imopen(im,strel($shape_string,$size)), in this method, $shape_string is equal to &#8216;disk&#8217; the variable $size is given by the analyzer. i.e. &#62;&#62; background = imopen(im,strel(&#8216;disk&#8217;,15)).</li>
			</ol>
			</li>
			<li>Subtract the filtered image with the background.
			<ol>
				<li>In the command line, enter: &#62;&#62; im2 = im - background</li>
			</ol>
			</li>
			<li>Depending on the quality of the results, perform image normalization with or without adaptive Otsu&#8217;s method28, which can be called using the function imadjust from the Image Processing Toolbox.
			<ol>
				<li>In the command line, enter: &#62;&#62; im3 = imadjust(im2)</li>
			</ol>
			</li>
			<li>Prepare the features of interest for segmentation, limiting the regions of interest by cropping the normalized image.
			<ol>
				<li>Using the imtool command, explore the region of interest that is to be cropped and provide the coordinates to the command: &#62;&#62; im3_crop = imcrop(im3, [x1 y1 x2 y2]), where the vector [x1 y1 x2 y2] corresponds to the square region to be cropped.</li>
			</ol>
			</li>
		</ol>
		</li>
		<li>Shape recognition/Supervised shape classification: Train the algorithm by providing specific examples for each different category of objects (linear traces in a 2D image across the features of interest).
		<ol>
			<li>Check that VLFEAT29 API is successfully installed and visit VLFEAT&#8217;s website for more in-depth documentation.</li>
			<li>In the command line, enter: &#62;&#62; [TREE,ASGN] = VL_HIKMEANS(im3_crop,$K,$NLEAVES) where $K is the number of cluster to be used or the number of classes the observer wants to arrange the data into, and $NLEAVES is the desired number of leaf clusters i.e. &#62;&#62; [TREE,ASGN] = VL_HIKMEANS(im3_crop,4,100)</li>
			<li>Use manually segmented features as the input for VLFeat. 
			NOTE: This open source C-based library will perform pixel patching, patch clustering, and cluster center positioning depending on the type of method chosen to work best for the datasets. The available options range from k-mean clustering to texton-based approaches30, and the output is a numerical array that describes the features desired based on the given exemplars.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Segmentation: Use this fully automated, although computationally expensive, approach to segment multiple classes of objects simultaneously, which will be written out as separate maps for further visualization and analysis.
	<ol>
		<li>Load the previously generated numeral array (model).</li>
		<li>Call the support vector machine (SVM) function in VLFeat, using the model and the image to be segmented as an input.
		<ol>
			<li>In the command line, enter: &#62;&#62; [w, b] = vl_svmtrain(x, y, 0.1), where x is the original cropped image im2_crop and y is the objective image, the image that has been manually segmented. Use &#62;&#62; ISEG = VL_IMSEG(I,LABELS) to color the results according to the labels generated by the clustering. 
			NOTE: Based on the characteristics of the model, VLFeat will classify the image on the number of classes (features of interest) assigned from the beginning. Depending on the grade of accuracy desired, it is possible to combine this method with other approaches or estimate cluster parameters such as hull and cluster centers. The output of the SVM algorithm is a probabilistic model and multiple binary masks of the desired classes in the new datasets.</li>
		</ol>
		</li>
		<li>Save results by entering the command: &#62;&#62; imwrite(im, $format, $filename) where $format is &#39;tiff&#39; and $filename is the path for the output file.</li>
		<li>For visualizing images, enter the command: &#62;&#62; imshow(im).</li>
	</ol>
	</li>
</ol>

<ol>
	<li>Auer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+electron+cryo-microscopy+as+a+powerful+structural+tool+in+molecular+medicine.">Three-dimensional electron cryo-microscopy as a powerful structural tool in molecular medicine.</a> J Mol Med (Berl). 78 (4), 191-202 (2000).</li><li>Johnson, M. C., Rudolph, F., Dreaden, T. M., Zhao, G., Barry, B. A., Schmidt-Krey, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+two-dimensional+crystallization+trials+of+small+membrane+proteins+for+structural+biology+studies+by+electron+crystallography.">Assessing two-dimensional crystallization trials of small membrane proteins for structural biology studies by electron crystallography.</a> Journal of visualized experiments JoVE. (44), e1846 (2010).</li><li>Jun, S., Zhao, G., Ning, J., Gibson, G. A., Watkins, S. C., Zhang, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Correlative+microscopy+for+3D+structural+analysis+of+dynamic+interactions.">Correlative microscopy for 3D structural analysis of dynamic interactions.</a> Journal of visualized experiments JoVE. (76), e50386 (2013).</li><li>Meng, X., Zhao, G., Zhang, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structure+of+HIV-1+capsid+assemblies+by+cryo-electron+microscopy+and+iterative+helical+real-space+reconstruction.">Structure of HIV-1 capsid assemblies by cryo-electron microscopy and iterative helical real-space reconstruction.</a> Journal of visualized experiments JoVE. (54), e3041 (2011).</li><li>Chen, S., McDowall, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Electron+Cryotomography+of+Bacterial+Cells.">Electron Cryotomography of Bacterial Cells.</a> Journal of visualized experiments JoVE. (39), e1943 (2010).</li><li>Meyerson, J. R., White, T. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Determination+of+molecular+structures+of+HIV+envelope+glycoproteins+using+cryo-electron+tomography+and+automated+sub-tomogram+averaging.">Determination of molecular structures of HIV envelope glycoproteins using cryo-electron tomography and automated sub-tomogram averaging.</a> Journal of visualized experiments JoVE. (58), e2770 (2011).</li><li>Lucic, V., Forster, F., Baumeister, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structural+studies+by+electron+tomography:+from+cells+to+molecules.">Structural studies by electron tomography: from cells to molecules.</a> Annu Rev Biochem. 74, 833-865 (2005).</li><li>Bajaj, C., Yu, Z., Auer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Volumetric+feature+extraction+and+visualization+of+tomographic+molecular+imaging.">Volumetric feature extraction and visualization of tomographic molecular imaging.</a> J Struct Biol. 144 (1-2), 132-143 (2003).</li><li>Lin, G., Adiga, U., Olson, K., Guzowski, J. F., Barnes, C. A., Roysam, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+hybrid+3D+watershed+algorithm+incorporating+gradient+cues+and+object+models+for+automatic+segmentation+of+nuclei+in+confocal+image+stacks.">A hybrid 3D watershed algorithm incorporating gradient cues and object models for automatic segmentation of nuclei in confocal image stacks.</a> Cytometry A. 56 (1), 23-36 (2003).</li><li>Volkmann, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+novel+three-dimensional+variant+of+the+watershed+transform+for+segmentation+of+electron+density+maps.">A novel three-dimensional variant of the watershed transform for segmentation of electron density maps.</a> Journal of Structural Biology. 138 (1), 123-129 (2002).</li><li>Rigort, A., G&#252;nther, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+segmentation+of+electron+tomograms+for+a+quantitative+description+of+actin+filament+networks.">Automated segmentation of electron tomograms for a quantitative description of actin filament networks.</a> Journal of structural biology. 177 (1), 135-144 (2012).</li><li>Cremers, D., Rousson, M., Deriche, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Review+of+Statistical+Approaches+to+Level+Set+Segmentation.">A Review of Statistical Approaches to Level Set Segmentation.</a> Integrating Color, Texture, Motion and Shape. International Journal of Computer Vision. 72 (2), 195-215 (2007).</li><li>Lin, Z., Davis, L. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Shape-based+human+detection+and+segmentation+via+hierarchical+part-template+matching.">Shape-based human detection and segmentation via hierarchical part-template matching.</a> IEEE Trans Pattern Anal Mach Intell. 32 (4), 604-618 (2010).</li><li>Pettersen, E. F., Goddard, T. D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=UCSF+Chimera--a+visualization+system+for+exploratory+research+and+analysis.">UCSF Chimera--a visualization system for exploratory research and analysis.</a> J Comput Chem. 25 (13), 1605-1612 (2004).</li><li>Kremer, J. R., Mastronarde, D. N., McIntosh, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Computer+visualization+of+three-dimensional+image+data+using+IMOD.">Computer visualization of three-dimensional image data using IMOD.</a> J Struct Biol. 116 (1), 71-76 (1996).</li><li>Zhang, Q., Bettadapura, R., Bajaj, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Macromolecular+structure+modeling+from+3D+EM+using+VolRover+2.0.">Macromolecular structure modeling from 3D EM using VolRover 2.0.</a> Biopolymers. 97 (9), 709-731 (2012).</li><li>Giannuzzi, L. A., Stevie, F. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+review+of+focused+ion+beam+milling+techniques+for+TEM+specimen+preparation.">A review of focused ion beam milling techniques for TEM specimen preparation.</a> Micron. 30 (3), 197-204 (1999).</li><li>Heymann, J. A. W., Hayles, M., Gestmann, I., Giannuzzi, L. A., Lich, B., Subramaniam, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Site-specific+3D+imaging+of+cells+and+tissues+with+a+dual+beam+microscope.">Site-specific 3D imaging of cells and tissues with a dual beam microscope.</a> Journal of structural biology. 155 (1), 63-73 (2006).</li><li>Knott, G., Rosset, S., Cantoni, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Focussed+ion+beam+milling+and+scanning+electron+microscopy+of+brain+tissue.">Focussed ion beam milling and scanning electron microscopy of brain tissue.</a> Journal of visualized experiments JoVE. (53), e2588 (2011).</li><li>Wirth, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Focused+Ion+Beam+(FIB)+combined+with+SEM+and+TEM:+Advanced+analytical+tools+for+studies+of+chemical+composition,+microstructure+and+crystal+structure+in+geomaterials+on+a+nanometre+scale.">Focused Ion Beam (FIB) combined with SEM and TEM: Advanced analytical tools for studies of chemical composition, microstructure and crystal structure in geomaterials on a nanometre scale.</a> Chemical Geology. 261 (3-4), 217-229 (2009).</li><li>Denk, W., Horstmann, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Serial+block-face+scanning+electron+microscopy+to+reconstruct+three-dimensional+tissue+nanostructure.">Serial block-face scanning electron microscopy to reconstruct three-dimensional tissue nanostructure.</a> PLoS Biol. 2 (11), e329 (2004).</li><li>Frangakis, A. S., Hegerl, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Noise+reduction+in+electron+tomographic+reconstructions+using+nonlinear+anisotropic+diffusion.">Noise reduction in electron tomographic reconstructions using nonlinear anisotropic diffusion.</a> Journal of structural biology. 135 (3), 239-250 (2001).</li><li>Jiang, W., Baker, M. L., Wu, Q., Bajaj, C., Chiu, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Applications+of+a+bilateral+denoising+filter+in+biological+electron+microscopy.">Applications of a bilateral denoising filter in biological electron microscopy.</a> Journal of Structural Biology. 144 (1), 114-122 (2003).</li><li>Van der Heide, P., Xu, X. P., Marsh, B. J., Hanein, D., Volkmann, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Efficient+automatic+noise+reduction+of+electron+tomographic+reconstructions+based+on+iterative+median+filtering.">Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering.</a> Journal of structural biology. 158 (2), 196-204 (2007).</li><li>Schneider, C. A., Rasband, W. S., Eliceiri, K. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=NIH+Image+to+ImageJ:+25+years+of+image+analysis.">NIH Image to ImageJ: 25 years of image analysis.</a> Nat Methods. 9 (7), 671-675 (2012).</li><li>Schindelin, J., Arganda-Carreras, I., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fiji:+an+open-source+platform+for+biological-image+analysis.">Fiji: an open-source platform for biological-image analysis.</a> Nat Methods. 9 (7), 676-682 (2012).</li><li>MathWorks. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> MATLAB. , (2012).</li><li>Otsu, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Threshold+Selection+Method+from+Gray-Level+Histograms.">A Threshold Selection Method from Gray-Level Histograms.</a> Systems, Man and Cybernetics, IEEE Transactions on. 9, 62-66 (1979).</li><li>Vedaldi, A., Fulkerson, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> VLFeat: An Open and Portable Library of Computer Vision Algorithms. , (2008).</li><li>Zhu, S. C., Guo, C., Wang, Y., Xu, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=What+are+Textons?.">What are Textons?.</a> International Journal of Computer Vision. 62 (1-2), 121-143 (2005).</li><li>Gagnon, L. H., Longo-Guess, C. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+chloride+intracellular+channel+protein+CLIC5+is+expressed+at+high+levels+in+hair+cell+stereocilia+and+is+essential+for+normal+inner+ear+function.">The chloride intracellular channel protein CLIC5 is expressed at high levels in hair cell stereocilia and is essential for normal inner ear function.</a> The Journal of neuroscience the official journal of the Society for Neuroscience. 26 (40), 10188-10198 (2006).</li><li>Briand, P., Petersen, O. W., Van Deurs, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+diploid+nontumorigenic+human+breast+epithelial+cell+line+isolated+and+propagated+in+chemically+defined+medium.">A new diploid nontumorigenic human breast epithelial cell line isolated and propagated in chemically defined medium.</a> In vitro cellular &amp; developmental biology journal of the Tissue Culture Association. 23 (3), 181-188 (1987).</li><li>Petersen, O. W., R&#248;nnov-Jessen, L., Howlett, A. R., Bissell, M. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interaction+with+basement+membrane+serves+to+rapidly+distinguish+growth+and+differentiation+pattern+of+normal+and+malignant+human+breast+epithelial+cells.">Interaction with basement membrane serves to rapidly distinguish growth and differentiation pattern of normal and malignant human breast epithelial cells.</a> Proceedings of the National Academy of Sciences of the United States of America. 89 (19), 9064-9068 (1992).</li><li>Shin, J. B., Krey, J. F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Molecular+architecture+of+the+chick+vestibular+hair+bundle.">Molecular architecture of the chick vestibular hair bundle.</a> Nature neuroscience. 16 (3), 365-374 (2013).</li><li>Yang, W., Zeng, Z., Max, N., Auer, M., Crivelli, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Simplified+Surface+Models+of+Tubular+Bacteria+and+Cytoskeleta.">Simplified Surface Models of Tubular Bacteria and Cytoskeleta.</a> Journal of Information &amp; Computational Science. 9 (6), 1589-1598 (2012).</li></ol>

The authors declare they have no competing financial interests.

Effective strategies for the extraction of relevant features from 3D EM volumes are urgently needed in order to keep up with the data tsunami that has recently hit biological imaging. While data can be generated in hours or days, it takes many months to analyze the 3D volumes in depth. Therefore, it is clear that the image analysis has become the bottleneck for scientific discoveries; without adequate solutions for these problems, imaging scientists become the victims of their own success. This is in part due to the high complexity of the data and also the macromolecular crowding typically found in biological cells, where proteins and protein complexes border one another and essentially appear as a continuous gradient of grayscale densities. The problem is complicated by sample preparation and imaging imperfections, and in some cases image reconstruction artifacts, leading to less than perfect volumetric data that can pose challenges for fully automated approaches. Most significant, however, is the fact that the experts in sample preparation, imaging, and the biological interpretation are seldom well versed in computational science, and hence require guidance on how to effectively approach feature extraction and analysis. Therefore, through the use of various examples, the protocol explains how to prepare data for segmentation, as well as the steps for manual abstracted model generation, automated density-based segmentation, manual tracing of features of interest, and custom-tailored automated segmentation. The manual and automatic approaches outlined in the procedure can be found in a large variety of segmentation software, some of which are mentioned here, but others perform similar functions and are equally well suited.
The results demonstrate that the effectiveness of each of the 3D segmentation approaches varies for each different type of data sets. Even though the different approaches produced qualitatively similar 3D renderings as the end product, the amount of time and effort spent on each during the segmentation process varied significantly. The recommendations for appropriate image characteristics and personal aims per segmentation approach are summarized in Figure 5, which is further explained in the following four subsections. These criteria were applied to the six datasets, as shown in the decision flow chart of Figure 6. Although Figures 5 and 6 are merely meant to provide a rationale for each data set and how each of the criteria were weighted in the decision making process, they do not provide a foolproof guidance, but rather a starting point. There are simply too many criteria that influence the decision-making process: some are objective criteria, such as data set characteristics, whereas others are more subjective criteria, such as the desired objective. It is safe to say that data sets that display a high level of contrast with sharp crisp boundaries, have features that are well separated and relatively homogeneous (not too diverse), and are processed with the objective of displaying a density model for a large number of objects, automated approaches will be superior, if not for the fact that manual approaches would simply be resource (time)-prohibitive. On the other hand, if contrast is low, the data is fuzzy and thus requires an expert&#8217;s knowledge, the objects are crowded, and the features show a high diversity and are thus heterogeneous, one may not have any other choice than manual feature extraction/segmentation.

Manual Abstracted Model Generation
Manual abstracted model tracing is particularly effective in segmenting linear elements, providing seeds points (balls) that can be automatically connected (sticks). Such balls and sticks-models can be very powerful to measure length and orientation of such model and provide an adequately abstracted model for both qualitative inspection and quantitative analysis. Manual abstracted model generation is commonly used when minimizing resources spent on the analysis is more important than absolute fidelity to the shapes of the original data. It is most successful with linear and homogenous features of interest (e.g., filaments, tubes). Data contrast, crispness, and crowdedness do not play a major role in determining this method&#8217;s success, as long as the human eye can recognize the object of interest. Sometimes such models can also be utilized as a skeleton to segment the 3D map in a zone around the skeleton. Although the model is abstract rather than a reflection of exact densities, it represents a skeletonized version of the 3D density and thus allows for clutter-free visualization and qualitative analysis. Quantitative measurements such as length can also be determined from the approximate model. For an example of software with manual abstracted model generation, please visit Chimera&#8217;s detailed user guide online at <a href="http://www.cgl.ucsf.edu/chimera/current/docs/UsersGuide/index.html">http://www.cgl.ucsf.edu/chimera/current/docs/UsersGuide/index.html</a>.

Manual Tracing of Features of Interest
Manual paintbrush tracing works well with almost all data characteristics, but it is also the most time consuming method. At times, it is the only technique for extracting a feature of interest from a complex image set containing a large variety of features, such as the thin and convoluted cell membrane. One useful tool available in some programs allows for interpolation between intermittently segmented slices when the feature of interest changes smoothly. Manual tracing can be applied most efficiently if the data is crisp and has medium to high contrast, but it can also be utilized for more challenging data sets, as long as the user is familiar with the object of interest. The data complexity can range from discrete objects to complex and crowded data sets, where objects are closely packed. In the latter case, manual segmentation may be the only choice, as automatic approaches often struggle to segment the desired volume and extract too much or too little. Difficult feature morphologies, such as convoluted sheets or volumes, can also be extracted by this method. However, the user should keep in mind that a dataset with several difficult characteristics can only be segmented if the population density of the features of interest is low, as segmentation of high population densities of the features of interest becomes time-prohibitive. For an example of software with manual tracing, please visit Amira&#8217;s detailed user guide online at <a href="http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf">http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf</a>.

Automated Density-based Segmentation
In contrast to the manual techniques, the automated approaches are generally less time-consuming, which is an important factor to consider when segmenting a large stack of images. However, simple thresholding may not be as accurate, and much more time may be spent on refinement and curation of the automatically segmented volume. Automated density-based segmentation works best on data sets that display a large number of similar features of interest that all require segmentation. If the data is more complex, these automated techniques can still serve as an initial step, but will likely require some manual intervention down the line in order to specify a subvolume containing the feature of interest. This strategy typically works well on linear morphologies or convoluted volumes, but it is rarely successful with thin convoluted sheets such as cell membranes. Minimal user intervention with automated approaches enables segmentation through large or small volumes, while expending few user resources such as time in return for high fidelity. For an example of software with automated density-based segmentation, please visit Amira&#8217;s detailed user guide online at <a href="http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf">http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf</a>.

Custom-Tailored Automated Segmentation
Custom-tailored automated segmentation allows the power customization of algorithms for a specific data set, but it is often specific to the data set or data type, appropriate for a limited number of feature characteristics, and cannot be generalized easily. The procedure showcased here differs from the general automated segmentation approaches, such as watershed immersion and other level set methods, which rely on a programmed determination of critical seed points, followed by fast-marching cube expansion from these seed points. A variation on this theme is boundary segmentation, where gradient vector information informs feature boundaries. In contrast, the customized script used here relies on a training stage where the user manually traces a few examples. Through machine learning, specific algorithms will detect and then learn to independently recognize properties and data characteristics consistently found in the traces. An expert user can retrain the algorithms and improve the accuracy of segmentation by including more example traces to provide a larger set of feature criteria. Overall, thresholding and related approaches, or even custom-tailored approaches may not be as useful to extract a single feature of interest from an image with complex diversity of organelles or shapes, as curation may be just as labor intensive as manual tracing.

Strategies for Triaging Data and Choosing a Segmentation Approach 

Given the subjective and objective criteria presented in Figure 4 and summary of suitable datasets in Figure 5, the decision making scheme depicted in Figure 6 can assist an effective assessment of feature extraction strategies for a large variety of data sets. The data sets are triaged in four consecutive decisions, each of which may include any one of the four respective objectives as well as the four subjective criteria introduced in Figure 4. As an example, Figure 6 is the rational for triaging each of the six data sets shown in Figure 3. Undoubtedly, for each data set there is not a single unique path, but rather different paths through this matrix following different criteria for decision-making that may lead to the same or different recommendation for data segmentation. While every data set will have its own set of properties, which cannot be anticipated, six examples are given, each paired with an explanation of the rationale behind the preferred feature extraction/segmentation approach. Most also include a proposition for an alternative decision route that either results in the use of the same or a different segmentation approach (Figure 6).
The kinocilium is a crisp data set with clearly defined boundaries, which makes automated approaches more likely to succeed. All features of interest are well separated, again favoring an automated approach. In addition, the features of interest are similar to one another, making it a relatively homogeneous data set ideal for custom-tailored segmentation. Lastly, the aim was to extract the entire feature, favoring a semi-automated approach. As a consequence, it was concluded that an automated thresholding (solid green line) as well as a custom-designed (e.g., shape supervised segmentation) approach (dotted green line) are both likely to do well on this data set.
Similar criteria, although placed in a different order in the decision making network, apply to the case of bacteria. A custom-tailored approach is recommended in part because this data set was very large; hence, limited resources prohibit a labor-intensive manual intervention/segmentation approach. While thresholding would have yielded acceptable results, the custom-designed approach was able to execute the study&#8217;s key objective to separate the roundish bacterial shapes from the extracellular metal deposits, located either in-between the bacteria or right next to the bacteria, and therefore the custom-tailored approach was preferred.
For stereocilia data sets, the first consideration was the desired objective: the goal can either be to show the entire density or to create geometrical models. The volume of interest was a crowded area, and the objective was to segment a large number of objects as separated objects in order to subsequently execute quantitative volumetric analysis, including lengths, numbers, distances, orientation, etc. It was helpful that the objects of interest were mainly linear, and this made geometrical model tracing the method of choice. However, if instead the objective has been to show the entire density, then the linear feature morphology as well as relatively high contrast with sharply defined boundaries would make an automated thresholding protocol feasible.
The cell membranes and mitochondria data cases are challenging for automated approaches due to their categories of feature morphology: convoluted sheets and volumes, respectively. The goal is to trace the cell or mitochondria outline accurately, but there are only finite resources to do so. In addition, the features of interest are complex and cannot be easily automatically detected or shape-encoded, although for the mitochondria data sets the customized scripting approach taken for the bacteria may possibly be applied with further customization. Fortunately, the membrane and mitochondria themselves only represent a small fraction of the entire volume and hence, manual tracing is a straightforward albeit time-consuming approach. Manual tracing is also the method of choice for such data sets when the contrast is rather low and the boundaries are rather fuzzy. As a result, even if they constitute a significant portion of the data sets, such convoluted sheets must be manually traced, simply due to the lack of a better alternative.
The plant data set posed its own challenges because the goal was to segment all objects, which are densely spaced and make up a crowded scenery. Displaying the density as-is would enable measurements about the shape and organization of the objects, but because manually segmenting each filamentous object is too costly, automatic thresholding was employed instead.
The various steps and corresponding results in creating a 3D model have been displayed here, but more importantly, the data characteristics and personal criteria found to be crucial in determining the best path of segmentation have also been elucidated. The important characteristics of the image data itself include what is described here as contrast, crowdedness, crispness, and the number of different shapes or features (such as organelles, filaments, membranes). Subjective criteria to consider include the desired objective of segmentation (measuring/counting, skeletonized representation of the data/displaying volumes in 3D renderings), morphological characteristics of the feature of interest (linear, elongated, networked, complex, convoluted), the density of features of interest in relation to the entire volume (the fraction of the objects that are important and need to be extracted), and balancing the tradeoffs of expending resources to the segmentation&#8217;s fidelity of the original data and the decreasing return on the investment resulting in incremental improvements for substantially higher allocation of resources.
The field of image segmentation has significantly matured over the recent years, yet there is no silver bullet, no algorithm or program that can do it all. Data set sizes have grown from hundreds of megabytes to routinely tens of gigabytes, and they are now starting to exceed terabytes, making manual segmentation near impossible. Thus, more resources need to be invested in the clever and time-effective feature extraction approaches that mimic the human decision making process. Such efforts will need to be combined with (1) geographic information system (GIS)-based semantic hierarchical data bases (similar to Google Earth), (2) data abstraction techniques (i.e., transitioning from a voxel to geometric/volumetric representation) compatible with computer assisted design (CAD) software in order to significantly reduce the amount of data and thus enabling the display of larger volumes35, (3) simulation techniques, as they are frequently used in the engineering disciplines, as well as (4) advanced animation and movie making capabilities, including fly-through animations (similar to what is developed for the gaming industry).&#160;
Clearly, efficient feature extraction and segmentation lies at the heart of this coming revolution in cellular high-resolution imaging, and while better approaches will always be needed, the principles presented here, as well as the examples of what approach was taken for different data types, will provide some valuable information for making a decision on which approach to take.

Traditionally, the electron microscopy (EM) field has been divided into 1) the structural biology branch using high and super-high resolution TEM, typically combined with implicit or explicit data averaging to investigate the three-dimensional (3D) structure of macromolecular complexes with a defined composition and typically a relatively small size1-4, and 2) the cellular imaging branch in which entire cellular sceneries are visualized1,5,6. While the structural biology branch has undergone a spectacular development over the last four decades, the cell biology branch was mostly restricted to two dimensions, often on less-than-optimally preserved samples. Only with the advent of electron tomography in the last decade has cell biological ultrastructural imaging expanded into the third dimension5,7, where typically averaging cannot be performed as the cellular sceneries, and thus the features of interest, are typically unique.
Although visualized cellular scenes are often stunning to the eye, efficient extraction of the features of interest and subsequent quantitative analysis of such highly complex cellular volumes lag behind, in part because the precise protein composition is usually unknown, therefore making it challenging to interpret these cellular 3D volumes. To this date, extensive biological expertise is often needed in order to interpret complex tomograms, or even to identify the important regions and essential components in the 3D volume. As a further complication, visualization of 3D volumes is remarkably non-trivial. 3D volumes can be thought of and thus visualized as stacks of 2D images. Slice-by-slice inspection of sequential 2D images reduces the complexity, but it also limits feature extraction and thus quantitative analysis to the two dimensions. However, for most 3D objects, the depiction of 3D volumes as merely a stack of consecutive planes leads to an incomplete and skewed perspective into a particular system&#8217;s 3D nature. Alternative modes of visual inspection require either volume rendering or surface rendering, which&#8212;given the often dense nature of a cellular volume&#8212;can easily lead to an obstructed view of nested objects or overwhelm a user altogether, thus making interactive manual segmentation difficult.
To remedy these barriers, a large variety of automated feature extraction (segmentation) approaches have been developed that are typically either density- or gradient-based8-10. However, these methods tend to segment the entire volume regardless of which areas or features are of interest to the expert, although some recent methods can target a specific feature of interest such as actin filaments11. In addition, the programs executing automated segmentation can sometimes result in the production of a large number of sub-volumes (e.g., when applying watershed immersion segmentation) that often need to be merged manually back into comprising the whole feature of interest or be subjected to further segmentation. This holds true particularly for complex and crowded data sets, thus most rendering computer algorithms are unable to extract only the features of interest with fidelity, and substantial curation efforts by an expert are often needed to produce a desired segmented volume.
Moreover, custom solutions to a highly specific problem are often published as a scientific meeting paper, with little to no emphasis on making them broad and comprehensive tools accessible to researchers who do not have intimate knowledge of the fields of mathematics, computer science and/or computer graphics. A customizable programming software environment, containing a range of image analysis libraries, can be a powerful tool set allowing users to efficiently write their own modules for accurate segmentation. However, this approach requires extensive training and a background in computer science in order to take advantage of its many features or capabilities for image analysis. One can work within such a versatile software environment for certain data sets where the features are more sparse, e.g., by utilizing powerful shape-based approaches which rely on the unique geometry of &#8220;templates&#8221; to separate objects of interest from their surroundings12,13.
A fair variety of computer graphics visualization packages exist for interactive manual segmentation and model building. Some packages are commercially available, while others are of academic origin and distributed free of charge, such as: University of California San Francisco Chimera14, University of Colorado IMOD15, and University of Texas Austin VolumeRover16. However, the wide range and complexity of features and capabilities these programs possess steepens the learning curve for each. Certain visualization programs provide simple geometrical models, such as balls and sticks of various sizes, which can be placed into the density maps in order to create a simplified model of the complex 3D volume. These models then allow simple geometric and volumetric measurements and therefore go beyond just the &#8220;pretty picture&#8221;. Such manual tracing of objects works well for volumes where only a small number of objects need to be traced and extracted. However, the recent development of large volume 3D ultrastructural imaging using either focused ion beam scanning electron microscopy (FIB-SEM)17-20 or serial block face scanning electron microscopy (SBF-SEM)21 presents the additional complication that the size of 3D data sets can range from gigabytes to tens and hundreds of gigabytes, and even the terabytes. Therefore, such large 3D volumes are virtually inaccessible to manual feature extraction, and hence efficient user-guided semi-automated feature extraction will be one of the bottlenecks for efficient analysis of 3D volumes in the foreseeable future.
Presented here are four different segmentation approaches that are routinely used on a large range of biological image types. These methods are then compared for their effectiveness for different types of data sets, allowing a compilation into a guide to help biologists decide what may be the best segmentation approach for effective feature extraction of their own data. As detailed user manuals are available for most of the programs described, the aim is not to make potential users familiar with any one of these particular packages. Instead, the goal is to demonstrate the respective strengths and limitations of these different segmentation strategies by applying them to six example data sets with diverse characteristics. Through this comparison, a set of criteria have been developed that are either based on the objective image characteristics of the 3D data sets, such as data contrast, crispness, crowdedness, and complexity, or stem from subjective considerations, such as the desired objective for segmentation, morphologies of the features to be segmented, population density of the features of interest, meaning the fraction of the volume occupied by the feature of interest, and how one proceeds optimally with finite resources such as time and availability of staff. These different example data sets illustrate how these objective and subjective criteria can be applied sequentially in a variety of combinations to yield a pairing of certain feature extraction approaches with certain types of data sets. The recommendations given will hopefully help novices faced with a large variety of segmentation options choose the most effective segmentation approach for their own 3D volume.
While the focus of this paper is feature extraction, attention to data collection and pre-processing data is crucial to efficient segmentation. Oftentimes staining of samples can be uneven, and hence, potential staining artifacts should be considered in the segmentation procedure. However, stain usually gives higher signal-to-noise, and therefore requires less filtering and other mathematical treatment of cellular volumes, which could potentially also result in artifacts. The respective raw image data sets need to be acquired at the correct contrast and camera pixel settings, aligned, and reconstructed into a 3D volume. For tomograms, aligned images are reconstructed typically using weighted back-projection, and then the data set is usually subjected to denoising algorithms such as non-linear anisotropic diffusion22, bilateral filtering23, or recursive median filtering24. FIB-SEM and SBF-SEM imaging data are aligned by cross-correlating consecutive slices in XY utilizing programs such as ImageJ25. Contrast enhancement and filtering can be applied to boost the features of interest and thus to de-noise the image stack. Filtering can be performed either on the entire volume prior to subvolume selection or on the selected subvolumes, as filtering approaches can be computationally expensive. Down-sampling of the data (binning), which is sometimes used for noise reduction and/or file size reduction, is only recommended if the data has been significantly oversampled compared to the anticipated resolution.
After noise-reduction, the processed images can then be segmented by various methods, and the focus in this study is on the following four: (1) manual abstracted model generation through creating a ball-and-stick model, (2) manual tracing of features of interest, (3) automated threshold-based density, and (4) custom-tailored automated segmentation via a script for project specific segmentation. Boundary segmentation8 and immersive watershed segmentation10 are better alternatives to simple thresholding, but they belong in the same category and have not been included explicitly in this discussion.
&#160;Manual tracing of densities requires outlining the features of interest, slice-by-slice, which allows the retention of the original density of respective sub-cellular areas. This approach allows maximal control of the segmentation process, but is a tedious and labor-intensive process.
Automated threshold-based (and related) density segmentation approaches are semi-automatic, where an algorithm chooses pixels based on a set of user-defined parameters. Several academic (free) visualization packages, such as UCSF Chimera, IMOD, Fiji26, and VolumeRover are available, as well as commercial (requiring paid licenses) packages, and both types typically include one or more of these segmentation approaches. The software packages used in this work to illustrate these different methods include both commercial programs and academic open source programs for manually generating an abstract model, as well as manual and automated density segmentation. However, open source software can sometimes offer more advanced options through the possibility of customization.
A comparison of these techniques using different types of data sets led to the following presentation of rules and guidance on how to approach the segmentation of diverse biological data 3D volumes, which to our knowledge has not yet been published. Thus, this is the first systematic comparison of the different approaches and their usefulness on data sets with varying characteristics for users with different aims.

<table border="0" cellpadding="0" cellspacing="0" width="647">
	<tbody>
		<tr height="43">
			<td height="43" style="height:43px;width:99px;">Material Name</td>
			<td style="width:151px;">Company</td>
			<td style="width:151px;"></td>
			<td style="width:248px;">Comments</td>
		</tr>
		<tr height="43">
			<td height="43" style="height:43px;width:99px;">Amira</td>
			<td style="width:151px;">FEI Visualization Sciences Group</td>
			<td style="width:151px;"></td>
			<td>http://www.vsg3d.com/amira/overview</td>
		</tr>
		<tr height="22">
			<td height="22" style="height:22px;width:99px;">Chimera</td>
			<td style="width:151px;">UCSF</td>
			<td style="width:151px;"></td>
			<td style="width:248px;"><a href="http://www.cgl.ucsf.edu/chimera/">http://www.cgl.ucsf.edu/chimera/</a></td>
		</tr>
		<tr height="43">
			<td height="43" style="height:43px;width:99px;"><a name="RANGE!A4">Fiji/ImageJ</a></td>
			<td style="width:151px;">National Institute of Health</td>
			<td style="width:151px;"></td>
			<td style="width:248px;">http://fiji.sc/Fiji, http://rsbweb.nih.gov/ij/</td>
		</tr>
		<tr height="64">
			<td height="64" style="height:64px;width:99px;">IMOD</td>
			<td style="width:151px;">Boulder Laboratory for 3D Electron Microscopy of Cells</td>
			<td style="width:151px;"></td>
			<td style="width:248px;">http://bio3d.colorado.edu/imod/</td>
		</tr>
		<tr height="35">
			<td height="35" style="height:35px;width:99px;">Photoshop</td>
			<td style="width:151px;">Adobe</td>
			<td style="width:151px;"></td>
			<td style="width:248px;"><a href="http://www.adobe.com/products/%20photoshopfamily.html">http://www.adobe.com/products/ photoshopfamily.html</a></td>
		</tr>
		<tr height="21">
			<td height="21" style="height:21px;width:99px;">MATLAB</td>
			<td style="width:151px;">MathWorks</td>
			<td style="width:151px;"></td>
			<td style="width:248px;"><a href="http://www.mathworks.com/">http://www.mathworks.com/</a></td>
		</tr>
		<tr height="17">
			<td height="17" style="height:17px;">VLFeat</td>
			<td>VLFeat</td>
			<td></td>
			<td>http://www.vlfeat.org/</td>
		</tr>
	</tbody>
</table>

from voxels to knowledge: a practical guide to the segmentation of complex electron microscopy 3d-data

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, enabling the visualization of large macromolecular machines, such as adhesion complexes, as well as higher-order structures, such as the cytoskeleton and cellular organelles in their respective cell and tissue context. Given the inherent complexity of cellular volumes, it is essential to first extract the features of interest in order to allow visualization, quantification, and therefore comprehension of their 3D organization. Each data set is defined by distinct characteristics, e.g., signal-to-noise ratio, crispness (sharpness) of the data, heterogeneity of its features, crowdedness of features, presence or absence of characteristic shapes that allow for easy identification, and the percentage of the entire volume that a specific region of interest occupies. All these characteristics need to be considered when deciding on which approach to take for segmentation.
The six different 3D ultrastructural data sets presented were obtained by three different imaging approaches: resin embedded stained electron tomography, focused ion beam- and serial block face- scanning electron microscopy (FIB-SEM, SBF-SEM) of mildly stained and heavily stained samples, respectively. For these data sets, four different segmentation approaches have been applied: (1) fully manual model building followed solely by visualization of the model, (2) manual tracing segmentation of the data followed by surface rendering, (3) semi-automated approaches followed by surface rendering, or (4) automated custom-designed segmentation algorithms followed by surface rendering and quantitative analysis. Depending on the combination of data set characteristics, it was found that typically one of these four categorical approaches outperforms the others, but depending on the exact sequence of criteria, more than one approach may be successful. Based on these data, we propose a triage scheme that categorizes both objective data set characteristics and subjective personal criteria for the analysis of the different data sets.

Watch this Scientific Journal Video about From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data at JoVE.com

From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

The bottleneck for cellular 3D electron microscopy is feature extraction (segmentation) in highly complex 3D density maps. We have developed a set of criteria, which provides guidance regarding which segmentation approach (manual, semi-automated, or automated) is best suited for different data types, thus providing a starting point for effective segmentation.

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, ...

from-voxels-to-knowledge-practical-guide-to-segmentation-complex

Research

JoVE Journal

Bioengineering

Low-Cost Cryo-Light Microscopy Stage Fabrication for Correlated Light/Electron Microscopy

The coupling of cryo-light microscopy (cryo-LM) and cryo-electron microscopy (cryo-EM) poses a number of advantages for understanding cellular 
dynamics and ultrastructure. First, cells can be imaged in a near native environment for both techniques. Second, due to the vitrification process, samples are
 preserved by rapid physical immobilization rather than slow chemical fixation. Third, imaging the same sample with both cryo-LM and cryo-EM provides correlation of
 data from a single cell, rather than a comparison of "representative samples". While these benefits are well known from prior studies, the widespread use of correlative 
 cryo-LM and cryo-EM remains limited due to the expense and complexity of buying or building a suitable cryogenic light microscopy stage. Here we demonstrate the 
 assembly, and use of an inexpensive cryogenic stage that can be fabricated in any lab for less than $40 with parts found at local hardware and grocery stores. This 
 cryo-LM stage is designed for use with reflected light microscopes that are fitted with long working distance air objectives. For correlative cryo-LM and cryo-EM 
 studies, we adapt the use of carbon coated standard 3-mm cryo-EM grids as specimen supports. After adsorbing the sample to the grid, previously established protocols 
 for vitrifying the sample and transferring/handling the grid are followed to permit multi-technique imaging. As a result, this setup allows any laboratory with a 
 reflected light microscope to have access to direct correlative imaging of frozen hydrated samples.

We demonstrate the fabrication of a low-cost cryogenic stage designed to fit most reflected light microscopes. This lab-built cryogenic stage enables efficient and reliable correlative imaging between cryo-light and cryo-electron microscopy.

The coupling of cryo-light microscopy (cryo-LM) and cryo-electron microscopy (cryo-EM) poses a number of advantages for understanding cellular 
dynamics ...

Design of a Cyclic Pressure Bioreactor for the Ex Vivo Study of Aortic Heart Valves

The aortic valve, located between the left ventricle and the aorta, allows for unidirectional blood flow, preventing backflow into the ventricle. Aortic valve leaflets are composed of interstitial cells suspended within an extracellular matrix (ECM) and are lined with an endothelial cell monolayer. The valve withstands a harsh, dynamic environment and is constantly exposed to shear, flexion, tension, and compression. Research has shown calcific lesions in diseased valves occur in areas of high mechanical stress as a result of endothelial disruption or interstitial matrix damage1-3. Hence, it is not surprising that epidemiological studies have shown high blood pressure to be a leading risk factor in the onset of aortic valve disease4. 
The only treatment option currently available for valve disease is surgical replacement of the diseased valve with a bioprosthetic or mechanical valve5. Improved understanding of valve biology in response to physical stresses would help elucidate the mechanisms of valve pathogenesis. In turn, this could help in the development of non-invasive therapies such as pharmaceutical intervention or prevention. Several bioreactors have been previously developed to study the mechanobiology of native or engineered heart valves6-9. Pulsatile bioreactors have also been developed to study a range of tissues including cartilage10, bone11 and bladder12. The aim of this work was to develop a cyclic pressure system that could be used to elucidate the biological response of aortic valve leaflets to increased pressure loads. 
The system consisted of an acrylic chamber in which to place samples and produce cyclic pressure, viton diaphragm solenoid valves to control the timing of the pressure cycle, and a computer to control electrical devices. The pressure was monitored using a pressure transducer, and the signal was conditioned using a load cell conditioner. A LabVIEW program regulated the pressure using an analog device to pump compressed air into the system at the appropriate rate. The system mimicked the dynamic transvalvular pressure levels associated with the aortic valve; a saw tooth wave produced a gradual increase in pressure, typical of the transvalvular pressure gradient that is present across the valve during diastole, followed by a sharp pressure drop depicting valve opening in systole. The LabVIEW program allowed users to control the magnitude and frequency of cyclic pressure. The system was able to subject tissue samples to physiological and pathological pressure conditions. This device can be used to increase our understanding of how heart valves respond to changes in the local mechanical environment.

Design of a Cyclic Pressure Bioreactor for the Ex Vivo Study of Aortic Heart Valves

A cyclic pressure bioreactor capable of subjecting heart valve tissue to physiological and pathological pressure conditions has been designed. A LabVIEW program allows users to control pressure magnitude, amplitude and frequency. This device can be used to study the mechanobiology of heart valve tissue or isolated cells.

The aortic valve, located between the left ventricle and the aorta, allows for unidirectional blood flow, preventing backflow into the ventricle. Aortic ...

Fluorescence-quenching of a Liposomal-encapsulated Near-infrared Fluorophore as a Tool for In Vivo Optical Imaging

Optical imaging offers a wide range of diagnostic modalities and has attracted a lot of interest as a tool for biomedical imaging. Despite the enormous number of imaging techniques currently available and the progress in instrumentation, there is still a need for highly sensitive probes that are suitable for in vivo imaging. One typical problem of available preclinical fluorescent probes is their rapid clearance in vivo, which reduces their imaging sensitivity. To circumvent rapid clearance, increase number of dye molecules at the target site, and thereby reduce background autofluorescence, encapsulation of the near-infrared fluorescent dye, DY-676-COOH in liposomes and verification of its potential for in vivo imaging of inflammation was done. DY-676 is known for its ability to self-quench at high concentrations. We first determined the concentration suitable for self-quenching, and then encapsulated this quenching concentration into the aqueous interior of PEGylated liposomes. To substantiate the quenching and activation potential of the liposomes we use a harsh freezing method which leads to damage of liposomal membranes without affecting the encapsulated dye. The liposomes characterized by a high level of fluorescence quenching were termed Lip-Q. We show by experiments with different cell lines that uptake of Lip-Q is predominantly by phagocytosis which in turn enabled the characterization of its potential as a tool for in vivo imaging of inflammation in mice models. Furthermore, we use a zymosan-induced edema model in mice to substantiate the potential of Lip-Q in optical imaging of inflammation in vivo. Considering possible uptake due to inflammation-induced enhanced permeability and retention (EPR) effect, an always-on liposome formulation with low, non-quenched concentration of DY-676-COOH (termed Lip-dQ) and the free DY-676-COOH were compared with Lip-Q in animal trials.

Fluorescence-quenching of a Liposomal-encapsulated Near-infrared Fluorophore as a Tool for In Vivo Optical Imaging

The use of fluorophores for in vivo imaging can be greatly limited by opsonization, rapid clearance, low detection sensitivity and cytotoxic effects on the host. Encapsulation of fluorophores in liposomes by film hydration and extrusion leads to fluorescence quenching and protection which enables in vivo imaging with high detection sensitivity.

Optical imaging offers a wide range of diagnostic modalities and has attracted a lot of interest as a tool for biomedical imaging. Despite the enormous ...

A Guide to Structured Illumination TIRF Microscopy at High Speed with Multiple Colors

Optical super-resolution imaging with structured illumination microscopy (SIM) is a key technology for the visualization of processes at the molecular level in the chemical and biomedical sciences. Although commercial SIM systems are available, systems that are custom designed in the laboratory can outperform commercial systems, the latter typically designed for ease of use and general purpose applications, both in terms of imaging fidelity and speed. This article presents an in-depth guide to building a SIM system that uses total internal reflection (TIR) illumination and is capable of imaging at up to 10 Hz in three colors at a resolution reaching 100 nm. Due to the combination of SIM and TIRF, the system provides better image contrast than rival technologies. To achieve these specifications, several optical elements are used to enable automated control over the polarization state and spatial structure of the illumination light for all available excitation wavelengths. Full details on hardware implementation and control are given to achieve synchronization between excitation light pattern generation, wavelength, polarization state, and camera control with an emphasis on achieving maximum acquisition frame rate. A step-by-step protocol for system alignment and calibration is presented and the achievable resolution improvement is validated on ideal test samples. The capability for video-rate super-resolution imaging is demonstrated with living cells.

This article provides an in depth guide for the assembly and operation of a structured illumination microscope operating with total internal reflection fluorescence illumination (TIRF-SIM) to image dynamic biological processes with optical super-resolution in multiple colors.

Optical super-resolution imaging with structured illumination microscopy (SIM) is a key technology for the visualization of processes at the molecular ...

Correlative Light- and Electron Microscopy Using Quantum Dot Nanoparticles

A method is described whereby quantum dot (QD) nanoparticles can be used for correlative immunocytochemical studies of human pathology tissue using widefield fluorescence light microscopy and transmission electron microscopy (TEM). To demonstrate the protocol we have immunolabeled ultrathin epoxy sections of human somatostatinoma tumor using a primary antibody to somatostatin, followed by a biotinylated secondary antibody and visualization with streptavidin conjugated 585 nm cadmium-selenium (CdSe) quantum dots (QDs). The sections are mounted on a TEM specimen grid then placed on a glass slide for observation by widefield fluorescence light microscopy. Light microscopy reveals 585 nm QD labeling as bright orange fluorescence forming a granular pattern within the tumor cell cytoplasm. At low to mid-range magnification by light microscopy the labeling pattern can be easily recognized and the level of non-specific or background labeling assessed. This is a critical step for subsequent interpretation of the immunolabeling pattern by TEM and evaluation of the morphological context. The same section is then blotted dry and viewed by TEM. QD probes are seen to be attached to amorphous material contained in individual secretory granules. Images are acquired from the same region of interest (ROI) seen by light microscopy for correlative analysis. Corresponding images from each modality may then be blended to overlay fluorescence data on TEM ultrastructure of the corresponding region.

A method is described whereby quantum dot (QD) nanoparticles can be used for correlative immunocytochemical studies of epoxy embedded human pathology tissue. We employ commercial antibody fragment conjugated QDs that are visualized by widefield fluorescence light microscopy and transmission electron microscopy.

A method is described whereby quantum dot (QD) nanoparticles can be used for correlative immunocytochemical studies of human pathology tissue using ...

Correlative Light Electron Microscopy (CLEM) for Tracking and Imaging Viral Protein Associated Structures in Cryo-immobilized Cells

Due to its high resolution, electron microscopy (EM) is an indispensable tool for virologists. However, one of the main difficulties when analyzing virus-infected or transfected cells via EM are the low efficiencies of infection or transfection, hindering the examination of these cells. In order to overcome this difficulty, light microscopy (LM) can be performed first to allocate the subpopulation of infected or transfected cells. Thus, taking advantage of the use of fluorescent proteins (FPs) fused to viral proteins, LM is used here to record the positions of the "positive-transfected" cells, expressing a FP and growing on a support with an alphanumeric pattern. Subsequently, cells are further processed for EM via high pressure freezing (HPF), freeze substitution (FS) and resin embedding. The ultra-rapid freezing step ensures excellent membrane preservation of the selected cells that can then be analyzed at the ultrastructural level by transmission electron microscopy (TEM). Here, a step-by-step correlative light electron microscopy (CLEM) workflow is provided, describing sample preparation, imaging and correlation in detail. The experimental design can be also applied to address many cell biology questions.

Correlative Light Electron Microscopy CLEM for Tracking and Imaging Viral Protein Associated Structures in Cryo-immobilized Cells

A correlative light electron microscopy (CLEM) method is applied to image virus-induced intracellular structures via electron microscopy (EM) in cells that are previously selected by light microscopy (LM). LM and EM are combined as a hybrid imaging approach to achieve an integrated view of virus-host interactions.

Due to its high resolution, electron microscopy (EM) is an indispensable tool for virologists. However, one of the main difficulties when analyzing ...

Gene Expression Analysis of Endothelial Cells Exposed to Shear Stress Using Multiple Parallel-plate Flow Chambers

We describe a workflow for the analysis of gene expression from endothelial cells subject to a steady laminar flow using multiple monitored parallel-plate flow chambers. Endothelial cells form the inner cellular lining of blood vessels and are chronically exposed to the frictional force of blood flow called shear stress. Under physiological conditions, endothelial cells function in the presence of various shear stress conditions. Thus, the application of shear stress conditions in in vitro models can provide greater insight into endothelial responses in vivo. The parallel-plate flow chamber previously published by Lane et al.9 is adapted to study endothelial gene regulation in the presence and absence of steady (non-pulsatile) laminar flow. Key adaptations in the set-up for laminar flow as presented here include a large, dedicated environment to house concurrent flow circuits, the monitoring of flow rates in real-time, and the inclusion of an exogenous reference RNA for the normalization of quantitative real-time PCR data. To assess multiple treatments/conditions with the application of shear stress, multiple flow circuits and pumps are used simultaneously within the same heated and humidified incubator. The flow rate of each flow circuit is measured continuously in real-time to standardize shear stress conditions throughout the experiments. Because these experiments have multiple conditions, we also use an exogenous reference RNA that is spiked-in at the time of RNA extraction for the normalization of RNA extraction and first-strand cDNA synthesis efficiencies. These steps minimize the variability between samples. This strategy is employed in our pipeline for the gene expression analysis with shear stress experiments using the parallel-plate flow chamber, but parts of this strategy, such as the exogenous reference RNA spike-in, can easily and cost-effectively be used for other applications.

Here, a workflow for the culture and gene expression analysis of endothelial cells under fluid shear stress is presented. Included is a physical arrangement for simultaneously housing and monitoring multiple flow chambers in a controlled environment and the use of an exogenous reference RNA for quantitative PCR.

We describe a workflow for the analysis of gene expression from endothelial cells subject to a steady laminar flow using multiple monitored parallel-plate ...

A Guide to Build a Highly Inclined Swept Tile Microscope for Extended Field-of-view Single-molecule Imaging

Single-molecule imaging has greatly advanced our understanding of molecular mechanisms in biological studies. However, it has been challenging to obtain large field-of-view, high-contrast images in thick cells and tissues. Here, we introduce highly inclined swept tile (HIST) microscopy that overcomes this problem. A pair of cylindrical lenses was implemented to generate an elongated excitation beam that was scanned over a large imaging area via a fast galvo mirror. A 4f configuration was used to position optical components. A scientific complementary metal-oxide semiconductor camera detected the fluorescence signal and blocked the out-of-focus background with a dynamic confocal slit synchronized with the beam sweeping. We present a step-by-step instruction on building the HIST microscope with all basic components.

A detailed instruction is described on how to build a highly inclined swept tile (HIST) microscope and its usage for single-molecule imaging.

Single-molecule imaging has greatly advanced our understanding of molecular mechanisms in biological studies. However, it has been challenging to obtain ...

Application of Atomic Force Microscopy to Detect Early Osteoarthritis

Biomechanical properties of cells and tissues not only regulate their shape and function but are also crucial for maintaining their vitality. Changes in elasticity can propagate or trigger the onset of major diseases like cancer or osteoarthritis (OA). Atomic force microscopy (AFM) has emerged as a strong tool to qualitatively and quantitatively characterize the biomechanical properties of specific biological target structures on a microscopic scale, measuring forces in a range from as small as the piconewton to the micronewton. Biomechanical properties are of special importance in musculoskeletal tissues, which are subjected to high levels of strain. OA as a degenerative disease of the cartilage results in the disruption of the pericellular matrix (PCM) and the spatial rearrangement of the chondrocytes embedded in their extracellular matrix (ECM). Disruption in PCM and ECM has been associated with changes in the biomechanical properties of cartilage. In the present study we used AFM to quantify these changes in relation to the specific spatial pattern changes of the chondrocytes. With each pattern change, significant changes in elasticity were observed for both the PCM and ECM. Measuring the local elasticity thus allows for drawing direct conclusions about the degree of local tissue degeneration in OA.

We present a method to investigate early osteoarthritic changes at the cellular level in articular cartilage by using atomic force microscopy (AFM).

Biomechanical properties of cells and tissues not only regulate their shape and function but are also crucial for maintaining their vitality. Changes in ...

Single-Cell Optical Action Potential Measurement in Human Induced Pluripotent Stem Cell-Derived Cardiomyocytes

Conventional intracellular microelectrode techniques to quantify cardiomyocyte electrophysiology are extremely complex, labor intensive, and typically carried out in low throughput. Rapid and ongoing expansion of induced pluripotent stem cell (iPSC) technology presents a new standard in cardiovascular research and alternate methods are now necessary to increase throughput of electrophysiological data at a single cell level. VF2.1Cl is a recently derived voltage sensitive dye which provides a rapid single channel, high magnitude response to fluctuations in membrane potential. It possesses kinetics superior to those of other existing voltage indicators and makes available functional data equivalent to that of traditional microelectrode techniques. Here, we demonstrate simplified, non-invasive action potential characterization in externally paced human iPSC derived cardiomyocytes using a modular and highly affordable photometry system.

Here we describe optical acquisition and characterization of action potentials from induced pluripotent stem cell derived cardiomyocytes using a high-speed modular photometry system.

Conventional intracellular microelectrode techniques to quantify cardiomyocyte electrophysiology are extremely complex, labor intensive, and typically ...