We would like to acknowledge and thank Tom Goddard at University of California San Francisco for his endless help with Chimera, Joel Mancuso and Chris Booth at Gatan, Inc. for their help with SBF-SEM data collection of bacteria dataset, Doug Wei at Zeiss, Inc. for his help with the FIB-SEM data collection of epithelial cell dataset, Kent McDonald at University of California Berkeley Electron Microscopy Lab for advice on sample preparation, TEM imaging and tomography, Roseann Csencsits at Lawrence Berkeley National Laboratory for her help taking the cryo-TEM image, Elena Bosneaga for cryo-sectioning of the plant dataset, Jocelyn Krey at Oregon Health and Science University for the dissection of utricle tissue, David Skinner at National Energy Research Scientific Computing Center (NERSC) and Jitendra Malik at University of California Berkeley for their advice in software infrastructure, and Pablo Arbelaez at&#160;University of California Berkeley&#160;for his codes contributions to the custom-tailored script presented in this article. 
Research was supported by the U.S. Department of Energy, Office of Science under contract No. DE-AC02-05CH11231 [David Skinner], as well as U.S. National Institutes of Health (NIH) grant No. P01 GM051487 [M.A.] for the inner ear hair cell project and microscopy instrumentation use.

1. Manual Abstracted Model Generation
Note: The details of the methodology described below are specific to Chimera, but other software packages may be used instead. Use this approach when the sole objective is to create a geometrical model (e.g., a ball and stick model) in order to make geometric measurements, rather than displaying the volume shape of the objects.
<ol>
	<li>Import the data volume into a suitable program for manual abstracted model generation.
	<ol>
		<li>Select File &#62; Open Map to pull up the Open File dialog. Navigate to the file location of the desired map.</li>
		<li>Pull up the Volume Viewer (Tools &#62; Volume Data &#62; Volume Viewer) and select Features &#62; Display Style to display data with different rendering styles.</li>
		<li>Adjust the threshold for the display by dragging the vertical bar on the histogram in the Volume Viewer window.</li>
	</ol>
	</li>
	<li>Navigate through the 3D volume (e.g., slice by slice) to select an area of interest for segmentation and crop out a smaller sub-volume if necessary.
	<ol>
		<li>In the Volume Viewer dialog, click Axis, then select X, Y, or Z.</li>
		<li>In the Volume Viewer dialog, select Features &#62; Planes. Click One to set Depth to display the plane corresponding to the number in the left box, and click All to display all planes.</li>
		<li>In the Volume Viewer dialog, select Features &#62; Subregion selection.
		<ol>
			<li>Click and drag to create a rectangular box around the region of interest.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Place markers along the feature of interest and connect them with linkers where appropriate (often done automatically by the program) until the model is complete.
	<ol>
		<li>From the Volume Viewer menu bar, select Tools &#62; Volume Tracer dialog to open the Volume Tracer dialog. In the Volume Tracer dialog, select File &#62; New Marker Set.</li>
		<li>In the Volume Tracer dialog, check Mouse &#62; Place markers on high quality, Place markers on data planes, Move and resize markers, Link new marker to selected marker, and Link consecutively selected markers.</li>
		<li>Click on the Marker Color swatch, and select a color. Repeat this step for Link color.</li>
		<li>Enter radii for the marker and link model-building elements.</li>
		<li>In the Volume Tracer Window, select Place markers using [right] mouse button, and insert radii for markers and links.</li>
		<li>Right click on the volume data to begin laying down markers. Markers will be connected automatically.</li>
		<li>In the Volume Tracer dialog, select File &#62; Save current marker set, then File &#62; Close marker set.</li>
	</ol>
	</li>
	<li>Open a new marker set (Step 1.3.1) to begin building a model into a second desired feature of interest. Utilize contrasting colors between marker sets to emphasize differences in features.</li>
</ol>
2. Manual Tracing of Features of Interest
Note: The details of the methodology described below are specific to Amira, but other software packages may be used instead. Use this approach when the population density is relatively small and when accuracy of feature extraction is paramount, as manual tracing is a time-consuming approach.
<ol>
	<li>Import volume data into a program with manual tracing options. Software with this capability generally offer at least a basic paintbrush tool.
	<ol>
		<li>For large volumes or tomograms (e.g., 16-bit 2,048 x 2,048 or larger .rec or .mrc tomograms generated in IMOD): Select Open Data &#62; Right click on filename.rec &#62; Format&#8230; &#62;Select Raw as LargeDiskData &#62; Ok &#62; Load. Select appropriate Raw Data Parameters from header information &#62; Ok. Toggle and Save As a new <a href="http://filename.am/">filename.am</a> file for use in the following steps.</li>
		<li>For smaller 3D image stack files (e.g., 3D .tif or .mrc or .rec): Open Data &#62; Select filename.tif or filename.mrc. Toggle and Right Click &#62; Save As <a href="http://filename.am/">filename.am</a>. If an error is generated or the program is unresponsive, the file may be too large and can be opened by following Step 2.1.1.</li>
	</ol>
	</li>
	<li>Navigate through the slices to select a 3D sub-volume for segmentation, and then crop to this area of interest.
	<ol>
		<li>In 3D Viewer window, select Orthoslice to open image file. Use slider at bottom to navigate through slices.</li>
		<li>To crop larger data opened as LargeDiskData, toggle file name in Pool window &#62; Right click &#62; LatticeAccess. Enter desired box size &#62; Apply. Save new file.</li>
	</ol>
	</li>
	<li>Create segmentation file.
	<ol>
		<li>Toggle the file in the Pool window &#62; Right Click &#62; Labeling &#62; LabelField. A new file will be created and automatically loaded in the Segmentation Editor tab.</li>
	</ol>
	</li>
	<li>Trace the border of the first feature of interest, then fill the trace by hand or by using a command specific to the software used. Follow the feature of interest through all slices and repeat the manual tracing segmentation. Use the following commands when using Amira:
	<ol>
		<li>To use the Paintbrush tool, alter brush size as desired, then use the mouse pointer to trace the border of the feature of interest.</li>
		<li>Fill the traced area with shortcut &#8220;f&#8221;. Add the selection by clicking the button with the plus symbol, or the shortcut &#8220;a&#8221;. If necessary, press &#8220;u&#8221; to undo, and &#8220;s&#8221; to subtract or erase.</li>
	</ol>
	</li>
	<li>Generate a surface rendering for visualization and basic qualitative or quantitative analysis per software user guide instruction.
	<ol>
		<li>In the Object Pool tab, toggle the filename-labels.am in the Pool window &#62; Right click &#62; SurfaceGen.</li>
		<li>Select desired Surface properties &#62; Apply. A new file filename.surf will be created in the Pool.</li>
		<li>To visualize the segmented volume, toggle filename.surf in Pool window &#62; Right click &#62; SurfaceView.</li>
		<li>Use the tools in the 3DViewer window to move, rotate, and zoom in the 3D volume.</li>
	</ol>
	</li>
	<li>Extract the exact densities and determine measurements such as volume or surface area. Export to other programs for more advanced display, analysis and simulation.
	<ol>
		<li>On 3DViewer window, click Measure tool &#62; Select appropriate option (2D length and 2D angle for measurements on a single 2D plane, 3D length and 3D Angle for measurements on a 3D volume).</li>
		<li>Click on mesh surface to measure desired length, distance, and angles. The values will be listed in the Properties window.</li>
	</ol>
	</li>
</ol>
3. Automated Density-based Segmentation
Note: The details of the methodology described below are specific to Amira, but other software packages may be used instead.
<ol>
	<li>Use this approach on data sets with any variety of contrast, crispness, or crowdedness to withdraw the densities of interest.</li>
	<li>Import volume data into a program equipped with thresholding, magic wand, or other density-based tools for automatic segmentation. Follow steps outlined in 2.1-2.1.2 in the directions for manual tracing.</li>
	<li>Navigate through slices and select area for segmentation. If necessary, crop out a smaller 3D sub-volume for segmentation. Follow steps outlined in 2.2-2.2.2 in the directions for manual tracing.</li>
	<li>Select the density of a feature of interest, usually by clicking or placing a mark or anchor point on the feature. If allowed in the software, enter a number range encompassing the feature&#8217;s pixel intensity and adjust this tolerance as desired. Densities belonging to the feature will be picked up in accordance to the intensity of the anchor&#8217;s pixel or tolerance value. Use the following commands when using Amira.
	<ol>
		<li>Use the Magic Wand Tool for features with distinguishable margins.
		<ol>
			<li>Click on the area of interest, then adjust sliders in Display and Masking to capture correct range of values so that the feature is fully highlighted. Add selection with shortcut &#8220;a&#8221;.</li>
		</ol>
		</li>
		<li>Use the Threshold Tool for features without clearly distinguishable margins.</li>
		<li>Select the Threshold icon. Adjust slider to adjust density within desirable range so that only the features of interest are masked. Click Select button, then add selection with shortcut &#8220;a&#8221;.</li>
		<li>To segment entire volume, select All slices before adding selection.</li>
		<li>To remove noise, select Segmentation &#62; Remove Islands and/or Segmentation &#62; Smooth labels.</li>
	</ol>
	</li>
	<li>Generate a surface for visualization and qualitative analysis as described in the manual tracing section 2.6-2.6.2. If desired, export to other programs for adequate 3D display, quantitative analysis and simulations.</li>
</ol>
4. Custom-tailored Automated Segmentation
Note: Use this approach to create customized scripts for automatic segmentation, which requires background experience in computer science, but allows the ability to create a precise density model from a large volume.
<ol>
	<li>Tools (Specific Example of Shape-Supervised Segmentation in MATLAB27)
	<ol>
		<li>Image pre-processing: Perform de-noising, background removal and image enhancement by using the following pipeline:
		<ol>
			<li>Load the image using the imread command.
			<ol>
				<li>In the command line, enter: &#62;&#62; im = imread($image_path), where $image_path is the location of the image to be analyzed.</li>
			</ol>
			</li>
			<li>From the Image Processing toolbox, call Wiener Filter using an estimated or known Noise-power-to-signal ratio (NSR).</li>
			<li>On the previously processed image, call the image opening function imopen to estimate the background layer, then allocate the outcome as a different mask.
			<ol>
				<li>In the command line, enter: &#62;&#62; background = imopen(im,strel($shape_string,$size)), in this method, $shape_string is equal to &#8216;disk&#8217; the variable $size is given by the analyzer. i.e. &#62;&#62; background = imopen(im,strel(&#8216;disk&#8217;,15)).</li>
			</ol>
			</li>
			<li>Subtract the filtered image with the background.
			<ol>
				<li>In the command line, enter: &#62;&#62; im2 = im - background</li>
			</ol>
			</li>
			<li>Depending on the quality of the results, perform image normalization with or without adaptive Otsu&#8217;s method28, which can be called using the function imadjust from the Image Processing Toolbox.
			<ol>
				<li>In the command line, enter: &#62;&#62; im3 = imadjust(im2)</li>
			</ol>
			</li>
			<li>Prepare the features of interest for segmentation, limiting the regions of interest by cropping the normalized image.
			<ol>
				<li>Using the imtool command, explore the region of interest that is to be cropped and provide the coordinates to the command: &#62;&#62; im3_crop = imcrop(im3, [x1 y1 x2 y2]), where the vector [x1 y1 x2 y2] corresponds to the square region to be cropped.</li>
			</ol>
			</li>
		</ol>
		</li>
		<li>Shape recognition/Supervised shape classification: Train the algorithm by providing specific examples for each different category of objects (linear traces in a 2D image across the features of interest).
		<ol>
			<li>Check that VLFEAT29 API is successfully installed and visit VLFEAT&#8217;s website for more in-depth documentation.</li>
			<li>In the command line, enter: &#62;&#62; [TREE,ASGN] = VL_HIKMEANS(im3_crop,$K,$NLEAVES) where $K is the number of cluster to be used or the number of classes the observer wants to arrange the data into, and $NLEAVES is the desired number of leaf clusters i.e. &#62;&#62; [TREE,ASGN] = VL_HIKMEANS(im3_crop,4,100)</li>
			<li>Use manually segmented features as the input for VLFeat. 
			NOTE: This open source C-based library will perform pixel patching, patch clustering, and cluster center positioning depending on the type of method chosen to work best for the datasets. The available options range from k-mean clustering to texton-based approaches30, and the output is a numerical array that describes the features desired based on the given exemplars.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Segmentation: Use this fully automated, although computationally expensive, approach to segment multiple classes of objects simultaneously, which will be written out as separate maps for further visualization and analysis.
	<ol>
		<li>Load the previously generated numeral array (model).</li>
		<li>Call the support vector machine (SVM) function in VLFeat, using the model and the image to be segmented as an input.
		<ol>
			<li>In the command line, enter: &#62;&#62; [w, b] = vl_svmtrain(x, y, 0.1), where x is the original cropped image im2_crop and y is the objective image, the image that has been manually segmented. Use &#62;&#62; ISEG = VL_IMSEG(I,LABELS) to color the results according to the labels generated by the clustering. 
			NOTE: Based on the characteristics of the model, VLFeat will classify the image on the number of classes (features of interest) assigned from the beginning. Depending on the grade of accuracy desired, it is possible to combine this method with other approaches or estimate cluster parameters such as hull and cluster centers. The output of the SVM algorithm is a probabilistic model and multiple binary masks of the desired classes in the new datasets.</li>
		</ol>
		</li>
		<li>Save results by entering the command: &#62;&#62; imwrite(im, $format, $filename) where $format is &#39;tiff&#39; and $filename is the path for the output file.</li>
		<li>For visualizing images, enter the command: &#62;&#62; imshow(im).</li>
	</ol>
	</li>
</ol>

<ol>
	<li>Auer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+electron+cryo-microscopy+as+a+powerful+structural+tool+in+molecular+medicine.">Three-dimensional electron cryo-microscopy as a powerful structural tool in molecular medicine.</a> J Mol Med (Berl). 78 (4), 191-202 (2000).</li><li>Johnson, M. C., Rudolph, F., Dreaden, T. M., Zhao, G., Barry, B. A., Schmidt-Krey, I. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Assessing+two-dimensional+crystallization+trials+of+small+membrane+proteins+for+structural+biology+studies+by+electron+crystallography.">Assessing two-dimensional crystallization trials of small membrane proteins for structural biology studies by electron crystallography.</a> Journal of visualized experiments JoVE. (44), e1846 (2010).</li><li>Jun, S., Zhao, G., Ning, J., Gibson, G. A., Watkins, S. C., Zhang, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Correlative+microscopy+for+3D+structural+analysis+of+dynamic+interactions.">Correlative microscopy for 3D structural analysis of dynamic interactions.</a> Journal of visualized experiments JoVE. (76), e50386 (2013).</li><li>Meng, X., Zhao, G., Zhang, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structure+of+HIV-1+capsid+assemblies+by+cryo-electron+microscopy+and+iterative+helical+real-space+reconstruction.">Structure of HIV-1 capsid assemblies by cryo-electron microscopy and iterative helical real-space reconstruction.</a> Journal of visualized experiments JoVE. (54), e3041 (2011).</li><li>Chen, S., McDowall, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Electron+Cryotomography+of+Bacterial+Cells.">Electron Cryotomography of Bacterial Cells.</a> Journal of visualized experiments JoVE. (39), e1943 (2010).</li><li>Meyerson, J. R., White, T. A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Determination+of+molecular+structures+of+HIV+envelope+glycoproteins+using+cryo-electron+tomography+and+automated+sub-tomogram+averaging.">Determination of molecular structures of HIV envelope glycoproteins using cryo-electron tomography and automated sub-tomogram averaging.</a> Journal of visualized experiments JoVE. (58), e2770 (2011).</li><li>Lucic, V., Forster, F., Baumeister, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structural+studies+by+electron+tomography:+from+cells+to+molecules.">Structural studies by electron tomography: from cells to molecules.</a> Annu Rev Biochem. 74, 833-865 (2005).</li><li>Bajaj, C., Yu, Z., Auer, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Volumetric+feature+extraction+and+visualization+of+tomographic+molecular+imaging.">Volumetric feature extraction and visualization of tomographic molecular imaging.</a> J Struct Biol. 144 (1-2), 132-143 (2003).</li><li>Lin, G., Adiga, U., Olson, K., Guzowski, J. F., Barnes, C. A., Roysam, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+hybrid+3D+watershed+algorithm+incorporating+gradient+cues+and+object+models+for+automatic+segmentation+of+nuclei+in+confocal+image+stacks.">A hybrid 3D watershed algorithm incorporating gradient cues and object models for automatic segmentation of nuclei in confocal image stacks.</a> Cytometry A. 56 (1), 23-36 (2003).</li><li>Volkmann, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+novel+three-dimensional+variant+of+the+watershed+transform+for+segmentation+of+electron+density+maps.">A novel three-dimensional variant of the watershed transform for segmentation of electron density maps.</a> Journal of Structural Biology. 138 (1), 123-129 (2002).</li><li>Rigort, A., G&#252;nther, D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Automated+segmentation+of+electron+tomograms+for+a+quantitative+description+of+actin+filament+networks.">Automated segmentation of electron tomograms for a quantitative description of actin filament networks.</a> Journal of structural biology. 177 (1), 135-144 (2012).</li><li>Cremers, D., Rousson, M., Deriche, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Review+of+Statistical+Approaches+to+Level+Set+Segmentation.">A Review of Statistical Approaches to Level Set Segmentation.</a> Integrating Color, Texture, Motion and Shape. International Journal of Computer Vision. 72 (2), 195-215 (2007).</li><li>Lin, Z., Davis, L. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Shape-based+human+detection+and+segmentation+via+hierarchical+part-template+matching.">Shape-based human detection and segmentation via hierarchical part-template matching.</a> IEEE Trans Pattern Anal Mach Intell. 32 (4), 604-618 (2010).</li><li>Pettersen, E. F., Goddard, T. D., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=UCSF+Chimera--a+visualization+system+for+exploratory+research+and+analysis.">UCSF Chimera--a visualization system for exploratory research and analysis.</a> J Comput Chem. 25 (13), 1605-1612 (2004).</li><li>Kremer, J. R., Mastronarde, D. N., McIntosh, J. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Computer+visualization+of+three-dimensional+image+data+using+IMOD.">Computer visualization of three-dimensional image data using IMOD.</a> J Struct Biol. 116 (1), 71-76 (1996).</li><li>Zhang, Q., Bettadapura, R., Bajaj, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Macromolecular+structure+modeling+from+3D+EM+using+VolRover+2.0.">Macromolecular structure modeling from 3D EM using VolRover 2.0.</a> Biopolymers. 97 (9), 709-731 (2012).</li><li>Giannuzzi, L. A., Stevie, F. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+review+of+focused+ion+beam+milling+techniques+for+TEM+specimen+preparation.">A review of focused ion beam milling techniques for TEM specimen preparation.</a> Micron. 30 (3), 197-204 (1999).</li><li>Heymann, J. A. W., Hayles, M., Gestmann, I., Giannuzzi, L. A., Lich, B., Subramaniam, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Site-specific+3D+imaging+of+cells+and+tissues+with+a+dual+beam+microscope.">Site-specific 3D imaging of cells and tissues with a dual beam microscope.</a> Journal of structural biology. 155 (1), 63-73 (2006).</li><li>Knott, G., Rosset, S., Cantoni, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Focussed+ion+beam+milling+and+scanning+electron+microscopy+of+brain+tissue.">Focussed ion beam milling and scanning electron microscopy of brain tissue.</a> Journal of visualized experiments JoVE. (53), e2588 (2011).</li><li>Wirth, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Focused+Ion+Beam+(FIB)+combined+with+SEM+and+TEM:+Advanced+analytical+tools+for+studies+of+chemical+composition,+microstructure+and+crystal+structure+in+geomaterials+on+a+nanometre+scale.">Focused Ion Beam (FIB) combined with SEM and TEM: Advanced analytical tools for studies of chemical composition, microstructure and crystal structure in geomaterials on a nanometre scale.</a> Chemical Geology. 261 (3-4), 217-229 (2009).</li><li>Denk, W., Horstmann, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Serial+block-face+scanning+electron+microscopy+to+reconstruct+three-dimensional+tissue+nanostructure.">Serial block-face scanning electron microscopy to reconstruct three-dimensional tissue nanostructure.</a> PLoS Biol. 2 (11), e329 (2004).</li><li>Frangakis, A. S., Hegerl, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Noise+reduction+in+electron+tomographic+reconstructions+using+nonlinear+anisotropic+diffusion.">Noise reduction in electron tomographic reconstructions using nonlinear anisotropic diffusion.</a> Journal of structural biology. 135 (3), 239-250 (2001).</li><li>Jiang, W., Baker, M. L., Wu, Q., Bajaj, C., Chiu, W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Applications+of+a+bilateral+denoising+filter+in+biological+electron+microscopy.">Applications of a bilateral denoising filter in biological electron microscopy.</a> Journal of Structural Biology. 144 (1), 114-122 (2003).</li><li>Van der Heide, P., Xu, X. P., Marsh, B. J., Hanein, D., Volkmann, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Efficient+automatic+noise+reduction+of+electron+tomographic+reconstructions+based+on+iterative+median+filtering.">Efficient automatic noise reduction of electron tomographic reconstructions based on iterative median filtering.</a> Journal of structural biology. 158 (2), 196-204 (2007).</li><li>Schneider, C. A., Rasband, W. S., Eliceiri, K. W. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=NIH+Image+to+ImageJ:+25+years+of+image+analysis.">NIH Image to ImageJ: 25 years of image analysis.</a> Nat Methods. 9 (7), 671-675 (2012).</li><li>Schindelin, J., Arganda-Carreras, I., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fiji:+an+open-source+platform+for+biological-image+analysis.">Fiji: an open-source platform for biological-image analysis.</a> Nat Methods. 9 (7), 676-682 (2012).</li><li>MathWorks. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> MATLAB. , (2012).</li><li>Otsu, N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+Threshold+Selection+Method+from+Gray-Level+Histograms.">A Threshold Selection Method from Gray-Level Histograms.</a> Systems, Man and Cybernetics, IEEE Transactions on. 9, 62-66 (1979).</li><li>Vedaldi, A., Fulkerson, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> VLFeat: An Open and Portable Library of Computer Vision Algorithms. , (2008).</li><li>Zhu, S. C., Guo, C., Wang, Y., Xu, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=What+are+Textons?.">What are Textons?.</a> International Journal of Computer Vision. 62 (1-2), 121-143 (2005).</li><li>Gagnon, L. H., Longo-Guess, C. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+chloride+intracellular+channel+protein+CLIC5+is+expressed+at+high+levels+in+hair+cell+stereocilia+and+is+essential+for+normal+inner+ear+function.">The chloride intracellular channel protein CLIC5 is expressed at high levels in hair cell stereocilia and is essential for normal inner ear function.</a> The Journal of neuroscience the official journal of the Society for Neuroscience. 26 (40), 10188-10198 (2006).</li><li>Briand, P., Petersen, O. W., Van Deurs, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+new+diploid+nontumorigenic+human+breast+epithelial+cell+line+isolated+and+propagated+in+chemically+defined+medium.">A new diploid nontumorigenic human breast epithelial cell line isolated and propagated in chemically defined medium.</a> In vitro cellular &amp; developmental biology journal of the Tissue Culture Association. 23 (3), 181-188 (1987).</li><li>Petersen, O. W., R&#248;nnov-Jessen, L., Howlett, A. R., Bissell, M. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interaction+with+basement+membrane+serves+to+rapidly+distinguish+growth+and+differentiation+pattern+of+normal+and+malignant+human+breast+epithelial+cells.">Interaction with basement membrane serves to rapidly distinguish growth and differentiation pattern of normal and malignant human breast epithelial cells.</a> Proceedings of the National Academy of Sciences of the United States of America. 89 (19), 9064-9068 (1992).</li><li>Shin, J. B., Krey, J. F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Molecular+architecture+of+the+chick+vestibular+hair+bundle.">Molecular architecture of the chick vestibular hair bundle.</a> Nature neuroscience. 16 (3), 365-374 (2013).</li><li>Yang, W., Zeng, Z., Max, N., Auer, M., Crivelli, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Simplified+Surface+Models+of+Tubular+Bacteria+and+Cytoskeleta.">Simplified Surface Models of Tubular Bacteria and Cytoskeleta.</a> Journal of Information &amp; Computational Science. 9 (6), 1589-1598 (2012).</li></ol>

The authors declare they have no competing financial interests.

Effective strategies for the extraction of relevant features from 3D EM volumes are urgently needed in order to keep up with the data tsunami that has recently hit biological imaging. While data can be generated in hours or days, it takes many months to analyze the 3D volumes in depth. Therefore, it is clear that the image analysis has become the bottleneck for scientific discoveries; without adequate solutions for these problems, imaging scientists become the victims of their own success. This is in part due to the high complexity of the data and also the macromolecular crowding typically found in biological cells, where proteins and protein complexes border one another and essentially appear as a continuous gradient of grayscale densities. The problem is complicated by sample preparation and imaging imperfections, and in some cases image reconstruction artifacts, leading to less than perfect volumetric data that can pose challenges for fully automated approaches. Most significant, however, is the fact that the experts in sample preparation, imaging, and the biological interpretation are seldom well versed in computational science, and hence require guidance on how to effectively approach feature extraction and analysis. Therefore, through the use of various examples, the protocol explains how to prepare data for segmentation, as well as the steps for manual abstracted model generation, automated density-based segmentation, manual tracing of features of interest, and custom-tailored automated segmentation. The manual and automatic approaches outlined in the procedure can be found in a large variety of segmentation software, some of which are mentioned here, but others perform similar functions and are equally well suited.
The results demonstrate that the effectiveness of each of the 3D segmentation approaches varies for each different type of data sets. Even though the different approaches produced qualitatively similar 3D renderings as the end product, the amount of time and effort spent on each during the segmentation process varied significantly. The recommendations for appropriate image characteristics and personal aims per segmentation approach are summarized in Figure 5, which is further explained in the following four subsections. These criteria were applied to the six datasets, as shown in the decision flow chart of Figure 6. Although Figures 5 and 6 are merely meant to provide a rationale for each data set and how each of the criteria were weighted in the decision making process, they do not provide a foolproof guidance, but rather a starting point. There are simply too many criteria that influence the decision-making process: some are objective criteria, such as data set characteristics, whereas others are more subjective criteria, such as the desired objective. It is safe to say that data sets that display a high level of contrast with sharp crisp boundaries, have features that are well separated and relatively homogeneous (not too diverse), and are processed with the objective of displaying a density model for a large number of objects, automated approaches will be superior, if not for the fact that manual approaches would simply be resource (time)-prohibitive. On the other hand, if contrast is low, the data is fuzzy and thus requires an expert&#8217;s knowledge, the objects are crowded, and the features show a high diversity and are thus heterogeneous, one may not have any other choice than manual feature extraction/segmentation.

Manual Abstracted Model Generation
Manual abstracted model tracing is particularly effective in segmenting linear elements, providing seeds points (balls) that can be automatically connected (sticks). Such balls and sticks-models can be very powerful to measure length and orientation of such model and provide an adequately abstracted model for both qualitative inspection and quantitative analysis. Manual abstracted model generation is commonly used when minimizing resources spent on the analysis is more important than absolute fidelity to the shapes of the original data. It is most successful with linear and homogenous features of interest (e.g., filaments, tubes). Data contrast, crispness, and crowdedness do not play a major role in determining this method&#8217;s success, as long as the human eye can recognize the object of interest. Sometimes such models can also be utilized as a skeleton to segment the 3D map in a zone around the skeleton. Although the model is abstract rather than a reflection of exact densities, it represents a skeletonized version of the 3D density and thus allows for clutter-free visualization and qualitative analysis. Quantitative measurements such as length can also be determined from the approximate model. For an example of software with manual abstracted model generation, please visit Chimera&#8217;s detailed user guide online at <a href="http://www.cgl.ucsf.edu/chimera/current/docs/UsersGuide/index.html">http://www.cgl.ucsf.edu/chimera/current/docs/UsersGuide/index.html</a>.

Manual Tracing of Features of Interest
Manual paintbrush tracing works well with almost all data characteristics, but it is also the most time consuming method. At times, it is the only technique for extracting a feature of interest from a complex image set containing a large variety of features, such as the thin and convoluted cell membrane. One useful tool available in some programs allows for interpolation between intermittently segmented slices when the feature of interest changes smoothly. Manual tracing can be applied most efficiently if the data is crisp and has medium to high contrast, but it can also be utilized for more challenging data sets, as long as the user is familiar with the object of interest. The data complexity can range from discrete objects to complex and crowded data sets, where objects are closely packed. In the latter case, manual segmentation may be the only choice, as automatic approaches often struggle to segment the desired volume and extract too much or too little. Difficult feature morphologies, such as convoluted sheets or volumes, can also be extracted by this method. However, the user should keep in mind that a dataset with several difficult characteristics can only be segmented if the population density of the features of interest is low, as segmentation of high population densities of the features of interest becomes time-prohibitive. For an example of software with manual tracing, please visit Amira&#8217;s detailed user guide online at <a href="http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf">http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf</a>.

Automated Density-based Segmentation
In contrast to the manual techniques, the automated approaches are generally less time-consuming, which is an important factor to consider when segmenting a large stack of images. However, simple thresholding may not be as accurate, and much more time may be spent on refinement and curation of the automatically segmented volume. Automated density-based segmentation works best on data sets that display a large number of similar features of interest that all require segmentation. If the data is more complex, these automated techniques can still serve as an initial step, but will likely require some manual intervention down the line in order to specify a subvolume containing the feature of interest. This strategy typically works well on linear morphologies or convoluted volumes, but it is rarely successful with thin convoluted sheets such as cell membranes. Minimal user intervention with automated approaches enables segmentation through large or small volumes, while expending few user resources such as time in return for high fidelity. For an example of software with automated density-based segmentation, please visit Amira&#8217;s detailed user guide online at <a href="http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf">http://www.vsg3d.com/sites/default/files/Amira_Users_Guide.pdf</a>.

Custom-Tailored Automated Segmentation
Custom-tailored automated segmentation allows the power customization of algorithms for a specific data set, but it is often specific to the data set or data type, appropriate for a limited number of feature characteristics, and cannot be generalized easily. The procedure showcased here differs from the general automated segmentation approaches, such as watershed immersion and other level set methods, which rely on a programmed determination of critical seed points, followed by fast-marching cube expansion from these seed points. A variation on this theme is boundary segmentation, where gradient vector information informs feature boundaries. In contrast, the customized script used here relies on a training stage where the user manually traces a few examples. Through machine learning, specific algorithms will detect and then learn to independently recognize properties and data characteristics consistently found in the traces. An expert user can retrain the algorithms and improve the accuracy of segmentation by including more example traces to provide a larger set of feature criteria. Overall, thresholding and related approaches, or even custom-tailored approaches may not be as useful to extract a single feature of interest from an image with complex diversity of organelles or shapes, as curation may be just as labor intensive as manual tracing.

Strategies for Triaging Data and Choosing a Segmentation Approach 

Given the subjective and objective criteria presented in Figure 4 and summary of suitable datasets in Figure 5, the decision making scheme depicted in Figure 6 can assist an effective assessment of feature extraction strategies for a large variety of data sets. The data sets are triaged in four consecutive decisions, each of which may include any one of the four respective objectives as well as the four subjective criteria introduced in Figure 4. As an example, Figure 6 is the rational for triaging each of the six data sets shown in Figure 3. Undoubtedly, for each data set there is not a single unique path, but rather different paths through this matrix following different criteria for decision-making that may lead to the same or different recommendation for data segmentation. While every data set will have its own set of properties, which cannot be anticipated, six examples are given, each paired with an explanation of the rationale behind the preferred feature extraction/segmentation approach. Most also include a proposition for an alternative decision route that either results in the use of the same or a different segmentation approach (Figure 6).
The kinocilium is a crisp data set with clearly defined boundaries, which makes automated approaches more likely to succeed. All features of interest are well separated, again favoring an automated approach. In addition, the features of interest are similar to one another, making it a relatively homogeneous data set ideal for custom-tailored segmentation. Lastly, the aim was to extract the entire feature, favoring a semi-automated approach. As a consequence, it was concluded that an automated thresholding (solid green line) as well as a custom-designed (e.g., shape supervised segmentation) approach (dotted green line) are both likely to do well on this data set.
Similar criteria, although placed in a different order in the decision making network, apply to the case of bacteria. A custom-tailored approach is recommended in part because this data set was very large; hence, limited resources prohibit a labor-intensive manual intervention/segmentation approach. While thresholding would have yielded acceptable results, the custom-designed approach was able to execute the study&#8217;s key objective to separate the roundish bacterial shapes from the extracellular metal deposits, located either in-between the bacteria or right next to the bacteria, and therefore the custom-tailored approach was preferred.
For stereocilia data sets, the first consideration was the desired objective: the goal can either be to show the entire density or to create geometrical models. The volume of interest was a crowded area, and the objective was to segment a large number of objects as separated objects in order to subsequently execute quantitative volumetric analysis, including lengths, numbers, distances, orientation, etc. It was helpful that the objects of interest were mainly linear, and this made geometrical model tracing the method of choice. However, if instead the objective has been to show the entire density, then the linear feature morphology as well as relatively high contrast with sharply defined boundaries would make an automated thresholding protocol feasible.
The cell membranes and mitochondria data cases are challenging for automated approaches due to their categories of feature morphology: convoluted sheets and volumes, respectively. The goal is to trace the cell or mitochondria outline accurately, but there are only finite resources to do so. In addition, the features of interest are complex and cannot be easily automatically detected or shape-encoded, although for the mitochondria data sets the customized scripting approach taken for the bacteria may possibly be applied with further customization. Fortunately, the membrane and mitochondria themselves only represent a small fraction of the entire volume and hence, manual tracing is a straightforward albeit time-consuming approach. Manual tracing is also the method of choice for such data sets when the contrast is rather low and the boundaries are rather fuzzy. As a result, even if they constitute a significant portion of the data sets, such convoluted sheets must be manually traced, simply due to the lack of a better alternative.
The plant data set posed its own challenges because the goal was to segment all objects, which are densely spaced and make up a crowded scenery. Displaying the density as-is would enable measurements about the shape and organization of the objects, but because manually segmenting each filamentous object is too costly, automatic thresholding was employed instead.
The various steps and corresponding results in creating a 3D model have been displayed here, but more importantly, the data characteristics and personal criteria found to be crucial in determining the best path of segmentation have also been elucidated. The important characteristics of the image data itself include what is described here as contrast, crowdedness, crispness, and the number of different shapes or features (such as organelles, filaments, membranes). Subjective criteria to consider include the desired objective of segmentation (measuring/counting, skeletonized representation of the data/displaying volumes in 3D renderings), morphological characteristics of the feature of interest (linear, elongated, networked, complex, convoluted), the density of features of interest in relation to the entire volume (the fraction of the objects that are important and need to be extracted), and balancing the tradeoffs of expending resources to the segmentation&#8217;s fidelity of the original data and the decreasing return on the investment resulting in incremental improvements for substantially higher allocation of resources.
The field of image segmentation has significantly matured over the recent years, yet there is no silver bullet, no algorithm or program that can do it all. Data set sizes have grown from hundreds of megabytes to routinely tens of gigabytes, and they are now starting to exceed terabytes, making manual segmentation near impossible. Thus, more resources need to be invested in the clever and time-effective feature extraction approaches that mimic the human decision making process. Such efforts will need to be combined with (1) geographic information system (GIS)-based semantic hierarchical data bases (similar to Google Earth), (2) data abstraction techniques (i.e., transitioning from a voxel to geometric/volumetric representation) compatible with computer assisted design (CAD) software in order to significantly reduce the amount of data and thus enabling the display of larger volumes35, (3) simulation techniques, as they are frequently used in the engineering disciplines, as well as (4) advanced animation and movie making capabilities, including fly-through animations (similar to what is developed for the gaming industry).&#160;
Clearly, efficient feature extraction and segmentation lies at the heart of this coming revolution in cellular high-resolution imaging, and while better approaches will always be needed, the principles presented here, as well as the examples of what approach was taken for different data types, will provide some valuable information for making a decision on which approach to take.

Traditionally, the electron microscopy (EM) field has been divided into 1) the structural biology branch using high and super-high resolution TEM, typically combined with implicit or explicit data averaging to investigate the three-dimensional (3D) structure of macromolecular complexes with a defined composition and typically a relatively small size1-4, and 2) the cellular imaging branch in which entire cellular sceneries are visualized1,5,6. While the structural biology branch has undergone a spectacular development over the last four decades, the cell biology branch was mostly restricted to two dimensions, often on less-than-optimally preserved samples. Only with the advent of electron tomography in the last decade has cell biological ultrastructural imaging expanded into the third dimension5,7, where typically averaging cannot be performed as the cellular sceneries, and thus the features of interest, are typically unique.
Although visualized cellular scenes are often stunning to the eye, efficient extraction of the features of interest and subsequent quantitative analysis of such highly complex cellular volumes lag behind, in part because the precise protein composition is usually unknown, therefore making it challenging to interpret these cellular 3D volumes. To this date, extensive biological expertise is often needed in order to interpret complex tomograms, or even to identify the important regions and essential components in the 3D volume. As a further complication, visualization of 3D volumes is remarkably non-trivial. 3D volumes can be thought of and thus visualized as stacks of 2D images. Slice-by-slice inspection of sequential 2D images reduces the complexity, but it also limits feature extraction and thus quantitative analysis to the two dimensions. However, for most 3D objects, the depiction of 3D volumes as merely a stack of consecutive planes leads to an incomplete and skewed perspective into a particular system&#8217;s 3D nature. Alternative modes of visual inspection require either volume rendering or surface rendering, which&#8212;given the often dense nature of a cellular volume&#8212;can easily lead to an obstructed view of nested objects or overwhelm a user altogether, thus making interactive manual segmentation difficult.
To remedy these barriers, a large variety of automated feature extraction (segmentation) approaches have been developed that are typically either density- or gradient-based8-10. However, these methods tend to segment the entire volume regardless of which areas or features are of interest to the expert, although some recent methods can target a specific feature of interest such as actin filaments11. In addition, the programs executing automated segmentation can sometimes result in the production of a large number of sub-volumes (e.g., when applying watershed immersion segmentation) that often need to be merged manually back into comprising the whole feature of interest or be subjected to further segmentation. This holds true particularly for complex and crowded data sets, thus most rendering computer algorithms are unable to extract only the features of interest with fidelity, and substantial curation efforts by an expert are often needed to produce a desired segmented volume.
Moreover, custom solutions to a highly specific problem are often published as a scientific meeting paper, with little to no emphasis on making them broad and comprehensive tools accessible to researchers who do not have intimate knowledge of the fields of mathematics, computer science and/or computer graphics. A customizable programming software environment, containing a range of image analysis libraries, can be a powerful tool set allowing users to efficiently write their own modules for accurate segmentation. However, this approach requires extensive training and a background in computer science in order to take advantage of its many features or capabilities for image analysis. One can work within such a versatile software environment for certain data sets where the features are more sparse, e.g., by utilizing powerful shape-based approaches which rely on the unique geometry of &#8220;templates&#8221; to separate objects of interest from their surroundings12,13.
A fair variety of computer graphics visualization packages exist for interactive manual segmentation and model building. Some packages are commercially available, while others are of academic origin and distributed free of charge, such as: University of California San Francisco Chimera14, University of Colorado IMOD15, and University of Texas Austin VolumeRover16. However, the wide range and complexity of features and capabilities these programs possess steepens the learning curve for each. Certain visualization programs provide simple geometrical models, such as balls and sticks of various sizes, which can be placed into the density maps in order to create a simplified model of the complex 3D volume. These models then allow simple geometric and volumetric measurements and therefore go beyond just the &#8220;pretty picture&#8221;. Such manual tracing of objects works well for volumes where only a small number of objects need to be traced and extracted. However, the recent development of large volume 3D ultrastructural imaging using either focused ion beam scanning electron microscopy (FIB-SEM)17-20 or serial block face scanning electron microscopy (SBF-SEM)21 presents the additional complication that the size of 3D data sets can range from gigabytes to tens and hundreds of gigabytes, and even the terabytes. Therefore, such large 3D volumes are virtually inaccessible to manual feature extraction, and hence efficient user-guided semi-automated feature extraction will be one of the bottlenecks for efficient analysis of 3D volumes in the foreseeable future.
Presented here are four different segmentation approaches that are routinely used on a large range of biological image types. These methods are then compared for their effectiveness for different types of data sets, allowing a compilation into a guide to help biologists decide what may be the best segmentation approach for effective feature extraction of their own data. As detailed user manuals are available for most of the programs described, the aim is not to make potential users familiar with any one of these particular packages. Instead, the goal is to demonstrate the respective strengths and limitations of these different segmentation strategies by applying them to six example data sets with diverse characteristics. Through this comparison, a set of criteria have been developed that are either based on the objective image characteristics of the 3D data sets, such as data contrast, crispness, crowdedness, and complexity, or stem from subjective considerations, such as the desired objective for segmentation, morphologies of the features to be segmented, population density of the features of interest, meaning the fraction of the volume occupied by the feature of interest, and how one proceeds optimally with finite resources such as time and availability of staff. These different example data sets illustrate how these objective and subjective criteria can be applied sequentially in a variety of combinations to yield a pairing of certain feature extraction approaches with certain types of data sets. The recommendations given will hopefully help novices faced with a large variety of segmentation options choose the most effective segmentation approach for their own 3D volume.
While the focus of this paper is feature extraction, attention to data collection and pre-processing data is crucial to efficient segmentation. Oftentimes staining of samples can be uneven, and hence, potential staining artifacts should be considered in the segmentation procedure. However, stain usually gives higher signal-to-noise, and therefore requires less filtering and other mathematical treatment of cellular volumes, which could potentially also result in artifacts. The respective raw image data sets need to be acquired at the correct contrast and camera pixel settings, aligned, and reconstructed into a 3D volume. For tomograms, aligned images are reconstructed typically using weighted back-projection, and then the data set is usually subjected to denoising algorithms such as non-linear anisotropic diffusion22, bilateral filtering23, or recursive median filtering24. FIB-SEM and SBF-SEM imaging data are aligned by cross-correlating consecutive slices in XY utilizing programs such as ImageJ25. Contrast enhancement and filtering can be applied to boost the features of interest and thus to de-noise the image stack. Filtering can be performed either on the entire volume prior to subvolume selection or on the selected subvolumes, as filtering approaches can be computationally expensive. Down-sampling of the data (binning), which is sometimes used for noise reduction and/or file size reduction, is only recommended if the data has been significantly oversampled compared to the anticipated resolution.
After noise-reduction, the processed images can then be segmented by various methods, and the focus in this study is on the following four: (1) manual abstracted model generation through creating a ball-and-stick model, (2) manual tracing of features of interest, (3) automated threshold-based density, and (4) custom-tailored automated segmentation via a script for project specific segmentation. Boundary segmentation8 and immersive watershed segmentation10 are better alternatives to simple thresholding, but they belong in the same category and have not been included explicitly in this discussion.
&#160;Manual tracing of densities requires outlining the features of interest, slice-by-slice, which allows the retention of the original density of respective sub-cellular areas. This approach allows maximal control of the segmentation process, but is a tedious and labor-intensive process.
Automated threshold-based (and related) density segmentation approaches are semi-automatic, where an algorithm chooses pixels based on a set of user-defined parameters. Several academic (free) visualization packages, such as UCSF Chimera, IMOD, Fiji26, and VolumeRover are available, as well as commercial (requiring paid licenses) packages, and both types typically include one or more of these segmentation approaches. The software packages used in this work to illustrate these different methods include both commercial programs and academic open source programs for manually generating an abstract model, as well as manual and automated density segmentation. However, open source software can sometimes offer more advanced options through the possibility of customization.
A comparison of these techniques using different types of data sets led to the following presentation of rules and guidance on how to approach the segmentation of diverse biological data 3D volumes, which to our knowledge has not yet been published. Thus, this is the first systematic comparison of the different approaches and their usefulness on data sets with varying characteristics for users with different aims.

<table><tbody><tr><td>Amira</td><td>FEI Visualization Sciences Group</td><td></td><td>http://www.vsg3d.com/amira/overview</td></tr><tr><td>Chimera</td><td>UCSF</td><td></td><td>http://www.cgl.ucsf.edu/chimera/</td></tr><tr><td>Fiji/ImageJ</td><td>National Institute of Health</td><td></td><td>http://fiji.sc/Fiji, http://rsbweb.nih.gov/ij/</td></tr><tr><td>IMOD</td><td>Boulder Laboratory for 3D Electron Microscopy of Cells</td><td></td><td>http://bio3d.colorado.edu/imod/</td></tr><tr><td>Photoshop</td><td>Adobe</td><td></td><td>http://www.adobe.com/products/ photoshopfamily.html</td></tr><tr><td>MATLAB</td><td>MathWorks</td><td></td><td>http://www.mathworks.com/</td></tr><tr><td>VLFeat</td><td>VLFeat</td><td></td><td>http://www.vlfeat.org/</td></tr></tbody></table>

from voxels to knowledge: a practical guide to the segmentation of complex electron microscopy 3d-data

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, enabling the visualization of large macromolecular machines, such as adhesion complexes, as well as higher-order structures, such as the cytoskeleton and cellular organelles in their respective cell and tissue context. Given the inherent complexity of cellular volumes, it is essential to first extract the features of interest in order to allow visualization, quantification, and therefore comprehension of their 3D organization. Each data set is defined by distinct characteristics, e.g., signal-to-noise ratio, crispness (sharpness) of the data, heterogeneity of its features, crowdedness of features, presence or absence of characteristic shapes that allow for easy identification, and the percentage of the entire volume that a specific region of interest occupies. All these characteristics need to be considered when deciding on which approach to take for segmentation.
The six different 3D ultrastructural data sets presented were obtained by three different imaging approaches: resin embedded stained electron tomography, focused ion beam- and serial block face- scanning electron microscopy (FIB-SEM, SBF-SEM) of mildly stained and heavily stained samples, respectively. For these data sets, four different segmentation approaches have been applied: (1) fully manual model building followed solely by visualization of the model, (2) manual tracing segmentation of the data followed by surface rendering, (3) semi-automated approaches followed by surface rendering, or (4) automated custom-designed segmentation algorithms followed by surface rendering and quantitative analysis. Depending on the combination of data set characteristics, it was found that typically one of these four categorical approaches outperforms the others, but depending on the exact sequence of criteria, more than one approach may be successful. Based on these data, we propose a triage scheme that categorizes both objective data set characteristics and subjective personal criteria for the analysis of the different data sets.

Watch this Scientific Journal Video about From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data at JoVE.com

From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

The bottleneck for cellular 3D electron microscopy is feature extraction (segmentation) in highly complex 3D density maps. We have developed a set of criteria, which provides guidance regarding which segmentation approach (manual, semi-automated, or automated) is best suited for different data types, thus providing a starting point for effective segmentation.

Modern 3D electron microscopy approaches have recently allowed unprecedented insight into the 3D ultrastructural organization of cells and tissues, ...

from-voxels-to-knowledge-practical-guide-to-segmentation-complex

Research

JoVE Journal

Bioengineering

24.5K Views.  Lawrence Berkeley National Laboratory.  De bottleneck voor mobiele 3D-elektronenmicroscopie is feature extractie (segmentatie) in zeer complexe 3D-density maps. We hebben een set van criteria, waarin aanbevelingen worden gedaan met betrekking tot welke segmentatie aanpak (handmatig, semi-automatisch of automatisch) is het meest geschikt voor verschillende soorten data ontwikkeld, en zo een uitgangspunt voor effectieve segmentatie. 

Video: From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

Cryo-EM and Single-Particle Analysis with Scipion

Cryo-electron microscopy has become one of the most important tools in biological research to reveal the structural information of macromolecules at near-atomic resolution. In single-particle analysis, the vitrified sample is imaged by an electron beam and the detectors at the end of the microscope column produce movies of that sample. These movies contain thousands of images of identical particles in random orientations. The data need to go through an image processing workflow with multiple steps to obtain the final 3D reconstructed volume. The goal of the image processing workflow is to identify the acquisition parameters to be able to reconstruct the specimen under study. Scipion provides all the tools to create this workflow using several image processing packages in an integrative framework, also allowing the traceability of the results. In this article the whole image processing workflow in Scipion is presented and discussed with data coming from a real test case, giving all the details necessary to go from the movies obtained by the microscope to a high resolution final 3D reconstruction. Also, the power of using consensus tools that allow combining methods, and confirming results along every step of the workflow, improving the accuracy of the obtained results, is discussed.

Single-particle analysis in cryo-electron microscopy is one of the main techniques used to determine the structure of biological ensembles at high resolution. Scipion provides the tools to create the whole pipeline to process the information acquired by the microscope and achieve a 3D reconstruction of the biological specimen.

Cryo-electron microscopy has become one of the most important tools in biological research to reveal the structural information of macromolecules at ...

Biochemistry

A Method for 3D Reconstruction and Virtual Reality Analysis of Glial and Neuronal Cells

Serial sectioning and subsequent high-resolution imaging of biological tissue using electron microscopy (EM) allow for the segmentation and reconstruction of high-resolution imaged stacks to reveal ultrastructural patterns that could not be resolved using 2D images. Indeed, the latter might lead to a misinterpretation of morphologies, like in the case of mitochondria; the use of 3D models is, therefore, more and more common and applied to the formulation of morphology-based functional hypotheses. To date, the use of 3D models generated from light or electron image stacks makes qualitative, visual assessments, as well as quantification, more convenient to be performed directly in 3D. As these models are often extremely complex, a virtual reality environment is also important to be set up to overcome occlusion and to take full advantage of the 3D structure. Here, a step-by-step guide from image segmentation to reconstruction and analysis is described in detail.

The described pipeline is designed for the segmentation of electron microscopy datasets larger than gigabytes, to extract whole-cell morphologies. Once the cells are reconstructed in 3D, customized software designed around individual needs can be used to perform a qualitative and quantitative analysis directly in 3D, also using virtual reality to overcome view occlusion.

Serial sectioning and subsequent high-resolution imaging of biological tissue using electron microscopy (EM) allow for the segmentation and reconstruction ...

Neuroscience

Whole&#45;Organ Pathogen Tracking via Fluorescent Reporter Quantification

High&#45;Resolution Three&#45;Dimensional Whole&#45;Organ Tomography of Microbial Infections

Most infections take place within three-dimensional host tissues with intricate anatomy and locally varying host physiology. The positioning of pathogen cells within this diverse environment significantly affects their stress levels, responses, fate, and contribution to the overall progression of the disease and treatment failure. However, due to the technical difficulties in locating &#181;m-sized pathogen cells within cm-sized host organs, this area of research has been relatively unexplored. Here, we present a method for addressing this challenge. We employ serial two-photon tomography and AI-enhanced image analysis to locate individual Salmonella cells throughout the entire spleen, liver lobes, and whole lymph nodes of infected mice. Using fluorescent reporters and in vivo antibody administration, the replication rate of single Salmonella cells, their local interaction with specific immune cells, and bacterial responses to antibiotics can be determined. These methodologies open avenues for a comprehensive examination of infections, their prevention, and treatment within the three-dimensional tissue context.

Author Spotlight: Unraveling Bacterial Responses to Antibiotics and Immune System in Tissues

Here, we describe a procedure that enables organ-wide detection of pathogenic bacteria during infection and quantification of fluorescent reporter activities.

Most infections take place within three-dimensional host tissues with intricate anatomy and locally varying host physiology. The positioning of pathogen ...

Immunology and Infection

3D Reconstruction and Analysis of Thin Subcellular Neuronal Structures using Focused-Ion Beam Scanning Electron Microscopy Data

Recent advances in scanning electron microscope technologies now permit the rapid three-dimensional (3D) analysis of ultrathin subcellular processes. Here, a methodological pipeline is presented to identify, visualize, and analyze thin neuronal processes, such as those that project into the presynaptic boutons of other neurons (termed 'spinules'). Using freely available software packages, this protocol demonstrates how to use a decision tree to identify common neuronal subcellular structures using morphological criteria within focused ion beam scanning electron microscopy (FIB-SEM) image volumes, with particular attention on identifying a diversity of spinules projecting into presynaptic boutons. In particular, this protocol describes how to trace spinules within neuronal synapses to produce 3D reconstructions of these thin subcellular projections, their parent neurites, and postsynaptic partners. Additionally, the protocol includes a list of freely available open-source software programs for analyzing FIB-SEM data and offers tips (e.g., smoothing, lighting) toward improving 3D reconstructions for visualization and publication. This adaptable protocol offers an entry point into the rapid nanoscale analysis of subcellular structures within FIB-SEM image volumes.

A flexible methodological pipeline to identify, visualize, and quantify thin subcellular neuronal processes within focused ion beam scanning electron microscopy image volumes using user-friendly open-source software packages.

Recent advances in scanning electron microscope technologies now permit the rapid three-dimensional (3D) analysis of ultrathin subcellular processes. ...

Correlative Confocal and 3D Electron Microscopy of a Specific Sensory Cell

Delineation of a cell&#8217;s ultrastructure is important for understanding its function. This can be a daunting project for rare cell types diffused throughout tissues made of diverse cell types, such as enteroendocrine cells of the intestinal epithelium. These gastrointestinal sensors of food and bacteria have been difficult to study because they are dispersed among other epithelial cells at a ratio of 1:1,000. Recently, transgenic reporter mice have been generated to identify enteroendocrine cells by means of fluorescence. One of those is the peptide YY-GFP mouse. Using this mouse, we developed a method to correlate confocal and serial block-face scanning electron microscopy. We named the method cocem3D and applied it to identify a specific enteroendocrine cell in tissue and unveil the cell&#8217;s ultrastructure in 3D. The resolution of cocem3D is sufficient to identify organelles as small as secretory vesicles and to distinguish cell membranes for volume rendering. Cocem3D can be easily adapted to study the 3D ultrastructure of other specific cell types in their native tissue.

Here, we introduce a method, cocem3D, to unveil the ultrastructure of a specific cell in its native tissue by bridging confocal and serial block-face scanning electron microscopy.

Delineation of a cell&#8217;s ultrastructure is important for understanding its function. This can be a daunting project for rare cell types diffused ...

Biology

Simplified Volumetric Models as an Effective Strategy for Segmenting Actin Networks in Cryo-Electron Tomograms

Efficient methods for the extraction of features of interest remain one of the biggest challenges for the interpretation of cryo-electron tomograms. Various automated approaches have been proposed, many of which work well for high-contrast datasets where the features of interest can be easily detected and are clearly separated from one another. Our inner ear stereocilia cryo-electron tomographic datasets are characterized by a dense array of hexagonally packed actin filaments that are frequently cross-connected. These features make automated segmentation very challenging, further aggravated by the high-noise environment of cryo-electron tomograms and the high complexity of the densely packed features. Using prior knowledge about the actin bundle organization, we have placed layers of a highly simplified ball-and-stick actin model to first obtain a global fit to the density map, followed by regional and local adjustments of the model. We show that volumetric model building not only allows us to deal with the high complexity, but also provides precise measurements and statistics about the actin bundle. Volumetric models also serve as anchoring points for local segmentation, such as in the case of the actin-actin cross connectors. Volumetric model building, particularly when further augmented by computer-based automated fitting approaches, can be a powerful alternative when conventional automated segmentation approaches are not successful.

Here, we present a protocol to place simplified volumetric models into noisy, complex, tomographic 3D volumes. This allows the fast segmentation of actin filament densities, detection of systematic filament bending and of gaps in hair bundle filaments, as well as convenient quantification of volumetric model properties, such as distances.

Efficient methods for the extraction of features of interest remain one of the biggest challenges for the interpretation of cryo-electron tomograms. ...

Targeted Studies Using Serial Block Face and Focused Ion Beam Scan Electron Microscopy

This protocol allows for the efficient and effective imaging of cell or tissue samples in three dimensions at the resolution level of electron microscopy. For many years electron microscopy (EM) has remained an inherently two-dimensional technique. With the advent of serial scanning electron microscope imaging techniques (volume EM), using either an integrated microtome or focused ion beam to slice then view embedded tissues, the third dimension becomes easily accessible. Serial block face scanning electron microscopy (SBF-SEM) uses an ultramicrotome enclosed in the SEM chamber. It has the capability to handle large specimens (1,000 &#181;m x 1,000 &#181;m) and image large fields of view at small X,Y pixel size, but is limited in the Z dimension by the diamond knife. Focused ion beam SEM (FIB-SEM) is not limited in 3D resolution, (isotropic voxels of &#8804;5 nm are achievable), but the field of view is much more limited. This protocol demonstrates a workflow for combining the two techniques to allow for finding individual regions of interest (ROIs) in a large field and then imaging the subsequent targeted volume at high isotropic voxel resolution. Preparing fixed cells or tissues is more demanding for volume EM techniques due to the extra contrasting needed for efficient signal generation in SEM imaging. Such protocols are time consuming and labor intensive. This protocol also incorporates microwave assisted tissue processing facilitating the penetration of reagents, which reduces the time needed for the processing protocol from days to hours.

Here, we present a protocol for efficiently combining serial block face and focused ion beam scanning electron microscopy to target an area of interest. This allows for efficient searching, in three dimensions, and locating rare events in a large field of view.

This protocol allows for the efficient and effective imaging of cell or tissue samples in three dimensions at the resolution level of electron microscopy. ...

Array Tomography Workflow for the Targeted Acquisition of Volume Information using Scanning Electron Microscopy

Electron microscopy is applied in biology and medicine for imaging of cellular and structural details at nanometer resolution. Historically, Transmission Electron Microscopy (TEM) provided insight into cell ultrastructure, but in the recent decade, the development of modern Scanning Electron Microscopes (SEM) has changed the way of looking inside the cells. Even though the resolution of TEM is superior when protein-level structural details are needed, SEM-resolution is sufficient for the majority of the organelle-level cell biology-related questions. The advancement in technology enabled automatic volume acquisition solutions such as Serial block-face imaging (SBF-SEM) and Focused ion beam SEM (FIB-SEM). Nevertheless, to this day, these methods remain inefficient when the identification and navigation to areas of interest are crucial. Without the means for precise localization of target areas before imaging, operators need to acquire much more data than they need (in SBF-SEM), or, even worse, prepare many grids and image them all (in TEM). We propose the strategy of "lateral screening" using Array Tomography in SEM, which facilitates the localization of areas of interest, followed by automated imaging of the relevant fraction of the total sample volume. Array tomography samples are conserved during imaging, and they can be arranged into section libraries ready for repeated imaging. Several examples are shown in which lateral screening enables us to analyze structural details that are incredibly challenging to access with any other method.

We describe the preparation of ribbons of serial sections and their collection on large transfer support for use as Array Tomography samples, along with automated imaging procedures in a scanning electron microscope. The protocol allows screening, retrieval, and targeted imaging of local, rare events, and the acquisition of large data volumes.

Electron microscopy is applied in biology and medicine for imaging of cellular and structural details at nanometer resolution. Historically, Transmission ...

Serial Block-Face Scanning Electron Microscopy (SBF-SEM) of Biological Tissue Samples

Serial block-face scanning electron microscopy (SBF-SEM) allows for the collection of hundreds to thousands of serially-registered ultrastructural images, offering an unprecedented three-dimensional view of tissue microanatomy. While SBF-SEM has seen an exponential increase in use in recent years, technical aspects such as proper tissue preparation and imaging parameters are paramount for the success of this imaging modality. This imaging system benefits from the automated nature of the device, allowing one to leave the microscope unattended during the imaging process, with the automated collection of hundreds of images possible in a single day. However, without appropriate tissue preparation cellular ultrastructure can be altered in such a way that incorrect or misleading conclusions might be drawn. Additionally, images are generated by scanning the block-face of a resin-embedded biological sample and this often presents challenges and considerations that must be addressed. The accumulation of electrons within the block during imaging, known as "tissue charging," can lead to a loss of contrast and an inability to appreciate cellular structure. Moreover, while increasing electron beam intensity/voltage or decreasing beam-scanning speed can increase image resolution, this can also have the unfortunate side effect of damaging the resin block and distorting subsequent images in the imaging series. Here we present a routine protocol for the preparation of biological tissue samples that preserves cellular ultrastructure and diminishes tissue charging. We also provide imaging considerations for the rapid acquisition of high-quality serial-images with minimal damage to the tissue block.

Serial Block-Face Scanning Electron Microscopy SBF-SEM of Biological Tissue Samples

This protocol outlines a routine method for using serial block-face scanning electron microscopy (SBF-SEM), a powerful 3D imaging technique. Successful application of SBF-SEM hinges on proper fixation and tissue staining techniques, as well as careful consideration of imaging settings. This protocol contains practical considerations for the entirety of this process.

Serial block-face scanning electron microscopy (SBF-SEM) allows for the collection of hundreds to thousands of serially-registered ultrastructural images, ...

Three-dimensional Characterization of Interorganelle Contact Sites in Hepatocytes using Serial Section Electron Microscopy

Transmission electron microscopy has been long considered to be the gold standard for the visualization of cellular ultrastructure. However, analysis is often limited to two dimensions, hampering the ability to fully describe the three-dimensional (3D) ultrastructure and functional relationship between organelles. Volume electron microscopy (vEM) describes a collection of techniques that enable the interrogation of cellular ultrastructure in 3D at mesoscale, microscale, and nanoscale resolutions. 
This protocol provides an accessible and robust method to acquire vEM data using serial section transmission EM (TEM) and covers the technical aspects of sample processing through to digital 3D reconstruction in a single, straightforward workflow. To demonstrate the usefulness of this technique, the 3D ultrastructural relationship between the endoplasmic reticulum and mitochondria and their contact sites in liver hepatocytes is presented. Interorganelle contacts serve vital roles in the transfer of ions, lipids, nutrients, and other small molecules between organelles. However, despite their initial discovery in hepatocytes, there is still much to learn about their physical features, dynamics, and functions. 
Interorganelle contacts can display a range of morphologies, varying in the proximity of the two organelles to one another (typically ~10-30 nm) and the extent of the contact site (from punctate contacts to larger 3D cisternal-like contacts). The examination of close contacts requires high-resolution imaging, and serial section TEM is well suited to visualize the 3D ultrastructural of interorganelle contacts during hepatocyte differentiation, as well as alterations in hepatocyte architecture associated with metabolic diseases.

A simple and comprehensive protocol to acquire three-dimensional details of membrane contact sites between organelles in hepatocytes from the liver or cells in other tissues.

Transmission electron microscopy has been long considered to be the gold standard for the visualization of cellular ultrastructure. However, analysis is ...

From Voxels to Knowledge: A Practical Guide to the Segmentation of Complex Electron Microscopy 3D-Data

Samenvatting

Meer video's verkennen