This research was an accumulated effort that began more than 10 years ago when Dr. Zhang was a graduate student at Stony Brook University. The current and previous students in our team at Iowa State University have contributed tremendously toward advancing this technology to where it is today. This work was partially sponsored by National Science Foundation under project number CMMI 1150711, and the William and Virginia Binger Foundation.

1. System Configuration
A schematic of the system is shown in Figure 1.
<ol>
	<li>Generate the fringe patterns for projection. These can be prepared well in advance by using an image programming environment such as MATLAB, OpenCV, or QT. Construct the patterns according to the three-step phase-shifting algorithm in18. Make three images, shifted in phase offset from each other by 2&#960;/3. For binary defocusing, use a dithering technique to generate sinusoidal patterns using only black and white pixels as described in19.</li>
	<li>Select the digital light processing projector. A high-speed binary defocusing system requires a faster, specialized projector such as the DLP LightCommander with ALP High Speed module. Be sure to use the binary or monochromatic setting to project the fringe images. Since the image values are purely on or off, nonlinearity adjustments are not necessary. Utilize the projector&#8217;s software program to upload the patterns for phase shifting.</li>
	<li>Select the camera. Choose a black-and-white CCD or CMOS camera with the correct capture rate for the system. Avoid color cameras for 3D capture since color is not needed and color cameras require nonlinearity and gamma adjustments. Keep in mind that the camera will need to capture the entire set of fringe images for each 3D video frame. High-quality sinusoidal systems require precise synchronization between the projector and the camera; for binary defocusing systems this requirement is more relaxed20.</li>
	<li>Determine the desired maximum (x, y) range and the distance from the projector to the object (d0). Choose a range that makes sense for the application, but be sure that the area is slightly larger than the subject to reduce any optical boundary effects. Since the light output of a projector is a frustum, this (x, y) range will drive d0. Move the projector relative to a large flat projection surface until the desired (x, y) range is found, then measure d0 with a tape measure.</li>
	<li>Select the camera lens with the proper focal length. Using the camera&#8217;s sensor size, find the focal length such that the field of view at the distance d0 is the same as the desired imaging range (x, y).</li>
	<li>Determine the separation distance between the projector and the camera. A trade-off occurs here between noise and shadowing. At a large angle between these components, triangulation between feature points is more obvious, but more features get lost in shadow from the camera&#8217;s perspective. At a small angle, triangulation becomes difficult, increasing noise in the results. Typically, 10-15&#176; is a good compromise.</li>
</ol>
2. System Calibration
This reference plane calibration is the simplest and most readily accessible method of calibration for the system. Therefore, it is the best for getting started. More accurate calibration methods are available in the literature for specific sinusoidal16 and binary defocusing17 applications. For maximum accuracy, calibration should be performed just before data capture. After calibration, the camera and projector should not be displaced relative to each other.
<ol>
	<li>Focus/defocus the projector. Carefully defocus the projection lens until the patterns at the imaging plane resemble high-quality sinusoids. This may require an iterative process of examining the data quality (Section 4) and adjusting the lens.</li>
	<li>Capture fringe images of a reference plane. Place a flat white board at the focal plane of the projector and in the field of view of the camera. A 3/16 in&#160;(5 mm) thick white foam core board works well, provided that the surface facing the system is not shiny or significantly blemished or torn. Record and save fringe images of this board for the data processing step.</li>
	<li>Capture fringe images of a reference object of known dimensions. For this step, a rigid foam cube is one simple example. Cover the cube with squares of 1/16 in&#160;(1.5 mm) white adhesive foam to make it diffuse. Place it at the focal plane of the camera in the camera&#8217;s field of view and record fringe images for the processing step.</li>
</ol>
3. Data Acquisition
<ol>
	<li>Place the object or invite the subject to sit at the focal plane of the camera. For a human subject, warn him or her that the projector light may be bright. A black cloth backdrop can be used behind the subject to hide extraneous surroundings.</li>
	<li>Adjust the camera aperture to optimize the light level. Sample fringe images should be as bright as possible, but not saturated. Dark images will have too much noise, while saturated areas (significant regions of maximum brightness) in images will result in the loss of details in the saturated region.</li>
	<li>Capture the desired number of frames. Be sure to bring a hard drive large enough to hold all of the captured images for both the subject and the calibration datasets. With the .OBJ file format, a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480 could be over 50 GB.</li>
</ol>
4. Data Analysis and Visualization
With software optimized for speed such as our in-house GUI, this step can take place during data capture. Real-time processing allows the user to immediately detect if the resulting data is desirable for the application and adjust if necessary. However, post processing can be more flexible and higher in accuracy. Post processing is also much simpler to implement and the best place to begin.
<ol>
	<li>Compute the wrapped phase for both the calibration and subject data. In the three-step phase-shifting algorithm in18, phase describes the position of a point within the cosine function. Since we have three equations and three unknowns, we can solve the equations used to generate the fringe images in step 1.1 for the phase at each point. Because of the arctangent function, the computed phase is in the range (-&#960;, &#960;]; hence it is called &#8220;wrapped phase.&#8221; To improve the processing speed, we developed a fast phase wrapping algorithm that is discussed in21.</li>
	<li>Unwrap the phase maps. Adopt a phase-unwrapping algorithm that detects the 2&#960; jumps in the phase and removes them by adding or subtracting multiples of 2&#960;. We have used the fast algorithm in22 in previous systems to unwrap the phase robustly yet quickly. In the video, we demonstrate the multifrequency technique described in15, which uses additional sets of three phase-shifted patterns at differing frequencies. The wrapped phase maps from each set of three can be combined to robustly yield a single unwrapped phase map. This technique increases the depth range for accurate capture with binary defocusing.</li>
	<li>Optional: Compute the 2D texture. Averaging sets of three neighboring fringe images will wash out the fringe stripes and generate the 2D texture map. This can be mapped onto the 3D data during visualization if desired.</li>
	<li>Convert the unwrapped phase maps into depth. As described in23, depth can be computed for the calibration cube as the difference between the calibration cube phase map and the reference plane phase map. Compare this computed depth to the known depth to compute the correct depth scaling factor c0. Then, compute the depth for the subject by subtracting the reference plane phase from the subject&#8217;s phase and multiplying the results by c0.</li>
	<li>Compute the x- and y-coordinates. Apply the scaling factor c0 to the calibration cube depth map. Determine the conversion factor &#961; from the cube dimensions in pixels to the known cube dimensions in the xy plane. Multiply the pixel count in the subject data by &#961; to compute the x and y coordinates.</li>
	<li>Visualize the data. Individual frames can be saved in our in-house format and viewed with simple MATLAB code or saved in .OBJ format and viewed with a commercial 3D modeling program such as Blender. Due to the large amount of data in each frame, these applications may be sluggish on some computers. For more responsiveness or for live video display, write software using a computer graphics library such as OpenGL or Direct3D. This software can take advantage of the graphics processing unit (GPU) to rapidly generate the x, y, and z coordinates from the unwrapped phase, form a triangle mesh, compute lighting normals, and display the results. Using the GPU, we have achieved up to 30 Hz live 3D data visualization with approximately 300,000 points per frame.</li>
</ol>

<ol>
	<li>Salvi, J., Fernandez, S., Pribanic, T., Llado, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+state+of+the+art+in+structured+light+patterns+for+surface+profilometry.">A state of the art in structured light patterns for surface profilometry.</a> Patt. Recogn. 43, 2666-2680 (2010).</li><li>Gorthi, S. S., Rastogi, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fringe+projection+techniques:+Whither+we+are.">Fringe projection techniques: Whither we are.</a> Opt. Laser Eng. 48 (2), 133-140 (2010).</li><li>Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Recent+progresses+on+real-time+3-D+shape+measurement+using+digital+fringe+projection+techniques.">Recent progresses on real-time 3-D shape measurement using digital fringe projection techniques.</a> Opt. Laser Eng. 48 (2), 149-158 (2010).</li><li>Su, X., Zhang, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Dynamic+3-D+shape+measurement+method:+A+review.">Dynamic 3-D shape measurement method: A review.</a> Opt. Laser Eng. 48 (2), 191-204 (2010).</li><li>Geng, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structured-light+3D+surface+imaging:+a+tutorial.">Structured-light 3D surface imaging: a tutorial.</a> Adv. Opt. Photonics. 3 (2), 128-160 (2011).</li><li>Zhang, S., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-resolution,+real-time+3-D+absolute+coordinate+measurement+based+on+a+phase-shifting+method.">High-resolution, real-time 3-D absolute coordinate measurement based on a phase-shifting method.</a> Opt. Express. 14 (7), 2644-2649 (2006).</li><li>Zhang, S., Royer, D., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=GPU-assisted+high-resolution,+real-time+3-D+shape+measurement.">GPU-assisted high-resolution, real-time 3-D shape measurement.</a> Opt. Express. 14 (20), 9120-9129 (2006).</li><li>Lei, S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Flexible+3-D+shape+measurement+using+projector+defocusing.">Flexible 3-D shape measurement using projector defocusing.</a> Opt. Lett. 34 (20), 3080-3082 (2009).</li><li>Mehta, R. P., Zhang, S., Hadlock, T. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Novel+3-D+video+for+quantification+of+facial+movement.">Novel 3-D video for quantification of facial movement.</a> Otolaryngol. Head Neck Surg. 138 (4), 468-472 (2008).</li><li>Wang, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High+resolution+acquisition,+learning+and+transfer+of+dynamic+3D+facial+expressions.">High resolution acquisition, learning and transfer of dynamic 3D facial expressions.</a> Comput. Graph. Forum. 23 (3), (2004).</li><li>Laughner, J. I., Zhang, S., Li, H., Efimov, I. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Mapping+cardiac+surface+mechanics+with+structured+light+imaging.">Mapping cardiac surface mechanics with structured light imaging.</a> Am. J. Physiol. Heart Circ. Physiol. , (2012).</li><li>Ekstrand, L., Wang, Y., Karpinsky, N., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Superfast+3D+profilometry+with+digital+fringe+projection+and+phase-shifting+techniques.">Superfast 3D profilometry with digital fringe projection and phase-shifting techniques.</a> Handbook of 3-D machine vision: Optical metrology and imaging. , (2012).</li><li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-resolution+real-time+three-dimensional+shape+measurement.">High-resolution real-time three-dimensional shape measurement.</a> Opt. Eng. 45 (12), 123-601 (2006).</li><li>Li, Y., Zhao, C., Qian, Y., Wang, H., Jin, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-speed+and+dense+three-dimensional+surface+acquisition+using+defocused+binary+patterns+for+spatially+isolated+objects.">High-speed and dense three-dimensional surface acquisition using defocused binary patterns for spatially isolated objects.</a> Opt. Express. 18, 21-635 (2010).</li><li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Superfast+multifrequency+phase-shifting+technique+with+optimal+pulse+width+modulation.">Superfast multifrequency phase-shifting technique with optimal pulse width modulation.</a> Opt. Express. 19 (6), 5143-5148 (2011).</li><li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Novel+method+for+structured+light+system+calibration.">Novel method for structured light system calibration.</a> Opt. Eng. 45 (8), 083601 (2006).</li><li>Merner, L., Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Accurate+calibration+for+3D+shape+measurement+system+using+a+binary+defocusing+technique.">Accurate calibration for 3D shape measurement system using a binary defocusing technique.</a> Opt. Laser Eng. 51 (5), 514-519 (2013).</li><li>Malacara, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Optical Shop Testing. , (2007).</li><li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+shape+measurement+with+binary+dithered+patterns.">Three-dimensional shape measurement with binary dithered patterns.</a> Appl. Opt. 51 (27), 6631-6636 (2012).</li><li>Ekstrand, L., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autoexposure+for+three-dimensional+shape+measurement+with+a+digital-light-processing+projector.">Autoexposure for three-dimensional shape measurement with a digital-light-processing projector.</a> Opt. Eng. 50 (12), 123-603 (2011).</li><li>Huang, P. S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+three-step+phase-shifting+algorithm.">Fast three-step phase-shifting algorithm.</a> Appl. Opt. 45 (21), 5086-5091 (2006).</li><li>Zhang, S., Li, X., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multilevel+quality-guided+phase+unwrapping+algorithm+for+real-time+3-D+shape+reconstruction.">Multilevel quality-guided phase unwrapping algorithm for real-time 3-D shape reconstruction.</a> Appl. Opt. 46 (1), 50-57 (2007).</li><li>Xu, Y., Ekstrand, L., Dai, J., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Phase+error+compensation+for+3-D+shape+measurement+with+projector+defocusing.">Phase error compensation for 3-D shape measurement with projector defocusing.</a> Appl. Opt. 50 (18), 2572-2581 (2011).</li></ol>

The authors have no competing financial interests.

This high-resolution, real-time to superfast 3D video imaging technology is a platform technology that could potentially benefit numerous and diverse scientific fields ranging from biological science to engineering practice. Biomedical applications include precision measurements of facial movements and organ surfaces.&#160;Other applications include 3D automated quality control with detection of warped surface features; 3D enhanced videoconferencing; detailed digitization of facial features for movies and videogames; dense and rapid deformation measurements for the design and analysis of structures; and fluid surface characterization.&#160;Many biological and engineering applications (e.g. beating rabbit hearts, fluid shockwaves) require the superfast imaging rates of a binary defocusing system to correctly resolve features without aliasing artifacts.&#160;
Nevertheless, many challenges remain to the widespread adoption of this technology.&#160;Conventional DFP technology requires the projector to display 8-bit grayscale sinusoidal fringe patterns.&#160;The speed of this technique is limited by the projector&#8217;s refresh rate (typically 120 Hz).&#160;This speed is sufficient for slow motion capture such as that in facial expressions.&#160;However, numerous applications exist that require faster capture rates.
Binary defocusing technology has relaxed this speed limitation, and we have successfully created a superfast 3D video imaging system.&#160;However, this system has two drawbacks.&#160;First, it requires an expensive projector such as the DLP Discovery platform and a costly high-speed video camera such as the Vision Research Phantom v9.1.&#160;Second, since it generates sinusoidal patterns via the defocusing of squared binary patterns, the binary defocusing technique has difficulty generating sinusoidal fringes of the same quality as the traditional DFP technique and a reduced depth measurement range (for further explanation, see23).&#160;Recent investigation indicates that dithered binary sinusoidal patterns can significantly alleviate the limitations on depth measurement range19.&#160;Future research will focus on overcoming the remaining issues while preserving the merits of binary defocusing.&#160;
Another challenge is compressing and storing the large amount of data generated by high-speed, high-resolution 3D video imaging systems.&#160;Uncompressed 3D videos are drastically larger than uncompressed 2D videos.&#160;For instance, for a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480, the .OBJ file size could be over 50 GB, making it extremely difficult to store.&#160;Since little progress has been made in the 3D video compression field, we will continue to focus on this in the future.

Digital fringe projection (DFP) techniques are based upon correlation and triangulation between two views of the same scene at different angles, the same principle employed by the human eyes and brain to achieve stereo vision. However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the object that the object&#8217;s surface distorts in the camera&#8217;s view. Three such distorted patterns (fringe images) at differing phase shifts from each other can be analyzed to retrieve the depth via triangulation. The use of a known pattern eliminates the difficult computational problem of identifying correspondence points, allowing the capture of depth measurements at the camera resolution. For example, with a 576 x 576 camera, the technique can capture 331,776 points. This allows DFP systems to measure very fine details such as the movement of facial muscles in human emotions.
3D optical imaging techniques for static or quasi-static events have been extensively studied over the past few decades and have seen great success in video game design, animation, movies, music videos, virtual reality, telesurgery, and many engineering disciplines5. Though numerous 3D profilometry techniques exist, they can be classified into two categories: surface contact methods and surface noncontact methods. Both the coordinate measurement machine (CMM) and the atomic force microscope (AFM) require contact with the measuring surface to obtain 3D profiles at high accuracy. This requirement places severe restrictions on the speed of contact methods. They cannot reach kHz measurement speed with thousands of points per scan.
Surface noncontact techniques typically utilize optical triangulation methods (e.g.&#160;stereo vision, spacetime stereo, structured light). By actively projecting known patterns onto the objects, structured light techniques can be used to measure surfaces without strong local texture variations1. Fringe analysis is a special group of structured light techniques that uses sinusoidal structured patterns (also known as fringe patterns). Because these patterns have intensities that vary continuously from point to point in a known manner, they boost the structured light techniques from projector-pixel resolution to camera-pixel resolution12. In the recent past, fringe analysis techniques were instrumental in achieving high-resolution 3D imaging.
The digital fringe projection (DFP) technique uses digital video projectors to generate sinusoidal fringe patterns. This technique has the merits of lower cost, higher speed, and simplicity of development, and it has been a very active research area within the past decade. Recent developments in DFP and similar digital structured light techniques are summarized in1-5. To achieve high-speed applications, a digital-light-processing (DLP) projector is preferable due to its fundamental operation mechanism. The speed and flexibility of this technique has allowed us to acquire 3D video at 40 Hz&#160;13 and then later at 60 Hz&#160;6,7.
Nevertheless, a fundamental speed limit exists for the traditional DFP technique. A DLP projector can only swap 8-bit color images at its maximum refresh rate (typically 120 Hz). Since the traditional fringe patterns are 8-bit grayscale images, we can encode three of them into one color image as the red, green, and blue color channels. The projector will swap each channel (and therefore each fringe pattern) at three times the refresh rate (typically 360 Hz). However, since each 3D video frame requires three fringe patterns, the maximum rate of 3D video capture is still only the refresh rate (120 Hz)3,14. To break past this hardware limitation, we have invented a modified DFP technique that uses binary defocusing8. Instead of 8-bit grayscale fringe patterns, this technique uses computer-generated 1-bit binary structured patterns. These patterns are defocused using the projector lens to become pseudo-sinusoidal patterns for DFP. Because DLP projectors can display binary images orders-of-magnitude faster than 8-bit grayscale images, the binary defocusing technology permits tens of kilohertz&#160;3D video imaging speed with the same resolution as the conventional DFP techniques15.
The overall goal of the following protocol is to demonstrate the basic implementation and operation of a binary defocusing three-step phase-shifting DFP system. First, the protocol will cover the selection and integration of the necessary components. Then, it will discuss the simplest, most readily accessible method of calibration for the system; more complex calibration methods are available in the literature for specific applications16,17. The protocol will then focus on the procedure for 3D video capture with the system and the process for converting the fringe images into visualized 3D measurements. Finally, we will present some representative results from our real-time and high-speed systems.

<table><tbody><tr><td colspan="4">&lt;p class=&quot;jove_content&quot;&gt;No materials or equipment used.&lt;/p&gt;</td></tr></tbody></table>

high-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses triangulation between matching points in two views of the same scene at different angles to compute depth.&#160;However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the subject, and the surface of the subject distorts these patterns in the camera&#8217;s field of view. Three distorted patterns (fringe images) from the camera can be used to compute the depth using triangulation.
Unlike other 3D measurement methods, DFP techniques lead to systems that tend to be faster, lower in equipment cost, more flexible, and easier to develop.&#160;DFP systems can also achieve the same measurement resolution as the camera.&#160;For this reason, DFP and other digital structured light techniques have recently been the focus of intense research (as summarized in1-5). Taking advantage of DFP, the graphics processing unit, and optimized algorithms, we have developed a system capable of 30&#160;Hz 3D video data acquisition, reconstruction, and display for over 300,000 measurement points per frame6,7.&#160;Binary defocusing DFP methods can achieve even greater speeds8.
Diverse applications can benefit from DFP techniques. Our collaborators have used our systems for facial function analysis9, facial animation10, cardiac mechanics studies11, and fluid surface measurements, but many other potential applications exist.&#160;This video will teach the fundamentals of DFP techniques and illustrate the design and operation of a binary defocusing DFP system.

Figure 1 shows the schematic of the system.&#160;The high-speed binary defocusing system in this video consists of a Logic PD DLP LightCommander projector and a Phantom v9.1 CMOS camera.
Figure 2 presents a single frame from our 3D real-time system of a human face.&#160;This system uses a 640 x 480 camera.&#160;Thanks to the aforementioned known sinusoidal pattern, we can capture 640 x 480 = 307,200 measurements, enough resolution to record very fine details.
Figure 3 shows an example of measuring human facial expressions in 3D at 60 Hz. Here, four frames selected from a video sequence clearly demonstrate the capability of the real-time system to capture dynamic changes in finely detailed geometry.
Figure 4 demonstrates our live visualization software used in conjunction with our real-time binary defocusing 3D video system.&#160;The 3D captured video of the subject is displayed in real time on the computer monitor to his right.&#160;This software was written in C++ using the OpenGL library, GLSL, and QT.&#160;The computer used is a Lenovo laptop.
Figure 5 shows 3D frames from live rabbit heart measurement with our newly developed superfast binary defocusing system.&#160;This system can record 3D frames at 667 Hz with an image resolution of 576 x 576. A superfast rate is required to measure the heart surface without motion-induced artifacts.&#160;The heart measurement research is in collaboration with Prof. Igor Efimov at Washington University-St. Louis (see11 for further details); note that the rabbit was humanely killed and that the images were taken while the heart was still beating.
<img alt="Figure 1" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig1highres.jpg" src="/files/ftp_upload/50421/50421fig1.jpg" /> 
Figure 1.&#160;Layout of the 3D video imaging system. In this system, a high-speed DLP projector projects three binary dithered phase-shifted images in rapid succession onto the subject.&#160;A high-speed CMOS camera is used to capture the three fringe images one by one for computation of the depth.
<img alt="Figure 2" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig2highres.jpg" src="/files/ftp_upload/50421/50421fig2.jpg" /> 
Figure 2.&#160;3D measurements of a human face at a resolution of 640 x 480, revealing fine details. Left to right shows the simultaneously captured texture perfectly aligned with the geometry, a shaded view of the geometry, the wireframe view depicting the density of the points, a close-up view of the nose area, and a close-up view of the eye region.
<img alt="Figure 3" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig3highres.jpg" src="/files/ftp_upload/50421/50421fig3.jpg" /> 
Figure 3.&#160;Four selected frames from 3D video of the formation of a facial expression.&#160;The video was captured at 60 Hz with a resolution of 640 x 480.&#160;These frames highlight the geometric changes in the woman&#8217;s face as she moves from a neutral expression to a smile.
<img alt="Figure 4" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig4highres.jpg" src="/files/ftp_upload/50421/50421fig4.jpg" /> 
Figure 4.&#160;Live 3D video capturing, processing, and rendering. The 3D measurements are displayed in real time on the computer screen to the subject&#8217;s right.
<img alt="Figure 5" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig5highres.jpg" src="/files/ftp_upload/50421/50421fig5.jpg" /> 
Figure 5.&#160;Capturing a live rabbit heart with our superfast 3D video imaging system. The heart is beating at approximately 200 beats/min. The 3D capture rate was 166 Hz with an image resolution of 576 x 576. See11 for further details.

Watch this Scientific Journal Video about High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques at JoVE.com

High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

This video describes the fundamentals of digital fringe projection techniques, which provide dense 3D measurements of dynamically changing surfaces.&#160;It also demonstrates the design and operation of a high-speed binary defocusing system based on these techniques.

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses ...

high-resolution-high-speed-three-dimensional-video-imaging-with

Research

JoVE Journal

Engineering

15.6K Views.  Iowa State University. This video describes the fundamentals of digital fringe projection techniques, which provide dense 3D measurements of dynamically changing surfaces. It also demonstrates the design and operation of a high-speed binary defocusing system based on these techniques.

Video: High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

High Resolution 3D Imaging of Ex-Vivo Biological Samples by Micro CT

Non-destructive volume visualization can be achieved only by tomographic techniques, of which the most efficient is the x-ray micro computerized tomography (&mu;CT).
High resolution &mu;CT is a very versatile yet accurate (1-2 microns of resolution) technique for 3D examination of ex-vivo biological samples1, 2. As opposed to electron tomography, the &mu;CT allows the examination of up to 4 cm thick samples. This technique requires only few hours of measurement as compared to weeks in histology. In addition, &mu;CT does not rely on 2D stereologic models, thus it may complement and in some cases can even replace histological methods3, 4, which are both time consuming and destructive. Sample conditioning and positioning in &mu;CT is straightforward and does not require high vacuum or low temperatures, which may adversely affect the structure. The sample is positioned and rotated 180&deg; or 360&deg;between a microfocused x-ray source and a detector, which includes a scintillator and an accurate CCD camera, For each angle a 2D image is taken, and then the entire volume is reconstructed using one of the different available algorithms5-7. The 3D resolution increases with the decrease of the rotation step. The present video protocol shows the main steps in preparation, immobilization and positioning of the sample followed by imaging at high resolution.

High Resolution 3D Imaging of Ex-Vivo Biological Samples by Micro CT

Non-destructive volume visualization can be achieved only by tomographic techniques, of which the most efficient is the x-ray micro computerized tomography ( CT).

Non-destructive volume visualization can be achieved only by tomographic techniques, of which the most efficient is the x-ray micro computerized ...

Bioengineering

Video-rate Scanning Confocal Microscopy and Microendoscopy

Confocal microscopy has become an invaluable tool in biology and the biomedical sciences, enabling rapid, high-sensitivity, and high-resolution optical sectioning of complex systems. Confocal microscopy is routinely used, for example, to study specific cellular targets1, monitor dynamics in living cells2-4, and visualize the three dimensional evolution of entire organisms5,6. Extensions of confocal imaging systems, such as confocal microendoscopes, allow for high-resolution imaging in vivo7 and are currently being applied to disease imaging and diagnosis in clinical settings8,9.
Confocal microscopy provides three-dimensional resolution by creating so-called "optical sections" using straightforward geometrical optics. In a standard wide-field microscope, fluorescence generated from a sample is collected by an objective lens and relayed directly to a detector. While acceptable for imaging thin samples, thick samples become blurred by fluorescence generated above and below the objective focal plane. In contrast, confocal microscopy enables virtual, optical sectioning of samples, rejecting out-of-focus light to build high resolution three-dimensional representations of samples.
Confocal microscopes achieve this feat by using a confocal aperture in the detection beam path. The fluorescence collected from a sample by the objective is relayed back through the scanning mirrors and through the primary dichroic mirror, a mirror carefully selected to reflect shorter wavelengths such as the laser excitation beam while passing the longer, Stokes-shifted fluorescence emission. This long-wavelength fluorescence signal is then passed to a pair of lenses on either side of a pinhole that is positioned at a plane exactly conjugate with the focal plane of the objective lens. Photons collected from the focal volume of the object are collimated by the objective lens and are focused by the confocal lenses through the pinhole. Fluorescence generated above or below the focal plane will therefore not be collimated properly, and will not pass through the confocal pinhole1, creating an optical section in which only light from the microscope focus is visible. (Fig 1). Thus the pinhole effectively acts as a virtual aperture in the focal plane, confining the detected emission to only one limited spatial location.
Modern commercial confocal microscopes offer users fully automated operation, making formerly complex imaging procedures relatively straightforward and accessible. Despite the flexibility and power of these systems, commercial confocal microscopes are not well suited for all confocal imaging tasks, such as many in vivo imaging applications. Without the ability to create customized imaging systems to meet their needs, important experiments can remain out of reach to many scientists.
In this article, we provide a step-by-step method for the complete construction of a custom, video-rate confocal imaging system from basic components. The upright microscope will be constructed using a resonant galvanometric mirror to provide the fast scanning axis, while a standard speed resonant galvanometric mirror will scan the slow axis. To create a precise scanned beam in the objective lens focus, these mirrors will be positioned at the so-called telecentric planes using four relay lenses. Confocal detection will be accomplished using a standard, off-the-shelf photomultiplier tube (PMT), and the images will be captured and displayed using a Matrox framegrabber card and the included software.

The complete construction of a custom, real-time confocal scanning imaging system is described. This system, which can be readily used for video-rate microscopy and microendoscopy, allows for an array of imaging geometries and applications not accessible using standard commercial confocal systems, at a fraction of the cost.

Confocal microscopy has become an invaluable tool in biology and the biomedical sciences, enabling rapid, high-sensitivity, and high-resolution optical ...

Time Multiplexing Super Resolving Technique for Imaging from a Moving Platform

We propose a method for increasing the resolution of an object and overcoming the diffraction limit of an optical system installed on top of a moving imaging system, such as an airborne platform or satellite. The resolution improvement is obtained in a two-step process. First, three low resolution differently defocused images are being captured and the optical phase is retrieved using an improved iterative Gerchberg-Saxton based algorithm. The phase retrieval allows to numerically back propagate the field to the aperture plane. Second, the imaging system is shifted and the first step is repeated. The obtained optical fields at the aperture plane are combined and a synthetically increased lens aperture is generated along the direction of movement, yielding higher imaging resolution. The method resembles a well-known approach from the microwave regime called the Synthetic Aperture Radar (SAR) in which the antenna size is synthetically increased along the platform propagation direction. The proposed method is demonstrated through laboratory experiment.

A method for overcoming the optical diffraction limit is presented. The method includes a two-step process: optical phase retrieval using iterative Gerchberg-Saxton algorithm, and imaging system shifting followed by repetition of the first step. A synthetically increased lens aperture is generated along the direction of movement, yielding higher imaging resolution.

We propose a method for increasing the resolution of an object and overcoming the diffraction limit of an optical system installed on top of a moving ...

Lensfree On-chip Tomographic Microscopy Employing Multi-angle Illumination and Pixel Super-resolution

Tomographic imaging has been a widely used tool in medicine as it can provide three-dimensional (3D) structural information regarding objects of different size scales. In micrometer and millimeter scales, optical microscopy modalities find increasing use owing to the non-ionizing nature of visible light, and the availability of a rich set of illumination sources (such as lasers and light-emitting-diodes) and detection elements (such as large format CCD and CMOS detector-arrays). Among the recently developed optical tomographic microscopy modalities, one can include optical coherence tomography, optical diffraction tomography, optical projection tomography and light-sheet microscopy. 1-6 These platforms provide sectional imaging of cells, microorganisms and model animals such as C. elegans, zebrafish and mouse embryos.
Existing 3D optical imagers generally have relatively bulky and complex architectures, limiting the availability of these equipments to advanced laboratories, and impeding their integration with lab-on-a-chip platforms and microfluidic chips. To provide an alternative tomographic microscope, we recently developed lensfree optical tomography (LOT) as a high-throughput, compact and cost-effective optical tomography modality. 7 LOT discards the use of lenses and bulky optical components, and instead relies on multi-angle illumination and digital computation to achieve depth-resolved imaging of micro-objects over a large imaging volume. LOT can image biological specimen at a spatial resolution of &lt;1 &mu;m x &lt;1 &mu;m x &lt;3 &mu;m in the x, y and z dimensions, respectively, over a large imaging volume of 15-100 mm3, and can be particularly useful for lab-on-a-chip platforms.

Lensfree optical tomography is a three-dimensional microscopy technique that offers a spatial resolution of &lt;1 &mu;m &times; &lt;1 &mu;m &times; &lt;3 &mu;m in x, y and z dimensions, respectively, over a large imaging-volume of 15-100 mm3, which can be particularly useful for integration with lab-on-a-chip platforms.

Tomographic imaging has been a widely used tool in medicine as it can provide three-dimensional (3D) structural information regarding objects of different ...

Compact Lens-less Digital Holographic Microscope for MEMS Inspection and Characterization

A micro-electro-mechanical-system (MEMS) is a widely used component in many industries, including energy, biotechnology, medical, communications, and automotive. However, effective inspection and characterization metrology systems are needed to ensure the functional reliability of MEMS. This study presents a system based on digital holography as a tool for MEMS metrology. Digital holography has gained increasing attention in the past 20 years. With the fast development and decreasing cost of sensor arrays, resolution of such systems has increased broadening potential applications. Thus, it has attracted attention from both research and industry sides as a potential reliable tool for industrial metrology. Indeed, by recording the interference pattern between an object beam (which contains sample height information) and a reference beam on a CCD camera, one can retrieve the quantitative phase information of an object. However, most of digital holographic systems are bulky and thus not easy to implement on industry production lines. The novelty of the system presented is that it is lens-less and thus very compact. In this study, it is shown that the Compact Digital Holographic Microscope (CDHM) can be used to evaluate several characteristics typically consider as criteria in MEMS inspections. The surface profiles of MEMS in both static and dynamic conditions are presented. Comparison with AFM is investigated to validate the accuracy of the CDHM.

We present a compact reflection digital holographic system (CDHM) for inspection and characterization of MEMS devices. A lens-less design using a diverging input wave providing natural geometrical magnification is demonstrated. Both static and dynamic studies are presented.

A micro-electro-mechanical-system (MEMS) is a widely used component in many industries, including energy, biotechnology, medical, communications, and ...

Rapid Acquisition of 3D Images Using High-resolution Episcopic Microscopy

High-resolution episcopic microscopic (HREM) technology enables rapid acquisition of high-resolution digital volumetric and three-dimensional (3D) morphometric data. Here, we describe the detailed protocol to image the entire mouse embryo. The protocol consists of four major sections: sample preparation, embedding, image acquisition and finally, 3D visualization. The technology requires specimens to be stained with a fluorescent dye, which can be problematic for large or dense specimens. To overcome this limitation, we have improved the existing protocol to enhance tissue penetration of the dye by pretreating the specimen with a solution containing urea and sodium dodecyl sulfate. The protocol uses only routine laboratory equipment and reagents for easy adaptation in standard laboratory settings. We show that the resulting high-resolution 3D images faithfully recapitulate the detailed morphologic features of the internal organs of mouse embryos, thereby permitting morphometric analyses. Together, we present a detailed and improved protocol using standard laboratory equipment to acquire high-resolution 3D images of small and large sized specimens.

We describe a detailed protocol using high-resolution episcopic microscopy to acquire three-dimensional (3D) images of mouse embryos. This improved protocol utilizes a modified tissue preparation method to enhance penetration of the fluorescent dye, thereby permitting morphometric analysis of both small and large-sized specimens.

High-resolution episcopic microscopic (HREM) technology enables rapid acquisition of high-resolution digital volumetric and three-dimensional (3D) ...

Developmental Biology

Recording Ultra-Realistic Full-Color Analog Holograms for Use in a Moving Hologram Display

This paper demonstrates a method to record a set of twelve ultra-realistic full-color analog holograms presenting the same brightness, transparency and homogeneous colors for the fabrication of a Fantatrope, a dynamic holographic 3D display, without the need for special viewing aids. The method involves the use of 3D printer technology, a single-beam full-color Denisyuk optical setup with three low-power lasers (red, green, and blue) and an iso-panchromatic high-sensitive silver-halide holographic emulsion specially designed for recording analog holograms without any diffusion. A cyclic animation is created with a 3D computer graphics program and different elements are 3D printed to form models for the holograms. Holograms are recorded with a full-color holographic setup and developed using two simple chemical baths. To prevent any emulsion thickness variations, the holograms are sealed with optical glue. Results confirm that all holograms recorded with this protocol present the same characteristics, which allow them to be used in the Fantatrope.

We present a protocol to record a set of ultra-realistic full-color analog holograms, showing the same brightness, transparency, and homogeneous colors, on ultra-fine-grain silver-halide holographic emulsions for the fabrication of a dynamic holographic 3D display.

This paper demonstrates a method to record a set of twelve ultra-realistic full-color analog holograms presenting the same brightness, transparency and ...

Visualization of Endosome Dynamics in Living Nerve Terminals with Four-dimensional Fluorescence Imaging

Four-dimensional (4D) light imaging has been used to study behavior of small structures within motor nerve terminals of the thin transversus abdominis muscle of the garter snake. Raw data comprises time-lapse sequences of 3D z-stacks. Each stack contains 4-20 images acquired with epifluorescence optics at focal planes separated by 400-1,500 nm. Steps in the acquisition of image stacks, such as adjustment of focus, switching of excitation wavelengths, and operation of the digital camera, are automated as much as possible to maximize image rate and minimize tissue damage from light exposure. After acquisition, a set of image stacks is deconvolved to improve spatial resolution, converted to the desired 3D format, and used to create a 4D "movie" that is suitable for variety of computer-based analyses, depending upon the experimental data sought. One application is study of the dynamic behavior of two classes of endosomes found in nerve terminals-macroendosomes (MEs) and acidic endosomes (AEs)-whose sizes (200-800 nm for both types) are at or near the diffraction limit. Access to 3D information at each time point provides several advantages over conventional time-lapse imaging. In particular, size and velocity of movement of structures can be quantified over time without loss of sharp focus. Examples of data from 4D imaging reveal that MEs approach the plasma membrane and disappear, suggesting that they are exocytosed rather than simply moving vertically away from a single plane of focus. Also revealed is putative fusion of MEs and AEs, by visualization of overlap between the two dye-containing structures as viewed in each three orthogonal projections.

Four-dimensional (4D) imaging is utilized to study the behavior and interactions among two types of endosomes in living vertebrate nerve terminals. Movement of these small structures is characterized in three dimensions, permitting confirmation of events such as endosome fusion and exocytosis.

Four-dimensional (4D) light imaging has been used to study behavior of small structures within motor nerve terminals of the thin transversus abdominis ...

Neuroscience

Three and Four-Dimensional Visualization and Analysis Approaches to Study Vertebrate Axial Elongation and Segmentation

Somitogenesis is a hallmark of vertebrate embryonic development. For years, researchers have been studying this process in a variety of organisms using a wide range of techniques encompassing ex vivo and in vitro approaches. However, most studies still rely on the analysis of two-dimensional (2D) imaging data, which limits proper evaluation of a developmental process like axial extension and somitogenesis involving highly dynamic interactions in a complex 3D space. Here we describe techniques that allow mouse live imaging acquisition, dataset processing, visualization and analysis in 3D and 4D to study the cells (e.g., neuromesodermal progenitors) involved in these developmental processes. We also provide a step-by-step protocol for optical projection tomography and whole-mount immunofluorescence microscopy in mouse embryos (from sample preparation to image acquisition) and show a pipeline that we developed to process and visualize 3D image data. We extend the use of some of these techniques and highlight specific features of different available software (e.g., Fiji/ImageJ, Drishti, Amira and Imaris) that can be used to improve our current understanding of axial extension and somite formation (e.g., 3D reconstructions). Altogether, the techniques here described emphasize the importance of 3D data visualization and analysis in developmental biology, and might help other researchers to better address 3D and 4D image data in the context of vertebrate axial extension and segmentation. Finally, the work also employs novel tools to facilitate teaching vertebrate embryonic development.

Here, we describe computational tools and methods that allow visualization and analysis of three and four-dimensional image data of mouse embryos in the context of axial elongation and segmentation, obtained by in toto optical projection tomography, and by live imaging and whole-mount immunofluorescence staining using multiphoton microscopy.

Somitogenesis is a hallmark of vertebrate embryonic development. For years, researchers have been studying this process in a variety of organisms using a ...

Digital Inline Holographic Microscopy (DIHM) of Weakly-scattering Subjects

Weakly-scattering objects, such as small colloidal particles and most biological cells, are frequently encountered in microscopy. Indeed, a range of techniques have been developed to better visualize these phase objects; phase contrast and DIC are among the most popular methods for enhancing contrast. However, recording position and shape in the out-of-imaging-plane direction remains challenging. This report introduces a simple experimental method to accurately determine the location and geometry of objects in three dimensions, using digital inline holographic microscopy (DIHM). Broadly speaking, the accessible sample volume is defined by the camera sensor size in the lateral direction, and the illumination coherence in the axial direction. Typical sample volumes range from 200 &#181;m&#160;x 200 &#181;m&#160;x 200 &#181;m using LED illumination, to 5 mm&#160;x 5 mm&#160;x 5 mm or larger using laser illumination. This illumination light is configured so that plane waves are incident on the sample. Objects in the sample volume then scatter light, which interferes with the unscattered light to form interference patterns perpendicular to the illumination direction. This image (the hologram) contains the depth information required for three-dimensional reconstruction, and can be captured on a standard imaging device such as a CMOS or CCD camera. The Rayleigh-Sommerfeld back propagation method is employed to numerically refocus microscope images, and a simple imaging heuristic based on the Gouy phase anomaly is used to identify scattering objects within the reconstructed volume. This simple but robust method results in an unambiguous, model-free measurement of the location and shape of objects in microscopic samples.

Digital Inline Holographic Microscopy DIHM of Weakly-scattering Subjects

The three-dimensional locations of weakly-scattering objects can be uniquely identified using digital inline holographic microscopy (DIHM), which involves a minor modification to a standard microscope. Our software uses a simple imaging heuristic coupled with Rayleigh-Sommerfeld back-propagation to yield the three-dimensional position and geometry of a microscopic phase object.

Weakly-scattering objects, such as small colloidal particles and most biological cells, are frequently encountered in microscopy. Indeed, a range of ...

High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

Summary

Explore More Videos

Chapters in this video

High Resolution 3D Imaging of Ex-Vivo Biological Samples by Micro CT

Video-rate Scanning Confocal Microscopy and Microendoscopy

Time Multiplexing Super Resolving Technique for Imaging from a Moving Platform

Lensfree On-chip Tomographic Microscopy Employing Multi-angle Illumination and Pixel Super-resolution

Compact Lens-less Digital Holographic Microscope for MEMS Inspection and Characterization

Rapid Acquisition of 3D Images Using High-resolution Episcopic Microscopy

Recording Ultra-Realistic Full-Color Analog Holograms for Use in a Moving Hologram Display

Visualization of Endosome Dynamics in Living Nerve Terminals with Four-dimensional Fluorescence Imaging

Three and Four-Dimensional Visualization and Analysis Approaches to Study Vertebrate Axial Elongation and Segmentation

Digital Inline Holographic Microscopy (DIHM) of Weakly-scattering Subjects