Name: high-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques
Uploaded: 2013-12-03T13:10:00
Duration: 11 min 34 s
Description: Watch this Scientific Journal Video about High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques at JoVE.com

This research was an accumulated effort that began more than 10 years ago when Dr. Zhang was a graduate student at Stony Brook University. The current and previous students in our team at Iowa State University have contributed tremendously toward advancing this technology to where it is today. This work was partially sponsored by National Science Foundation under project number CMMI 1150711, and the William and Virginia Binger Foundation.

1. System Configuration
A schematic of the system is shown in Figure 1.
<ol>
	<li>Generate the fringe patterns for projection. These can be prepared well in advance by using an image programming environment such as MATLAB, OpenCV, or QT. Construct the patterns according to the three-step phase-shifting algorithm in18. Make three images, shifted in phase offset from each other by 2&#960;/3. For binary defocusing, use a dithering technique to generate sinusoidal patterns using only black and white pixels as described in19.</li>
	<li>Select the digital light processing projector. A high-speed binary defocusing system requires a faster, specialized projector such as the DLP LightCommander with ALP High Speed module. Be sure to use the binary or monochromatic setting to project the fringe images. Since the image values are purely on or off, nonlinearity adjustments are not necessary. Utilize the projector&#8217;s software program to upload the patterns for phase shifting.</li>
	<li>Select the camera. Choose a black-and-white CCD or CMOS camera with the correct capture rate for the system. Avoid color cameras for 3D capture since color is not needed and color cameras require nonlinearity and gamma adjustments. Keep in mind that the camera will need to capture the entire set of fringe images for each 3D video frame. High-quality sinusoidal systems require precise synchronization between the projector and the camera; for binary defocusing systems this requirement is more relaxed20.</li>
	<li>Determine the desired maximum (x, y) range and the distance from the projector to the object (d0). Choose a range that makes sense for the application, but be sure that the area is slightly larger than the subject to reduce any optical boundary effects. Since the light output of a projector is a frustum, this (x, y) range will drive d0. Move the projector relative to a large flat projection surface until the desired (x, y) range is found, then measure d0 with a tape measure.</li>
	<li>Select the camera lens with the proper focal length. Using the camera&#8217;s sensor size, find the focal length such that the field of view at the distance d0 is the same as the desired imaging range (x, y).</li>
	<li>Determine the separation distance between the projector and the camera. A trade-off occurs here between noise and shadowing. At a large angle between these components, triangulation between feature points is more obvious, but more features get lost in shadow from the camera&#8217;s perspective. At a small angle, triangulation becomes difficult, increasing noise in the results. Typically, 10-15&#176; is a good compromise.</li>
</ol>
2. System Calibration
This reference plane calibration is the simplest and most readily accessible method of calibration for the system. Therefore, it is the best for getting started. More accurate calibration methods are available in the literature for specific sinusoidal16 and binary defocusing17 applications. For maximum accuracy, calibration should be performed just before data capture. After calibration, the camera and projector should not be displaced relative to each other.
<ol>
	<li>Focus/defocus the projector. Carefully defocus the projection lens until the patterns at the imaging plane resemble high-quality sinusoids. This may require an iterative process of examining the data quality (Section 4) and adjusting the lens.</li>
	<li>Capture fringe images of a reference plane. Place a flat white board at the focal plane of the projector and in the field of view of the camera. A 3/16 in&#160;(5 mm) thick white foam core board works well, provided that the surface facing the system is not shiny or significantly blemished or torn. Record and save fringe images of this board for the data processing step.</li>
	<li>Capture fringe images of a reference object of known dimensions. For this step, a rigid foam cube is one simple example. Cover the cube with squares of 1/16 in&#160;(1.5 mm) white adhesive foam to make it diffuse. Place it at the focal plane of the camera in the camera&#8217;s field of view and record fringe images for the processing step.</li>
</ol>
3. Data Acquisition
<ol>
	<li>Place the object or invite the subject to sit at the focal plane of the camera. For a human subject, warn him or her that the projector light may be bright. A black cloth backdrop can be used behind the subject to hide extraneous surroundings.</li>
	<li>Adjust the camera aperture to optimize the light level. Sample fringe images should be as bright as possible, but not saturated. Dark images will have too much noise, while saturated areas (significant regions of maximum brightness) in images will result in the loss of details in the saturated region.</li>
	<li>Capture the desired number of frames. Be sure to bring a hard drive large enough to hold all of the captured images for both the subject and the calibration datasets. With the .OBJ file format, a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480 could be over 50 GB.</li>
</ol>
4. Data Analysis and Visualization
With software optimized for speed such as our in-house GUI, this step can take place during data capture. Real-time processing allows the user to immediately detect if the resulting data is desirable for the application and adjust if necessary. However, post processing can be more flexible and higher in accuracy. Post processing is also much simpler to implement and the best place to begin.
<ol>
	<li>Compute the wrapped phase for both the calibration and subject data. In the three-step phase-shifting algorithm in18, phase describes the position of a point within the cosine function. Since we have three equations and three unknowns, we can solve the equations used to generate the fringe images in step 1.1 for the phase at each point. Because of the arctangent function, the computed phase is in the range (-&#960;, &#960;]; hence it is called &#8220;wrapped phase.&#8221; To improve the processing speed, we developed a fast phase wrapping algorithm that is discussed in21.</li>
	<li>Unwrap the phase maps. Adopt a phase-unwrapping algorithm that detects the 2&#960; jumps in the phase and removes them by adding or subtracting multiples of 2&#960;. We have used the fast algorithm in22 in previous systems to unwrap the phase robustly yet quickly. In the video, we demonstrate the multifrequency technique described in15, which uses additional sets of three phase-shifted patterns at differing frequencies. The wrapped phase maps from each set of three can be combined to robustly yield a single unwrapped phase map. This technique increases the depth range for accurate capture with binary defocusing.</li>
	<li>Optional: Compute the 2D texture. Averaging sets of three neighboring fringe images will wash out the fringe stripes and generate the 2D texture map. This can be mapped onto the 3D data during visualization if desired.</li>
	<li>Convert the unwrapped phase maps into depth. As described in23, depth can be computed for the calibration cube as the difference between the calibration cube phase map and the reference plane phase map. Compare this computed depth to the known depth to compute the correct depth scaling factor c0. Then, compute the depth for the subject by subtracting the reference plane phase from the subject&#8217;s phase and multiplying the results by c0.</li>
	<li>Compute the x- and y-coordinates. Apply the scaling factor c0 to the calibration cube depth map. Determine the conversion factor &#961; from the cube dimensions in pixels to the known cube dimensions in the xy plane. Multiply the pixel count in the subject data by &#961; to compute the x and y coordinates.</li>
	<li>Visualize the data. Individual frames can be saved in our in-house format and viewed with simple MATLAB code or saved in .OBJ format and viewed with a commercial 3D modeling program such as Blender. Due to the large amount of data in each frame, these applications may be sluggish on some computers. For more responsiveness or for live video display, write software using a computer graphics library such as OpenGL or Direct3D. This software can take advantage of the graphics processing unit (GPU) to rapidly generate the x, y, and z coordinates from the unwrapped phase, form a triangle mesh, compute lighting normals, and display the results. Using the GPU, we have achieved up to 30 Hz live 3D data visualization with approximately 300,000 points per frame.</li>
</ol>

<ol>
	<li>Salvi, J., Fernandez, S., Pribanic, T., Llado, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+state+of+the+art+in+structured+light+patterns+for+surface+profilometry.">A state of the art in structured light patterns for surface profilometry.</a> Patt. Recogn. 43, 2666-2680 (2010).</li><li>Gorthi, S. S., Rastogi, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fringe+projection+techniques:+Whither+we+are.">Fringe projection techniques: Whither we are.</a> Opt. Laser Eng. 48 (2), 133-140 (2010).</li><li>Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Recent+progresses+on+real-time+3-D+shape+measurement+using+digital+fringe+projection+techniques.">Recent progresses on real-time 3-D shape measurement using digital fringe projection techniques.</a> Opt. Laser Eng. 48 (2), 149-158 (2010).</li><li>Su, X., Zhang, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Dynamic+3-D+shape+measurement+method:+A+review.">Dynamic 3-D shape measurement method: A review.</a> Opt. Laser Eng. 48 (2), 191-204 (2010).</li><li>Geng, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Structured-light+3D+surface+imaging:+a+tutorial.">Structured-light 3D surface imaging: a tutorial.</a> Adv. Opt. Photonics. 3 (2), 128-160 (2011).</li><li>Zhang, S., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-resolution,+real-time+3-D+absolute+coordinate+measurement+based+on+a+phase-shifting+method.">High-resolution, real-time 3-D absolute coordinate measurement based on a phase-shifting method.</a> Opt. Express. 14 (7), 2644-2649 (2006).</li><li>Zhang, S., Royer, D., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=GPU-assisted+high-resolution,+real-time+3-D+shape+measurement.">GPU-assisted high-resolution, real-time 3-D shape measurement.</a> Opt. Express. 14 (20), 9120-9129 (2006).</li><li>Lei, S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Flexible+3-D+shape+measurement+using+projector+defocusing.">Flexible 3-D shape measurement using projector defocusing.</a> Opt. Lett. 34 (20), 3080-3082 (2009).</li><li>Mehta, R. P., Zhang, S., Hadlock, T. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Novel+3-D+video+for+quantification+of+facial+movement.">Novel 3-D video for quantification of facial movement.</a> Otolaryngol. Head Neck Surg. 138 (4), 468-472 (2008).</li><li>Wang, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High+resolution+acquisition,+learning+and+transfer+of+dynamic+3D+facial+expressions.">High resolution acquisition, learning and transfer of dynamic 3D facial expressions.</a> Comput. Graph. Forum. 23 (3), (2004).</li><li>Laughner, J. I., Zhang, S., Li, H., Efimov, I. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Mapping+cardiac+surface+mechanics+with+structured+light+imaging.">Mapping cardiac surface mechanics with structured light imaging.</a> Am. J. Physiol. Heart Circ. Physiol. , (2012).</li><li>Ekstrand, L., Wang, Y., Karpinsky, N., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Superfast+3D+profilometry+with+digital+fringe+projection+and+phase-shifting+techniques.">Superfast 3D profilometry with digital fringe projection and phase-shifting techniques.</a> Handbook of 3-D machine vision: Optical metrology and imaging. , (2012).</li><li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-resolution+real-time+three-dimensional+shape+measurement.">High-resolution real-time three-dimensional shape measurement.</a> Opt. Eng. 45 (12), 123-601 (2006).</li><li>Li, Y., Zhao, C., Qian, Y., Wang, H., Jin, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=High-speed+and+dense+three-dimensional+surface+acquisition+using+defocused+binary+patterns+for+spatially+isolated+objects.">High-speed and dense three-dimensional surface acquisition using defocused binary patterns for spatially isolated objects.</a> Opt. Express. 18, 21-635 (2010).</li><li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Superfast+multifrequency+phase-shifting+technique+with+optimal+pulse+width+modulation.">Superfast multifrequency phase-shifting technique with optimal pulse width modulation.</a> Opt. Express. 19 (6), 5143-5148 (2011).</li><li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Novel+method+for+structured+light+system+calibration.">Novel method for structured light system calibration.</a> Opt. Eng. 45 (8), 083601 (2006).</li><li>Merner, L., Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Accurate+calibration+for+3D+shape+measurement+system+using+a+binary+defocusing+technique.">Accurate calibration for 3D shape measurement system using a binary defocusing technique.</a> Opt. Laser Eng. 51 (5), 514-519 (2013).</li><li>Malacara, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Optical Shop Testing. , (2007).</li><li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+shape+measurement+with+binary+dithered+patterns.">Three-dimensional shape measurement with binary dithered patterns.</a> Appl. Opt. 51 (27), 6631-6636 (2012).</li><li>Ekstrand, L., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autoexposure+for+three-dimensional+shape+measurement+with+a+digital-light-processing+projector.">Autoexposure for three-dimensional shape measurement with a digital-light-processing projector.</a> Opt. Eng. 50 (12), 123-603 (2011).</li><li>Huang, P. S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Fast+three-step+phase-shifting+algorithm.">Fast three-step phase-shifting algorithm.</a> Appl. Opt. 45 (21), 5086-5091 (2006).</li><li>Zhang, S., Li, X., Yau, S. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multilevel+quality-guided+phase+unwrapping+algorithm+for+real-time+3-D+shape+reconstruction.">Multilevel quality-guided phase unwrapping algorithm for real-time 3-D shape reconstruction.</a> Appl. Opt. 46 (1), 50-57 (2007).</li><li>Xu, Y., Ekstrand, L., Dai, J., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Phase+error+compensation+for+3-D+shape+measurement+with+projector+defocusing.">Phase error compensation for 3-D shape measurement with projector defocusing.</a> Appl. Opt. 50 (18), 2572-2581 (2011).</li></ol>

The authors have no competing financial interests.

This high-resolution, real-time to superfast 3D video imaging technology is a platform technology that could potentially benefit numerous and diverse scientific fields ranging from biological science to engineering practice. Biomedical applications include precision measurements of facial movements and organ surfaces.&#160;Other applications include 3D automated quality control with detection of warped surface features; 3D enhanced videoconferencing; detailed digitization of facial features for movies and videogames; dense and rapid deformation measurements for the design and analysis of structures; and fluid surface characterization.&#160;Many biological and engineering applications (e.g. beating rabbit hearts, fluid shockwaves) require the superfast imaging rates of a binary defocusing system to correctly resolve features without aliasing artifacts.&#160;
Nevertheless, many challenges remain to the widespread adoption of this technology.&#160;Conventional DFP technology requires the projector to display 8-bit grayscale sinusoidal fringe patterns.&#160;The speed of this technique is limited by the projector&#8217;s refresh rate (typically 120 Hz).&#160;This speed is sufficient for slow motion capture such as that in facial expressions.&#160;However, numerous applications exist that require faster capture rates.
Binary defocusing technology has relaxed this speed limitation, and we have successfully created a superfast 3D video imaging system.&#160;However, this system has two drawbacks.&#160;First, it requires an expensive projector such as the DLP Discovery platform and a costly high-speed video camera such as the Vision Research Phantom v9.1.&#160;Second, since it generates sinusoidal patterns via the defocusing of squared binary patterns, the binary defocusing technique has difficulty generating sinusoidal fringes of the same quality as the traditional DFP technique and a reduced depth measurement range (for further explanation, see23).&#160;Recent investigation indicates that dithered binary sinusoidal patterns can significantly alleviate the limitations on depth measurement range19.&#160;Future research will focus on overcoming the remaining issues while preserving the merits of binary defocusing.&#160;
Another challenge is compressing and storing the large amount of data generated by high-speed, high-resolution 3D video imaging systems.&#160;Uncompressed 3D videos are drastically larger than uncompressed 2D videos.&#160;For instance, for a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480, the .OBJ file size could be over 50 GB, making it extremely difficult to store.&#160;Since little progress has been made in the 3D video compression field, we will continue to focus on this in the future.

Digital fringe projection (DFP) techniques are based upon correlation and triangulation between two views of the same scene at different angles, the same principle employed by the human eyes and brain to achieve stereo vision. However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the object that the object&#8217;s surface distorts in the camera&#8217;s view. Three such distorted patterns (fringe images) at differing phase shifts from each other can be analyzed to retrieve the depth via triangulation. The use of a known pattern eliminates the difficult computational problem of identifying correspondence points, allowing the capture of depth measurements at the camera resolution. For example, with a 576 x 576 camera, the technique can capture 331,776 points. This allows DFP systems to measure very fine details such as the movement of facial muscles in human emotions.
3D optical imaging techniques for static or quasi-static events have been extensively studied over the past few decades and have seen great success in video game design, animation, movies, music videos, virtual reality, telesurgery, and many engineering disciplines5. Though numerous 3D profilometry techniques exist, they can be classified into two categories: surface contact methods and surface noncontact methods. Both the coordinate measurement machine (CMM) and the atomic force microscope (AFM) require contact with the measuring surface to obtain 3D profiles at high accuracy. This requirement places severe restrictions on the speed of contact methods. They cannot reach kHz measurement speed with thousands of points per scan.
Surface noncontact techniques typically utilize optical triangulation methods (e.g.&#160;stereo vision, spacetime stereo, structured light). By actively projecting known patterns onto the objects, structured light techniques can be used to measure surfaces without strong local texture variations1. Fringe analysis is a special group of structured light techniques that uses sinusoidal structured patterns (also known as fringe patterns). Because these patterns have intensities that vary continuously from point to point in a known manner, they boost the structured light techniques from projector-pixel resolution to camera-pixel resolution12. In the recent past, fringe analysis techniques were instrumental in achieving high-resolution 3D imaging.
The digital fringe projection (DFP) technique uses digital video projectors to generate sinusoidal fringe patterns. This technique has the merits of lower cost, higher speed, and simplicity of development, and it has been a very active research area within the past decade. Recent developments in DFP and similar digital structured light techniques are summarized in1-5. To achieve high-speed applications, a digital-light-processing (DLP) projector is preferable due to its fundamental operation mechanism. The speed and flexibility of this technique has allowed us to acquire 3D video at 40 Hz&#160;13 and then later at 60 Hz&#160;6,7.
Nevertheless, a fundamental speed limit exists for the traditional DFP technique. A DLP projector can only swap 8-bit color images at its maximum refresh rate (typically 120 Hz). Since the traditional fringe patterns are 8-bit grayscale images, we can encode three of them into one color image as the red, green, and blue color channels. The projector will swap each channel (and therefore each fringe pattern) at three times the refresh rate (typically 360 Hz). However, since each 3D video frame requires three fringe patterns, the maximum rate of 3D video capture is still only the refresh rate (120 Hz)3,14. To break past this hardware limitation, we have invented a modified DFP technique that uses binary defocusing8. Instead of 8-bit grayscale fringe patterns, this technique uses computer-generated 1-bit binary structured patterns. These patterns are defocused using the projector lens to become pseudo-sinusoidal patterns for DFP. Because DLP projectors can display binary images orders-of-magnitude faster than 8-bit grayscale images, the binary defocusing technology permits tens of kilohertz&#160;3D video imaging speed with the same resolution as the conventional DFP techniques15.
The overall goal of the following protocol is to demonstrate the basic implementation and operation of a binary defocusing three-step phase-shifting DFP system. First, the protocol will cover the selection and integration of the necessary components. Then, it will discuss the simplest, most readily accessible method of calibration for the system; more complex calibration methods are available in the literature for specific applications16,17. The protocol will then focus on the procedure for 3D video capture with the system and the process for converting the fringe images into visualized 3D measurements. Finally, we will present some representative results from our real-time and high-speed systems.

No materials or equipment used.

high-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses triangulation between matching points in two views of the same scene at different angles to compute depth.&#160;However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the subject, and the surface of the subject distorts these patterns in the camera&#8217;s field of view. Three distorted patterns (fringe images) from the camera can be used to compute the depth using triangulation.
Unlike other 3D measurement methods, DFP techniques lead to systems that tend to be faster, lower in equipment cost, more flexible, and easier to develop.&#160;DFP systems can also achieve the same measurement resolution as the camera.&#160;For this reason, DFP and other digital structured light techniques have recently been the focus of intense research (as summarized in1-5). Taking advantage of DFP, the graphics processing unit, and optimized algorithms, we have developed a system capable of 30&#160;Hz 3D video data acquisition, reconstruction, and display for over 300,000 measurement points per frame6,7.&#160;Binary defocusing DFP methods can achieve even greater speeds8.
Diverse applications can benefit from DFP techniques. Our collaborators have used our systems for facial function analysis9, facial animation10, cardiac mechanics studies11, and fluid surface measurements, but many other potential applications exist.&#160;This video will teach the fundamentals of DFP techniques and illustrate the design and operation of a binary defocusing DFP system.

Figure 1 shows the schematic of the system.&#160;The high-speed binary defocusing system in this video consists of a Logic PD DLP LightCommander projector and a Phantom v9.1 CMOS camera.
Figure 2 presents a single frame from our 3D real-time system of a human face.&#160;This system uses a 640 x 480 camera.&#160;Thanks to the aforementioned known sinusoidal pattern, we can capture 640 x 480 = 307,200 measurements, enough resolution to record very fine details.
Figure 3 shows an example of measuring human facial expressions in 3D at 60 Hz. Here, four frames selected from a video sequence clearly demonstrate the capability of the real-time system to capture dynamic changes in finely detailed geometry.
Figure 4 demonstrates our live visualization software used in conjunction with our real-time binary defocusing 3D video system.&#160;The 3D captured video of the subject is displayed in real time on the computer monitor to his right.&#160;This software was written in C++ using the OpenGL library, GLSL, and QT.&#160;The computer used is a Lenovo laptop.
Figure 5 shows 3D frames from live rabbit heart measurement with our newly developed superfast binary defocusing system.&#160;This system can record 3D frames at 667 Hz with an image resolution of 576 x 576. A superfast rate is required to measure the heart surface without motion-induced artifacts.&#160;The heart measurement research is in collaboration with Prof. Igor Efimov at Washington University-St. Louis (see11 for further details); note that the rabbit was humanely killed and that the images were taken while the heart was still beating.
<img alt="Figure 1" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig1highres.jpg" src="/files/ftp_upload/50421/50421fig1.jpg" /> 
Figure 1.&#160;Layout of the 3D video imaging system. In this system, a high-speed DLP projector projects three binary dithered phase-shifted images in rapid succession onto the subject.&#160;A high-speed CMOS camera is used to capture the three fringe images one by one for computation of the depth.
<img alt="Figure 2" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig2highres.jpg" src="/files/ftp_upload/50421/50421fig2.jpg" /> 
Figure 2.&#160;3D measurements of a human face at a resolution of 640 x 480, revealing fine details. Left to right shows the simultaneously captured texture perfectly aligned with the geometry, a shaded view of the geometry, the wireframe view depicting the density of the points, a close-up view of the nose area, and a close-up view of the eye region.
<img alt="Figure 3" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig3highres.jpg" src="/files/ftp_upload/50421/50421fig3.jpg" /> 
Figure 3.&#160;Four selected frames from 3D video of the formation of a facial expression.&#160;The video was captured at 60 Hz with a resolution of 640 x 480.&#160;These frames highlight the geometric changes in the woman&#8217;s face as she moves from a neutral expression to a smile.
<img alt="Figure 4" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig4highres.jpg" src="/files/ftp_upload/50421/50421fig4.jpg" /> 
Figure 4.&#160;Live 3D video capturing, processing, and rendering. The 3D measurements are displayed in real time on the computer screen to the subject&#8217;s right.
<img alt="Figure 5" fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig5highres.jpg" src="/files/ftp_upload/50421/50421fig5.jpg" /> 
Figure 5.&#160;Capturing a live rabbit heart with our superfast 3D video imaging system. The heart is beating at approximately 200 beats/min. The 3D capture rate was 166 Hz with an image resolution of 576 x 576. See11 for further details.

Watch this Scientific Journal Video about High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques at JoVE.com

High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

This video describes the fundamentals of digital fringe projection techniques, which provide dense 3D measurements of dynamically changing surfaces.&#160;It also demonstrates the design and operation of a high-speed binary defocusing system based on these techniques.

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses ...

high-resolution-high-speed-three-dimensional-video-imaging-with

Research

JoVE Journal

Engineering

Process of Making Three-dimensional Microstructures using Vaporization of a Sacrificial Component

Vascular structures in natural systems are able to provide high mass transport through high surface areas and optimized structure. Few synthetic material fabrication techniques are able to mimic the complexity of these structures while maintaining scalability. The Vaporization of a Sacrificial Component (VaSC) process is able to do so. This process uses sacrificial fibers as a template to form hollow, cylindrical microchannels embedded within a matrix. Tin (II) oxalate (SnOx) is embedded within poly(lactic) acid (PLA) fibers which facilitates the use of this process. The SnOx catalyzes the depolymerization of the PLA fibers at lower temperatures. The lactic acid monomers are gaseous at these temperatures and can be removed from the embedded matrix at temperatures that do not damage the matrix. Here we show a method for aligning these fibers using micromachined plates and a tensioning device to create complex patterns of three-dimensionally arrayed microchannels. The process allows the exploration of virtually any arrangement of fiber topologies and structures.

The Vaporization of a Sacrificial Component (VaSC) process is used to fabricate microvascular structures. This procedure uses sacrificial poly(lactic) acid fibers to form hollow microchannels with precise 3D geometric positioning provided by laser micromachined guide plates.

Vascular structures in natural systems are able to provide high mass transport through high surface areas and optimized structure. Few synthetic material ...

Construction of a High Resolution Microscope with Conventional and Holographic Optical Trapping Capabilities

High resolution microscope systems with optical traps allow for precise manipulation of various refractive objects, such as dielectric beads 1 or cellular organelles 2,3, as well as for high spatial and temporal resolution readout of their position relative to the center of the trap. The system described herein has one such "traditional" trap operating at 980 nm. It additionally provides a second optical trapping system that uses a commercially available holographic package to simultaneously create and manipulate complex trapping patterns in the field of view of the microscope 4,5 at a wavelength of 1,064 nm. The combination of the two systems allows for the manipulation of multiple refractive objects at the same time while simultaneously conducting high speed and high resolution measurements of motion and force production at nanometer and piconewton scale.

The system described herein employs a traditional optical trap as well as an independent holographic optical trapping line, capable of creating and manipulating multiple traps. This allows for the creation of complex geometric arrangements of refractive particles while also permitting simultaneous high-speed, high-resolution measurements of the activity of biological enzymes.

High resolution microscope systems with optical traps allow for precise manipulation of various refractive objects, such as dielectric beads 1 or cellular ...

High-speed Particle Image Velocimetry Near Surfaces

Multi-dimensional and transient flows play a key role in many areas of science, engineering, and health sciences but are often not well understood. The complex nature of these flows may be studied using particle image velocimetry (PIV), a laser-based imaging technique for optically accessible flows. Though many forms of PIV exist that extend the technique beyond the original planar two-component velocity measurement capabilities, the basic PIV system consists of a light source (laser), a camera, tracer particles, and analysis algorithms. The imaging and recording parameters, the light source, and the algorithms are adjusted to optimize the recording for the flow of interest and obtain valid velocity data. 
Common PIV investigations measure two-component velocities in a plane at a few frames per second. However, recent developments in instrumentation have facilitated high-frame rate (&gt; 1 kHz) measurements capable of resolving transient flows with high temporal resolution. Therefore, high-frame rate measurements have enabled investigations on the evolution of the structure and dynamics of highly transient flows. These investigations play a critical role in understanding the fundamental physics of complex flows.
A detailed description for performing high-resolution, high-speed planar PIV to study a transient flow near the surface of a flat plate is presented here. Details for adjusting the parameter constraints such as image and recording properties, the laser sheet properties, and processing algorithms to adapt PIV for any flow of interest are included.

A procedure for studying transient flows near boundaries using high-resolution, high-speed particle image velocimetry (PIV) is described here. PIV is a non-intrusive measurement technique applicable to any optically accessible flow by optimizing several parameter constraints such as the image and recording properties, the laser sheet properties, and analysis algorithms.

Multi-dimensional and transient flows play a key role in many areas of science, engineering, and health sciences but are often not well understood. The ...

Real-Time DC-dynamic Biasing Method for Switching Time Improvement in Severely Underdamped Fringing-field Electrostatic MEMS Actuators

Mechanically underdamped electrostatic fringing-field MEMS actuators are well known for their fast switching operation in response to a unit step input bias voltage. However, the tradeoff for the improved switching performance is a relatively long settling time to reach each gap height in response to various applied voltages. Transient applied bias waveforms are employed to facilitate reduced switching times for electrostatic fringing-field MEMS actuators with high mechanical quality factors. Removing the underlying substrate of the fringing-field actuator creates the low mechanical damping environment necessary to effectively test the concept. The removal of the underlying substrate also a has substantial improvement on the reliability performance of the device in regards to failure due to stiction. Although DC-dynamic biasing is useful in improving settling time, the required slew rates for typical MEMS devices may place aggressive requirements on the charge pumps for fully-integrated on-chip designs. Additionally, there may be challenges integrating the substrate removal step into the back-end-of-line commercial CMOS processing steps. Experimental validation of fabricated actuators demonstrates an improvement of 50x in switching time when compared to conventional step biasing results. Compared to theoretical calculations, the experimental results are in good agreement.

The robust device design of fringing-field electrostatic MEMS actuators results in inherently low squeeze-film damping conditions and long settling times when performing switching operations using conventional step biasing. Real-time switching time improvement with DC-dynamic waveforms reduces the settling time of fringing-field MEMS actuators when transitioning between up-to-down and down-to-up states.

Mechanically underdamped electrostatic fringing-field MEMS actuators are well known for their fast switching operation in response to a unit step input ...

Visualization of High Speed Liquid Jet Impaction on a Moving Surface

Two apparatuses for examining liquid jet impingement on a high-speed moving surface are described: an air cannon device (for examining surface speeds between 0 and 25 m/sec) and a spinning disk device (for examining surface speeds between 15 and 100 m/sec). The air cannon linear traverse is a pneumatic energy-powered system that is designed to accelerate a metal rail surface mounted on top of a wooden projectile. A pressurized cylinder fitted with a solenoid valve rapidly releases pressurized air into the barrel, forcing the projectile down the cannon barrel. The projectile travels beneath a spray nozzle, which impinges a liquid jet onto its metal upper surface, and the projectile then hits a stopping mechanism. A camera records the jet impingement, and a pressure transducer records the spray nozzle backpressure. The spinning disk set-up consists of a steel disk that reaches speeds of 500 to 3,000 rpm via a variable frequency drive (VFD) motor. A spray system similar to that of the air cannon generates a liquid jet that impinges onto the spinning disc, and cameras placed at several optical access points record the jet impingement. Video recordings of jet impingement processes are recorded and examined to determine whether the outcome of impingement is splash, splatter, or deposition. The apparatuses are the first that involve the high speed impingement of low-Reynolds-number liquid jets on high speed moving surfaces. In addition to its rail industry applications, the described technique may be used for technical and industrial purposes such as steelmaking and may be relevant to high-speed 3D printing.

Two experimental devices for examining liquid jet impingement on a high-speed moving surface are described: an air cannon device and a spinning disk device. The apparatuses are used to determine optimal approaches to the application of liquid friction modifier (LFM) onto rail tracks for top-of-rail friction control.

Two apparatuses for examining liquid jet impingement on a high-speed moving surface are described: an air cannon device (for examining surface speeds ...

Using Synchrotron Radiation Microtomography to Investigate Multi-scale Three-dimensional Microelectronic Packages

Synchrotron radiation micro-tomography (SR&#181;T) is a non-destructive three-dimensional (3D) imaging technique that offers high flux for fast data acquisition times with high spatial resolution. In the electronics industry there is serious interest in performing failure analysis on 3D microelectronic packages, many which contain multiple levels of high-density interconnections. Often in tomography there is a trade-off between image resolution and the volume of a sample that can be imaged. This inverse relationship limits the usefulness of conventional computed tomography (CT) systems since a microelectronic package is often large in cross sectional area 100-3,600 mm2, but has important features on the micron scale. The micro-tomography beamline at the Advanced Light Source (ALS), in Berkeley, CA USA, has a setup which is adaptable and can be tailored to a sample&#39;s properties, i.e., density, thickness, etc., with a maximum allowable cross-section of 36 x 36 mm. This setup also has the option of being either monochromatic in the energy range ~7-43 keV or operating with maximum flux in white light mode using a polychromatic beam. Presented here are details of the experimental steps taken to image an entire 16 x 16 mm system within a package, in order to obtain 3D images of the system with a spatial resolution of 8.7 &#181;m all within a scan time of less than 3 min. Also shown are results from packages scanned in different orientations and a sectioned package for higher resolution imaging. In contrast a conventional CT system would take hours to record data with potentially poorer resolution. Indeed, the ratio of field-of-view to throughput time is much higher when using the synchrotron radiation tomography setup. The description below of the experimental setup can be implemented and adapted for use with many other multi-materials.

For this study synchrotron radiation micro-tomography, a non-destructive three-dimensional imaging technique, is employed to investigate an entire microelectronic package with a cross-sectional area of 16 x 16 mm. Due to the synchrotron&#39;s high flux and brightness the sample was imaged in just 3 min&#160;with an 8.7 &#181;m spatial resolution.

Synchrotron radiation micro-tomography (SR&#181;T) is a non-destructive three-dimensional (3D) imaging technique that offers high flux for fast data ...

High Resolution Phonon-assisted Quasi-resonance Fluorescence Spectroscopy

High resolution optical spectroscopy methods are demanding in terms of either technology, equipment, complexity, time or a combination of these. Here we demonstrate an optical spectroscopy method that is capable of resolving spectral features beyond that of the spin fine structure and homogeneous linewidth of single quantum dots (QDs) using a standard, easy-to-use spectrometer setup. This method incorporates both laser and photoluminescence spectroscopy, combining the advantage of laser line-width limited resolution with multi-channel photoluminescence detection. Such a scheme allows for considerable improvement of resolution over that of a common single-stage spectrometer. The method uses phonons to assist in the measurement of the photoluminescence of a single quantum dot after resonant excitation of its ground state transition. The phonon&#39;s energy difference allows one to separate and filter out the laser light exciting the quantum dot. An advantageous feature of this method is its straight forward integration into standard spectroscopy setups, which are accessible to most researchers.

The manuscript describes a method of phonon-assisted quasi-resonant fluorescence spectroscopy that incorporates both laser-limited resolution and photoluminescence (PL) spectroscopy. This method utilizes optical phonons to provide linewidth-limited resolution spectra of atom-like semiconductor structures in the energy domain. The method is also easily realized with a single spectrometer optical spectroscopy setup.

High resolution optical spectroscopy methods are demanding in terms of either technology, equipment, complexity, time or a combination of these. Here we ...

High-resolution Thermal Micro-imaging Using Europium Chelate Luminescent Coatings

Micro-electronic devices often undergo significant self-heating when biased to their typical operating conditions. This paper describes a convenient optical micro-imaging technique which can be used to map and quantify such behavior. Europium thenoyltrifluoroacetonate (EuTFC) has a 612 nm luminescence line whose activation efficiency drops strongly with increasing temperature, due to T-dependent interactions between the Eu3+ ion and the organic chelating compound. This material may be readily coated on to a sample surface by thermal sublimation in vacuum. When the coating is excited with ultraviolet light (337 nm) an optical micro-image of the 612 nm luminescent response can be converted directly into a map of the sample surface temperature. This technique offers spatial resolution limited only by the microscope optics (about 1 micron) and time resolution limited by the speed of the camera employed. It offers the additional advantages of only requiring comparatively simple and non-specialized equipment, and giving a quantitative probe of sample temperature.

Europium thenoyltrifluoroacetonate (EuTFC) has an optical luminescence line at 612 nm, whose activation efficiency decreases strongly with temperature. If a sample coated with a thin film of this material is micro-imaged, the 612 nm luminescent response intensity may be converted into a direct map of sample surface temperature.

Micro-electronic devices often undergo significant self-heating when biased to their typical operating conditions. This paper describes a convenient ...

High-speed Continuous-wave Stimulated Brillouin Scattering Spectrometer for Material Analysis

Recent years have witnessed a significant increase in the use of spontaneous Brillouin spectrometers for non-contact analysis of soft matter, such as aqueous solutions and biomaterials, with fast acquisition times. Here, we discuss the assembly and operation of a Brillouin spectrometer that uses stimulated Brillouin scattering (SBS) to measure stimulated Brillouin gain (SBG) spectra of water and lipid emulsion-based tissue-like samples in transmission mode with &lt;10 MHz spectral-resolution and &lt;35 MHz Brillouin-shift measurement precision at &lt;100 ms. The spectrometer consists of two nearly counter-propagating continuous-wave (CW) narrow-linewidth lasers at 780 nm whose frequency detuning is scanned through the material Brillouin shift. By using an ultra-narrowband hot rubidium-85 vapor notch filter and a phase-sensitive detector, the signal-to-noise-ratio of the SBG signal is significantly enhanced compared to that obtained with existing CW-SBS spectrometers. This improvement enables measurement of SBG spectra with up to 100-fold faster acquisition times, thereby facilitating high spectral-resolution and high-precision Brillouin analysis of soft materials at high speed.

We describe the construction of a rapid continuous-wave-stimulated-Brillouin-scattering (CW-SBS) spectrometer. The spectrometer employs single-frequency diode-lasers and an atomic vapor notch-filter to acquire transmission spectra of turbid/non-turbid samples with high spectral-resolution at speeds up to 100-fold faster than those of existing CW-SBS spectrometers. This improvement enables high-speed Brillouin material analysis.

Recent years have witnessed a significant increase in the use of spontaneous Brillouin spectrometers for non-contact analysis of soft matter, such as ...

Three-Dimensional Ultrasonic Needle Tip Tracking with a Fiber-Optic Ultrasound Receiver

Ultrasound is frequently used for guiding minimally invasive procedures, but visualizing medical devices is often challenging with this imaging modality. When visualization is lost, the medical device can cause trauma to critical tissue structures. Here, a method to track the needle tip during ultrasound image-guided procedures is presented. This method involves the use of a fiber-optic ultrasound receiver that is affixed within the cannula of a medical needle to communicate ultrasonically with the external ultrasound probe. This custom probe comprises a central transducer element array and side element arrays. In addition to conventional two-dimensional (2D) B-mode ultrasound imaging provided by the central array, three-dimensional (3D) needle tip tracking is provided by the side arrays. For B-mode ultrasound imaging, a standard transmit-receive sequence with electronic beamforming is performed. For ultrasonic tracking, Golay-coded ultrasound transmissions from the 4 side arrays are received by the hydrophone sensor, and subsequently the received signals are decoded to identify the needle tip&#39;s spatial location with respect to the ultrasound imaging probe. As a preliminary validation of this method, insertions of the needle/hydrophone pair were performed in clinically realistic contexts. This novel ultrasound imaging/tracking method is compatible with current clinical workflow, and it provides reliable device tracking during in-plane and out-of-plane needle insertions.

Accurate and efficient visualization of invasive medical devices is extremely important in many ultrasound-guided minimally invasive procedures. Here, a method for localizing the spatial position of a needle tip relative to the ultrasound imaging probe is presented.

Ultrasound is frequently used for guiding minimally invasive procedures, but visualizing medical devices is often challenging with this imaging modality. ...