Name: robotized testing of camera positions to determine ideal configuration for stereo 3d visualization of open-heart surgery
Uploaded: 2021-08-12T14:45:00
Description: Watch this Scientific Journal Video about Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery at JoVE.com

The research was carried out with funding from Vinnova (2017-03728, 2018-05302 and 2018-03651), Heart-Lung Foundation (20180390), Family Kamprad Foundation (20190194), and Anna-Lisa and Sven Eric Lundgren Foundation (2017 and 2018).

The experiments were approved by the local Ethics Committee in Lund, Sweden. The participation was voluntary, and the patients&#39; legal guardians provided informed written consent.
1. Robot setup and configuration
NOTE: This experiment used a dual-arm collaborative industrial robot and the standard control panel with a touch display. The robot is controlled with RobotWare 6.10.01 controller software and robot integrated development environment (IDE) RobotStudio 2019...

<ol>
	<li>Held, R. T., Hui, T. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+guide+to+stereoscopic+3D+displays+in+medicine.">A guide to stereoscopic 3D displays in medicine.</a> Academic Radiology. 18 (8), 1035-1048 (2011).</li><li>van Beurden, M. H. P. H., IJsselsteijn, W. A., Juola, J. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effectiveness+of+stereoscopic+displays+in+medicine:+A+review.">Effectiveness of stereoscopic displays in medicine: A review.</a> 3D Research. 3 (1), 1-13 (2012).</li><li>Luursema, J. M., Verwey, W. B., Kommers, P. A. M., Geelkerken, R. H., Vos, H. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Optimizing+conditions+for+computer-assisted+anatomical+learning.">Optimizing conditions for computer-assisted anatomical learning.</a> Interacting with Computers. 18 (5), 1123-1138 (2006).</li><li>Takano, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Usefulness+and+capability+of+three-dimensional,+full+high-definition+movies+for+surgical+education.">Usefulness and capability of three-dimensional, full high-definition movies for surgical education.</a> Maxillofacial Plastic and Reconstructive Surgery. 39 (1), 10 (2017).</li><li>Triepels, C. P. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Does+three-dimensional+anatomy+improve+student+understanding.">Does three-dimensional anatomy improve student understanding.</a> Clinical Anatomy. 33 (1), 25-33 (2020).</li><li>Beermann, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+visualisation+improves+understanding+of+surgical+liver+anatomy.">Three-dimensional visualisation improves understanding of surgical liver anatomy.</a> Medical Education. 44 (9), 936-940 (2010).</li><li>Battulga, B., Konishi, T., Tamura, Y., Moriguchi, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+Effectiveness+of+an+interactive+3-dimensional+computer+graphics+model+for+medical+education.">The Effectiveness of an interactive 3-dimensional computer graphics model for medical education.</a> Interactive Journal of Medical Research. 1 (2), (2012).</li><li>Yammine, K., Violato, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+meta-analysis+of+the+educational+effectiveness+of+three-dimensional+visualization+technologies+in+teaching+anatomy.">A meta-analysis of the educational effectiveness of three-dimensional visualization technologies in teaching anatomy.</a> Anatomical Sciences Education. 8 (6), 525-538 (2015).</li><li>Fitzgerald, J. E. F., White, M. J., Tang, S. W., Maxwell-Armstrong, C. A., James, D. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Are+we+teaching+sufficient+anatomy+at+medical+school?+The+opinions+of+newly+qualified+doctors.">Are we teaching sufficient anatomy at medical school? The opinions of newly qualified doctors.</a> Clinical Anatomy. 21 (7), 718-724 (2008).</li><li>Bergman, E. M., Van Der Vleuten, C. P. M., Scherpbier, A. J. J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Why+don't+they+know+enough+about+anatomy?+A+narrative+review.">Why don't they know enough about anatomy? A narrative review.</a> Medical Teacher. 33 (5), 403-409 (2011).</li><li>Terzi&#263;, K., Hansard, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Methods+for+reducing+visual+discomfort+in+stereoscopic+3D:+A+review.">Methods for reducing visual discomfort in stereoscopic 3D: A review.</a> Signal Processing: Image Communication. 47, 402-416 (2016).</li><li>Fan, Z., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=3D+interactive+surgical+visualization+system+using+mobile+spatial+information+acquisition+and+autostereoscopic+display.">3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display.</a> Journal of Biomedical Informatics. 71, 154-164 (2017).</li><li>Fan, Z., Zhang, S., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=3D+quantitative+evaluation+system+for+autostereoscopic+display.">3D quantitative evaluation system for autostereoscopic display.</a> Journal of Display Technology. 12 (10), 1185-1196 (2016).</li><li>McIntire, J. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Binocular+fusion+ranges+and+stereoacuity+predict+positional+and+rotational+spatial+task+performance+on+a+stereoscopic+3D+display.">Binocular fusion ranges and stereoacuity predict positional and rotational spatial task performance on a stereoscopic 3D display.</a> Journal of Display Technology. 11 (11), 959-966 (2015).</li><li>Kalia, M., Navab, N., Fels, S. S., Salcudean, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+method+to+introduce+&amp;+evaluate+motion+parallax+with+stereo+for+medical+AR/MR.">A method to introduce &amp; evaluate motion parallax with stereo for medical AR/MR.</a> IEEE Conference on Virtual Reality and 3D User Interfaces. , 1755-1759 (2019).</li><li>Kyt&#246;, M., Hakala, J., Oittinen, P., H&#228;kkinen, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Effect+of+camera+separation+on+the+viewing+experience+of+stereoscopic+photographs.">Effect of camera separation on the viewing experience of stereoscopic photographs.</a> Journal of Electronic Imaging. 21 (1), 1-9 (2012).</li><li>Moorthy, A. K., Su, C. C., Mittal, A., Bovik, A. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Subjective+evaluation+of+stereoscopic+image+quality.">Subjective evaluation of stereoscopic image quality.</a> Signal Processing: Image Communication. 28 (8), 870-883 (2013).</li><li>Yilmaz, G. N. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+depth+perception+evaluation+metric+for+immersive+3D+video+services.">A depth perception evaluation metric for immersive 3D video services.</a> 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video. , 1-4 (2017).</li><li>Lebreton, P., Raake, A., Barkowsky, M., Le Callet, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Evaluating+depth+perception+of+3D+stereoscopic+videos.">Evaluating depth perception of 3D stereoscopic videos.</a> IEEE Journal on Selected Topics in Signal Processing. 6, 710-720 (2012).</li><li>L&#243;pez, J. P., Rodrigo, J. A., Jim&#233;nez, D., Men&#233;ndez, J. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stereoscopic+3D+video+quality+assessment+based+on+depth+maps+and+video+motion.">Stereoscopic 3D video quality assessment based on depth maps and video motion.</a> EURASIP Journal on Image and Video Processing. 2013 (1), 62 (2013).</li><li>Banks, M. S., Read, J. C., Allison, R. S., Watt, S. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Stereoscopy+and+the+human+visual+system.">Stereoscopy and the human visual system.</a> SMPTE Motion Imaging Journal. 121 (4), 24-43 (2012).</li><li>Kyt&#246;, M., Nuutinen, M., Oittinen, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Method+for+measuring+stereo+camera+depth+accuracy+based+on+stereoscopic+vision.">Method for measuring stereo camera depth accuracy based on stereoscopic vision.</a> Three-Dimensional Imaging, Interaction, and Measurement. 7864, 168-176 (2011).</li><li>Kang, Y. S., Ho, Y. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Geometrical+compensation+algorithm+of+multiview+image+for+arc+multi-camera+arrays.">Geometrical compensation algorithm of multiview image for arc multi-camera arrays.</a> Advances in Multimedia Information Processing. 2008, 543-552 (2008).</li><li>Product Specification IRB 14000. DocumentID: 3HAC052982-001 Revision J. ABB Robotics Available from: <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2018)</li><li>Operating Manual IRB 14000. Document ID: 3HAC052986-001 Revision F. ABB Robotics Available from: <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2019)</li><li>. Github repository Available from: <a target="_blank" href="https://github.com/majstenmark/stereo2">https://github.com/majstenmark/stereo2</a> (2021)</li><li>Operating manual RobotStudio. Document ID: 3HAC032104-001 Revision Y. ABB Robotics Available from: <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2019)</li><li>Won, C. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Adaptive+interpolation+for+3D+stereoscopic+video+in+frame-compatible+top-bottom+packing.">Adaptive interpolation for 3D stereoscopic video in frame-compatible top-bottom packing.</a> IEEE International Conference on Consumer Electronics. 2011, 179-180 (2011).</li><li>Kim, S. K., Lee, C., Kim, K. T., Javidi, B., Okano, F., Son, J. Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multi-view+image+acquisition+and+display.">Multi-view image acquisition and display.</a> Three-Dimensional Imaging, Visualization, and Display. , 227-249 (2009).</li><li>Liu, F., Niu, Y., Jin, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Keystone+correction+for+stereoscopic+cinematography.">Keystone correction for stereoscopic cinematography.</a> IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012, 1-7 (2012).</li><li>Kang, W., Lee, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Horizontal+parallax+distortion+correction+method+in+toed-in+camera+with+wide-angle+lens.">Horizontal parallax distortion correction method in toed-in camera with wide-angle lens.</a> 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video. 2009, 1-4 (2009).</li></ol>

During live surgery, the total time of the experiment used for 3D video data collection was limited to be safe for the patient. If the object is unfocused or overexposed, the data cannot be used. The critical steps are during camera tool calibration and setup (step 2). The camera aperture and focus cannot be changed when the surgery has started; the same lighting conditions and distance should be used during setup and surgery. The camera calibration in steps 2.1-2.4 must be carried out carefully to ensure that the heart ...

In surgery, 3D visualization can be used for education, diagnoses, pre-operative planning, and post-operative evaluation1,2. Realistic depth perception can improve understanding3,4,5,6 of normal and abnormal anatomies. Simple 2D video recordings of surgical procedures are a good start. However, the lack of depth perception can make it hard for the non-surgical colleagues to fully understand the antero-posterior relationships between different anatomical structures and therefore also introduce a risk of misinterpretation of the anatomy7,8,9,10.
The 3D viewing experience is affected by five factors: (1) Camera configuration can either be parallel or toed-in as shown in Figure 1, (2) Baseline distance (the separation between the cameras). (3) Distance to the object of interest and other scene characteristics such as the background. (4) Characteristics of viewing devices such as screen size and viewing position1,11,12,13. (5) Individual preferences of the viewers14,15.
Designing a 3D camera setup begins with the capture of test videos recorded at various camera baseline distances and configurations to be used for subjective or automatic evaluation16,17,18,19,20. The camera distance must be constant to the surgical field to capture sharp images. Fixed focus is preferred because autofocus will adjust to focus on hands, instruments, or heads that may come into view. However, this is not easily achievable when the scene of interest is the surgical field. Operating rooms are restricted access areas because these facilities must be kept clean and sterile. Technical equipment, surgeons, and scrub nurses are often clustered closely around the patient to secure a good visual overview and an efficient workflow. To compare and evaluate the effect of camera positions on the 3D viewing experience, one complete test range of camera positions should be recording the same scene because the object characteristics such as shape, size, and color can affect the 3D viewing experience21.
For the same reason, complete test ranges of camera positions should be repeated on different surgical procedures. The entire sequence of positions must be repeated with high accuracy. In a surgical setting, existing methods that require either manual adjustment of the baseline distance22 or different camera pairs with fixed baseline distances23 are not feasible because of both space and time constraints. To address this challenge, this robotized solution was designed.
The data was collected with a dual-arm collaborative industrial robot mounted in the ceiling in the operating room. Cameras were attached to the wrists of the robot and moved along an arc-shaped trajectory with increasing baseline distance, as shown in Figure 2.
To demonstrate the approach, 10 test series were recorded from 4 different patients with 4 different congenital heart defects. Scenes were chosen when a pause in surgery was feasible: with the beating hearts just before and after surgical repair. Series were also made when the hearts were arrested. The surgeries were paused for 3 min and 20 s to collect forty 5-ssequences with different camera convergence distances and baseline distances to capture the scene. The videos were later post-processed, displayed in 3D for the clinical team, who rated how realistic the 3D video was along a scale from 0-5.
The convergence point for toed-in stereo cameras is where the center points of both images meet. The convergence point can, by principle, be placed either in front, within, or behind the object, see Figure 1A-C. When the convergence point is in front of the object, the object will be captured and displayed left of the midline for the left camera image and right of the midline for the right camera image (Figure 1A). The opposite applies when the convergence point is behind the object (Figure 1B). When the convergence point is on the object, the object will also appear in the midline of the camera images (Figure 1C), which presumably should yield the most comfortable viewing since no squinting is required to merge the images. To achieve comfortable stereo 3D video, the convergence point must be located on, or slightly behind, the object of interest, else the viewer is required to voluntarily squint outwards (exotropia).
The data was collected using a dual-arm collaborative industrial robot to position the cameras (Figure 2A-B). The robot weighs 38 kg without equipment. The robot is intrinsically safe; when it detects an unexpected impact, it stops moving. The robot was programmed to position the 5 Megapixel cameras with C-mount lenses along an arc-shaped trajectory stopping at predetermined baseline distances (Figure 2C). The cameras were attached to the robot hands using adaptor plates, as shown in Figure 3. Each camera recorded at 25 frames per second. Lenses were set at f-stop 1/8 with focus fixed on the object of interest (approximated geometrical center of the heart). Every image frame had a timestamp which was used to synchronize the two video streams.
Offsets between the robot wrist and the camera were calibrated. This can be achieved by aligning the crosshairs of the camera images, as shown in Figure 4. In this setup, the total translational offset from the mounting point on the robot wrist and the center of the camera image sensor was 55.3 mm in the X-direction and 21.2 mm in the Z-direction, displayed in Figure 5. The rotational offsets were calibrated at a convergence distance of 1100 mm and a baseline distance of 50 mm and adjusted manually with the joystick on the robot control panel. The robot in this study had a specified accuracy of 0.02 mm in Cartesian space and 0.01 degrees rotational resolution24. At a radius of 1100 m, an angle difference of 0.01 degrees offsets the center point 0.2 mm. During the full robot motion from 50-240 mm separation, the crosshair for each camera was within 2 mm from the ideal center of convergence.
The baseline distance was increased stepwise by symmetrical separation of the cameras around the center of the field of view in increments of 10 mm ranging from 50-240 mm (Figure 2). The cameras were kept at a standstill for 5 s in each position and moved between the positions at a velocity of 50 mm/s. The convergence point could be adjusted in X and Z directions using a graphical user interface (Figure 6). The robot followed accordingly within its working range.
The accuracy of the convergence point was estimated using the uniform triangles and the variable names in Figure 7A and B. The height &#39;z&#39; was calculated from the convergence distance &#39;R&#39; with the Pythagorean theorem as
<img alt="Equation 1" src="/files/ftp_upload/62786/62786eq01.jpg" />
When the real convergence point was closer than the desired point, as shown in Figure 7A, the error distance &#39;f1&#39; was calculated as
<img alt="Equation 2" src="/files/ftp_upload/62786/62786eq02.jpg" />
Similarly, when the convergence point was distal to the desired point, the error distance &#39;f2&#39; was calculated as
<img alt="Equation 3" src="/files/ftp_upload/62786/62786eq03.jpg" />
Here, &#39;e&#39; was the maximum separation between the crosshairs, at most 2 mm at maximum baseline separation during calibration (D = 240 mm). For R = 1100 mm (z = 1093 mm), the error was less than &#177; 9.2 mm. For R = 1400 mm (z = 1395 mm), the error was &#177; 11.7 mm. That is, the error of the placement of the convergence point was within 1% of the desired. The two test distances of 1100 mm and 1400 mm were therefore well separated.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>2 C-mount lenses (35 mm F2.1, 5 M pixel)</td>
			<td>Tamron</td>
			<td>M112FM35</td>
			<td>Rated for 5 Mpixel</td>
		</tr>
		<tr>
			<td>3D glasses (DLP-link active shutter)</td>
			<td>Celexon</td>
			<td>G1000</td>
			<td>Any compatible 3D glasses can be used</td>
		</tr>
		<tr>
			<td>3D Projector</td>
			<td>Viewsonic</td>
			<td>X10-4K</td>
			<td>Displays 3D in 1080, can be exchanged for other 3D projectors</td>
		</tr>
		<tr>
			<td>6 M2 x 8 screws</td>
			<td></td>
			<td></td>
			<td>To attach the cXimea cameras to the camera adaptor plates</td>
		</tr>
		<tr>
			<td>8 M2.5 x 8 screws</td>
			<td></td>
			<td></td>
			<td>To attach the circular mounting plates to the robot wrist</td>
		</tr>
		<tr>
			<td>8 M5 x 40 screws</td>
			<td></td>
			<td></td>
			<td>To mount the robot</td>
		</tr>
		<tr>
			<td>8 M6 x 10 screws with flat heads</td>
			<td></td>
			<td></td>
			<td>For attaching the circular mounting plate and the camera adaptor plates</td>
		</tr>
		<tr>
			<td>Calibration checker board plate (25 by 25 mm)</td>
			<td></td>
			<td></td>
			<td>Any standard checkerboard can be used, including printed, as long as the grid is clearly visible in the cameras</td>
		</tr>
		<tr>
			<td>Camera adaptor plates, x2</td>
			<td></td>
			<td></td>
			<td>Designed by the authors in robot_camera_adaptor_plates.dwg, milled in aluminium.</td>
		</tr>
		<tr>
			<td>Circular mounting plates, x2</td>
			<td></td>
			<td></td>
			<td>Distributed with the permission of the designer Julius Klein and printed with ABS plastic on an FDM 3D printer. License Tecnalia Research &#38; Innovation 2017. Attached as Mountingplate_ROBOT_SIDE_ 
			NewDesign_4.stl</td>
		</tr>
		<tr>
			<td>Fix focus usb cameras, x2 (5 Mpixel)</td>
			<td>Ximea</td>
			<td>MC050CG-SY-UB</td>
			<td>With Sony IMX250LQR sensor</td>
		</tr>
		<tr>
			<td>Flexpendant</td>
			<td>ABB</td>
			<td>3HAC028357-001</td>
			<td>robot touch display</td>
		</tr>
		<tr>
			<td>Liveview</td>
			<td></td>
			<td></td>
			<td>recording application</td>
		</tr>
		<tr>
			<td>RobotStudio</td>
			<td></td>
			<td></td>
			<td>&#160;robot integrated development environment (IDE)</td>
		</tr>
		<tr>
			<td>USB3 active cables (10.0 m), x2</td>
			<td></td>
			<td></td>
			<td>Thumbscrew lock connector, water proofed.</td>
		</tr>
		<tr>
			<td>YuMi dual-arm robot</td>
			<td>ABB</td>
			<td>IRB14000</td>
			<td></td>
		</tr>
	</tbody>
</table>

robotized testing of camera positions to determine ideal configuration for stereo 3d visualization of open-heart surgery

Stereo 3D video from surgical procedures can be highly valuable for medical education and improve clinical communication. But access to the operating room and the surgical field is restricted. It is a sterile environment, and the physical space is crowded with surgical staff and technical equipment. In this setting, unobscured capture and realistic reproduction of the surgical procedures are difficult. This paper presents a method for rapid and reliable data collection of stereoscopic 3D videos at different camera baseline distances and distances of convergence. To collect test data with minimum interference during surgery, with high precision and repeatability, the cameras were attached to each hand of a dual-arm robot. The robot was ceiling-mounted in the operating room. It was programmed to perform a timed sequence of synchronized camera movements stepping through a range of test positions with baseline distance between 50-240 mm at incremental steps of 10 mm, and at two convergence distances of 1100 mm and 1400 mm. Surgery was paused to allow 40 consecutive 5-s video samples. A total of 10 surgical scenarios were recorded.

An acceptable evaluation video with the right image placed at the top in top-bottom stereoscopic 3D is shown in Video1. A successful sequence should be sharp, focused, and without unsynchronized image frames. Unsynchronized video streams will cause blur, as shown in the file Video 2. The convergence point should be centered horizontally, independent of the camera separation, as seen in Figure 9A,B. When the robot transitions between the posi...

Watch this Scientific Journal Video about Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery at JoVE.com

Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery

The human depth perception of 3D stereo videos depends on the camera separation, point of convergence, distance to, and familiarity of the object. This paper presents a robotized method for rapid and reliable test data collection during live open-heart surgery to determine the ideal camera configuration.

Stereo 3D video from surgical procedures can be highly valuable for medical education and improve clinical communication. But access to the operating room ...

It enables capturing of high-quality 3D videos for training and education. It can be applied to most operating room settings with open surgery. With a robotized approach, we can quickly and accurately collect the data to evaluate the camera positions.This method is used to rapidly test camera configuration for 3D visions applications. It can also be used to investigate how different subjects perceive depth. To begin mounting the camera to the robot, first, mount the lens to the camera, and then the camera to the adapter plate with three M2 screws.Next, mount the circular mounting plate to the camera adapter plate with four M4 screws to the opposite side of the camera. Once all the cameras are mounted to the robot, observe the mirroring of the resulting assemblies. Mount the adapter plate to the robot wrist with four M2.5 screws.For a table-mounted robot, attach the left camera to the right robot arm. Connect the USB cables to the cameras and the Ubuntu computer. On the robot touch display, press the Menu button and select Stereo 2 to start the robot application.On the main screen, press Go to start for 1, 100 millimeters in the robot application, and wait for the robot to move to the start position. Remove the protective lens cap from the cameras. Place a printed calibration grid at 1, 100 millimeters from the camera sensors.To correctly identify the corresponding squares, place a small screw nut or mark somewhere in enter of the grid. Start the recording application on the Ubuntu computer to start the interface. And then, adjust the aperture and focus on the lens with the aperture and focus rings.In the recording application, check Crosshair to visualize the crosshairs. In the recording application, ensure that the crosshairs align with the calibration grid in the same position in both camera images. After pressing the gear icon in the robot application, position the cameras relative to the patient.Change the X direction by pressing the plus or minus for Hand distance from robot, and Z direction by pressing the plus or minus for Height to capture the surgical field in the images. Change the Y direction by manually moving the robot or patient. After pausing the surgery and informing the OR personnel about starting the experiment, press Record in the recording application, and then press Run Experiment in the robot application.After a while, when the experiment is finished, Done will appear on the touch display. Then, press Stop Recording in the recording application to stop recording. Inform the OR personnel that the experiment has finished and resume the surgery.For evaluation, display the video in top-bottom 3D format with an active 3D projector. The correct evaluation video with a right image placed at the top in top-bottom stereoscopic 3D is shown. A successful sequence should be sharp, focused, and without unsynchronized image frames.The unsynchronized video streams create blur. The convergence point should be centered horizontally, independent of the camera separation. With a very large separation between the right and left images, the brain cannot fuse the images into one 3D image.Several reasons can cause failure in the center positioning of the heart, such as the distance from the convergence point to the heart. For correct configuration of the camera tool coordinate system, ensure that the crosshairs in the recording application point at the same object, the calibration object is in the center position, and the correct debayering format of colors. 3D videos with changing baseline distances enable us to study how depth information affects anatomical understanding among medical staff.

robotized-testing-camera-positions-to-determine-ideal-configuration

Research

JoVE Journal

Medicine

A Pipeline for 3D Multimodality Image Integration and Computer-assisted Planning in Epilepsy Surgery

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality image integration can be technically demanding, and is underutilised in clinical practice. We have developed a single software platform for image integration, 3D visualization and surgical planning. Here, our pipeline is described in step-by-step fashion, starting with image acquisition, proceeding through image co-registration, manual segmentation, brain and vessel extraction, 3D visualization and manual planning of stereoEEG (SEEG) implantations. With dissemination of the software this pipeline can be reproduced in other centres, allowing other groups to benefit from 3DMMI. We also describe the use of an automated, multi-trajectory planner to generate stereoEEG implantation plans. Preliminary studies suggest this is a rapid, safe and efficacious adjunct for planning SEEG implantations. Finally, a simple solution for the export of plans and models to commercial neuronavigation systems for implementation of plans in the operating theater is described. This software is a valuable tool that can support clinical decision making throughout the epilepsy surgery pathway.

We describe the steps to use our custom designed software for image integration, visualization and planning in epilepsy surgery.

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality ...

Visualization of Vascular and Parenchymal Regeneration after 70% Partial Hepatectomy in Normal Mice

A modified silicone injection procedure was used for visualization of the hepatic vascular tree. This procedure consisted of in-vivo injection of the silicone compound, via a 26 G catheter, into the portal or hepatic vein. After silicone injection, organs were explanted and prepared for ex-vivo micro-CT (&#181;CT) scanning. The silicone injection procedure is technically challenging. Achieving a successful outcome requires extensive microsurgical experience from the surgeon. One of the challenges of this procedure involves determining the adequate perfusion rate for the silicone compound. The perfusion rate for the silicone compound needs to be defined based on the hemodynamic of the vascular system of interest. Inappropriate perfusion rate can lead to an incomplete perfusion, artificial dilation and rupturing of vascular trees.
The 3D reconstruction of the vascular system was based on CT scans and was achieved using preclinical software such as HepaVision. The quality of the reconstructed vascular tree was directly related to the quality of silicone perfusion. Subsequently computed vascular parameters indicative of vascular growth, such as total vascular volume, were calculated based on the vascular reconstructions. Contrasting the vascular tree with silicone allowed for subsequent histological work-up of the specimen after &#181;CT scanning. The specimen can be subjected to serial sectioning, histological analysis and whole slide scanning, and thereafter to 3D reconstruction of the vascular trees based on histological images. This is the prerequisite for the detection of molecular events and their distribution with respect to the vascular tree. This modified silicone injection procedure can also be used to visualize and reconstruct the vascular systems of other organs. This technique has the potential to be extensively applied to studies concerning vascular anatomy and growth in various animal and disease models.

Tools used for visualizing vascular regeneration require methods for contrasting the vascular trees. This film demonstrated a delicate injection technique used to achieve optimal contrasting of the vascular trees and illustrate the potential benefits resulting from a detailed analysis of the resulting specimen using &#181;CT and histological serial sections.

A modified silicone injection procedure was used for visualization of the hepatic vascular tree. This procedure consisted of in-vivo injection of the ...

A Quantitative Sensory Testing Paradigm to Obtain Measures of Pain Processing in Patients Undergoing Breast Cancer Surgery

Chronic pain following surgery, persistent postsurgical pain, is an important highly prevalent condition contributing to significant symptom burden and lower quality of life. Persistent postsurgical pain is relatively refractory to treatment hence generating a high need for preventive strategies and treatments. Therefore, the identification of patients at risk of developing persistent pain is an area of active ongoing research. Recently it was demonstrated that peri-operative disruptions in central pain processing may be able to predict persistent postsurgical pain at long term follow-up in breast cancer patients. The aim of the current report is to present a short protocol to obtain pain thresholds to different stimuli at multiple sites and a measure of endogenous analgesia in breast cancer patients. We have used this method successfully in a clinical context and detail some representative results from a clinical study.

Persistent postsurgical pain may be related to changes in pain processing. By assessing pain in response to standardized stimuli, changes in pain processing can be elucidated. We present methods to obtain pain thresholds to different stimuli and a measure of endogenous analgesia in patients undergoing breast cancer surgery.

Chronic pain following surgery, persistent postsurgical pain, is an important highly prevalent condition contributing to significant symptom burden and ...

A Clinical Trial Assessing the Safety, Efficacy, and Delivery of Olive-Oil-Based Three-Chamber Bags for Parenteral Nutrition

Limited evidence exists to precisely estimate efficacy and safety differences between parenteral nutrition (PN) prepared using olive-oil-based three-chamber bags (3CBs) and soybean-oil-based compounded bags (CoBs) in hospitalized adult patients. We designed a multicenter, randomized, prospective, open-label, noninferiority protocol to compare the efficacy, safety, and distribution of olive-oil-based 3CBs and soybean-oil-based CoB formulations in adult Chinese patients requiring PN during surgical intervention. Subjects were randomized to receive either one of the study treatments using an interactive voice or web-based recognition system in accordance with the randomization code. Randomization was further stratified based on the study site and surgical category. Both treatment groups received similar amounts of calories and protein. In addition, the two study treatments contained a similar composition of the amino-acid component. The only difference between the two PN formulations was the lipid constitution. The duration of administration of study treatments was a minimum of 5 days up to a maximum of 14 days after the surgical procedure. The primary efficacy endpoint was serum prealbumin levels on day 5 of the study. Noninferiority was proved if the anti-log of the lower bound of the 95% confidence interval (CI) of the treatment difference was at least 0.80. Other efficacy measures included treatment preparation time; duration to achieve tolerability of oral nutrition; associated infectious complications; length of hospitalization; and laboratory assessment of markers of nutrition, inflammation, metabolism, and oxidative stress. A total of 458 patients were enrolled in the study. The results showed that olive-oil-based 3CBs were non-inferior to soybean-based CoBs, besides being well tolerated. The infection rate was found to be significantly lower in the olive-oil-based 3CB group. Thus, this study may be used as a reference for future research on lipid emulsion and 3CBs.

Here, we present a protocol for a study comparing the efficacy, safety, and delivery of olive-oil-based 3CB and soybean-oil-based CoB formulations in adults requiring parenteral nutrition. The results revealed that olive-oil-based 3CBs is non-inferior and well tolerated compared to soybean formulations.

Limited evidence exists to precisely estimate efficacy and safety differences between parenteral nutrition (PN) prepared using olive-oil-based ...

A Training and Testing System for Performing Vascular Reconstruction In Vitro

Manual vascular reconstruction training is essential for a beginner surgeon. However, an optimal training system for vascular reconstruction in vitro has yet to be developed. In this study, we introduce an in vitro training and testing system using a magnetic anchoring technique with which a trainee can practice manual vascular reconstruction individually. Additionally, this system can also be used to test the quality of the reconstruction. The described system includes a vascular reconstruction training machine, magnetic tractors, and a magnetic suture puller. In this manuscript, we detail an end-to-end vein anastomosis using porcine right and left iliac veins. To identify the potential damage caused by a magnetic suture puller on the suture, we created three groups with six segments of 4-0 polypropylene sutures each: a control group with no intervention on the polypropylene suture, a group in which the polypropylene suture is manually pulled with sterile gloves 20x, and a magnetic puller group in which the magnetic puller pulled the polypropylene suture 20x. These groups were tested by light microscopy and breaking strength tests, and the effect of reconstruction was assessed. In the light microscopy test, the control group was less likely to be damaged (p &#60; 0.05) and the number of damaged points of the manual group and magnetic puller group were similar (p &#62; 0.05). The results of the breaking strength test were compared across groups and no significant difference was observed (p &#62; 0.05). The end-to-end anastomosis of the porcine iliac veins was successfully performed using this training system, and the reconstructed veins could undergo 2.0 kPa perfusion pressure. Using this training and testing system the trainee can practice manual vascular reconstruction in vitro individually with the aid of magnetic tractors and a magnetic suture puller, and the quality of the reconstruction can be tested.

Here we present a training and testing system where a trainee can complete manual vascular reconstruction in vitro individually using a magnetic anchoring technique. The system can also be used to test the quality of reconstruction.

Manual vascular reconstruction training is essential for a beginner surgeon. However, an optimal training system for vascular reconstruction in vitro has ...

A Postoperative Evaluation Guideline for Computer-Assisted Reconstruction of the Mandible

Valid comparisons of postoperative accuracy results in computer-assisted reconstruction of the mandible are difficult due to heterogeneity in imaging modalities, mandibular defect classification, and evaluation methodologies between studies. This guideline uses a step-by-step approach guiding the process of imaging, classification of mandibular defects and volume assessment of three-dimensional (3D) models, after which a legitimized quantitative accuracy evaluation method can be performed between the postoperative clinical situation and the preoperative virtual plan. The condyles and the vertical and horizontal corners of the mandible are used as bony landmarks to define virtual lines in the computer-assisted surgery (CAS) software. Between these lines the axial, coronal, and both sagittal mandibular angles are calculated on both pre- and postoperative 3D models of the (neo)mandible and subsequently the deviations are calculated. By superimposing the postoperative 3D model to the preoperative virtually planned 3D model, which is fixed to the XYZ axis, the deviation between pre- and postoperative virtually planned dental implant positions can be calculated. This protocol continues and specifies an earlier publication of this evaluation guideline.

Here, we propose a practical, feasible and reproducible evaluation guideline for computer-assisted reconstruction of the mandible in order to create uniformity between studies regarding postoperative accuracy evaluation. This protocol continues and specifies an earlier publication of this evaluation guideline.

Valid comparisons of postoperative accuracy results in computer-assisted reconstruction of the mandible are difficult due to heterogeneity in imaging ...

3D Planning and Printing of Patient Specific Implants for Reconstruction of Bony Defects

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical field using the constantly developing 3D planning and printing capabilities. Computer-assisted design (CAD) and computer assisted manufacturing (CAM) are used to describe the 3D planning and manufacturing of the product. The planning and manufacturing of 3D surgical guides and reconstruction implants is performed almost exclusively by engineers. As technology advances and software interfaces become more user-friendly, it raises a question regarding the possibility of transferring the planning and manufacturing to the clinician. The reasons for such a shift are clear: the surgeon has the idea of what he wants to design, and he also knows what is feasible and could be used in the operating room. It allows him to be prepared for any scenario/unexpected results during the operation and allows the surgeon to be creative and express his new ideas using the CAD software. The purpose of this method is to provide clinicians with the ability to create their own surgical guides and reconstruction implants. In this manuscript, a detailed protocol will provide a simple method for segmentation using segmentation software and implant planning using a 3D design software. Following the segmentation and stl file production using segmentation software, the clinician could create a simple patient specific reconstruction plate or a more complex plate with a cradle for bone graft positioning. Surgical guides can be created for accurate resection, hole preparation for proper reconstruction plate positioning or for bone graft harvesting and re-contouring. A case of lower jaw reconstruction following plate fracture and nonunion healing of a trauma sustained injury is detailed.

This protocol describes the use of 3D planning and printing for reconstruction of bony defects. We use segmentation tools to create 3D models followed by 3D design software to create patient specific implants for reconstruction purposes concomitant to ablative surgery or as a second stage.

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical ...

Implantation of Combined Telemetric ECG and Blood Pressure Transmitters to Determine Spontaneous Baroreflex Sensitivity in Conscious Mice

Blood pressure (BP) and heart rate (HR) are both controlled by the autonomic nervous system (ANS) and are closely intertwined due to reflex mechanisms. The baroreflex is a key homeostatic mechanism to counteract acute, short-term changes in arterial BP and to maintain BP in a relatively narrow physiological range. BP is sensed by baroreceptors located in the aortic arch and carotid sinus. When BP changes, signals are transmitted to the central nervous system and are then communicated to the parasympathetic and sympathetic branches of the autonomic nervous system to adjust HR. A rise in BP causes a reflex decrease in HR, a drop in BP causes a reflex increase in HR.
Baroreflex sensitivity (BRS) is the quantitative relationship between changes in arterial BP and corresponding changes in HR. Cardiovascular diseases are often associated with impaired baroreflex function. In various studies reduced BRS has been reported in e.g., heart failure, myocardial infarction, or coronary artery disease. 
Determination of BRS requires information from both BP and HR, which can be recorded simultaneously using telemetric devices. The surgical procedure is described beginning with the insertion of the pressure sensor into the left carotid artery and positioning of its tip in the aortic arch to monitor arterial pressure followed by the subcutaneous placement of the transmitter and ECG electrodes. We also describe postoperative intensive care and analgesic management. After a two-week period of post-surgery recovery long-term ECG and BP recordings are performed in conscious and unrestrained mice. Finally, we include examples of high-quality recordings and the analysis of spontaneous baroreceptor sensitivity using the sequence method.

The baroreflex is a heart-rate regulation mechanism by the autonomic nervous system in response to blood-pressure changes. We describe a surgical technique to implant telemetry transmitters for continuous and simultaneous measurement of electrocardiogram and blood pressure in mice. This can determine spontaneous baroreflex sensitivity, an important prognostic marker for cardiovascular disease.

Blood pressure (BP) and heart rate (HR) are both controlled by the autonomic nervous system (ANS) and are closely intertwined due to reflex mechanisms. ...

Use of 3D Robotic Ultrasound for In Vivo Analysis of Mouse Kidneys

Common modalities for in vivo imaging of rodents include positron emission tomography (PET), computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound (US). Each method has limitations and advantages, including availability, ease of use, cost, size, and the use of ionizing radiation or magnetic fields. This protocol describes the use of 3D robotic US for in vivo imaging of rodent kidneys and heart, subsequent data analysis, and possible research applications. Practical applications of robotic US are the quantification of total kidney volume (TKV), as well as the measurement of cysts, tumors, and vasculature. Although the resolution is not as high as other modalities, robotic US allows for more practical high throughput data collection. Furthermore, using US M-mode imaging, cardiac function may be quantified. Since the kidneys receive 20%-25% of the cardiac output, assessing cardiac function is critical to the understanding of kidney physiology and pathophysiology.

Use of 3D Robotic Ultrasound for In Vivo Analysis of Mouse Kidneys

This protocol demonstrates robotic ultrasound (US) as a practical, cost-effective, and quick alternative to traditional non-invasive image modalities.

Common modalities for in vivo imaging of rodents include positron emission tomography (PET), computed tomography (CT), magnetic resonance imaging (MRI), ...

Combining 3D-Printing and Electrospinning to Manufacture Biomimetic Heart Valve Leaflets

Electrospinning has become a widely used technique in cardiovascular tissue engineering as it offers the possibility to create (micro-)fibrous scaffolds with adjustable properties. The aim of this study was to create multilayered scaffolds mimicking the architectural fiber characteristics of human heart valve leaflets using conductive 3D-printed collectors.
Models of aortic valve cusps were created using commercial computer-aided design (CAD) software. Conductive polylactic acid was used to fabricate 3D-printed leaflet templates. These cusp negatives were integrated into a specifically designed, rotating electrospinning mandrel. Three layers of polyurethane were spun onto the collector, mimicking the fiber orientation of human heart valves. Surface and fiber structure was assessed with a scanning electron microscope (SEM). The application of fluorescent dye additionally permitted the microscopic visualization of the multilayered fiber structure. Tensile testing was performed to assess the biomechanical properties of the scaffolds.
3D-printing of essential parts for the electrospinning rig was possible in a short time for a low budget. The aortic valve cusps created following this protocol were three-layered, with a fiber diameter of 4.1 &#177; 1.6 &#181;m. SEM imaging revealed an even distribution of fibers. Fluorescence microscopy revealed individual layers with differently aligned fibers, with each layer precisely reaching the desired fiber configuration. The produced scaffolds showed high tensile strength, especially along the direction of alignment. The printing files for the different collectors are available as Supplemental File 1, Supplemental File 2, Supplemental File 3, Supplemental File 4, and Supplemental File 5.
With a highly specialized setup and workflow protocol, it is possible to mimic tissues with complex fiber structures over multiple layers. Spinning directly on 3D-printed collectors creates considerable flexibility in manufacturing 3D shapes at low production costs.

The presented method offers an innovative way for engineering biomimetic fiber structures in three-dimensional (3D) scaffolds (e.g., heart valve leaflets). 3D-printed, conductive geometries were used to determine shape and dimensions. Fiber orientation and characteristics were individually adjustable for each layer. Multiple samples could be manufactured in one setup.

Electrospinning has become a widely used technique in cardiovascular tissue engineering as it offers the possibility to create (micro-)fibrous scaffolds ...