The research was carried out with funding from Vinnova (2017-03728, 2018-05302 and 2018-03651), Heart-Lung Foundation (20180390), Family Kamprad Foundation (20190194), and Anna-Lisa and Sven Eric Lundgren Foundation (2017 and 2018).

The experiments were approved by the local Ethics Committee in Lund, Sweden. The participation was voluntary, and the patients&#39; legal guardians provided informed written consent.
1. Robot setup and configuration
NOTE: This experiment used a dual-arm collaborative industrial robot and the standard control panel with a touch display. The robot is controlled with RobotWare 6.10.01 controller software and robot integrated development environment (IDE) RobotStudio 2019.525. Software developed by the authors, including the robot application, recording application, and postprocessing scripts, are available at the GitHub repository26.
CAUTION: Use protective eyeglasses and reduced speed during setup and testing of the robot program.
<ol>
	<li>Mount the robot to the ceiling or a table using bolts dimensioned for 100 kg as described on page 25 in the product specification24, following the manufacturer&#39;s specifications. Ensure that the arms can move freely and the line of sight to the field of view is unobscured. 
	CAUTION: Use lift or safety ropes when mounting the robot in a high position.</li>
	<li>Start the robot by turning the start switch located at the base of the robot. Calibrate the robot by following the procedure described in the operating manual on pages 47-5625.</li>
	<li>Start the robot IDE on a Windows computer.</li>
	<li>Connect to the physical robot system (operating manual page 14027).</li>
	<li>Load the code for the robot program and application libraries for the user interface to the robot:
	<ol>
		<li>The robot code for a ceiling-mounted robot is in the folder Robot/InvertedCode and for a table mounted robot in Robot/TableMountedCode. For each of the files left/Data.mod, left/MainModule.mod, right/Data.mod and right/MainModule.mod:</li>
		<li>Create a new program module (see operating manual page 31827) with the same name as the file (Data or MainModule) and copy the file content to the new module.</li>
		<li>Press on Apply in the robot IDE to save the files to the robot.</li>
	</ol>
	</li>
	<li>Use File Transfer (operating manual page 34627) to transfer the robot application files TpSViewStereo2.dll, TpsViewStereo2.gtpu.dll, and TpsViewStereo2.pdb located in the FPApp folder to the robot. After this step, the robot IDE will not be further used.</li>
	<li>Press the Reset button on the back of the robot touch display (FlexPendant) to reload the graphical interface. The robot application Stereo2 will now be visible under the touch display menu.</li>
	<li>Install the recording application (Liveview) and postprocessing application on an Ubuntu 20.04 computer by running the script install_all_linux.sh, located in the root folder in the Github repository.</li>
	<li>Mount each camera to the robot. The components needed for mounting are displayed in Figure 3A.
	<ol>
		<li>Mount the lens to the camera.</li>
		<li>Mount the camera to the camera adaptor plate with three M2 screws.</li>
		<li>Mount the circular mounting plate to the camera adaptor plate with four M6 screws on the opposite side of the camera.</li>
	</ol>
	</li>
	<li>Repeat steps 1.9.1-1.9.3 for the other camera. The resulting assemblies are mirrored, as shown in Figure 3B and Figure 3C.</li>
	<li>Mount the adaptor plate to the robot wrist with four M2.5 screws, as shown in Figure 3D.
	<ol>
		<li>For a ceiling-mounted robot: attach the left camera in Figure 3C to the left robot arm as shown in Figure 2A.</li>
		<li>For a table-mounted robot: attach the left camera in Figure 3C to the right robot arm.</li>
	</ol>
	</li>
	<li>Connect the USB cables to the cameras, as shown in Figure 3E, and to the Ubuntu computer.</li>
</ol>
2. Verify the camera calibration
<ol>
	<li>On the robot touch display, press the Menu button and select Stereo2 to start the robot application. This will open the main screen, as shown in Figure 6A.
	<ol>
		<li>On the main screen, press on Go to start for 1100 mm in the robot application and wait for the robot to move to the start position.</li>
		<li>Remove the protective lens caps from the cameras and connect the USB cables to the Ubuntu computer.</li>
		<li>Place a printed calibration grid (CalibrationGrid.png in the repository) 1100 mm from the camera sensors. To facilitate correct identification of the corresponding squares, place a small screw-nut or mark somewhere in the center of the grid.</li>
	</ol>
	</li>
	<li>Start the recording application on the Ubuntu computer (run the script start.sh located in the liveview folder inside the Github repository). This starts the interface, as shown in Figure 4.
	<ol>
		<li>Adjust the aperture and focus on the lens with the aperture and focus rings.</li>
		<li>In the recording application, check Crosshair to visualize the crosshairs.</li>
	</ol>
	</li>
	<li>In the recording application, ensure that the crosshairs align with the calibration grid in the same position in both camera images, as shown in Figure 4. Most likely, some adjustment will be required as follows:
	<ol>
		<li>If the crosses do not overlap, press the Gear icon (bottom left Figure 6A) in the robot application on the robot touch display to open the setting screen, as shown in Figure 6B.</li>
		<li>Press on 1. Go to Start Pos, as shown in Figure 6B.</li>
		<li>Jog the robot with the joystick to adjust the camera position (operating manual page 3123).</li>
		<li>Update the tool position for each robot arm. Press 3. Update Left Tool and 4. Update Right Tool to save the calibration for the left and the right arm, respectively.</li>
		<li>Press on the Back Arrow icon (top right, Figure 6B) to return to the main screen.</li>
	</ol>
	</li>
	<li>Press on Run Experiment (Figure 6A) in the robot application and verify that the crosshairs align. Otherwise, repeat steps 2.3-2.3.5.</li>
	<li>Add and test any changes to the distances and/or time at this point. This requires changes in the robot program code and advanced robot programming skills. Change the following variables in the Data module in the left task (arm): the desired separation distances in the integer array variable Distances, the convergence distances in the integer array ConvergencePos and edit the time at each step by editing the variable Nwaittime (value in seconds). 
	CAUTION: Never run an untested robot program during live surgery.</li>
	<li>When the calibration is complete, press on Raise to raise the robot arms to the standby position.</li>
	<li>Optionally turn off the robot. 
	&#8203;NOTE: The procedure can be paused between any of the steps above.</li>
</ol>
3. Preparation at the start of the surgery
<ol>
	<li>Dust the robot.
	<ol>
		<li>If the robot was turned off, start it by turning on the Start switch located at the base of the robot.</li>
	</ol>
	</li>
	<li>Start the robot application on the touch display and recording application described in steps 2.1 and 2.2.</li>
	<li>In the recording application, create and then select the folder where to save the video (press Change Folder).</li>
	<li>In the robot application: press the gear icon, position the cameras in relation to the patient. Change X and Z direction by pressing +/- for Hand Distance from Robot and Height, respectively, so that the image captures the surgical field. Perform the positioning in the Y-direction by manually moving the robot or patient. 
	&#8203;NOTE: The preparations can be paused between the preparation steps 3.1-3.4.</li>
</ol>
4. Experiment
CAUTION: All personnel should be informed about the experiment beforehand.
<ol>
	<li>Pause the surgery.
	<ol>
		<li>Inform the OR personnel that the experiment is started.</li>
	</ol>
	</li>
	<li>Press Record in the recording application.</li>
	<li>Press Run experiment in the robot application.</li>
	<li>Wait while the program is running; the robot displays &#34;Done&#34; in the robot application on the touch display when finished.</li>
	<li>Stop recording in the recording application by pressing Quit.
	<ol>
		<li>Inform the OR personnel that the experiment has finished.</li>
	</ol>
	</li>
	<li>Resume surgery. 
	&#8203;NOTE: The experiment cannot be paused during steps 4.1-4.6.</li>
</ol>
5. Repeat
<ol>
	<li>Repeat steps 4.1-4.6 to capture another sequence and steps 3.1-3.4 and steps 4.1-4.6 to capture sequences from different surgeries. Capture around ten full sequences.</li>
</ol>
6. Postprocessing
NOTE: The following steps can be carried out using most video editing software or the provided scripts in the postprocessing folder.
<ol>
	<li>In this case, debayer the video as it is saved in the RAW format:
	<ol>
		<li>Run the script postprocessing/debayer/run.sh to open the debayer application shown in Figure 8A.</li>
		<li>Press Browse Input Directory and select the folder with the RAW video.</li>
		<li>Press Browse Output Directory and select a folder for the resulting debayered and color-adjusted video files.</li>
		<li>Press Debayer! and wait until the process is finished - both progress bars are full, as shown in Figure 8B.</li>
	</ol>
	</li>
	<li>Merge the right and left synchronized videos to 3D stereo format28:
	<ol>
		<li>Run the script postprocessing/merge_tb/run.sh to start the merge application; it opens the graphical user interface shown in Figure 8C.</li>
		<li>Press Browse Input Directory and select the folder with the debayered video files.</li>
		<li>Press Browse Output Directory and select a folder for the resulting merged 3D stereo file.</li>
		<li>Press Merge! and wait until the finish screen in Figure 8D is shown.</li>
	</ol>
	</li>
	<li>Use off-the-shelf video editing software such as Premiere Pro to add text labels to each camera distance in the video. 
	&#8203;NOTE: In the video, there is a visible shake every time the robot moved, and the camera distance increased. In this experiment, labels A-T were used for the camera distances.</li>
</ol>
7. Evaluation
<ol>
	<li>Display the video in top-bottom 3D format with an active 3D projector.</li>
	<li>Viewing experience depends on the viewing angle and the distance to screen; evaluate the video using intended audience and setup.</li>
</ol>

<ol><li>Held, R. T., Hui, T. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+guide+to+stereoscopic+3D+displays+in+medicine%5BTitle%5D)+AND+%22Academic+Radiology%22%5BJournal%5D)">A guide to stereoscopic 3D displays in medicine.</a> Academic Radiology. 18 (8), 1035-1048 (2011).</li>
<li>van Beurden, M. H. P. H., IJsselsteijn, W. A., Juola, J. F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Effectiveness+of+stereoscopic+displays+in+medicine%3A+A+review%5BTitle%5D)+AND+%223D+Research%22%5BJournal%5D)">Effectiveness of stereoscopic displays in medicine: A review.</a> 3D Research. 3 (1), 1-13 (2012).</li>
<li>Luursema, J. M., Verwey, W. B., Kommers, P. A. M., Geelkerken, R. H., Vos, H. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Optimizing+conditions+for+computer-assisted+anatomical+learning%5BTitle%5D)+AND+%22Interacting+with+Computers%22%5BJournal%5D)">Optimizing conditions for computer-assisted anatomical learning.</a> Interacting with Computers. 18 (5), 1123-1138 (2006).</li>
<li>Takano, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Usefulness+and+capability+of+three-dimensional%2C+full+high-definition+movies+for+surgical+education%5BTitle%5D)+AND+%22Maxillofacial+Plastic+and+Reconstructive+Surgery%22%5BJournal%5D)">Usefulness and capability of three-dimensional, full high-definition movies for surgical education.</a> Maxillofacial Plastic and Reconstructive Surgery. 39 (1), 10(2017).</li>
<li>Triepels, C. P. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Does+three-dimensional+anatomy+improve+student+understanding%5BTitle%5D)+AND+%22Clinical+Anatomy%22%5BJournal%5D)">Does three-dimensional anatomy improve student understanding.</a> Clinical Anatomy. 33 (1), 25-33 (2020).</li>
<li>Beermann, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Three-dimensional+visualisation+improves+understanding+of+surgical+liver+anatomy%5BTitle%5D)+AND+%22Medical+Education%22%5BJournal%5D)">Three-dimensional visualisation improves understanding of surgical liver anatomy.</a> Medical Education. 44 (9), 936-940 (2010).</li>
<li>Battulga, B., Konishi, T., Tamura, Y., Moriguchi, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+Effectiveness+of+an+interactive+3-dimensional+computer+graphics+model+for+medical+education%5BTitle%5D)+AND+%22Interactive+Journal+of+Medical+Research%22%5BJournal%5D)">The Effectiveness of an interactive 3-dimensional computer graphics model for medical education.</a> Interactive Journal of Medical Research. 1 (2), (2012).</li>
<li>Yammine, K., Violato, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+meta-analysis+of+the+educational+effectiveness+of+three-dimensional+visualization+technologies+in+teaching+anatomy%5BTitle%5D)+AND+%22Anatomical+Sciences+Education%22%5BJournal%5D)">A meta-analysis of the educational effectiveness of three-dimensional visualization technologies in teaching anatomy.</a> Anatomical Sciences Education. 8 (6), 525-538 (2015).</li>
<li>Fitzgerald, J. E. F., White, M. J., Tang, S. W., Maxwell-Armstrong, C. A., James, D. K. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Are+we+teaching+sufficient+anatomy+at+medical+school%3F+The+opinions+of+newly+qualified+doctors%5BTitle%5D)+AND+%22Clinical+Anatomy%22%5BJournal%5D)">Are we teaching sufficient anatomy at medical school? The opinions of newly qualified doctors.</a> Clinical Anatomy. 21 (7), 718-724 (2008).</li>
<li>Bergman, E. M., Van Der Vleuten, C. P. M., Scherpbier, A. J. J. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Why+don%27t+they+know+enough+about+anatomy%3F+A+narrative+review%5BTitle%5D)+AND+%22Medical+Teacher%22%5BJournal%5D)">Why don't they know enough about anatomy? A narrative review.</a> Medical Teacher. 33 (5), 403-409 (2011).</li>
<li>Terzić, K., Hansard, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Methods+for+reducing+visual+discomfort+in+stereoscopic+3D%3A+A+review%5BTitle%5D)+AND+%22Signal+Processing%3A+Image+Communication%22%5BJournal%5D)">Methods for reducing visual discomfort in stereoscopic 3D: A review.</a> Signal Processing: Image Communication. 47, 402-416 (2016).</li>
<li>Fan, Z., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+interactive+surgical+visualization+system+using+mobile+spatial+information+acquisition+and+autostereoscopic+display%5BTitle%5D)+AND+%22Journal+of+Biomedical+Informatics%22%5BJournal%5D)">3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display.</a> Journal of Biomedical Informatics. 71, 154-164 (2017).</li>
<li>Fan, Z., Zhang, S., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+quantitative+evaluation+system+for+autostereoscopic+display%5BTitle%5D)+AND+%22Journal+of+Display+Technology%22%5BJournal%5D)">3D quantitative evaluation system for autostereoscopic display.</a> Journal of Display Technology. 12 (10), 1185-1196 (2016).</li>
<li>McIntire, J. P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Binocular+fusion+ranges+and+stereoacuity+predict+positional+and+rotational+spatial+task+performance+on+a+stereoscopic+3D+display%5BTitle%5D)+AND+%22Journal+of+Display+Technology%22%5BJournal%5D)">Binocular fusion ranges and stereoacuity predict positional and rotational spatial task performance on a stereoscopic 3D display.</a> Journal of Display Technology. 11 (11), 959-966 (2015).</li>
<li>Kalia, M., Navab, N., Fels, S. S., Salcudean, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+method+to+introduce+%26+evaluate+motion+parallax+with+stereo+for+medical+AR%2FMR%5BTitle%5D)+AND+%22IEEE+Conference+on+Virtual+Reality+and+3D+User+Interfaces%22%5BJournal%5D)">A method to introduce & evaluate motion parallax with stereo for medical AR/MR.</a> IEEE Conference on Virtual Reality and 3D User Interfaces. , 1755-1759 (2019).</li>
<li>Kytö, M., Hakala, J., Oittinen, P., Häkkinen, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Effect+of+camera+separation+on+the+viewing+experience+of+stereoscopic+photographs%5BTitle%5D)+AND+%22Journal+of+Electronic+Imaging%22%5BJournal%5D)">Effect of camera separation on the viewing experience of stereoscopic photographs.</a> Journal of Electronic Imaging. 21 (1), 1-9 (2012).</li>
<li>Moorthy, A. K., Su, C. C., Mittal, A., Bovik, A. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Subjective+evaluation+of+stereoscopic+image+quality%5BTitle%5D)+AND+%22Signal+Processing%3A+Image+Communication%22%5BJournal%5D)">Subjective evaluation of stereoscopic image quality.</a> Signal Processing: Image Communication. 28 (8), 870-883 (2013).</li>
<li>Yilmaz, G. N. A depth perception evaluation metric for immersive 3D video services. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aYilmaz%2C+GN+%223DTV+Conference%3A+The+True+Vision+-+Capture%2C+Transmission+and+Display+of+3D+Video%22">3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video</a>. , 1-4 (2017).</li>
<li>Lebreton, P., Raake, A., Barkowsky, M., Le Callet, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Evaluating+depth+perception+of+3D+stereoscopic+videos%5BTitle%5D)+AND+%22IEEE+Journal+on+Selected+Topics+in+Signal+Processing%22%5BJournal%5D)">Evaluating depth perception of 3D stereoscopic videos.</a> IEEE Journal on Selected Topics in Signal Processing. 6, 710-720 (2012).</li>
<li>López, J. P., Rodrigo, J. A., Jiménez, D., Menéndez, J. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Stereoscopic+3D+video+quality+assessment+based+on+depth+maps+and+video+motion%5BTitle%5D)+AND+%22EURASIP+Journal+on+Image+and+Video+Processing%22%5BJournal%5D)">Stereoscopic 3D video quality assessment based on depth maps and video motion.</a> EURASIP Journal on Image and Video Processing. 2013 (1), 62(2013).</li>
<li>Banks, M. S., Read, J. C., Allison, R. S., Watt, S. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Stereoscopy+and+the+human+visual+system%5BTitle%5D)+AND+%22SMPTE+Motion+Imaging+Journal%22%5BJournal%5D)">Stereoscopy and the human visual system.</a> SMPTE Motion Imaging Journal. 121 (4), 24-43 (2012).</li>
<li>Kytö, M., Nuutinen, M., Oittinen, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Method+for+measuring+stereo+camera+depth+accuracy+based+on+stereoscopic+vision%5BTitle%5D)+AND+%22Three-Dimensional+Imaging%2C+Interaction%2C+and+Measurement%22%5BJournal%5D)">Method for measuring stereo camera depth accuracy based on stereoscopic vision.</a> Three-Dimensional Imaging, Interaction, and Measurement. 7864, 168-176 (2011).</li>
<li>Kang, Y. S., Ho, Y. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Geometrical+compensation+algorithm+of+multiview+image+for+arc+multi-camera+arrays%5BTitle%5D)+AND+%22Advances+in+Multimedia+Information+Processing%22%5BJournal%5D)">Geometrical compensation algorithm of multiview image for arc multi-camera arrays.</a> Advances in Multimedia Information Processing. 2008, 543-552 (2008).</li>
<li>Product Specification IRB 14000. DocumentID: 3HAC052982-001 Revision J. ABB Robotics. , Available from: 
 <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2018).</li>
<li>Operating Manual IRB 14000. Document ID: 3HAC052986-001 Revision F. ABB Robotics. , Available from: 
 <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2019).</li>
<li>Github repository. , Available from: 
 <a target="_blank" href="https://github.com/majstenmark/stereo2">https://github.com/majstenmark/stereo2</a> (2021).</li>
<li>Operating manual RobotStudio. Document ID: 3HAC032104-001 Revision Y. ABB Robotics. , Available from: 
 <a target="_blank" href="https://library.abb.com/en/results">https://library.abb.com/en/results</a> (2019).</li>
<li>Won, C. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Adaptive+interpolation+for+3D+stereoscopic+video+in+frame-compatible+top-bottom+packing%5BTitle%5D)+AND+%22IEEE+International+Conference+on+Consumer+Electronics%22%5BJournal%5D)">Adaptive interpolation for 3D stereoscopic video in frame-compatible top-bottom packing.</a> IEEE International Conference on Consumer Electronics. 2011, 179-180 (2011).</li>
<li>Kim, S. K., Lee, C., Kim, K. T. Multi-view image acquisition and display. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aKim%2C+SK+%22Three-Dimensional+Imaging%2C+Visualization%2C+and+Display%22">Three-Dimensional Imaging, Visualization, and Display</a>. Javidi, B., Okano, F., Son, J. Y. , Springer. New York. 227-249 (2009).</li>
<li>Liu, F., Niu, Y., Jin, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Keystone+correction+for+stereoscopic+cinematography%5BTitle%5D)+AND+%22IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition+Workshops%22%5BJournal%5D)">Keystone correction for stereoscopic cinematography.</a> IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012, 1-7 (2012).</li>
<li>Kang, W., Lee, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Horizontal+parallax+distortion+correction+method+in+toed-in+camera+with+wide-angle+lens%5BTitle%5D)+AND+%223DTV+Conference%3A+The+True+Vision+-+Capture%2C+Transmission+and+Display+of+3D+Video%22%5BJournal%5D)">Horizontal parallax distortion correction method in toed-in camera with wide-angle lens.</a> 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video. 2009, 1-4 (2009).</li>
</ol>

The authors have nothing to disclose.

During live surgery, the total time of the experiment used for 3D video data collection was limited to be safe for the patient. If the object is unfocused or overexposed, the data cannot be used. The critical steps are during camera tool calibration and setup (step 2). The camera aperture and focus cannot be changed when the surgery has started; the same lighting conditions and distance should be used during setup and surgery. The camera calibration in steps 2.1-2.4 must be carried out carefully to ensure that the heart is centered in the captured video. To troubleshoot the calibration, the values of the camera tool coordinate system can be verified separately by jogging the robot in the coordinate system (step 2.3.3). It is critical to test the full robot program and cameras together with the recording application before the surgery. The height of the operating table is sometimes adjusted during surgery; the height of the robot cameras can also be modified live in the robot application (step 3.4) to keep the desired distance to the heart. The distances and wait times of the robot program can be modified as described in step 2.5.
One limitation of this technique is that it requires that the surgery is paused; therefore, data collection can only be carried out when it is safe for the patient to pause the surgery. Another limitation is that it requires physical adaptation of the operating room to mount the robot in the ceiling and the programmed robot motion assumes that the robot is centered above the heart. Additionally, the cameras are toed-in instead of parallel, which can cause a keystone effect. The keystone effect can be adjusted in postproduction29,30,31.
An array of multiple cameras placed on an arc can be used to collect similar data23. The camera array can capture images simultaneously from all cameras; thus, surgery can be paused for a shorter time. A source of error for a camera array is that the cameras can have a slightly different focus, aperture, and calibration and when videos from different camera pairs are compared, other parameters than the baseline distance can affect the image quality and depth perception. Another drawback with a camera array is that the step size between baseline distances is limited by the physical size of the cameras. For example, the lens used in this study has a diameter of 30 mm, which would equal the minimum possible step size. With the setup presented in the study, step sizes of 10 mm were tested but could be set smaller if necessary. Also, with the array setup, height and convergence distance cannot be dynamically adjusted.
Another alternative is to manually move the cameras to predefined positions22. This is not feasible during live heart surgery because it would infringe on critical surgical workspace and time.
This method is applicable to many types of open surgery, including orthopedic, vascular, and general surgery, where optimal baseline and convergence distances are yet to be determined.
This method can also be adapted to collect images for purposes other than 3D visualization. Many computer vision applications use the disparity between images to calculate the distance to an object. A precise camera motion can be used to 3D scan stationary objects from multiple directions to create 3D models. For 3D localization, the 3D viewing experience is less important if the same points on the object can be identified in different images, depending on accurate camera positioning, camera calibration, light conditions, and frame synchronization.
Robot-controlled camera positioning is both safe and effective for collecting video data for the identification of optimal camera positions for stereoscopic 3D video.

In surgery, 3D visualization can be used for education, diagnoses, pre-operative planning, and post-operative evaluation1,2. Realistic depth perception can improve understanding3,4,5,6 of normal and abnormal anatomies. Simple 2D video recordings of surgical procedures are a good start. However, the lack of depth perception can make it hard for the non-surgical colleagues to fully understand the antero-posterior relationships between different anatomical structures and therefore also introduce a risk of misinterpretation of the anatomy7,8,9,10.
The 3D viewing experience is affected by five factors: (1) Camera configuration can either be parallel or toed-in as shown in Figure 1, (2) Baseline distance (the separation between the cameras). (3) Distance to the object of interest and other scene characteristics such as the background. (4) Characteristics of viewing devices such as screen size and viewing position1,11,12,13. (5) Individual preferences of the viewers14,15.
Designing a 3D camera setup begins with the capture of test videos recorded at various camera baseline distances and configurations to be used for subjective or automatic evaluation16,17,18,19,20. The camera distance must be constant to the surgical field to capture sharp images. Fixed focus is preferred because autofocus will adjust to focus on hands, instruments, or heads that may come into view. However, this is not easily achievable when the scene of interest is the surgical field. Operating rooms are restricted access areas because these facilities must be kept clean and sterile. Technical equipment, surgeons, and scrub nurses are often clustered closely around the patient to secure a good visual overview and an efficient workflow. To compare and evaluate the effect of camera positions on the 3D viewing experience, one complete test range of camera positions should be recording the same scene because the object characteristics such as shape, size, and color can affect the 3D viewing experience21.
For the same reason, complete test ranges of camera positions should be repeated on different surgical procedures. The entire sequence of positions must be repeated with high accuracy. In a surgical setting, existing methods that require either manual adjustment of the baseline distance22 or different camera pairs with fixed baseline distances23 are not feasible because of both space and time constraints. To address this challenge, this robotized solution was designed.
The data was collected with a dual-arm collaborative industrial robot mounted in the ceiling in the operating room. Cameras were attached to the wrists of the robot and moved along an arc-shaped trajectory with increasing baseline distance, as shown in Figure 2.
To demonstrate the approach, 10 test series were recorded from 4 different patients with 4 different congenital heart defects. Scenes were chosen when a pause in surgery was feasible: with the beating hearts just before and after surgical repair. Series were also made when the hearts were arrested. The surgeries were paused for 3 min and 20 s to collect forty 5-ssequences with different camera convergence distances and baseline distances to capture the scene. The videos were later post-processed, displayed in 3D for the clinical team, who rated how realistic the 3D video was along a scale from 0-5.
The convergence point for toed-in stereo cameras is where the center points of both images meet. The convergence point can, by principle, be placed either in front, within, or behind the object, see Figure 1A-C. When the convergence point is in front of the object, the object will be captured and displayed left of the midline for the left camera image and right of the midline for the right camera image (Figure 1A). The opposite applies when the convergence point is behind the object (Figure 1B). When the convergence point is on the object, the object will also appear in the midline of the camera images (Figure 1C), which presumably should yield the most comfortable viewing since no squinting is required to merge the images. To achieve comfortable stereo 3D video, the convergence point must be located on, or slightly behind, the object of interest, else the viewer is required to voluntarily squint outwards (exotropia).
The data was collected using a dual-arm collaborative industrial robot to position the cameras (Figure 2A-B). The robot weighs 38 kg without equipment. The robot is intrinsically safe; when it detects an unexpected impact, it stops moving. The robot was programmed to position the 5 Megapixel cameras with C-mount lenses along an arc-shaped trajectory stopping at predetermined baseline distances (Figure 2C). The cameras were attached to the robot hands using adaptor plates, as shown in Figure 3. Each camera recorded at 25 frames per second. Lenses were set at f-stop 1/8 with focus fixed on the object of interest (approximated geometrical center of the heart). Every image frame had a timestamp which was used to synchronize the two video streams.
Offsets between the robot wrist and the camera were calibrated. This can be achieved by aligning the crosshairs of the camera images, as shown in Figure 4. In this setup, the total translational offset from the mounting point on the robot wrist and the center of the camera image sensor was 55.3 mm in the X-direction and 21.2 mm in the Z-direction, displayed in Figure 5. The rotational offsets were calibrated at a convergence distance of 1100 mm and a baseline distance of 50 mm and adjusted manually with the joystick on the robot control panel. The robot in this study had a specified accuracy of 0.02 mm in Cartesian space and 0.01 degrees rotational resolution24. At a radius of 1100 m, an angle difference of 0.01 degrees offsets the center point 0.2 mm. During the full robot motion from 50-240 mm separation, the crosshair for each camera was within 2 mm from the ideal center of convergence.
The baseline distance was increased stepwise by symmetrical separation of the cameras around the center of the field of view in increments of 10 mm ranging from 50-240 mm (Figure 2). The cameras were kept at a standstill for 5 s in each position and moved between the positions at a velocity of 50 mm/s. The convergence point could be adjusted in X and Z directions using a graphical user interface (Figure 6). The robot followed accordingly within its working range.
The accuracy of the convergence point was estimated using the uniform triangles and the variable names in Figure 7A and B. The height &#39;z&#39; was calculated from the convergence distance &#39;R&#39; with the Pythagorean theorem as
<img alt="Equation 1" src="/files/ftp_upload/62786/62786eq01.jpg" />
When the real convergence point was closer than the desired point, as shown in Figure 7A, the error distance &#39;f1&#39; was calculated as
<img alt="Equation 2" src="/files/ftp_upload/62786/62786eq02.jpg" />
Similarly, when the convergence point was distal to the desired point, the error distance &#39;f2&#39; was calculated as
<img alt="Equation 3" src="/files/ftp_upload/62786/62786eq03.jpg" />
Here, &#39;e&#39; was the maximum separation between the crosshairs, at most 2 mm at maximum baseline separation during calibration (D = 240 mm). For R = 1100 mm (z = 1093 mm), the error was less than &#177; 9.2 mm. For R = 1400 mm (z = 1395 mm), the error was &#177; 11.7 mm. That is, the error of the placement of the convergence point was within 1% of the desired. The two test distances of 1100 mm and 1400 mm were therefore well separated.

<table><tbody><tr><td>2 C-mount lenses (35 mm F2.1, 5 M pixel)</td><td>Tamron</td><td>M112FM35</td><td>Rated for 5 Mpixel</td></tr><tr><td>3D glasses (DLP-link active shutter)</td><td>Celexon</td><td>G1000</td><td>Any compatible 3D glasses can be used</td></tr><tr><td>3D Projector</td><td>Viewsonic</td><td>X10-4K</td><td>Displays 3D in 1080, can be exchanged for other 3D projectors</td></tr><tr><td>6 M2 x 8 screws</td><td></td><td></td><td>To attach the cXimea cameras to the camera adaptor plates</td></tr><tr><td>8 M2.5 x 8 screws</td><td></td><td></td><td>To attach the circular mounting plates to the robot wrist</td></tr><tr><td>8 M5 x 40 screws</td><td></td><td></td><td>To mount the robot</td></tr><tr><td>8 M6 x 10 screws with flat heads</td><td></td><td></td><td>For attaching the circular mounting plate and the camera adaptor plates</td></tr><tr><td>Calibration checker board plate (25 by 25 mm)</td><td></td><td></td><td>Any standard checkerboard can be used, including printed, as long as the grid is clearly visible in the cameras</td></tr><tr><td>Camera adaptor plates, x2</td><td></td><td></td><td>Designed by the authors in &lt;em&gt;robot_camera_adaptor_plates.dwg&lt;/em&gt;, milled in aluminium.</td></tr><tr><td>Circular mounting plates, x2</td><td></td><td></td><td>Distributed with the permission of the designer Julius Klein and printed with ABS plastic on an FDM 3D printer. License Tecnalia Research &amp;amp; Innovation 2017. Attached as Mountingplate_ROBOT_SIDE_&lt;br/&gt; NewDesign_4.stl</td></tr><tr><td>Fix focus usb cameras, x2 (5 Mpixel)</td><td>Ximea</td><td>MC050CG-SY-UB</td><td>With Sony IMX250LQR sensor</td></tr><tr><td>Flexpendant</td><td>ABB</td><td>3HAC028357-001</td><td>robot touch display</td></tr><tr><td>Liveview</td><td></td><td></td><td>recording application</td></tr><tr><td>RobotStudio</td><td></td><td></td><td>&amp;nbsp;robot integrated development environment (IDE)</td></tr><tr><td>USB3 active cables (10.0 m), x2</td><td></td><td></td><td>Thumbscrew lock connector, water proofed.</td></tr><tr><td>YuMi dual-arm robot</td><td>ABB</td><td>IRB14000</td><td></td></tr></tbody></table>

robotized testing of camera positions to determine ideal configuration for stereo 3d visualization of open-heart surgery

Stereo 3D video from surgical procedures can be highly valuable for medical education and improve clinical communication. But access to the operating room and the surgical field is restricted. It is a sterile environment, and the physical space is crowded with surgical staff and technical equipment. In this setting, unobscured capture and realistic reproduction of the surgical procedures are difficult. This paper presents a method for rapid and reliable data collection of stereoscopic 3D videos at different camera baseline distances and distances of convergence. To collect test data with minimum interference during surgery, with high precision and repeatability, the cameras were attached to each hand of a dual-arm robot. The robot was ceiling-mounted in the operating room. It was programmed to perform a timed sequence of synchronized camera movements stepping through a range of test positions with baseline distance between 50-240 mm at incremental steps of 10 mm, and at two convergence distances of 1100 mm and 1400 mm. Surgery was paused to allow 40 consecutive 5-s video samples. A total of 10 surgical scenarios were recorded.

An acceptable evaluation video with the right image placed at the top in top-bottom stereoscopic 3D is shown in Video1. A successful sequence should be sharp, focused, and without unsynchronized image frames. Unsynchronized video streams will cause blur, as shown in the file Video 2. The convergence point should be centered horizontally, independent of the camera separation, as seen in Figure 9A,B. When the robot transitions between the positions, there is a small shake in the video, which is to be expected at a transition velocity of 50 mm/s. With the too large separation between the right and left image, the brain cannot fuse the images into one 3D image, see Figure 9C and Video 3.
The position of the heart in the images should be centered during the entire video, as shown in Figure 1C. Several reasons can cause this to fail: (1) The convergence point is too far away from the heart, see Figure 7. The camera positions relative to the patient can be modified from the robot application setting screen (Figure 6B). (2) The camera tool coordinate system is not properly configured. The robot program will simultaneously move the camera symmetrically in a radial motion around the convergence point (Figure 2C) and rotate the cameras around the camera tool coordinate system (Figure 5). If the camera adaptor plates (Figure 3) are assembled or mounted incorrectly, the default values will not work. Rerun step 2.1-2.4 and ensure that the crosshairs in the recording application (Figure 6) point at the same object during the full robot motion. When adjusting the coordinate frames, ensure that the object used for calibration (Figure 4) is centered between the cameras; otherwise, the calibration will result in non-symmetrical coordinate frames.
If the colors are incorrect after debayering with the debayering application (Figure 8), the captured videos have the wrong debayering format. This requires the user to modify the code for the debayering application or use another debayering tool. Similarly, if the automatic synchronization between the stereo videos failed, the user should use video editing programs such as Premiere Pro to align the videos.
To analyze the results, the video should be displayed on a 3D projector for the intended audience. The audience can subjectively rate how well the 3D video corresponds to the real-life situation. The labels added in step 6.3 can be used to score different distances.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/62786/62786fig01.jpg" /> 
Figure 1. Placement of convergence point. Different placement of convergence points relative to the object of interest (grey dot). (A) Convergence point in front of the object, (B) behind the object, and (C) on the object. The midline for each camera image is shown with a dotted line. The surgeon is shown from above, standing between the cameras. At the top, the resulting position of the object in the left and the right camera images are displayed relative to the midline. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/62786/62786fig02.jpg" /> 
Figure 2: Robot motion. The camera separation was increased from (A) 50 mm to (B) 240 mm with incremental steps of 10 mm. (C) The robot moved the cameras radially, always pointing the cameras toward the convergence point - the heart. Here the distance D is the distance between the cameras, R is the radius 1100 or 1400 mm, and a is the angle of the cameras, sin(a) = D/2R. The right and left cameras were angled a degree in the negative and positive direction, respectively, around the tool Z-axis. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/62786/62786fig03.jpg" /> 
Figure 3: Mounting cameras on the robot. (A) Exploded view of the components for one camera: lens, camera sensor, camera adaptor plate, circular mounting plate, robot wrist, and screws. The two assembled camera adaptors are shown from (B) the robot side and (C) the front. (D)Adaptors attached to the robot wrist with four M2.5 screws. (E) USB cables connected to the cameras. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/62786/62786fig04.jpg" /> 
Figure 4: Camera calibration with the recording application. A calibration grid and a screw-nut were used to calibrate the camera tool coordinate systems relative to the robot writs. The cameras should be angled so that the nut is in the center of the images. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/62786/62786fig05.jpg" /> 
Figure 5: Camera tool coordinate system. The X-axis (red), Y-axis (green), and Z-axis (blue) of the camera tool coordinate system. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig05large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/62786/62786fig06.jpg" /> 
Figure 6: The robot application. (A) Display of the main screen on the touch display for running the experiments. (B) The setup screen for tool calibration and adjustment of the convergence point. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig06large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/62786/62786fig07.jpg" /> 
Figure 7: Error estimation. Convergence error (A) above and (B) below the desired convergence point. The horizontal baseline distance (D = 240 mm), the distance between the cameras, and the convergence point (R = 1100). The vertical distance between the cameras and the convergence point (z = 1093 mm), the maximum separation between the image center points (crosshairs) (e = 2 mm), the vertical error distance when the real convergence point is above the desired convergence position (f1 = 9 mm). The vertical error distance when the real convergence point is below the desired convergence position (f2 = 9.2 mm). Figure not drawn to scale. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig07large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 8" class="xfigimg" src="/files/ftp_upload/62786/62786fig08.jpg" /> 
Figure 8: Postprocessing applications for debayering and merging. (A) Start and (B) Finish screens of the debayer application. (C) Start and (D) Finish screens of the merge application. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig08large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 9" class="xfigimg" src="/files/ftp_upload/62786/62786fig09.jpg" /> 
Figure 9: Snapshots of finished stereo videos. Only every other pixel row was used from the original images to comply with standard top/bottom 3D stereo format. Upper images are from the right camera and lower from the left camera. (A) 3D stereo image with 50 mm baseline distance and the convergence point on the OR-table behind the heart. (B) 3D stereo image with 240 mm baseline distance and the convergence point at the OR-table behind the heart. (C) 3D stereo image with 240 mm baseline distance and the convergence point 300 mm behind the heart. <a href="https://www.jove.com/files/ftp_upload/62786/62786fig09large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Video 1. Stereo 3D video at 1100 mm. The convergence point is on the heart, 1100 mm from the cameras. The video starts with a baseline distance of 50 mm (A) and increases with steps of 10 mm to 240 mm (T). <a href="https://www.jove.com/files/ftp_upload/62786/Video1.mp4" target="_blank">Please click here to download this Video.</a>
Video 2. Unsynchronized stereo 3D video. The right and left videos are not synchronized which causes blur when viewed in 3D. <a href="https://www.jove.com/files/ftp_upload/62786/Video2.mp4" target="_blank">Please click here to download this Video.</a>
Video 3. Stereo 3D video at 1400 mm. The convergence point is behind the heart, 1400 mm from the cameras. The video <a href="https://www.jove.com/files/ftp_upload/62786/Video3.mp4" target="_blank">Please click here to download this Video.</a>

Watch this Scientific Journal Video about Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery at JoVE.com

Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery

The human depth perception of 3D stereo videos depends on the camera separation, point of convergence, distance to, and familiarity of the object. This paper presents a robotized method for rapid and reliable test data collection during live open-heart surgery to determine the ideal camera configuration.

Stereo 3D video from surgical procedures can be highly valuable for medical education and improve clinical communication. But access to the operating room ...

robotized-testing-camera-positions-to-determine-ideal-configuration

Nanyang Technological University

Research

JoVE Journal

Medicine

2.0K Views.  Skåne University Hospital. The human depth perception of 3D stereo videos depends on the camera separation, point of convergence, distance to, and familiarity of the object. This paper presents a robotized method for rapid and reliable test data collection during live open-heart surgery to determine the ideal camera configuration. 

Video: Robotized Testing of Camera Positions to Determine Ideal Configuration for Stereo 3D Visualization of Open-Heart Surgery

Measuring Sensitivity to Viewpoint Change with and without Stereoscopic Cues

The speed and accuracy of object recognition is compromised by a change in viewpoint; demonstrating that human observers are sensitive to this transformation. Here we discuss a novel method for simulating the appearance of an object that has undergone a rotation-in-depth, and include an exposition of the differences between perspective and orthographic projections. Next we describe a method by which human sensitivity to rotation-in-depth can be measured. Finally we discuss an apparatus for creating a vivid percept of a 3-dimensional rotation-in-depth; the Wheatstone Eight Mirror Stereoscope. By doing so, we reveal a means by which to evaluate the role of stereoscopic cues in the discrimination of viewpoint rotated shapes and objects.

We discuss a novel method forviewpoint-rotation of visual stimuli, and demonstrate using a mirror stereoscopethe three-dimensional percept of rotation-in-depth. The technique can be used to investigate the role of stereoscopic cues in encoding viewpoint-rotated figures.

The speed and accuracy of object recognition is compromised by a change in viewpoint; demonstrating that human observers are sensitive to this ...

Behavior

Novel 3D/VR Interactive Environment for MD Simulations, Visualization and Analysis

The increasing development of computing (hardware and software) in the last decades has impacted scientific research in many fields including materials science, biology, chemistry and physics among many others. A new computational system for the accurate and fast simulation and 3D/VR visualization of nanostructures is presented here, using the open-source molecular dynamics (MD) computer program LAMMPS. This alternative computational method uses modern graphics processors, NVIDIA CUDA technology and specialized scientific codes to overcome processing speed barriers common to traditional computing methods. In conjunction with a virtual reality system used to model materials, this enhancement allows the addition of accelerated MD simulation capability. The motivation is to provide a novel research environment which simultaneously allows visualization, simulation, modeling and analysis. The research goal is to investigate the structure and properties of inorganic nanostructures (e.g., silica glass nanosprings) under different conditions using this innovative computational system. The work presented outlines a description of the 3D/VR Visualization System and basic components, an overview of important considerations such as the physical environment, details on the setup and use of the novel system, a general procedure for the accelerated MD enhancement, technical information, and relevant remarks. The impact of this work is the creation of a unique computational system combining nanoscale materials simulation, visualization and interactivity in a virtual environment, which is both a research and teaching instrument at UC Merced.

A new computational system featuring GPU-accelerated molecular dynamics simulation and 3D/VR visualization, analysis and manipulation of nanostructures has been implemented, representing a novel approach to advance materials research and promote innovative investigation and alternative methods to learn about material structures with dimensions invisible to the human eye.

The increasing development of computing (hardware and software) in the last decades has impacted scientific research in many fields including materials ...

Engineering

A Pipeline for 3D Multimodality Image Integration and Computer-assisted Planning in Epilepsy Surgery

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality image integration can be technically demanding, and is underutilised in clinical practice. We have developed a single software platform for image integration, 3D visualization and surgical planning. Here, our pipeline is described in step-by-step fashion, starting with image acquisition, proceeding through image co-registration, manual segmentation, brain and vessel extraction, 3D visualization and manual planning of stereoEEG (SEEG) implantations. With dissemination of the software this pipeline can be reproduced in other centres, allowing other groups to benefit from 3DMMI. We also describe the use of an automated, multi-trajectory planner to generate stereoEEG implantation plans. Preliminary studies suggest this is a rapid, safe and efficacious adjunct for planning SEEG implantations. Finally, a simple solution for the export of plans and models to commercial neuronavigation systems for implementation of plans in the operating theater is described. This software is a valuable tool that can support clinical decision making throughout the epilepsy surgery pathway.

We describe the steps to use our custom designed software for image integration, visualization and planning in epilepsy surgery.

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality ...

Biplanar Videoradiography to Study the Wrist and Distal Radioulnar Joints

Accurate measurement of skeletal kinematics in vivo is essential for understanding normal joint function, the influence of pathology, disease progression, and the effects of treatments. Measurement systems that use skin surface markers to infer skeletal motion have provided important insight into normal and pathological kinematics, however, accurate arthrokinematics cannot be attained using these systems, especially during dynamic activities. In the past two decades, biplanar videoradiography (BVR) systems have enabled many researchers to directly study the skeletal kinematics of the joints during activities of daily living. To implement BVR systems for the distal upper extremity, videoradiographs of the distal radius and the hand are acquired from two calibrated X-ray sources while a subject performs a designated task. Three-dimensional (3D) rigid-body positions are computed from the videoradiographs via a best-fit registrations of 3D model projections onto&#160;to each BVR view. The 3D models are density-based image volumes of the specific bone derived from independently acquired computed-tomography data. Utilizing graphics processor units and high-performance computing systems, this model-based tracking approach is shown to be fast and accurate in evaluating the wrist and distal radioulnar joint biomechanics. In this study, we first summarized the previous studies that have established the submillimeter and subdegree agreement of BVR with an in vitro&#160;optical motion capture system in evaluating the wrist and distal radioulnar joint kinematics. Furthermore, we used BVR to compute the center of rotation behavior of the wrist joint, to evaluate the articulation pattern of the components of the implant upon one another, and to assess the dynamic change of ulnar variance during pronosupination of the forearm. In the future, carpal bones may be captured in greater detail with the addition of flat panel X-ray detectors, more X-ray sources (i.e., multiplanar videoradiography), or advanced computer vision algorithms.

Biplanar videoradiography (BVR) is an advanced imaging technique for understanding the three-dimensional movement of skeletal bones and implants. Combining density-based image volumes and videoradiographs of the distal upper extremity, BVR is used to study the in vivo motion of the wrist and distal radioulnar joint, as well as joint arthroplasties.

Accurate measurement of skeletal kinematics in vivo is essential for understanding normal joint function, the influence of pathology, disease progression, ...

Bioengineering

Measuring 3D In-vivo Shoulder Kinematics using Biplanar Videoradiography

The shoulder is one of the human body's most complex joint systems, with motion occurring through the coordinated actions of four individual joints, multiple ligaments, and approximately 20 muscles. Unfortunately, shoulder pathologies (e.g., rotator cuff tears, joint dislocations, arthritis) are common, resulting in substantial pain, disability, and decreased quality of life. The specific etiology for many of these pathologic conditions is not fully understood, but it is generally accepted that shoulder pathology is often associated with altered joint motion. Unfortunately, measuring shoulder motion with the necessary level of accuracy to investigate motion-based hypotheses is not trivial. However, radiographic-based motion measurement techniques have provided the advancement necessary to investigate motion-based hypotheses and provide a mechanistic understanding of shoulder function. Thus, the purpose of this article is to describe the approaches for measuring shoulder motion using a custom biplanar videoradiography system. The specific objectives of this article are to describe the protocols to acquire biplanar videoradiographic images of the shoulder complex, acquire CT scans, develop 3D bone models, locate anatomical landmarks, track the position and orientation of the humerus, scapula, and torso from the biplanar radiographic images, and calculate the kinematic outcome measures. In addition, the article will describe special considerations unique to the shoulder when measuring joint kinematics using this approach.

Biplane videoradiography can quantify shoulder kinematics with a high degree of accuracy. The protocol described herein was specifically designed to track the scapula, humerus, and the ribs during planar humeral elevation, and outlines the procedures for data collection, processing, and analysis. Unique considerations for data collection are also described.

The shoulder is one of the human body's most complex joint systems, with motion occurring through the coordinated actions of four individual joints, ...

Translational Brain Mapping at the University of Rochester Medical Center: Preserving the Mind Through Personalized Brain Mapping

The Translational Brain Mapping Program at the University of Rochester is an interdisciplinary effort that integrates cognitive science, neurophysiology, neuroanesthesia, and neurosurgery. Patients who have tumors or epileptogenic tissue in eloquent brain areas are studied preoperatively with functional and structural MRI, and intraoperatively with direct electrical stimulation mapping. Post-operative neural and cognitive outcome measures fuel basic science studies about the factors that mediate good versus poor outcome after surgery, and how brain mapping can be further optimized to ensure the best outcome for future patients. In this article, we describe the interdisciplinary workflow that allows our team to meet the synergistic goals of optimizing patient outcome and advancing scientific understanding of the human brain.

This article provides an overview of a multi-modal brain mapping program designed to identify regions of the brain that support critical cognitive functions in individual neurosurgery patients.

The Translational Brain Mapping Program at the University of Rochester is an interdisciplinary effort that integrates cognitive science, neurophysiology, ...

Neuroscience

An Endoscopic Transcanal Transpromontorial Approach for Vestibular Schwannomas using a Computer-Based Three-Dimensional Imaging System

A 2D monocular endoscope has been used in transcanal transpromontory vestibular schwannoma surgery instead of craniotomy. However, the absence of depth perception is the limitation of this approach. With the loss of depth perception, the surgeon will be not able to perform delicate and particularly complicated surgery. A binocular endoscope has been developed to provide stereoscopic vision with better depth perception for complicated anatomic structures and has been applied in some endoscopic surgeries. However, the diameter of the endoscope is a limitation in the performance of transcanal otologic surgeries. A small diameter endoscope facilitates easier surgery in a restricted space. A computer-based 3D imaging system can obtain 3D images in real-time using a small monocular endoscope. In this study, to evaluate the feasibility of a computer-based 3D imaging system for endoscopic lateral skull base surgery, we applied this 3D imaging system in a transcanal transpromontorial approach in two patients with vestibular schwannomas. The surgical procedure was completed without complication in these two cases. There was no mortality, perioperative complications, nor notable postoperative complications. Using this computer-based 3D imaging system, a better depth perception and stereoscopic vision was observed compared to a conventional 2D endoscope. The improvement in depth perception offers superior management of the complicated surgical anatomy.

Here, we present a transcanal transpromontorial approach for vestibular schwannomas using a computer-based three-dimensional (3D) imaging system combined with a two-dimensional (2D) endoscope. This system provided stereoscopic vision, better depth perception and reduced visual fatigue. This 3D imaging system enabled the application of 3D vision technology in endoscopic lateral skull base surgery.

A 2D monocular endoscope has been used in transcanal transpromontory vestibular schwannoma surgery instead of craniotomy. However, the absence of depth ...

Virtual Reality-3D Modeling for Minimally Invasive Neuro-Interventional Planning

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular ...

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) for minimally invasive pedicle screw placement. The cadaveric porcine specimens were placed on a surgical table and draped with sterile covers. The levels of interest were identified using fluoroscopy, and a dynamic reference frame was attached to the spinous process of a vertebra in the region of interest. Cone beam computerized tomography (CBCT) was performed, and a 3D rendering was automatically generated, which was used for the subsequent planning of the pedicle screw placements. Each surgeon was fitted with an HMD that was individually eye-calibrated and connected to the spinal navigation system.
Navigated instruments, tracked by the navigation system and displayed in 2D and 3D in the HMD, were used for 33 pedicle cannulations, each with a diameter of 4.5 mm. Postprocedural CBCT scans were assessed by an independent reviewer to measure the technical (deviation from the planned path) and clinical (Gertzbein grade) accuracy of each cannulation. The navigation time for each cannulation was measured. The technical accuracy was 1.0 mm &#177; 0.5 mm at the entry point and 0.8 mm &#177; 0.1 mm at the target. The angular deviation was 1.5&#176; &#177; 0.6&#176;, and the mean insertion time per cannulation was 141 s &#177; 71 s. The clinical accuracy was 100% according to the Gertzbein grading scale (32 grade 0; 1 grade 1). When used for minimally invasive pedicle cannulations in a porcine model, submillimeter technical accuracy and 100% clinical accuracy could be achieved with this protocol.

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) ...

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...