Name: pioneering patient-specific approaches for precision surgery using imaging and virtual reality
Uploaded: 2024-04-05T13:20:00.000+00:00
Duration: 6 min 18 s
Description: Watch this Scientific Journal Video about Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions at JoVE.com

We extend a special thanks to the review committee for their insightful feedback, and to the editorial for their invaluable comments, expertise, guidance, and support throughout the writing process of this article. We greatly appreciate the collaborative environment fostered by the Mission Partners at OSF HealthCare System, which enhanced the quality of this work. Thanks to OSF HealthCare System for providing resources and support and to the Advanced Imaging and Modeling Lab at Jump Simulation and Education Center for their assistance.

De-identified human DICOMs or DICOMs for patient care are used in accordance with institutional guidelines for patient care, the Health Insurance Portability and Accountability Act of 1996 (HIPAA), and collaboration with the Institutional Review Board (IRB) when appropriate.
1. Segment patient-specific&#160;anatomy
<ol>
	<li>Acquisition of medical scans
	<ol>
		<li>Begin the segmentation process with a physician or surgeon ordering medical scans. These scans are a part of the standard patient care protocol and do not introduce extra procedures.</li>
		<li>If the physician knows they will request segmentation, ensure they ask for thinly sliced data sets to be exported from the MRI or CT scanner. In most cases, these thin slices are less than 1 mm thick; however, this resolution can vary between scanners. Take CTA scans with contrast in order to ensure proper segmentation of vasculature and blood pools.
		<ol>
			<li>For data acquisition, acquire MRI 3D sequence with the following suggested parameters: run in axial, ensure slice thickness and space between slices of 0.625 mm or smaller, zero spacing. Acquire CT 3D series with the following suggested parameters: slice scanner in helical mode, slice thickness and space between slices of 0.625mm, e.g., Neuro: Kvp of 120, Smart mA range of 100-740, rotation speed at .5ms, or Cardiac: Kvp of 70, Smart mA range of 201-227 (smart MA mode 226), rotation speed at 0.28 ms14. Follow the institution&#39;s helical parameters for each body part. 
			NOTE: The 3D sequence should be acquired so that in the reconstruction, there is a near isotropic resolution in the axial, coronal, and sagittal planes. In most cases, the thinnest slices should be acquired. The 3D sequence is run in addition to the institution&#39;s standard protocol for imaging. However, it is run at the same time, so there is minimal additional work, radiation exposure, and expense for the clinical team and patient.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Ask the physician to request the segmentation of the model, specifying what anatomy will be the central focus for the segmentation process (A physician or surgeon usually completes this step).</li>
	<li>Download scan data and save it locally.
	<ol>
		<li>If more than one scan was taken, ensure that DICOM&#160;datasets from the scan are compared to determine which scan set has the thinnest slice setting and best contrast, as this will provide the highest resolution 3D models when segmented.</li>
		<li>Once the best image set is determined, download it from the image database for segmentation, anonymize, or leave the data as is with protected health information (PHI). This protocol will work with an anonymized DICOM.</li>
	</ol>
	</li>
	<li>Import the DICOM dataset into the segmentation software. 
	NOTE: The following instruction set uses terms specific to the Materialise Mimics segmentation software. While the Materialise Suite is subscription-based software, there are open-source alternatives such as 3DSlicer. Tool names and terminology may vary across other segmentation tools.</li>
	<li>Create a rough mask of the target anatomy, such as bone, blood pool, aneurysm, etc.
	<ol>
		<li>Under the SEGMENT tab, select New Mask tool.</li>
		<li>Set the upper and lower threshold boundaries by clicking and dragging both to capture as much of the relevant target anatomy as possible while limiting the capture of surrounding tissues. Click and drag the boundaries within the threshold tool or input the desired Hounsfield Unit (HU).</li>
		<li>While setting thresholds, crop to a specific area of the scan to avoid excess selection of surrounding tissue. The upper and lower threshold boundaries vary greatly depending on scan type, sequence type, amount of contrast, and patient.</li>
		<li>Click OK to finalize the rough mask.</li>
	</ol>
	</li>
	<li>Use other tools within the SEGMENT tab to remove unnecessary portions of the mask or add missing tissue as needed.
	<ol>
		<li>Use the Region Grow tool to separate all voxels of the mask directly connected to a voxel selected by the user; use Edit Mask to add or remove voxels in the mask via both the 2D and 3D windows; use Multiple Slices Edit to add or remove voxels through interpolation between slices farther apart; and use Fill Holes or Smart Fill to fill holes of a user-defined size within the mask.</li>
		<li>Continue refining the mask making use of tools in the tab until the 2D to 3D interpolation is as accurate as possible.</li>
	</ol>
	</li>
	<li>Repeat steps 1.5 and 1.6 for all target anatomies.</li>
	<li>Consult a physician regarding the completed segmentation to ensure accuracy.
	<ol>
		<li>Show the completed masks to a physician to ensure important anatomy has not been omitted and excess anatomy has not been included. In most cases, consult the physician who requests the segmentation for quality control. The physician ensures the portion of the DICOM highlighted by the mask on each slice is as accurate as possible (see Figure 1).</li>
	</ol>
	</li>
	<li>Export segmentation for further processing.
	<ol>
		<li>Convert finalized masks to parts by using the Calculate Part tool located in the Project Management menu on the right side.</li>
		<li>Export calculated parts to 3D files by right-clicking on The Part and selecting Export STL.</li>
	</ol>
	</li>
</ol>
2. Prepare the model for virtual reality
<ol>
	<li>Create a new Blender project and remove the default scene elements. Press the a key to highlight all visible elements, and then x followed by Enter to remove them from the scene. 
	NOTE: Blender is a free and open-source modeling software. While other modeling software may be able to accomplish the same tasks, the terminology used in this step will be specific to Blender.</li>
	<li>Import the anatomy files through File &#62; Import &#62; Stl (.stl).</li>
	<li>Align the patient&#39;s anatomy with the world&#39;s origin.
	<ol>
		<li>Select all patient anatomy in order to maintain relative positioning. To do this, press the a key after all files are imported.</li>
		<li>Use the Move and Rotate tools to align the anatomy with the world&#39;s origin. Ensure the patient&#39;s nose is aligned with one axis, with the perpendicular axis contacting the ear area and the top of the skull, respectively. Use orthographic views, which can be activated with the widget in the top right corner of the blender interface.</li>
	</ol>
	</li>
	<li>Import the VR protractor and align it to the patient&#39;s anatomy. This protractor has been specially designed by the OSF engineering team to aid in acquiring C-arm angles in VR, based on angles in 3D space.
	<ol>
		<li>Import the protractor .stl file from Supplementary File 1.</li>
		<li>Align (0,0) on the protractor, represented by the longest measurement mark, with the patient&#39;s nose. Orient the gap in the protractor arms toward the patient&#39;s feet.</li>
		<li>Scale the protractor accordingly. For most cases, scale the protractor quite small in order to ensure ease of measurement after scaling in VR. In the case of aneurysms, try to scale the protractor in a way that it lies just outside of the aneurysm area.</li>
	</ol>
	</li>
	<li>Align anatomy origins to the world origin.
	<ol>
		<li>Right-click click in the main viewport and select Snap &#62; Cursor to World Origin. This ensures the 3D cursor is aligned with the global origin.</li>
		<li>Select all visible models with the a key.</li>
		<li>Right-click, click in the viewport and select Set Origin &#62; Origin to 3D Cursor. This aligns the 3D origin of all models to the same point, ensuring they will line up and scale properly when imported to VR.</li>
	</ol>
	</li>
	<li>Add texture or color to the models as desired for better distinction in VR.
	<ol>
		<li>This is an optional step. Select the individual .stl files, then click the Material Properties tab on the right-hand side of the screen. Under this tab, the base color can be adjusted to the desired color. Repeat this step for each object to add color.</li>
	</ol>
	</li>
	<li>Export the finalized model as a single. glb/.gltf file. Ensure that none of the Limit to&#160;options under the Include tab of the export window are selected. 
	NOTE: The. glb/.gltf file format reflects the file type required for use in the VR software, as well as for upload to the NIH 3D library. Other export types may be needed for different software.</li>
</ol>
3. Training medical professionals in virtual reality
NOTE: The following instructions are written to be used with the Enduvo digital classroom software. While it may be possible to use other 3D viewing software, the ability to move models, place cameras, and record physician positioning are some features that make this software ideal for this procedure. Different VR headsets, controllers, and software combinations may have different controls.
<ol>
	<li>Create a new lesson.
	<ol>
		<li>Import the .gltf file exported in step 2.7 in the lesson creation menu. The software may produce a message stating: The file type (GLB) that you are attempting to upload is not currently fully supported. Ignore this message and click the Confirm button.</li>
	</ol>
	</li>
	<li>Open the lesson in VR for finalization.
	<ol>
		<li>Using the transparency menu accessed by pressing down on the thumbpad or joystick of one controller, hide all models except the target anatomy. The aneurysm should be the only model visible.</li>
	</ol>
	</li>
	<li>Place the surgeon or physician in VR and allow them some time to familiarize themselves with the 3D space and functions and the anatomy in the lesson.</li>
	<li>Once the surgeon is comfortable with the anatomy, begin recording.
	<ol>
		<li>Start the record function using either the virtual button in the VR space or the record button on the secondary monitor.</li>
		<li>Let the surgeon rotate the target anatomy to find preferred viewing angles for both anteroposterior (AP) and lateral fluoroscopy views. After finding a preferred angle, ask the surgeon to pause briefly and state they have found a preferred angle and whether the current viewing angle is AP or lateral.</li>
		<li>After all preferred angles are found, stop recording with either the VR button or external monitor.</li>
	</ol>
	</li>
</ol>
4. Generation of fluoroscopy roadmap in VR
<ol>
	<li>Acquire fluoroscopy analogs using recorded surgeon placement.
	<ol>
		<li>Place an image mimicking the gray hue background of a fluoroscopy image behind the model in the virtual space. Use the select button on the controller, often the trigger on the backside of the controller, to manipulate the image as needed. This creates a consistent background color which makes it easier to view the anatomy and is more representative of fluoroscopy.</li>
		<li>Place the camera in line with the surgeon&#39;s view at the time declared as a preferred viewing angle, ensuring the camera is pointed roughly at the center of the target anatomy. The surgeon will appear as a set of floating glasses and two controllers in VR.</li>
		<li>Capture a 2D snapshot with the camera in the desired position. Repeat the step for each preferred angle.</li>
	</ol>
	</li>
	<li>Acquire C-arm angles using the surgeon&#39;s movements and protractor.
	<ol>
		<li>Pause the recorded lesson when the surgeon declares a preferred viewing angle.</li>
		<li>Click the Trackpad to open the quick menu and select the On/Off check box to unhide the protractor attached to the model.</li>
		<li>Use the controller&#39;s grab button to select and manipulate a pointer or straightedge in line with the surgeon&#39;s viewpoint, also passing through the origin of the protractor.</li>
		<li>Step back from the model and view the angles from the orthographic viewpoints corresponding with the C-arm motions.</li>
		<li>For a neurology case with the patient&#39;s nose oriented to 0&#176; on all C-arm axes, take the AP angles from the sagittal and axial planes. Take the lateral angles from the coronal and axial planes. In both cases, the axial plane corresponds with the right and left angles of the C-arm, while the sagittal and coronal planes correspond with the cranial and caudal angles.</li>
		<li>Repeat steps above for each preferred angle.</li>
	</ol>
	</li>
</ol>

Virtual Reality-3D Modeling for Minimally Invasive Neuro-Interventional Planning

<ol>
	<li>Xia, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+virtual+reality+surgical+planning+and+simulation+workbench+for+orthognathic+surgery.">Three-dimensional virtual reality surgical planning and simulation workbench for orthognathic surgery.</a> Int J Adult Orthodon Orthognath Surg. 15 (4), 265-282 (2000).</li><li>Boedecker, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+virtual+3D-models+in+surgical+planning:+workflow+of+an+immersive+virtual+reality+application+in+liver+surgery.">Using virtual 3D-models in surgical planning: workflow of an immersive virtual reality application in liver surgery.</a> Langenbecks Arch Surg. 406, 911-915 (2021).</li><li>Reitinger, B., Bornik, A., Beichel, R., Schmalstieg, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Liver+surgery+planning+using+virtual+reality.">Liver surgery planning using virtual reality.</a> IEEE Comput Graph Appl. 26 (6), 36-47 (2006).</li><li>Robiony, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+surgical+planning+for+maxillofacial+distraction+osteogenesis:+The+role+of+Reverse+Engineering+Rapid+Prototyping+and+cooperative+work.">Virtual reality surgical planning for maxillofacial distraction osteogenesis: The role of Reverse Engineering Rapid Prototyping and cooperative work.</a> J Oral Maxillofacial Surg. 65 (6), 1198-1208 (2007).</li><li>Xia, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three-dimensional+virtual-reality+surgical+planning+and+soft-tissue+prediction+for+orthognathic+surgery.">Three-dimensional virtual-reality surgical planning and soft-tissue prediction for orthognathic surgery.</a> IEEE Trans Info Tech Biomed. 5 (2), 97-107 (2001).</li><li>Kim, Y., Kim, H., Kim, Y. O. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+and+augmented+reality+in+plastic+surgery:+A+Review.">Virtual reality and augmented reality in plastic surgery: A Review.</a> Arch Plastic Surg. 44 (3), 179-187 (2017).</li><li>Ayoub, A., Pulijala, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+application+of+virtual+reality+and+augmented+reality+in+oral+&amp;+maxillofacial+surgery.">The application of virtual reality and augmented reality in oral &amp; maxillofacial surgery.</a> BMC Oral Health. 19 (1), 238 (2019).</li><li>Ulbrich, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Advantages+of+a+training+course+for+surgical+planning+in+virtual+reality+for+oral+and+maxillofacial+surgery:+Crossover+study.">Advantages of a training course for surgical planning in virtual reality for oral and maxillofacial surgery: Crossover study.</a> JMIR Serious Games. 11, e40541 (2023).</li><li>Lyuksemburg, V., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+for+preoperative+planning+in+complex+surgical+oncology:+A+single-center+experience.">Virtual reality for preoperative planning in complex surgical oncology: A single-center experience.</a> J Surg Res. 291, 546-556 (2023).</li><li>Macario, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=What+does+one+minute+of+operating+room+time+cost.">What does one minute of operating room time cost.</a> J Clin Anesth. 22 (4), 233-236 (2010).</li><li>Bramlet, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+visualization+of+patient+specific+heart+model.">Virtual reality visualization of patient specific heart model.</a> J Cardiovasc Mag Res. 18 (1), 13 (2016).</li><li>Guillot, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Relationship+between+spatial+abilities,+mental+rotation+and+functional+anatomy+learning.">Relationship between spatial abilities, mental rotation and functional anatomy learning.</a> Adv Health Sci Edu. 12 (4), 491-507 (2006).</li><li>Juhnke, B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Use+of+virtual+reality+for+pre-surgical+planning+in+separation+of+conjoined+twins:+A+case+report.">Use of virtual reality for pre-surgical planning in separation of conjoined twins: A case report.</a> Proc Inst Mech Eng H. 233 (12), 1327-1332 (2019).</li><li>Mattus, M. S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Creation+of+patient-specific+silicone+cardiac+models+with+applications+in+pre-surgical+plans+and+hands-on+training.">Creation of patient-specific silicone cardiac models with applications in pre-surgical plans and hands-on training.</a> J Vis Exp. (180), e62805 (2022).</li><li>Kern, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Angiographic+projections+made+simple:+An+easy+guide+to+understanding+oblique+views.">Angiographic projections made simple: An easy guide to understanding oblique views.</a> Cath Lab Digest. 19 (8), (2011).</li><li>Reinschluessel, A. V., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+for+surgical+planning+-+evaluation+based+on+two+liver+tumor+resections.">Virtual reality for surgical planning - evaluation based on two liver tumor resections.</a> Front Surg. 9, 821060 (2022).</li><li>Tang, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+technology+for+preoperative+planning+and+intraoperative+navigation+during+Hepatobiliary+Surgery:+A+review+of+current+methods.">Augmented reality technology for preoperative planning and intraoperative navigation during Hepatobiliary Surgery: A review of current methods.</a> Hepatobiliary Pancreat Dis Int. 17 (2), 101-112 (2018).</li><li>Manzie, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Virtual+reality+digital+surgical+planning+for+jaw+reconstruction:+A+usability+study.">Virtual reality digital surgical planning for jaw reconstruction: A usability study.</a> ANZ J Surg. 93 (5), 1341-1347 (2023).</li><li>Corren, Y. B., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=CT-based+Simulation+of+Projection+Angiography+Using+the+SlicerHeart+Virtual+Cath+Lab+[Poster+Presentation].">CT-based Simulation of Projection Angiography Using the SlicerHeart Virtual Cath Lab [Poster Presentation].</a> World Congress of Pediatric Cardiology and Cardiac Surgery. , (2023).</li></ol>

Matthew Bramlet is co-founder of Enduvo, Inc. The remaining authors declare that they have no relevant or material financial interests that relate to the research described in this paper.

3D modeling was introduced to medical workflows with the advent of 3D printing technologies2,3,4,6,7,9,11, but VR affords novel applications of 3D technology beyond a physical 3D object. Efforts to replicate anatomy and scenarios in a virtual world allow for personalized medical practice on individual patients1,2,3,4,9,11,13,16. This work demonstrates the expansive capability of creating new pre-surgical simulations in a digital world with minimal effort.
Throughout the presented protocol, there are several steps that are critical to the success of a case. The most important factor in producing adequate results with proper resolution is acquiring the correct medical imaging. The presented process does not require additional scans on the patient, using the standard CTA scan that is scheduled for every intracranial aneurysm case. Most scanners will store scans for a short time, depending on the scanner model and health system protocol, allowing the imaging technician to upload the acquired thin slices of the scans typically less than 1 mm thick slices are often not stored longer than a few days due to the storage size. These thin slices allow for greater detail and the inclusion of smaller anatomy, such as blood vessels. After segmentation has taken place, physician quality control must be completed to ensure the 3D models generated represent the patient anatomy as accurately as possible in future steps. Quality control of all models should be a part of the segmentation process, minimizing the potential for propagation of error throughout the remainder of the protocol. Quality control includes blood vessel borders and segmenting the aneurysm separately from the surrounding vessels, similar to how it would present with contrast. Quality control with a physician is of utmost importance as the physician holds the entirety of the responsibility for the accuracy of the models, especially if the models are to be used in further decision-making of patient&#39;s treatment. In some circumstances, it may be feasible or practical for the physician to complete the segmentation step themselves.
The next important step in the protocol is maintaining spatial model alignment while integrating the protractor measurement tool. Blender has proved to be an extremely helpful tool for this step as it allows for the combination of multiple STL file types into one combined file with multiple layers, each of which is spatially aligned and can be colored or textured for added clarity. Additionally, during this step, the protractor STL is added so that angle data can be gathered in VR. This protractor model was specifically developed using a computer aided design (CAD) tool, SolidWorks. Taking advantage of high-precision dimensioning tools within the software, an arc with tic marks denoting every 5&#176; in all three axes was created. The protractor also has crosshairs denoting the true center of that model and allowing for alignment to the center of the patient&#39;s anatomy. There is also a large bar within the model signifying (0,0) and is to be aligned with the patient nose. Also, it is important to note that this was done manually and could have increased the error percentage. Alignment is of utmost importance to ensure the accuracy of all potential angle measurements. Once properly aligned, the model is ready for VR, where recording of the physician placement of the model allows for future determination of the angles at which the model has been placed. During the recording, everything within the virtual space is recorded in reference to one another, most importantly the physician&#39;s point-of-view (POV) and the models&#39; movements and rotations. Taking full advantage of this recording and the pause feature, a straight edge is placed from the physician&#39;s POV through the protractor model&#39;s crosshair, and measurements can be observed in a manner remarkably similar to the use of an actual protractor.
This methodology does have some limitations. One such limitation is that there is not necessarily a single correct orientation for the aneurysm when viewing it in fluoroscopy. This led to multiple validation attempts simply due to the different viewing angles. This limitation can be viewed as a possible benefit from the perspective that with additional familiarity that comes from manipulating the 3D model, it is possible that the physician will find an optimal view as compared to the current method of determining angles within the operating suite. Another potential limitation of this protocol is that it is possible to determine a viewing angle in VR that would not actually be possible for the C-arms to get to. This limitation would be taken into account and known by the physician in VR so specifications could be made if this became part of surgical planning. Another limitation, proving the importance of the quality control step, is that in some instances, vessels that are distal of the aneurysm, in reality, are not seen as prominently in fluoroscopy procedures as they would be if included in the model in VR. This can force the physician to be mindful of a vessel that would not necessarily be in the way during the procedure in VR, leading to a suboptimal viewing angle being generated in VR. In segmentation, it is possible to segment out the majority of the blood vessels and the area of interest; the interventionalist could choose to toggle between models of vessels to ensure there would be no additional vessels in their viewing angle, the use of contract minimizes this risk as well.
The development of a 3D model protractor and a protocol that can provide angle measurements in multiple axes within VR holds immense importance and promises a wide array of potential applications. The benefits could prove to be multifaceted, potentially enhancing various industries from architecture and engineering to manufacturing and military applications. However, as shown in this protocol, its true potential shines in the realm of healthcare, directly within the surgical planning portions of patient care. Surgeons can utilize this tool to meticulously assess and plan all types of procedures by being able to visualize and measure angles directly in VR. This technique is similar to work done for cardiac catheterization19. One direct benefit of knowing particular angles pre-procedure is the significant reduction in the need for a full 360-degree spin during fluoroscopy, a commonly employed imaging technique during aneurysm repair. By determining the angles required to mimic the virtual surgical roadmap, the surgeons can position the equipment more accurately, thus minimizing the radiation exposure to the patient. This not only contributes to patient safety by minimizing risks associated with radiation exposure but also streamlines the surgical procedure. With reduced time spent on fluoroscopy adjustments, surgical teams can operate more efficiently, ultimately leading to shorter procedure times.
Recent advancements in 3D modeling and virtual reality technology allow medical staff to avoid improvisational thinking during surgeries by obtaining a deep understanding of a patient&#39;s internal anatomy prior to operation in all but the most urgent cases1,2,3,4,6,9,11,13,16. If time allows, medical staff should leverage the use of medical image segmentation and VR diagnostics to further their understanding of the case prior to placing the patient on the operating table. This will ultimately lead to a better understanding of each unique patient, as well as reduced surgery time and time under anesthesia.

The treatment of intracranial aneurysms is a challenging aspect of neuro-interventional surgery, necessitating precise surgical planning to ensure optimal patient outcomes. In recent years, virtual reality (VR) technology has become a promising tool for enhancing surgical planning by providing surgeons access to immersive, patient-specific anatomical models in a virtual 3D environment1,2,3,4,5,6,7,8. This article presents a comprehensive protocol for the use of medical imaging and segmentation, 3D modeling, VR surgical planning, and idealized virtual roadmap generation to aid in surgical planning for the treatment of aneurysms.
The combination of these steps culminates in a virtual surgical planning approach, allowing physicians to immerse themselves in a virtual environment and gain a comprehensive understanding of a patient&#39;s unique anatomy prior to a surgical procedure. This immersive approach empowers surgeons to explore optimal positioning and simulate various procedural scenarios. Recording these scenarios can provide insight into the placement of real-world surgical equipment, such as C-arm positioning.
In addition to positioning angles, it is also possible to measure anatomy in a virtual environment using measurement tools designed for 3D space. These measurements can provide insight into the correct sizing and shape of the device to be used in an intracranial aneurysm case9.
This protocol presents a comprehensive process&#160;that seamlessly combines medical imaging, image segmentation, VR model preparation, and virtual surgical roadmap generation to enhance the surgical planning process. Using a combination of leading-edge technologies, this protocol provides opportunities to save valuable time in the operating room10, as well as a boost to surgeon confidence and understanding of complex surgical cases11,12,13.

<table><tbody><tr><td>3D Slicer&amp;nbsp;</td><td></td><td>N/A&amp;nbsp;</td><td>Open source segmentation software&amp;nbsp;</td></tr><tr><td>Blender&amp;nbsp;</td><td></td><td>N/A&amp;nbsp;</td><td>Open-source CAD software that can import and edit organic models created through segmentation&amp;nbsp;</td></tr><tr><td>Enduvo&amp;nbsp;</td><td>Enduvo&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>A proprietary VR viewer built for education, and our VR viewer of choice &amp;nbsp;</td></tr><tr><td>McKesson PACS Change Healthcare Radiology Solution&amp;nbsp;</td><td>McKesson&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>Any Picture Archiving and Communication System should be suffiecient, McKessen is simply our PACS software solution of choice. &amp;nbsp;</td></tr><tr><td>Mimics&amp;nbsp;</td><td>Materialise&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>Segmentation software&amp;nbsp;</td></tr><tr><td>Quest&amp;nbsp;</td><td>Oculus&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>Virtual Reality Headset&amp;nbsp;</td></tr><tr><td>Steam VR&amp;nbsp;</td><td>Steam&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>Computer to headset connection software. &amp;nbsp;</td></tr><tr><td>VR capable computer&amp;nbsp;</td><td></td><td></td><td>See Steam VR for minimal requirements.</td></tr><tr><td>VR-STL-Viewer&amp;nbsp;</td><td>GitHub&amp;nbsp;</td><td>N/A&amp;nbsp;</td><td>A open-source VR viewer capable of importing and viewing .stl and&amp;nbsp;can be used, however we cannot guarantee all functionalities mentioned in this paper will be available&amp;nbsp;</td></tr></tbody></table>

pioneering patient-specific approaches for precision surgery using imaging and virtual reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Following the presented protocol, virtual surgical roadmaps can be generated for both the AP and lateral fluoroscopy views. These roadmaps are created by placing a camera at the viewpoint of the surgeon in VR to capture their ideal AP and lateral views while also placing a colored background behind the target anatomy to better replicate a fluoroscopy image. The VR protractor is used at this point to record the angle from which the surgeon is viewing the target anatomy, recorded as right, or left anterior oblique (RAO/LAO - camera offset to the patient&#39;s right or left, respectively), and cranial or caudal anterior (CRA/CAA - camera offset to toward the patient&#39;s head or feet, respectively)15. When developing this process, retrospective cases were used to provide the ability to compare angles measured in VR with the actual angles used on the C-arm machines in surgery. Three different retrospective cases were selected for this process, each case having been treated with a different surgical device. The diversity of these three cases shows the versatility of the presented protocol. The surgeon was asked to find preferred AP and lateral angles without referencing the C-arm angles used during the procedure, and the VR measurements were then compared to these pre-existing C-arm positions.
In case 1, the declared preferred AP viewing angle was measured in VR as 16&#176; CRA, 12&#176; RAO. The actual measurements used in surgery for this case were 11&#176; CRA and 13&#176; RAO. The maximum error among these measurements is 5&#176; on the cranial/caudal axis. Figure 2A shows the surgeon&#39;s declared AP view in virtual reality, followed by Figure 2B, which shows the actual angle used in surgery as seen in VR, and Figure 2C, which shows the surgical fluoroscopy image. Comparing the three images shows the VR images to be extraordinarily similar to the actual fluoroscopy image at the same angle.
The lateral view of the same case displayed one of the many challenges of this process due to the 3D model being inadequately reviewed. Because of this faulty review, there were some extraneous vessels segmented that, according to the surgeon, inhibited their view of the aneurysm in VR and are not connected to the target anatomy and, as such, are not accurately reflected in VR. These discrepancies were a result of miscommunication in the required target anatomy during the quality control session with the physician. These discrepancies can be seen in Figure 2D-F, which shows the surgeon&#39;s declared lateral, the VR representation based on surgical fluoroscopy angles, and the actual fluoroscopy images from left to right, respectively. With the exception of extraneous vessels, the surgeon&#39;s declared AP view closely resembles the actual fluoroscopy image, despite the measurements taken being 6&#176; and 26&#176; off in the coronal and axial planes, respectively. The replication of actual measurements in VR, as shown in Figure 2E, also depicts a similar view to the real fluoroscopy shown on the right of Figure 2F, with the main discrepancy being the anomalous extra vessels. This case employed a less reliable manual placement of the protractor tool, which may account for the slight difference in measurement. Future cases employ a protractor that is bound to the anatomy in order to ensure maximum accuracy of angle measurements taken in VR.
In cases 2 and 3, the views selected to be optimal in VR were not representative of the views used in the actual procedure. This was a consequence of the initial placement of models in VR being a blinded study. It is important to note the surgeon expressed that fluoroscopy procedures can have multiple acceptable treatment angles, and there is not necessarily a correct angle. For the purpose of comparison, images were taken in VR from the reported surgical angles. Figure 3 shows the VR AP view in Figure 3A and the surgical AP view in Figure 3B. In Figure 3, a similar comparison can be made between lateral views in Figure 3C,D for case 2. For case 3, Figure 4 shows the AP comparison Figure 4A,B, as well as the lateral comparison Figure 4C,D. The similarities between the VR and fluoroscopy images of these cases further demonstrate the ability of VR to be used in surgical planning.
An important benefit of this protocol is the improvement of surgical planning by leveraging 3D models in a VR environment. A previous study on the effectiveness of VR in surgical planning for complex oncological cases showed that roughly 50% of cases that employed the use of VR altered the surgical approach from the plan made using only 2D datasets9. VR has also been proven useful in the surgical planning process for liver tumor resection16,17, as well as procedures involving head and neck pathology18. The surgeon participating in the creation of this protocol stated that : in VR I can see [the anatomy] so much better, showing the benefit of VR for endovascular neurosurgical applications.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/66227/66227fig01.jpg" /> 
Figure 1: Screenshot from within segmentation software. The screenshot shows the highlighted anatomy based on Masks. <a href="https://www.jove.com/files/ftp_upload/66227/66227fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/66227/66227fig02.jpg" /> 
Figure 2: Case 1. (A) Anteroposterior view of Case 1 as placed by the operating surgeon in VR. (B) Anteroposterior view of Case 1 in VR based on angle measurements taken during surgery. (C) Anteroposterior fluoroscopy view captured during surgery. (D) Lateral view of Case 1 as placed by the operating surgeon in VR. (E) Lateral view of Case 1 in VR based on angle measurements taken during surgery. (F) Lateral fluoroscopy view captured during surgery. <a href="https://www.jove.com/files/ftp_upload/66227/66227fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/66227/66227fig03.jpg" /> 
Figure 3: Case 2. (A) Anteroposterior view of Case 2 in VR based on angle measurements taken during surgery. (B) Anteroposterior fluoroscopy view of Case 2 captured during surgery. (C) Lateral view of Case 2 in VR based on angle measurements taken during surgery. (D) Lateral fluoroscopy view of Case 2 captured during surgery. <a href="https://www.jove.com/files/ftp_upload/66227/66227fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/66227/66227fig04.jpg" /> 
Figure 4: Case 3. (A) Anteroposterior view of Case 3 in VR based on angle measurements taken during surgery. (B) Anteroposterior fluoroscopy view of Case 3 captured during surgery. (C) Lateral view of Case 3 in VR based on angle measurements taken during surgery. (D) Lateral fluoroscopy view of Case 3 captured during surgery. <a href="https://www.jove.com/files/ftp_upload/66227/66227fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Supplementary File 1: A 3D model of protractors was developed and used for the protocol in STL file format. <a href="https://www.jove.com/files/ftp_upload/66227/protractors.zip" target="_blank">Please click here to download this File.</a>

Watch this Scientific Journal Video about Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions at JoVE.com

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular ...

pioneering-patient-specific-approaches-for-precision-surgery-using

Research

JoVE Journal

Medicine

926 Views.  OSF HealthCare. Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time. 

Video: Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

A Teleoperated Robotic System-Assisted Percutaneous Transiliac-Transsacral Screw Fixation Technique

Transiliac-transsacral screw fixation is challenging in clinical practice as the screws need to break through six layers of cortical bone. Transiliac-transsacral screws provide a longer lever arm to withstand the perpendicular vertical shear forces. However, the screw channel is so long that a minor discrepancy can lead to iatrogenic neurovascular injuries. The development of medical robots has improved the precision of surgery. The present protocol describes how to use a new teleoperated robotic system to execute transiliac-transacral screw fixation. The Robot was operated remotely to position the entry point and adjust the orientation of the sleeve. The screw positions were evaluated using postoperative computed tomography (CT). All the screws were safely implanted, as confirmed using intraoperative fluoroscopy. Postoperative CT confirmed that all the screws were in the cancellous bone. This system combines the doctor's initiative with the Robot's stability. The remote control of this procedure is possible. Robot-assisted surgery has a higher position-retention capacity compared with conventional methods. In contrast to active robotic systems, surgeons have full control over the operation. The robot system is fully compatible with operating room systems and does not require additional equipment.

Teleoperated robotic system-assisted percutaneous transiliac-transsacral screw fixation is a feasible technique. Screw channels can be implemented with high accuracy owing to the excellent freedom of movement and stability of the robotic arms.

Transiliac-transsacral screw fixation is challenging in clinical practice as the screws need to break through six layers of cortical bone. ...

Engineering

A Pipeline for 3D Multimodality Image Integration and Computer-assisted Planning in Epilepsy Surgery

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality image integration can be technically demanding, and is underutilised in clinical practice. We have developed a single software platform for image integration, 3D visualization and surgical planning. Here, our pipeline is described in step-by-step fashion, starting with image acquisition, proceeding through image co-registration, manual segmentation, brain and vessel extraction, 3D visualization and manual planning of stereoEEG (SEEG) implantations. With dissemination of the software this pipeline can be reproduced in other centres, allowing other groups to benefit from 3DMMI. We also describe the use of an automated, multi-trajectory planner to generate stereoEEG implantation plans. Preliminary studies suggest this is a rapid, safe and efficacious adjunct for planning SEEG implantations. Finally, a simple solution for the export of plans and models to commercial neuronavigation systems for implementation of plans in the operating theater is described. This software is a valuable tool that can support clinical decision making throughout the epilepsy surgery pathway.

We describe the steps to use our custom designed software for image integration, visualization and planning in epilepsy surgery.

Epilepsy surgery is challenging and the use of 3D multimodality image integration (3DMMI) to aid presurgical planning is well-established. Multimodality ...

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

3D Planning and Printing of Patient Specific Implants for Reconstruction of Bony Defects

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical field using the constantly developing 3D planning and printing capabilities. Computer-assisted design (CAD) and computer assisted manufacturing (CAM) are used to describe the 3D planning and manufacturing of the product. The planning and manufacturing of 3D surgical guides and reconstruction implants is performed almost exclusively by engineers. As technology advances and software interfaces become more user-friendly, it raises a question regarding the possibility of transferring the planning and manufacturing to the clinician. The reasons for such a shift are clear: the surgeon has the idea of what he wants to design, and he also knows what is feasible and could be used in the operating room. It allows him to be prepared for any scenario/unexpected results during the operation and allows the surgeon to be creative and express his new ideas using the CAD software. The purpose of this method is to provide clinicians with the ability to create their own surgical guides and reconstruction implants. In this manuscript, a detailed protocol will provide a simple method for segmentation using segmentation software and implant planning using a 3D design software. Following the segmentation and stl file production using segmentation software, the clinician could create a simple patient specific reconstruction plate or a more complex plate with a cradle for bone graft positioning. Surgical guides can be created for accurate resection, hole preparation for proper reconstruction plate positioning or for bone graft harvesting and re-contouring. A case of lower jaw reconstruction following plate fracture and nonunion healing of a trauma sustained injury is detailed.

This protocol describes the use of 3D planning and printing for reconstruction of bony defects. We use segmentation tools to create 3D models followed by 3D design software to create patient specific implants for reconstruction purposes concomitant to ablative surgery or as a second stage.

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical ...

Treatment of Facial Deformities using 3D Planning and Printing of Patient-Specific Implants

Technological advancements in surgical planning and patient-specific implants are constantly evolving. One can either adopt the technology to achieve better results, even in the less experienced hand, or continue without it. As technology develops and becomes more user-friendly, we believe it is time to allow the surgeon the option to plan his/her operations and create his/her own patient-specific surgical guides and fixation plates allowing him full control over the process. We present here a protocol for 3D planning of the operation followed by 3D planning and printing of surgical guides and patient-specific fixation implants. During this process we use two commercial computer-assisted design (CAD) software. We also use a fused deposition&#160;modeling printer for the surgical guides and a selective laser sintering printer for the titanium patient-specific fixation implants. The process includes computed tomography (CT) imaging acquisition, 3D segmentation of the skull and facial bones from the CT, 3D planning of the operations, 3D planning of patient-specific fixation implant according to the final position of the bones, 3D planning of surgical guides for performing an accurate osteotomy and preparing the bone for the fixation plates, and 3D printing of the surgical guides and the patient-specific fixation plates. The advantages of the method include full control over the surgery, planned osteotomies and fixation plates, significant reduction in price, reduction in operation duration, superior performance and highly accurate results. Limitations include the need to master the CAD programs.

As technology develops and becomes more user-friendly, planning of operations and patient-specific surgical guides and fixation plates should be performed by the surgeon. We present a protocol for 3D planning of orthognathic skeletal movements and 3D planning and printing of patient-specific fixation plates and surgical guides.

Technological advancements in surgical planning and patient-specific implants are constantly evolving. One can either adopt the technology to achieve ...

Digital Hybrid Model Preparation for Virtual Planning of Reconstructive Dentoalveolar Surgical Procedures

Virtual, hybrid three-dimensional (3D) model acquisition is presented in this article, utilizing the sequence of radiographic image segmentation, spatial registration, and free-form surface modeling. Firstly cone-beam computed tomography datasets were reconstructed with a semi-automatic segmentation method. Alveolar bone and teeth are separated into different segments, allowing 3D morphology, and localization of periodontal intrabony defects to be assessed. The severity, extent, and morphology of acute and chronic alveolar ridge defects are validated concerning adjacent teeth. On virtual complex tissue models, positions of dental implants can be planned in 3D. Utilizing spatial registration of IOS and CBCT data and subsequent free-form surface modeling, realistic 3D hybrid models can be acquired, visualizing alveolar bone, teeth, and soft tissues. With the superimposition of IOS and CBCT soft tissue, thickness above the edentulous ridge can be assessed about the underlying bone dimensions; therefore, flap design and surgical flap management can be determined, and occasional complications may be avoided.

A workflow for creating three-dimensional (3D) virtual hybrid models has been designed based on cone-beam computed tomography dataset and intraoral optical scans utilizing radiographic image segmentation methods and free-form surface modeling. Digital models are used for the virtual planning of reconstructive dentoalveolar surgical procedures.

Virtual, hybrid three-dimensional (3D) model acquisition is presented in this article, utilizing the sequence of radiographic image segmentation, spatial ...

Advanced CAD Imaging for Surgical Margin Analysis in Cancer Care

Enhanced Communication of Tumor Margins Using 3D Scanning and Mapping

After oncologic resection of malignant tumors, specimens are sent to pathology for processing to determine the surgical margin status. These results are communicated in the form of a written pathology report. The current standard-of-care pathology report provides a written description of the specimen and the sites of margin sampling without any visual representation of the resected tissue. The specimen itself is typically destroyed during sectioning and analysis. This often leads to challenging communication between pathologists and surgeons when the final pathology report is confirmed. Furthermore, surgeons and pathologists are the only members of the multidisciplinary cancer care team to visualize the resected cancer specimen. We have developed a 3D scanning and specimen mapping protocol to address this unmet need. Computer-aided design (CAD) software is used to annotate the virtual specimen clearly showing sites of inking and margin sampling. This map can be utilized by various members of the multidisciplinary cancer care team.

Author Spotlight: 3D Scanning and Augmented Reality for Enhanced Cancer Surgery Communication

A novel method for 3D scanning and virtual mapping of cancer resections is proposed with the goal of improving communication among the multidisciplinary cancer care team.

After oncologic resection of malignant tumors, specimens are sent to pathology for processing to determine the surgical margin status. These results are ...

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) for minimally invasive pedicle screw placement. The cadaveric porcine specimens were placed on a surgical table and draped with sterile covers. The levels of interest were identified using fluoroscopy, and a dynamic reference frame was attached to the spinous process of a vertebra in the region of interest. Cone beam computerized tomography (CBCT) was performed, and a 3D rendering was automatically generated, which was used for the subsequent planning of the pedicle screw placements. Each surgeon was fitted with an HMD that was individually eye-calibrated and connected to the spinal navigation system.
Navigated instruments, tracked by the navigation system and displayed in 2D and 3D in the HMD, were used for 33 pedicle cannulations, each with a diameter of 4.5 mm. Postprocedural CBCT scans were assessed by an independent reviewer to measure the technical (deviation from the planned path) and clinical (Gertzbein grade) accuracy of each cannulation. The navigation time for each cannulation was measured. The technical accuracy was 1.0 mm &#177; 0.5 mm at the entry point and 0.8 mm &#177; 0.1 mm at the target. The angular deviation was 1.5&#176; &#177; 0.6&#176;, and the mean insertion time per cannulation was 141 s &#177; 71 s. The clinical accuracy was 100% according to the Gertzbein grading scale (32 grade 0; 1 grade 1). When used for minimally invasive pedicle cannulations in a porcine model, submillimeter technical accuracy and 100% clinical accuracy could be achieved with this protocol.

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) ...

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

Mixed Reality Assisted Radical Endoscopic Thyroidectomy

Radical endoscopic thyroidectomy (ET) offers superior cosmetic outcomes and enhanced visibility of the surgical field compared to open surgery. However, the thyroid&#39;s unique physiological functions and intricate surrounding anatomy may result in various surgical complications. Mixed reality (MR), a real-time holographic visualization technology, enables the creation of highly realistic 3D models in the real world and facilitates multiple human-computer interactions. MR can be utilized for both preoperative evaluation and intraoperative navigation. First, semi-automatic 3D reconstruction of the neck from enhanced computed tomography images is performed using 3Dslicer. Next, the 3D model is imported into Unity3D to create a virtual hologram that can be displayed on an MR helmet-mounted display (HMD). During surgery, surgeons can wear the MR HMD to locate lesions and surrounding anatomy through the virtual hologram. In this study, patients requiring radical ET were randomly assigned to either the experimental group or the control group. Surgeons performed MR-assisted radical ET in the experimental group. A comparative analysis of surgical outcomes and the results of scales was conducted.&#160;This study successfully developed the neck 3D model and the virtual hologram. According to the NASA Task Load Index Scale, the experimental group exhibited significantly higher scores in &#39;Own Performance&#39; and lower scores in &#39;Effort&#39; compared to the control group (p = 0.002). Additionally, on the Likert Subjective Evaluation Scale, the mean scores for all questions exceeded 3. Although the incidence of surgical complications was lower in the experimental group than in the control group, the differences in surgical outcomes were not statistically significant.MR is beneficial for enhancing performance and alleviating the burden of surgeons during the perioperative period. Furthermore, MR has demonstrated the potential to enhance the safety of ET. Therefore, it is essential to further investigate the surgical applications of MR.

Radical endoscopic thyroidectomy is associated with various surgical complications. This study utilizes mixed reality techniques to assist surgeons in performing radical endoscopic thyroidectomy, aiming to enhance its safety and lower the surgical threshold.

Radical endoscopic thyroidectomy (ET) offers superior cosmetic outcomes and enhanced visibility of the surgical field compared to open surgery. However, ...