The authors acknowledge no funding for this research.

1. Surgical environment
<ol>
	<li>Design a surgery environment as shown in Figure 1. Make sure the environment has an object-carrying platter, a robotic arm, and two side-hanging arms, one to hold a camera placeholder and the other to have a consistent white background along with the weighing module for balance.</li>
	<li>Develop two drivers, one for the snapshot of the live surgical environment, as mentioned in steps 2.1 to 2.10, and the other to control the revolving mechanism that supports 360&#176; surveillance, as mentioned in steps 3.1 to 3.4.</li>
	<li>Before implementing the above two modules, enable Bluetooth of the mobile device and the laptop, which serve as the surgeon&#39;s HoloLens emulator.</li>
	<li>Pair the devices for uninterrupted image transmission.</li>
</ol>
2. Setting up the driver to control the two hanging arms
<ol>
	<li>Make sure that the hanging arms are controlled by a stepper motor with the arrangement as shown in Figure 2 for a flawless revolution of 360&#176;.</li>
	<li>Connect the motor to the microcontroller board using the TB 6600 driver. To run the motor, install the microcontroller IDE from the browser.</li>
	<li>Click the Download button to download the software. Then, in the microcontroller IDE, go to File &#62; Open a New Sketch to write the code.</li>
	<li>Make sure to connect the microcontroller board to interface with the new sketch through a dedicated connection port, say COM 4. Check the Com Port and verify that it shows the microcontroller board.</li>
	<li>Check the hardware switch settings of the stepper motor driver TB 6600. Ensure that the settings are such that the current flow is 2 A, which can be attained by setting SW4 ON and SW5 and SW6 OFF.</li>
	<li>Ensure the switch positions of SW1, SW2, and SW3 are set so that the micro-step is 1/8 steps to attain the revolution steps as per the requirement. Ensure the settings are SW1 OFF, SW2 ON, and SW3 OFF in TB6600.</li>
	<li>Connect RTC 3231 with the microcontroller to have real global time synchronization. Make sure that the revolution step size is 12&#176; and that the motor step increment is triggered only when the real-time unit, i.e., the seconds read from the RTC module, is odd in number.</li>
	<li>Connect the 5 V pin of the microcontroller board to the RTC VCC and the microcontroller&#39;s GND to the RTC&#39;s GND.</li>
	<li>Connect the SCL pin of RTC to the A0 pin and the SDA to the A1 pin of the microcontroller. This module can ensure a step size of 12&#176;, making 30 steps in one revolution. Ensure that the step increment happens every odd second. Let this software module drive the stepper motor21.</li>
	<li>Verify the setup is working correctly by running the code, which is available on the GitHub page:&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
	<li>Download Android Studio to develop the automatic camera app. Ensure the system requirements are met, then download the software from the website.</li>
</ol>
3. Developing a driver for mobile-based scene surveillance and image transmission as a client module
<ol>
	<li>Develop a camera application in the Android operating system that can take snapshots every 2 s, especially when the seconds are odd numbers.</li>
	<li>Connect the mobile phone with the system. In Android Studio, click New &#62; New Project and choose Empty Views Activity. Click on Next to develop an Android code, which is available at&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
	<li>Ensure that the app captures the images automatically and sends them to a remote device consistently.</li>
	<li>Transmit snapshots from the mobile app immediately after taking the snapshot to the paired device, i.e., to the remote surgeon&#39;s system, through Bluetooth. 
	NOTE: Make sure the modules mentioned in sections 2 and 3 run in time synchronization, one for every even number of seconds and the other for every odd number of seconds.</li>
</ol>
4. Developing a client module to receive and handle surveillance images
<ol>
	<li>Open the server module, which is a graphical user interface.</li>
	<li>Enter the VVID port number in the text field VVID, whose default value is 94f39d29 7d6d 437d 973b fba39e49d4ee.</li>
	<li>Click on Create Socket to create and bind the socket. Click on Connect to establish a connection with the mobile app.</li>
	<li>Click on Capture to capture and save the scene surveillance images in the local folder</li>
	<li>Enter the local folder name in the field folder name if it needs to be other than the default one mentioned.</li>
</ol>
5. Operating the robotic arm
<ol>
	<li>Let the client module include a robotic arm as well. Design the arm to have a rotational movement in its base, shoulder, elbow, wrist, and fingers.</li>
	<li>Make sure that MG 996R servos are used for governing the rotational movement at the base, shoulder and elbow. Ensure that the SG 90 servo motor is used to control the rotational movement at the elbow and fingers.</li>
	<li>Compile the code given in&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR in the microcontroller IDE to drive the robotic arm based on the commands received from the remote surgeon.</li>
</ol>
6. 3D reconstruction for augmented reality
<ol>
	<li>Read two images at a time in a sequence, one by one, from the local folder to obtain the possible overlapping (as the images are collected in close proximity, there will be an overlapping between the consecutive images) between them.</li>
	<li>Design a tertiary filter as per the requirement of the Directional Intensified Feature Description using the Tertiary Filtering22 (DITF)&#160;algorithm to obtain the gradient and orientation.</li>
	<li>Extract the features using the DITF method22, as shown in Figure 3.</li>
	<li>Reconstruct 3D images from the collected features using SFM23.</li>
</ol>
7. Hand gesture recognition at the surgeon&#39;s location
<ol>
	<li>Facilitate the surgeon to inspect the 3D reconstructed image features by enabling him/her to visualize the environment from all perspectives by providing hand gesture-based rotation and zoom in/out of reconstructed features.</li>
	<li>Normalize and map the distance between the tip of the surgeon&#39;s thumb and the index finger of the right hand into a corresponding angle of rotation. Let the normalization be in such a way that the bare minimum distance corresponds to 0&#176; and the maximum to 180&#176;.</li>
	<li>Transmit the hand gesture control, through Bluetooth, to the remote surgery environment as well for the rotation of the object platter, which makes it revolve on its axis as the 3D reconstructed features revolve at the surgeon&#39;s end.</li>
	<li>Find the distance between the tip and thumb of the surgeon&#39;s left hand to control the movement of the fingers of the robot arm.</li>
	<li>Measure the angle of elevation from the spatial distance between the tip of the thumb and index fingers of the surgeon&#39;s left hand with respect to an imaginary x-y ground plane to determine the elevation angle. Map this angle into an elevation angle that the robot arm can make with the x-y plane.</li>
	<li>Find the azimuth angle that the hand of the surgeon makes with that of the virtual y-z plane. Identify these angles through hand gesture-based recognition.</li>
	<li>Map the distance, elevation, and azimuth angles to control the robot&#39;s finger movement and arm rotation, which both correspond to elevation and azimuth angles.</li>
	<li>Let the surgeon inspect the reconstructed features by zooming and rotating. Let the surgeon transmit commands to the robot arm to perform surgery from a remote location.</li>
	<li>Make sure that the surgery commands are transmitted as a control string of sequence starting with a string concurrence followed by the values to control the platter rotation and robotic arm controller. Let [&#952;b, &#952;s, &#952;e, &#952;w, &#952;f] be the angle of the vector that consists of values, each corresponding to the control signal corresponding to the base, shoulder, elbow, wrist, and finger of the robot arm. 
	NOTE: The GitHub link provides the code to enable hand gesture control in the surgical field.&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
</ol>

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

<ol><li>Wang, L. J., Casto, B., Reyes-Molyneux, N., Chance, W. W., Wang, S. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Smartphone-based+augmented+reality+patient+education+in+radiation+oncology%5BTitle%5D)+AND+%22Tech+Innov+Patient+Supp+Radiation+Oncol%22%5BJournal%5D)">Smartphone-based augmented reality patient education in radiation oncology.</a> Tech Innov Patient Supp Radiation Oncol. 29, 100229(2024).</li>
<li>Guerroudji, M. A., Amara, K., Lichouri, M., Zenati, N., Masmoudi, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+3D+visualization-based+augmented+reality+application+for+brain+tumor+segmentation%5BTitle%5D)+AND+%22Comput+Anim+Virtual+Worlds%22%5BJournal%5D)">A 3D visualization-based augmented reality application for brain tumor segmentation.</a> Comput Anim Virtual Worlds. 35 (1), e2223(2024).</li>
<li>Xia, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+and+indoor+positioning+based+mobile+production+monitoring+system+to+support+workers+with+human-in-the-loop%5BTitle%5D)+AND+%22Robotic+Comput+Integ+Manufac%22%5BJournal%5D)">Augmented reality and indoor positioning based mobile production monitoring system to support workers with human-in-the-loop.</a> Robotic Comput Integ Manufac. 86, 102664(2024).</li>
<li>Preece, C., Skandalis, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Time+to+imagine+an+escape%3A+investigating+the+consumer+timework+at+play+in+augmented+reality%5BTitle%5D)+AND+%22Eur+J+Market%22%5BJournal%5D)">Time to imagine an escape: investigating the consumer timework at play in augmented reality.</a> Eur J Market. 58, 92-118 (2024).</li>
<li>Suresh, D., Aydin, A., James, S., Ahmed, K., Dasgupta, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((The+role+of+augmented+reality+in+surgical+training%3A+a+systematic+review%5BTitle%5D)+AND+%22Surg+Innov%22%5BJournal%5D)">The role of augmented reality in surgical training: a systematic review.</a> Surg Innov. 30, 366-382 (2023).</li>
<li>Moreta-Martinez, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Combining+augmented+reality+and+3D+printing+to+display+patient+models+on+a+smartphone%5BTitle%5D)+AND+%22J+Vis+Exp%22%5BJournal%5D)">Combining augmented reality and 3D printing to display patient models on a smartphone.</a> J Vis Exp. (155), 60618(2020).</li>
<li>Ma, L., Huang, T., Wang, J., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Visualization%2C+registration+and+tracking+techniques+for+augmented+reality+guided+surgery%3A+a+review%5BTitle%5D)+AND+%22Phys+Med+Biol%22%5BJournal%5D)">Visualization, registration and tracking techniques for augmented reality guided surgery: a review.</a> Phys Med Biol. 68, 04TR02(2023).</li>
<li>Hofman, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((First-in-human+real-time+ai-assisted+instrument+deocclusion+during+augmented+reality+robotic+surgery%5BTitle%5D)+AND+%22Healthc+Technol+Lett%22%5BJournal%5D)">First-in-human real-time ai-assisted instrument deocclusion during augmented reality robotic surgery.</a> Healthc Technol Lett. 11 (2-3), 33-39 (2023).</li>
<li>Thiene, G., Rizzo, S., Basso, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Bicuspid+aortic+valve%3A+The+most+frequent+and+not+so+benign+congenital+heart+disease%5BTitle%5D)+AND+%22Cardiovasc+Pathol%22%5BJournal%5D)">Bicuspid aortic valve: The most frequent and not so benign congenital heart disease.</a> Cardiovasc Pathol. 70, 107604(2024).</li>
<li>Mack, M. J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Transcatheter+aortic-valve+replacement+in+low-risk+patients+at+five+years%5BTitle%5D)+AND+%22New+Engl+J+Med%22%5BJournal%5D)">Transcatheter aortic-valve replacement in low-risk patients at five years.</a> New Engl J Med. 389, 1949-1960 (2023).</li>
<li>Vitanova, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Aortic+valve+versus+root+surgery+after+failed+transcatheter+aortic+valve+replacement%5BTitle%5D)+AND+%22J+Thorac+Cardiovas+Surg%22%5BJournal%5D)">Aortic valve versus root surgery after failed transcatheter aortic valve replacement.</a> J Thorac Cardiovas Surg. 166, 1418-1430 (2023).</li>
<li>Bydlon, T. M., Torjesen, A., Fokkenrood, S., Di Tullio, A., Flexman, M. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3d+visualisation+of+navigation+catheters+for+endovascular+procedures+using+a+3d+hub+and+fiber+optic+realshape+technology%3A+phantom+study+results%5BTitle%5D)+AND+%22EJVES+Vascular+Forum%22%5BJournal%5D)">3d visualisation of navigation catheters for endovascular procedures using a 3d hub and fiber optic realshape technology: phantom study results.</a> EJVES Vascular Forum. 59, 24-30 (2023).</li>
<li>Faris, H., Harfouche, C., Bandle, J., Wisbach, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Surgical+tele-mentoring+using+a+robotic+platform%3A+initial+experience+in+a+military+institution%5BTitle%5D)+AND+%22Surg+Endosc%22%5BJournal%5D)">Surgical tele-mentoring using a robotic platform: initial experience in a military institution.</a> Surg Endosc. 37, 9159-9166 (2023).</li>
<li>Fitzgerald, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Mentoring+approaches+in+a+safe+surgery+program+in+tanzania%3A+Lessons+learned+during+covid-19+and+recommendations+for+the+future%5BTitle%5D)+AND+%22Surg+Open+Sci%22%5BJournal%5D)">Mentoring approaches in a safe surgery program in tanzania: Lessons learned during covid-19 and recommendations for the future.</a> Surg Open Sci. 14, 109-113 (2023).</li>
<li>de Turenne, A., Eugène, F., Blanc, R., Szewczyk, J., Haigron, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Catheter+navigation+support+for+mechanical+thrombectomy+guidance%3A+3d%2F2d+multimodal+catheter-based+registration+with+no+contrast+dye+fluoroscopy%5BTitle%5D)+AND+%22Int+J+Comput+Assist+Radiol+Surg%22%5BJournal%5D)">Catheter navigation support for mechanical thrombectomy guidance: 3d/2d multimodal catheter-based registration with no contrast dye fluoroscopy.</a> Int J Comput Assist Radiol Surg. 19 (3), 459-468 (2023).</li>
<li>Tang, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+multiple+catheter+tips+tracking+method+in+x-ray+fluoroscopy+images+by+a+new+lightweight+segmentation+network+and+bayesian+filtering%5BTitle%5D)+AND+%22Int+J+Med+Robotics+Comput+Assist+Surg%22%5BJournal%5D)">A multiple catheter tips tracking method in x-ray fluoroscopy images by a new lightweight segmentation network and bayesian filtering.</a> Int J Med Robotics Comput Assist Surg. 19, e2569(2023).</li>
<li>Annabestani, M., Caprio, A., Wong, S. C., Mosadegh, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+machine+learning-based+roll+angle+prediction+for+intracardiac+echocardiography+catheter+during+bi-plane+fluoroscopy%5BTitle%5D)+AND+%22Appl+Sci%22%5BJournal%5D)">A machine learning-based roll angle prediction for intracardiac echocardiography catheter during bi-plane fluoroscopy.</a> Appl Sci. 13, 3483(2023).</li>
<li>Thourani, V. H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Survival+after+surgical+aortic+valve+replacement+in+low-risk+patients%3A+a+contemporary+trial+benchmark%5BTitle%5D)+AND+%22Ann+Thorac+Surg%22%5BJournal%5D)">Survival after surgical aortic valve replacement in low-risk patients: a contemporary trial benchmark.</a> Ann Thorac Surg. 117, 106-112 (2024).</li>
<li>Domínguez-Velasco, C. F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+simulation+as+training+model+of+ventricular+puncture%3A+Evidence+in+the+improvement+of+the+quality+of+punctures%5BTitle%5D)+AND+%22Int+J+Med+Robotics+Comput+Assist+Surg%22%5BJournal%5D)">Augmented reality simulation as training model of ventricular puncture: Evidence in the improvement of the quality of punctures.</a> Int J Med Robotics Comput Assist Surg. 19, e2529(2023).</li>
<li>Wang, Q., Xie, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Arias%3A+An+ar-based+interactive+advertising+system%5BTitle%5D)+AND+%22Plos+One%22%5BJournal%5D)">Arias: An ar-based interactive advertising system.</a> Plos One. 18, e0285838(2023).</li>
<li>Ji, F., Chong, F., Wang, F., Bai, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+platform+for+the+unmanned+mining+process+in+underground+mines%5BTitle%5D)+AND+%22Mining%2CMetal+Explor%22%5BJournal%5D)">Augmented reality platform for the unmanned mining process in underground mines.</a> Mining,Metal Explor. 39 (2), 385-395 (2022).</li>
<li>Indhumathi, S., Clement, J. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Directional+intensified+feature+description+using+tertiary+filtering+for+augmented+reality+tracking%5BTitle%5D)+AND+%22Sci+Rep%22%5BJournal%5D)">Directional intensified feature description using tertiary filtering for augmented reality tracking.</a> Sci Rep. 13, 20311(2023).</li>
<li>Gao, L., Zhao, Y., Han, J., Liu, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Research+on+multi-view+3D+reconstruction+technology+based+on+SFM%5BTitle%5D)+AND+%22Sensors%22%5BJournal%5D)">Research on multi-view 3D reconstruction technology based on SFM.</a> Sensors. 22 (12), 4366(2022).</li>
<li>Suárez, I., Sfeir, G., Buenaposada, J. M., Baumela, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((BEBLID%3A+Boosted+efficient+binary+local+image+descriptor%5BTitle%5D)+AND+%22Pattern+Recog+Lett%22%5BJournal%5D)">BEBLID: Boosted efficient binary local image descriptor.</a> Pattern Recog Lett. 133, 366-372 (2020).</li>
<li>Borman, R. I., Harjoko, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Improved+ORB+algorithm+through+feature+point+optimization+and+Gaussian+pyramid%5BTitle%5D)+AND+%22Int+J+Adv+Comp+Sci+Appl%22%5BJournal%5D)">Improved ORB algorithm through feature point optimization and Gaussian pyramid.</a> Int J Adv Comp Sci Appl. 15 (2), 268-275 (2024).</li>
</ol>

The authors declare no conflicts of interest.

In an existing work15, X-ray and CT scans are examined to locate the catheter in the heart. However, AR TAVR replacement establishes a new possibility in TAVR18 surgical procedure by the implementation of an automated model using 3D reconstruction. As mentioned in the protocol section this work has five stages to design. The first stage of DITF22, mentioned in section 6, which we proposed in our previous work22, is enhanced in...

AR can superimpose the 3D model in a real-world environment. The technological development towards AR has made a paradigm shift in many fields, namely education1, medical2, manufacturing3, and entertainment4. AR technology, along with ultra-reliable low-latency communication, proves its inevitable role in the medical field. From the learning stage of human anatomy to surgical guidance, the stages of learning can be visualized with AR-powered software5,6 and hardware. AR provides a crucial and reliable solution to the medical practitioner in a surgical environment7,8.
Aortic valve stenosis is the heart valve disease, which is most common among mankind9. The root cause of the disease is poor food habits and irregular routines of day-to-day life. The symptom and result of the disease is the narrowing of the heart valve, followed by a reduction in the blood flow. This problem needs to be addressed before any damage takes place to the human heart. Thus, the heart is overburdened to process the blood flow. So, before any damage happens, surgery needs to be done, which, owing to technological developments in recent days, can also be done using the TAVR procedure. The procedure can be adopted based on the condition of the heart and other body parts of patients. In this TAVR10,11, the catheter is inserted to replace the damaged valve in the heart. However, placing the catheter position12 to replace the valve is tedious for the practitioner. This idea motivated us to design an automated surgery model based on AR13,14, which helps the surgeon to precisely position the valve during the replacement process. Moreover, the surgery can be performed by a motion mapping algorithm, which maps the surgeon&#39;s movement captured from a remote location to the robotic arm.
In the existing work15,16,17, the visualization of the TAVR18 procedure is monitored through fluoroscopy. Hence, it is difficult to analyze the heart valve and tedious to find the replacement location. This sets up a barrier to positioning the catheter in the human heart. In addition, the remote motion is mapped to the surgical field to make the process automated. However, to overcome the research gap, we propose an automated robotic-based surgery for valve replacement using AR-assisted technology.
The protocol is a generic model that can be applied to all surgical environments. In the beginning stage of the work, 2D images are captured all around the surgical environment with the fullest spatial resolution of the largest degree of freedom. This means that enough images are captured for 3D reconstruction19 purpose, followed by motion mapping through hand gesture tracking20.

<table><tbody><tr><td>android IDE</td><td>software</td><td>https://developer.android.com/studio</td><td>software can be downloaded from this link</td></tr><tr><td>Arduino Board</td><td>Ardunio Uno</td><td>Ardunio Uno</td><td>Microcontroller for processing</td></tr><tr><td>arduino software</td><td>software</td><td>https://www.arduino.cc/en/software.</td><td>software can be downloaded from this link</td></tr><tr><td>Human Heart phantom model</td><td>Biology Lab Equipment Manufacturer and Exporter</td><td>B071YBLX2V(8B-ZB2Q-H3MS-1)</td><td>light weight model with 3parts to the deep analysis of heart.</td></tr><tr><td>mobile holder</td><td>Humble universal monopoad holder</td><td>B07S9KNGVS</td><td>To carry the mobile in surgical field</td></tr><tr><td>pycharm IDE</td><td>software</td><td>https://www.jetbrains.com/pycharm/</td><td>software can be downloaded from this link</td></tr><tr><td>Robot arm</td><td>Printed-bots</td><td>B08R2JLKYM(P0-E2UT-JSOU)</td><td>arm can be controlled through control signal.it has 5 degree of freedom to access.</td></tr><tr><td>servo motor</td><td>Kollmorgen Co-Engineers Motors</td><td>MG-966R</td><td>high-torque servo motor,servo pulses ranging from 500 to 2500 microseconds (&amp;micro;s), with a frequency of 50Hz to 333Hz.&amp;nbsp;</td></tr><tr><td>servomotor</td><td>Kollmorgen Co-Engineers&amp;nbsp;Motors</td><td>SG-90R</td><td>1.8 kg-cm to 2.5 kg-cm load can be applied to SG-90R servo.</td></tr><tr><td>Stepper Motor</td><td>28BYJ-48</td><td>28BYJ-48</td><td>Steper motor, 5V DC, 100 Hz frequency, torque 1200 Gf.cm</td></tr><tr><td>Stepper Motor</td><td>Nema 23</td><td>Nema</td><td>Steper motor, 9V - 42 V DC, 100 Hz frequency</td></tr></tbody></table>

automatic surgery in transcatheter aortic valve replacement using augmented reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

The protocol was tested with the heart phantom model. Figure 2 shows the expected setup for the live surveillance of the surgical field with the help of spatially distributed cameras. The distributed cameras, as shown in Figure 2, help to increase the spatial resolution of the field for effective 3D reconstruction. However, realizing the physical placement of those cameras in various spatial locations involves complexity. So, we have optimized the setup design a...

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

automatic-surgery-transcatheter-aortic-valve-replacement-using

Research

JoVE Journal

Medicine

698 Views.  Vellore Institute of Technology, Vellore. This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity. 

Video: Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

A Teleoperated Robotic System-Assisted Percutaneous Transiliac-Transsacral Screw Fixation Technique

Transiliac-transsacral screw fixation is challenging in clinical practice as the screws need to break through six layers of cortical bone. Transiliac-transsacral screws provide a longer lever arm to withstand the perpendicular vertical shear forces. However, the screw channel is so long that a minor discrepancy can lead to iatrogenic neurovascular injuries. The development of medical robots has improved the precision of surgery. The present protocol describes how to use a new teleoperated robotic system to execute transiliac-transacral screw fixation. The Robot was operated remotely to position the entry point and adjust the orientation of the sleeve. The screw positions were evaluated using postoperative computed tomography (CT). All the screws were safely implanted, as confirmed using intraoperative fluoroscopy. Postoperative CT confirmed that all the screws were in the cancellous bone. This system combines the doctor's initiative with the Robot's stability. The remote control of this procedure is possible. Robot-assisted surgery has a higher position-retention capacity compared with conventional methods. In contrast to active robotic systems, surgeons have full control over the operation. The robot system is fully compatible with operating room systems and does not require additional equipment.

Teleoperated robotic system-assisted percutaneous transiliac-transsacral screw fixation is a feasible technique. Screw channels can be implemented with high accuracy owing to the excellent freedom of movement and stability of the robotic arms.

Transiliac-transsacral screw fixation is challenging in clinical practice as the screws need to break through six layers of cortical bone. ...

Engineering

Improved Registration of 3D CT Angiography with X-ray Fluoroscopy for Image Fusion During Transcatheter Aortic Valve Implantation

The fusion of 3D anatomical models derived from high-fidelity pre-interventional computed tomography angiography (CTA), and x-ray (XR) fluoroscopy to facilitate anatomical guidance is of huge interest for complex cardiac interventions like TAVI procedures with cerebral protection. Co-registration of CTA and XR has been introduced either based on additional intraoperative non-/contrast-enhanced cone-beam computed tomography (CBCT) or two separate aortograms. With the related increase of radiation exposure and/or contrast agent (CA) dose, a potential additional risk for the patient is introduced. Here, we propose a modified co-registration approach making use of arteriograms of the iliofemoral arteries, routinely performed during the femoral puncture and sheath introduction. On-the-fly refinement of the co-registration during the on-going procedure enables accurate co-registration without any additional angiograms, thus reducing CA, XR dose and procedure time, while simultaneously improving operator confidence and procedure safety.

The aim of this study was to improve the co-registration for image fusion (IF) of pre-interventional CT data with real-time x-ray (XR) fluoroscopy during transfemoral transcatheter aortic valve implantation (TAVI).

The fusion of 3D anatomical models derived from high-fidelity pre-interventional computed tomography angiography (CTA), and x-ray (XR) fluoroscopy to ...

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

Four-Dimensional Computed Tomography-Guided Valve Sizing for Transcatheter Pulmonary Valve Replacement

The measurements of the right ventricle (RV) and pulmonary artery (PA), for selecting the optimal prosthesis size for transcatheter pulmonary valve replacement (TPVR), vary considerably. Three-dimensional (3D) computed tomography (CT) imaging for device size prediction is insufficient to assess the displacement of the right ventricular outflow tract (RVOT) and PA, which could increase the risk of stent misplacement and paravalvular leak. The aim of this study is to provide a dynamic model to visualize and quantify the anatomy of the RVOT to PA over the entire cardiac cycle by four-dimensional (4D) cardiac CT reconstruction to obtain an accurate quantitative evaluation of the required valve size. In this pilot study, cardiac CT from sheep J was chosen to illustrate the procedures. 3D cardiac CT was imported into 3D reconstruction software to build a 4D sequence which was divided into eleven frames over the cardiac cycle to visualize the deformation of the heart. Diameter, cross-sectional area, and circumference of five imaging planes at the main PA, sinotubular junction, sinus, basal plane of the pulmonary valve (BPV), and RVOT were measured at each frame in 4D straightened models prior to valve implantation to predict the valve size. Meanwhile, dynamic changes in the RV volume were also measured to evaluate right ventricular ejection fraction (RVEF). 3D measurements at the end of the diastole were obtained for comparison with the 4D measurements. In sheep J, 4D CT measurements from the straightened model resulted in the same choice of valve size for TPVR (30 mm) as 3D measurements. The RVEF of sheep J from pre-CT was 62.1 %. In contrast with 3D CT, the straightened 4D reconstruction model not only enabled accurate prediction for valve size selection for TPVR but also provided an ideal virtual reality, thus presenting a promising method for TPVR and the innovation of TPVR devices.

This study assessed a new methodology with a straightened model generated from the four-dimensional cardiac computed tomography sequence to obtain the desired measurements for valve sizing in the application of transcatheter pulmonary valve replacement.

The measurements of the right ventricle (RV) and pulmonary artery (PA), for selecting the optimal prosthesis size for transcatheter pulmonary valve ...

Intravascular Lithotripsy-Assisted Transfemoral Transcatheter Aortic Valve Implantation

During the last decade, transcatheter aortic valve implantation (TAVI) has evolved as a well-established therapy for aging patients suffering from symptomatic severe aortic valve stenosis. This is also reflected in the recently updated international guidelines on managing patients with valvular heart disease. A transfemoral&#160;(TF) TAVI&#160;approach has proven superior to alternative access strategies. With the introduction of intravascular lithotripsy (IVL), patients with calcified iliofemoral vascular disease and borderline intraluminal diameters have also become candidates for percutaneous TF-TAVI. Moreover, IVL reduces the risk of major vascular complications by modifying the superficial and deep vascular calcium, thereby changing the vessel compliance and controlling luminal expansion. In this way, IVL has shown to safely facilitate TF delivery of TAVI devices in patients with calcified peripheral artery disease. The present article aims to provide a detailed step-by-step description on how to perform IVL-assisted TF-TAVI safely and efficiently. Furthermore, a literature review on the outcomes obtained with this technology is included, along with a concise discussion on this unique TAVI approach.

Transcatheter aortic valve implantation (TAVI) has been shown to generate the best clinical outcomes when performed by the percutaneous transfemoral approach. Intravascular lithotripsy (IVL) can facilitate a transfemoral process in patients with calcified iliofemoral vascular disease and borderline intraluminal diameters. The present protocol describes IVL-assisted transfemoral TAVI.

During the last decade, transcatheter aortic valve implantation (TAVI) has evolved as a well-established therapy for aging patients suffering from ...

Augmented Reality Navigation-Guided Core Decompression for Osteonecrosis of Femoral Head

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For early-stage ONFH, core decompression surgery is a classical and effective hip preservation therapy. In traditional procedures of core decompression with Kirschner wire, there are still many problems such as X-ray exposure, repeated puncture verification, and damage to normal bone tissue. The blindness of the puncture process and the inability to provide real-time visualization are crucial reasons for these problems.
To optimize this procedure, our team developed an intraoperative navigation system on the basis of augmented reality (AR) technology. This surgical system can intuitively display the anatomy of the surgical areas and render preoperative images and virtual needles to intraoperative video in real-time. With the guide of the navigation system, surgeons can accurately insert Kirschner wires into the targeted lesion area and minimize the collateral damage. We conducted 10 cases of core decompression surgery with this system. The efficiency of positioning and fluoroscopy is greatly improved compared to the traditional procedures, and the accuracy of puncture is also guaranteed.

Augmented reality technology was applied to core decompression for osteonecrosis of the femoral head to realize real-time visualization of this surgical procedure. This method can effectively improve the safety and precision of core decompression.

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For ...

Virtual Reality-3D Modeling for Minimally Invasive Neuro-Interventional Planning

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular ...

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) for minimally invasive pedicle screw placement. The cadaveric porcine specimens were placed on a surgical table and draped with sterile covers. The levels of interest were identified using fluoroscopy, and a dynamic reference frame was attached to the spinous process of a vertebra in the region of interest. Cone beam computerized tomography (CBCT) was performed, and a 3D rendering was automatically generated, which was used for the subsequent planning of the pedicle screw placements. Each surgeon was fitted with an HMD that was individually eye-calibrated and connected to the spinal navigation system.
Navigated instruments, tracked by the navigation system and displayed in 2D and 3D in the HMD, were used for 33 pedicle cannulations, each with a diameter of 4.5 mm. Postprocedural CBCT scans were assessed by an independent reviewer to measure the technical (deviation from the planned path) and clinical (Gertzbein grade) accuracy of each cannulation. The navigation time for each cannulation was measured. The technical accuracy was 1.0 mm &#177; 0.5 mm at the entry point and 0.8 mm &#177; 0.1 mm at the target. The angular deviation was 1.5&#176; &#177; 0.6&#176;, and the mean insertion time per cannulation was 141 s &#177; 71 s. The clinical accuracy was 100% according to the Gertzbein grading scale (32 grade 0; 1 grade 1). When used for minimally invasive pedicle cannulations in a porcine model, submillimeter technical accuracy and 100% clinical accuracy could be achieved with this protocol.

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) ...

Mixed Reality Assisted Radical Endoscopic Thyroidectomy

Radical endoscopic thyroidectomy (ET) offers superior cosmetic outcomes and enhanced visibility of the surgical field compared to open surgery. However, the thyroid&#39;s unique physiological functions and intricate surrounding anatomy may result in various surgical complications. Mixed reality (MR), a real-time holographic visualization technology, enables the creation of highly realistic 3D models in the real world and facilitates multiple human-computer interactions. MR can be utilized for both preoperative evaluation and intraoperative navigation. First, semi-automatic 3D reconstruction of the neck from enhanced computed tomography images is performed using 3Dslicer. Next, the 3D model is imported into Unity3D to create a virtual hologram that can be displayed on an MR helmet-mounted display (HMD). During surgery, surgeons can wear the MR HMD to locate lesions and surrounding anatomy through the virtual hologram. In this study, patients requiring radical ET were randomly assigned to either the experimental group or the control group. Surgeons performed MR-assisted radical ET in the experimental group. A comparative analysis of surgical outcomes and the results of scales was conducted.&#160;This study successfully developed the neck 3D model and the virtual hologram. According to the NASA Task Load Index Scale, the experimental group exhibited significantly higher scores in &#39;Own Performance&#39; and lower scores in &#39;Effort&#39; compared to the control group (p = 0.002). Additionally, on the Likert Subjective Evaluation Scale, the mean scores for all questions exceeded 3. Although the incidence of surgical complications was lower in the experimental group than in the control group, the differences in surgical outcomes were not statistically significant.MR is beneficial for enhancing performance and alleviating the burden of surgeons during the perioperative period. Furthermore, MR has demonstrated the potential to enhance the safety of ET. Therefore, it is essential to further investigate the surgical applications of MR.

Radical endoscopic thyroidectomy is associated with various surgical complications. This study utilizes mixed reality techniques to assist surgeons in performing radical endoscopic thyroidectomy, aiming to enhance its safety and lower the surgical threshold.

Radical endoscopic thyroidectomy (ET) offers superior cosmetic outcomes and enhanced visibility of the surgical field compared to open surgery. However, ...

Augmented Reality-Based Visualization of a Resected Tumor Specimen

This video demonstrates using augmented reality (AR) for tumor resection, where a 3D-scanned tumor specimen is visualized as a hologram on a head-mounted display. Surgeons can manipulate and lock the hologram in place to ensure precise tumor removal.

All procedures involving sample collection have been performed in accordance with the institute&#39;s IRB guidelines.
NOTE: A schematic detailing the ...