The authors acknowledge no funding for this research.

1. Surgical environment
<ol>
	<li>Design a surgery environment as shown in Figure 1. Make sure the environment has an object-carrying platter, a robotic arm, and two side-hanging arms, one to hold a camera placeholder and the other to have a consistent white background along with the weighing module for balance.</li>
	<li>Develop two drivers, one for the snapshot of the live surgical environment, as mentioned in steps 2.1 to 2.10, and the other to control the revolving mechanism that supports 360&#176; surveillance, as mentioned in steps 3.1 to 3.4.</li>
	<li>Before implementing the above two modules, enable Bluetooth of the mobile device and the laptop, which serve as the surgeon&#39;s HoloLens emulator.</li>
	<li>Pair the devices for uninterrupted image transmission.</li>
</ol>
2. Setting up the driver to control the two hanging arms
<ol>
	<li>Make sure that the hanging arms are controlled by a stepper motor with the arrangement as shown in Figure 2 for a flawless revolution of 360&#176;.</li>
	<li>Connect the motor to the microcontroller board using the TB 6600 driver. To run the motor, install the microcontroller IDE from the browser.</li>
	<li>Click the Download button to download the software. Then, in the microcontroller IDE, go to File &#62; Open a New Sketch to write the code.</li>
	<li>Make sure to connect the microcontroller board to interface with the new sketch through a dedicated connection port, say COM 4. Check the Com Port and verify that it shows the microcontroller board.</li>
	<li>Check the hardware switch settings of the stepper motor driver TB 6600. Ensure that the settings are such that the current flow is 2 A, which can be attained by setting SW4 ON and SW5 and SW6 OFF.</li>
	<li>Ensure the switch positions of SW1, SW2, and SW3 are set so that the micro-step is 1/8 steps to attain the revolution steps as per the requirement. Ensure the settings are SW1 OFF, SW2 ON, and SW3 OFF in TB6600.</li>
	<li>Connect RTC 3231 with the microcontroller to have real global time synchronization. Make sure that the revolution step size is 12&#176; and that the motor step increment is triggered only when the real-time unit, i.e., the seconds read from the RTC module, is odd in number.</li>
	<li>Connect the 5 V pin of the microcontroller board to the RTC VCC and the microcontroller&#39;s GND to the RTC&#39;s GND.</li>
	<li>Connect the SCL pin of RTC to the A0 pin and the SDA to the A1 pin of the microcontroller. This module can ensure a step size of 12&#176;, making 30 steps in one revolution. Ensure that the step increment happens every odd second. Let this software module drive the stepper motor21.</li>
	<li>Verify the setup is working correctly by running the code, which is available on the GitHub page:&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
	<li>Download Android Studio to develop the automatic camera app. Ensure the system requirements are met, then download the software from the website.</li>
</ol>
3. Developing a driver for mobile-based scene surveillance and image transmission as a client module
<ol>
	<li>Develop a camera application in the Android operating system that can take snapshots every 2 s, especially when the seconds are odd numbers.</li>
	<li>Connect the mobile phone with the system. In Android Studio, click New &#62; New Project and choose Empty Views Activity. Click on Next to develop an Android code, which is available at&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
	<li>Ensure that the app captures the images automatically and sends them to a remote device consistently.</li>
	<li>Transmit snapshots from the mobile app immediately after taking the snapshot to the paired device, i.e., to the remote surgeon&#39;s system, through Bluetooth. 
	NOTE: Make sure the modules mentioned in sections 2 and 3 run in time synchronization, one for every even number of seconds and the other for every odd number of seconds.</li>
</ol>
4. Developing a client module to receive and handle surveillance images
<ol>
	<li>Open the server module, which is a graphical user interface.</li>
	<li>Enter the VVID port number in the text field VVID, whose default value is 94f39d29 7d6d 437d 973b fba39e49d4ee.</li>
	<li>Click on Create Socket to create and bind the socket. Click on Connect to establish a connection with the mobile app.</li>
	<li>Click on Capture to capture and save the scene surveillance images in the local folder</li>
	<li>Enter the local folder name in the field folder name if it needs to be other than the default one mentioned.</li>
</ol>
5. Operating the robotic arm
<ol>
	<li>Let the client module include a robotic arm as well. Design the arm to have a rotational movement in its base, shoulder, elbow, wrist, and fingers.</li>
	<li>Make sure that MG 996R servos are used for governing the rotational movement at the base, shoulder and elbow. Ensure that the SG 90 servo motor is used to control the rotational movement at the elbow and fingers.</li>
	<li>Compile the code given in&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR in the microcontroller IDE to drive the robotic arm based on the commands received from the remote surgeon.</li>
</ol>
6. 3D reconstruction for augmented reality
<ol>
	<li>Read two images at a time in a sequence, one by one, from the local folder to obtain the possible overlapping (as the images are collected in close proximity, there will be an overlapping between the consecutive images) between them.</li>
	<li>Design a tertiary filter as per the requirement of the Directional Intensified Feature Description using the Tertiary Filtering22 (DITF)&#160;algorithm to obtain the gradient and orientation.</li>
	<li>Extract the features using the DITF method22, as shown in Figure 3.</li>
	<li>Reconstruct 3D images from the collected features using SFM23.</li>
</ol>
7. Hand gesture recognition at the surgeon&#39;s location
<ol>
	<li>Facilitate the surgeon to inspect the 3D reconstructed image features by enabling him/her to visualize the environment from all perspectives by providing hand gesture-based rotation and zoom in/out of reconstructed features.</li>
	<li>Normalize and map the distance between the tip of the surgeon&#39;s thumb and the index finger of the right hand into a corresponding angle of rotation. Let the normalization be in such a way that the bare minimum distance corresponds to 0&#176; and the maximum to 180&#176;.</li>
	<li>Transmit the hand gesture control, through Bluetooth, to the remote surgery environment as well for the rotation of the object platter, which makes it revolve on its axis as the 3D reconstructed features revolve at the surgeon&#39;s end.</li>
	<li>Find the distance between the tip and thumb of the surgeon&#39;s left hand to control the movement of the fingers of the robot arm.</li>
	<li>Measure the angle of elevation from the spatial distance between the tip of the thumb and index fingers of the surgeon&#39;s left hand with respect to an imaginary x-y ground plane to determine the elevation angle. Map this angle into an elevation angle that the robot arm can make with the x-y plane.</li>
	<li>Find the azimuth angle that the hand of the surgeon makes with that of the virtual y-z plane. Identify these angles through hand gesture-based recognition.</li>
	<li>Map the distance, elevation, and azimuth angles to control the robot&#39;s finger movement and arm rotation, which both correspond to elevation and azimuth angles.</li>
	<li>Let the surgeon inspect the reconstructed features by zooming and rotating. Let the surgeon transmit commands to the robot arm to perform surgery from a remote location.</li>
	<li>Make sure that the surgery commands are transmitted as a control string of sequence starting with a string concurrence followed by the values to control the platter rotation and robotic arm controller. Let [&#952;b, &#952;s, &#952;e, &#952;w, &#952;f] be the angle of the vector that consists of values, each corresponding to the control signal corresponding to the base, shoulder, elbow, wrist, and finger of the robot arm. 
	NOTE: The GitHub link provides the code to enable hand gesture control in the surgical field.&#160;https://github.com/Johnchristopherclement/Automatic_Surgery_model_using_AR.</li>
</ol>

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

<ol>
	<li>Wang, L. J., Casto, B., Reyes-Molyneux, N., Chance, W. W., Wang, S. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Smartphone-based+augmented+reality+patient+education+in+radiation+oncology.">Smartphone-based augmented reality patient education in radiation oncology.</a> Tech Innov Patient Supp Radiation Oncol. 29, 100229 (2024).</li><li>Guerroudji, M. A., Amara, K., Lichouri, M., Zenati, N., Masmoudi, M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+3D+visualization-based+augmented+reality+application+for+brain+tumor+segmentation.">A 3D visualization-based augmented reality application for brain tumor segmentation.</a> Comput Anim Virtual Worlds. 35 (1), e2223 (2024).</li><li>Xia, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+and+indoor+positioning+based+mobile+production+monitoring+system+to+support+workers+with+human-in-the-loop.">Augmented reality and indoor positioning based mobile production monitoring system to support workers with human-in-the-loop.</a> Robotic Comput Integ Manufac. 86, 102664 (2024).</li><li>Preece, C., Skandalis, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Time+to+imagine+an+escape:+investigating+the+consumer+timework+at+play+in+augmented+reality.">Time to imagine an escape: investigating the consumer timework at play in augmented reality.</a> Eur J Market. 58, 92-118 (2024).</li><li>Suresh, D., Aydin, A., James, S., Ahmed, K., Dasgupta, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=The+role+of+augmented+reality+in+surgical+training:+a+systematic+review.">The role of augmented reality in surgical training: a systematic review.</a> Surg Innov. 30, 366-382 (2023).</li><li>Moreta-Martinez, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Combining+augmented+reality+and+3D+printing+to+display+patient+models+on+a+smartphone.">Combining augmented reality and 3D printing to display patient models on a smartphone.</a> J Vis Exp. (155), 60618 (2020).</li><li>Ma, L., Huang, T., Wang, J., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Visualization,+registration+and+tracking+techniques+for+augmented+reality+guided+surgery:+a+review.">Visualization, registration and tracking techniques for augmented reality guided surgery: a review.</a> Phys Med Biol. 68, 04TR02 (2023).</li><li>Hofman, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=First-in-human+real-time+ai-assisted+instrument+deocclusion+during+augmented+reality+robotic+surgery.">First-in-human real-time ai-assisted instrument deocclusion during augmented reality robotic surgery.</a> Healthc Technol Lett. 11 (2-3), 33-39 (2023).</li><li>Thiene, G., Rizzo, S., Basso, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bicuspid+aortic+valve:+The+most+frequent+and+not+so+benign+congenital+heart+disease.">Bicuspid aortic valve: The most frequent and not so benign congenital heart disease.</a> Cardiovasc Pathol. 70, 107604 (2024).</li><li>Mack, M. J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Transcatheter+aortic-valve+replacement+in+low-risk+patients+at+five+years.">Transcatheter aortic-valve replacement in low-risk patients at five years.</a> New Engl J Med. 389, 1949-1960 (2023).</li><li>Vitanova, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Aortic+valve+versus+root+surgery+after+failed+transcatheter+aortic+valve+replacement.">Aortic valve versus root surgery after failed transcatheter aortic valve replacement.</a> J Thorac Cardiovas Surg. 166, 1418-1430 (2023).</li><li>Bydlon, T. M., Torjesen, A., Fokkenrood, S., Di Tullio, A., Flexman, M. L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=3d+visualisation+of+navigation+catheters+for+endovascular+procedures+using+a+3d+hub+and+fiber+optic+realshape+technology:+phantom+study+results.">3d visualisation of navigation catheters for endovascular procedures using a 3d hub and fiber optic realshape technology: phantom study results.</a> EJVES Vascular Forum. 59, 24-30 (2023).</li><li>Faris, H., Harfouche, C., Bandle, J., Wisbach, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Surgical+tele-mentoring+using+a+robotic+platform:+initial+experience+in+a+military+institution.">Surgical tele-mentoring using a robotic platform: initial experience in a military institution.</a> Surg Endosc. 37, 9159-9166 (2023).</li><li>Fitzgerald, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Mentoring+approaches+in+a+safe+surgery+program+in+tanzania:+Lessons+learned+during+covid-19+and+recommendations+for+the+future.">Mentoring approaches in a safe surgery program in tanzania: Lessons learned during covid-19 and recommendations for the future.</a> Surg Open Sci. 14, 109-113 (2023).</li><li>de Turenne, A., Eug&#232;ne, F., Blanc, R., Szewczyk, J., Haigron, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Catheter+navigation+support+for+mechanical+thrombectomy+guidance:+3d/2d+multimodal+catheter-based+registration+with+no+contrast+dye+fluoroscopy.">Catheter navigation support for mechanical thrombectomy guidance: 3d/2d multimodal catheter-based registration with no contrast dye fluoroscopy.</a> Int J Comput Assist Radiol Surg. 19 (3), 459-468 (2023).</li><li>Tang, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+multiple+catheter+tips+tracking+method+in+x-ray+fluoroscopy+images+by+a+new+lightweight+segmentation+network+and+bayesian+filtering.">A multiple catheter tips tracking method in x-ray fluoroscopy images by a new lightweight segmentation network and bayesian filtering.</a> Int J Med Robotics Comput Assist Surg. 19, e2569 (2023).</li><li>Annabestani, M., Caprio, A., Wong, S. C., Mosadegh, B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+machine+learning-based+roll+angle+prediction+for+intracardiac+echocardiography+catheter+during+bi-plane+fluoroscopy.">A machine learning-based roll angle prediction for intracardiac echocardiography catheter during bi-plane fluoroscopy.</a> Appl Sci. 13, 3483 (2023).</li><li>Thourani, V. H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Survival+after+surgical+aortic+valve+replacement+in+low-risk+patients:+a+contemporary+trial+benchmark.">Survival after surgical aortic valve replacement in low-risk patients: a contemporary trial benchmark.</a> Ann Thorac Surg. 117, 106-112 (2024).</li><li>Dom&#237;nguez-Velasco, C. F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+simulation+as+training+model+of+ventricular+puncture:+Evidence+in+the+improvement+of+the+quality+of+punctures.">Augmented reality simulation as training model of ventricular puncture: Evidence in the improvement of the quality of punctures.</a> Int J Med Robotics Comput Assist Surg. 19, e2529 (2023).</li><li>Wang, Q., Xie, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Arias:+An+ar-based+interactive+advertising+system.">Arias: An ar-based interactive advertising system.</a> Plos One. 18, e0285838 (2023).</li><li>Ji, F., Chong, F., Wang, F., Bai, D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+platform+for+the+unmanned+mining+process+in+underground+mines.">Augmented reality platform for the unmanned mining process in underground mines.</a> Mining,Metal Explor. 39 (2), 385-395 (2022).</li><li>Indhumathi, S., Clement, J. C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Directional+intensified+feature+description+using+tertiary+filtering+for+augmented+reality+tracking.">Directional intensified feature description using tertiary filtering for augmented reality tracking.</a> Sci Rep. 13, 20311 (2023).</li><li>Gao, L., Zhao, Y., Han, J., Liu, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Research+on+multi-view+3D+reconstruction+technology+based+on+SFM.">Research on multi-view 3D reconstruction technology based on SFM.</a> Sensors. 22 (12), 4366 (2022).</li><li>Su&#225;rez, I., Sfeir, G., Buenaposada, J. M., Baumela, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=BEBLID:+Boosted+efficient+binary+local+image+descriptor.">BEBLID: Boosted efficient binary local image descriptor.</a> Pattern Recog Lett. 133, 366-372 (2020).</li><li>Borman, R. I., Harjoko, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Improved+ORB+algorithm+through+feature+point+optimization+and+Gaussian+pyramid.">Improved ORB algorithm through feature point optimization and Gaussian pyramid.</a> Int J Adv Comp Sci Appl. 15 (2), 268-275 (2024).</li></ol>

The authors declare no conflicts of interest.

In an existing work15, X-ray and CT scans are examined to locate the catheter in the heart. However, AR TAVR replacement establishes a new possibility in TAVR18 surgical procedure by the implementation of an automated model using 3D reconstruction. As mentioned in the protocol section this work has five stages to design. The first stage of DITF22, mentioned in section 6, which we proposed in our previous work22, is enhanced in...

AR can superimpose the 3D model in a real-world environment. The technological development towards AR has made a paradigm shift in many fields, namely education1, medical2, manufacturing3, and entertainment4. AR technology, along with ultra-reliable low-latency communication, proves its inevitable role in the medical field. From the learning stage of human anatomy to surgical guidance, the stages of learning can be visualized with AR-powered software5,6 and hardware. AR provides a crucial and reliable solution to the medical practitioner in a surgical environment7,8.
Aortic valve stenosis is the heart valve disease, which is most common among mankind9. The root cause of the disease is poor food habits and irregular routines of day-to-day life. The symptom and result of the disease is the narrowing of the heart valve, followed by a reduction in the blood flow. This problem needs to be addressed before any damage takes place to the human heart. Thus, the heart is overburdened to process the blood flow. So, before any damage happens, surgery needs to be done, which, owing to technological developments in recent days, can also be done using the TAVR procedure. The procedure can be adopted based on the condition of the heart and other body parts of patients. In this TAVR10,11, the catheter is inserted to replace the damaged valve in the heart. However, placing the catheter position12 to replace the valve is tedious for the practitioner. This idea motivated us to design an automated surgery model based on AR13,14, which helps the surgeon to precisely position the valve during the replacement process. Moreover, the surgery can be performed by a motion mapping algorithm, which maps the surgeon&#39;s movement captured from a remote location to the robotic arm.
In the existing work15,16,17, the visualization of the TAVR18 procedure is monitored through fluoroscopy. Hence, it is difficult to analyze the heart valve and tedious to find the replacement location. This sets up a barrier to positioning the catheter in the human heart. In addition, the remote motion is mapped to the surgical field to make the process automated. However, to overcome the research gap, we propose an automated robotic-based surgery for valve replacement using AR-assisted technology.
The protocol is a generic model that can be applied to all surgical environments. In the beginning stage of the work, 2D images are captured all around the surgical environment with the fullest spatial resolution of the largest degree of freedom. This means that enough images are captured for 3D reconstruction19 purpose, followed by motion mapping through hand gesture tracking20.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>android IDE</td>
			<td>software</td>
			<td>https://developer.android.com/studio</td>
			<td>software can be downloaded from this link</td>
		</tr>
		<tr>
			<td>Arduino Board</td>
			<td>Ardunio Uno</td>
			<td>Ardunio Uno</td>
			<td>Microcontroller for processing</td>
		</tr>
		<tr>
			<td>arduino software</td>
			<td>software</td>
			<td>https://www.arduino.cc/en/software.</td>
			<td>software can be downloaded from this link</td>
		</tr>
		<tr>
			<td>Human Heart phantom model</td>
			<td>Biology Lab Equipment Manufacturer and Exporter</td>
			<td>B071YBLX2V(8B-ZB2Q-H3MS-1)</td>
			<td>light weight model with 3parts to the deep analysis of heart.</td>
		</tr>
		<tr>
			<td>mobile holder</td>
			<td>Humble universal monopoad holder</td>
			<td>B07S9KNGVS</td>
			<td>To carry the mobile in surgical field</td>
		</tr>
		<tr>
			<td>pycharm IDE</td>
			<td>software</td>
			<td>https://www.jetbrains.com/pycharm/</td>
			<td>software can be downloaded from this link</td>
		</tr>
		<tr>
			<td>Robot arm</td>
			<td>Printed-bots</td>
			<td>B08R2JLKYM(P0-E2UT-JSOU)</td>
			<td>arm can be controlled through control signal.it has 5 degree of freedom to access.</td>
		</tr>
		<tr>
			<td>servo motor</td>
			<td>Kollmorgen Co-Engineers Motors</td>
			<td>MG-966R</td>
			<td>high-torque servo motor,servo pulses ranging from 500 to 2500 microseconds (&#181;s), with a frequency of 50Hz to 333Hz.&#160;</td>
		</tr>
		<tr>
			<td>servomotor</td>
			<td>Kollmorgen Co-Engineers&#160;Motors</td>
			<td>SG-90R</td>
			<td>1.8 kg-cm to 2.5 kg-cm load can be applied to SG-90R servo.</td>
		</tr>
		<tr>
			<td>Stepper Motor</td>
			<td>28BYJ-48</td>
			<td>28BYJ-48</td>
			<td>Steper motor, 5V DC, 100 Hz frequency, torque 1200 Gf.cm</td>
		</tr>
		<tr>
			<td>Stepper Motor</td>
			<td>Nema 23</td>
			<td>Nema</td>
			<td>Steper motor, 9V - 42 V DC, 100 Hz frequency</td>
		</tr>
	</tbody>
</table>

automatic surgery in transcatheter aortic valve replacement using augmented reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

The protocol was tested with the heart phantom model. Figure 2 shows the expected setup for the live surveillance of the surgical field with the help of spatially distributed cameras. The distributed cameras, as shown in Figure 2, help to increase the spatial resolution of the field for effective 3D reconstruction. However, realizing the physical placement of those cameras in various spatial locations involves complexity. So, we have optimized the setup design a...

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

automatic-surgery-transcatheter-aortic-valve-replacement-using

Research

JoVE Journal

Medicine

565 Views.  Vellore Institute of Technology, Vellore. This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity. 

Video: Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

Transplantation of Pulmonary Valve Using a Mouse Model of Heterotopic Heart Transplantation

Tissue engineered heart valves, especially decellularized valves, are starting to gain momentum in clinical use of reconstructive surgery with mixed results. However, the cellular and molecular mechanisms of the neotissue development, valve thickening, and stenosis development are not researched extensively. To answer the above questions, we developed a murine heterotopic heart valve transplantation model. A heart valve was harvested from a valve donor mouse and transplanted to a heart donor mouse. The heart with a new valve was transplanted heterotopically to a recipient mouse. The transplanted heart showed its own heartbeat, independent of the recipient&#8217;s heartbeat. The blood flow was quantified using a high frequency ultrasound system with a pulsed wave Doppler. The flow through the implanted pulmonary valve showed forward flow with minimal regurgitation and the peak flow was close to 100 mm/sec. This murine model of heart valve transplantation is highly versatile, so it can be modified and adapted to provide different hemodynamic environments and/or can be used with various transgenic mice to study neotissue development in a tissue engineered heart valve.

In order to understand the cellular and molecular mechanisms underlying neotissue formation and stenosis development in tissue engineered heart valves, a murine model of heterotopic heart valve transplantation was developed. A pulmonary heart valve was transplanted to recipient using the heterotopic heart transplantation technique.

Tissue engineered heart valves, especially decellularized valves, are starting to gain momentum in clinical use of reconstructive surgery with mixed ...

Full-root Aortic Valve Replacement by Stentless Aortic Xenografts in Patients with Small Aortic Roots

In patients with small aortic roots who need an aortic valve replacement with biological valve substitutes, the implantation of the stented pericardial valve might not meet the functional needs. The implantation of a too-small stented pericardial valve, leading to an effective orifice area indexed to a body surface area less than 0.85 cm2/m2, is regarded as prosthesis-patient mismatch (PPM). A PPM negatively affects the regression of left ventricular hypertrophy and thus the normalization of left ventricular function and the alleviation of symptoms. Persistent left ventricular hypertrophy is associated with an increased risk of arrhythmias and sudden cardiac death. In the case of predictable PPM, there are three options: 1) accept the PPM resulting from the implantation of a stented pericardial valve when comorbidities of the patient forbid the more technically demanding operative technique of implanting a larger prosthesis, 2) enlarge the aortic root to accommodate a larger stented valve substitute, or 3) implant a stentless biological valve or a homograft. Compared to classical aortic valve replacement with stented pericardial valves, the full-root implantation of stentless aortic xenografts offers the possibility of implanting a 3-4 mm larger valve in a given patient, thus allowing significant reduction in transvalvular gradients. However, a number of cardiac surgeons are reluctant to transform a classical aortic valve replacement with stented pericardial valves into the more technically challenging full-root implantation of stentless aortic xenografts. Given the potential hemodynamic advantages of stentless aortic xenografts, we have adopted full-root implantation to avoid PPM in patients with small aortic roots necessitating an aortic valve replacement. Here, we describe in detail a technique for the full-root implantation of stentless aortic xenografts, with emphasis on the management of the proximal suture line and coronary anastomoses. Limitations of this technique and alternative options are discussed.

Full-root aortic valve replacement by stentless aortic xenograft is a viable option in patients with small aortic roots. We describe, a technique for the full-root implantation of stentless aortic xenografts, with emphasis on the management of the proximal suture line and coronary anastomoses, and discuss its limitations and alternative options.

In patients with small aortic roots who need an aortic valve replacement with biological valve substitutes, the implantation of the stented pericardial ...

Standardized Technique of Aortic Valve Re-implantation for Valve-sparing Aortic Root Replacement

Despite the obvious advantages of the preservation of a normal aortic valve during aortic root replacement, the complexity of valve sparing procedures prevents a number of cardiac surgeons from incorporating them into their practice. The aim of this protocol is to describe a simplified and user-friendly technique of an aortic valve-sparing root replacement (VSRR) procedure by re-implantation of the aortic valve. Proper selection of patients and limitations of the technique are discussed.
In 54 consecutive patients, normal appearing aortic valves were re-implanted in a commercially available polyester prosthesis with pre-shaped sinuses by a simplified and standardized technique. Placement of the first row of the proximal suture line, choice of the prosthesis size, and adjustment of the height of the commissures of the patient to the fixed height of the sinus portion of the prosthesis were slightly modified from the reference techniques with the aim of increasing its feasibility for use by other cardiac surgeons. Early mortality and morbidity as well as 5-year survival, freedom from aortic valve reoperation, and freedom from recurrent moderate regurgitation were collected in all patients.
Thirty-day mortality, re-sternotomy for bleeding, re-sternotomy for mediastinitis, and the incidence of stroke were very low, 1.8% for each (1 of 54). No patient required permanent pace-maker implantation. At 5 years, survival, freedom from aortic valve reoperation, and freedom from recurrent moderate regurgitation were 97.5%, 95.2%, and 91.6%, respectively.
Mid-term results of our standardized technique of re-implantation of the aortic valve for valve-sparing aortic root replacement are very good and compare with more complex techniques reported by experienced surgeons. By following the present protocol of the standardized re-implantation technique, a greater number of cardiac surgeons can perform this procedure with comparable good results.

Valve-sparing aortic root replacement has the advantage of preserving the patient&#39;s own aortic valve. The complexity of the reported techniques to date restricts their use to a limited number of cardiac surgeons. This protocol describes step-by-step a standardized technique reproducible by a greater number of cardiac surgeons.

Despite the obvious advantages of the preservation of a normal aortic valve during aortic root replacement, the complexity of valve sparing procedures ...

Technique and Patient Selection Criteria of Right Anterior Mini-Thoracotomy for Minimal Access Aortic Valve Replacement

Aortic valve stenosis has become the most prevalent valvular heart disease in developed countries, and is due to the aging of these populations. The incidence of the pathology increases with growing age after 65 years. Conventional surgical aortic valve replacement through median sternotomy has been the gold standard of patient care for symptomatic aortic valve stenosis. However, as the risk profile of patients worsens, other therapeutic strategies have been introduced in an attempt to maintain the excellent results obtained by the established surgical treatment. One of these approaches is represented by transcatheter aortic valve implantation. Although the outcomes of high-risk patients undergoing treatment for symptomatic aortic valve stenosis have improved with transcatheter aortic valve replacement, many patients with this condition remain candidates for surgical aortic valve replacement. In order to reduce the surgical trauma in patients who are candidates for surgical aortic valve replacement, minimally invasive approaches have garnered interest during the past decade. Since the introduction of right anterior thoracotomy for aortic valve replacement in 1993, right anterior mini-thoracotomy and upper hemi-sternotomy have become the predominant incisional approaches among cardiac surgeons performing minimal access aortic valve replacement. Beside the location of the incision, the arterial cannulation site represents the second major landmark of minimal access techniques for aortic valve replacement. The two most frequently used arterial cannulation sites include central aortic and peripheral femoral approaches. With the purpose of reducing surgical trauma in these patients, we have opted for a right anterior mini-thoracotomy approach with a central aortic cannulation site. This protocol describes in detail a technique for minimally invasive aortic valve replacement and provides recommendations for patient selection criteria, including cardiac computer tomography measurements. The indications and limitations of this technique, as well as its alternatives, are discussed.

The goal of this protocol is to describe in detail the technique of minimally invasive aortic valve replacement through a right anterior mini-thoracotomy and central aortic cannulation. This technique can potentially enhance patients&#39; comfort and, by reducing post-operative morbidity, promote lowering the length of stay and global costs.

Aortic valve stenosis has become the most prevalent valvular heart disease in developed countries, and is due to the aging of these populations. The ...

Improved Registration of 3D CT Angiography with X-ray Fluoroscopy for Image Fusion During Transcatheter Aortic Valve Implantation

The fusion of 3D anatomical models derived from high-fidelity pre-interventional computed tomography angiography (CTA), and x-ray (XR) fluoroscopy to facilitate anatomical guidance is of huge interest for complex cardiac interventions like TAVI procedures with cerebral protection. Co-registration of CTA and XR has been introduced either based on additional intraoperative non-/contrast-enhanced cone-beam computed tomography (CBCT) or two separate aortograms. With the related increase of radiation exposure and/or contrast agent (CA) dose, a potential additional risk for the patient is introduced. Here, we propose a modified co-registration approach making use of arteriograms of the iliofemoral arteries, routinely performed during the femoral puncture and sheath introduction. On-the-fly refinement of the co-registration during the on-going procedure enables accurate co-registration without any additional angiograms, thus reducing CA, XR dose and procedure time, while simultaneously improving operator confidence and procedure safety.

The aim of this study was to improve the co-registration for image fusion (IF) of pre-interventional CT data with real-time x-ray (XR) fluoroscopy during transfemoral transcatheter aortic valve implantation (TAVI).

The fusion of 3D anatomical models derived from high-fidelity pre-interventional computed tomography angiography (CTA), and x-ray (XR) fluoroscopy to ...

Integrating Augmented Reality Tools in Breast Cancer Related Lymphedema Prognostication and Diagnosis

Breast cancer related lymphedema (BCRL) is a detrimental condition characterized by fluid accumulation in the upper limb in breast cancer patients subjected to axillary surgery and/or radiations. Its etiology is multifactorial and include also tumor-specific pathological features, such as lymphovascular invasion (LVI) and extranodal extension (ENE). To date, no widely employed guidelines for the early diagnosis of BCRL are available. Here, we illustrate a protocol for a digitally assisted BCRL assessment using a 3D laser scanner (3DLS) and a tablet computer. It has been specifically optimized in a discovery cohort of high-risk breast cancer patients. This study provides a proof-of-principle that augmented reality tools, such as 3DLS, can be incorporated into the clinical workup of BCRL to allow for a precise, reproducible, reliable, and cheap diagnosis.

Breast cancer related lymphedema is frequent in breast cancer survivors, but there are not widely employed guidelines for its diagnosis and quantification. Here, we introduce a reliable and cost-effective protocol to define, quantify, and compare upper limb volume in breast cancer patients.

Breast cancer related lymphedema (BCRL) is a detrimental condition characterized by fluid accumulation in the upper limb in breast cancer patients ...

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

A Simplified Model for Heterotopic Heart Valve Transplantation in Rodents

There is an urgent clinical need for heart valve replacements that can grow in children. Heart valve transplantation is proposed as a new type of transplant with the potential to deliver durable heart valves capable of somatic growth with no requirement for anticoagulation. However, the immunobiology of heart valve transplants remains unexplored, highlighting the need for animal models to study this new type of transplant. Previous rat models for heterotopic aortic valve transplantation into the abdominal aorta have been described, though they are technically challenging and costly. For addressing this challenge, a renal subcapsular transplant model was developed in rodents as a practical and more straightforward method for studying heart valve transplant immunobiology. In this model, a single aortic valve leaflet is harvested and inserted into the renal subcapsular space. The kidney is easily accessible, and the transplanted tissue is securely contained in a subcapsular space that is well vascularized and can accommodate a variety of tissue sizes. Furthermore, because a single rat can provide three donor aortic leaflets and a single kidney can provide multiple sites for transplanted tissue, fewer rats are required for a given study. Here, the transplantation technique is described, providing a significant step forward in studying the transplant immunology of heart valve transplantation.

This protocol describes a simple and efficient method for the transplantation of aortic valve leaflets under the renal capsule to allow for the study of alloreactivity of heart valves.

There is an urgent clinical need for heart valve replacements that can grow in children. Heart valve transplantation is proposed as a new type of ...

Four-Dimensional Computed Tomography-Guided Valve Sizing for Transcatheter Pulmonary Valve Replacement

The measurements of the right ventricle (RV) and pulmonary artery (PA), for selecting the optimal prosthesis size for transcatheter pulmonary valve replacement (TPVR), vary considerably. Three-dimensional (3D) computed tomography (CT) imaging for device size prediction is insufficient to assess the displacement of the right ventricular outflow tract (RVOT) and PA, which could increase the risk of stent misplacement and paravalvular leak. The aim of this study is to provide a dynamic model to visualize and quantify the anatomy of the RVOT to PA over the entire cardiac cycle by four-dimensional (4D) cardiac CT reconstruction to obtain an accurate quantitative evaluation of the required valve size. In this pilot study, cardiac CT from sheep J was chosen to illustrate the procedures. 3D cardiac CT was imported into 3D reconstruction software to build a 4D sequence which was divided into eleven frames over the cardiac cycle to visualize the deformation of the heart. Diameter, cross-sectional area, and circumference of five imaging planes at the main PA, sinotubular junction, sinus, basal plane of the pulmonary valve (BPV), and RVOT were measured at each frame in 4D straightened models prior to valve implantation to predict the valve size. Meanwhile, dynamic changes in the RV volume were also measured to evaluate right ventricular ejection fraction (RVEF). 3D measurements at the end of the diastole were obtained for comparison with the 4D measurements. In sheep J, 4D CT measurements from the straightened model resulted in the same choice of valve size for TPVR (30 mm) as 3D measurements. The RVEF of sheep J from pre-CT was 62.1 %. In contrast with 3D CT, the straightened 4D reconstruction model not only enabled accurate prediction for valve size selection for TPVR but also provided an ideal virtual reality, thus presenting a promising method for TPVR and the innovation of TPVR devices.

This study assessed a new methodology with a straightened model generated from the four-dimensional cardiac computed tomography sequence to obtain the desired measurements for valve sizing in the application of transcatheter pulmonary valve replacement.

The measurements of the right ventricle (RV) and pulmonary artery (PA), for selecting the optimal prosthesis size for transcatheter pulmonary valve ...

Transcatheter Pulmonary Valve Replacement from Autologous Pericardium with a Self-Expandable Nitinol Stent in an Adult Sheep Model

Transcatheter pulmonary valve replacement has been established as a viable alternative approach for patients suffering from right ventricular outflow tract or bioprosthetic valve dysfunction, with excellent early and late clinical outcomes. However, clinical challenges such as stented heart valve deterioration, coronary occlusion, endocarditis, and other complications must be addressed for lifetime application, particularly in pediatric patients. To facilitate the development of a lifelong solution for patients, transcatheter autologous pulmonary valve replacement was performed in an adult sheep model. The autologous pericardium was harvested from the sheep via left anterolateral minithoracotomy under general anesthesia with ventilation. The pericardium was placed on a 3D shaping heart valve model for non-toxic cross-linking for 2 days and 21 h. Intracardiac echocardiography (ICE) and angiography were performed to assess the position, morphology, function, and dimensions of the native pulmonary valve (NPV). After trimming, the crosslinked pericardium was sewn onto a self-expandable Nitinol stent and crimped into a self-designed delivery system. The autologous pulmonary valve (APV) was implanted at the NPV position via left jugular vein catheterization. ICE and angiography were repeated to evaluate the position, morphology, function, and dimensions of the APV. An APV was successfully implanted in&#160;sheep J. In this paper, sheep J was selected to obtain representative results. A 30 mm APV with a Nitinol stent was accurately implanted at the NPV position without any significant hemodynamic change. There was no paravalvular leak, no new pulmonary valve insufficiency, or stented pulmonary valve migration. This study demonstrated the feasibility and safety, in a long-time follow-up, of developing an APV for implantation at the NPV position with a self-expandable Nitinol stent via jugular vein catheterization in an adult sheep model.

This study demonstrates the feasibility and safety of developing an autologous pulmonary valve for implantation at the native pulmonary valve position by using a self-expandable Nitinol stent in an adult sheep model. This is a step toward developing transcatheter pulmonary valve replacement for patients with right ventricular outflow tract dysfunction.

Transcatheter pulmonary valve replacement has been established as a viable alternative approach for patients suffering from right ventricular outflow ...