This report was supported by projects PI18/01625 and PI15/02121 (Ministerio de Ciencia, Innovaci&#243;n y Universidades, Instituto de Salud Carlos III and European Regional Development Fund "Una manera de hacer Europa") and IND2018/TIC-9753 (Comunidad de Madrid).

<ol>
	<li>Coles, T. R., John, N. W., Gould, D., Caldwell, D. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Integrating+Haptics+with+Augmented+Reality+in+a+Femoral+Palpation+and+Needle+Insertion+Training+Simulation.">Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation.</a> IEEE Transactions on Haptics. 4 (3), 199-209 (2011).</li><li>Pelargos, P. E., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Utilizing+virtual+and+augmented+reality+for+educational+and+clinical+enhancements+in+neurosurgery.">Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery.</a> Journal of Clinical Neuroscience. 35, 1-4 (2017).</li><li>Abhari, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Training+for+Planning+Tumour+Resection:+Augmented+Reality+and+Human+Factors.">Training for Planning Tumour Resection: Augmented Reality and Human Factors.</a> IEEE Transactions on Biomedical Engineering. 62 (6), 1466-1477 (2015).</li><li>Uppot, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Implementing+Virtual+and+Augmented+Reality+Tools+for+Radiology+Education+and+Training,+Communication,+and+Clinical+Care.">Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care.</a> Radiology. 291, 182210 (2019).</li><li>Pratt, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Through+the+HoloLensTM+looking+glass:+augmented+reality+for+extremity+reconstruction+surgery+using+3D+vascular+models+with+perforating+vessels.">Through the HoloLensTM looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels.</a> European Radiology Experimental. 2 (1), 2 (2018).</li><li>Rose, A. S., Kim, H., Fuchs, H., Frahm, J. -. M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+augmented-reality+applications+in+otolaryngology-head+and+neck+surgery.">Development of augmented-reality applications in otolaryngology-head and neck surgery.</a> The Laryngoscope. , (2019).</li><li>Zhou, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Robot-Assisted+Surgery+for+Mandibular+Angle+Split+Osteotomy+Using+Augmented+Reality:+Preliminary+Results+on+Clinical+Animal+Experiment.">Robot-Assisted Surgery for Mandibular Angle Split Osteotomy Using Augmented Reality: Preliminary Results on Clinical Animal Experiment.</a> Aesthetic Plastic Surgery. 41 (5), 1228-1236 (2017).</li><li>Heinrich, F., Joeres, F., Lawonn, K., Hansen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparison+of+Projective+Augmented+Reality+Concepts+to+Support+Medical+Needle+Insertion.">Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion.</a> IEEE Transactions on Visualization and Computer Graphics. 25 (6), 1 (2019).</li><li>Deng, W., Li, F., Wang, M., Song, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Easy-to-Use+Augmented+Reality+Neuronavigation+Using+a+Wireless+Tablet+PC.">Easy-to-Use Augmented Reality Neuronavigation Using a Wireless Tablet PC.</a> Stereotactic and Functional Neurosurgery. 92 (1), 17-24 (2014).</li><li>Fan, Z., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=3D+interactive+surgical+visualization+system+using+mobile+spatial+information+acquisition+and+autostereoscopic+display.">3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display.</a> Journal of Biomedical Informatics. 71, 154-164 (2017).</li><li>Arnal-Burr&#243;, J., P&#233;rez-Ma&#241;anes, R., Gallo-del-Valle, E., Igualada-Blazquez, C., Cuervas-Mons, M., Vaquero-Mart&#237;n, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Three+dimensional-printed+patient-specific+cutting+guides+for+femoral+varization+osteotomy:+Do+it+yourself.">Three dimensional-printed patient-specific cutting guides for femoral varization osteotomy: Do it yourself.</a> The Knee. 24 (6), 1359-1368 (2017).</li><li>Vaquero, J., Arnal, J., Perez-Ma&#241;anes, R., Calvo-Haro, J., Chana, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=3D+patient-specific+surgical+printing+cutting+blocks+guides+and+spacers+for+open-+wedge+high+tibial+osteotomy+(HTO)+-+do+it+yourself.">3D patient-specific surgical printing cutting blocks guides and spacers for open- wedge high tibial osteotomy (HTO) - do it yourself.</a> Revue de Chirurgie Orthop&#233;dique et Traumatologique. 102, 131 (2016).</li><li>De La Pe&#241;a, A., De La Pe&#241;a-Brambila, J., P&#233;rez-De La Torre, J., Ochoa, M., Gallardo, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Low-cost+customized+cranioplasty+using+a+3D+digital+printing+model:+a+case+report.">Low-cost customized cranioplasty using a 3D digital printing model: a case report.</a> 3D Printing in Medicine. 4 (1), 1-9 (2018).</li><li>Kamio, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Utilizing+a+low-cost+desktop+3D+printer+to+develop+a+"one-stop+3D+printing+lab"+for+oral+and+maxillofacial+surgery+and+dentistry+fields.">Utilizing a low-cost desktop 3D printer to develop a "one-stop 3D printing lab" for oral and maxillofacial surgery and dentistry fields.</a> 3D Printing in Medicine. 4 (1), 1-7 (2018).</li><li>Punyaratabandhu, T., Liacouras, P., Pairojboriboon, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Using+3D+models+in+orthopedic+oncology:+presenting+personalized+advantages+in+surgical+planning+and+intraoperative+outcomes.">Using 3D models in orthopedic oncology: presenting personalized advantages in surgical planning and intraoperative outcomes.</a> 3D Printing in Medicine. 4 (1), 1-13 (2018).</li><li>Wake, N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Patient-specific+3D+printed+and+augmented+reality+kidney+and+prostate+cancer+models:+impact+on+patient+education.">Patient-specific 3D printed and augmented reality kidney and prostate cancer models: impact on patient education.</a> 3D Printing in Medicine. 5 (1), 1-8 (2019).</li><li>Barber, S. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+Reality,+Surgical+Navigation,+and+3D+Printing+for+Transcanal+Endoscopic+Approach+to+the+Petrous+Apex.">Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex.</a> OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation. 2 (4), (2018).</li><li>Witowski, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+and+three-dimensional+printing+in+percutaneous+interventions+on+pulmonary+arteries.">Augmented reality and three-dimensional printing in percutaneous interventions on pulmonary arteries.</a> Quantitative Imaging in Medicine and Surgery. 9 (1), (2019).</li><li>Moreta-Mart&#237;nez, R., Garc&#237;a-Mato, D., Garc&#237;a-Sevilla, M., P&#233;rez-Ma&#241;anes, R., Calvo-Haro, J., Pascau, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Augmented+reality+in+computer-assisted+interventions+based+on+patient-specific+3D+printed+reference.">Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference.</a> Healthcare Technology Letters. , (2018).</li></ol>

The authors have nothing to disclose.

AR holds great potential in education, training, and surgical guidance in the medical field. Its combination with 3D printing opens may open new possibilities in clinical applications. This protocol describes a methodology that enables inexperienced users to create a smartphone app combining AR and 3DP for the visualization of anatomical 3D models of patients with 3D-printed reference markers.
In general, one of the most interesting clinical applications of AR and 3DP is to improve patient-to-physician communication by giving the patient a different perspective of the case, improving explanations of specific medical conditions or treatments. Another possible application includes surgical guidance for target localization, in which 3D-printed patient-specific tools (with a reference AR marker attached) can be placed on rigid structures (i.e., bone) and used as a reference for navigation. This application is especially useful for orthopedic and maxillofacial surgical procedures, in which bone tissue surface is easily accessed during surgery.
The protocol starts with section 1, describing the workstation set-up and software tools necessary. Section 2 describes how to use 3D Slicer software to easily segment target anatomies of the patient from any medical imaging modality to obtain 3D models. This step is crucial, as the virtual 3D models created are those displayed in the final AR application.
In section 3, 3D Slicer is used to register the 3D models created in the previous section with an AR marker. During this registration procedure, patient 3D models are efficiently and simply positioned with respect to the AR marker. The position defined in this section will determine the hologram relative position in the final app. It is believed that this solution reduces complexity and multiplies the possible applications. Section 3 describes two different options to define the spatial relationships between the models and AR markers: "visualization" and "registration" mode. The first option, "visualization" mode, allows the 3D models to be positioned anywhere with respect to the marker and displayed as the whole biomodel. This mode provides a realistic, 3D perspective of the patient&#39;s anatomy and allows moving and rotating of the biomodels by moving the tracked AR marker. The second option, "registration" mode, allows attachment and combining of a marker adaptor to any part of the biomodel, offering an automatic registration process. With this option, a small section of the 3D model, including the marker adaptor, can be 3D-printed, and the app can display the rest of the model as a hologram.
Section 4 provides guidelines for the 3D printing process. First, the user can choose between two different markers: the "dual color marker" and "sticker marker". The whole "dual color marker" can be 3D-printed but requires a dual extruder 3D printer. In case this printer is not available, the "sticker marker" is proposed. This is a simpler marker that can be obtained by 3D-printing the cubic structure, then pasting the images of the cube with sticker paper or sticker glue. Furthermore, both markers were designed with extensible sections to perfectly fit in a specific adaptor. Thus, the marker can be reused in several cases.
Section 5 describes the process to create a Unity project for AR using the Vuforia software development kit. This step may be the hardest portion for users with no programming experience, but with these guidelines, it should be easier to obtain the final application that is presented in section 6. The app displays the patient&#39;s virtual models over the smartphone screen when the camera recognizes the 3D-printed marker. In order for the app to detect the 3D marker, a minimum distance of approximately 40 cm or less from the phone to the marker as well as good lighting conditions are required.
The final application of this protocol allows the user to choose the specific biomodels to visualize and in which positions. Addtionally, the app can perform automatic patient-hologram registration using a 3D-printed marker and adaptor attached to the biomodel. This solves the challenge of registering virtual models with the environment in a direct and convenient manner. Moreover, this methodology does not require broad knowledge of medical imaging or software development, does not depend on complex hardware and expensive software, and can be implemented over a short time period. It is expected that this method will help accelerate the adoption of AR and 3DP technologies by medical professionals.

AR and 3DP are technologies that provide increasing numbers of applications in the medical field. In the case of AR, its interaction with virtual 3D models and the real environment benefits physicians in regards to education and training1,2,3, communication and interactions with other physicians4, and guidance during clinical interventions5,6,7,8,9,10. Likewise, 3DP has become a powerful solution for physicians when developing patient-specific customizable tools11,12,13 or creating 3D models of a patient&rsquo;s anatomy, which can help improve preoperative planning and clinical interventions14,15.
Both AR and 3DP technologies help to improve orientation, guidance, and spatial skills in medical procedures; thus, their combination is the next logical step. Previous work has demonstrated that their joint use can increase value in patient education16, facilitating explanations of medical conditions and proposed treatment, optimizing surgical workflow17,18 and improving patient-to-model registration19. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this work is to describe a step-by-step methodology that enables the use of AR and 3DP by inexperienced users without the need for broad technical knowledge.
This protocol describes how to develop an AR smartphone app that allows the superimposing of any patient-based 3D model onto a real-world environment using a 3D-printed marker tracked by the smartphone camera. In addition, an alternative approach is described to provide automatic registration between a 3D-printed biomodel (i.e., a 3D model created from a patient&rsquo;s anatomy) and the projected holograms. The protocol described is entirely based on free and multi-platform software.
In previous work, AR patient-to-image registration has been calculated manually5 with surface recognition algorithms10 or has been unavailable2. These methods have been considered somewhat limited when an accurate registration is required19. To overcome these limitations, this work provides tools to perform accurate and simple patient-to-image registration in AR procedures by combining AR technology and 3DP.
The protocol is generic and can be applied to any medical imaging modality or patient. As an example, a real clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. The first step describes how to easily segment the affected anatomy from computed tomography (CT) medical images to generate 3D virtual models. Afterward, positioning of the 3D models is performed, then the required tools and models are 3D-printed. Finally, the desired AR app is deployed. This app allows for the visualization of patient 3D models overlaid on a smartphone camera in real-time.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>3D Printing material: Acrylonitrile Butadiene Styrene (ABS)</td>
			<td></td>
			<td></td>
			<td>Thermoplastic polymer material usually used in domestic 3D printers.</td>
		</tr>
		<tr>
			<td>3D Printing material: Polylactic Acid (PLA)</td>
			<td></td>
			<td></td>
			<td>Bioplastic material usually used in domestic 3D printers.</td>
		</tr>
		<tr>
			<td>3D Slicer</td>
			<td></td>
			<td></td>
			<td>Open-source software platform for medical image informatics, image processing, and three-dimensional visualization</td>
		</tr>
		<tr>
			<td>Android</td>
			<td>Alphabet, Inc.</td>
			<td></td>
			<td>Android is a mobile operating system developed by Google. It is based on a modified version of the Linux kernel and other open source software, and is designed primarily for touchscreen mobile devices such as smartphones and tablets.</td>
		</tr>
		<tr>
			<td>Autodesk Meshmixer</td>
			<td>Autodesk, Inc.</td>
			<td></td>
			<td>Meshmixer is state-of-the-art software for working with triangle meshes. Free software.</td>
		</tr>
		<tr>
			<td>iPhone OS</td>
			<td>Apple, Inc.</td>
			<td></td>
			<td>iPhone OS is a mobile operating system created and developed by Apple Inc. exclusively for its hardware.</td>
		</tr>
		<tr>
			<td>Ultimaker 3 Extended</td>
			<td>Ultimaker BV</td>
			<td></td>
			<td>Fused deposition modeling 3D printer.</td>
		</tr>
		<tr>
			<td>Unity</td>
			<td>Unity Technologies</td>
			<td></td>
			<td>Unity is a real-time development platform to create 3D, 2D VR &#38; AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering &#38; more. Free software.</td>
		</tr>
		<tr>
			<td>Xcode</td>
			<td>Apple, Inc.</td>
			<td></td>
			<td>Xcode is a complete developer toolset for creating apps for Mac, iPhone, iPad, Apple Watch, and Apple TV. Free software.</td>
		</tr>
	</tbody>
</table>

combining augmented reality and 3d printing to display patient models on a smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

The protocol was applied to data from a patient suffering from distal leg sarcoma in order to visualize the affected anatomical region from a 3D perspective. Using the method described in section 2, the portion of the affected bone (here, the tibia and fibula) and tumor were segmented from the patient&#39;s CT scan. Then, using the segmentation tools from 3D Slicer, two biomodels were created: the bone (section of the tibia and fibula) (Figure 1A) and tumor (Figure 1B).
Next, the two 3D models were positioned virtually with respect to the marker for optimal visualization. Both modes described in section 3 were followed for this example. For visualization mode, the models were centered in the upper face of the marker (Figure 2). For registration mode, the marker adaptor was positioned in the bone (specifically, the tibia [Figure 3]). Then, a small section of the tibia was selected to be 3D-printed with a 3D marker adaptor (Figure 4). An Ultimaker 3 extended 3D printer with PLA material was used to create the 3D-printed markers (Figure 5A, B), marker holder base (Figure 5C) for the "visualization" mode, and section of the tibia for "registration" mode (Figure 5D). Figure 5E shows how the marker was attached to the "visualization" mode 3D-printed base. Figure 5F shows the attachment with the "registration" mode 3D-printed biomodel. Finally, Unity was used to create the app and deploy it on the smartphone.
Figure 6 shows how the app worked for "visualization" mode. The hologram was accurately located in the upper part of the cube as previously defined. Figure 7 shows the application for "registration" mode, in which the app positioned the complete bone model on top of the 3D-printed section. The final visualization of the holograms was clear and realistic, maintained the real sizes of the biomodels, and positioned accurately. When using the smartphone application, the AR marker needs to be visible by the camera for the app to correctly display the holograms. In addition, the light conditions in the scene must be of good quality and constant for proper marker detection. Bad light conditions or reflections on the marker surface hinder the tracking of the AR marker and cause malfunctioning of the app.
The time required to create the app depends on several factors. The duration of section 1 is limited by the download speed. Regarding anatomy segmentation (section 2), factors affecting segmentation time include complexity of the region and medical imaging modality (i.e., CT is easily segmented, while MRI is more difficult). For the representative example of the tibia, approximately 10 min was required to generate both 3D models from the CT scan. Biomodel positioning (section 3) is simple and straightforward. Here, it took approximately 5 min to define the biomodel position with respect to the AR marker. For the 3D printing step, the duration is highly dependent on the selected mode. The "dual color marker" was manufactured at high quality in a period of 5 h and 20 min. The "sticker marker" was manufactured in a period of 1 h and 30 min, plus the time required to paste the stickers. The final step for app development can be time-consuming for those with no previous experience in Unity, but it can be easily completed following the protocol steps. Once the AR markers have been 3D-printed, the development of an entirely new AR app can be performed in less than 1 h. This duration can be further reduced with additional experience.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/60618/60618fig01.jpg" /> 
Figure 1: Representation of 3D models created from a CT image of a patient suffering from distal leg sarcoma. (A) Bone tissue represented in white (tibia and fibula). (B) Tumor represented in red. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/60618/60618fig02.jpg" /> 
Figure 2: Results showing how "visualization" mode in 3D Slicer positions the virtual 3D models of the bone and tumor with respect to 3D-printed marker reference. The patient 3D models (A) are positioned above the upper face of the marker cube (B). <a href="https://www.jove.com/files/ftp_upload/60618/60618fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/60618/60618fig03.jpg" /> 
Figure 3: Results showing how "registration" mode in 3D Slicer positions the virtual 3D models of the bone and tumor (A) with respect to 3D-printed marker reference (B). The marker adaptor is attached to the bone tissue model. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/60618/60618fig04.jpg" /> 
Figure 4: Small section of the bone tissue and 3D marker adaptor. The two components are combined then 3D-printed. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/60618/60618fig05.jpg" /> 
Figure 5: 3D printed tools required for the final application. (A) "Two color cube marker" 3D-printed with two colors of materials. (B) "Sticker cube marker" 3D-printed, with stickers pasted. (C) Marker base cube adaptor. (D) Section of the patient&#39;s bone tissue 3D model and marker cube adaptor. (E) "Sticker cube marker" placed in the marker base cube adaptor. (F) "Two color cube marker" placed in the marker adaptor attached to the patient&#39;s anatomy. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig05large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/60618/60618fig06.jpg" /> 
Figure 6: App display when using "visualization" mode. The patient&#39;s affected anatomy 3D models are positioned above the upper face of the 3D-printed cube. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig06large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/60618/60618fig07.jpg" /> 
Figure 7: AR visualization when using "registration" mode. The 3D-printed marker enables registration of the 3D-printed biomodel with the virtual 3D models. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig07large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone at JoVE.com

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

combining-augmented-reality-3d-printing-to-display-patient-models-on

Research

JoVE Journal

Medicine

18.1K Views.  Universidad Carlos III de Madrid. Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Video: Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Step By Step: Microsurgical training method combining two nonliving animal models

The learning of microsurgical techniques and the maintenance of microsurgical skills have been traditionally based on the use of living animals, mainly laboratory rats. This method although extremely valuable can be economically demanding both for the surgeon and the sponsoring institution; it also requires special training facilities that may not always be available or accessible. Furthermore ethical concerns can limit the use of living animals for training purposes. Alternative training methods, such as inert tubes and gloves have not gained popularity among surgeons since they do not offer an experience similar to that of a clinical situation. Non-living animal models include the use of chicken thighs and wings; they offer a practice experience that resembles a clinical situation to a considerable extent. This type of training is relatively cheap and easily available. The microscope and instruments required can be acquired over the internet, and the chicken pieces can be bought at the local supermarket.
This approach allows a motivated trainee to rehearse different types of surgical techniques several times at a reasonable expense, helping to develop or maintain his surgical expertise if more complex facilities are not available. On the current manuscript we describe how to setup a small practice station, how to dissect the specimens, and how to practice both with the chicken thighs and with the chicken wings in a progressive fashion. This approach takes advantage on the versatility of the chicken thigh model and the small size of the chicken wing Brachial artery.

Here we describe how to set up a small microsurgical practice station and a simple and inexpensive method for the training of microsurgery with non-living animal models.

The learning of microsurgical techniques and the maintenance of microsurgical skills have been traditionally based on the use of living animals, mainly ...

Smartphone Fundus Photography

Smartphone fundus photography is a simple technique to obtain ocular fundus pictures using a smartphone camera and a conventional handheld indirect ophthalmoscopy lens. This technique is indispensable when picture documentation of optic nerve, retina, and retinal vessels is necessary but a fundus camera is not available. The main advantage of this technique is the widespread&#160;availability of smartphones that allows documentation of macula and optic nerve changes in many settings that was not previously possible. Following the well-defined steps detailed here, such as proper alignment of the phone camera, handheld lens, and the patient&#39;s pupil, is the key for obtaining a clear retina picture with no interfering light reflections and aberrations. In this paper, the optical principles of indirect ophthalmoscopy and fundus photography will be reviewed first. Then, the step-by-step method to record a good quality retinal image using a smartphone will be explained.

Fundus photography normally requires specialized fundus cameras that are not always available in all clinical settings.&#160;Here, a simple method to record ocular fundus images using a smartphone camera and a conventional high-plus handheld indirect ophthalmoscopy lens is described.

Smartphone fundus photography is a simple technique to obtain ocular fundus pictures using a smartphone camera and a conventional handheld indirect ...

Integrating Augmented Reality Tools in Breast Cancer Related Lymphedema Prognostication and Diagnosis

Breast cancer related lymphedema (BCRL) is a detrimental condition characterized by fluid accumulation in the upper limb in breast cancer patients subjected to axillary surgery and/or radiations. Its etiology is multifactorial and include also tumor-specific pathological features, such as lymphovascular invasion (LVI) and extranodal extension (ENE). To date, no widely employed guidelines for the early diagnosis of BCRL are available. Here, we illustrate a protocol for a digitally assisted BCRL assessment using a 3D laser scanner (3DLS) and a tablet computer. It has been specifically optimized in a discovery cohort of high-risk breast cancer patients. This study provides a proof-of-principle that augmented reality tools, such as 3DLS, can be incorporated into the clinical workup of BCRL to allow for a precise, reproducible, reliable, and cheap diagnosis.

Breast cancer related lymphedema is frequent in breast cancer survivors, but there are not widely employed guidelines for its diagnosis and quantification. Here, we introduce a reliable and cost-effective protocol to define, quantify, and compare upper limb volume in breast cancer patients.

Breast cancer related lymphedema (BCRL) is a detrimental condition characterized by fluid accumulation in the upper limb in breast cancer patients ...

Multicolor 3D Printing of Complex Intracranial Tumors in Neurosurgery

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. The model can be used for planning and simulating critical steps of a surgical approach. Therefore, it is important that anatomical structures such as blood vessels inside a tumor can be printed to be colored not only on their surface, but throughout their whole volume. During simulation this allows for the removal of certain parts (e.g., with a high-speed drill) and revealing internally located structures of a different color. Thus, diagnostic information from various imaging modalities (e.g., CT, MRI) can be combined in a single compact and tangible object.
However, preparation and printing of such a fully colored anatomical model remains a difficult task. Therefore, a step-by-step guide is provided, demonstrating the fusion of different cross-sectional imaging data sets, segmentation of anatomical structures, and creation of a virtual model. In a second step the virtual model is printed with volumetrically colored anatomical structures using a plaster-based color 3D binder jetting technique. This method allows highly accurate reproduction of patient-specific anatomy as shown in a series of 3D-printed petrous apex chondrosarcomas. Furthermore, the models created can be cut and drilled, revealing internal structures that allow for simulation of surgical procedures.

The protocol describes the fabrication of fully colored three-dimensional prints of patient-specific, anatomical skull models to be used for surgical simulation. The crucial steps of combining different imaging modalities, image segmentation, three-dimensional model extraction, and production of the prints are explained.

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. ...

3D Planning and Printing of Patient Specific Implants for Reconstruction of Bony Defects

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical field using the constantly developing 3D planning and printing capabilities. Computer-assisted design (CAD) and computer assisted manufacturing (CAM) are used to describe the 3D planning and manufacturing of the product. The planning and manufacturing of 3D surgical guides and reconstruction implants is performed almost exclusively by engineers. As technology advances and software interfaces become more user-friendly, it raises a question regarding the possibility of transferring the planning and manufacturing to the clinician. The reasons for such a shift are clear: the surgeon has the idea of what he wants to design, and he also knows what is feasible and could be used in the operating room. It allows him to be prepared for any scenario/unexpected results during the operation and allows the surgeon to be creative and express his new ideas using the CAD software. The purpose of this method is to provide clinicians with the ability to create their own surgical guides and reconstruction implants. In this manuscript, a detailed protocol will provide a simple method for segmentation using segmentation software and implant planning using a 3D design software. Following the segmentation and stl file production using segmentation software, the clinician could create a simple patient specific reconstruction plate or a more complex plate with a cradle for bone graft positioning. Surgical guides can be created for accurate resection, hole preparation for proper reconstruction plate positioning or for bone graft harvesting and re-contouring. A case of lower jaw reconstruction following plate fracture and nonunion healing of a trauma sustained injury is detailed.

This protocol describes the use of 3D planning and printing for reconstruction of bony defects. We use segmentation tools to create 3D models followed by 3D design software to create patient specific implants for reconstruction purposes concomitant to ablative surgery or as a second stage.

We are in the midst of the 3D era in most aspects of life, and especially in medicine. The surgical discipline is one of the major players in the medical ...

Treatment of Facial Deformities using 3D Planning and Printing of Patient-Specific Implants

Technological advancements in surgical planning and patient-specific implants are constantly evolving. One can either adopt the technology to achieve better results, even in the less experienced hand, or continue without it. As technology develops and becomes more user-friendly, we believe it is time to allow the surgeon the option to plan his/her operations and create his/her own patient-specific surgical guides and fixation plates allowing him full control over the process. We present here a protocol for 3D planning of the operation followed by 3D planning and printing of surgical guides and patient-specific fixation implants. During this process we use two commercial computer-assisted design (CAD) software. We also use a fused deposition&#160;modeling printer for the surgical guides and a selective laser sintering printer for the titanium patient-specific fixation implants. The process includes computed tomography (CT) imaging acquisition, 3D segmentation of the skull and facial bones from the CT, 3D planning of the operations, 3D planning of patient-specific fixation implant according to the final position of the bones, 3D planning of surgical guides for performing an accurate osteotomy and preparing the bone for the fixation plates, and 3D printing of the surgical guides and the patient-specific fixation plates. The advantages of the method include full control over the surgery, planned osteotomies and fixation plates, significant reduction in price, reduction in operation duration, superior performance and highly accurate results. Limitations include the need to master the CAD programs.

As technology develops and becomes more user-friendly, planning of operations and patient-specific surgical guides and fixation plates should be performed by the surgeon. We present a protocol for 3D planning of orthognathic skeletal movements and 3D planning and printing of patient-specific fixation plates and surgical guides.

Technological advancements in surgical planning and patient-specific implants are constantly evolving. One can either adopt the technology to achieve ...

Combining 3D-Printing and Electrospinning to Manufacture Biomimetic Heart Valve Leaflets

Electrospinning has become a widely used technique in cardiovascular tissue engineering as it offers the possibility to create (micro-)fibrous scaffolds with adjustable properties. The aim of this study was to create multilayered scaffolds mimicking the architectural fiber characteristics of human heart valve leaflets using conductive 3D-printed collectors.
Models of aortic valve cusps were created using commercial computer-aided design (CAD) software. Conductive polylactic acid was used to fabricate 3D-printed leaflet templates. These cusp negatives were integrated into a specifically designed, rotating electrospinning mandrel. Three layers of polyurethane were spun onto the collector, mimicking the fiber orientation of human heart valves. Surface and fiber structure was assessed with a scanning electron microscope (SEM). The application of fluorescent dye additionally permitted the microscopic visualization of the multilayered fiber structure. Tensile testing was performed to assess the biomechanical properties of the scaffolds.
3D-printing of essential parts for the electrospinning rig was possible in a short time for a low budget. The aortic valve cusps created following this protocol were three-layered, with a fiber diameter of 4.1 &#177; 1.6 &#181;m. SEM imaging revealed an even distribution of fibers. Fluorescence microscopy revealed individual layers with differently aligned fibers, with each layer precisely reaching the desired fiber configuration. The produced scaffolds showed high tensile strength, especially along the direction of alignment. The printing files for the different collectors are available as Supplemental File 1, Supplemental File 2, Supplemental File 3, Supplemental File 4, and Supplemental File 5.
With a highly specialized setup and workflow protocol, it is possible to mimic tissues with complex fiber structures over multiple layers. Spinning directly on 3D-printed collectors creates considerable flexibility in manufacturing 3D shapes at low production costs.

The presented method offers an innovative way for engineering biomimetic fiber structures in three-dimensional (3D) scaffolds (e.g., heart valve leaflets). 3D-printed, conductive geometries were used to determine shape and dimensions. Fiber orientation and characteristics were individually adjustable for each layer. Multiple samples could be manufactured in one setup.

Electrospinning has become a widely used technique in cardiovascular tissue engineering as it offers the possibility to create (micro-)fibrous scaffolds ...

Augmented Reality Navigation-Guided Core Decompression for Osteonecrosis of Femoral Head

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For early-stage ONFH, core decompression surgery is a classical and effective hip preservation therapy. In traditional procedures of core decompression with Kirschner wire, there are still many problems such as X-ray exposure, repeated puncture verification, and damage to normal bone tissue. The blindness of the puncture process and the inability to provide real-time visualization are crucial reasons for these problems.
To optimize this procedure, our team developed an intraoperative navigation system on the basis of augmented reality (AR) technology. This surgical system can intuitively display the anatomy of the surgical areas and render preoperative images and virtual needles to intraoperative video in real-time. With the guide of the navigation system, surgeons can accurately insert Kirschner wires into the targeted lesion area and minimize the collateral damage. We conducted 10 cases of core decompression surgery with this system. The efficiency of positioning and fluoroscopy is greatly improved compared to the traditional procedures, and the accuracy of puncture is also guaranteed.

Augmented reality technology was applied to core decompression for osteonecrosis of the femoral head to realize real-time visualization of this surgical procedure. This method can effectively improve the safety and precision of core decompression.

Osteonecrosis of the femoral head (ONFH) is a common joint disease in young and middle-aged patients, which seriously burdens their lives and work. For ...

3D Printing Model of a Patient's Specific Lumbar Vertebra

Selective dorsal rhizotomy (SDR) is a difficult, risky, and sophisticated operation, in which a laminectomy should not only expose an adequate surgical field of view but also protect the patient&#39;s spinal nerves from injury. Digital models play an important role in the pre-and intra-operation of SDR, because they can not only make doctors more familiar with the anatomical structure of the surgical site, but also provide precise surgical navigation coordinates for the manipulator. This study aims to create a 3D digital model of a patient-specific lumbar vertebra that can be used for planning, surgical navigation, and training of the SDR operation. The 3D printing model is also manufactured for more effective work during these processes.
Traditional orthopedic digital models rely almost entirely on computed tomography (CT) data, which is less sensitive to soft tissues. Fusion of the bone structure from CT and the neural structure from magnetic resonance imaging (MRI) is the key element for the model reconstruction in this study. The patient&#39;s specific 3D digital model is reconstructed for the real appearance of the surgical area and shows the accurate measurement of inter-structural distances and regional segmentation, which can effectively help in the preoperative planning and training of SDR. The transparent bone structure material of the 3D-printed model allows surgeons to clearly distinguish the relative relationship between the spinal nerve and the vertebral plate of the operated segment, enhancing their anatomical understanding and spatial sense of the structure. The advantages of the individualized 3D digital model and its accurate relationship between spinal nerve and bone structures make this method a good choice for preoperative planning of SDR surgery.

This study aims to create a 3D-printed model of a patient-specific lumbar vertebra, which contains both the vertebra and spinal nerve models fused from high-resolution computed tomography (HRCT) and MRI-Dixon data.

Selective dorsal rhizotomy (SDR) is a difficult, risky, and sophisticated operation, in which a laminectomy should not only expose an adequate surgical ...

Plasma Polishing as a New Polishing Option to Reduce the Surface Roughness of Porous Titanium Alloy for 3D Printing

Porous titanium alloy implants with simulated trabecular bone fabricated by 3D printing technology have broad prospects. However, due to the fact that some powder adheres to the surface of the workpiece during the manufacturing process, the surface roughness in direct printing pieces is relatively high. At the same time, since the internal pores of the porous structure cannot be polished by conventional mechanical polishing, an alternative method needs to be found. As a surface technology, plasma polishing technology is especially suitable for parts with complex shapes that are difficult to polish mechanically. It can effectively remove particles and fine splash residues attached to the surface of 3D printed porous titanium alloy workpieces. Therefore, it can reduce surface roughness. Firstly, titanium alloy powder is used to print the porous structure of the simulated trabecular bone with a metal 3D printer. After printing, heat treatment, removal of the supporting structure, and ultrasonic cleaning is carried out. Then, plasma polishing is performed, consisting of adding a polishing electrolyte with the pH set to 5.7, preheating the machine to 101.6 &#176;C, fixing the workpiece on the polishing fixture, and setting the voltage (313 V), current (59 A), and polishing time (3 min). After polishing, the surface of the porous titanium alloy workpiece is analyzed by a confocal microscope, and the surface roughness is measured. Scanning electron microscopy is used to characterize the surface condition of porous titanium. The results show that the surface roughness of the whole porous titanium alloy workpiece changed from Ra (average roughness) = 126.9 &#181;m to Ra = 56.28 &#181;m, and the surface roughness of the trabecular structure changed from Ra = 42.61 &#181;m to Ra = 26.25 &#181;m. Meanwhile, semi-molten powders and ablative oxide layers are removed, and surface quality is improved.

Plasma polishing is a promising surface processing technology, especially suitable for 3D printing of porous titanium alloy workpieces. It can remove semi-molten powders and ablative oxide layers, thereby effectively reducing surface roughness and improving surface quality.

Porous titanium alloy implants with simulated trabecular bone fabricated by 3D printing technology have broad prospects. However, due to the fact that ...