This report was supported by projects PI18/01625 and PI15/02121 (Ministerio de Ciencia, Innovaci&#243;n y Universidades, Instituto de Salud Carlos III and European Regional Development Fund "Una manera de hacer Europa") and IND2018/TIC-9753 (Comunidad de Madrid).

<ol><li>Coles, T. R., John, N. W., Gould, D., Caldwell, D. G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Integrating+Haptics+with+Augmented+Reality+in+a+Femoral+Palpation+and+Needle+Insertion+Training+Simulation%5BTitle%5D)+AND+%22IEEE+Transactions+on+Haptics%22%5BJournal%5D)">Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation.</a> IEEE Transactions on Haptics. 4 (3), 199-209 (2011).</li>
<li>Pelargos, P. E., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Utilizing+virtual+and+augmented+reality+for+educational+and+clinical+enhancements+in+neurosurgery%5BTitle%5D)+AND+%22Journal+of+Clinical+Neuroscience%22%5BJournal%5D)">Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery.</a> Journal of Clinical Neuroscience. 35, 1-4 (2017).</li>
<li>Abhari, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Training+for+Planning+Tumour+Resection%3A+Augmented+Reality+and+Human+Factors%5BTitle%5D)+AND+%22IEEE+Transactions+on+Biomedical+Engineering%22%5BJournal%5D)">Training for Planning Tumour Resection: Augmented Reality and Human Factors.</a> IEEE Transactions on Biomedical Engineering. 62 (6), 1466-1477 (2015).</li>
<li>Uppot, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Implementing+Virtual+and+Augmented+Reality+Tools+for+Radiology+Education+and+Training%2C+Communication%2C+and+Clinical+Care%5BTitle%5D)+AND+%22Radiology%22%5BJournal%5D)">Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care.</a> Radiology. 291, 182210(2019).</li>
<li>Pratt, P., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Through+the+HoloLensTM+looking+glass%3A+augmented+reality+for+extremity+reconstruction+surgery+using+3D+vascular+models+with+perforating+vessels%5BTitle%5D)+AND+%22European+Radiology+Experimental%22%5BJournal%5D)">Through the HoloLensTM looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels.</a> European Radiology Experimental. 2 (1), 2(2018).</li>
<li>Rose, A. S., Kim, H., Fuchs, H., Frahm, J. -M. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Development+of+augmented-reality+applications+in+otolaryngology-head+and+neck+surgery%5BTitle%5D)+AND+%22The+Laryngoscope%22%5BJournal%5D)">Development of augmented-reality applications in otolaryngology-head and neck surgery.</a> The Laryngoscope. , (2019).</li>
<li>Zhou, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Robot-Assisted+Surgery+for+Mandibular+Angle+Split+Osteotomy+Using+Augmented+Reality%3A+Preliminary+Results+on+Clinical+Animal+Experiment%5BTitle%5D)+AND+%22Aesthetic+Plastic+Surgery%22%5BJournal%5D)">Robot-Assisted Surgery for Mandibular Angle Split Osteotomy Using Augmented Reality: Preliminary Results on Clinical Animal Experiment.</a> Aesthetic Plastic Surgery. 41 (5), 1228-1236 (2017).</li>
<li>Heinrich, F., Joeres, F., Lawonn, K., Hansen, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Comparison+of+Projective+Augmented+Reality+Concepts+to+Support+Medical+Needle+Insertion%5BTitle%5D)+AND+%22IEEE+Transactions+on+Visualization+and+Computer+Graphics%22%5BJournal%5D)">Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion.</a> IEEE Transactions on Visualization and Computer Graphics. 25 (6), 1(2019).</li>
<li>Deng, W., Li, F., Wang, M., Song, Z. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Easy-to-Use+Augmented+Reality+Neuronavigation+Using+a+Wireless+Tablet+PC%5BTitle%5D)+AND+%22Stereotactic+and+Functional+Neurosurgery%22%5BJournal%5D)">Easy-to-Use Augmented Reality Neuronavigation Using a Wireless Tablet PC.</a> Stereotactic and Functional Neurosurgery. 92 (1), 17-24 (2014).</li>
<li>Fan, Z., Weng, Y., Chen, G., Liao, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+interactive+surgical+visualization+system+using+mobile+spatial+information+acquisition+and+autostereoscopic+display%5BTitle%5D)+AND+%22Journal+of+Biomedical+Informatics%22%5BJournal%5D)">3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display.</a> Journal of Biomedical Informatics. 71, 154-164 (2017).</li>
<li>Arnal-Burró, J., Pérez-Mañanes, R., Gallo-del-Valle, E., Igualada-Blazquez, C., Cuervas-Mons, M., Vaquero-Martín, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Three+dimensional-printed+patient-specific+cutting+guides+for+femoral+varization+osteotomy%3A+Do+it+yourself%5BTitle%5D)+AND+%22The+Knee%22%5BJournal%5D)">Three dimensional-printed patient-specific cutting guides for femoral varization osteotomy: Do it yourself.</a> The Knee. 24 (6), 1359-1368 (2017).</li>
<li>Vaquero, J., Arnal, J., Perez-Mañanes, R., Calvo-Haro, J., Chana, F. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((3D+patient-specific+surgical+printing+cutting+blocks+guides+and+spacers+for+open-+wedge+high+tibial+osteotomy+%28HTO%29+-+do+it+yourself%5BTitle%5D)+AND+%22Revue+de+Chirurgie+Orthop%C3%A9dique+et+Traumatologique%22%5BJournal%5D)">3D patient-specific surgical printing cutting blocks guides and spacers for open- wedge high tibial osteotomy (HTO) - do it yourself.</a> Revue de Chirurgie Orthopédique et Traumatologique. 102, Supplement 131(2016).</li>
<li>De La Peña, A., De La Peña-Brambila, J., Pérez-De La Torre, J., Ochoa, M., Gallardo, G. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Low-cost+customized+cranioplasty+using+a+3D+digital+printing+model%3A+a+case+report%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Low-cost customized cranioplasty using a 3D digital printing model: a case report.</a> 3D Printing in Medicine. 4 (1), 1-9 (2018).</li>
<li>Kamio, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Utilizing+a+low-cost+desktop+3D+printer+to+develop+a+%22one-stop+3D+printing+lab%22+for+oral+and+maxillofacial+surgery+and+dentistry+fields%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Utilizing a low-cost desktop 3D printer to develop a "one-stop 3D printing lab" for oral and maxillofacial surgery and dentistry fields.</a> 3D Printing in Medicine. 4 (1), 1-7 (2018).</li>
<li>Punyaratabandhu, T., Liacouras, P., Pairojboriboon, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Using+3D+models+in+orthopedic+oncology%3A+presenting+personalized+advantages+in+surgical+planning+and+intraoperative+outcomes%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Using 3D models in orthopedic oncology: presenting personalized advantages in surgical planning and intraoperative outcomes.</a> 3D Printing in Medicine. 4 (1), 1-13 (2018).</li>
<li>Wake, N., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Patient-specific+3D+printed+and+augmented+reality+kidney+and+prostate+cancer+models%3A+impact+on+patient+education%5BTitle%5D)+AND+%223D+Printing+in+Medicine%22%5BJournal%5D)">Patient-specific 3D printed and augmented reality kidney and prostate cancer models: impact on patient education.</a> 3D Printing in Medicine. 5 (1), 1-8 (2019).</li>
<li>Barber, S. R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+Reality%2C+Surgical+Navigation%2C+and+3D+Printing+for+Transcanal+Endoscopic+Approach+to+the+Petrous+Apex%5BTitle%5D)+AND+%22OTO+Open%3A+The+Official+Open+Access+Journal+of+the+American+Academy+of+Otolaryngology-Head+and+Neck+Surgery+Foundation%22%5BJournal%5D)">Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex.</a> OTO Open: The Official Open Access Journal of the American Academy of Otolaryngology-Head and Neck Surgery Foundation. 2 (4), (2018).</li>
<li>Witowski, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+and+three-dimensional+printing+in+percutaneous+interventions+on+pulmonary+arteries%5BTitle%5D)+AND+%22Quantitative+Imaging+in+Medicine+and+Surgery%22%5BJournal%5D)">Augmented reality and three-dimensional printing in percutaneous interventions on pulmonary arteries.</a> Quantitative Imaging in Medicine and Surgery. 9 (1), (2019).</li>
<li>Moreta-Martínez, R., García-Mato, D., García-Sevilla, M., Pérez-Mañanes, R., Calvo-Haro, J., Pascau, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Augmented+reality+in+computer-assisted+interventions+based+on+patient-specific+3D+printed+reference%5BTitle%5D)+AND+%22Healthcare+Technology+Letters%22%5BJournal%5D)">Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference.</a> Healthcare Technology Letters. , (2018).</li>
</ol>

The authors have nothing to disclose.

AR holds great potential in education, training, and surgical guidance in the medical field. Its combination with 3D printing opens may open new possibilities in clinical applications. This protocol describes a methodology that enables inexperienced users to create a smartphone app combining AR and 3DP for the visualization of anatomical 3D models of patients with 3D-printed reference markers.
In general, one of the most interesting clinical applications of AR and 3DP is to improve patient-to-physician communication by giving the patient a different perspective of the case, improving explanations of specific medical conditions or treatments. Another possible application includes surgical guidance for target localization, in which 3D-printed patient-specific tools (with a reference AR marker attached) can be placed on rigid structures (i.e., bone) and used as a reference for navigation. This application is especially useful for orthopedic and maxillofacial surgical procedures, in which bone tissue surface is easily accessed during surgery.
The protocol starts with section 1, describing the workstation set-up and software tools necessary. Section 2 describes how to use 3D Slicer software to easily segment target anatomies of the patient from any medical imaging modality to obtain 3D models. This step is crucial, as the virtual 3D models created are those displayed in the final AR application.
In section 3, 3D Slicer is used to register the 3D models created in the previous section with an AR marker. During this registration procedure, patient 3D models are efficiently and simply positioned with respect to the AR marker. The position defined in this section will determine the hologram relative position in the final app. It is believed that this solution reduces complexity and multiplies the possible applications. Section 3 describes two different options to define the spatial relationships between the models and AR markers: "visualization" and "registration" mode. The first option, "visualization" mode, allows the 3D models to be positioned anywhere with respect to the marker and displayed as the whole biomodel. This mode provides a realistic, 3D perspective of the patient&#39;s anatomy and allows moving and rotating of the biomodels by moving the tracked AR marker. The second option, "registration" mode, allows attachment and combining of a marker adaptor to any part of the biomodel, offering an automatic registration process. With this option, a small section of the 3D model, including the marker adaptor, can be 3D-printed, and the app can display the rest of the model as a hologram.
Section 4 provides guidelines for the 3D printing process. First, the user can choose between two different markers: the "dual color marker" and "sticker marker". The whole "dual color marker" can be 3D-printed but requires a dual extruder 3D printer. In case this printer is not available, the "sticker marker" is proposed. This is a simpler marker that can be obtained by 3D-printing the cubic structure, then pasting the images of the cube with sticker paper or sticker glue. Furthermore, both markers were designed with extensible sections to perfectly fit in a specific adaptor. Thus, the marker can be reused in several cases.
Section 5 describes the process to create a Unity project for AR using the Vuforia software development kit. This step may be the hardest portion for users with no programming experience, but with these guidelines, it should be easier to obtain the final application that is presented in section 6. The app displays the patient&#39;s virtual models over the smartphone screen when the camera recognizes the 3D-printed marker. In order for the app to detect the 3D marker, a minimum distance of approximately 40 cm or less from the phone to the marker as well as good lighting conditions are required.
The final application of this protocol allows the user to choose the specific biomodels to visualize and in which positions. Addtionally, the app can perform automatic patient-hologram registration using a 3D-printed marker and adaptor attached to the biomodel. This solves the challenge of registering virtual models with the environment in a direct and convenient manner. Moreover, this methodology does not require broad knowledge of medical imaging or software development, does not depend on complex hardware and expensive software, and can be implemented over a short time period. It is expected that this method will help accelerate the adoption of AR and 3DP technologies by medical professionals.

AR and 3DP are technologies that provide increasing numbers of applications in the medical field. In the case of AR, its interaction with virtual 3D models and the real environment benefits physicians in regards to education and training1,2,3, communication and interactions with other physicians4, and guidance during clinical interventions5,6,7,8,9,10. Likewise, 3DP has become a powerful solution for physicians when developing patient-specific customizable tools11,12,13 or creating 3D models of a patient&rsquo;s anatomy, which can help improve preoperative planning and clinical interventions14,15.
Both AR and 3DP technologies help to improve orientation, guidance, and spatial skills in medical procedures; thus, their combination is the next logical step. Previous work has demonstrated that their joint use can increase value in patient education16, facilitating explanations of medical conditions and proposed treatment, optimizing surgical workflow17,18 and improving patient-to-model registration19. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this work is to describe a step-by-step methodology that enables the use of AR and 3DP by inexperienced users without the need for broad technical knowledge.
This protocol describes how to develop an AR smartphone app that allows the superimposing of any patient-based 3D model onto a real-world environment using a 3D-printed marker tracked by the smartphone camera. In addition, an alternative approach is described to provide automatic registration between a 3D-printed biomodel (i.e., a 3D model created from a patient&rsquo;s anatomy) and the projected holograms. The protocol described is entirely based on free and multi-platform software.
In previous work, AR patient-to-image registration has been calculated manually5 with surface recognition algorithms10 or has been unavailable2. These methods have been considered somewhat limited when an accurate registration is required19. To overcome these limitations, this work provides tools to perform accurate and simple patient-to-image registration in AR procedures by combining AR technology and 3DP.
The protocol is generic and can be applied to any medical imaging modality or patient. As an example, a real clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. The first step describes how to easily segment the affected anatomy from computed tomography (CT) medical images to generate 3D virtual models. Afterward, positioning of the 3D models is performed, then the required tools and models are 3D-printed. Finally, the desired AR app is deployed. This app allows for the visualization of patient 3D models overlaid on a smartphone camera in real-time.

<table><tbody><tr><td>3D Printing material: Acrylonitrile Butadiene Styrene (ABS)</td><td></td><td></td><td>Thermoplastic polymer material usually used in domestic 3D printers.</td></tr><tr><td>3D Printing material: Polylactic Acid (PLA)</td><td></td><td></td><td>Bioplastic material usually used in domestic 3D printers.</td></tr><tr><td>3D Slicer</td><td></td><td></td><td>Open-source software platform for medical image informatics, image processing, and three-dimensional visualization</td></tr><tr><td>Android</td><td>Alphabet, Inc.</td><td></td><td>Android is a mobile operating system developed by Google. It is based on a modified version of the Linux kernel and other open source software, and is designed primarily for touchscreen mobile devices such as smartphones and tablets.</td></tr><tr><td>Autodesk Meshmixer</td><td>Autodesk, Inc.</td><td></td><td>Meshmixer is state-of-the-art software for working with triangle meshes. Free software.</td></tr><tr><td>iPhone OS</td><td>Apple, Inc.</td><td></td><td>iPhone OS is a mobile operating system created and developed by Apple Inc. exclusively for its hardware.</td></tr><tr><td>Ultimaker 3 Extended</td><td>Ultimaker BV</td><td></td><td>Fused deposition modeling 3D printer.</td></tr><tr><td>Unity</td><td>Unity Technologies</td><td></td><td>Unity is a real-time development platform to create 3D, 2D VR &amp;amp; AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering &amp;amp; more. Free software.</td></tr><tr><td>Xcode</td><td>Apple, Inc.</td><td></td><td>Xcode is a complete developer toolset for creating apps for Mac, iPhone, iPad, Apple Watch, and Apple TV. Free software.</td></tr></tbody></table>

combining augmented reality and 3d printing to display patient models on a smartphone

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient&rsquo;s anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient&rsquo;s anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

The protocol was applied to data from a patient suffering from distal leg sarcoma in order to visualize the affected anatomical region from a 3D perspective. Using the method described in section 2, the portion of the affected bone (here, the tibia and fibula) and tumor were segmented from the patient&#39;s CT scan. Then, using the segmentation tools from 3D Slicer, two biomodels were created: the bone (section of the tibia and fibula) (Figure 1A) and tumor (Figure 1B).
Next, the two 3D models were positioned virtually with respect to the marker for optimal visualization. Both modes described in section 3 were followed for this example. For visualization mode, the models were centered in the upper face of the marker (Figure 2). For registration mode, the marker adaptor was positioned in the bone (specifically, the tibia [Figure 3]). Then, a small section of the tibia was selected to be 3D-printed with a 3D marker adaptor (Figure 4). An Ultimaker 3 extended 3D printer with PLA material was used to create the 3D-printed markers (Figure 5A, B), marker holder base (Figure 5C) for the "visualization" mode, and section of the tibia for "registration" mode (Figure 5D). Figure 5E shows how the marker was attached to the "visualization" mode 3D-printed base. Figure 5F shows the attachment with the "registration" mode 3D-printed biomodel. Finally, Unity was used to create the app and deploy it on the smartphone.
Figure 6 shows how the app worked for "visualization" mode. The hologram was accurately located in the upper part of the cube as previously defined. Figure 7 shows the application for "registration" mode, in which the app positioned the complete bone model on top of the 3D-printed section. The final visualization of the holograms was clear and realistic, maintained the real sizes of the biomodels, and positioned accurately. When using the smartphone application, the AR marker needs to be visible by the camera for the app to correctly display the holograms. In addition, the light conditions in the scene must be of good quality and constant for proper marker detection. Bad light conditions or reflections on the marker surface hinder the tracking of the AR marker and cause malfunctioning of the app.
The time required to create the app depends on several factors. The duration of section 1 is limited by the download speed. Regarding anatomy segmentation (section 2), factors affecting segmentation time include complexity of the region and medical imaging modality (i.e., CT is easily segmented, while MRI is more difficult). For the representative example of the tibia, approximately 10 min was required to generate both 3D models from the CT scan. Biomodel positioning (section 3) is simple and straightforward. Here, it took approximately 5 min to define the biomodel position with respect to the AR marker. For the 3D printing step, the duration is highly dependent on the selected mode. The "dual color marker" was manufactured at high quality in a period of 5 h and 20 min. The "sticker marker" was manufactured in a period of 1 h and 30 min, plus the time required to paste the stickers. The final step for app development can be time-consuming for those with no previous experience in Unity, but it can be easily completed following the protocol steps. Once the AR markers have been 3D-printed, the development of an entirely new AR app can be performed in less than 1 h. This duration can be further reduced with additional experience.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/60618/60618fig01.jpg" /> 
Figure 1: Representation of 3D models created from a CT image of a patient suffering from distal leg sarcoma. (A) Bone tissue represented in white (tibia and fibula). (B) Tumor represented in red. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/60618/60618fig02.jpg" /> 
Figure 2: Results showing how "visualization" mode in 3D Slicer positions the virtual 3D models of the bone and tumor with respect to 3D-printed marker reference. The patient 3D models (A) are positioned above the upper face of the marker cube (B). <a href="https://www.jove.com/files/ftp_upload/60618/60618fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/60618/60618fig03.jpg" /> 
Figure 3: Results showing how "registration" mode in 3D Slicer positions the virtual 3D models of the bone and tumor (A) with respect to 3D-printed marker reference (B). The marker adaptor is attached to the bone tissue model. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 4" class="xfigimg" src="/files/ftp_upload/60618/60618fig04.jpg" /> 
Figure 4: Small section of the bone tissue and 3D marker adaptor. The two components are combined then 3D-printed. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig04large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 5" class="xfigimg" src="/files/ftp_upload/60618/60618fig05.jpg" /> 
Figure 5: 3D printed tools required for the final application. (A) "Two color cube marker" 3D-printed with two colors of materials. (B) "Sticker cube marker" 3D-printed, with stickers pasted. (C) Marker base cube adaptor. (D) Section of the patient&#39;s bone tissue 3D model and marker cube adaptor. (E) "Sticker cube marker" placed in the marker base cube adaptor. (F) "Two color cube marker" placed in the marker adaptor attached to the patient&#39;s anatomy. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig05large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 6" class="xfigimg" src="/files/ftp_upload/60618/60618fig06.jpg" /> 
Figure 6: App display when using "visualization" mode. The patient&#39;s affected anatomy 3D models are positioned above the upper face of the 3D-printed cube. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig06large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 7" class="xfigimg" src="/files/ftp_upload/60618/60618fig07.jpg" /> 
Figure 7: AR visualization when using "registration" mode. The 3D-printed marker enables registration of the 3D-printed biomodel with the virtual 3D models. <a href="https://www.jove.com/files/ftp_upload/60618/60618fig07large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>

Watch this Scientific Journal Video about Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone at JoVE.com

Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) ...

combining-augmented-reality-3d-printing-to-display-patient-models-on

Universidad Científica del Sur

Research

JoVE Journal

Medicine

18.2K Views.  Universidad Carlos III de Madrid. Presented here is a method to design an augmented reality smartphone application for the visualization of anatomical three-dimensional models of patients using a 3D-printed reference marker.

Video: Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

Scaled Anatomical Model Creation of Biomedical Tomographic Imaging Data and Associated Labels for Subsequent Sub-surface Laser Engraving (SSLE) of Glass Crystals

Biomedical imaging modalities like computed tomography (CT) and magnetic resonance (MR) provide excellent platforms for collecting three-dimensional data sets of patient or specimen anatomy in clinical or preclinical settings. However, the use of a virtual, on-screen display limits the ability of these tomographic images to fully convey the anatomical information embedded within. One solution is to interface a biomedical imaging data set with 3D printing technology to generate a physical replica. Here we detail a complementary method to visualize tomographic imaging data with a hand-held model: Sub Surface Laser Engraving (SSLE) of crystal glass. SSLE offers several unique benefits including: the facile ability to include anatomical labels, as well as a scale bar; streamlined multipart assembly of complex structures in one medium; high resolution in the X, Y, and Z planes; and semi-transparent shells for visualization of internal anatomical substructures. Here we demonstrate the process of SSLE with CT data sets derived from pre-clinical and clinical sources. This protocol will serve as a powerful and inexpensive new tool with which to visualize complex anatomical structures for scientists and students in a number of educational and research settings.

Scaled Anatomical Model Creation of Biomedical Tomographic Imaging Data and Associated Labels for Subsequent Sub-surface Laser Engraving SSLE of Glass Crystals

A methodology is described herein for representing anatomical imaging data within crystals. We create scaled three-dimensional models of biomedical imaging data for use in Sub-Surface Laser Engraving (SSLE) of crystal glass. This tool offers a useful complement to computational display or three-dimensionally printed models used within clinical or educational settings.

Biomedical imaging modalities like computed tomography (CT) and magnetic resonance (MR) provide excellent platforms for collecting three-dimensional data ...

Bioengineering

Voxel Printing Anatomy: Design and Fabrication of Realistic, Presurgical Planning Models through Bitmap Printing

Most applications of 3-dimensional (3D) printing for presurgical planning have been limited to bony structures and simple morphological descriptions of complex organs due to the fundamental limitations in accuracy, quality, and efficiency of the current modeling paradigm. This has largely ignored the soft tissue critical to most surgical specialties where the interior of an object matters and anatomical boundaries transition gradually. Therefore, the needs of the biomedical industry to replicate human tissue, which displays multiple scales of organization and varying material distributions, necessitate new forms of representation. 
Presented here is a novel technique to create 3D models directly from medical images, which are superior in spatial and contrast resolution to current 3D modeling methods and contain previously unachievable spatial fidelity and soft tissue differentiation. Also presented are empirical measurements of novel, additively manufactured composites that span the gamut of material stiffnesses seen in soft biological tissues from MRI and CT. These unique volumetric design and printing methods allow for deterministic and continuous adjustment of material stiffness and color. This capability enables an entirely new application of additive manufacturing to presurgical planning: mechanical realism. As a natural complement to existing models that provide appearance matching, these new models also allow medical professionals to "feel" the spatially varying material properties of a tissue simulant-a critical addition to a field in which tactile sensation plays a key role.

This method demonstrates a voxel-based 3D printing workflow, which prints directly from medical images with exact spatial fidelity and spatial/contrast resolution. This enables the precise, graduated control of material distributions through morphologically complex, graduated materials correlated to radiodensity without loss or alteration of data.

Most applications of 3-dimensional (3D) printing for presurgical planning have been limited to bony structures and simple morphological descriptions of ...

3D Printing of Preclinical X-ray Computed Tomographic Data Sets

Three-dimensional printing allows for the production of highly detailed objects through a process known as additive manufacturing. Traditional, mold-injection methods to create models or parts have several limitations, the most important of which is a difficulty in making highly complex products in a timely, cost-effective manner.1 However, gradual improvements in three-dimensional printing technology have resulted in both high-end and economy instruments that are now available for the facile production of customized models.2 These printers have the ability to extrude high-resolution objects with enough detail to accurately represent in vivo images generated from a preclinical X-ray CT scanner. With proper data collection, surface rendering, and stereolithographic editing, it is now possible and inexpensive to rapidly produce detailed skeletal and soft tissue structures from X-ray CT data. Even in the early stages of development, the anatomical models produced by three-dimensional printing appeal to both educators and researchers who can utilize the technology to improve visualization proficiency. 3, 4 The real benefits of this method result from the tangible experience a researcher can have with data that cannot be adequately conveyed through a computer screen. The translation of pre-clinical 3D data to a physical object that is an exact copy of the test subject is a powerful tool for visualization and communication, especially for relating imaging research to students, or those in other fields. Here, we provide a detailed method for printing plastic models of bone and organ structures derived from X-ray CT scans utilizing an Albira X-ray CT system in conjunction with PMOD, ImageJ, Meshlab, Netfabb, and ReplicatorG software packages.

Using modern plastic extrusion and printing technologies, it is now possible to quickly and inexpensively produce physical models of X-ray CT data taken in a laboratory. The three -dimensional printing of tomographic data is a powerful visualization, research, and educational tool that may now be accessed by the preclinical imaging community.

Three-dimensional printing allows for the production of highly detailed objects through a process known as additive manufacturing.  Traditional, ...

Biology

Multicolor 3D Printing of Complex Intracranial Tumors in Neurosurgery

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. The model can be used for planning and simulating critical steps of a surgical approach. Therefore, it is important that anatomical structures such as blood vessels inside a tumor can be printed to be colored not only on their surface, but throughout their whole volume. During simulation this allows for the removal of certain parts (e.g., with a high-speed drill) and revealing internally located structures of a different color. Thus, diagnostic information from various imaging modalities (e.g., CT, MRI) can be combined in a single compact and tangible object.
However, preparation and printing of such a fully colored anatomical model remains a difficult task. Therefore, a step-by-step guide is provided, demonstrating the fusion of different cross-sectional imaging data sets, segmentation of anatomical structures, and creation of a virtual model. In a second step the virtual model is printed with volumetrically colored anatomical structures using a plaster-based color 3D binder jetting technique. This method allows highly accurate reproduction of patient-specific anatomy as shown in a series of 3D-printed petrous apex chondrosarcomas. Furthermore, the models created can be cut and drilled, revealing internal structures that allow for simulation of surgical procedures.

The protocol describes the fabrication of fully colored three-dimensional prints of patient-specific, anatomical skull models to be used for surgical simulation. The crucial steps of combining different imaging modalities, image segmentation, three-dimensional model extraction, and production of the prints are explained.

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. ...

Digital Hybrid Model Preparation for Virtual Planning of Reconstructive Dentoalveolar Surgical Procedures

Virtual, hybrid three-dimensional (3D) model acquisition is presented in this article, utilizing the sequence of radiographic image segmentation, spatial registration, and free-form surface modeling. Firstly cone-beam computed tomography datasets were reconstructed with a semi-automatic segmentation method. Alveolar bone and teeth are separated into different segments, allowing 3D morphology, and localization of periodontal intrabony defects to be assessed. The severity, extent, and morphology of acute and chronic alveolar ridge defects are validated concerning adjacent teeth. On virtual complex tissue models, positions of dental implants can be planned in 3D. Utilizing spatial registration of IOS and CBCT data and subsequent free-form surface modeling, realistic 3D hybrid models can be acquired, visualizing alveolar bone, teeth, and soft tissues. With the superimposition of IOS and CBCT soft tissue, thickness above the edentulous ridge can be assessed about the underlying bone dimensions; therefore, flap design and surgical flap management can be determined, and occasional complications may be avoided.

A workflow for creating three-dimensional (3D) virtual hybrid models has been designed based on cone-beam computed tomography dataset and intraoral optical scans utilizing radiographic image segmentation methods and free-form surface modeling. Digital models are used for the virtual planning of reconstructive dentoalveolar surgical procedures.

Virtual, hybrid three-dimensional (3D) model acquisition is presented in this article, utilizing the sequence of radiographic image segmentation, spatial ...

The Use of Mixed Reality in Custom-Made Revision Hip Arthroplasty: A First Case Report

The technology of 3D printing and visualization of anatomical structures is rapidly growing in various fields of medicine. A custom-made implant and mixed reality were used to perform complex revision hip arthroplasty in January 2019. The use of mixed reality allowed for a very good visualization of the structures and resulted in precise implant fixation. According to the authors' knowledge, this is the first described case report of the combined use of these two innovations. The diagnosis preceding the qualification for the procedure was the loosening of the left hip's acetabular component. Mixed reality headset and holograms prepared by engineers were used during the surgery. The operation was successful, and it was followed by early verticalization and patient rehabilitation. The team sees opportunities for technology development in joint arthroplasty, trauma, and orthopedic oncology.

A complex revision hip arthroplasty was performed using a custom-made implant and mixed reality technology. According to the authors&#39; knowledge, this is the first report of such procedure described in the literature.

The technology of 3D printing and visualization of anatomical structures is rapidly growing in various fields of medicine. A custom-made implant and mixed ...

Virtual Reality-3D Modeling for Minimally Invasive Neuro-Interventional Planning

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular ...

Pedicle Screw Placement Using an Augmented Reality Head-Mounted Display in a Porcine Model

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) for minimally invasive pedicle screw placement. The cadaveric porcine specimens were placed on a surgical table and draped with sterile covers. The levels of interest were identified using fluoroscopy, and a dynamic reference frame was attached to the spinous process of a vertebra in the region of interest. Cone beam computerized tomography (CBCT) was performed, and a 3D rendering was automatically generated, which was used for the subsequent planning of the pedicle screw placements. Each surgeon was fitted with an HMD that was individually eye-calibrated and connected to the spinal navigation system.
Navigated instruments, tracked by the navigation system and displayed in 2D and 3D in the HMD, were used for 33 pedicle cannulations, each with a diameter of 4.5 mm. Postprocedural CBCT scans were assessed by an independent reviewer to measure the technical (deviation from the planned path) and clinical (Gertzbein grade) accuracy of each cannulation. The navigation time for each cannulation was measured. The technical accuracy was 1.0 mm &#177; 0.5 mm at the entry point and 0.8 mm &#177; 0.1 mm at the target. The angular deviation was 1.5&#176; &#177; 0.6&#176;, and the mean insertion time per cannulation was 141 s &#177; 71 s. The clinical accuracy was 100% according to the Gertzbein grading scale (32 grade 0; 1 grade 1). When used for minimally invasive pedicle cannulations in a porcine model, submillimeter technical accuracy and 100% clinical accuracy could be achieved with this protocol.

The augmented reality head-mounted display, Magic Leap, was used in combination with a conventional navigation system to place pedicle screws in a porcine model by adhering to a novel workflow. With a median insertion time of &lt;2.5 min, submillimeter technical accuracy and 100% clinical accuracy were achieved according to Gertzbein.

This protocol helps assess the accuracy and workflow of an augmented reality (AR) hybrid navigation system using the Magic Leap head-mounted display (HMD) ...

Optimized 3D Reconstruction and Motion Mapping for Remote AR-Assisted Surgery

Automatic Surgery in Transcatheter Aortic Valve Replacement Using Augmented Reality

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter Aortic Valve Replacement (TAVR). TAVR is the alternate medical procedure for open-heart surgery. TAVR replaces the injured valve with the new one using a catheter. In the existing model, remote guidance is given, while the surgery is not automated based on AR. In this article, we deployed a spatially aligned camera that is connected to a motor for the automation of image capture in the surgical environment. The camera tracks the 2D high-resolution image of the patient's heart along with the catheter testbed. These captured images are uploaded using the mobile app to a remote surgeon who is a cardiology expert. This image is utilized for the 3D reconstruction from 2D image tracking. This is viewed in a HoloLens like an emulator in a laptop. The surgeon can remotely inspect the 3D reconstructed images with additional transformation features such as rotation and scaling. These transformation features are enabled through hand gestures. The surgeon's guidance is transmitted to the surgical environment to automate the process in real-time scenarios. The catheter testbed in the surgical field is controlled by the hand gesture guidance of the remote surgeon. The developed prototype model demonstrates the effectiveness of remote surgical guidance through AR.

Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility

This article presents the design and implementation of an automatic surgery module based on augmented reality (AR)- based 3D reconstruction. The system enables remote surgery by allowing surgeons to inspect reconstructed features and replicate surgical hand movements as if they were performing the surgery in close proximity.

Augmented Reality (AR) is in high demand in medical applications. The aim of the paper is to provide automatic surgery using AR for the Transcatheter ...

Mixed Reality Technology and Three-Dimensional Printing in Teaching: Heart Anatomy as an Example

Mixed reality technology and three-dimensional (3D) printing are becoming more and morecommon in the field of medicine. During the COVID-19 pandemic and immediately after the restrictions had beeneased, many innovations were implemented in the teaching of future doctors. There was also interest in immersive techniques and 3D printing technology in anatomy teaching. However, these are not common implementations. In 2023, 3D prints and holograms in mixed reality technology were prepared for classes focused on the structure of the heart. They were used to teach students, who, with the support of engineers, could learn about the detailed structure of the heart and familiarize themselves with the new technologies that support the traditional model of learning on human cadavers. Students findthis possibility to be highly valuable. The article presents the process of preparing materials for classes and further implementation possibilities. The authors see an opportunity for the development of the presented technologies in students'teaching at various levels of education and the justification for increasingly widespread implementation.

Here, we describe a protocol for constructing a heart model from scratch based on computed tomography and present it to medical students using three-dimensional (3D) printing and mixed reality technology to learn anatomy.

Mixed reality technology and three-dimensional (3D) printing are becoming more and morecommon in the field of medicine. During the COVID-19 pandemic and ...