Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone

This protocol gives unexperienced users the tools to bring augmented reality solutions into the medical field through the building of their own smartphone app. This method combines augmented reality and 3D printing allowing a quick and easy visualization of 3D models created from basing data on top of a 3D printed reference marker. This technique can be applied to any medical scenario in which a 3D printed reference can be rigidly attached to the patient.

The visualization of visual 3D models created from basing data localization that is targeted and can also improve patient This protocol has been specifically designed for users with no prior knowledge of medical imaging or software development to aid in the use of augmented reality in the medical field. To create 3D models of a patient's anatomy, first track the medical image file into the 3D Slicer software window and click OK.To segment the anatomy of the patient, go to the segment editor module in 3D Slicer. A segmentation item will be created automatically upon entering the module.

Select the desired volume in the master volume section and right-click below the image to select add to create a segment. In the effects panel, select the most convenient tool for the target and segment the medical image of the patient. To export the segmentation in a 3D model file format, open the segmentations module.

In export/import models and labelmaps, select export and models and click export to create the 3D model from the segmented area. Click save and select the elements to be saved. Then change the file format of the 3D model to OBJ.

The segmentation can be repeated to create additional 3D models of different anatomical regions. For positioning of a 3D patient model at any position with respect to the augmented reality marker, open the AR Health model position module and select visualization mode. Click load marker model to load the marker for this option, the ellipsis button to select the path of the saved 3D model, and load model to load the model in 3D Slicer.

Click finish and center to center all of the models within the marker. Use the slide bars to adjust the position, orientation and scaling of the 3D models with respect to the marker as desired. Then select the path to store the files and click save models to save the models at this position.

To combine the augmented reality marker with a 3D biomodel at any desired position, open the AR Health model position module and in the initialization section select the registration mode. Click load marker model to load the marker for this option and move the 3D models until they intersect with the supporting structure of the cube marker, modifying the height of the marker base as necessary. To save the model at this position, select the path to store the files and click save models.

If the anatomy model is too large, load the biomodel and supporting structure of the cube marker model in the mesh mixer software. Select both models in the object browser window to combine the models and use the plain cut tool from the edit menu to remove any unwanted sections of the model that will not be 3D printed. To save the model to be 3D printed, select file and export and select the desired format.

To 3D print the physical models required for the final augmented reality application, in the 3D printing software, select a white colored material for the TwoColorCuebMarker_white. obj file and a black colored material for the TwoColorCubeMarker_white. obj file.

Then use a dual extruder 3D printer to 3D print the cubic marker in black and white in high quality mode with a small layer height. To design a smartphone app in Unity Engine that includes the 3D models, open Vuforia Developer and create an account. Select get development key to obtain a free development license key and in the license manager menu select and copy the key.

To set up the smartphone, in the Unity version 2019 application, under build settings in the file menu select the appropriate platform for the device. To enable Vuforia into the project, select edit, project setting, player settings, and XR settings and check the box labeled Vuforia Augmented Reality Support. To create an augmented reality camera, select menu bar, game object, Vuforia Engine, and AR camera and import the Vuforia components when prompted.

To add the Vuforia license key into Vuforia configuration settings, select the resources folder, click Vuforia configuration, and paste the license key into the app license key section. Import the Vuforia target file containing the files that Vuforia requires to detect the markers into Unity and select menu bar, game object, Vuforia Engine, and multi image to create a Vuforia multi target. Click the multi target to select the marker type that will be used for detection and in the database option under multi target behavior, select ARHealth_3dPrintedCube_30x30x30.

In the multi target option under multi target behavior, select either TwoColorCubeMarker or StickerCubeMarker depending on the marker. Drag the 3D models into the models folder and drag the folder under the multi target item. The models should become visible in the Unity 3D view scene.

To change the colors of the 3D models, create new materials and assign the new materials to the models. If a webcam is available, click play to test the application on the computer. If the marker is visible to the webcam, it should be detected and the 3D models should appear in the scene.

If an Android smartphone will be used for the app development, select build settings in Unity and select the plugged phone from the list. Then save the file with a apk extension and allow the process to finish. If the app will be deployed on an iOS device, select file in build settings and save the file.

To visualize the app, open the app on the smartphone and use the smartphone camera to look at the marker from within the app at a minimum distance of 40 centimeters. Once the app detects the marker, the previously created 3D models should appear on the smartphone screen at the exact location defined during the procedure. Using the method as demonstrated, this portion of the affected tibia and fibula from a patient suffering from distal leg sarcoma and tumor were segmented from the patient's CT scan.

Using the segmentation tools, one biomodel was created for the bone and one biomodel was created for the tumor. For visualization mode, the models were centered in the upper phase of the marker. For registration mode, the marker adapter was positioned in the tibia and a small section of the tibia was selected to be 3D printed with a 3D marker adapter.

Polylactic acid can be used to create the 3D printed markers, marker holder bases and bone sections as demonstrated. Here a marker is attached to a visualization mode 3D printed base and here the attachment is shown with the registration mode 3D printed biomodel. This representation shows how the app works in visualization mode with the hologram accurately located in the upper part of the cube as previously defined.

In registration mode, the complete bone model can be positioned on top of the 3D printed section with a clear and realistic visualization of the marker at the bone site. To use augmented reality to visualize important patient information, you will need several software tools that are freely available as well as access to a 3D printer and smartphone. This procedure can be applied to any model obtained from medical imaging.

Its use can be extended to other interventions such as radiation therapy positioning of needle insertion. We are now extending the applications of this development into new clinical areas including myofascial or orthopedic surgery. Our initial research are promising and the surgeon feedback is very positive.