application of the differential transformation for cross-device co-registration of two preclinical imaging modalities

Effective Co-Registration of Two Preclinical Imaging Modalities Across Devices

Different preclinical imaging modalities have distinct advantages and disadvantages. For instance, X-ray computed tomography (CT) is well-suited for examining anatomical structures with different radio densities, such as bones and lungs. It is widely used due to its rapid acquisition speed, high three-dimensional resolution, relative ease of image assessment, and versatility with or without contrast agents1,2,3. Magnetic resonance imaging (MRI) provides the most versatile soft tissue contrast without ionizing radiation4. On the other hand, tracer-based modalities like positron emission tomography (PET), single-photon emission computed tomography (SPECT), fluorescence-mediated tomography (FMT), and magnetic particle imaging (MPI) are established tools for quantitatively assessing molecular processes, metabolism, and the biodistribution of radiolabeled diagnostic or therapeutic compounds with high sensitivity. However, they lack resolution and anatomical information5,6. Therefore, more anatomy-oriented modalities are typically paired with highly sensitive ones having their strength in tracer detection7. These combinations make the quantification of tracer concentrations within a specific region of interest possible8,9. For combined imaging devices, modality co-registration is usually a built-in feature. However, it is also useful to co-register scans from different devices, e.g., if the devices were purchased separately or if a hybrid device is not available.
This article focuses on cross-modality fusion in small animal imaging, which is essential for basic research and drug development. A previous study10 points out that this can be achieved with feature recognition, contour mapping, or fiducials markers (fiducials). Fiducials are reference points for accurately aligning and correlating images from different imaging modalities. In special cases, fiducials can even be dots of Chinese ink on the skin of nude mice11; however, often, an imaging cartridge with built-in fiducial markers is used. While this is a robust and well-developed method10, using it for every scan presents practical problems. MRI-detectable fiducials are often liquid-based and tend to dry out during storage. PET requires radioactive markers, the signal of which decays according to the emitter&#39;s half-life period, which is usually short for biomedical applications, necessitating preparation shortly before the scan. Other issues, such as the mismatch in the dynamic range of the signal from the fiducial marker and the examined object, strongly impact in vivo imaging. The wide range of dynamic contrast requires frequent adaptation of the marker signal strength to the object being examined. Consequently, while a weak marker signal may not be detected in the analysis, a strong marker signal may create artifacts that impair image quality. Additionally, to consistently include the markers, the field of view must be unnecessarily large for many applications, potentially leading to higher radiation exposure, larger data volumes, longer scan times, and, in some cases, lower resolution. This may affect the health of laboratory animals and the quality of the generated data.
Transformation and differential transformation 
An image dataset consists of voxel data and metadata. Each voxel is associated with an intensity value (Figure 1A). The metadata includes a transformation specifying the dataset placement in the imaging device&#39;s coordinate system (Figure 1B) and the voxel size used to scale the coordinate system. Additional information, such as device type or scan date, can be optionally stored in the metadata. The mentioned transformation is mathematically called a rigid body transformation. Rigid body transformations are used to change the orientation or position of objects in an image or geometric space while preserving the distance between each pair of points, meaning the transformed object retains its size and shape while being rotated and translated in space. Any series of such transformations can be described as a single transformation consisting of a rotation followed by a translation. The formula used by the software to move from the data coordinate to the metric target coordinate is shown in Figure 1C, where R is an orthonormal rotation matrix, d and v are voxel indices and sizes, and t is a 3 x 1 translation vector12. The rotation is detailed in Figure 1D.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/65701/65701fig01.jpg" /> 
Figure 1: 2D representation of an image dataset structure and placement in a global coordinate system. (A) An image dataset consists of voxel data and metadata. The transformation specifying the placement and voxel size are essential metadata components. (B) The image is rendered into the device&#39;s coordinate system. The necessary transformation to place the object consists of a rotation (blue) followed by a translation (green). (C) To move from the data coordinate to the target coordinate, the software uses this formula where R is an orthonormal rotation matrix, d, and v are voxel indices and sizes, and t is a 3 x 1 translation vector. (D) A rotation matrix (blue in plane A) represents the linear transformation of rotating points. Multiplying a point&#39;s coordinates by this matrix results in the new, rotated coordinates. <a href="https://www.jove.com/files/ftp_upload/65701/65701fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
A differential transformation is a rigid body transformation converting coordinates from one coordinate system to another, e.g., from PET to X-ray microtomography (&#181;CT), and it can be computed using fiducial markers. At least three common points - the fiducials - are selected in both coordinate systems. From their coordinates, a mathematical transformation can be derived that converts the coordinates. The software uses the least squares method, which provides a best-fit solution to a system of equations with errors or noise in the measured data. This is called the Procrustes Problem13 and is solved using singular value decomposition. The method is reliable and robust because it leads to a unique and well-defined solution (if at least three non-collinear markers are given). Six free parameters are calculated: three for translation and three for rotation. In the following, we will use the term transformation matrix even though it technically consists of a rotation matrix and translation vector.
Each imaging device has its own coordinate system, and the software calculates a differential transformation to align them. Figure 2A,B describe how the differential transformation is determined, while Figure 2C,D describe how it is applied. The images of both modalities can have different dimensions and keep them in the process, as is shown in the example image with the fusion of CT and PET in Figure 2E.
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/65701/65701fig02.jpg" /> 
Figure 2: Differential transformation. (A-D) Simplified to 2D. While applicable to other modalities, it is assumed that the modalities are CT and PET for this example. (A,C) A CT image with a red bounding box is positioned in the coordinate system. Applied to the same coordinate system, the PET image with a yellow bounding box is positioned deviating.(B) Using fiducial markers that can be located in both CT and PET, a differential transformation T can be determined. This is symbolized by the arrow. The differential transformation matrix is stored. (D) The previously saved differential transformation matrix T can then be applied to each PET image. This results in a new transformation that replaces the original transformation in the metadata. (E) A CT image fused with a PET image. The transformations in the metadata of both images refer to the same coordinate system. <a href="https://www.jove.com/files/ftp_upload/65701/65701fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
Method and requirements 
For the presented method, a phantom containing markers visible in both modalities is scanned in both devices. It is then sufficient to mark these fiducials in the proposed software to calculate a differential transformation between the two modalities. The differential transformation must be created individually for each pair of devices. It can be saved and later applied to any new image, thereby eliminating the need for fiducial markers in subsequent scans. The final placement of the image in the coordinate system of another device can again be described as a transformation and be stored in the metadata of the image, replacing the original transformation there.
Four requirements for this method can be formulated: (1) Multimodal phantom: A phantom containing markers visible in both modalities must be available. A large selection of phantoms is commercially obtainable, and the use of 3D printing for phantom construction has been widely described14, even including the incorporation of radioisitopes15. The phantoms utilized in the following examples are listed in the Table of Materials. At least three non-collinear points are required16. The markers might be cavities that can be filled with an appropriate tracer, small objects made of a material that is easily detectable in each modality, or simply holes, cuts, or edges in the phantom itself as long as they can be identified in both modalities. (2) Multimodal carrier: A carrier, such as a mouse bed, is needed that can be fixed in a reproducible position in both devices. Ideally, it should not be possible to use it in a reversed position to avoid errors. The carrier is particularly important for in vivo imaging because it is needed to transport a sedated animal from one imaging device to the other without changing its position. Based on our experience, sedated mice are more likely to change their position in a flat mouse bed compared to a concave-shaped one. Additionally, a custom 3D-printed jig for holding the mouse&#39;s tibia to minimize motion has been previously suggested17. (3) Self-consistency: Each imaging device must provide the rotation and translation of the reconstructed volume in its reference frame in a reproducible and coherent manner. This also means that a coordinate system for the entire device is preserved when only a small region is scanned. It is part of the protocol to test an imaging device for its self-consistency. (4) Software support: The proposed software must be able to interpret the metadata (voxel size, translation, orientation) stored with the reconstructed volume provided by the device. The volume can be in DICOM, NIfTI, Analyze, or GFF file format. For an overview of various file formats, see Yamoah et al.12.
While the co-registration of two modalities is described, the procedure is also applicable to three or more modalities, for example, by co-registering two modalities to one reference modality.

Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities

Multimodal Cross-Device and Marker-Free Co-Registration of Preclinical Imaging Modalities

We develop software tools for preclinical imaging, including image reconstruction, fusion, segmentation, and quantification. Our software is meant to be user-friendly, efficient, and compatible with any small animal imaging device to enable reproducible quantitative analysis of the acquired multimodal image data. Automation is an important topic in our field because manual analysis is often too time-consuming, error-prone, and hardly reproducible.Software-based automation can make the analysis efficient and reproducible, but the challenge is in making the automation robust enough. Small animal imaging devices provide volumetric image data of anesthetized laboratory mice and rats, and allow non-invasive assessment of pathological and physiological changes. Integrated multimodal devices combine the strengths of multiple modalities, such as Micro-CT and PET.It is also possible to combine two separate devices. We developed software for the automated fusion of special modalities, such as Micro-CT and FMT. We also provided efficient software for interactive image fusion, and defined a new file format to store the image data, considering aspects from data curation.Last year, we published an approach for retrospective respiratory gating under high throughput settings. Our protocol resides in a transformation matrix, which is generated using a calibration step that involves visible structures such as fiducial markers. Then, the image fusion can be automated using this transformation matrix, which eliminates the need for markers.

Application of the Differential Transformation for Cross-Device Co-Registration of Two Preclinical Imaging Modalities

To begin, launch the analysis software. Import the CT file as an underlay by clicking on file underlay and load underlay. In the ensuing dialogue, select the CT image file to open it.Load the PET file as an overlay by sequentially selecting file overlay and load overlay. In the dialogue that appears, choose the PET image file. To enable the display of both layers, go to menu view followed by layer visibility and check both underlay and overlay.To load and apply the previously saved differential transformation matrix, click on fusion overlay transformation and then load and apply transform. Then select the file containing the matrix from the calibration process to open it.

application-differential-transformation-for-cross-device-co

Research

JoVE Journal

Bioengineering

164 Views. To begin, launch the analysis software. Import the CT file as an underlay by clicking on file underlay and load underlay. In the ensuing dialogue, select the CT image file to open it.Load the PET file as an overlay by sequentially selecting file overlay and load overlay. In the dialogue that appears, choose the PET image file. To enable the display of both layers, go to menu view followed by layer visibility and check both underlay and overlay.To load and apply the previously sa...

Video: Application of the Differential Transformation for Cross-Device Co-Registration of Two Preclinical Imaging Modalities

Integrated preclinical multimodal imaging systems, such as X-ray computed tomography (CT) combined with positron emission tomography (PET) or magnetic resonance imaging (MRI) combined with PET, are widely available and typically provide robustly co-registered volumes. However, separate devices are often needed to combine a standalone MRI with an existing PET-CT or to incorporate additional data from optical tomography or high-resolution X-ray microtomography. This necessitates image co-registration, which involves complex aspects such as multimodal mouse bed design, fiducial marker inclusion, image reconstruction, and software-based image fusion. Fiducial markers often pose problems for in vivo data due to dynamic range issues, limitations on the imaging field of view, difficulties in marker placement, or marker signal loss over time (e.g., from drying or decay). These challenges must be understood and addressed by each research group requiring image co-registration, resulting in repeated efforts, as the relevant details are rarely described in existing publications.
This protocol outlines a general workflow that overcomes these issues. Although a differential transformation is initially created using fiducial markers or visual structures, such markers are not required in production scans. The requirements for the volume data and the metadata generated by the reconstruction software are detailed. The discussion covers achieving and verifying requirements separately for each modality. A phantom-based approach is described to generate a differential transformation between the coordinate systems of two imaging modalities. This method showcases how to co-register production scans without fiducial markers. Each step is illustrated using available software, with recommendations for commercially available phantoms. The feasibility of this approach with different combinations of imaging modalities installed at various sites is showcased.

The combination of multiple imaging modalities is often necessary to gain a comprehensive understanding of pathophysiology. This approach utilizes phantoms to generate a differential transformation between the coordinate systems of two modalities, which is then applied for co-registration. This method eliminates the need for fiducials in production scans.

Integrated preclinical multimodal imaging systems, such as X-ray computed tomography (CT) combined with positron emission tomography (PET) or magnetic ...

Calculation of the Differential Transformation for Cross-Device Co-Registration of Two Preclinical Imaging Modalities

Begin by loading the images of both modalities. To do so, open the analysis software. To load the CT image A as underlay, go to File followed by Underlay and click Load underlay.In the dialogue that appears, choose and open the image file. Similarly, to load the PET image A as an overlay, click File, Overlay, Load overlay and choose the image file to open it. For marker-based fusion, hide the underlay by navigating to Menu, View and Layer visibility.Then uncheck Overlay and keep Underlay. Switch the selected layer to the underlay if required. To show multiple slice views, press Alt plus A for the axial, Alt plus S for the sagittal, and Alt plus C for the coronal views.If necessary, press Control plus W and adjust the left and right vertical bars in the windowing dialogue to better see the fiducials. Click on OK to close the dialogue. Click the marker symbol on the vertical toolbar on the left side to activate the mouse action mode, Create Marker.Navigate to one fiducial of the phantom. Place the mouse pointer in its center and left click. Select OK to save the marker.Place the mouse pointer over the view of a plane and use Alt plus mouse wheel to slice through the planes. Adjust the viewing settings by navigating to Menu, View, Layer visibility, and check Overlay to show the overlay. Switch the selected layer to Overlay.Adjust the windowing as demonstrated if necessary. Navigate to each fiducial of the phantom. Place the mouse pointer in the center of the fiducial and left click.Click on OK to save the marker. Place the mouse pointer over the view of a plane and use Alt plus mouse wheel to slice through the planes. Align the markers of Underlay and Overlay by navigating to Fusion, followed by Register overlay to underlay and selecting Compute rotation and translation.Note this measurement and click OK.Check the result of the alignment. As demonstrated previously, activate the views of underlay and overlay layers. Click on the symbol on the left side vertical toolbar to activate the mouse mode interactive image fusion.Check Overlay, uncheck Underlay and segmentation. Perform translation by placing the mouse pointer near the center of a view. Hold the left mouse button and move the mouse to move the overlay.Perform rotation by placing the pointer near the edge of a view, axial, coronal, or sagittal. Hold the left mouse button and move the mouse to rotate the overlay. Keep performing rotations and translations on the different views until the underlay and overlay are aligned as best as possible.Create and save the differential transformation by navigating Fusion, Overlay transformation, and select Create and save differential transformation. In the first dialogue, select the original overlay file and click on Open. In the second dialogue, enter a file name for the differential transformation and press Save.