This work was supported by the National Research Foundation of Korea (NRF), grant funded by the Ministry of Science and ICT of Korea (MSIT) (number: 2020R1C1C1010079). For the CMC-ORBIT dataset, the central Institutional Review Board (IRB) of the Catholic Medical Center provided approval (XC19REGI0076).&#160;This work was supported by 2022 Hongik University Research Fund.

The present work was performed with the approval of the Institutional Review Board (IRB) of the Catholic Medical Center, and the privacy, confidentiality, and security of the health information were protected. The orbital CT data were collected (from de-identified human subjects) from hospitals affiliated with the College of Medicine, the Catholic University of Korea (CMC; Seoul St. Mary&#39;s Hospital, Yeouido St. Mary&#39;s Hospital, Daejeon St. Mary&#39;s Hospital, and St. Vincent Hospital). The orbital CT scans were obtained from January 2016 to December 2020. The dataset contained 46 orbital CT scans from Korean men and women ranging in age from 20 years to 60 years. The runtime environment (RTE) is summarized in Supplementary Table 1.
1. Masking the eyeball, optic nerve, and extraocular muscles on the orbital CT scans
<ol>
	<li>Run the masking software program. 
	NOTE: The masking software program (MediLabel, see the Table of Materials) is a medical image labeling software program for segmentation, which requires few clicks and has high speed.</li>
	<li>Load the orbital CT by clicking on the open file icon and selecting the target CT file. Then, the CT scans are shown on the screen.</li>
	<li>Mask the eyeball, optic nerve, and extraocular muscles using super pixels.
	<ol>
		<li>Run the SmartPencil by clicking on the SmartPencil wizard in MediLabel (Video 1).</li>
		<li>Control the resolution of the super pixel map if necessary (e.g., 100, 500, 1,000, and 2,000 super pixels).</li>
		<li>Click on the cluster of super pixels of the eyeball, optic nerve, and extraocular muscles on the super pixel map, where pixels of similar image intensity values are clustered.</li>
	</ol>
	</li>
	<li>Refine the masks with the autocorrection functions in MediLabel.
	<ol>
		<li>Click on the SmartFill wizard after masking some of the super pixels on the slices (Video 2).</li>
		<li>Click on the AutoCorrection icon, and ensure that the corrected mask labels are computed (Video 3).</li>
	</ol>
	</li>
	<li>Repeat step 1.3 and step 1.4 until the refinement of the masking is complete.</li>
	<li>Save the masked images.</li>
</ol>
2. Pre-processing: Window clipping and cropping the VOIs
<ol>
	<li>Extract the VOIs with preprocessing_multilabel.py (the file is downloadable from GitHub).
	<ol>
		<li>Run preprocessing_multilabel.py.</li>
		<li>Check the scans and masks, which are cropped and saved in the VOIs folder.</li>
	</ol>
	</li>
	<li>Transform the VOIs to the set of three sequential CT slices for the input to SEQ-UNET with builder_multilabel.py (the file is downloadable from GitHub).
	<ol>
		<li>Run sequence_builder_multilabel.py.</li>
		<li>Ensure that the slices and masks are resized to 64 pixels by 64 pixels during the transformation.</li>
		<li>During the transformation, perform clipping with the 48 HU clipping level and the 400 HU window.</li>
		<li>Check the saved transformed CT scans (nii file) and masks (nii file) in the scan folder and the mask folder under pre-processed folders, respectively.</li>
	</ol>
	</li>
</ol>
3. Four cross-validations of the orbital segmentation model
<ol>
	<li>Build the model following the steps below.
	<ol>
		<li>Run main.py.</li>
		<li>When running main.py, give the fold number of the four cross-validations by &#34;-fold num x&#34;, where x is 0, 1, 2, or 3.</li>
		<li>When running main.py, use the epoch, which is the number of training iterations, as an option, such as &#34;-epoch x&#34;, where x is the epoch number. The default number is 500.</li>
		<li>When running main.py, set the batch size, which is the number of training samples in a single training session. The default number is 32.</li>
		<li>In main.py, load the CT scans and masks, and initialize the SEQ-UNET with the pre-trained parameters using the LIDC-IDRI dataset (downloadable from the cancer imaging archive).</li>
		<li>In main.py, perform the testing of the model after training. Calculate the evaluation metrics, dice score, and volume similarity, and save them in the metrics folder.</li>
		<li>Check the results in the segmented folder.</li>
	</ol>
	</li>
</ol>

<ol>
	<li>Li, Z., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+learning-based+CT+radiomics+for+feature+representation+and+analysis+of+aging+characteristics+of+Asian+bony+orbit.">Deep learning-based CT radiomics for feature representation and analysis of aging characteristics of Asian bony orbit.</a> Journal of Craniofacial Surgery. 33 (1), 312-318 (2022).</li><li>Hamwood, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+deep+learning+method+for+automatic+segmentation+of+the+bony+orbit+in+MRI+and+CT+images.">A deep learning method for automatic segmentation of the bony orbit in MRI and CT images.</a> Scientific Reports. 11, 1-12 (2021).</li><li>Kim, K. S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Schwannoma+of+the+orbit.">Schwannoma of the orbit.</a> Archives of Craniofacial Surgery. 16 (2), 67-72 (2015).</li><li>Baur, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autoencoders+for+unsupervised+anomaly+segmentation+in+brain+MR+images:+A+comparative+study.">Autoencoders for unsupervised anomaly segmentation in brain MR images: A comparative study.</a> Medical Image Analysis. 69, 101952 (2021).</li><li>Tr&#233;meau, A., Colantoni, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Regions+adjacency+graph+applied+to+color+image+segmentation.">Regions adjacency graph applied to color image segmentation.</a> IEEE Transactions on Image Processing. 9 (4), 735-744 (2000).</li><li>Boykov, Y. Y., Jolly, M. -. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interactive+graph+cuts+for+optimal+boundary+&amp;+region+segmentation+of+objects+in+ND+images.">Interactive graph cuts for optimal boundary &amp; region segmentation of objects in ND images.</a> Proceedings of Eighth IEEE International Conference on Computer Vision. International Conference on Computer Vision. 1, 105-122 (2001).</li><li>Rother, C., Kolmogorov, V., Blake, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=&amp;#34;GrabCut&amp;#34;+interactive+foreground+extraction+using+iterated+graph+cuts.">&amp;#34;GrabCut&amp;#34; interactive foreground extraction using iterated graph cuts.</a> ACM Transactions on Graphics. 23 (3), 309-314 (2004).</li><li>Suh, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Supervised+segmentation+with+domain+adaptation+for+small+sampled+orbital+CT+images.">Supervised segmentation with domain adaptation for small sampled orbital CT images.</a> Journal of Computational Design and Engineering. 9 (2), 783-792 (2022).</li><li>Ronneberger, O., Fischer, P., Brox, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=U-net:+Convolutional+networks+for+biomedical+image+segmentation.">U-net: Convolutional networks for biomedical image segmentation.</a> International Conference on Medical Image Computing and Computer-Assisted Intervention.Medical Image Computing and Computer-Assisted Intervention - MICCAI. , 234-241 (2015).</li><li>Qamar, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+variant+form+of+3D-UNet+for+infant+brain+segmentation.">A variant form of 3D-UNet for infant brain segmentation.</a> Future Generation Computer Systems. 108, 613-623 (2020).</li><li>Nguyen, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Ocular+structures+segmentation+from+multi-sequences+MRI+using+3D+UNet+with+fully+connected+CRFS.">Ocular structures segmentation from multi-sequences MRI using 3D UNet with fully connected CRFS.</a> Computational Pathology and Ophthalmic Medical Image Analysis. , 167-175 (2018).</li><li>Liu, Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bidirectional-convolutional+LSTM+based+spectral-spatial+feature+learning+for+hyperspectral+image+classification.">Bidirectional-convolutional LSTM based spectral-spatial feature learning for hyperspectral image classification.</a> Remote Sensing. 9 (12), 1330 (2017).</li><li>Yeghiazaryan, V., Voiculescu, I. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Family+of+boundary+overlap+metrics+for+the+evaluation+of+medical+image+segmentation.">Family of boundary overlap metrics for the evaluation of medical image segmentation.</a> Journal of Medical Imaging. 5 (1), 015006 (2018).</li><li>Zhang, G., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparable+performance+of+deep+learning-based+to+manual-based+tumor+segmentation+in+KRAS/NRAS/BRAF+mutation+prediction+with+MR-based+radiomics+in+rectal+cancer.">Comparable performance of deep learning-based to manual-based tumor segmentation in KRAS/NRAS/BRAF mutation prediction with MR-based radiomics in rectal cancer.</a> Frontiers in Oncology. 11, 696706 (2021).</li><li>Christopher, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Performance+of+deep+learning+architectures+and+transfer+learning+for+detecting+glaucomatous+optic+neuropathy+in+fundus+photographs.">Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs.</a> Scientific Reports. 8, 16685 (2018).</li></ol>

The authors declare no conflicts of interest.

Deep learning-based medical image analysis is widely used for disease detection. In the ophthalmology domain, detection and segmentation models are used in diabetic retinopathy, glaucoma, age-related macular degeneration, and retinopathy of prematurity. However, other rare diseases apart from those in ophthalmology have not been studied due to the limited access to large open public datasets for deep learning analysis. When applying this method in situations when no public dataset is available, the masking step, which is a labor-intensive and time-consuming task, is unavoidable. However, the proposed masking step (protocol section, step 1) helps to generate masking with high accuracy within a short time. Using super pixels and neural network-based filling, which cluster pixels that are similar in low-level image properties, clinicians can label the masks by clicking the groups of pixels instead of pointing out the specific pixels. Also, the automatic correction functions help refine the mask processes. This method's efficiency and effectiveness will help generate more masked images in medical research.
Among the many possibilities in pre-processing, extracting VOIs and window clipping are effective methods. Here, extracting VOIs and window clipping are introduced in step 2 of the protocol. When the clinicians prepare the dataset, extracting the VOI from the given dataset is the most important step in the process because most segmentation cases focus on small and specific regions in the whole medical image. Regarding the VOIs, the regions of the eyeball, optic nerve, and extraocular muscles are cropped based on the location, but more effective methods for extracting VOIs have the potential to improve segmentation performance14. 
For the segmentation, SEQ-UNET is employed in the study. The 3D medical images have large volumes, so deep neural network models require large memory capacities. In SEQ-UNET, the segmentation model is implemented with a small number of slices to reduce the required memory size without losing the features of the 3D information. 
The model was trained with 46 VOIs, which is not a large number for model training. Due to the small number of training datasets, the performance of optic nerve and extraocular muscle segmentation is limited. Transfer learning15 and domain adaptation8 could provide a solution for improving the segmentation performance.
The whole segmentation process introduced here is not limited to orbital CT segmentation. The efficient labeling method helps create a new medical image dataset for when the application domain is unique to the research area. The python codes of GitHub concerning the pre-processing and segmentation modeling can be applied to other domains with the modification of the cropping region, the window clipping level, and the model hyper-parameters, such as the number of sequential slices, the U-Net architectures, and so on.

The orbit is a small and complicated space of approximately 30.1 cm3 that contains important structures such as the eyeball, nerves, extraocular muscles, supportive tissues, and vessels for vision and eyeball movements1. Orbital tumors are abnormal tissue growths in the orbit, and some of them threaten patients&#39; vision or eyeball movement, which may lead to fatal dysfunction. To conserve patients&#39; visual function, clinicians must decide on the treatment modalities based on the tumor characteristics, and a surgical biopsy is generally inevitable. This compact and crowded area often makes it challenging for clinicians to perform a biopsy without damaging the normal structure. The deep learning-based image analysis of pathology for determining the condition of the orbit could help in avoiding unnecessary or avoidable injury to the orbital tissues during biopsy2. One method of image analysis for orbital tumors is tumor detection and segmentation. However, the collection of large amounts of data for CT images containing orbital tumors is limited due to their low incidence3. The other efficient method for computational tumor diagnosis4 involves comparing the tumor to the normal structures of the orbit. The number of orbital CT images in normal structures is relatively larger than that in tumors. Therefore, the segmentation of normal orbital structures is the first step to achieve this goal.
This study presents the entire process of deep learning-based orbital structure segmentation, including the data collection, pre-processing, and subsequent modeling. The study is intended to be a resource for clinicians interested in using the current method to efficiently generate a masked dataset and for ophthalmologists requiring information about pre-processing and modeling for orbital CT images.This article presents a new method for orbital structure segmentation and sequential U-Net, a sequential 2D segmentation model based on a representative deep-learning solution in U-Net for medical image segmentation. The protocol describes the detailed procedure of orbit segmentation, including (1) how to use a masking tool for the ground truth of orbit structure segmentation, (2) the steps required for the pre-processing of the orbital images, and (3) how to train the segmentation model and evaluate the segmentation performance.
For supervised learning, four experienced ophthalmologists who had been board certified for over 5 years manually annotated the masks of the eyeball, optic nerve, and extraocular muscles. All the ophthalmologists used the masking software program (MediLabel, see the Table of Materials), which uses super-resolution for efficient masking on CT scans. The masking software has the following semi-automatic features: (1) SmartPencil, which generates super pixel map clusters with similar values of image intensity5; (2) SmartFill, which generates segmentation masks by computing the energy function of the ongoing foreground and background6,7; and (3) AutoCorrection, which makes the borders of the segmentation masks clean and consistent with the original image. Example images of the semi-automatic features are shown in Figure 1. The detailed steps of manual masking are provided in the protocol section (step 1).
The next step is the pre-processing of the orbital CT scans. To obtain the orbital volumes of interest (VOIs), the areas of the orbit where the eyeball, muscle, and nerve are located in normal conditions are identified, and these areas are cropped. The dataset has a high resolution, with &#60;1 mm in-plane voxel resolution and slice thickness, so the interpolation process is skipped. Instead, window clipping is conducted at the 48 HU clipping level and the 400 HU window. After the cropping and window clipping, three serial slices of the orbit VOIs are generated for the segmentation model input8.&#160;The protocol section (step 2) provides details on the pre-processing steps.
U-Net9 is a widely used segmentation model for medical images. The U-Net architecture comprises an encoder, which extracts the features of the medical images, and a decoder, which presents the discriminative features semantically. When employing U-Net for CT scans, the convolutional layers consist of 3D filters10,11. This is a challenge because the computation of 3D filters requires a large memory capacity. To reduce the memory requirements for 3D U-Net, SEQ-UNET8, wherein a set of sequential 2D slices are used in the U-Net, was proposed. To prevent the loss of spatiotemporal correlations between the 2D image slices of the 3D CT scan, two bi-directional convolutional long-term short memories (C-LSTMs)12 are employed in basic U-Net. The first bi-directional C-LSTM extracts the inter-slice correlations at the end of the encoder. The second bi-directional C-LSTM, after the output of the decoder, transforms the semantic segmentation information in the dimensions of the slice sequence into a single image segmentation. The architecture of SEQ-UNET is shown in Figure 2. The implementation codes are available at github.com/SleepyChild1005/OrbitSeg, and the usage of the codes is detailed in the protocol section (step 3).

<table><tbody><tr><td>GitHub link</td><td></td><td></td><td>github.com/SleepyChild1005/OrbitSeg</td></tr><tr><td>MediLabel</td><td>INGRADIENT (Seoul, Korea)</td><td></td><td>a medical image labeling software promgram for segmentation with fewer click and higher speed</td></tr><tr><td>SEQ-UNET</td><td></td><td></td><td>downloadable from GitHub</td></tr><tr><td>SmartFil</td><td></td><td></td><td>wizard in MediLabel&amp;nbsp;</td></tr><tr><td>SmartPencil&amp;nbsp;</td><td></td><td></td><td>wizard in MediLabel&amp;nbsp;</td></tr></tbody></table>

application of deep learning-based medical image segmentation via orbital computed tomography

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of constructing an orbital computed tomography (CT) segmentation model based on U-Net. For supervised learning, a labor-intensive and time-consuming process is required. The method of labeling with super-resolution to efficiently mask the ground truth on orbital CT images is introduced. Also, the volume of interest is cropped as part of the pre-processing of the dataset. Then, after extracting the volumes of interest of the orbital structures, the model for segmenting the key structures of the orbital CT is constructed using U-Net, with sequential 2D slices that are used as inputs and two bi-directional convolutional long-term short memories for conserving the inter-slice correlations. This study primarily focuses on the segmentation of the eyeball, optic nerve, and extraocular muscles. The evaluation of the segmentation reveals the potential application of segmentation to orbital CT images using deep learning methods.

For the quantitative evaluation, two evaluation metrics were adopted, which were used in the CT image segmentation task. These were two similarity metrics, including dice score (DICE) and volume similarity (VS)13:
DICE (%) = 2 &#215; TP/(2 &#215; TP + FP + FN) 
VS (%) = 1 &#8722; |FN &#8722; FP|/(2 &#215; TP + FP + FN)
where TP, FP, and FN denote the true positive, false positive, and false negative values, respectively, when the segmentation result and the segmentation mask are given.
The performance of SEQ-UNET for orbital structure segmentation was evaluated by four cross-validations. The results are shown in Table 1. The eyeball segmentation using SEQ-UNET achieved a dice score of 0.86 and a VS of 0.83. The segmentation of the extraocular muscles and optic nerve achieved low dice scores (0.54 and 0.34, respectively). The dice score of the eyeball segmentation was over 80% because it had a large portion of the VOIs and little heterogeneity between CT scans. The dice scores of the extraocular muscles and optic nerve were relatively low because they infrequently appeared in the CT volume and were found in a relatively small number of the CT slices. However, the visual similarity scores of the extraocular muscles and optic nerve (0.65 and 0.80, respectively) were higher than their dice scores. This result indicates that the specificity of segmentation was low. Overall, the dice score and visual similarity of SEQ-UNET for the segmentation of all the orbital substructures were 0.79 and 0.82, respectively. Examples of the visual results of orbital structure segmentation are shown in Figure 3. In Figure 3A-C, blue is the predicted segmentation result, and red is the ground truth mask. In Figure 3D, red, green, and orange are the eyeball, optic muscle, and nerve segmentation, respectively.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/64500/64500fig01.jpg" /> 
Figure 1: Semi-automatic masking features. Masking the eyeball, extraocular muscles, and optic nerve on orbital CT scans&#160;using (A) SmartPencil, (B)&#160;SmartFill, and&#160;(C)&#160;AutoCorrection. The mask of the eyeball is labeled by SmartPencil, which computes the super pixels of the slices, and the mask is made by clicking on the super pixels. After clicking some of the eyeball super pixels, the entire eyeball mask can be computed by SmartFill. In the case of masking the optic nerve, the masking refinement is made by AutoCorrection. Blue color labeled eyeballs are shown in (A) and (B). <a href="https://www.jove.com/files/ftp_upload/64500/64500fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64500/64500fig02.jpg" /> 
Figure 2: SEQ U-Net architecture. Sequential 2D slices as input and output; two bi-directional C-LSTMs are applied to the end of the encoding and decoding blocks based on the U-Net architecture. <a href="https://www.jove.com/files/ftp_upload/64500/64500fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/64500/64500fig03.jpg" /> 
Figure 3: Segmentation results of the orbital structures. (A) Eyeball (label 1),&#160;(B) optic muscle (label 2),&#160;(C) optic nerve (label 3), and (D) multi-label (labels 1, 2, and 3).&#160;The left image is the VOI of the orbit, the center image is the predicted segmentation, and the right image is the ground truth. In (A), (B), and (C), blue is the predicted segmentation result, and red is the ground truth mask. In (D), red, green, and orange are the eyeball, extraocular muscle, and optic nerve segmentation, respectively. The predicted segmentation showed high performance&#160;(DICE: 0.86 vs. 0.82) in the case of the eyeball but low performance in the case of theextraocular muscle (DICE: 0.54 vs. 0.65) and optic nerve(DICE: 0.34 vs. 0.8). <a href="https://www.jove.com/files/ftp_upload/64500/64500fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td rowspan="2"></td>
			<td colspan="2">Multi-Label</td>
			<td colspan="2">Label 1 (Eyeball)</td>
			<td colspan="2">Label 2 (Extraocular muscle)</td>
			<td colspan="2">Label 3 (Optic nerve)</td>
		</tr>
		<tr>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
		</tr>
		<tr>
			<td>SEQ-UNET</td>
			<td>0.79</td>
			<td>0.82</td>
			<td>0.86</td>
			<td>0.83</td>
			<td>0.54</td>
			<td>0.65</td>
			<td>0.34</td>
			<td>0.8</td>
		</tr>
	</tbody>
</table>
Table 1: Segmentation results for the dice score and visual similarity. The eyeball, which has a relatively large number of slices, was segmented well with a DICE of 0.8, but the extraocular muscle and optic nerve, which have small numbers of slices and line shape, were partially segmented with DICE values of 0.54 and 0.34, respectively.
Video 1: SmartPencil wizard in the masking software program. A demonstration of annotating multiple pixels for eyeball masking. The masking tasks are enabled with one click on clustered super pixels. <a href="https://www.jove.com/files/ftp_upload/64500/SmartPencil_Figure1(A).mp4" target="_blank">Please click here to download this Video.</a>
Video 2: SmartFill wizard in the masking software program. A demonstration of annotating multiple pixels for eyeball masking. After selecting some pixels in the annotating area, this function generates full segmentation masks with similar intensities to the selected pixels. <a href="https://www.jove.com/files/ftp_upload/64500/SmartFill_Figure1(B).mp4" target="_blank">Please click here to download this Video.</a>
Video 3: AutoCorrection in the masking software program. A demonstration of the automatic correction of a masked pixel using a pre-trained convolutional neural network algorithm. <a href="https://www.jove.com/files/ftp_upload/64500/AutoCorrection_Figure(C).mp4" target="_blank">Please click here to download this Video.</a>
Supplementary Table 1: Runtime environment (RTE) of masking, pre-processing, and segmentation modeling. <a href="https://www.jove.com/files/ftp_upload/64500/Supplemtary Table 1.xlsx" target="_blank">Please click here to download this Table.</a>

Watch this Scientific Journal Video about Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography at JoVE.com

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

An object segmentation protocol for orbital computed tomography (CT) images is introduced. The methods of labeling the ground truth of orbital structures by using super-resolution, extracting the volume of interest from CT images, and modeling multi-label segmentation using 2D sequential U-Net for orbital CT images are explained for supervised learning.

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of ...

application-deep-learning-based-medical-image-segmentation-via

Research

JoVE Journal

Engineering

2.6K Views.  The Catholic University of Korea. We introduceerden in deze studie het objectsegmentatieprotocol voor orbitale computertomografiebeelden. Het is de eerste proef ter wereld. Met behulp van dit programma kunt u eenvoudig maskeren op elk anatomisch deel waar u maar wilt.Het visualiseert en bespaart uw tijd en moeite. We hopen dat deze studie een hoeksteen kan zijn voor de diagnose van orbitale ziekten, die moeilijk zijn voor biopsie. Begin met het uitvoeren van het maskeringssoftwareprogramma.Als u de orbitale CT w...

Video: Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Longitudinal Morphological and Physiological Monitoring of Three-dimensional Tumor Spheroids Using Optical Coherence Tomography

Tumor spheroids have been developed as a three-dimensional (3D) cell culture model in cancer research and anti-cancer drug discovery. However, currently, high-throughput imaging modalities utilizing bright field or fluorescence detection, are unable to resolve the overall 3D structure of the tumor spheroid due to limited light penetration, diffusion of fluorescent dyes and depth-resolvability. Recently, our lab demonstrated the use of optical coherence tomography (OCT), a label-free and non-destructive 3D imaging modality, to perform longitudinal characterization of multicellular tumor spheroids in a 96-well plate. OCT was capable of obtaining 3D morphological and physiological information of tumor spheroids growing up to about 600 &#181;m in height. In this article, we demonstrate a high-throughput OCT (HT-OCT) imaging system that scans the whole multi-well plate and obtains 3D OCT data of tumor spheroids automatically. We describe the details of the HT-OCT system and construction guidelines in the protocol. From the 3D OCT data, one can visualize the overall structure of the spheroid with 3D rendered and orthogonal slices, characterize the longitudinal growth curve of the tumor spheroid based on the morphological information of size and volume, and monitor the growth of the dead-cell regions in the tumor spheroid based on optical intrinsic attenuation contrast. We show that HT-OCT can be used as a high-throughput imaging modality for drug screening as well as characterizing biofabricated samples.

Optical coherence tomography (OCT), a three-dimensional imaging technology, was used to monitor and characterize the growth kinetics of multicellular tumor spheroids. Precise volumetric quantification of tumor spheroids using a voxel counting approach, and label-free dead tissue detection in the spheroids based on intrinsic optical attenuation contrast, were demonstrated.

Tumor spheroids have been developed as a three-dimensional (3D) cell culture model in cancer research and anti-cancer drug discovery. However, currently, ...

Cancer Research

Brain Infarct Segmentation and Registration on MRI or CT for Lesion-symptom Mapping

In lesion-symptom mapping (LSM), brain function is inferred by relating the location of acquired brain lesions to behavioral or cognitive symptoms in a group of patients. With recent advances in brain imaging and image processing, LSM has become a popular tool in cognitive neuroscience. LSM can provide fundamental insights into the functional architecture of the human brain for a variety of cognitive and non-cognitive functions. A crucial step in performing LSM studies is the segmentation of lesions on brains scans of a large group of patients and registration of each scan to a common stereotaxic space (also called standard space or a standardized brain template). Described here is an open-access, standardized method for infarct segmentation and registration for the purpose of LSM, as well as a detailed and hands-on walkthrough based on exemplary cases. A comprehensive tutorial for the manual segmentation of brain infarcts on CT scans and DWI or FLAIR MRI sequences is provided, including criteria for infarct identification and pitfalls for different scan types. The registration software provides multiple registration schemes that can be used for processing of CT and MRI data with heterogeneous acquisition parameters. A tutorial on using this registration software and performing visual quality checks and manual corrections (which are needed in some cases) is provided. This approach provides researchers with a framework for the entire process of brain image processing required to perform an LSM study, from gathering of the data to final quality checks of the results.

Provided here is a practical tutorial for an open-access, standardized image processing pipeline for the purpose of lesion-symptom mapping. A step-by-step walkthrough is provided for each processing step, from manual infarct segmentation on CT/MRI to subsequent registration to standard space, along with practical recommendations and illustrations with exemplary cases.

In lesion-symptom mapping (LSM), brain function is inferred by relating the location of acquired brain lesions to behavioral or cognitive symptoms in a ...

Behavior

Multiparametric Tumor Organoid Drug Screening Using Widefield Live-Cell Imaging for Bulk and Single-Organoid Analysis

Patient-derived tumor organoids (PDTOs) hold great promise for preclinical and translational research and predicting the patient therapy response from ex vivo drug screenings. However, current adenosine triphosphate (ATP)-based drug screening assays do not capture the complexity of a drug response (cytostatic or cytotoxic) and intratumor heterogeneity that has been shown to be retained in PDTOs due to a bulk readout. Live-cell imaging is a powerful tool to overcome this issue and visualize drug responses more in-depth. However, image analysis software is often not adapted to the three-dimensionality of PDTOs, requires fluorescent viability dyes, or is not compatible with a 384-well microplate format. This paper describes a semi-automated methodology to seed, treat, and image PDTOs in a high-throughput, 384-well format using conventional, widefield, live-cell imaging systems. In addition, we developed viability marker-free image analysis software to quantify growth rate-based drug response metrics that improve reproducibility and correct growth rate variations between different PDTO lines. Using the normalized drug response metric, which scores drug response based on the growth rate normalized to a positive and negative control condition, and a fluorescent cell death dye, cytotoxic and cytostatic drug responses can be easily distinguished, profoundly improving the classification of responders and non-responders. In addition, drug-response heterogeneity can by quantified from single-organoid drug response analysis to identify potential, resistant clones. Ultimately, this method aims to improve the prediction of clinical therapy response by capturing a multiparametric drug response signature, which includes kinetic growth arrest and cell death quantification.

This protocol describes a semi-automated method for medium- to high-throughput organoid drug screenings and microscope-agnostic, automated image analysis software to quantify and visualize multiparametric, single-organoid drug responses to capture intratumor heterogeneity.

Patient-derived tumor organoids (PDTOs) hold great promise for preclinical and translational research and predicting the patient therapy response from ex ...

Swin-PSAxialNet: A Streamlined Approach for Segmenting Multiple Organs

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Abdominal multi-organ segmentation is one of the most important topics in the field of medical image analysis, and it plays an important role in supporting clinical workflows such as disease diagnosis and treatment planning. In this study, an efficient multi-organ segmentation method called Swin-PSAxialNet based on the nnU-Net architecture is proposed. It was designed specifically for the precise segmentation of 11 abdominal organs in CT images. &#65279;The proposed network has made the following improvements compared to nnU-Net. Firstly, Space-to-depth (SPD) modules and parameter-shared axial attention (PSAA) feature extraction blocks were introduced, enhancing the capability of 3D image feature extraction. Secondly, a multi-scale image fusion approach was employed to capture detailed information and spatial features, improving the capability of extracting subtle features and edge features. Lastly, a parameter-sharing method was introduced to reduce the model's computational cost and training speed. &#65279;The proposed network achieves an average Dice coefficient of 0.93342 for the segmentation task involving 11 organs. Experimental results indicate the notable superiority of Swin-PSAxialNet over previous mainstream segmentation methods. The method shows excellent accuracy and low computational costs in segmenting major abdominal organs.

The present protocol describes an efficient multi-organ segmentation method called Swin-PSAxialNet, which has achieved excellent accuracy compared to previous segmentation methods. The key steps of this procedure include dataset collection, environment configuration, data preprocessing, model training and comparison, and ablation experiments.

Abdominal multi-organ segmentation is one of the most important topics in the field of medical image analysis, and it plays an important role in ...

Automated Segmentation of Cortical Grey Matter from T1-Weighted MRI Images

Within neuroimaging research, a number of recent studies have discussed the impact of between-study differences in volumetric findings that are thought to result from the use of different segmentation tools to generate brain volumes. Here, processing pipelines for seven automated tools that can be used to segment grey matter within the brain are presented. The protocol provides an initial step for researchers aiming to find the most accurate method for generating grey matter volumes from T1-weighted MRI scans. Steps to undertake detailed visual quality control are also included in the manuscript. This protocol covers a range of potential segmentation tools and encourages users to compare the performance of these tools within a subset of their data before selecting one to apply to a full cohort. Furthermore, the protocol may be further generalized to the segmentation of other brain regions.

This protocol describes the process of applying seven different automated segmentation tools to structural T1-weighted MRI scans to delineate grey matter regions that can be used for the quantification of grey matter volume.

Within neuroimaging research, a number of recent studies have discussed the impact of between-study differences in volumetric findings that are thought to ...

Neuroscience

Multicolor 3D Printing of Complex Intracranial Tumors in Neurosurgery

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. The model can be used for planning and simulating critical steps of a surgical approach. Therefore, it is important that anatomical structures such as blood vessels inside a tumor can be printed to be colored not only on their surface, but throughout their whole volume. During simulation this allows for the removal of certain parts (e.g., with a high-speed drill) and revealing internally located structures of a different color. Thus, diagnostic information from various imaging modalities (e.g., CT, MRI) can be combined in a single compact and tangible object.
However, preparation and printing of such a fully colored anatomical model remains a difficult task. Therefore, a step-by-step guide is provided, demonstrating the fusion of different cross-sectional imaging data sets, segmentation of anatomical structures, and creation of a virtual model. In a second step the virtual model is printed with volumetrically colored anatomical structures using a plaster-based color 3D binder jetting technique. This method allows highly accurate reproduction of patient-specific anatomy as shown in a series of 3D-printed petrous apex chondrosarcomas. Furthermore, the models created can be cut and drilled, revealing internal structures that allow for simulation of surgical procedures.

The protocol describes the fabrication of fully colored three-dimensional prints of patient-specific, anatomical skull models to be used for surgical simulation. The crucial steps of combining different imaging modalities, image segmentation, three-dimensional model extraction, and production of the prints are explained.

Three-dimensional (3D) printing technologies offer the possibility of visualizing patient-specific pathologies in a physical model of correct dimensions. ...

Medicine

Enhancing Neuroimaging Analysis in Brain Disorders

Manual Segmentation of the Human Choroid Plexus Using Brain MRI

The choroid plexus has been implicated in neurodevelopment and a range of brain disorders. Evidence demonstrates that the choroid plexus is critical for brain maturation, immune/ inflammatory regulation, and behavioral/cognitive functioning. However, current automated neuroimaging segmentation tools are poor at accurately and reliably segmenting the lateral ventricle choroid plexus. Furthermore, there is no existing tool that segments the choroid plexus located in the third and fourth ventricles of the brain. Thus, a protocol delineating how to segment the choroid plexus in the lateral, third, and fourth ventricle is needed to increase the reliability and replicability of studies examining the choroid plexus in neurodevelopmental and brain disorders. This protocol provides detailed steps to create separately labeled files in 3D Slicer for the choroid plexus based on DICOM or NIFTI images. The choroid plexus will be manually segmented using the axial, sagittal, and coronal planes of T1w images making sure to exclude voxels from gray or white matter structures bordering the ventricles. Windowing will be adjusted to assist in the localization of the choroid plexus and its anatomical boundaries. Methods for assessing accuracy and reliability will be demonstrated as part of this protocol. Gold standard segmentation of the choroid plexus using manual delineations can be used to develop better and more reliable automated segmentation tools that can be openly shared to elucidate changes in the choroid plexus across the lifespan and within various brain disorders.

Author Spotlight: Bridging Gaps in Anatomy and Establishing a Foundation for Algorithmic Studies 

Despite the crucial role of the choroid plexus in the brain, neuroimaging studies of this structure are scarce due to the lack of reliable automated segmentation tools. The present protocol aims to ensure gold-standard manual segmentation of the choroid plexus that can inform future neuroimaging studies.

The choroid plexus has been implicated in neurodevelopment and a range of brain disorders. Evidence demonstrates that the choroid plexus is critical for ...

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application

Tongue diagnosis is an essential technique of traditional Chinese medicine (TCM) diagnosis, and the need for objectifying tongue images through image processing technology is growing. The present study provides an overview of the progress made in tongue objectification over the past decade and compares segmentation models. Various deep learning models are constructed to verify and compare algorithms using real tongue image sets. The strengths and weaknesses of each model are analyzed. The findings indicate that the U-Net algorithm outperforms other models regarding precision accuracy (PA), recall, and mean intersection over union (MIoU) metrics. However, despite the significant progress in tongue image acquisition and processing, a uniform standard for objectifying tongue diagnosis has yet to be established. To facilitate the widespread application of tongue images captured using mobile devices in tongue diagnosis objectification, further research could address the challenges posed by tongue images captured in complex environments.

The present study employed U-Net and other deep learning algorithms to segment a tongue image and compared the segmentation results to investigate the objectification of tongue diagnosis.

Tongue diagnosis is an essential technique of traditional Chinese medicine (TCM) diagnosis, and the need for objectifying tongue images through image ...

Virtual Reality-3D Modeling for Minimally Invasive Neuro-Interventional Planning

Pioneering Patient-Specific Approaches for Precision Surgery Using Imaging and Virtual Reality

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular procedural solutions. Complex open surgical procedures used to be the only option for the treatment of a myriad of conditions like pulmonary and aortic valve replacement as well as cerebral aneurysm repair. However, due to advancements in catheter-delivered devices and operator expertise, these procedures (along with many others) can now be performed through minimally invasive procedures delivered through a central or peripheral vein or artery. The decision to shift from an open procedure to an endovascular approach is based on multi-modal imaging, often including 3D Digital Imaging and Communications in Medicine (DICOM) imaging datasets. Utilizing these 3D images, our lab generates 3D models of the pathologic anatomy, thereby allowing the pre-procedural analysis necessary to pre-plan critical components of the catheterization lab procedure, namely, C-arm positioning, 3D measurement, and idealized road-map generation. This article describes how to take segmented 3D models of patient-specific pathology and predict generalized C-arm positions, how to measure critical two-dimensional (2D) measurements of 3D structures relevant to the 2D fluoroscopy projections, and how to generate 2D fluoroscopy roadmap analogs that can assist in proper C-arm positioning during catheterization lab procedures.

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Advancements in endovascular treatment have replaced complex open surgical procedures with minimally invasive options, like valve replacement and aneurysm repair. This paper proposes using three-dimensional (3D) modeling and virtual reality to aid in C-arm positioning, angle measurements, and roadmap generation for neuro-interventional catheterization lab procedural planning, minimizing procedure time.

Endovascular treatment of complex vascular anomalies shifts the risk of open surgical procedures to the benefit of minimally invasive endovascular ...

Deep Learning-Based Segmentation of Cryo-Electron Tomograms

Cryo-electron tomography (cryo-ET) allows researchers to image cells in their native, hydrated state at the highest resolution currently possible. The technique has several limitations, however, that make analyzing the data it generates time-intensive and difficult. Hand segmenting a single tomogram can take from hours to days, but a microscope can easily generate 50 or more tomograms a day. Current deep learning segmentation programs for cryo-ET do exist, but are limited to segmenting one structure at a time. Here, multi-slice U-Net convolutional neural networks are trained and applied to automatically segment multiple structures simultaneously within cryo-tomograms. With proper preprocessing, these networks can be robustly inferred to many tomograms without the need for training individual networks for each tomogram. This workflow dramatically improves the speed with which cryo-electron tomograms can be analyzed by cutting segmentation time down to under 30 min in most cases. Further, segmentations can be used to improve the accuracy of filament tracing within a cellular context and to rapidly extract coordinates for subtomogram averaging.

This is a method for training a multi-slice U-Net for multi-class segmentation of cryo-electron tomograms using a portion of one tomogram as a training input. We describe how to infer this network to other tomograms and how to extract segmentations for further analyses, such as subtomogram averaging and filament tracing.

Cryo-electron tomography (cryo-ET) allows researchers to image cells in their native, hydrated state at the highest resolution currently possible. The ...

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Samenvatting

Meer video's verkennen