This work was supported by the National Research Foundation of Korea (NRF), grant funded by the Ministry of Science and ICT of Korea (MSIT) (number: 2020R1C1C1010079). For the CMC-ORBIT dataset, the central Institutional Review Board (IRB) of the Catholic Medical Center provided approval (XC19REGI0076).&#160;This work was supported by 2022 Hongik University Research Fund.

The present work was performed with the approval of the Institutional Review Board (IRB) of the Catholic Medical Center, and the privacy, confidentiality, and security of the health information were protected. The orbital CT data were collected (from de-identified human subjects) from hospitals affiliated with the College of Medicine, the Catholic University of Korea (CMC; Seoul St. Mary&#39;s Hospital, Yeouido St. Mary&#39;s Hospital, Daejeon St. Mary&#39;s Hospital, and St. Vincent Hospital). The orbital CT scans were obtained from January 2016 to December 2020. The dataset contained 46 orbital CT scans from Korean men and women ranging in age from 20 years to 60 years. The runtime environment (RTE) is summarized in Supplementary Table 1.
1. Masking the eyeball, optic nerve, and extraocular muscles on the orbital CT scans
<ol>
	<li>Run the masking software program. 
	NOTE: The masking software program (MediLabel, see the Table of Materials) is a medical image labeling software program for segmentation, which requires few clicks and has high speed.</li>
	<li>Load the orbital CT by clicking on the open file icon and selecting the target CT file. Then, the CT scans are shown on the screen.</li>
	<li>Mask the eyeball, optic nerve, and extraocular muscles using super pixels.
	<ol>
		<li>Run the SmartPencil by clicking on the SmartPencil wizard in MediLabel (Video 1).</li>
		<li>Control the resolution of the super pixel map if necessary (e.g., 100, 500, 1,000, and 2,000 super pixels).</li>
		<li>Click on the cluster of super pixels of the eyeball, optic nerve, and extraocular muscles on the super pixel map, where pixels of similar image intensity values are clustered.</li>
	</ol>
	</li>
	<li>Refine the masks with the autocorrection functions in MediLabel.
	<ol>
		<li>Click on the SmartFill wizard after masking some of the super pixels on the slices (Video 2).</li>
		<li>Click on the AutoCorrection icon, and ensure that the corrected mask labels are computed (Video 3).</li>
	</ol>
	</li>
	<li>Repeat step 1.3 and step 1.4 until the refinement of the masking is complete.</li>
	<li>Save the masked images.</li>
</ol>
2. Pre-processing: Window clipping and cropping the VOIs
<ol>
	<li>Extract the VOIs with preprocessing_multilabel.py (the file is downloadable from GitHub).
	<ol>
		<li>Run preprocessing_multilabel.py.</li>
		<li>Check the scans and masks, which are cropped and saved in the VOIs folder.</li>
	</ol>
	</li>
	<li>Transform the VOIs to the set of three sequential CT slices for the input to SEQ-UNET with builder_multilabel.py (the file is downloadable from GitHub).
	<ol>
		<li>Run sequence_builder_multilabel.py.</li>
		<li>Ensure that the slices and masks are resized to 64 pixels by 64 pixels during the transformation.</li>
		<li>During the transformation, perform clipping with the 48 HU clipping level and the 400 HU window.</li>
		<li>Check the saved transformed CT scans (nii file) and masks (nii file) in the scan folder and the mask folder under pre-processed folders, respectively.</li>
	</ol>
	</li>
</ol>
3. Four cross-validations of the orbital segmentation model
<ol>
	<li>Build the model following the steps below.
	<ol>
		<li>Run main.py.</li>
		<li>When running main.py, give the fold number of the four cross-validations by &#34;-fold num x&#34;, where x is 0, 1, 2, or 3.</li>
		<li>When running main.py, use the epoch, which is the number of training iterations, as an option, such as &#34;-epoch x&#34;, where x is the epoch number. The default number is 500.</li>
		<li>When running main.py, set the batch size, which is the number of training samples in a single training session. The default number is 32.</li>
		<li>In main.py, load the CT scans and masks, and initialize the SEQ-UNET with the pre-trained parameters using the LIDC-IDRI dataset (downloadable from the cancer imaging archive).</li>
		<li>In main.py, perform the testing of the model after training. Calculate the evaluation metrics, dice score, and volume similarity, and save them in the metrics folder.</li>
		<li>Check the results in the segmented folder.</li>
	</ol>
	</li>
</ol>

<ol>
	<li>Li, Z., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Deep+learning-based+CT+radiomics+for+feature+representation+and+analysis+of+aging+characteristics+of+Asian+bony+orbit.">Deep learning-based CT radiomics for feature representation and analysis of aging characteristics of Asian bony orbit.</a> Journal of Craniofacial Surgery. 33 (1), 312-318 (2022).</li><li>Hamwood, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+deep+learning+method+for+automatic+segmentation+of+the+bony+orbit+in+MRI+and+CT+images.">A deep learning method for automatic segmentation of the bony orbit in MRI and CT images.</a> Scientific Reports. 11, 1-12 (2021).</li><li>Kim, K. S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Schwannoma+of+the+orbit.">Schwannoma of the orbit.</a> Archives of Craniofacial Surgery. 16 (2), 67-72 (2015).</li><li>Baur, C., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Autoencoders+for+unsupervised+anomaly+segmentation+in+brain+MR+images:+A+comparative+study.">Autoencoders for unsupervised anomaly segmentation in brain MR images: A comparative study.</a> Medical Image Analysis. 69, 101952 (2021).</li><li>Tr&#233;meau, A., Colantoni, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Regions+adjacency+graph+applied+to+color+image+segmentation.">Regions adjacency graph applied to color image segmentation.</a> IEEE Transactions on Image Processing. 9 (4), 735-744 (2000).</li><li>Boykov, Y. Y., Jolly, M. -. P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Interactive+graph+cuts+for+optimal+boundary+&amp;+region+segmentation+of+objects+in+ND+images.">Interactive graph cuts for optimal boundary &amp; region segmentation of objects in ND images.</a> Proceedings of Eighth IEEE International Conference on Computer Vision. International Conference on Computer Vision. 1, 105-122 (2001).</li><li>Rother, C., Kolmogorov, V., Blake, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=&amp;#34;GrabCut&amp;#34;+interactive+foreground+extraction+using+iterated+graph+cuts.">&amp;#34;GrabCut&amp;#34; interactive foreground extraction using iterated graph cuts.</a> ACM Transactions on Graphics. 23 (3), 309-314 (2004).</li><li>Suh, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Supervised+segmentation+with+domain+adaptation+for+small+sampled+orbital+CT+images.">Supervised segmentation with domain adaptation for small sampled orbital CT images.</a> Journal of Computational Design and Engineering. 9 (2), 783-792 (2022).</li><li>Ronneberger, O., Fischer, P., Brox, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=U-net:+Convolutional+networks+for+biomedical+image+segmentation.">U-net: Convolutional networks for biomedical image segmentation.</a> International Conference on Medical Image Computing and Computer-Assisted Intervention.Medical Image Computing and Computer-Assisted Intervention - MICCAI. , 234-241 (2015).</li><li>Qamar, S., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+variant+form+of+3D-UNet+for+infant+brain+segmentation.">A variant form of 3D-UNet for infant brain segmentation.</a> Future Generation Computer Systems. 108, 613-623 (2020).</li><li>Nguyen, H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Ocular+structures+segmentation+from+multi-sequences+MRI+using+3D+UNet+with+fully+connected+CRFS.">Ocular structures segmentation from multi-sequences MRI using 3D UNet with fully connected CRFS.</a> Computational Pathology and Ophthalmic Medical Image Analysis. , 167-175 (2018).</li><li>Liu, Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Bidirectional-convolutional+LSTM+based+spectral-spatial+feature+learning+for+hyperspectral+image+classification.">Bidirectional-convolutional LSTM based spectral-spatial feature learning for hyperspectral image classification.</a> Remote Sensing. 9 (12), 1330 (2017).</li><li>Yeghiazaryan, V., Voiculescu, I. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Family+of+boundary+overlap+metrics+for+the+evaluation+of+medical+image+segmentation.">Family of boundary overlap metrics for the evaluation of medical image segmentation.</a> Journal of Medical Imaging. 5 (1), 015006 (2018).</li><li>Zhang, G., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparable+performance+of+deep+learning-based+to+manual-based+tumor+segmentation+in+KRAS/NRAS/BRAF+mutation+prediction+with+MR-based+radiomics+in+rectal+cancer.">Comparable performance of deep learning-based to manual-based tumor segmentation in KRAS/NRAS/BRAF mutation prediction with MR-based radiomics in rectal cancer.</a> Frontiers in Oncology. 11, 696706 (2021).</li><li>Christopher, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Performance+of+deep+learning+architectures+and+transfer+learning+for+detecting+glaucomatous+optic+neuropathy+in+fundus+photographs.">Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs.</a> Scientific Reports. 8, 16685 (2018).</li></ol>

The authors declare no conflicts of interest.

Deep learning-based medical image analysis is widely used for disease detection. In the ophthalmology domain, detection and segmentation models are used in diabetic retinopathy, glaucoma, age-related macular degeneration, and retinopathy of prematurity. However, other rare diseases apart from those in ophthalmology have not been studied due to the limited access to large open public datasets for deep learning analysis. When applying this method in situations when no public dataset is available, the masking step, which is a labor-intensive and time-consuming task, is unavoidable. However, the proposed masking step (protocol section, step 1) helps to generate masking with high accuracy within a short time. Using super pixels and neural network-based filling, which cluster pixels that are similar in low-level image properties, clinicians can label the masks by clicking the groups of pixels instead of pointing out the specific pixels. Also, the automatic correction functions help refine the mask processes. This method's efficiency and effectiveness will help generate more masked images in medical research.
Among the many possibilities in pre-processing, extracting VOIs and window clipping are effective methods. Here, extracting VOIs and window clipping are introduced in step 2 of the protocol. When the clinicians prepare the dataset, extracting the VOI from the given dataset is the most important step in the process because most segmentation cases focus on small and specific regions in the whole medical image. Regarding the VOIs, the regions of the eyeball, optic nerve, and extraocular muscles are cropped based on the location, but more effective methods for extracting VOIs have the potential to improve segmentation performance14. 
For the segmentation, SEQ-UNET is employed in the study. The 3D medical images have large volumes, so deep neural network models require large memory capacities. In SEQ-UNET, the segmentation model is implemented with a small number of slices to reduce the required memory size without losing the features of the 3D information. 
The model was trained with 46 VOIs, which is not a large number for model training. Due to the small number of training datasets, the performance of optic nerve and extraocular muscle segmentation is limited. Transfer learning15 and domain adaptation8 could provide a solution for improving the segmentation performance.
The whole segmentation process introduced here is not limited to orbital CT segmentation. The efficient labeling method helps create a new medical image dataset for when the application domain is unique to the research area. The python codes of GitHub concerning the pre-processing and segmentation modeling can be applied to other domains with the modification of the cropping region, the window clipping level, and the model hyper-parameters, such as the number of sequential slices, the U-Net architectures, and so on.

The orbit is a small and complicated space of approximately 30.1 cm3 that contains important structures such as the eyeball, nerves, extraocular muscles, supportive tissues, and vessels for vision and eyeball movements1. Orbital tumors are abnormal tissue growths in the orbit, and some of them threaten patients&#39; vision or eyeball movement, which may lead to fatal dysfunction. To conserve patients&#39; visual function, clinicians must decide on the treatment modalities based on the tumor characteristics, and a surgical biopsy is generally inevitable. This compact and crowded area often makes it challenging for clinicians to perform a biopsy without damaging the normal structure. The deep learning-based image analysis of pathology for determining the condition of the orbit could help in avoiding unnecessary or avoidable injury to the orbital tissues during biopsy2. One method of image analysis for orbital tumors is tumor detection and segmentation. However, the collection of large amounts of data for CT images containing orbital tumors is limited due to their low incidence3. The other efficient method for computational tumor diagnosis4 involves comparing the tumor to the normal structures of the orbit. The number of orbital CT images in normal structures is relatively larger than that in tumors. Therefore, the segmentation of normal orbital structures is the first step to achieve this goal.
This study presents the entire process of deep learning-based orbital structure segmentation, including the data collection, pre-processing, and subsequent modeling. The study is intended to be a resource for clinicians interested in using the current method to efficiently generate a masked dataset and for ophthalmologists requiring information about pre-processing and modeling for orbital CT images.This article presents a new method for orbital structure segmentation and sequential U-Net, a sequential 2D segmentation model based on a representative deep-learning solution in U-Net for medical image segmentation. The protocol describes the detailed procedure of orbit segmentation, including (1) how to use a masking tool for the ground truth of orbit structure segmentation, (2) the steps required for the pre-processing of the orbital images, and (3) how to train the segmentation model and evaluate the segmentation performance.
For supervised learning, four experienced ophthalmologists who had been board certified for over 5 years manually annotated the masks of the eyeball, optic nerve, and extraocular muscles. All the ophthalmologists used the masking software program (MediLabel, see the Table of Materials), which uses super-resolution for efficient masking on CT scans. The masking software has the following semi-automatic features: (1) SmartPencil, which generates super pixel map clusters with similar values of image intensity5; (2) SmartFill, which generates segmentation masks by computing the energy function of the ongoing foreground and background6,7; and (3) AutoCorrection, which makes the borders of the segmentation masks clean and consistent with the original image. Example images of the semi-automatic features are shown in Figure 1. The detailed steps of manual masking are provided in the protocol section (step 1).
The next step is the pre-processing of the orbital CT scans. To obtain the orbital volumes of interest (VOIs), the areas of the orbit where the eyeball, muscle, and nerve are located in normal conditions are identified, and these areas are cropped. The dataset has a high resolution, with &#60;1 mm in-plane voxel resolution and slice thickness, so the interpolation process is skipped. Instead, window clipping is conducted at the 48 HU clipping level and the 400 HU window. After the cropping and window clipping, three serial slices of the orbit VOIs are generated for the segmentation model input8.&#160;The protocol section (step 2) provides details on the pre-processing steps.
U-Net9 is a widely used segmentation model for medical images. The U-Net architecture comprises an encoder, which extracts the features of the medical images, and a decoder, which presents the discriminative features semantically. When employing U-Net for CT scans, the convolutional layers consist of 3D filters10,11. This is a challenge because the computation of 3D filters requires a large memory capacity. To reduce the memory requirements for 3D U-Net, SEQ-UNET8, wherein a set of sequential 2D slices are used in the U-Net, was proposed. To prevent the loss of spatiotemporal correlations between the 2D image slices of the 3D CT scan, two bi-directional convolutional long-term short memories (C-LSTMs)12 are employed in basic U-Net. The first bi-directional C-LSTM extracts the inter-slice correlations at the end of the encoder. The second bi-directional C-LSTM, after the output of the decoder, transforms the semantic segmentation information in the dimensions of the slice sequence into a single image segmentation. The architecture of SEQ-UNET is shown in Figure 2. The implementation codes are available at github.com/SleepyChild1005/OrbitSeg, and the usage of the codes is detailed in the protocol section (step 3).

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>GitHub link</td>
			<td></td>
			<td></td>
			<td>github.com/SleepyChild1005/OrbitSeg</td>
		</tr>
		<tr>
			<td>MediLabel</td>
			<td>INGRADIENT (Seoul, Korea)</td>
			<td></td>
			<td>a medical image labeling software promgram for segmentation with fewer click and higher speed</td>
		</tr>
		<tr>
			<td>SEQ-UNET</td>
			<td></td>
			<td></td>
			<td>downloadable from GitHub</td>
		</tr>
		<tr>
			<td>SmartFil</td>
			<td></td>
			<td></td>
			<td>wizard in MediLabel&#160;</td>
		</tr>
		<tr>
			<td>SmartPencil&#160;</td>
			<td></td>
			<td></td>
			<td>wizard in MediLabel&#160;</td>
		</tr>
	</tbody>
</table>

application of deep learning-based medical image segmentation via orbital computed tomography

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of constructing an orbital computed tomography (CT) segmentation model based on U-Net. For supervised learning, a labor-intensive and time-consuming process is required. The method of labeling with super-resolution to efficiently mask the ground truth on orbital CT images is introduced. Also, the volume of interest is cropped as part of the pre-processing of the dataset. Then, after extracting the volumes of interest of the orbital structures, the model for segmenting the key structures of the orbital CT is constructed using U-Net, with sequential 2D slices that are used as inputs and two bi-directional convolutional long-term short memories for conserving the inter-slice correlations. This study primarily focuses on the segmentation of the eyeball, optic nerve, and extraocular muscles. The evaluation of the segmentation reveals the potential application of segmentation to orbital CT images using deep learning methods.

For the quantitative evaluation, two evaluation metrics were adopted, which were used in the CT image segmentation task. These were two similarity metrics, including dice score (DICE) and volume similarity (VS)13:
DICE (%) = 2 &#215; TP/(2 &#215; TP + FP + FN) 
VS (%) = 1 &#8722; |FN &#8722; FP|/(2 &#215; TP + FP + FN)
where TP, FP, and FN denote the true positive, false positive, and false negative values, respectively, when the segmentation result and the segmentation mask are given.
The performance of SEQ-UNET for orbital structure segmentation was evaluated by four cross-validations. The results are shown in Table 1. The eyeball segmentation using SEQ-UNET achieved a dice score of 0.86 and a VS of 0.83. The segmentation of the extraocular muscles and optic nerve achieved low dice scores (0.54 and 0.34, respectively). The dice score of the eyeball segmentation was over 80% because it had a large portion of the VOIs and little heterogeneity between CT scans. The dice scores of the extraocular muscles and optic nerve were relatively low because they infrequently appeared in the CT volume and were found in a relatively small number of the CT slices. However, the visual similarity scores of the extraocular muscles and optic nerve (0.65 and 0.80, respectively) were higher than their dice scores. This result indicates that the specificity of segmentation was low. Overall, the dice score and visual similarity of SEQ-UNET for the segmentation of all the orbital substructures were 0.79 and 0.82, respectively. Examples of the visual results of orbital structure segmentation are shown in Figure 3. In Figure 3A-C, blue is the predicted segmentation result, and red is the ground truth mask. In Figure 3D, red, green, and orange are the eyeball, optic muscle, and nerve segmentation, respectively.
<img alt="Figure 1" class="xfigimg" src="/files/ftp_upload/64500/64500fig01.jpg" /> 
Figure 1: Semi-automatic masking features. Masking the eyeball, extraocular muscles, and optic nerve on orbital CT scans&#160;using (A) SmartPencil, (B)&#160;SmartFill, and&#160;(C)&#160;AutoCorrection. The mask of the eyeball is labeled by SmartPencil, which computes the super pixels of the slices, and the mask is made by clicking on the super pixels. After clicking some of the eyeball super pixels, the entire eyeball mask can be computed by SmartFill. In the case of masking the optic nerve, the masking refinement is made by AutoCorrection. Blue color labeled eyeballs are shown in (A) and (B). <a href="https://www.jove.com/files/ftp_upload/64500/64500fig01large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 2" class="xfigimg" src="/files/ftp_upload/64500/64500fig02.jpg" /> 
Figure 2: SEQ U-Net architecture. Sequential 2D slices as input and output; two bi-directional C-LSTMs are applied to the end of the encoding and decoding blocks based on the U-Net architecture. <a href="https://www.jove.com/files/ftp_upload/64500/64500fig02large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<img alt="Figure 3" class="xfigimg" src="/files/ftp_upload/64500/64500fig03.jpg" /> 
Figure 3: Segmentation results of the orbital structures. (A) Eyeball (label 1),&#160;(B) optic muscle (label 2),&#160;(C) optic nerve (label 3), and (D) multi-label (labels 1, 2, and 3).&#160;The left image is the VOI of the orbit, the center image is the predicted segmentation, and the right image is the ground truth. In (A), (B), and (C), blue is the predicted segmentation result, and red is the ground truth mask. In (D), red, green, and orange are the eyeball, extraocular muscle, and optic nerve segmentation, respectively. The predicted segmentation showed high performance&#160;(DICE: 0.86 vs. 0.82) in the case of the eyeball but low performance in the case of theextraocular muscle (DICE: 0.54 vs. 0.65) and optic nerve(DICE: 0.34 vs. 0.8). <a href="https://www.jove.com/files/ftp_upload/64500/64500fig03large.jpg" target="_blank">Please click here to view a larger version of this figure.</a>
<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td rowspan="2"></td>
			<td colspan="2">Multi-Label</td>
			<td colspan="2">Label 1 (Eyeball)</td>
			<td colspan="2">Label 2 (Extraocular muscle)</td>
			<td colspan="2">Label 3 (Optic nerve)</td>
		</tr>
		<tr>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
			<td>DICE</td>
			<td>VS</td>
		</tr>
		<tr>
			<td>SEQ-UNET</td>
			<td>0.79</td>
			<td>0.82</td>
			<td>0.86</td>
			<td>0.83</td>
			<td>0.54</td>
			<td>0.65</td>
			<td>0.34</td>
			<td>0.8</td>
		</tr>
	</tbody>
</table>
Table 1: Segmentation results for the dice score and visual similarity. The eyeball, which has a relatively large number of slices, was segmented well with a DICE of 0.8, but the extraocular muscle and optic nerve, which have small numbers of slices and line shape, were partially segmented with DICE values of 0.54 and 0.34, respectively.
Video 1: SmartPencil wizard in the masking software program. A demonstration of annotating multiple pixels for eyeball masking. The masking tasks are enabled with one click on clustered super pixels. <a href="https://www.jove.com/files/ftp_upload/64500/SmartPencil_Figure1(A).mp4" target="_blank">Please click here to download this Video.</a>
Video 2: SmartFill wizard in the masking software program. A demonstration of annotating multiple pixels for eyeball masking. After selecting some pixels in the annotating area, this function generates full segmentation masks with similar intensities to the selected pixels. <a href="https://www.jove.com/files/ftp_upload/64500/SmartFill_Figure1(B).mp4" target="_blank">Please click here to download this Video.</a>
Video 3: AutoCorrection in the masking software program. A demonstration of the automatic correction of a masked pixel using a pre-trained convolutional neural network algorithm. <a href="https://www.jove.com/files/ftp_upload/64500/AutoCorrection_Figure(C).mp4" target="_blank">Please click here to download this Video.</a>
Supplementary Table 1: Runtime environment (RTE) of masking, pre-processing, and segmentation modeling. <a href="https://www.jove.com/files/ftp_upload/64500/Supplemtary Table 1.xlsx" target="_blank">Please click here to download this Table.</a>

Watch this Scientific Journal Video about Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography at JoVE.com

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

An object segmentation protocol for orbital computed tomography (CT) images is introduced. The methods of labeling the ground truth of orbital structures by using super-resolution, extracting the volume of interest from CT images, and modeling multi-label segmentation using 2D sequential U-Net for orbital CT images are explained for supervised learning.

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of ...

application-deep-learning-based-medical-image-segmentation-via

Research

JoVE Journal

Engineering

2.5K Views.  The Catholic University of Korea. An object segmentation protocol for orbital computed tomography (CT) images is introduced. The methods of labeling the ground truth of orbital structures by using super-resolution, extracting the volume of interest from CT images, and modeling multi-label segmentation using 2D sequential U-Net for orbital CT images are explained for supervised learning.

Video: Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Echo Particle Image Velocimetry

The transport of mass, momentum, and energy in fluid flows is ultimately determined by spatiotemporal distributions of the fluid velocity field.1 Consequently, a prerequisite for understanding, predicting, and controlling fluid flows is the capability to measure the velocity field with adequate spatial and temporal resolution.2 For velocity measurements in optically opaque fluids or through optically opaque geometries, echo particle image velocimetry (EPIV) is an attractive diagnostic technique to generate "instantaneous" two-dimensional fields of velocity.3,4,5,6 In this paper, the operating protocol for an EPIV system built by integrating a commercial medical ultrasound machine7 with a PC running commercial particle image velocimetry (PIV) software8 is described, and validation measurements in Hagen-Poiseuille (i.e., laminar pipe) flow are reported.
For the EPIV measurements, a phased array probe connected to the medical ultrasound machine is used to generate a two-dimensional ultrasound image by pulsing the piezoelectric probe elements at different times. Each probe element transmits an ultrasound pulse into the fluid, and tracer particles in the fluid (either naturally occurring or seeded) reflect ultrasound echoes back to the probe where they are recorded. The amplitude of the reflected ultrasound waves and their time delay relative to transmission are used to create what is known as B-mode (brightness mode) two-dimensional ultrasound images. Specifically, the time delay is used to determine the position of the scatterer in the fluid and the amplitude is used to assign intensity to the scatterer. The time required to obtain a single B-mode image, t, is determined by the time it take to pulse all the elements of the phased array probe. For acquiring multiple B-mode images, the frame rate of the system in frames per second (fps) = 1/&delta;t. (See 9 for a review of ultrasound imaging.)
For a typical EPIV experiment, the frame rate is between 20-60 fps, depending on flow conditions, and 100-1000 B-mode images of the spatial distribution of the tracer particles in the flow are acquired. Once acquired, the B-mode ultrasound images are transmitted via an ethernet connection to the PC running the PIV commercial software. Using the PIV software, tracer particle displacement fields, D(x,y)[pixels], (where x and y denote horizontal and vertical spatial position in the ultrasound image, respectively) are acquired by applying cross correlation algorithms to successive ultrasound B-mode images.10 The velocity fields, u(x,y)[m/s], are determined from the displacements fields, knowing the time step between image pairs, &Delta;T[s], and the image magnification, M[meter/pixel], i.e., u(x,y) = MD(x,y)/&Delta;T. The time step between images &Delta;T = 1/fps + D(x,y)/B, where B[pixels/s] is the time it takes for the ultrasound probe to sweep across the image width. In the present study, M = 77[&mu;m/pixel], fps = 49.5[1/s], and B = 25,047[pixels/s]. Once acquired, the velocity fields can be analyzed to compute flow quantities of interest.

An echo particle image velocimetry (EPIV) system capable of acquiring two-dimensional fields of velocity in optically opaque fluids or through optically opaque geometries is described, and validation measurements in pipe flow are reported.

The transport of mass, momentum, and energy in fluid flows is ultimately determined by spatiotemporal distributions of the fluid velocity field.1 ...

High-speed Particle Image Velocimetry Near Surfaces

Multi-dimensional and transient flows play a key role in many areas of science, engineering, and health sciences but are often not well understood. The complex nature of these flows may be studied using particle image velocimetry (PIV), a laser-based imaging technique for optically accessible flows. Though many forms of PIV exist that extend the technique beyond the original planar two-component velocity measurement capabilities, the basic PIV system consists of a light source (laser), a camera, tracer particles, and analysis algorithms. The imaging and recording parameters, the light source, and the algorithms are adjusted to optimize the recording for the flow of interest and obtain valid velocity data. 
Common PIV investigations measure two-component velocities in a plane at a few frames per second. However, recent developments in instrumentation have facilitated high-frame rate (&gt; 1 kHz) measurements capable of resolving transient flows with high temporal resolution. Therefore, high-frame rate measurements have enabled investigations on the evolution of the structure and dynamics of highly transient flows. These investigations play a critical role in understanding the fundamental physics of complex flows.
A detailed description for performing high-resolution, high-speed planar PIV to study a transient flow near the surface of a flat plate is presented here. Details for adjusting the parameter constraints such as image and recording properties, the laser sheet properties, and processing algorithms to adapt PIV for any flow of interest are included.

A procedure for studying transient flows near boundaries using high-resolution, high-speed particle image velocimetry (PIV) is described here. PIV is a non-intrusive measurement technique applicable to any optically accessible flow by optimizing several parameter constraints such as the image and recording properties, the laser sheet properties, and analysis algorithms.

Multi-dimensional and transient flows play a key role in many areas of science, engineering, and health sciences but are often not well understood. The ...

In Depth Analyses of LEDs by a Combination of X-ray Computed Tomography (CT) and Light Microscopy (LM) Correlated with Scanning Electron Microscopy (SEM)

In failure analysis, device characterization and reverse engineering of light emitting diodes (LEDs), and similar electronic components of micro-characterization, plays an important role. Commonly, different techniques like X-ray computed tomography (CT), light microscopy (LM) and scanning electron microscopy (SEM) are used separately. Similarly, the results have to be treated for each technique independently. Here a comprehensive study is shown which demonstrates the potentials leveraged by linking CT, LM and SEM. In depth characterization is performed on a white emitting LED, which can be operated throughout all characterization steps. Major advantages are: planned preparation of defined cross sections, correlation of optical properties to structural and compositional information, as well as reliable identification of different functional regions. This results from the breadth of information available from identical regions of interest (ROIs): polarization contrast, bright and dark-field LM images, as well as optical images of the LED cross section in operation. This is supplemented by SEM imaging techniques and micro-analysis using energy dispersive X-ray spectroscopy.

In Depth Analyses of LEDs by a Combination of X-ray Computed Tomography CT and Light Microscopy LM Correlated with Scanning Electron Microscopy SEM

A workflow for comprehensive micro-characterization of active optical devices is outlined. It contains structural as well as functional investigations by means of CT, LM and SEM. The method is demonstrated for a white LED which can be still be operated during characterization.

In failure analysis, device characterization and reverse engineering of light emitting diodes (LEDs), and similar electronic components of ...

Measuring Spray Droplet Size from Agricultural Nozzles Using Laser Diffraction

When making an application of any crop protection material such as an herbicide or pesticide, the applicator uses a variety of skills and information to make an application so that the material reaches the target site (i.e., plant). Information critical in this process is the droplet size that a particular spray nozzle, spray pressure, and spray solution combination generates, as droplet size greatly influences product efficacy and how the spray moves through the environment. Researchers and product manufacturers commonly use laser diffraction equipment to measure the spray droplet size in laboratory wind tunnels. The work presented here describes methods used in making spray droplet size measurements with laser diffraction equipment for both ground and aerial application scenarios that can be used to ensure inter- and intra-laboratory precision while minimizing sampling bias associated with laser diffraction systems. Maintaining critical measurement distances and concurrent airflow throughout the testing process is key to this precision. Real time data quality analysis is also critical to preventing excess variation in the data or extraneous inclusion of erroneous data. Some limitations of this method include atypical spray nozzles, spray solutions or application conditions that result in spray streams that do not fully atomize within the measurement distances discussed. Successful adaption of this method can provide a highly efficient method for evaluation of the performance of agrochemical spray application nozzles under a variety of operational settings. Also discussed are potential experimental design considerations that can be included to enhance functionality of the data collected.

We present protocols to be used in the measurement of spray droplet size from agricultural nozzles used in both aerial and ground based agrochemical applications. These methods presented were developed to provide consistent and repeatable droplet size data both inter- and intra-laboratory, when using laser diffraction systems.

When making an application of any crop protection material such as an herbicide or pesticide, the applicator uses a variety of skills and information to ...

New Application of an Atmospheric Pressure Plasma Jet as a Neuro-protective Agent Against Glucose Deprivation-induced Injury of SH-SY5Y Cells

The atmospheric pressure plasma jet (APPJ) has attracted the attention of many researchers from multiple disciplines in recent years because its emissions include multiple types of reactive nitrogen species (RNS) and reactive oxygen species (ROS). Our previous study has shown the cytoprotective effect of the APPJ against oxidative stress-induced injuries. The aim of the present study is to provide a detailed in vitro treatment protocol regarding the neuroprotective applications of helium APPJs on glucose deprivation-induced injury in SH-SY5Y cells. The SH-SY5Y human neuroblastoma-derived cell line was maintained in RPMI 1640 medium supplemented with 15% fetal calf serum. The culture medium was then changed to RPMI 1640 without glucose before APPJ treatment. After a 1 h incubation in a cell incubator, cell viability was determined using Cell Counting Kit 8. The results showed that, compared to the glucose deprivation group, cells treated with APPJ exhibited significantly increased cell viability in a dose-dependent manner, with 8 s/well observed as an optimal dose. Meanwhile, helium flow had no effect on the glucose deprivation-induced cell impairment. Our results indicated that APPJ could be potentially used as a treatment method for the diseases in the central nervous system related to glucose deprivation. This protocol could also be used as a cytoprotective application for other cells with different impairments, but the cell culture and APPJ treatment conditions should be readjusted, and the treatment dose must be relatively low.

A protocol for the neuroprotective application of low-dose atmospheric pressure plasma treatment on glucose deprivation-induced SH-SY5Y injuries.

The atmospheric pressure plasma jet (APPJ) has attracted the attention of many researchers from multiple disciplines in recent years because its emissions ...

Novel Photoacoustic Microscopy and Optical Coherence Tomography Dual-modality Chorioretinal Imaging in Living Rabbit Eyes

Photoacoustic ocular imaging is an emerging ophthalmic imaging technology that can noninvasively visualize ocular tissue by converting light energy into sound waves and is currently under intensive investigation. However, most reported work to date is focused on the imaging of the posterior segment of the eyes of small animals, such as rats and mice, which poses challenges for clinical human translation due to small eyeball sizes. This manuscript describes a novel photoacoustic microscopy (PAM) and optical coherence tomography (OCT) dual-modality system for posterior segment imaging of the eyes of larger animals, such as rabbits. The system configuration, system alignment, animal preparation, and dual-modality experimental protocols for in vivo, noninvasive, label-free chorioretinal imaging in rabbits are detailed. The effectiveness of the method is demonstrated through representative experimental results, including retinal and choroidal vasculature obtained by the PAM and OCT. This manuscript provides a practical guide to reproducing the imaging results in rabbits and advancing photoacoustic ocular imaging in larger animals.

This manuscript describes the novel setup and operating procedure of a photoacoustic microscopy and optical coherence tomography dual-modality system for noninvasive, label-free chorioretinal imaging of larger animals, such as rabbits.

Photoacoustic ocular imaging is an emerging ophthalmic imaging technology that can noninvasively visualize ocular tissue by converting light energy into ...

Application of Design Aspects in Uniaxial Loading Machine Development

In terms of accurate and precise mechanical testing, machines run the continuum. Whereas commercial platforms offer excellent accuracy, they can be cost-prohibitive, often priced in the $100,000 - $200,000 price range. At the other extreme are stand-alone manual devices that often lack repeatability and accuracy (e.g., a manual crank device). However, if a single use is indicated, it is over-engineering to design and machine something overly elaborate. Nonetheless, there are occasions where machines are designed and built in-house to accomplish a motion not attainable with the existing machines in the laboratory. Described in detail here is one such device. It is a loading platform that enables pure uniaxial loading. Standard loading machines typically are biaxial in that linear loading occurs along the axis and rotary loading occurs about the axis. During testing with these machines, a load is applied to one end of the specimen while the other end remains fixed. These systems are not capable of conducting pure axial testing in which tension/compression is applied equally to the specimen ends. The platform developed in this paper enables the equal and opposite loading of specimens. While it can be used for compression, here the focus is on its use in pure tensile loading. The device incorporates commercial load cells and actuators (movers) and, as is the case with machines built in-house, a frame is machined to hold the commercial parts and fixtures for testing.

Here we present a protocol to develop a pure uniaxial loading machine. Critical design aspects are employed to ensure accurate and reproducible testing results.

In terms of accurate and precise mechanical testing, machines run the continuum. Whereas commercial platforms offer excellent accuracy, they can be ...

Automated Delivery of Microfabricated Targets for Intense Laser Irradiation Experiments

Described is an experimental procedure that enables high-power laser irradiation of microfabricated targets. Targets are brought to the laser focus by a closed feedback loop that operates between the target manipulator and a ranging sensor. The target fabrication process is explained in detail. Representative results of MeV-level proton beams generated by irradiation of 600 nm thick gold foils at a rate of 0.2 Hz are given. The method is compared with other replenishable target systems and the prospects of increasing the shot rates to above 10 Hz are discussed.

A protocol is presented for automated irradiation of thin gold foils with high intensity laser pulses. The protocol includes a step-by-step description of the micromachining target fabrication process and a detailed guide for how targets are brought to the laser&#39;s focus at a rate of 0.2 Hz.

Described is an experimental procedure that enables high-power laser irradiation of microfabricated targets. Targets are brought to the laser focus by a ...

Deep Neural Networks for Image-Based Dietary Assessment

Due to the issues and costs associated with manual dietary assessment approaches, automated solutions are required to ease and speed up the work and increase its quality. Today, automated solutions are able to record a person's dietary intake in a much simpler way, such as by taking an image with a smartphone camera. In this article, we will focus on such image-based approaches to dietary assessment. For the food image recognition problem, deep neural networks have achieved the state of the art in recent years, and we present our work in this field. In particular, we first describe the method for food and beverage image recognition using a deep neural network architecture, called NutriNet. This method, like most research done in the early days of deep learning-based food image recognition, is limited to one output per image, and therefore unsuitable for images with multiple food or beverage items. That is why approaches that perform food image segmentation are considerably more robust, as they are able to identify any number of food or beverage items in the image. We therefore also present two methods for food image segmentation - one is based on fully convolutional networks (FCNs), and the other on deep residual networks (ResNet).

The goal of the work presented in this article is to develop technology for automated recognition of food and beverage items from images taken by mobile devices. The technology comprises of two different approaches - the first one performs food image recognition while the second one performs food image segmentation.

Due to the issues and costs associated with manual dietary assessment approaches, automated solutions are required to ease and speed up the work and ...

Particle Image Velocimetry Investigation of Hemodynamics via Aortic Phantom

Aortic valve dysfunction and stroke have recently been reported in transcatheter aortic valve implantation (TAVI) patients. Thrombus in the aortic sinus and neo-sinus due to hemodynamic changes has been suspected. In vitro experiments help investigate the hemodynamic characteristics in the cases where an in vivo assessment proves to be limited. In vitro experiments are also more robust, and the variable parameters are controlled readily. Particle image velocimetry (PIV) is a popular velocimetry method for in vitro studies. It provides a high-resolution velocity field such that even small-scale flow features are observed. The purpose of this study is to show how PIV is used to investigate the flow field in the aortic sinus after TAVI. The in vitro setup of the aortic phantom, TAVI for PIV, and the data acquisition process and post-processing flow analysis are described. The hemodynamic parameters are derived, including the velocity, flow stasis, vortex, vorticity, and particle residence. The results confirm that in vitro experiments and PIV help investigate the hemodynamic features in the aortic sinus.

Particle Image Velocimetry Investigation of Hemodynamics via Aortic Phantom

The present protocol describes particle image velocimetry (PIV) measurements performed to investigate the sinus flow through the in vitro setup of the transcatheter aortic valve (TAV). The hemodynamic parameters based on velocity are also determined.

Aortic valve dysfunction and stroke have recently been reported in transcatheter aortic valve implantation (TAVI) patients. Thrombus in the aortic sinus ...