A.A. received funding support from the Office of Student Research, Yale School of Medicine.
L.J.S. receives grants from the National Institutes of Health (NIH/NCI R01CA206180), Leopoldina Postdoctoral Fellowship, and the Rolf W. Guenther Foundation of Radiological Sciences (Aachen, Germany).
J.C. receives grants from the National Institutes of Health (NIH/NCI R01CA206180), Philips Healthcare, and the German-Israeli Foundation for Scientific Research and Development (Jerusalem, Israel and Neuherberg, Germany); and scholarships from the Rolf W. Guenther Foundation of Radiological Sciences and the Charite Berlin Institute of Health Clinical Scientist Program (Berlin, Germany).
J.S.D. and M.L. receive grants from the National Institutes of Health (NIH/NCI R01CA206180) and Philips Healthcare (Best, The Netherlands).
J.F.G. receives grants from the National Institutes of Health (NIH/NCI R01CA206180), Philips Healthcare, BTG (London, United Kingdom), Boston Scientific (Marlborough, Massachusetts), and Guerbet Healthcare (Villepinte, France)

1 . Workstation Setup for Machine Learning
<ol>
	<li>Use a system with the following: 
	Intel Core 2 Duo or higher CPU at 2.0 GHz 
	4 GB or more system memory 
	POSIX-compliant operating system (Linux or Mac OS) or Microsoft Windows 7 
	User permissions for executing programs and saving files</li>
	<li>Install the following tools: 
	Anaconda Python3: https://www.anaconda.com/download 
	DICOM to NIfTI converter (dcm2niix) - https://github.com/rordenlab/dcm2niix 
	Sublime Text Editor: https://www.sublimetext.com/ 
	itk-SNAP (optional): http://www.itksnap.org
	<ol>
		<li>Install&#160;Anaconda Python3, dcm2nii and Sublime Text, visit their respective websites for operating system specific installation steps.</li>
		<li>Create and activate an Anaconda environment. 
		conda create --name mlenv 
		conda activate mlenv</li>
		<li>Install Anaconda packages for machine learning. 
		conda install numpy scipy scikit-learn nltk nibabel 
		NOTE: The nltk package is useful for parsing plaintext clinical notes, while the nibabel package provides useful functions for medical image manipulation. itk-SNAP may be installed for segmenting organs and tumors from medical images. It is useful for constraining features to specific regions.</li>
	</ol>
	</li>
</ol>
2 . Feature Extraction from Plaintext Clinical Notes and Structured Clinical Data
<ol>
	<li>Create a parent directory for the project and create a folder for each patient within the parent folder. Directory structure should resemble: 
	Project/ 
	Project/Patient_1/ 
	Project/Patient_2/ 
	Project/Patient_3/ 
	...</li>
	<li>Obtain plaintext clinical notes from the electronic medical record (EMR). Retrieve notes manually through the EMR or by means of the hospital information technology (IT) office through a data-dump. Store each patient&#39;s notes in their respective folders. 
	Project/Patient_1/History_and_Physical.txt 
	&#8203;Project/Patient_1/Procedure_Note.txt
	<ol>
		<li>Decided which clinical features to include in the model. Parse the plaintext clinic notes for these features. The Python Natural Language Toolkit (nltk) library provides useful commands for splitting documents into sentences. Each sentence may be searched for appropriate terms such as jaundice. Store each patient&#39;s features in a file with one feature per line. 
		Project/Patient_1/Features.txt: 
		age 67 
		sex male 
		albumin 3.1 
		cirrhotic no 
		hepatitis_c no 
		...</li>
		<li>For non-binary features, take the median value of each feature across all patients. Binarize each feature as a true(1) or false(0) value based on the median value. 
		Project/Patient_1/Binary_Features.txt: 
		age_over_60 0 
		male_sex 1 
		albumin_less_than_3.5 1 
		presence_of_cirrhosis 0 
		hepatitis_c 0 
		...</li>
	</ol>
	</li>
</ol>
3 . Feature Extraction from Medical Images
NOTE: See Step 3 Supplementary Materials for Code Examples.
<ol>
	<li>Download pre- and post- therapy magnetic resonance DICOM images from the hospital PACS. Store images in the corresponding patient folders. 
	Project/ 
	Project/Patient_1/Pre_TACE_MRI_Pre-Contrast.dcm 
	Project/Patient_1/Pre_TACE_MRI_Arterial.dcm 
	Project/Patient_1/Post_TACE_MRI_Pre-Constrast.dcm 
	Project/Patient_1/Post_TACE_MRI_Arterial.dcm</li>
	<li>Convert DICOM images into NIfTI format using the dcm2niix program. The following commands converts all .dcm images in specified folder. Repeat for all patients. 
	dcm2niix Project/Patient_1/ 
	dcm2niix Project/Patient_2/</li>
	<li>Load each NIfTI file into Python. 
	import nibabel 
	&#8203;image = nibabel.load(&#39;Project/Patient_1/Pre_TACE_MRI_Pre-Contrast.dcm&#39;)
	<ol>
		<li>Canonicalize the orientation of each image. This ensures that the x, y, and z axes are identical, irrespective of the machine used to acquire the images. 
		cImage = nibabel.as_closest_canonical(image)</li>
	</ol>
	</li>
	<li>Use itk-SNAP (or an equivalent software package) to segment binary liver and tumor masks for each image. 
	Project/Patient_1/Pre_TACE_MRI_Pre-Contrast_Liver_Mask.bin 
	Project/Patient_1/Pre_TACE_MRI_Pre-Contrast_Tumor_Mask.bin</li>
	<li>Read the liver and tumor masks into Python. The code below demonstrates how to correct orientation issues in order to orient the masks along the same canonical axes as the MR images. 
	import numpy as np 
	with open(liver_mask_file, &#39;rb&#39;) as f: 
	liver_mask = f.read() 
	liver_mask = np.fromstring(liver_mask, dtype=&#39;uint8&#39;) 
	liver_mask = np.reshape(liver_mask, diff.shape, order=&#39;F&#39;) 
	liver_mask = liver_mask[:,::-1,:] 
	&#8203;liver_mask[liver_mask &#62; 0] = 1
	<ol>
		<li>Use the liver and tumor masks to isolate voxels containing liver and tumor. 
		liver = np.copy(cImage) 
		liver[liver_mask &#60;= 0] = 0</li>
	</ol>
	</li>
	<li>Calculate mean liver enhancement feature. 
	&#8203;mean_liver_enhancement = mean(liver)
	<ol>
		<li>Calculate liver volume feature. 
		pixdim = cImage.header[&#39;pixdim&#39;] 
		units = pre.header[&#39;xyzt_units&#39;] 
		dx, dy, dz = pre_pixdim[1:4] 
		liver_volume = length(liver) * dx * dx * dz</li>
		<li>(Optional) Calculate additional features as desired.</li>
	</ol>
	</li>
	<li>Update patient-specific features file with the image features. 
	Project/Patient_1/Features.txt: 
	age 67 
	sex male 
	albumin 3.1 
	cirrhotic no 
	hepatitis_c no 
	pre_tace_mean_liver_enhancement 78 
	&#8203;pre_tace_liver_volume 10000
	<ol>
		<li>Calculate median values for each imaging feature and binarize as in Step 2.2.2. 
		Project/Patient_1/Binary_Features.txt: 
		age_over_60 0 
		male_sex 1 
		albumin_less_than_3.5 1 
		presence_of_cirrhosis 0 
		hepatitis_c 0 
		pre_tace_mean_liver_enhancement 1 
		pre_tace_liver_volume 0</li>
	</ol>
	</li>
</ol>
4. Feature Aggregation and Reduction
NOTE: See Step 4 Supplementary Materials for Code Examples.
<ol>
	<li>Combine the Binary_Features.txt files for each patient into a spreadsheet with patients on the y-axis and features on the x-axis.
	<table fo:keep-together.within-page="1" fo:margin-left="-5mm">
		<tbody>
			<tr>
				<td>Patient</td>
				<td>Age &#62; 60</td>
				<td>Male Sex</td>
				<td>Albumin &#60; 3.5</td>
				<td>Presence of Cirrhosis</td>
				<td>Hepatitis C Present</td>
				<td>mean liver enhancement &#62; 50</td>
				<td>liver volume &#62; 20000</td>
			</tr>
			<tr>
				<td>1</td>
				<td>0</td>
				<td>1</td>
				<td>1</td>
				<td>0</td>
				<td>0</td>
				<td>1</td>
				<td>0</td>
			</tr>
			<tr>
				<td>2</td>
				<td>1</td>
				<td>1</td>
				<td>1</td>
				<td>0</td>
				<td>0</td>
				<td>0</td>
				<td>0</td>
			</tr>
			<tr>
				<td>3</td>
				<td>0</td>
				<td>1</td>
				<td>1</td>
				<td>0</td>
				<td>1</td>
				<td>0</td>
				<td>0</td>
			</tr>
		</tbody>
	</table>
	<ol>
		<li>Add qEASL outcome response labels as the final column.
		<table fo:keep-together.within-page="1" fo:margin-left="-15mm">
			<tbody>
				<tr>
					<td>Patient</td>
					<td>Age &#62; 60</td>
					<td>Male Sex</td>
					<td>Albumin &#60; 3.5</td>
					<td>Presence of Cirrhosis</td>
					<td>Hepatitis C Present</td>
					<td>mean liver enhancement &#62; 50</td>
					<td>liver volume &#62; 20000</td>
					<td>qEASL Responder</td>
				</tr>
				<tr>
					<td>1</td>
					<td>0</td>
					<td>1</td>
					<td>1</td>
					<td>0</td>
					<td>0</td>
					<td>1</td>
					<td>0</td>
					<td>1</td>
				</tr>
				<tr>
					<td>2</td>
					<td>1</td>
					<td>1</td>
					<td>1</td>
					<td>0</td>
					<td>0</td>
					<td>0</td>
					<td>0</td>
					<td>1</td>
				</tr>
				<tr>
					<td>3</td>
					<td>0</td>
					<td>1</td>
					<td>1</td>
					<td>0</td>
					<td>1</td>
					<td>0</td>
					<td>0</td>
					<td>0</td>
				</tr>
			</tbody>
		</table>
		</li>
		<li>Export the spreadsheet as a tab-delimited file. 
		Project/ML_Matrix.tsv: 
		PatientAge &#62; 60&#160; &#160; &#160;Male Sex&#160; &#160; &#160;Albumin &#60; 3.5&#160; &#160; &#160;Presence of Cirrhosis&#160; &#160; &#160;Hepatitis&#160;C&#160;Present&#160; &#160; &#160;mean liver enhancement &#62; 50&#160; &#160; &#160;liver volume &#62; 20000&#160; &#160; &#160;qEASL Responder 
		1&#160; &#160;0&#160; &#160;1&#160; &#160;1&#160; &#160;0&#160; &#160;0&#160; &#160;1&#160; &#160;0&#160; &#160;1 
		2&#160; &#160;1&#160; &#160;1&#160; &#160;1&#160; &#160;0&#160; &#160;0&#160; &#160;0&#160; &#160;0&#160; &#160;1 
		3&#160; &#160;0&#160; &#160;1&#160; &#160;1&#160; &#160;0&#160; &#160;1&#160; &#160;0&#160; &#160;0&#160; &#160;0</li>
	</ol>
	</li>
	<li>Remove low-variance features from consideration. 
	import numpy as np 
	from sklearn.feature_selection import VarianceThreshold 
	 
	#&#160;Read in the binary matrix. 
	features = [] 
	labels = [] 
	for i, L in enumerate(sys.stdin): 
	if i == 0 
	continue 
	n_fs_L = L.strip().split(&#39;\t&#39;) 
	features.append([float(_) for _ in n_fs_L[1:-1]]) 
	labels.append(n_fs_L[-1]) 
	X = np.array(features) 
	y = np.array(labels) 
	 
	# Compute features appearing in at least 20% of both responders and non-respnders. 
	model = VarianceThreshold(threshold=0.8 * (1 - 0.8)) 
	X_new = model.fit_transform(X, y) 
	The male sex, albumin &#60; 3.5, presence of cirrhosis, and liver volume &#62; 2000 features have been removed.
	<table fo:keep-together.within-page="1">
		<tbody>
			<tr>
				<td>Patient</td>
				<td>Age &#62; 60</td>
				<td>Hepatitis C Present</td>
				<td>mean liver enhancement &#62; 50</td>
				<td>qEASL Responder</td>
			</tr>
			<tr>
				<td>1</td>
				<td>0</td>
				<td>0</td>
				<td>1</td>
				<td>1</td>
			</tr>
			<tr>
				<td>2</td>
				<td>1</td>
				<td>0</td>
				<td>0</td>
				<td>1</td>
			</tr>
			<tr>
				<td>3</td>
				<td>0</td>
				<td>1</td>
				<td>0</td>
				<td>0</td>
			</tr>
		</tbody>
	</table>
	</li>
	<li>Remove features with low univariate-association with the outcome. Filter only those features that passed 4.2. Retain ceil(log2(N)) features, where N is the number patients. Ceil(Log2(3)) = 2. 
	 
	import math 
	from sklearn.feature_selection import SelectKBest 
	from sklearn.feature_selection import chi2 
	 
	#&#160;Read in the binary matrix as in 4.2.1 
	... 
	 
	#&#160;Compute top ceil(log2(N)) features by univariate association. 
	k = math.ceil(log2(length(y))) 
	model = SelectKBest(chi2, k=k) 
	X_new = model.fit_transform(X, y) 
	 
	The male sex age &#62; 60 feature has been removed from the remaining features from 4.2.1.</li>
</ol>
<table fo:keep-together.within-page="1">
	<tbody>
		<tr>
			<td>Patient</td>
			<td>Hepatitis C Present</td>
			<td>mean liver enhancement &#62; 50</td>
			<td>qEASL Responder</td>
		</tr>
		<tr>
			<td>1</td>
			<td>0</td>
			<td>1</td>
			<td>1</td>
		</tr>
		<tr>
			<td>2</td>
			<td>0</td>
			<td>0</td>
			<td>1</td>
		</tr>
		<tr>
			<td>3</td>
			<td>1</td>
			<td>0</td>
			<td>0</td>
		</tr>
	</tbody>
</table>
5 . Model Training and Testing
See Step 5 Supplementary Materials for Code Examples
<ol>
	<li>Train a logistic regression model using the binary features matrix from 4.3. 
	import math 
	from sklearn.linear_model import LogisticRegression 
	 
	#&#160;Read in the binary matrix as in 4.2 and 4.3. 
	... 
	 
	#&#160;For each patient, train a model on all other patients. 
	score = 0.0 
	models = [] 
	for patient in len(X): 
	#&#160;Train model on all but one of the patients. 
	train_x = np.array([_ for i, _ in enumerate(X) if i != patient]) 
	train_y = np.array([_ for i, _ in enumerate(y) if i != patient]) 
	model = LogisticRegression(C=1e15) 
	model.fit(train_x, train_y) 
	 
	#&#160;Test on the left-out patient. 
	y_prediction = model.predict(X[patient]) 
	if y_prediction == y[patient]: 
	&#160; &#160; &#160;score += 1 
	&#160; &#160; &#160;models.append(model)</li>
	<li>Train a random forest model using the binary features matrix from 4.2.2. Steps are identical to 5.2.1, except the model instantiation should be updated as follows: 
	from sklearn.ensemble import RandomForestClassifier 
	... 
	model = RandomForestClassifier(n_estimators=100) 
	...</li>
	<li>Print out score / len(X)for 5.1 and 5.2. This represents the average accuracy of all logistic regression models and all random forest models, respectively. All N models should be applied to new patients with the average classification taken as the prediction outcome</li>
</ol>

<ol>
	<li>Benson, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=NCCN+clinical+practice+guidelines+in+oncology:+hepatobiliary+cancers.">NCCN clinical practice guidelines in oncology: hepatobiliary cancers.</a> J National Comprehensive Cancer Network. 7 (4), 350-391 (2009).</li><li>Siegel, R., Miller, K., Jemal, A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cancer+statistics,+2016.">Cancer statistics, 2016.</a> CA Cancer J Clin. 66 (1), 7-30 (2016).</li><li>Bruix, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Clinical+management+of+hepatocellular+carcinoma.+Conclusions+of+the+Barcelona-2000+European+Association+for+the+Study+of+the+Liver+conference.">Clinical management of hepatocellular carcinoma. Conclusions of the Barcelona-2000 European Association for the Study of the Liver conference.</a> Journal of Hepatology. 35 (3), 421-430 (2001).</li><li>Eisenhauer, E., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=New+response+evaluation+criteria+in+solid+tumours:+revised+RECIST+guideline+(version+1.1).">New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1).</a> European Journal of Cancer. 45 (2), 228-247 (2009).</li><li>Gillmore, R., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=EASL+and+mRECIST+responses+are+independent+prognostic+factors+for+survival+in+hepatocellular+cancer+patients+treated+with+transarterial+embolization.">EASL and mRECIST responses are independent prognostic factors for survival in hepatocellular cancer patients treated with transarterial embolization.</a> Journal of Hepatology. 55 (6), 1309-1316 (2011).</li><li>Lin, M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Quantitative+and+volumetric+European+Association+for+the+Study+of+the+Liver+and+Response+Evaluation+Criteria+in+Solid+Tumors+measurements:+feasibility+of+a+semiautomated+software+method+to+assess+tumor+response+after+transcatheter+arterial+chemoembolization.">Quantitative and volumetric European Association for the Study of the Liver and Response Evaluation Criteria in Solid Tumors measurements: feasibility of a semiautomated software method to assess tumor response after transcatheter arterial chemoembolization.</a> Journal of Vascular and Interventional Radiology. 23 (12), 1629-1637 (2012).</li><li>Tacher, V., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Comparison+of+Existing+Response+Criteria+in+Patients+with+Hepatocellular+Carcinoma+Treated+with+Transarterial+Chemoembolization+Using+a+3D+Quantitative+Approach.">Comparison of Existing Response Criteria in Patients with Hepatocellular Carcinoma Treated with Transarterial Chemoembolization Using a 3D Quantitative Approach.</a> Radiology. 278 (1), 275-284 (2016).</li><li>Pedregosa, F., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Scikit-learn:+Machine+Learning+in+Python.">Scikit-learn: Machine Learning in Python.</a> Journal of Machine Learning Research. 12, 2825-2830 (2011).</li><li>Bishop, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Pattern recognition and machine learning. , 738 (2006).</li><li>Alpaydin, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=.">.</a> Introduction to machine learning. Third edition. , 613 (2014).</li><li>Kim, S., Cho, K., Oh, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Development+of+machine+learning+models+for+diagnosis+of+glaucoma.">Development of machine learning models for diagnosis of glaucoma.</a> PLoS One. 12 (5), (2017).</li><li>Son, Y., Kim, H., Kim, E., Choi, S., Lee, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Application+of+support+vector+machine+for+prediction+of+medication+adherence+in+heart+failure+patients.">Application of support vector machine for prediction of medication adherence in heart failure patients.</a> Healthcare Informatics Research. 16 (4), 253-259 (2010).</li><li>Wang, S., Summers, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Machine+learning+and+radiology.">Machine learning and radiology.</a> Medical Image Analysis. 16 (5), 933-951 (2012).</li><li>Abajian, A., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predicting+Treatment+Response+to+Intra-arterial+Therapies+for+Hepatocellular+Carcinoma+with+the+Use+of+Supervised+Machine+Learning-An+Artificial+Intelligence+Concept.">Predicting Treatment Response to Intra-arterial Therapies for Hepatocellular Carcinoma with the Use of Supervised Machine Learning-An Artificial Intelligence Concept.</a> Journal of Vascular and Interventional Radiology. , (2018).</li></ol>

A.A. works as a software consult for Health Fidelity, Inc. that employs similar machine learning techniques on clinical notes for optimizing medical reimbursement.
J.F.G. receives personal fees from Guerbet Healthcare, BTG, Threshold Pharmaceuticals (San Francisco, California), Boston Scientific, and Terumo (Elkton, Maryland); and has a paid consultancy for Prescience Labs (Westport, Connecticut).
None of the other authors have identified a conflict of interest.

Patients with hepatocellular carcinoma who are not candidates for surgical resection are offered intra-arterial therapies. Few methods exist to determine if a patient will respond pre-treatment. Post-treatment evaluation techniques rely upon changes in tumor size or tumor contrast uptake. These are called response criteria, with the most accurate being the Quantitative European Association for the Study of the Liver (qEASL) criterion. qEASL relies upon both volumetric and enhancement changes following therapy to...

Patients with hepatocellular carcinoma who are not surgical candidates are offered intra-arterial therapies1,2,3. There is no single metric that determines whether a patient will respond to an intra-arterial therapy before the treatment is administered. The objective of this study was to demonstrate a method that predicts treatment response by applying methods from machine learning. Such models provide guidance to practitioners and patients when choosing whether to proceed with a treatment.
The protocol entails a reproducible process for training and updating a model starting from primary patient data (clinical notes, demographics, laboratory data, and imaging). The data is initially parsed for specific features, with each patient represented by a set of binary features and a binary outcome target label. The outcome label is determined using an established imaging-based response criterion for hepatocellular therapy4,5,6,7. The features and target labels are passed to machine learning software that learns the mapping between features and outcomes under a specific learning model (logistic regression or random forest)8,9,10. Similar techniques have been applied in radiology and other areas of cancer research for diagnosis and treatment prediction11,12,13.
The method adapts techniques from computer science to the field of interventional radiology. Traditional significance studies in interventional radiology, and medicine in general, rely upon mono- or oligo- feature analyses. For example, the Model for End-Stage Liver Disease incorporates five clinical metrics to assess the extent of liver disease. The benefit of the proposed method is the ability to add features liberally; twenty-five features are considered in the example analysis. Additional features may be added as desired.
The technique may be applied to other radiographic interventions where pre- and post- intervention imaging data are available. For example, outcomes following percutaneous treatments could be predicted in a similar manner. The main limitation of the study is the need to manual curate features for inclusion in the model. Data curation and feature extraction is time-consuming for the practitioner and may impede clinical adoption of such machine learning models.

<table><tbody><tr><td>Computer workstation</td><td>N/A</td><td>N/A</td><td>Intel Core 2 Duo or higher CPU at 2.0 GHz; 4 GB or more system memory; POSIX-compliant operating system (Linux or Mac OS) or Microsoft Windows 7; User permissions for executing programs and saving files</td></tr><tr><td>Anaconda Python 3</td><td>Anaconda, Inc.</td><td>Version 3.6</td><td>Python 3 system and libraries packaged for scientists and researchers</td></tr><tr><td>DICOM to NIfTI</td><td>NeuroImaging Tools &amp;amp; Resources Collaboratory</td><td>Version 1.0 (4/4/2018 release)</td><td>Standalone program for converting DICOM imaging files to NIfTI format</td></tr><tr><td>Sublime Text Editor</td><td>Sublime HQ Pty Ltd</td><td>Version 3 (Build 3143)</td><td>Text-editor for writing Python code</td></tr><tr><td>Required Python Libraries</td><td>N/A</td><td>Version 3.2.25 (nltk)&lt;br/&gt; Version 0.19.1 (scikit-learn)</td><td>Natural Language Toolkit (nltk)&lt;br/&gt; Scikit-learn</td></tr><tr><td>ITK-SNAP</td><td>N/A</td><td>Version 3.6.0</td><td>Optional toolkit for performing segmentation of organ systems in medical images.</td></tr></tbody></table>

predicting treatment response to image-guided therapies using machine learning: an example for trans-arterial treatment of hepatocellular carcinoma

Intra-arterial therapies are the standard of care for patients with hepatocellular carcinoma who cannot undergo surgical resection. The objective of this study was to develop a method to predict response to intra-arterial treatment prior to intervention.
The method provides a general framework for predicting outcomes prior to intra-arterial therapy. It involves pooling clinical, demographic and imaging data across a cohort of patients and using these data to train a machine learning model. The trained model is applied to new patients in order to predict their likelihood of response to intra-arterial therapy.
The method entails the acquisition and parsing of clinical, demographic and imaging data from N patients who have already undergone trans-arterial therapies. These data are parsed into discrete features (age, sex, cirrhosis, degree of tumor enhancement, etc.) and binarized into true/false values (e.g., age over 60, male gender, tumor enhancement beyond a set threshold, etc.). Low-variance features and features with low univariate associations with the outcome are removed. Each treated patient is labeled according to whether they responded or did not respond to treatment. Each training patient is thus represented by a set of binary features and an outcome label. Machine learning models are trained using N - 1 patients with testing on the left-out patient. This process is repeated for each of the N patients. The N models are averaged to arrive at a final model.
The technique is extensible and enables inclusion of additional features in the future. It is also a generalizable process that may be applied to clinical research questions outside of interventional radiology. The main limitation is the need to derive features manually from each patient. A popular modern form of machine learning called deep learning does not suffer from this limitation, but requires larger datasets.

The proposed method was applied to 36 patients who had undergone trans-arterial therapies for hepatocellular carcinoma. Twenty-five features were identified and binarized using steps 1-5. Five features satisfied both the variance and univariate association filters (see steps 5.1 and 5.2) and were used for model training. Each patient was labeled as either a responder or non-responder under the qEASL response criteria. The features matrix was thus a 36 x 5 array while the target labels vec...

Watch this Scientific Journal Video about Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma at JoVE.co

Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma

Intra-arterial therapies are the standard of care for patients with hepatocellular carcinoma who cannot undergo surgical resection. A method for predicting response to these therapies is proposed. The technique uses pre-procedural clinical, demographic, and imaging information to train machine learning models capable of predicting response prior to treatment.

Intra-arterial therapies are the standard of care for patients with hepatocellular carcinoma who cannot undergo surgical resection. The objective of this ...

predicting-treatment-response-to-image-guided-therapies-using-machine

Research

JoVE Journal

Medicine

8.2K Views.  Yale School of Medicine. Deze methode kan helpen bij het voorspellen van de respons op intra arteriële therapieën. Machine learning-toepassingen in interventionele oncologie veranderen de manier waarop we leverkanker en andere ziekten behandelen met beeldgestuurde therapieën. Deze methode heeft het potentieel om de manier waarop twee belangrijke beslissingen worden genomen te verbeteren.Over het algemeen zullen personen die nieuw zijn bij deze methode worstelen vanwege de noodzaak om de functies te generere...

Video: Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma

Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model

Machine learning (ML) algorithms permit the integration of different features into a model to perform classification or regression tasks with an accuracy exceeding its constituents. This protocol describes the development of an ML algorithm to predict the growth of breast cancer bone macrometastases in a rat model before any abnormalities are observable with standard imaging methods. Such an algorithm can facilitate the detection of early metastatic disease (i.e., micrometastasis) that is regularly missed during staging examinations.
The applied metastasis model is site-specific, meaning that the rats develop metastases exclusively in their right hind leg. The model&#8217;s tumor-take rate is 60%&#8211;80%, with macrometastases becoming visible in magnetic resonance imaging (MRI) and positron emission tomography/computed tomography (PET/CT) in a subset of animals 30 days after induction, whereas a second subset of animals exhibit no tumor growth.
Starting from image examinations acquired at an earlier time point, this protocol describes the extraction of features that indicate tissue vascularization detected by MRI, glucose metabolism by PET/CT, and the subsequent determination of the most relevant features for the prediction of macrometastatic disease. These features are then fed into a model-averaged neural network (avNNet) to classify the animals into one of two groups: one that will develop metastases and the other that will not develop any tumors. The protocol also describes the calculation of standard diagnostic parameters, such as overall accuracy, sensitivity, specificity, negative/positive predictive values, likelihood ratios, and the development of a receiver operating characteristic. An advantage of the proposed protocol is its flexibility, as it can be easily adapted to train a plethora of different ML algorithms with adjustable combinations of an unlimited number of features. Moreover, it can be used to analyze different problems in oncology, infection, and inflammation.

This protocol was designed to train a machine learning algorithm to use a combination of imaging parameters derived from magnetic resonance imaging (MRI) and positron emission tomography/computed tomography (PET/CT) in a rat model of breast cancer bone metastases to detect early metastatic disease and predict subsequent progression to macrometastases.

Machine learning (ML) algorithms permit the integration of different features into a model to perform classification or regression tasks with an accuracy ...

Cancer Research

Forecasting Hepatocellular Carcinoma Mortality using a Weighted Regression Model to Estimate Cohort Effects in Taiwan

To eliminate the influence of age and period in age cycle contingency table data, a multistage method was adopted to evaluate the cohort effect. The most general primary malignant tumor of the liver is hepatocellular carcinoma (HCC). HCC is associated with liver cirrhosis with alcohol and viral etiologies. In epidemiology, long-term trends in HCC mortality were delineated (or forecasted) by using an age-period-cohort (APC) model. The HCC deaths were determined for each cohort with its weighted influence. The confidence interval (CI) of the weighted mean is fairly narrow (compared to the equally weighted estimates). Due to the fairly narrow CI with less uncertainty, the weighted mean estimation was used as a means for forecasting. With the multistage method, it is recommended to use weighted mean estimation based on a regression model to evaluate the cohort effect in the age-period contingency table data.

We depict a multistage method to measure a cohort effect with age data, thereby allowing data to be eliminated in many situations without sacrificing data quality. The protocol demonstrates the strategy and provides a weighted regression model for analyzing the hepatocellular carcinoma data.

To eliminate the influence of age and period in age cycle contingency table data, a multistage method was adopted to evaluate the cohort effect. The most ...

Dual-phase Cone-beam Computed Tomography to See, Reach, and Treat Hepatocellular Carcinoma during Drug-eluting Beads Transarterial Chemo-embolization

The advent of cone-beam computed tomography (CBCT) in the angiography suite has been revolutionary in interventional radiology. CBCT offers 3 dimensional&#160;(3D) diagnostic imaging in the interventional suite and can enhance minimally-invasive therapy beyond the limitations of 2D angiography alone. The role of CBCT has been recognized in transarterial chemo-embolization (TACE) treatment of hepatocellular carcinoma (HCC). The recent introduction of a CBCT technique: dual-phase CBCT (DP-CBCT) improves intra-arterial HCC treatment with drug-eluting beads (DEB-TACE). DP-CBCT can be used to localize liver tumors with the diagnostic accuracy of multi-phasic multidetector computed tomography (M-MDCT) and contrast enhanced magnetic resonance imaging (CE-MRI) (See the tumor), to guide intra-arterially guidewire and microcatheter to the desired location for selective therapy (Reach the tumor), and to evaluate treatment success during the procedure (Treat the tumor). The purpose of this manuscript is to illustrate how DP-CBCT is used in DEB-TACE to see, reach, and treat HCC.

Dual-phase cone-beam computed tomography (DP-CBCT) is a useful intraprocedural imaging technique for transarterial chemo-embolization treatment with drug-eluting beads of hepatocellular carcinoma. DP-CBCT has been used to perform three major steps in oncologic interventional radiology: tumor localization (see), navigation and intraprocedural catheter guidance (reach), and intraprocedural evaluation of treatment success (treat).

The advent of cone-beam computed tomography (CBCT) in the angiography suite has been revolutionary in interventional radiology. CBCT offers 3 ...

A Three-Dimensional Spheroid Model to Investigate the Tumor-Stromal Interaction in Hepatocellular Carcinoma

The aggressiveness and lack of well-tolerated and widely effective treatments for advanced hepatocellular carcinoma (HCC), the predominant form of liver cancer, rationalize its rank as the second most common cause of cancer-related death. Preclinical models need to be adapted to recapitulate the human conditions to select the best therapeutic candidates for clinical development and aid the delivery of personalized medicine. Three-dimensional (3D) cellular spheroid models show promise as an emerging in vitro alternative to two-dimensional (2D) monolayer cultures. Here, we describe a 3D tumor spheroid&#160;model which exploits the ability of individual cells to aggregate when maintained in hanging droplets, and is more representative of an in vivo environment than standard monolayers. Furthermore, 3D spheroids can be produced by combining homotypic or heterotypic cells, more reflective of the cellular heterogeneity in vivo, potentially enabling the study of environmental interactions that can influence progression and treatment responses. The current research optimized the cell density to form 3D homotypic and heterotypic tumor spheroids by immobilizing cell suspensions on the lids of standard 10 cm3 Petri dishes. Longitudinal analysis was performed to generate growth curves for homotypic versus heterotypic tumor/fibroblasts spheroids. Finally, the proliferative impact of fibroblasts (COS7 cells) and liver myofibroblasts (LX2) on homotypic tumor (Hep3B) spheroids was investigated. A seeding density of 3,000 cells (in 20 &#181;L media) successfully yielded Huh7/COS7 heterotypic spheroids, which displayed a steady increase in size up to culture day 8, followed by growth retardation. This finding was corroborated using Hep3B homotypic spheroids cultured in LX2 (human hepatic stellate cell line) conditioned medium (CM). LX2 CM triggered the proliferation of Hep3B spheroids compared to control tumor spheroids. In conclusion, this protocol has shown that 3D tumor spheroids can be used as a simple, economical, and prescreen in vitro tool to study tumor-stromal interactions more comprehensively.

Comprehensive in vitro models that faithfully recapitulate the relevant human disease are lacking. The current study presents three-dimensional (3D) tumor spheroid creation and culture, a reliable in vitro tool to study the tumor-stromal interaction in human hepatocellular carcinoma.

The aggressiveness and lack of well-tolerated and widely effective treatments for advanced hepatocellular carcinoma (HCC), the predominant form of liver ...

Non-Invasive PET/MR Imaging in an Orthotopic Mouse Model of Hepatocellular Carcinoma

Preclinical experimental models of hepatocellular carcinoma (HCC) that recapitulate human disease represent an important tool to study tumorigenesis and evaluate novel therapeutic approaches. Non-invasive whole-body imaging using positron emission tomography (PET) provides critical insights into the in vivo characteristics of tissues at the molecular level in real-time. We present here a protocol for orthotopic HCC xenograft creation with and without hepatic artery ligation (HAL) to induce tumor hypoxia and the assessment of their tumor metabolism in vivo using [18F]Fluoromisonidazole ([18F]FMISO) and [18F]Fluorodeoxyglucose ([18F]FDG) PET/magnetic resonance (MR) imaging. Tumor hypoxia could be readily visualized using the hypoxia marker [18F]FMISO, and it was found that the [18F]FMISO uptake was higher in HCC mice that underwent HAL than in the non-HAL group, whereas [18F]FDG could not distinguish tumor hypoxia between the two groups. HAL tumors also displayed a higher level of hypoxia-inducible factor (HIF)-1&#945; expression in response to hypoxia. Quantification of HAL tumors showed a 2.3-fold increase in [18F]FMISO uptake based on the standardized value uptake (SUV) approach.

Here, we present a protocol to create orthotopic hepatocellular carcinoma xenografts with and without hepatic artery ligation and perform non-invasive positron emission tomography (PET) imaging of tumor hypoxia using [18F]Fluoromisonidazole ([18F]FMISO) and [18F]Fluorodeoxyglucose ([18F]FDG).

Preclinical experimental models of hepatocellular carcinoma (HCC) that recapitulate human disease represent an important tool to study tumorigenesis and ...

Prognostic Significance of Immune Cell Infiltration in Hepatocellular Carcinoma

Mast Cells in the Microenvironment of Hepatocellular Carcinoma Confer Favorable Prognosis: A Retrospective Study using QuPath Image Analysis Software

The insights provided by in-situ detection of immune cells within hepatocellular carcinoma (HCC) might present information on patient outcomes. Studies investigating the expression and localization of immune cells within tumor tissues are associated with several challenges, including a lack of precise annotation for tumor regions and random selection of microscopic fields of view. QuPath is an open-source, user-friendly software that could meet the growing need for digital pathology in whole-slide image (WSI) analysis.
The infiltration of HCC and adjacent tissues by CD1a+ immature dendritic cells (iDCs), CD117+ mast cells, and NKp46+ natural killer cells (NKs) cells was assessed immunohistochemically in representative specimens of 67 patients with HCC who underwent curative resection. The area fraction (AF) of positively stained cells was assessed automatically in WSIs using QuPath in the tumor center (TC), inner margin (IM), outer margin (OM), and peritumor (PT) area. The prognostic significance of immune cells was evaluated for time to recurrence (TTR), disease-free survival (DFS), and overall survival (OS).
The AF of mast cells was significantly greater than the AF of NKs, and the AF of iDCs was significantly lower compared to NKs in each region of interest. High AFs of mast cells in the IM and PT areas were associated with longer DFS. In addition, high AF of mast cells in IM was associated with longer OS.
Computer-assisted analysis using this software is a suitable tool for obtaining prognostic information for tumor-infiltrating immune cells (iDCs, mast cells, and NKs) in different regions of HCC after resection. Mast cells displayed the greatest AF in all regions of interest (ROIs). Mast cells in the peritumor region and IM showed a positive prognostic significance.

Author Spotlight: Investigating Immune Cell Dynamics in the Tumor Microenvironment &#8212; Challenges and Innovations in Cancer Prognosis

The presence of mast cells in the inner margin and peritumor areas of hepatocellular carcinoma after resection confers a favorable prognosis. This study endorses QuPath image analysis software as a promising platform that could meet the need for reproducibility, consistency, and accuracy in digital pathology.

The insights provided by in-situ detection of immune cells within hepatocellular carcinoma (HCC) might present information on patient outcomes. Studies ...

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of constructing an orbital computed tomography (CT) segmentation model based on U-Net. For supervised learning, a labor-intensive and time-consuming process is required. The method of labeling with super-resolution to efficiently mask the ground truth on orbital CT images is introduced. Also, the volume of interest is cropped as part of the pre-processing of the dataset. Then, after extracting the volumes of interest of the orbital structures, the model for segmenting the key structures of the orbital CT is constructed using U-Net, with sequential 2D slices that are used as inputs and two bi-directional convolutional long-term short memories for conserving the inter-slice correlations. This study primarily focuses on the segmentation of the eyeball, optic nerve, and extraocular muscles. The evaluation of the segmentation reveals the potential application of segmentation to orbital CT images using deep learning methods.

Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography

An object segmentation protocol for orbital computed tomography (CT) images is introduced. The methods of labeling the ground truth of orbital structures by using super-resolution, extracting the volume of interest from CT images, and modeling multi-label segmentation using 2D sequential U-Net for orbital CT images are explained for supervised learning.

Recently, deep learning-based segmentation models have been widely applied in the ophthalmic field. This study presents the complete process of ...

Engineering

Treatment of Liver Metastases Using an Internal Target Volume Method for Stereotactic Body Radiotherapy

The prognosis of patients with metastatic cancers has improved in the past decades due to effective chemotherapy and oligometastatic surgery. For inoperable patients, local ablation therapies, such as stereotactic body radiotherapy (SBRT), can provide effective local tumor control with minimal toxicity. Because of its high precision and accuracy, SBRT delivers a higher radiation dose per fraction, is more effective, and targets smaller irradiation volumes than does conventional radiotherapy. In addition, steep dose gradients from target lesions to surrounding normal tissues are achieved using SBRT; thus, SBRT provides more effective tumor control and exhibits fewer side effects than conventional radiotherapy. The use of SBRT is prevalent for treating intracranial lesions (known as stereotactic radiosurgery); however, it is now also used for treating spinal and adrenal metastases. Because of advancements in image-guided assistance and respiratory motion management, several studies have investigated the use of SBRT for treating lung or liver tumors, which move as a patient breathes. The results of these studies have suggested that SBRT favorably controls tumors in the case of moving lesions.
Four-dimensional computed tomography (4D-CT) with an abdominal compressor (AC) is clinically convenient for effective respiratory motion management. Because this method is noninvasive and allows free breathing, its use reduces complications. Furthermore, patients consider this method convenient. Moreover, it is considered more efficient than other methods of respiratory motion management by physicians and therapists. The use of 4D-CT with an AC for treating pulmonary lesions has also been widely investigated, and the technique is gaining acceptance for treating hepatic lesions. However, the protocols for using 4D-CT with an AC for treating hepatic lesions are different from those used for treating pulmonary lesions. In this article, we describe a new protocol for SBRT with 4D-CT and an AC for treating liver metastases.

Stereotactic body radiotherapy (SBRT) requires rigorous accuracy and precision for delivering high radiation doses per fraction to small treatment volumes to improve tumor control and simultaneously reduce toxicity. Herein, we present a noninvasive and clinically convenient respiratory motion management protocol for SBRT for liver metastases.

The prognosis of patients with metastatic cancers has improved in the past decades due to effective chemotherapy and oligometastatic surgery. For ...

Novel In Vivo Micro-Computed Tomography Imaging Techniques for Assessing the Progression of Non-Alcoholic Fatty Liver Disease

Non-alcoholic fatty liver disease (NAFLD) is a growing global health issue, and the impact of NAFLD is compounded by the current lack of effective treatments. Considerable limiting factors hindering the timely and accurate diagnosis (including grading) and monitoring of NAFLD, as well as the development of potential therapies, are the current inadequacies in the characterization of the hepatic microenvironment structure and the scoring of the disease stage in a spatiotemporal and non-invasive manner. Using a diet-induced NAFLD mouse model, we investigated the use of in vivo micro-computed tomography (CT) imaging techniques as a non-invasive method to assess the progression stages of NAFLD, focusing predominantly on the hepatic vascular network due to its significant involvement in NAFLD-related hepatic dysregulation. This imaging methodology allows for longitudinal analysis of liver steatosis and functional tissue uptake, as well as the evaluation of the relative blood volume, portal vein diameter, and density of the vascular network. Understanding the adaptations of the hepatic vascular network during NAFLD progression and correlating this with other ways of characterizing the disease progression (steatosis, inflammation, fibrosis) using the proposed method can pave the way toward the establishment of new, more efficient, and reproducible approaches for NAFLD research in mice. This protocol is also expected to upgrade the value of preclinical animal models for investigating&#160;the development of novel therapies against disease progression.

Novel In Vivo Micro-Computed Tomography Imaging Techniques for Assessing the Progression of Non-Alcoholic Fatty Liver Disease

Using a diet-induced non-alcoholic fatty liver disease (NAFLD) mouse model, we describe the use of novel in vivo micro-computed tomography imaging techniques as a non-invasive method to assess the progression stages of NAFLD, focusing predominantly on the hepatic vascular network due to its significant involvement in NAFLD-related hepatic dysregulation.

Non-alcoholic fatty liver disease (NAFLD) is a growing global health issue, and the impact of NAFLD is compounded by the current lack of effective ...

Biology

Trans-Arterial Indocyanine Green Staining for Precise Anatomical Hepatectomy of Liver Segments VII and VIII

Anatomical hepatectomy for resectable hepatocellular carcinoma (HCC) is performed with precision using the Glissonean system, ultrasound imaging, and indocyanine green (ICG) fluorescence staining to define liver segments, enhancing surgical radicality and preserving liver volume. However, laparoscopic clamping and ultrasound-guided ICG staining for deep-seated tumors in liver segments VII and VIII pose challenges due to the depth of the Glissonean ducts and other technical limitations. This study aimed to overcome these obstacles by exploring a protocol using trans-arterial ICG staining, a technique specifically tailored for complex anatomical hepatectomy of liver segments VII and VIII. In this method, an interventionalist accessed the right femoral artery and advanced to the celiac trunk for arteriography, followed by strategic placement of a microcatheter into the tumor's blood supply vessel to facilitate surgical resection. During the operation, ICG was injected through the microcatheter to fluorescently label the tumor's liver segment, enabling precise anatomical resection under fluorescence guidance. This trans-arterial ICG staining approach allows for the accurate identification of tumor subsegments, facilitates complete resection, and optimizes liver function preservation, ultimately improving oncological outcomes without increasing surgical complication rates.

Laparoscopic anatomical hepatectomy has gained global recognition in liver surgery. However, delineating tumor boundaries in liver segments VII and VIII presents distinct technical challenges. This protocol utilizes trans-arterial indocyanine green (ICG) staining to accurately identify liver subsegments, facilitating complete tumor resection while preserving functional liver volume.

Anatomical hepatectomy for resectable hepatocellular carcinoma (HCC) is performed with precision using the Glissonean system, ultrasound imaging, and ...

Predicting Treatment Response to Image-Guided Therapies Using Machine Learning: An Example for Trans-Arterial Treatment of Hepatocellular Carcinoma

Samenvatting

Meer video's verkennen

Hoofdstukken in deze video