Name: an r-based landscape validation of a competing risk model
Uploaded: 2022-09-16T13:10:00
Description: Watch this Scientific Journal Video about An R-Based Landscape Validation of a Competing Risk Model at JoVE.com

The study was supported by grants from the Medical Science &#38; Technology Plan Project of Zhejiang Province (grant numbers 2013KYA212), the general program of Zhejiang Province Natural Science Foundation (grant number Y19H160126), and the key program of the Jinhua Municipal Science &#38; Technology Bureau (grant number 2016-3-005, 2018-3-001d, and 2019-3-013).

The Surveillance, Epidemiology, and End Results (SEER) database is an open-access cancer database that only contains deidentified patient data (SEER ID: 12296-Nov2018). Therefore, this study was exempted from the approval of the review board of the Affiliated Jinhua Hospital, Zhejiang University School of Medicine.
1. Data preparation and R packages preparation
<ol>
	<li>Prepare and import the data. 
	&#62; Dataset &#60;- read.csv(&#34;&#8230;/Breast cancer Data.xls...

<ol>
	<li>Andersen, P. K., Gill, R. D. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Cox's+regression+model+for+counting+processes:+A+large+sample+study.">Cox's regression model for counting processes: A large sample study.</a> The Annals of Statistics. 10 (4), 1100-1120 (1982).</li><li>Lubsen, J., Pool, J., vander Does, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+practical+device+for+the+application+of+a+diagnostic+or+prognostic+function.">A practical device for the application of a diagnostic or prognostic function.</a> Methods of Information in Medicine. 17 (2), 127-129 (1978).</li><li>Harrell, F. E., Lee, K. L., Mark, D. B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Multivariable+prognostic+models:+Issues+in+developing+models,+evaluating+assumptions+and+adequacy,+and+measuring+and+reducing+errors.">Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.</a> Statistics In Medicine. 15 (4), 361-387 (1996).</li><li>Hung, H., Chiang, C. -. T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Estimation+methods+for+time-dependent+AUC+models+with+survival+data.">Estimation methods for time-dependent AUC models with survival data.</a> The Canadian Journal of Statistics / La Revue Canadienne de Statistique. 38 (1), 8-26 (2010).</li><li>Moons, K. G. M., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Risk+prediction+models:+I.+Development,+internal+validation,+and+assessing+the+incremental+value+of+a+new+(bio)marker.">Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker.</a> Heart. 98 (9), 683-690 (2012).</li><li>Fu, J., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Real-world+impact+of+non-breast+cancer-specific+death+on+overall+survival+in+resectable+breast+cancer.">Real-world impact of non-breast cancer-specific death on overall survival in resectable breast cancer.</a> Cancer. 123 (13), 2432-2443 (2017).</li><li>Fine, J. P., Gray, R. J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=A+proportional+hazards+model+for+the+subdistribution+of+a+competing+risk.">A proportional hazards model for the subdistribution of a competing risk.</a> Journal of the American Statistical Association. 94 (446), 496-509 (1999).</li><li>Wu, L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Establishing+a+competing+risk+regression+nomogram+model+for+survival+data.">Establishing a competing risk regression nomogram model for survival data.</a> Journal of Visualized Experiments. (164), e60684 (2020).</li><li>Zhang, Z., Geskus, R. B., Kattan, M. W., Zhang, H., Liu, T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Nomogram+for+survival+analysis+in+the+presence+of+competing+risks.">Nomogram for survival analysis in the presence of competing risks.</a> Annals of Translational Medicine. 5 (20), 403 (2017).</li><li>Zhang, Z. H., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Overview+of+model+validation+for+survival+regression+model+with+competing+risks+using+melanoma+study+data.">Overview of model validation for survival regression model with competing risks using melanoma study data.</a> Annals Of Translational Medicine. 6 (16), 325 (2018).</li><li>Newson, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Confidence+intervals+for+rank+statistics:+Somers'+D+and+extensions.">Confidence intervals for rank statistics: Somers' D and extensions.</a> Stata Journal. 6 (3), 309-334 (2006).</li><li>Davison, A. C., Hinkley, D. V., Schechtman, E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Efficient+bootstrap+simulation.">Efficient bootstrap simulation.</a> Biometrika. 73 (3), 555-566 (1986).</li><li>Roecker, E. B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Prediction+error+and+its+estimation+for+subset-selected+models.">Prediction error and its estimation for subset-selected models.</a> Technometrics. 33 (4), 459-468 (1991).</li><li>Steyerberg, E. W., Harrell, F. E. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Prediction+models+need+appropriate+internal,+internal-external,+and+external+validation.">Prediction models need appropriate internal, internal-external, and external validation.</a> Journal of Clinical Epidemiology. 69, 245-247 (2016).</li><li>Zhang, Z., Chen, L., Xu, P., Hong, Y. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed&cmd=Search&doptcmdl=Citation&defaultField=Title+Word&term=Predictive+analytics+with+ensemble+modeling+in+laparoscopic+surgery:+A+technical+note.">Predictive analytics with ensemble modeling in laparoscopic surgery: A technical note.</a> Laparoscopic, Endoscopic and Robotic Surgery. 5 (1), 25-34 (2022).</li></ol>

This study compared competing risk nomograms established by two distinct methods and conducted evaluation and validation of the established nomograms. Specifically, this study provided a step-by-step tutorial for establishing the nomogram based on a direct method, as well as calculating the C-index and plotting the calibration curves.
The rms package in R software is widely used for the construction and evaluation of Cox proportional hazard models, but it is not applicable for competi...

In recent years, emerging prognostic factors have been identified with the development of precision medicine, and prognostic models combining molecular and clinicopathological factors are drawing increasing attention in clinical settings. However, non-graphical models, such as the Cox proportional hazard model, with results of coefficient values, are difficult for clinicians to understand1. In comparison, a nomogram is a visualization tool of regression models (including the Cox regression model, competing risk model, etc.), a two-dimensional diagram designed for the approximate graphical computation of a mathematical function2. It enables the valuation of different levels of variables in a clinical model and the calculation of risk scores (RS) to predict prognosis.
Model evaluation is essential in model construction, and two characteristics are generally accepted for evaluation: discrimination and calibration. In clinical models, discrimination refers to the ability of a model to separate individuals who develop events from those who do not, such as patients who die versus those who remain alive, and the concordance index (C-index) or the area under the receiver operating characteristic curve (AUC) are typically used to characterize it3,4. Calibration is a process of comparing the predicted probabilities of a model with the actual probabilities, and calibration curves have been widely used to represent it. In addition, model validation (internal and external validation) is an important step in model construction, and only validated models can be further extrapolated5.
The Cox proportional hazard model is a regression model used in medical research for investigating the associations between prognostic factors and survival status. However, the Cox proportional hazard model only considers two statuses of outcome [Y (0, 1)], while study subjects often face more than two statuses, and competing risks arise [Y (0, 1, 2)]1. Overall survival (OS), which is defined as the time from the date of origin (e.g., treatment) to the date of death due to any cause, is the most important endpoint in survival analysis. However, the OS fails to differentiate cancer-specific death from non-cancer-specific death (e.g., cardiovascular events and other unrelated causes), thus ignoring competing risks6. In these situations, the competing risk model is preferred for the prediction of survival status with the consideration of competing risks7. The methodology of constructing and validating Cox proportional hazard models is well-established, while there have been few reports regarding the validation of competing risk models.
In our previous study, a specific competing risk nomogram, a combination of a nomogram and competing risk model, and a risk score estimation based on a competing risk model were established8. This study aims to present different methods of evaluation and validation of the established competing risk nomogram, which should serve as a useful tool for clinicians to predict prognosis with the consideration of competing risks.

<table border="1" fo:keep-together.within-page="1" fo:keep-with-next.within-page="always">
	<tbody>
		<tr>
			<td>R software</td>
			<td>None</td>
			<td>Not Applicable</td>
			<td>Version 3.6.2 or higher&#160;</td>
		</tr>
		<tr>
			<td>Computer system</td>
			<td>Microsoft&#160;</td>
			<td>Windows 10</td>
			<td>&#160;Windows 10 or higher</td>
		</tr>
	</tbody>
</table>

an r-based landscape validation of a competing risk model

The Cox proportional hazard model is widely applied for survival analyses in clinical settings, but it is not able to cope with multiple survival outcomes. Different from the traditional Cox proportional hazard model, competing risk models consider the presence of competing events and their combination with a nomogram, a graphical calculating device, which is a useful tool for clinicians to conduct a precise prognostic prediction. In this study, we report a method for establishing the competing risk nomogram, that is, the evaluation of its discrimination (i.e., concordance index and area under the curve) and calibration (i.e., calibration curves) abilities, as well as the net benefit (i.e., decision curve analysis). In addition, internal validation using bootstrap resamples of the original dataset and external validation using an external dataset of the established competing risk nomogram were also performed to demonstrate its extrapolation ability. The competing risk nomogram should serve as a useful tool for clinicians to predict prognosis with the consideration of competing risks.

In this study, data of patients with breast cancer were retrieved from the SEER database and served as example data. The SEER database provides data on cancer representing around 34.6% of the United States population, and permission to access the database was obtained (reference number 12296-Nov2018).
Two nomograms (Figure 1), both including histological type, differentiated grade, T stage, and N stage, were established using the direct method and the weighted met...

Watch this Scientific Journal Video about An R-Based Landscape Validation of a Competing Risk Model at JoVE.com

An R-Based Landscape Validation of a Competing Risk Model

The present protocol describes codes in R for evaluating the discrimination and calibration abilities of a competing risk model, as well as codes for the internal and external validation of it.

The Cox proportional hazard model is widely applied for survival analyses in clinical settings, but it is not able to cope with multiple survival ...

Our protocol presents different methods of evaluation and validation of competing risk nomogram;which considers the presence of competing events in survival analysis. This protocol serves as a supplemental to the risk regression package in R.Such as the calculation of C-index and the internal validation, using bootstrap method. To begin for C-index discrimination, fit the matrix cob into the competing risk model mod CRR and get a predicted matrix SUV by executing the command.Get the cumulative incidences in a certain month from SUV and calculate the C-index with the function R-core sends. For AUC discrimination, score the predictive performance of the competing risk model, using the function score, risk regression package. Then extract the AUC from the score by executing the command.To obtain calibration curves with a 95%confidence interval of the competing risk model, get a data frame with the cumulative incidences of each individual in a certain failure time. Then divide the cohort according to the estimated cumulative incidences into five subgroups and calculate the average predicted cumulative incidences of each subgroup. Calculate the observed cumulative incidences that is the actual cumulative incidences using the function cuminc.And then get the observed cumulative incidences with a 95%confidence interval in a certain failure time by executing the command. Plot the calibration curve with the predicted cumulative incidences as the X axis and the observed cumulative incidences as the Y axis. Using the function gg plot.For calibration curve with risk scores of the competing risk model, valuate each level of all the variables and obtain the total RS by executing the command. Count the frequencies and calculate the observed cumulative incidences of the different total risk scores. Set the range of the X axis and calculate the predicted cumulative incidences of the total risk scores.Then plot the calibration curve with risk scores by executing the command. To get the average predicted cumulative incidences using the bootstrap method, re-sample the original dataset, dataset with replace, to generate the bootstrap dataset. Dataset in.Then establish a competing risk model:mod NCRR with the bootstrap dataset and use the function predict CRR to predict mod NCRR in loop B times to generate SUV all in. Next, get the average predicted cumulative incidences in a certain month. Calculate the C-index using interval cross validation with the function R-core sends.For calibration, using external validation, get the predicted cumulative incidences using external data and cumulative incidences with the matrix of external data variables:Code-X by executing the command. Then calculate the C-index using external validation By executing the command. Two nomograms were obtained using the direct method and the weighted method, demonstrating that the points of each level of variables and the probabilities corresponding to the total points were almost the same.While some slight differences were observed. The calibration curve for the competing risk model was close to the equivalence line and the 95%confidence interval of the observed frequency, fell into the equivalence line in each group. Indicating the accurate calibration ability of the model.Calibration curves using internal and external validation were shown, indicating that the constructed model had a good calibration ability in the internal validation;but a poor one in the external validation. The results of the decision curve analysis of the competing risk nomogram were obtained. Demonstrating the changes in net benefit with increasing threshold probability.Re-sampling the original dataset was replaced to generate the bootstrap dataset as important, in performing the internal validation of the competing risk nomogram. Besides the bootstrap method, the randomized display and K4 method can also be performed to generate data sets used for internal validation. Using our R based landscape validation of the competing risk model, clinicians can perform a prognosis analysis in real world considering the competing risk more easily.

Research

JoVE Journal

Cancer Research

Isolation of Circulating Tumor Cells in an Orthotopic Mouse Model of Colorectal Cancer

Despite the advantages of easy applicability and cost-effectiveness, subcutaneous mouse models have severe limitations and do not accurately simulate ...

An In Vivo Murine Sciatic Nerve Model of Perineural Invasion

An In Vivo Murine Sciatic Nerve Model of Perineural Invasion

Cancer cells invade nerves through a process termed perineural invasion (PNI), in which cancer cells proliferate and migrate in the nerve ...

Immunophenotyping of Orthotopic Homograft (Syngeneic) of Murine Primary KPC Pancreatic Ductal Adenocarcinoma by Flow Cytometry

Immunophenotyping of Orthotopic Homograft Syngeneic of Murine Primary KPC Pancreatic Ductal Adenocarcinoma by Flow Cytometry

Homograft (syngeneic) tumors are the workhorse of today&#39;s immuno-oncology (I/O) preclinical research. The tumor microenvironment (TME), particularly ...

Morphology-Based Distinction Between Healthy and Pathological Cells Utilizing Fourier Transforms and Self-Organizing Maps

The appearance and the movements of immune cells are driven by their environment. As a reaction to a pathogen invasion, the immune cells are recruited to ...

Evaluation of Colorectal Cancer Risk and Prevalence by Stool DNA Integrity Detection

Nowadays, stool DNA can be isolated and analyzed by several methods. The long fragments of DNA in stool can be detected by a qPCR assay, which provides a ...

Spatial and Temporal Control of Murine Melanoma Initiation from Mutant Melanocyte Stem Cells

Cutaneous melanoma is well known as the most aggressive skin cancer. Although the risk factors and major genetic alterations continue to be documented ...

In Vitro Assay to Study Tumor-macrophage Interaction

Tumor associated macrophages (TAMs) account for a large percentage of cells in the tumor mass for different types of cancers. Glioblastoma (GBM), a ...

Establishing a Competing Risk Regression Nomogram Model for Survival Data

The Kaplan&#8211;Meier method and Cox proportional hazards regression model are the most common analyses in the survival framework. These are relatively ...

An Orthotopic Resectional Mouse Model of Pancreatic Cancer

There is a lack of satisfactory animal models to study adjuvant and/or neoadjuvant therapy in patients being considered for surgery of pancreatic cancer ...