This work was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB13040400) and the startup grant from Jilin University. Anonymous reviewers and biomedical testing users were appreciated for their constructive comments on improving the usability and functionality of kSolutionVis.

NOTE: The following protocol describes the details of the informatics analytic procedure and pseudo-codes of the major modules. The automatic analysis system was developed using Python version 3.6.0 and the Python modules pandas, abc, numpy, scipy, sklearn, sys, PyQt5, sys, mRMR, math and matplotlib. The materials used in this study are listed in the Table of Materials.
1. Prepare the Data Matrix and Class Labels
<ol>
	<li>Prepare the data matrix file as a TAB- or comma-deli...

We have no conflicts of interest related to this report.

This study presents an easy-to-follow multi-solution biomarker detection and characterization protocol for a user-specified binary classification dataset. The software puts an emphasis on user-friendliness and flexible import/export interfaces for various file formats, allowing a biomedical researcher to investigate their dataset easily using the GUI of the software. This study also highlights the necessity of generating more than one solution with similarly effective modeling performances, previously ignored by many exi...

Binary classification, one of the most commonly investigated and challenging data mining problems in the biomedical area, is used to build a classification model trained on two groups of samples with the most accurate discrimination power1,2,3,4,5,6,7. However, the big data generated in the biomedical field has the inherent &#34;large p small n&#34; paradigm, with the number of features usually much larger than the number of samples6,8,9. Therefore, biomedical researchers have to reduce the feature dimension before utilizing the classification algorithms to avoid the overfitting problem8,9. Diagnosis biomarkers are defined as a subset of detected features separating patients of a given disease from healthy control samples10,11. Patients are usually defined as the positive samples, and the healthy controls are defined as the negative samples12.
Recent studies have suggested that there exists more than one solution with identical or similarly effective classification performances for a biomedical dataset5. Almost all the feature selection algorithms are deterministic algorithms, producing only one solution for the same dataset. Genetic algorithms may simultaneously generate multiple solutions with similar performances, but they still try to select one solution with the best fitness function as the output for a given dataset13,14.
Feature selection algorithms can be roughly grouped as either filters or wrappers12. A filter algorithm chooses the top-k features ranked by their significant individual association with the binary class labels based on the assumption that features are independent of each other15,16,17. Although this assumption does not hold true for almost all real-world datasets, the heuristic filter rule performs well in many cases, for instance, the mRMR (Minimum Redundancy and Maximum Relevance) algorithm, the Wilcoxon test based feature filtering (WRank) algorithm, and the ROC (Receiver operating characteristic) plot based filtering (ROCRank) algorithm. mRMR, is an efficient filter algorithm because it approximates the combinatorial estimation problem with a series of much smaller problems, comparing to the maximum-dependency feature selection algorithm, each of which only involves two variables, and therefore uses pairwise joint probabilities which are more robust18,19. However, mRMR may underestimate the usefulness of some features as it does not measure the interactions between features which can increase relevancy, and thus misses some feature combinations that are individually useless but are useful only when combined. The WRank algorithm calculates a non-parametric score of how discriminative a feature is between two classes of samples, and is known for its robustness for outliers20,21. Furthermore, the ROCRank algorithm evaluates how significant the Area Under the ROC Curve (AUC) of a particular feature is for the investigated binary classification performance22,23.
On the other hand, a wrapper evaluates the pre-defined classifier&#39;s performance of a given feature subset, iteratively generated by a heuristic rule, and creates the feature subset with the best performance measurement24. A wrapper generally outperforms a filter in the classification performance but runs slower25. For example, the Regularized Random Forest (RRF)26,27 algorithm uses a greedy rule, by evaluating the features on a subset of the training data at each random forest node, whose feature importance scores are evaluated by the Gini index. The choice of a new feature will be penalized if its information gain does not improve that of the chosen features. Additionally, the Prediction Analysis for Microarrays (PAM)28,29 algorithm, also a wrapper algorithm, calculates a centroid for each of the class labels, and then selects features to shrink the gene centroids toward the overall class centroid. PAM is robust for outlying features.
Multiple solutions with the top classification performance may be necessary for any given dataset. Firstly, the optimization goal of a deterministic algorithm is defined by a mathematical formula, e.g., minimum error rate30, which is not necessarily ideal for biological samples. Secondly, a dataset may have multiple, significantly different, solutions with similar effective or even identical performances. Almost all existing feature selection algorithms will randomly select one of these solutions as the output31.
This study will introduce an informatics analytic protocol for generating multiple feature selection solutions with similar performances for any given binary classification dataset. Considering that most biomedical researchers are not familiar with informatic techniques or computer coding, a user-friendly graphical user interface (GUI) was developed to facilitate the rapid analysis of biomedical binary classification datasets. The analytic protocol consists of data loading and summarizing, parameter tuning, pipeline execution, and result interpretations. With a simple click, the researcher is able to generate the biomarker subsets and publication-quality visualization plots. The protocol has been tested using the transcriptomes of two binary classification datasets of Acute Lymphoblastic Leukemia (ALL), i.e., ALL1 and ALL212. The datasets of ALL1 and ALL2 were downloaded from the Broad Institute Genome Data Analysis Center, available at http://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi.&#160;ALL1 contains 128 samples with 12,625 features. Of these samples, 95 are B-cell ALL and 33 are T-cell ALL. ALL2 includes 100 samples with 12,625 features as well. Of these samples, there are 65 patients that suffered relapse and 35 patients that did not. ALL1 was an easy binary classification dataset, with a minimum accuracy of four filters and four wrappers being 96.7%, and 6 of the 8 feature selection algorithms achieving 100%12. While ALL2 was a more difficult dataset, with the above 8 feature selection algorithms achieving no better than 83.7% accuracy12. This best accuracy was achieved with 56 features detected by the wrapper algorithm, Correlation-based Feature Selection (CFS).

<table border="0" cellpadding="0" cellspacing="0" style="width:669px;" width="669">
	<tbody>
		<tr height="22">
			<td height="22" style="height:22px;width:81px;">Hardware</td>
			<td style="width:88px;"></td>
			<td style="width:137px;"></td>
			<td style="width:363px;"></td>
		</tr>
		<tr height="64">
			<td height="64" style="height:64px;width:81px;">laptop</td>
			<td style="width:88px;">Lenovo</td>
			<td style="width:137px;">X1 carbon</td>
			<td style="width:363px;">Any computer works. Recommended minimum configuration: 1GB extra hard disk space, 1 GB memory, 2.0MHz CPU</td>
		</tr>
		<tr height="22">
			<td height="22" style="height:22px;width:81px;">Name</td>
			<td style="width:88px;">Company</td>
			<td style="width:137px;">Catalog Number</td>
			<td style="width:363px;">Comments</td>
		</tr>
		<tr height="22">
			<td height="22" style="height:22px;width:81px;">Software</td>
			<td style="width:88px;"></td>
			<td style="width:137px;"></td>
			<td style="width:363px;"></td>
		</tr>
		<tr height="43">
			<td height="43" style="height:43px;width:81px;">Python 3.0</td>
			<td style="width:88px;">WingWare</td>
			<td style="width:137px;">Wing Personal</td>
			<td style="width:363px;">Any python programming and running environments support Python version 3.0 or above</td>
		</tr>
	</tbody>
</table>

selecting multiple biomarker subsets with similarly effective binary classification performances

Biomarker detection is one of the more important biomedical questions for high-throughput &#39;omics&#39; researchers, and almost all existing biomarker detection algorithms generate one biomarker subset with the optimized performance measurement for a given dataset. However, a recent study demonstrated the existence of multiple biomarker subsets with similarly effective or even identical classification performances. This protocol presents a simple and straightforward methodology for detecting biomarker subsets with binary classification performances, better than a user-defined cutoff. The protocol consists of data preparation and loading, baseline information summarization, parameter tuning, biomarker screening, result visualization and interpretation, biomarker gene annotations, and result and visualization exportation at publication quality. The proposed biomarker screening strategy is intuitive and demonstrates a general rule for developing biomarker detection algorithms. A user-friendly graphical user interface (GUI) was developed using the programming language Python, allowing biomedical researchers to have direct access to their results. The source code and manual of kSolutionVis can be downloaded from http://www.healthinformaticslab.org/supp/resources.php.

The goal of this workflow (Figure 6) is to detect multiple biomarker subsets with similar efficiencies for a binary classification dataset. The whole process is illustrated by two example datasets ALL1 and ALL2 extracted from a recently-published biomarker detection study12,48. A user may install kSolutionVis by following the instructions in the supplementary materials.
<p class="jove_content" fo:...

Watch this Scientific Journal Video about Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances at JoVE.com

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances

Existing algorithms generate one solution for a biomarker detection dataset. This protocol demonstrates the existence of multiple similarly effective solutions and presents a user-friendly software to help biomedical researchers investigate their datasets for the proposed challenge. Computer scientists may also provide this feature in their biomarker detection algorithms.

Biomarker detection is one of the more important biomedical questions for high-throughput &#39;omics&#39; researchers, and almost all existing biomarker ...

This measure can help answer key questions in the biomedical detection field about generating multiple solutions. The main advantage of this technique is that it provides a user-friendly graphical user interface for assisting biomedical researchers in the detection of multiple feature subsides. Begin by loading the data Matrix and class labels into the software.Click load data matrix to select the user specified data metrics file and load class labels to select the corresponding class label file. To determine the class labels in the number of top-ranked features, select the names of the positive and negative classes in the appropriate drop down boxes and select 10 as the number of top-ranked features in the top X drop down box for a comprehensive screen of the feature subset. To tune the system parameters for different performances, select the performance measurement accuracy as the accuracy balanced accuracy drop down box for the selected extreme learning machine classifier.Then, select a cut-off value of 0.7 for the specified performance measurement in the performance cutoff input box. To run the pipeline, click analyze and select 0.7 as the default value of the performance measurement cut off. And, 10 as the default number of the best feature subsets.Then collect and interpret the features detected by the software. To generate a 3D scatter plot of the top 10 features of the subsets with the best classification performances detected by the software, click analyze and sort the three features in a feature subset in ascending order of their ranks, using the ranks of the three features as the F1, F2, and F3 axes. Change the performance cut off value to 0.7 and click analyze to generate a 3D scatter plot of the feature subsets with a greater than or equal to performance cut off performance measurement value.Then click 3D tuning to open a new window for manual tuning of the viewing angles of the 3D scatter plot and reduce to reduce the redundancy of the detected feature subsets. To annotate a gene in both the DNA and protein sequence levels, open the David database web page and click on the gene ID conversion link to input the feature IDs of the first biomarker subset of the prepared data set. Click the gene list link and click submit list to retrieve the annotations of interest, and show Gene list to obtain the list of Gene symbols.Next, open the GeneCards database web page and enter the name of the gene of interest into the database query input box to find the annotations of this gene. Open the Online Mendelian Inheritance in Man database and search for the gene to find the annotations of this gene from the database. To annotate the encoded proteins, open the UniProt knowledge base database page and search for the annotations of the gene from this database.Open the group based prediction system, or GPS web server, and retrieve the protein sequence encoded by the biomarker gene from the UniProt knowledge base database and use the online GPS tool to predict the proteins post-transitional modification residues. To annotate the protein-protein interactions and there enriched functional modules, open the string web server page and use the string database to search the lift for the genes of interest to find their orchestrated properties. To export the detected biomarker subsets for further analysis, click export the table and select the appropriate text format for saving the files.Then, export the visualization plots as individual image files, clicking save under each plot and selecting the appropriate image format for saving each file. In this representative experiment, two data sets were formatted as CSV files and loaded into the software as demonstrated. In the first data set, 128 samples with 12, 625 features and individual class labels were loaded with the final data Matrix containing 95 negative samples and 33 positive samples.Similar operations were also conducted for the second difficult data set. Searching for a user-specific keyword in the feature names reveals a histogram of the features for each data set. After executing the pipeline algorithm for each data set, 120 qualified biomarker subsets were detected for the easy to discriminate data set, with 57 triplet biomarker subsets demonstrating a 100%accuracy.Only 76 biomarker subsets where detected for the difficult data set, however. And, with a lower biomarkers subset accuracy suggesting that biomarkers are phenotype specific, another major challenge in biomarker detection. While using this procedure, it's important to remember that a future selection problem has multiple solutions.Read the SIM best of performance. After its development, this technique paved the way for biomedical researchers to explore biomedical detection with multiple solutions.

selecting-multiple-biomarker-subsets-with-similarly-effective-binary

Research

JoVE Journal

Cancer Research

7.4K vistas.  Jilin University. Esta medida puede ayudar a responder preguntas clave en el campo de la detección biomédica sobre la generación de múltiples soluciones. La principal ventaja de esta técnica es que proporciona una interfaz gráfica de usuario fácil de usar para ayudar a los investigadores biomédicos en la detección de múltiples características disminuye. Comience cargando la matriz de datos y las etiquetas de clase en el software.Haga clic en cargar matriz de datos para seleccionar el archivo ...