SRBCT sPLS-DA Case Study, but with permuted class labels
This case study will use the exact same methodology as the sPLS-DA SRBCT Case Study. However, prior to model construction, the class labels (srbct$class) will be randomly permuted. The proportions of each class will be maintained, but the instances they are associated with will be different. This is done to exemplify a case where the distinction between the provided classes is minimal and/or the data are quite noisy.
sPLS-DA is fairly ineffective when looking at classes defined by linear or non-linear relationships. It is effective when the classes cluster according to a set of “signal” features, even in the presence of large quantities of noise attributes [1].
Rather than undergoing the entire tuning process again, the optimal values yielded from the sPLS-DA SRBCT Case Study will be used here – they can be seen directly below. Using these values, the sPLS-DA model can be constructed.
optimal.ncomp
## [1] 3
optimal.keepX
## [1] 9 260 30
Variable Stability
sPLS-DA is ultimately a feature selection tool – such that variables that best separate the classes will be extracted. Hence, asssessing the stability (frequency of feature selection over cross validated folds) can give an indicator of the performance of said feature selection.
Below, the number of features with a stability of above 0.4 and the mean stability value of all selected features can be seen. For all three components, both these values were lower in the permuted case compared to the normal case. As this was over 5 folds and 10 repeats, this difference can be assumed to be significant. By permuting the labels, the ability of the model to locate signal features is significantly hindered. More noisy features are interpreted as useful by the model, resulting in lower average stabilities of selected features.
## [1] "Normal component 1" ## - Features with high stability: 11 ## - Mean stability: 0.1323529 ## [1] "Normal component 2" ## - Features with high stability: 205 ## - Mean stability: 0.2105263 ## [1] "Normal component 3" ## - Features with high stability: 25 ## - Mean stability: 0.1648352 ## [1] "Permutted component 1" ## - Features with high stability: 1 ## - Mean stability: 0.05357143 ## [1] "Permutted component 2" ## - Features with high stability: 86 ## - Mean stability: 0.1356286 ## [1] "Permutted component 3" ## - Features with high stability: 3 ## - Mean stability: 0.05217391
Plots
Sample Plots
First, observe the sample plots found with the sPLS-DA SRBCT Case Study – seen in Figure 1. The components are useful discriminators between classes, for instance the first component is best suited to defining the BL class from the others. Overall, the classes separate quite well (save for the NB and RMS classes on the second component).
FIGURE 1: Sample plots from sPLS-DA performed on the SRBCT gene expression data including 95% confidence ellipses. Samples are projected into the space spanned by the first three components. (a) Components 1 and 2 and (b) Components 1 and 3. Samples are coloured by their tumour subtypes.
The equivalent plots using the permuted labels can be seen in Figure 2. There is a stark decrease in the quality of separation of classes. Large degrees of overlap are present across all three components. The lack of distinction between the classes is a key indicator of a failure of the sPLS-DA methodology to produce useful components.
FIGURE 2: Sample plots from sPLS-DA performed on the SRBCT gene expression data after class label permutation. Samples are projected into the space spanned by the first three components. (a) Components 1 and 2 and (b) Components 1 and 3.
Variable Plots
Next, the correlation circle plots will be evaluated. The features of the permuted data seemingly have a lower average, absolute correlation with the sPLS-DA components. The clusters of features are also more dispersed. Separation of these features along the first component is severely reduced.
In combination, Figures 1 and 3 provide useful insights into what features are associated with specific class labels (ie. features positively correlated with the first component are likely key features in defining the BL class). The same inferences cannot be made using the permuted form of this data (Figures 2 and 4).
FIGURE 3: Correlation circle plot representing the genes selected by sPLS-DA performed on the SRBCT gene expression data. Gene names are truncated to the first 10 characters. Only the genes selected by sPLS-DA are shown in components 1 and 2.
FIGURE 4: Correlation circle plot representing the genes selected by sPLS-DA performed on the SRBCT gene expression data after class label permutation. Gene names are truncated to the first 10 characters. Only the genes selected by sPLS-DA are shown in components 1 and 2.
Prediction Performance
Cross validated error rate is usually the best way to evaluate the performance of a sPLS-DA model [1]. It is the key indicator of when it is appropriate to use this type of model. Other metrics such as precision, recall and the F1 score are also useful indicators, but are secondary to that of the error rate.
Across 5 folds and 100 repeats, these four performance metrics are shown for each of the classes.
## [1] "Normal model performance metrics" ## EWS BL NB RMS ## error 0.04166667 0.0160000 0.0125000 0.02716667 ## precision 0.96181825 0.8184000 0.9071667 0.94714286 ## recall 0.92200238 0.8410000 0.9311333 0.94888810 ## f1 0.93345597 0.8216952 0.9125063 0.94156841 ## [1] "Permuted model performance metrics" ## EWS NB RMS BL ## error 0.4510000 0.3578333 0.5036667 0.22616667 ## precision 0.3755810 0.1480381 0.1972437 0.02963333 ## recall 0.3695238 0.2020333 0.1849929 0.02700000 ## f1 0.3385008 0.1477177 0.1683331 0.02454286
It is clear that in the permuted case, all these metrics worsen (ie. error increases; precision, recall and F1 decrease). In the normal case, there is a set of signal features, each of which are involved in defining one or more classes from the remaining classes. By permuting the labels, the samples which allowed the model to determine which features were associated with defining a specific class are no longer part of that class. This mimics data that has few signal features, representing the case where sPLS-DA becomes an ineffective classifier and feature selection tool.