Searching Data with blkbox

Training & Testing

This involves using a singular portion of the dataset to generate the model and testing or validating that models assumptions on a singular holdout portion of the dataset. Therefore this requires the user to partition the data into two sets, with each data.frame having a corresponding labels vector as described prior.

Data can be partitioned manually or using the Partition function, which can be supplied to blkbox using via the data parameter.

The last step before running the model is to decide on which models you will be running on the data, blkbox currently includes eight algorithms by default. Depending on the size of the dataset it may not be feasible to run all eight algorithms. The neural network algorithm package nnet and the bayesian regression trees package bartMachine can be quite slow or memory intensive if the dataset is large (features > 5,000) and therefore should be used after aggressive feature reduction (e.g. within blkboxNCV).

The function for generating a Training & Testing model is blkbox

# Partitioning Data
my_partition = Partition(data = my_data,
                         labels = my_labels)

# Creating a Training & Testing Model
model_1 <- blkbox(data = my_partition)

Cross-fold Validation

Cross-fold Validation is often useful when the data being used has a limited number of samples or one wants to determine the robustness of a set of features for prediction. The process will seperate the data into k folds, where each fold is the same size (or as close to the same size as possible). Splitting of the data into folds is with respect to samples, where each fold will have the same features.

Models are then generated using k - 1 folds, leaving the remaining holdout fold to assess the model, this process is repeated k times until each fold has been used as a holdout.

The performance of each model can be determined by the effectiveness in which they predicted the classification of the holdout, this can be left as seperate values over holdouts or averaged to assess the overall robustness.

Cross-fold Validation requires no partitioning beforehand and k = 10 by default.

# Creating a Cross-fold Validation Model
model_2 <- blkboxCV(data = my_data,
                    labels = my_labels)

Nested Cross-fold Validation

Nested Cross-fold Validation is an extension upon Cross-fold Validation which can be useful to avoid overfitting when sample space may be particularly low. The process begins the same way as Cross-fold Validation, k fold are created and each will rotate as the holdout.

However Nested will then take the k - 1 folds and execute a complete round of Cross-fold Validation, the contribution of each feature to the model on average is determined. Features are then selected based upon an AUC cutoff, this is defined cumulatively, therfore sum the importance of the top ranking features until that value reaches the total importance of all features multiplied by the AUC cutoff.

This will allow the most important features to be retained whilst removing those will little contribution. blkboxNCV must therefore specify a Method and AUC parameter to determine which algorithm will be used to determine the importance of features internally and which cutoff is to be used. More depth on this will be covered in the worked examples.

Once the features are selected a model is generated from the data before Cross-fold Validation was run but only using those features deemed important and then testing it on the original holdout (reduced to the same features as the model).

This process is then repeated for each holdout, the main difference between Nested and standard Cross-fold Validation is that the features used in each holdout can vary and therefore each holdout cannot be expected to contain the same features. This can be useful for detecting reoccurent features and their robustness, moreso than with Cross-fold Validation.

# Creating a Nested Cross-fold Validation Model
model_3 <- blkboxNCV(data = my_data,
                     labels = my_labels,
                     Method = "randomforest",
                     AUC = 0.9)

Because feature importance must be determined within the process the Performance function is intergrated automatically within blkboxNCV and defaults to all available metrics.

ROC Curves

Receiver operating characteristic (ROC) curves are a measure of true positive and false positive rates, the area under this curve is often used as a metric for evaluating model performance. blkbox offers the blkboxROC function that uses the pROC package to calculate the curve and then feeds that to ggplot2 for an aesthetic overhaul.

blkboxROC works on all model types (blkbox, blkboxCV, blkboxNCV), however the representation of each may vary.

For Training and Testing as well as Cross-fold Validation the ROC curve is generated after calculating Performance.

# Calculate Performance
perf = Performance(model_1)
# Standard ROC curve
blkboxROC(perf)

If repeatitions were run with blkboxCV for the models then the plot becomes faceted by repeat number. Similiarly when running blkboxNCV the plot can be generated for holdouts individually or combined, the former is faceted whilst the later is a singular ROC curve.

# Standard ROC curve for Cross-fold Validation with 2 repeats
model_2r <- blkboxCV(data = my_data,
                    labels = my_labels,
                    repeats = 2)
perf_2r = Performance(model_2r)
blkboxROC(perf_2r)

# Alternvatively to avoid Faceting 
# perf_2r = Performance(model_2r, consensus = F)
# blkboxROC(perf_2r)

# Standard ROC curve for Cross-fold Validation

    # Need to adjust blkboxROC to accept NCV input (data is already in output)

Performance Measures

blkbox currently supports five different performance metrics; error rate, area under the receiver operating characteristic curve, Matthews correlation coefficient and F-score.

Metrics can be applied using the Performance function which will allow one more of the metrics to be specified. These can then be passed onto functions which will graphically represent the result. The results can also be merged and averaged over repititions…

cv.plot …

# Example placeholder cv.plot

ncv.plot …

# Example placeholder ncv.plot

Venn Diagrams

When comparing multiple algorithms it can be insightful to visualise the overlap between them, therefore cv.venn and ncv.venn allow comparison of what features were found to be important in each algorithm

cv.venn …

# Example placeholder

ncv.venn …

# Example placeholder

rJava & bartMachine

blkbox utilises rJava for the models created with bartMachine, therefore the memory allowed for use by java proccess must be set with options(java.parameters = 'Xmx4g') with the desired memory. blkbox will raise the memory to four gigabytes by default, however this is not “locked in” until bartMachine is called for modelling and therefore can be adjusted after calling library(blkbox).

library(blkbox)
# Allowed for 16 gigabytes of memory for rJava processes
options(java.parameters = 'Xmx16g')