Introduction

Background

One key task in environmental science is obtaining information of environmental variables in space or in space and time, usually based on limited field data. In that respect, machine learning algorithms have been proven to be an important tool to learn patterns in nonlinear and complex systems. However, standard machine learning applications are not suitable for spatio-temporal data, as they usually ignore the spatio-temporal dependencies in the data. This becomes problematic in (at least) two aspects of predictive modelling: Overfitting during model setup as well as objective error assessment (see Meyer et al 2018). To approach these problems, CAST supports the well-known caret package (Kuhn 2018 to allow accounting for spatio-temporal dependencies in prediction tasks.

This tutorial shows how to set up a spatio-temporal prediction model that includes objective and reliable error estimation. It further shows how spatio-temporal overfitting can be detected by comparison between validation strategies. It will be shown that certain variables are responsible for the problem of overfitting due to spatio-temporal autocorrelation patterns. Therefore, this tutorial also shows how to automatically exclude variables that lead to overfitting with the aim to improve the spatio-temporal prediction model.

In order to follow this tutorial, I assume that the reader is familiar with the basics of predictive modelling nicely explained in Kuhn and Johnson 2013 as well as machine learning applications via the caret package.

How to start

To work with the tutorial, first install the CAST package and load the library:

#install.packages("CAST")
library(CAST)

If you need help, see

help(CAST)

Example of a typical spatio-temporal prediction task

The example prediction task for this tutorial is the following: We have a set of data loggers distributed over a farm, and we want to map soil moisture, based on a set of spatial and temporal predictor variables. We will use Random Forests as a machine learning algorithm in this tutorial.

Description of the example dataset

To do so, we will work with the cookfarm dataset, described in e.g. Gasch et al 2015 and available via the GSIF package (Hengl 2017). The dataset included in CAST is a re-structured dataset which was used for the analysis in Meyer et al 2018.

data <- get(load(system.file("extdata","Cookfarm.RData",package="CAST")))
head(data)
##        SOURCEID    VW  Easting Northing altitude      DEM      TWI
## 92743    CAF316 0.441 494076.9  5180950     -0.3 793.7094 5.068904
## 204055   CAF316 0.495 494076.9  5180950     -0.6 793.7094 5.068904
## 315367   CAF316 0.464 494076.9  5180950     -0.9 793.7094 5.068904
## 426679   CAF316 0.505 494076.9  5180950     -1.2 793.7094 5.068904
## 537991   CAF316 0.462 494076.9  5180950     -1.5 793.7094 5.068904
## 92744    CAF316 0.440 494076.9  5180950     -0.3 793.7094 5.068904
##           NDRE.M   NDRE.Sd     Bt  BLD  PHI Crop       Date Precip_wrcc
## 92743  0.2185383 0.3310146 0.0000 1.43 5.75   WW 2007-04-20           0
## 204055 0.2185383 0.3310146 0.0000 1.53 6.28   WW 2007-04-20           0
## 315367 0.2185383 0.3310146 0.0701 1.62 6.49   WW 2007-04-20           0
## 426679 0.2185383 0.3310146 0.4730 1.68 6.72   WW 2007-04-20           0
## 537991 0.2185383 0.3310146 0.7250 1.66 6.74   WW 2007-04-20           0
## 92744  0.2185383 0.3310146 0.0000 1.43 5.75   WW 2007-04-21           0
##        MaxT_wrcc MinT_wrcc Precip_cum  cday     cdayt
## 92743       11.7      -1.1          0 13624 1.0000000
## 204055      11.7      -1.1          0 13624 1.0000000
## 315367      11.7      -1.1          0 13624 1.0000000
## 426679      11.7      -1.1          0 13624 1.0000000
## 537991      11.7      -1.1          0 13624 1.0000000
## 92744       13.9       3.3          0 13625 0.9998477

I want to point out on the following information of this dataset: The “SOURCEID” represents the ID for the data logger, “VW” is soil moisture which is our response variable, “Easting” and “Northing” are the coordinates of the data loggers, “altitude” indicates the depth of the soil in which VW was measured, and the remaining columns represent different potential predictor variables which are terrain related (e.g. “DEM”, “TWI”), vegetation indices (e.g. “NDRE”), soil properties (e.g. “BLD”) or climate-related predictors (e.g. “Precip_wrcc”). See Gasch et al 2015 for further description on the dataset.

To get an impression on the spatial properties of the dataset, Let’s have a look on the spatial distribution of the data loggers on the cookfarm:

library(mapview)
library(sp)
data_sp <- unique(data[,c("SOURCEID","Easting","Northing")])
coordinates(data_sp) <- ~Easting+Northing
proj4string(data_sp) <- CRS("+init=epsg:26911")
#plot the data:
mapviewOptions(basemaps = c("Esri.WorldImagery"))
mapview(data_sp)

We see that the data are taken at 42 locations (SOURCEID) over the field. The loggers recorded data between 2007 and 2013. The data are given here on a daily basis.

Data subsetting

To reduce the data to an amount that can be handeld in a tutorial, let’s restrict the data to the depth of -0.3 and to two weeks of the year 2012. After subsetting let’s have an overview on the soil moisture time series measured by teh data loggers.

library(lubridate)
library(ggplot2)
trainDat <- data[data$altitude==-0.3&
                   year(data$Date)==2012&
                   week(data$Date)%in%c(13:14),]
ggplot(data = trainDat, aes(x=Date, y=VW)) +
  geom_line(aes(colour=SOURCEID))

What we can see is that (as expected) each logger location has a unique time series of soil moisture.

Model training and prediction

In the following we will use this subset of the cookfarm data as an example to spatially predict soil moisture with (and without) consideration of the spatio-temporal dependencies. To start with, lets use this dataset to create a “default” Random Forest model that predicts soil moisture based on a hand full of predictor variables. To keep computation time at a minimum, we don’t include hyperparameter tuning (hence we set tuneLength to 1) which is reasonable as Random Forests are comparably insensitive to tuning.

library(caret)
predictors <- c("DEM","BLD","TWI","Precip_cum","cday",
                "MaxT_wrcc","Precip_wrcc",
                "Northing","Easting","NDRE.M")
set.seed(10)
model <- train(trainDat[,predictors],trainDat$VW,
               method="rf",tuneLength=1,importance=TRUE,
               trControl=trainControl(method="cv",number=5))

Based on the trained model we can make spatial predictions of soil moisture. To do this we load a RasterStack that contains spatial data of all predictor variables for the 25th of March 2012. We then apply the trained model on this data set.

library(raster)
predictors_sp <- stack(system.file("extdata","predictors_2012-03-25.grd",package="CAST"))
prediction <- predict(predictors_sp,model)
spplot(prediction)

The result is a spatially comprehensive map of soil moisture for this day. We see that simply creating a map using machine learning and caret is an easy task, however accurately measuring its performance is less simple. Though the map looks good on a first sight we now have to follow the question of how accurate this map is, hence we need to ask how well the model is able to map soil moisture.

From a visible inspection it is noticeable that the model produces a strange linear features at the eastern side of the farm which looks suspicious. But let’s come back to this later and first focus on a statistical validation of the model.

Cross validation strategies for spatio-temporal data

Amongst validation strategies, k-fold cross validation (CV) is popular to estimate the performance of the model in view to data that have not been used for model training. During CV, models are repeatedly trained (k models) and in each model run, the data of one fold are put to the side and are not used for model training but for model validation. In this way, the performance of the model can be estimated using data that have not been included in the model training.

The Standard approach: Random k-fold CV

In the example above we used a random k-fold CV that we defined in caret’s trainControl argument. More specifically, we used a random 5-fold CV. Hence, the data points in our dataset were RANDOMLY split into 5 folds. To assess the performance of the model let’s have a look on the output of the Random CV:

model
## Random Forest 
## 
## 490 samples
##  10 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (5 fold) 
## Summary of sample sizes: 391, 391, 392, 392, 394 
## Resampling results:
## 
##   RMSE        Rsquared  MAE        
##   0.01216135  0.970192  0.006805217
## 
## Tuning parameter 'mtry' was held constant at a value of 3

We see that soil moisture could be modelled with an R² of 0.97 which indicates a nearly perfect fit of the data. Sounds good, but unfortunately, the random k fold CV does not give us a good indication for the model performance. Random k-fold CV means that each of the five folds (with the highest certainty) contains data points from each data logger. Therefore, a random CV cannot indicate the ability of the model to make predictions beyond the location of the training data (i.e. to map soil moisture). Since our aim is to map soil moisture, we rather need to perform a target-oriented validation which validates the model in view to spatial mapping.

Target-oriented validation

We are not interested in the model performance in view to random subsets of our data loggers, but we need to know how well the model is able to make predictions for areas without data loggers. To find this out, we need to repeatedly leave the complete time series of one or more data loggers out and use them as test data during CV.

To do this we first need to create meaningful folds rather than random folds. CAST’s function “CreateSpaceTimeFolds” is designed to provide index arguments used by caret’s trainControl. The index defines which data points are used for model training during each model run and reversely defines which data points are held back. Hence, using the index argument we can account for the dependencies in the data by leaving the complete data from one or more data loggers out (LLO CV), from one or more time steps out (LTO CV) or from data loggers and time steps out (LLTO CV). In this example we’re focusing on LLO CV, therefore we use the column “SOURCEID” to define the location of a data logger and split the data into folds using this information. Analogous to the random CV we split the data into five folds, hence five model runs are performed each leaving one fifth of all data loggers out for validation.

set.seed(10)
indices <- CreateSpacetimeFolds(trainDat,spacevar = "SOURCEID",
                                k=5)
set.seed(10)
model_LLO <- train(trainDat[,predictors],trainDat$VW,
                   method="rf",tuneLength=1, importance=TRUE,
                   trControl=trainControl(method="cv",
                                          index = indices$index))
model_LLO
## Random Forest 
## 
## 490 samples
##  10 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 382, 402, 378, 392, 406 
## Resampling results:
## 
##   RMSE        Rsquared   MAE       
##   0.06456067  0.2375651  0.05046414
## 
## Tuning parameter 'mtry' was held constant at a value of 3

By inspecting the output of the model, we see that in view to new locations, the R² is only 0.24 so the performance is much lower than expected from the random CV (R² = 0.97). Apparently, there is considerable overfitting in the model, causing a good random performance but a poor performance in view to new locations. This might partly be attributed to the choice of variables where we must suspect that certain variables are misinterpreted by the model (see Meyer et al 2018).

Let’s have a look at the variable importance ranking of Random Forest and see if we find something suspicious:

plot(varImp(model_LLO))

The importance ranking indicates that amongst others, “Easting” is an important variable. This fits to the observation of an inappropriate linear features in the predicted map. Apparently the model assigns a high importance to this variable which causes a high random CV performance. But at the same time the model fails in the prediction on new locations because the variable is unsuitable for predictions beyond the locations of the data loggers used for model training.

Assuming that certain variables are misinterpreted by the algorithm we should be able to produce a higher LLO performance when such variables are removed. Let’s see if this is true…

Removing variables that cause overfitting

CAST’s forward feature selection (ffs) selects variables that make sense in view to the selected CV method and excludes those which are counterproductive in view to the selected CV method. When we use LLO as CV method, ffs selects variables that lead in combination to the highest LLO performance (i.e. the best spatial model). All variables that have no spatial meaning or are even counterproductive won’t improve or even reduce the LLO performance and are therefore excluded from the model by the ffs.

ffs is doing this job by first training models using all possible pairs of two predictor variables. The best model of these initial models is kept. On the basis of this best model the predictor variables are iteratively increased and each of the remaining variables is tested for its improvement of the currently best model. The process stops if none of the remaining variables increases the model performance when added to the current best model.

So let’s run the ffs on our case study using R² as a metric to select the optimal variables. This process will take 1-2 minutes…

set.seed(10)
ffsmodel_LLO <- ffs(trainDat[,predictors],trainDat$VW,metric="Rsquared",
                    method="rf",tuneLength=1, verbose=FALSE,
                    trControl=trainControl(method="cv",
                                           index = indices$index))
ffsmodel_LLO
## Random Forest 
## 
## 490 samples
##   4 predictor
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 382, 402, 378, 392, 406 
## Resampling results:
## 
##   RMSE        Rsquared   MAE       
##   0.05591788  0.3615628  0.04614253
## 
## Tuning parameter 'mtry' was held constant at a value of 2
ffsmodel_LLO$selectedvars
## [1] "TWI"    "NDRE.M" "BLD"    "cday"

Using the ffs with LLO CV, the R² could be increased from 0.24 to 0.36. The variables that are used for this model are “TWI”,“NDRE.M”,“BLD” and “cday”. All others are removed because they have (at least in this small example) no spatial meaning or are even counterproductive.

Using the plot\(\_\)ffs function we can visualize how the performance of the model changed depending on the variables being used:

plot_ffs(ffsmodel_LLO)

See that the best model using two variables led to an R² of slightly below 0.35. Using the third and forth variable could slightly increase the R². Any fifth variable could not improve the LLO performance. Note that the R² features a high standard deviation regardless of the variables being used. This is due to the small dataset that was used which cannot lead to robust results.

What effect does the new model has on the spatial representation of soil moisture?

prediction_ffs <- predict(predictors_sp,ffsmodel_LLO)
spplot(prediction_ffs)

We see that the variable selection does not only have an effect on the statistical performance but also the predicted spatial patterns change considerably. It is of note that the linear feature is not any more in the resulting soil moisture map most likely because “Easting” was removed from the set of predictor variables by ffs.

Conclusions

To conclude, the tutorial has shown how CAST can be used to facilitate target-oriented CV on spatial and spatio-temporal data which is crucial to obtain meaningful validation results. Using the ffs in conjunction with target-oriented validation, variables can be excluded that are counterproductive in view to the target-orineted performance due to misinterpretations by the algorithm. ffs therefore helps to select the ideal set of predictor variables for spatio-temporal prediction tasks and gives objective error estimates.

Final notes

The intention of this tutorial is to describe the motivation that led to the development of CAST as well as its functionality. Priority is not on modelling soil moisture of the cookfarm in the best possible way but to provide an example for the motivation and functionality of CAST that can run within a few minutes. Hence, only a very small subset of the entire cookfarm dataset was used. Keep in mind that due to the small subset the example is not robust and quite different results might be obtained depending on small changes in the settings.

The intention of showing the motivation of CAST is also the reason why the coordinates are used here as predictor variables. Though coordinates are used as predictors in quite some scientific studies they rather provide here an extreme example of how misleading variables can lead to overfitting.