# Catching errors

#### Phil Chalmers

#### 2020-01-20

Error catching is an important area to consider when creating Monte Carlo simulations. Sometimes, iterative algorithms will ‘fail to converge’, or otherwise crash for other reasons (e.g., sparse data). However, `SimDesign` makes this process much easier because the internal functions are automatically wrapped within `try` blocks, and therefore simulations will not terminate unexpectedly. This type of information is also collected in the final simulation object since it may be relevant to the writer that something unknown is going wrong in the code-base. Below we demonstrate what happens when errors are thrown and caught, and how this information is tracked in the returned object.

# 1 Define the functions

As usual, define the functions of interest.

    library(SimDesign)
    # SimFunctions(comments=FALSE)

    Design <- createDesign(N = c(10,20,30))

    Generate <- function(condition, fixed_objects = NULL) {
        ret <- with(condition, rnorm(N))
        ret
    }

    Analyse <- function(condition, dat, fixed_objects = NULL) {
        whc <- sample(c(0,1,2,3), 1, prob = c(.7, .20, .05, .05))
        if(whc == 0){
           ret <- mean(dat)
        } else if(whc == 1){
            ret <- t.test() # missing arguments
        } else if(whc == 2){
            ret <- t.test('invalid') # invalid arguments
        } else if(whc == 3){
            # throw error manually
            stop('Manual error thrown')
        }
        # manual warnings
        if(sample(c(TRUE, FALSE), 1, prob = c(.1, .9)))
            warning('This warning happens rarely')
        if(sample(c(TRUE, FALSE), 1, prob = c(.5, .5)))
            warning('This warning happens much more often')
        ret
    }

    Summarise <- function(condition, results, fixed_objects = NULL) {
        ret <- c(bias = bias(results, 0))
        ret
    }

The above simulation is just an example of how errors are tracked in `SimDesign`, as well as how to throw a manual error in case the data should be re-drawn based on the user’s decision (e.g., when a model converges, but fails to do so before some number of predefined iterations).

# 2 Run the simulation

    result <- runSimulation(Design, replications = 100,
                           generate=Generate, analyse=Analyse, summarise=Summarise)

    ##
    ##
    Design row: 1/3;   Started: Mon Jan 20 18:42:54 2020;   Total elapsed time: 0.00s
    ##
    ##
    Design row: 2/3;   Started: Mon Jan 20 18:42:54 2020;   Total elapsed time: 0.09s
    ##
    ##
    Design row: 3/3;   Started: Mon Jan 20 18:42:54 2020;   Total elapsed time: 0.17s

    print(result)

    ## # A tibble: 3 x 8
    ##       N   bias REPLICATIONS SIM_TIME COMPLETED              SEED ERRORS WARNINGS
    ##   <dbl>  <dbl>        <int>    <dbl> <chr>                 <int>  <int>    <int>
    ## 1    10 0.0611          100   0.0930 Mon Jan 20 18:42:…   1.14e9     53       59
    ## 2    20 0.0143          100   0.0790 Mon Jan 20 18:42:…   3.13e8     52       60
    ## 3    30 0.0179          100   0.066  Mon Jan 20 18:42:…   8.66e8     42       56

What you’ll immediately notice from this output object is that counts of the error and warning messages have been appended to the `result` object. This is useful to determine just how problematic the errors and warnings are based on their frequency alone. Furthermore, the specific frequency in which the errors/warnings occurred are also included for each design condition (here the `t.test.default()` error, where no inputs were supplied, occurred more often than the manually thrown error as well as the invalid-input error) after extracting and inspecting `SimExtract(results, what = 'errors')` and `SimExtract(results, what = 'warnings')`.

    SimExtract(result, what = 'errors')

    ##    N ERROR: .Error in t.test.default("invalid") : not enough 'x' observations\n
    ## 1 10                                                                         12
    ## 2 20                                                                          9
    ## 3 30                                                                         10
    ##   ERROR: .Error in t.test.default() : argument "x" is missing, with no default\n
    ## 1                                                                             31
    ## 2                                                                             38
    ## 3                                                                             25
    ##   ERROR: .Manual error thrown\n
    ## 1                            10
    ## 2                             5
    ## 3                             7

Finally, `SimDesign` has a built-in safety feature controlled by with `max_errors` argument to avoid getting stuck in infinite redrawing loops. By default, if more than 50 errors are consecutively returned then the simulation condition will be halted, and a warnning message will be printed to the console indicating the last observed fatal error. These safety features are built-in because too many consecutive `stop()` calls generally indicates a major problem in the simulation code which should be fixed before continuing. However, when encountering fatal errors in a given simulation condition the remainer of the simulation experiment will still be executed as normal, where for the problematic conditions combinations `NA` placeholders will be assigned to these rows in the final output object. This is so that the entire experiment does not unexpectedly terminate due to one or more problmematic row connditions in `Design`, and instead these conidtions can be inspected and debugged at a later time. Of course, if inspecting the code directly, the simulation could be manually halted so that these terminal errors can be attended to immediately (e.g., using `Ctrl + c`, or clicking the ‘Stop’ icon in Rstudio).

# 3 What to do (explicit debug catch)

If errors occur too often (but not in a fatal way) then the respective design conditions should either be extracted out of the simulation or further inspected to determine if they can be fixed (e.g., providing better starting values, increasing convergence criteria/number of iterations, etc). The use of the debugging features can also be useful to track down issues as well. For example, manually wrap the problematic functions in a `try()` call, and add the line `if(is(object, 'try-error')) browser()` to jump into the location/replication where the object unexpectedly witnessed an error. Jumping into the exact location where the error occurred will greatly help you determine what exactly went wrong in the simulation state, allowing you to quickly locate and fix the issue.

# 4 What to do (stored error seed debuging)

An alternative approach to locating errors in general is to use information stored within the `SimDesign` objects at the time of completion. By default, all `.Random.seed` states associated with errors are stored within the final object, and these can be extracted using the `SimExtract(..., what='error_seeds')` option. This function returns a `data.frame` object with each seed stored column-wise, where the associated error message is contained in the column name itself (and allowed to be coerced into a valid column name to make it easier to use the `$` operator). For example,

    seeds <- SimExtract(result, what = 'error_seeds')
    head(seeds[,1:3])

    ##   Design_row_1.1..Error.in.t.test.default.....argument..x..is.missing..with.no.default.
    ## 1                                                                                 10403
    ## 2                                                                                   624
    ## 3                                                                            -159038368
    ## 4                                                                            1905303777
    ## 5                                                                            -371375826
    ## 6                                                                           -1012234281
    ##   Design_row_1.2..Manual.error.thrown.
    ## 1                                10403
    ## 2                                   21
    ## 3                           -230613222
    ## 4                            203707493
    ## 5                           1161141503
    ## 6                            549195142
    ##   Design_row_1.3..Error.in.t.test.default.....argument..x..is.missing..with.no.default.
    ## 1                                                                                 10403
    ## 2                                                                                    65
    ## 3                                                                            -230613222
    ## 4                                                                             203707493
    ## 5                                                                            1161141503
    ## 6                                                                             549195142

Given these seeds, replicating an exact error can be achieved by a) extracting a single column into an `integer` vector, and b) passing this vector to the `load_seed` input. For example, replicating the first error message can be achieved as follows, where it makes the most sense to immediately go into the debugging mode via the `debug` inputs.

Note: It is important to manually select the correct `Design` row using this error extraction approach; otherwise, the seed will clearly not replicate the exact problem state.

    picked_seed <- seeds$Design_row_1.1..Error.in.t.test.default..invalid.....not.enough..x..observations.
    runSimulation(Design[1,], replications = 100, load_seed=picked_seed, debug='analyse',
                  generate=Generate, analyse=Analyse, summarise=Summarise)

The `.Random.seed` state will be loaded at this exact state, and will always be related at this state as well (in case `c` is typed in the debugger, or somehow the error is harder to find while walking through the debug mode). Hence, users must type `Q` to exit the debugger after they have better understood the nature of the error message first-hand.