# Understanding lime

#### Thomas Lin Pedersen & Michaël Benesty

#### 2018-11-21

In order to be able to understand the explanations produced by `lime` it is necessary to have at least some knowledge of how these explanations are achieved. To this end, you are encouraged to read through the article that introduced the lime framework as well as the additional resources linked to from the original Python repository. This vignette will provide an overview to allow you to get up to speed on the framework and let you efficiently understand the output it produces.

## How lime explains stuff

Behind the workings of lime lies the (big) assumption that every complex model is linear on a local scale. While this is not justified in the paper it is not difficult to convince yourself that this is generally sound — you usually expect two very similar observations to behave predictably even in a complex model. `lime` then takes this assumption to its natural conclusion by asserting that it is possible to fit a simple model around a single observation that will mimic how the global model behaves at that locality. The simple model can then be used to explain the predictions of the more complex model locally.

The general approach `lime` takes to achieving this goal is as follows:

1. For each prediction to explain, permute the observation `n` times.
2. Let the complex model predict the outcome of all permuted observations.
3. Calculate the distance from all permutations to the original observation.
4. Convert the distance to a similarity score.
5. Select `m` features best describing the complex model outcome from the permuted data.
6. Fit a simple model to the permuted data, explaining the complex model outcome with the `m` features from the permuted data weighted by its similarity to the original observation.
7. Extract the feature weights from the simple model and use these as explanations for the complex models local behavior.

It is clear from the above that there’s much wiggle-room in order to optimize the explanation. Chief among the choices that influence the quality of the explanation is how permutations are created, how permutation similarity is calculated, how, and how many, features are selected, and which model is used as the simple model. Some of these choices are hard-coded into `lime`, while others can be influenced by the user — all of them will be discussed below.

### How to permute an observation

When it comes to permuting an observation, `lime` depends on the type of input data. Currently two types of inputs are supported: tabular and text

Tabular Data
When dealing with tabular data, the permutations are dependent on the training set. During the creation of the explainer the statistics for each variable are extracted and permutations are then sampled from the variable distributions. This means that permutations are in fact independent from the explained variable making the similarity computation even more important as this is the only thing establishing the locality of the analysis.

Text Data
When the outcome of text predictions are to be explained the permutations are performed by randomly removing words from the original observation. Depending on whether the model uses word location or not, words occurring multiple times will be removed one-by-one or as a whole.

### Calculating similarities with permutations

Just as permutations are created differently based on the input data, so the similarities are calculated in different ways. For text data the cosine similarity measure is used, which is the standard in text analysis (it effectively measures the angle difference between the two feature vectors). For tabular data a bit more thought is required and the optimal solution will depend on the type of input data. First, the categorical features will be recoded based on whether or not they are equal to the observation. Second, if continuous features are binned (the default) these features will be recoded based on whether they are in the same bin as the observation. Using the recoded data the distance to the original observation is then calculated based on a user-chosen distance measure (euclidean by default), and converted to a similarity using an exponential kernel of a user defined width (defaults to 0.75 times the square root of the number of features).

### Selecting the features to use

Feature selection is a complete sub-field of modelling in itself and `lime` has no silver bullet for this. Instead, it implements a range of different feature selection approaches that the user is free to choose from. First though, the number of features needs to be chosen. The number must strike a balance between the complexity of the model and the simplicity of the explanation, but try to keep it below 10 (personal opinion). As for selecting the features, `lime` supports the following algorithms:

- none: Use all features for the explanation. Not advised unless you have very few features.
- forward selection: Features are added one by one based on their improvements to a ridge regression fit of the complex model outcome.
- highest weights: The `m` features with highest absolute weight in a ridge regression fit of the complex model outcome are chosen.
- lasso: The `m` features that are least prone to shrinkage based on the regularization path of a lasso fit of the complex model outcome is chosen.
- tree: A tree is fitted with `log2(m)` splits, to use at max `m` features. It may possibly select less.
- auto: Uses forward selection if `m <= 6` and otherwise highest weights.

### Fitting a model to the permuted and feature-reduced data

Once permutations have been created, similarities calculated and features selected it is time to fit a model. If the complex model is a regressor, the simple model will predict the output of the complex model directly. If the complex model is a classifier, the simple model will predict the probability of the chosen class (for classifiers it is possible to either specify the classes to explain, or let `lime` chose the top `k` most probable classes).

The only requirement for the simple model is that it can work with weighted input and that it is easy to extract understanding from the resulting fit. While multiple types of models support this, `lime` uses a ridge regression as the model of choice. In the future this might be expanded to other types of models.

## An example - Tabular Data

The following is a simple example which seeks to explain the outcome of a model predicting cancer based on biopsy results:

    library(MASS)
    library(lime)
    data(biopsy)

    # First we'll clean up the data a bit
    biopsy$ID <- NULL
    biopsy <- na.omit(biopsy)
    names(biopsy) <- c('clump thickness', 'uniformity of cell size',
                       'uniformity of cell shape', 'marginal adhesion',
                       'single epithelial cell size', 'bare nuclei',
                       'bland chromatin', 'normal nucleoli', 'mitoses',
                       'class')

    # Now we'll fit a linear discriminant model on all but 4 cases
    set.seed(4)
    test_set <- sample(seq_len(nrow(biopsy)), 4)
    prediction <- biopsy$class
    biopsy$class <- NULL
    model <- lda(biopsy[-test_set, ], prediction[-test_set])

If we use the model to predict the 4 remaining cases we get some pretty solid predictions:

    predict(model, biopsy[test_set, ])

    ## $class
    ## [1] benign    benign    malignant benign
    ## Levels: benign malignant
    ##
    ## $posterior
    ##           benign    malignant
    ## 416 9.943635e-01 5.636486e-03
    ## 7   9.527375e-01 4.726245e-02
    ## 207 1.456491e-05 9.999854e-01
    ## 195 9.999977e-01 2.300862e-06
    ##
    ## $x
    ##            LD1
    ## 416 -0.2070616
    ## 7    0.2382330
    ## 207  3.1409787
    ## 195 -1.8101622

But lets see how these predictions came to be, using lime.

    explainer <- lime(biopsy[-test_set,], model, bin_continuous = TRUE, quantile_bins = FALSE)
    explanation <- explain(biopsy[test_set, ], explainer, n_labels = 1, n_features = 4)
    # Only showing part of output for better printing
    explanation[, 2:9]

    ## # A tibble: 16 x 8
    ##    case  label label_prob model_r2 model_intercept model_prediction feature
    ##    <chr> <chr>      <dbl>    <dbl>           <dbl>            <dbl> <chr>
    ##  1 416   beni…      0.994    0.543           0.130            1.01  margin…
    ##  2 416   beni…      0.994    0.543           0.130            1.01  bare n…
    ##  3 416   beni…      0.994    0.543           0.130            1.01  clump …
    ##  4 416   beni…      0.994    0.543           0.130            1.01  unifor…
    ##  5 7     beni…      0.953    0.659           0.678            0.343 mitoses
    ##  6 7     beni…      0.953    0.659           0.678            0.343 bare n…
    ##  7 7     beni…      0.953    0.659           0.678            0.343 clump …
    ##  8 7     beni…      0.953    0.659           0.678            0.343 unifor…
    ##  9 207   mali…      1.000    0.157           0.313            0.698 bare n…
    ## 10 207   mali…      1.000    0.157           0.313            0.698 clump …
    ## 11 207   mali…      1.000    0.157           0.313            0.698 unifor…
    ## 12 207   mali…      1.000    0.157           0.313            0.698 bland …
    ## 13 195   beni…      1.000    0.538           0.113            1.04  mitoses
    ## 14 195   beni…      1.000    0.538           0.113            1.04  bare n…
    ## 15 195   beni…      1.000    0.538           0.113            1.04  clump …
    ## 16 195   beni…      1.000    0.538           0.113            1.04  unifor…
    ## # ... with 1 more variable: feature_value <int>

We can see that all the explanations for a benign outcome have chosen the same features, strongly indicating that these are both locally and globally important features. In order to get a more intuitive representation, we can use the provided `plot_features()` to get a visual overview of the explanations.

    plot_features(explanation, ncol = 1)

In this overview it is clear to see how case 195 and 416 behave alike, while the third benign case (7) has an unusual large bare nuclei which are detracting from its status as benign without affecting the final prediction (indicating that the values of its other features are making up for this odd one). To no surprise it is clear that high values in the measurements are indicative of a malignant tumor.

## An example - text data

The following is a simple example which seeks to explain the outcome of a model classifying sentences from 30 scientific papers as being about (or not) the author’s own work, e.g. methods, results or conclusions.

Most of the things written for `data.frame` can be applied to textual data.

    library(lime)
    library(xgboost) # the classifier
    library(text2vec) # used to build the BoW matrix

    # load data
    data(train_sentences)
    data(test_sentences)

    # Data are stored in a 2 columns data.frame, one for the sentences, one for the
    # labels.
    print(str(train_sentences))

    ## 'data.frame':    2517 obs. of  2 variables:
    ##  $ class.text: chr  "MISC" "MISC" "AIMX" "OWNX" ...
    ##  $ text      : chr  "although the internet as level topology has been extensively studied over the past few years  little is known a"| __truncated__ "an as  node  can represent a wide variety of organizations  e g   large isp  or small private business  univers"| __truncated__ "in this paper  we introduce a radically new approach based on machine learning techniques to map all the ases i"| __truncated__ "we believe that this dataset will serve as an invaluable addition to further understanding of the structure and"| __truncated__ ...
    ## NULL

    # The list of possible classes for a sentence
    # We are only interested in the class "OWNX"
    print(unique(train_sentences$class.text))

    ##  [1] "MISC"                "AIMX"                "OWNX"
    ##  [4] "BASE"                "CONT"                "MISC--the"
    ##  [7] "AIMX--on"            "OWNX--after"         "MISC--in"
    ## [10] "MISC--specifically," "MISC--on"            "MISC--several"
    ## [13] "CONT--these"         "OWNX--we"            "MISC--for"

    # Tokenize data
    get_matrix <- function(text) {
      it <- itoken(text, progressbar = FALSE)
      create_dtm(it, vectorizer = hash_vectorizer())
    }

    # BoW matrix generation
    dtm_train = get_matrix(train_sentences$text)
    dtm_test = get_matrix(test_sentences$text)

    # Create boosting model for binary classification (-> logistic loss)
    # Other parameters are quite standard
    param <- list(max_depth = 7,
                  eta = 0.1,
                  objective = "binary:logistic",
                  eval_metric = "error",
                  nthread = 1)

    xgb_model <- xgb.train(
      param,
      xgb.DMatrix(dtm_train, label = train_sentences$class.text == "OWNX"),
      nrounds = 50
    )

We will test our model on test data.

    # We use a (standard) threshold of 0.5
    predictions <- predict(xgb_model, dtm_test) > 0.5
    test_labels <- test_sentences$class.text == "OWNX"

    # Accuracy
    print(mean(predictions == test_labels))

    ## [1] 0.8433333

Now we are sure that the model works, we may want to understand what are the most important words for the predictions.

    # We select 10 sentences from the label OWNX
    sentence_to_explain <- head(test_sentences[test_labels,]$text, 5)
    explainer <- lime(sentence_to_explain, model = xgb_model,
                      preprocess = get_matrix)
    explanation <- explain(sentence_to_explain, explainer, n_labels = 1,
                           n_features = 2)

    # Most of the words choosen by Lime
    # are related to the team (we, our)
    # or part of the paper (Section, in)
    explanation[, 2:9]

    ## # A tibble: 10 x 8
    ##     case label label_prob model_r2 model_intercept model_prediction feature
    ##    <int> <chr>      <dbl>    <dbl>           <dbl>            <dbl> <chr>
    ##  1     1 1          0.642    0.988           0.332            0.644 in
    ##  2     1 1          0.642    0.988           0.332            0.644 we
    ##  3     2 1          0.802    0.846           0.322            0.735 in
    ##  4     2 1          0.802    0.846           0.322            0.735 We
    ##  5     3 0          0.543    0.874           0.694            0.496 will
    ##  6     3 0          0.543    0.874           0.694            0.496 we
    ##  7     4 1          0.872    0.671           0.487            0.840 in
    ##  8     4 1          0.872    0.671           0.487            0.840 Section
    ##  9     5 1          0.732    0.345           0.355            0.667 of
    ## 10     5 1          0.732    0.345           0.355            0.667 our
    ## # ... with 1 more variable: feature_value <chr>

    # Another more graphical view of the same information (2 first sentences only)
    plot_features(explanation)

The graph view gives lots of detailed information about the selected words. Another approach to explain a text classification model is to see the most important words in their context.

    plot_text_explanations(explanation)

For some sentences, the prediction is wrong. For instance, look at the third sentence. The word we is in red and the predicted label is 0 (with a low probability), meaning the model thinks it’s not a sentence from the category `OWNX`, mainly because of the presence of the word those but the word we in red doesn’t support this (false) prediction.

## Interactive text model explanations

Text explanations are also available through `shine` to explore interactively the model.

After loading the application, both text and parameters can be modified. Prediction and selected words are updated in real time.

    # Launching the application is done in one command
    interactive_text_explanations(explainer)

Interactive text explanations

## Session Info

    ## ─ Session info ──────────────────────────────────────────────────────────
    ##  setting  value
    ##  version  R version 3.5.1 (2018-07-02)
    ##  os       macOS  10.14.1
    ##  system   x86_64, darwin15.6.0
    ##  ui       X11
    ##  language (EN)
    ##  collate  C
    ##  ctype    en_US.UTF-8
    ##  tz       Europe/Copenhagen
    ##  date     2018-11-21
    ##
    ## ─ Packages ──────────────────────────────────────────────────────────────
    ##  package        * version    date       lib
    ##  assertthat       0.2.0      2017-04-11 [2]
    ##  backports        1.1.2      2017-12-13 [2]
    ##  base64enc        0.1-3      2015-07-28 [2]
    ##  bindr            0.1.1      2018-03-13 [2]
    ##  bindrcpp         0.2.2      2018-03-29 [2]
    ##  callr            3.0.0      2018-08-24 [2]
    ##  cli              1.0.1      2018-09-25 [2]
    ##  codetools        0.2-15     2016-10-05 [2]
    ##  colorspace       1.3-2      2016-12-14 [2]
    ##  crayon           1.3.4      2017-09-16 [2]
    ##  data.table       1.11.8     2018-09-30 [2]
    ##  desc             1.2.0      2018-05-01 [2]
    ##  devtools         2.0.1      2018-10-26 [2]
    ##  digest           0.6.18     2018-10-10 [2]
    ##  dplyr            0.7.8      2018-11-10 [2]
    ##  evaluate         0.12       2018-10-09 [2]
    ##  fansi            0.4.0      2018-10-05 [2]
    ##  foreach          1.4.4      2017-12-12 [2]
    ##  formatR          1.5        2017-04-25 [2]
    ##  fs               1.2.6      2018-08-23 [2]
    ##  futile.logger    1.4.3      2016-07-10 [2]
    ##  futile.options   1.0.1      2018-04-20 [2]
    ##  ggplot2          3.1.0.9000 2018-11-19 [2]
    ##  glmnet           2.0-16     2018-04-02 [2]
    ##  glue             1.3.0      2018-07-17 [2]
    ##  gower            0.1.2      2017-02-23 [2]
    ##  gtable           0.2.0.9000 2018-11-19 [2]
    ##  htmltools        0.3.6      2017-04-28 [2]
    ##  htmlwidgets      1.3        2018-09-30 [2]
    ##  httpuv           1.4.5      2018-07-19 [2]
    ##  iterators        1.0.10     2018-07-13 [2]
    ##  jsonlite         1.5        2017-06-01 [2]
    ##  knitr            1.20.19    2018-10-11 [2]
    ##  labeling         0.3        2014-08-23 [2]
    ##  lambda.r         1.2.3      2018-05-17 [2]
    ##  later            0.7.5      2018-09-18 [2]
    ##  lattice          0.20-38    2018-11-04 [2]
    ##  lazyeval         0.2.1      2017-10-29 [2]
    ##  lime           * 0.4.1      2018-11-21 [1]
    ##  magrittr         1.5        2014-11-22 [2]
    ##  MASS           * 7.3-51.1   2018-11-01 [2]
    ##  Matrix           1.2-15     2018-11-01 [2]
    ##  memoise          1.1.0      2017-04-21 [2]
    ##  mime             0.6        2018-10-05 [2]
    ##  mlapi            0.1.0      2017-12-17 [2]
    ##  munsell          0.5.0      2018-06-12 [2]
    ##  pillar           1.3.0      2018-07-14 [2]
    ##  pkgbuild         1.0.2      2018-10-16 [2]
    ##  pkgconfig        2.0.2      2018-08-16 [2]
    ##  pkgload          1.0.2      2018-10-29 [2]
    ##  plyr             1.8.4      2016-06-08 [2]
    ##  prettyunits      1.0.2      2015-07-13 [2]
    ##  processx         3.2.0      2018-08-16 [2]
    ##  promises         1.0.1      2018-04-13 [2]
    ##  ps               1.2.1      2018-11-06 [2]
    ##  purrr            0.2.5      2018-05-29 [2]
    ##  R6               2.3.0      2018-10-04 [2]
    ##  Rcpp             1.0.0      2018-11-07 [2]
    ##  RcppParallel     4.4.1      2018-07-19 [2]
    ##  remotes          2.0.2      2018-10-30 [2]
    ##  rlang            0.3.0.1    2018-10-25 [2]
    ##  rmarkdown        1.10       2018-06-11 [2]
    ##  rprojroot        1.3-2      2018-01-03 [2]
    ##  scales           1.0.0      2018-08-09 [2]
    ##  sessioninfo      1.1.1      2018-11-05 [2]
    ##  shiny            1.2.0      2018-11-02 [2]
    ##  shinythemes      1.1.2      2018-11-06 [2]
    ##  stringi          1.2.4      2018-07-20 [2]
    ##  stringr          1.3.1      2018-05-10 [2]
    ##  testthat         2.0.1      2018-10-13 [2]
    ##  text2vec       * 0.5.1      2018-01-11 [2]
    ##  tibble           1.4.2      2018-01-22 [2]
    ##  tidyselect       0.2.5      2018-10-11 [2]
    ##  usethis          1.4.0      2018-08-14 [2]
    ##  utf8             1.1.4      2018-05-24 [2]
    ##  withr            2.1.2      2018-03-15 [2]
    ##  xfun             0.4        2018-10-23 [2]
    ##  xgboost        * 0.71.2     2018-06-09 [2]
    ##  xtable           1.8-3      2018-08-29 [2]
    ##  yaml             2.2.0      2018-07-25 [2]
    ##  source
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.1)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.1)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.1)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  Github (tidyverse/ggplot2@828e948)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  Github (r-lib/gtable@cdbf7c0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  Github (yihui/knitr@3abb642)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  local
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##  CRAN (R 3.5.0)
    ##
    ## [1] /private/var/folders/wq/20rfm4212kx8fdtc4q_bvfd00000gn/T/Rtmp49yxvA/Rinst1869a579ad47f
    ## [2] /Library/Frameworks/R.framework/Versions/3.5/Resources/library
    ## [3] /private/var/folders/wq/20rfm4212kx8fdtc4q_bvfd00000gn/T/RtmpJcXshk/temp_libpath1868274a67aad