# Compliance Documentation

#### Peter Hurford

#### 2020-02-21

Compliance documentation is a premium add-on product to DataRobot. It allows users to automatically generate and download documentation to assist with deploying models in highly regulated industries.

## Connect to DataRobot

To explore compliance documentation, let’s first connect to DataRobot. First, you must load the DataRobot R package library.

If you have set up a credentials file, `library(datarobot)` will initialize a connection to DataRobot automatically. Otherwise, you can specify your `endpoint` and `apiToken` as in this example to connect to DataRobot directly. For more information on connecting to DataRobot, see the “Introduction to DataRobot” vignette.

    library(datarobot)
    endpoint <- "https://<YOUR DATAROBOT URL GOES HERE>/api/v2"
    apiToken <- "<YOUR API TOKEN GOES HERE>"
    ConnectToDataRobot(endpoint = endpoint, token = apiToken)

## Download Compliance Documentation

To download compliance documentation for a particular model, call `DownloadComplianceDocumentation` on a particular model and specify a filepath to download the documentation to. Note that it downloads in DOCX format.

    DownloadComplianceDocumentation(model, "path/to/filename.docx")

## Creating a Custom Template

You can also use your own custom compliance documentation templates.

#### The Default Template

First, let’s get the default template. This can be done just by using `GetComplianceDocTemplate`. It downloads as a JSON file.

    GetComplianceDocTemplate("path/to/filename.json")

#### Updating the Default Template

A common workflow for building your own template is downloading the default template and modifying it.

    DownloadComplianceDocTemplate("path/to/filename.json")
    # ...then modify the compliance doc template in your favorite editor.
    UploadComplianceDocTemplate(name = "myNewTemplate", filename = "path/to/modified_file.json")

Alternatively, you can construct a template via a list:

    sections <- list(list("title" = "Missing Values Report",
                          "highlightedText" = "NOTICE",
                          "regularText" = "This dataset had a lot of Missing Values. See the chart below: {{missingValues}}",
                          "type" = "user"),
                     list("title" = "Blueprints",
                          "regularText" = "{{blueprintDiagram}} /n Blueprint for this model",
                          "type" = "user"))
    UploadComplianceDocTemplate(name = "myNewTemplateFromSections", sections = sections)

You can then get and download your template:

    myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplateFromSections")[[1]]
    DownloadComplianceDocTemplate(myTemplate)

#### Creating Custom Compliance Documentation from Custom Template

Once you have a custom template made, you can use it to create custom compliance documentation:

    myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplate")[[1]]
    CreateComplianceDocumentation(model, myTemplate)

#### Keyword Tags

Custom keyword tags are supported for templates, embedding DataRobot generated content into the template. Each tag looks like {{ keyword }} and on generation will be replaced with corresponding content. Below you can find a table of currently supported tags:

    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | Tag                            | Type           | Content                                              | Web Application UI Analog                                      |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ blueprint_diagram }}        | Image          | Graphical representation of the modeling pipeline.   | Leaderboard >> Model >> Describe >> Blueprint                  |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ alternative_models }}       | Table          | Comparison of the model with alternatives            | Leaderboard                                                    |
    |                                |                | built in the same project.                           |                                                                |
    |                                |                | Also known as challenger models.                     |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ model_features }}           | Table          | Description of the model features                    | Data >> Project Data                                           |
    |                                |                | and corresponding EDA statistics.                    |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ missing_values }}           | Table          | Description of the missing values and their          | Leaderboard >> Model >> Describe >> Missing Values             |
    |                                |                | processing in the model.                             |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ partitioning }}             | Image          | Graphical representation of the data partitioning.   | Data >> Show Advanced Options >> Partitioning                  |
    |                                |                |                                                      | (only available before project start)                          |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ model_scores }}             | Table          | Metric scores of the model on different data sources | Leaderboard >> Model                                           |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ lift_chart }}               | Image          | Lift chart.                                          | Leaderboard >> Model >> Evaluate >> Lift Chart                 |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ feature_impact }}           | Image          | Feature Impact chart.                                | Leaderboard >> Model >> Understand >> Feature Impact           |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ feature_impact_table }}     | Table          | Table representation of Feature Impact data.         | Leaderboard >> Model >> Understand >> Feature Impact >> Export |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ feature_effects }}          | List of images | Feature Effects charts for the top 3 features.       | Leaderboard >> Model >> Understand >> Feature Effects          |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ accuracy_over_time }}       | Image          | Accuracy over time chart.                            | Leaderboard >> Model >> Evaluate >> Accuracy Over Time         |
    |                                |                | Available only for datetime partitioned projects.    |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ cv_scores }}                | Table          | Project metric scores for each fold.                 | Currently unavailable in the UI                                |
    |                                |                | Available only for projects with cross validation.   |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ roc_curve }}                |                | ROC Curve.                                           | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
    |                                | Image          | Available only for binary classification projects.   |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ confusion_matrix_summary }} | Table          | Confusion matrix summary for the threshold with      | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
    |                                |                | maximal F1 score value (default suggestion in UI).   |                                                                |
    |                                |                | Available only for binary classification projects.   |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
    | {{ prediction_distribution }}  | Image          | Prediction distribution.                             | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
    |                                |                | Available only for binary classification projects.   |                                                                |
    +--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+