# Operators

#### John Mount, Nina Zumel

#### 2019-04-20

As of `cdata` version `1.0.8` `cdata` implements an operator notation for data transform.

The idea is simple, yet powerful. In fact it helps us find new names for the transforms: multiply to block records and factor to row records. These terms will become clear as we work examples.

First let’s start with some data.

    d <- wrapr::build_frame(
       "model_id"  , "measure", "value" |
         1         , "AUC"    , 0.7     |
         1         , "R2"     , 0.4     |
         2         , "AUC"    , 0.8     |
         2         , "R2"     , 0.5     )

    knitr::kable(d)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

In the above data we have two measurements each for two individuals (individuals identified by the “`model_id`” column). Using `cdata`’s `rowrecs_to_blocks_spec()` method we can capture a description of this record structure and transformation details.

    library("cdata")

    transform <- rowrecs_to_blocks_spec(
      wrapr::qchar_frame(
        "measure", "value" |
        "AUC"    , AUC     |
        "R2"     , R2      ),
      recordKeys = "model_id")

    print(transform)
    #> {
    #>  row_record <- wrapr::qchar_frame(
    #>    "model_id"  , "AUC", "R2" |
    #>      .         , AUC  , R2   )
    #>  row_keys <- c('model_id')
    #>
    #>  # becomes
    #>
    #>  block_record <- wrapr::qchar_frame(
    #>    "model_id"  , "measure", "value" |
    #>      .         , "AUC"    , AUC     |
    #>      .         , "R2"     , R2      )
    #>  block_keys <- c('model_id', 'measure')
    #>
    #>  # args: c(checkNames = TRUE, checkKeys = TRUE, strict = FALSE)
    #> }

Once we have this specification we can transform the data using operator notation.

We can collect the record blocks into rows by a “factor-out” (or aggregation/projection) step.

    knitr::kable(d)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

    d2 <- d %//% t(transform)

    knitr::kable(d2)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

We can expand record rows into blocks by a “multiplication” (or join) step.

    knitr::kable(d2)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

    d3 <- d2 %**% transform

    knitr::kable(d3)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

(`%//%` and `%**%` being two operators introduced by the `cdata` package.)

And the two operators have an inverse/adjoint relation.

    knitr::kable(d)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

    # identity
    d4 <- d %//% t(transform) %**% transform

    knitr::kable(d4)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

We can also pipe into the spec (and into its adjoint) using the `wrapr` `dot pipe operator`.

    # reverse or adjoint/transpose operation specification
    t_record_spec <- t(transform)

    d %.>%
      t_record_spec %.>%
      knitr::kable(.)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

    # using dot-pipe's bquote style .() execute immediate notation
    d %.>%
      .(t(transform)) %.>%
      knitr::kable(.)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

    # identity
    d %.>%
      .(t(transform)) %.>%
      transform %.>%
      knitr::kable(.)

model_id | measure | value
--- | --- | ---
1 | AUC | 0.7
1 | R2 | 0.4
2 | AUC | 0.8
2 | R2 | 0.5

And, of course, the exact same functionality for database tables.

    have_db <- requireNamespace("DBI", quietly = TRUE) &&
       requireNamespace("RSQLite", quietly = TRUE)

    raw_connection <- DBI::dbConnect(RSQLite::SQLite(),
                                     ":memory:")
    RSQLite::initExtension(raw_connection)
    db <- rquery::rquery_db_info(
      connection = raw_connection,
      is_dbi = TRUE,
      connection_options = rquery::rq_connection_tests(raw_connection))

    d_td <- rquery::rq_copy_to(db, "d", d)

    ops <- d_td %//% t(transform)
    cat(format(ops))
    #> table(`d`;
    #>   model_id,
    #>   measure,
    #>   value) %.>%
    #>  non_sql_node(., blocks_to_rowrecs(.))

    rquery::execute(db, ops) %.>%
      knitr::kable(.)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

    d_td %.>%
      .(t(transform)) %.>%
      rquery::execute(db, .) %.>%
      knitr::kable(.)

model_id | AUC | R2
--- | --- | ---
1 | 0.7 | 0.4
2 | 0.8 | 0.5

    DBI::dbDisconnect(raw_connection)

And that is truly fluid data manipulation.