---
title: "Working with SO files in R with libsoc"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Working with SO files in R with libsoc}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

## Reading an SO file
It is easy to read a Standard Output file into R.

```{r read_so}
filename <- system.file("extdata", "pheno.SO.xml",  package="libsoc")
so <- libsoc::so_SO_read(filename)
```

## Navigating the SO structure
The SO R object closely follows the tree structure of the SO XML. We can extract the population parameter estimates like this:

```{r estimates}
param <- so$SOBlock[[1]]$Estimation$PopulationEstimates$MLE
param
```

The result is a data.frame with the population estimates and the names of the parameters as column names. Note the way we go deeper in the SO tree with successive use of the $-opterator. As an SO can contain more than one SOBlock the SOBlock-structure is a list. In the example we simply select the first (and only) SOBlock.

libsoc also has a special function for extracting the parameter estimates across all SOBlocks

```{r parameter_estimates}
so$all_population_estimates()
```

## Getting information on parameters
If the SO is linked to a PharmML (for example if it was generated with "nmoutput2so -generate_pharmml") parameter information can be retrieved from the ParameterModel.

```{r parameter_types}
cols <- colnames(param)
so$variability_type(cols)
```

It is also possible to get information on which parameters represent correlations.
```{r correlations}
so$correlation_parameters(cols)
```

This example has no correlation parameters.


The names of the random variables corresponding to the variability parameters can be retrieved:
```{r random_variables}
so$random_variable_from_variability_parameter(cols)
```


## Obtaining some parameter uncertainty measures

The covariance matrix is of standard R matrix type:

```{r covariance_matrix}
so$SOBlock[[1]]$Estimation$PrecisionPopulationEstimates$MLE$CovarianceMatrix
```

The standard error for the parameters can be extracted for each SOBlock separately:
```{r standard_error}
so$SOBlock[[1]]$Estimation$PrecisionPopulationEstimates$MLE$StandardError
```

Or more conveniently for all SOBlocks together:
```{r standard_error2}
so$all_standard_errors()
```

Relative standard errors can currently only be retrieved for each SOBlock separately:
```{r relative_standard_error}
so$SOBlock[[1]]$Estimation$PrecisionPopulationEstimates$MLE$RelativeStandardError
```

## Getting the OFV

```{r OFV}
so$SOBlock[[1]]$Estimation$OFMeasures$Deviance
```

## Getting the Predictions
```{r predictions}
pred <- so$SOBlock[[1]]$Estimation$Predictions
head(pred, 20)
```

## Getting information on table columns
Each table in the SO structure has a columnType attribute that gives an array of columnTypes taken directly from the xml.

```{r columnType}
attributes(pred)$columnType
```

The portable way of finding the column name/number for id, idv and dv columns is to use the provided functions.

```{r id_column}
libsoc::id_column(pred)
```

```{r id_column_name}
libsoc::id_column_name(pred)
```

```{r idv_column}
libsoc::idv_column(pred)
```

```{r idv_column_name}
libsoc::idv_column_name(pred)
```

## Getting the Residuals
```{r residuals}
res <- so$SOBlock[[1]]$Estimation$Residuals$ResidualTable
head(res, 20)
```

## Reading messages

Messages from a run is stored in the TaskInformation structure. Each message contains a Name a Severity from 1 to 10, the message content and the name of the tool that emitted the message.
```{r messages}
messages <- so$SOBlock[[1]]$TaskInformation$Message
messages[[1]]$Name
messages[[1]]$Content
messages[[1]]$Severity
messages[[1]]$Toolname
messages[[13]]$Content
```
## iOFV
OFV values for each individual separately is in the OFMeasures section

```{r iofv}
iofv <- so$SOBlock[[1]]$Estimation$OFMeasures$IndividualContribToLL
head(iofv)
```