---
title: "Vignette, Create Seasonally Detrended Flow and Salinity Data Sets"
author: "Jon.Harcum@tetratech.com and Erik.Leppo@tetratech.com"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Data Sets}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
<!-- Data is in vignettes\data folder  -->
```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Purpose
The purpose of this vignette is to cover the basics for creating seasonally detrended flow and seasonally detrended salinity data sets. These data sets are used in support of applying **gam4**. In gam4, the dependent variable is modeled with a non-linear term as a function of year; a seasonality term includes an interaction term which allows seasonality to vary over the period of record; and  includes a hydrologic term that allows for factoring wet/dry conditions in the model.

The seasonally detrended flow and salinity data sets are created using the detrended.flow and detrended.salinity functions included in baytrends. Refer to the help for each of these functions (e.g., ??detrended.flow or ??detrended.salinity) for the specific computational steps involved with each function.

## Load baytrends

The first step is to load the baytrends library. Loading baytrends will result in loading multiple additional libraries as well.  

```{r initialize,echo=TRUE}
library(baytrends)
```

## Seasonally Detrended Flow

The function, **detrended.flow**, returns a list of seasonally detrended flow and companion statistics. It relies on USGS' dataRetrieval package to retrieve daily flow data.

**It is the user's responsibility to save the list that is returned from the detrended.flow function as flow.detrended for integration with baytrends.**

For purposes of the Chesapeake Bay Program, it is expected that the user would typically identify all nine of the *River Input Monitoring* (RIM) stations. The nine RIM stations for the Chesapeake Bay have been loaded as a data frame in baytrends, **usgsGages**.

```{r usgsgages, echo=FALSE, results='asis'}
.T("Chesapeake Bay River Input Monitoring Stations",1,'e')
knitr::kable(usgsGages)
```

The below code chunk is an example of retrieving and seasonally detrending daily flow data for all nine of the RIM stations from 1983-2017. To retrieve data for other USGS gages, the variables, usgsGagesID and siteName would be assigned with vectors that contain the list of USGS gage IDs and their site names. The vector of gage IDs are used to retrieve data while the vector of siteNames is used for diagnostic plots created by the function, detrend.flow.  

To best match up with water quality data, we recommend retrieving flow data for one year prior to the first year of water quality data. This allows for creating a time-averaged flow data set and not loose the first few months of water quality data due to lack of matching flow data. Data retrievals should also be made in light of the time needed by the USGS to review and approve their flow records.  

Information about the settings for dvAvgWinSel, dvAvgWgtSel, dvAvgSidesSel, lowess.f, span, and max.fill are described in the help file for this function.    

```{r flow, eval=FALSE, results='asis', fig.height=6.5, fig.width=6.5}
# Define Function Inputs
usgsGageID    <- usgsGages$usgsGageID # all RIM stations
siteName      <- usgsGages$siteName   # all RIM stations
yearStart     <- 1983
yearEnd       <- 2017
dvAvgWinSel   <- c(1, 5, 10, 15, 20, 30, 40, 50, 60, 90, 120, 150, 180, 210)
dvAvgWgtSel   <- "uniform"
dvAvgSidesSel <- 1
lowess.f      <- 0.2

# Run detrended.flow function
flow.detrended <- detrended.flow(usgsGageID, siteName, yearStart, yearEnd
                                 , dvAvgWinSel, dvAvgWgtSel, dvAvgSidesSel
                                 , lowess.f)

# Save list to data file separate use
save(flow.detrended, file='mySeasonallyDetrendedFlow.rda')
```

### Example 

The following five figures Figures 1-5 provide an example analysis for the USGS gage on the Susquehanna River at Conowingo, Maryland. Figure 1 displays the daily flow values [cms] as a function of the day of year for data from 2003-2017. The solid red line represents the predicted GAM using the function Log(Q)~s(doy,bs='cc').  

Subtracting the predicted GAM from the observations yields the observations plotted in Figure 2. The ordinate is plotted on a log scale since the predicted GAM and observations were log transformed. We also refer to this as the 1-day-smoothed residuals. A 10-day-smoothed residual is displayed in Figure 3. Using the above settings for dvAvgWgtSel ("uniform") and dvAvgSidesSel (1), the 10-day smoothed residual is the average of the preceeding nine days and the current day of the 1-day-smoothed residual. That is, the value on June 10, 2015 is equal to the average of the values from June 1-10, 2015. As the number of days used in the averaging process increases, the range of residuals decreases. These x-day smoothed residuals are used in gam4 and can be thought of an indicator of how wet or dry a particular time period is relative to the average.

Figures 4 and 5 display summary statistics (mean, standard deviation [sd], and number of observations [Nobs.] for the 1-day and 10-day smoothed residuals. The red line on the standard deviation plot is based on a lowess smooth. The standard deviations associated with the red line are used for computing confidence intervals.

```{r flow1, eval=TRUE, echo=FALSE, results='asis', fig.height=4.5, fig.width=6.5}

# Define Function Inputs
usgsGageID    <- c("01578310")
siteName      <- c("Susquehanna River at Conowingo, MD")
yearStart     <- 2003 #1983
yearEnd       <- 2017
dvAvgWinSel   <- c(1, 10)
dvAvgWgtSel   <- "uniform"
dvAvgSidesSel <- 1
lowess.f      <- 0.2

# Run detrended.flow function
flow.detrended <- detrended.flow(usgsGageID, siteName, yearStart, yearEnd
                                 , dvAvgWinSel, dvAvgWgtSel, dvAvgSidesSel
                                 , lowess.f)
```

## Seasonally Detrended Salinity

The function, **detrended.salinity**, returns a list of seasonally detrended salinity and companion statistics. 

**It is the user's responsibility to save the list that is returned from the detrended.salinity function as salinity.detrended for integration with baytrends.**

The detrended.salinity evaluates a user-supplied data set. baytrends includes a sample salinity data set, **sal**, that corresponds to the water quality data stored in **dataCensored**. Table 2 and Table 3 display some sample data and the structure of **sal**.


```{r salinity0, echo=FALSE, results='asis'}
.T("Example Salinity Data",2,'e')
knitr::kable(head(sal[sal$layer %in% c('S', 'B') , ]),row.names = FALSE)

.T("Data Structure Associated with sal",3,'e')
```
```{r salinity1, echo=FALSE, results='markup'}
str(sal)
```

The below code chunk is an example for analyzing all of the data contained in the sample data frame, sal. As currently developed, detrended.salinity will average salinity in the surface ('S') and above pycnocline ('AP') layers to create an average salinity value for the surface and above pycnocline (SAP) layer.  detrended.salinity will also average salinity in the bottom ('B') and below pycnocline ('AP') layers to create an average salinity value for the bottom and below pycnocline (BAP) layer.  

The use and settings for dvAvgWinSel, lowess.f, minObs, and minObs.sd are described in the help file. The detrened.salinity function will assume the user wants to evaluate the salinity for all of the stations in sal and for all the years of available data.  

```{r salinity2, eval=FALSE, results='asis', fig.height=6.5, fig.width=6.5}

# Define Function Inputs
df.sal      <- sal
dvAvgWinSel <- 30
lowess.f    <- 0.2
minObs      <- 40
minObs.sd   <- 10
                 
# Run Function
salinity.detrended <- detrended.salinity(df.sal, dvAvgWinSel, 
                                 lowess.f, minObs, minObs.sd)   

# Save list to data file separate use
save(salinity.detrended, file='mySeasonallyDetrendedSalinity.rda')
```

### Example

The below figures display summary statistics (mean, standard deviation [sd], and number of observations [Nobs.] for the SAP and BBP layers, respectively. The red line on the standard deviation plot is based on a lowess smooth. The standard deviations associated with the red line are used for computing confidence intervals.

The reader is referred to the help file for detrend.salinity to learn more about the procedure used to compute these statistics and how windowing is used address the issues related to sparse sampling.  

```{r salinity3, eval=TRUE, echo=FALSE, results='asis', fig.height=4.5, fig.width=6.5}
# Define Function Inputs
df.sal      <- sal[sal$station=="CB5.4", ]
dvAvgWinSel <- 30
lowess.f    <- 0.2
minObs      <- 40
minObs.sd   <- 10
                 
# Run Function
salinity.detrended <- detrended.salinity(df.sal, dvAvgWinSel, lowess.f, minObs, minObs.sd) 
```

## Example GAM Analysis 

It is now possible to run gam4 using the results of the above analyses. The below code chunk will run the gam4 for surface total nitrogen at station CB5.4.  

```{r gamTrendExample, eval=TRUE, results='asis', fig.height=3.5, fig.width=6.5}
dfr       <- analysisOrganizeData(dataCensored, reports = NA)
df        <- dfr[["df"]]
analySpec <- dfr[["analySpec"]]

# down select analyzed GAM
analySpec$gamModels <- analySpec$gamModels[5]

stat = 'CB5.4'; dep = 'tn'; layer = 'S';
gamResult <- gamTest(df, dep, stat, layer, analySpec
                     , salinity.detrended=salinity.detrended)

```